Multifunctional vehicle with remote driving function and remote driving method

Abstract

The present invention relates to a multi-functional vehicle with a remote driving function and a remote driving method. The multi-functional vehicle is provided with a driving input device, an information processing device, wearable glasses, and a communication device. The driving input device is operated by the driver of the multi-functional vehicle. The information processing device processes the operation information of the driving input device. The wearable glasses communicate directly with other vehicles or communicate via the information processing device. The communication device communicates with other vehicles. The information processing device switches between a normal driving mode for driving the multi-functional vehicle and a remote driving mode for driving other vehicles and executes. In the normal driving mode, vehicle control of the multi-functional vehicle is executed using control information generated based on the operation information. In the remote driving mode, the control information is sent to other vehicles via the communication device. The wearable glasses display images around other vehicles during the execution of the remote driving mode based on the information processing device.

Description

Technical Field

[0001] The present invention relates to a multifunctional vehicle having an autonomous driving function and a function of remotely driving other vehicles, and a remote driving method using the multifunctional vehicle. Background Art

[0002] Japanese Patent Application Laid-Open No. 2004-295360 discloses a device for remotely driving a vehicle. This conventional device is provided in a control center that communicates with the vehicle. This conventional device further includes a control device, a display, a driver's seat, various sensors, etc. Images from a camera of the vehicle are output to the display. The driver's seat has the same structure as that of an actual vehicle driver's seat. While observing the camera images output to the display, a remote operator sitting in the driver's seat operates input devices such as a steering wheel, an accelerator pedal, and a brake pedal. Various sensors detect the amount of operation of the input devices by the remote operator. The control device calculates control amounts for driving devices such as a steering device, a driving device, and a braking device of the vehicle based on this operation amount. Summary of the Invention

[0003] In ordinary vehicles, various structures for a driver to drive the vehicle are usually provided. Thus, remote driving of other vehicles using typical structures of ordinary vehicles (e.g., a privately owned personal-owned vehicle POV) is considered. In this case, the driver of the ordinary vehicle becomes a remote operator who operates the input devices of the vehicle in order to drive a vehicle at a remote location. As a specific example, in order to transport passengers in a vehicle such as a taxi or a hire car existing at a remote location, the driver operates the input devices of his own vehicle and acts as a remote operator.

[0004] However, in order to implement a remote driving service that utilizes the typical structures of such ordinary vehicles, it is necessary to enrich the operation environment of the input devices of the vehicle on the remote driving side. In particular, in order to ensure the safety of the vehicle on the remotely driven side, it is required to sufficiently ensure the vision of the driver sitting in the vehicle on the remote driving side. In addition, it is required to be improved so that not only the vehicle on the remote driving side does not autonomously drive during the implementation of the remote driving service, but also does not transmit information that is unnecessary for pedestrians and the like around the vehicle.

[0005] One object of the present invention is to provide a technology that can implement a remote driving service that utilizes the typical structures of ordinary vehicles.

[0006] A first invention is a multifunctional vehicle having an autonomous driving function and a function of remotely driving other vehicles, and has the following characteristics.

[0007] The multi-functional vehicle includes a driving input device, an information processing device, wearable glasses, and a communication device. The driving input device is operated by the driver of the multi-functional vehicle. The information processing device processes the operation information of the driving input device operated by the driver. The wearable glasses are worn by the driver. The wearable glasses communicate with other vehicles through at least one of direct communication with the other vehicles and indirect communication via the information processing device. The communication device communicates with the other vehicles.

[0008] The information processing device switches between the normal driving mode of driving the multi-functional vehicle and the remote driving mode of driving the other vehicle and executes. In the normal driving mode, vehicle control of the multi-functional vehicle is performed using control information generated based on the operation information. In the remote driving mode, the control information is sent to the other vehicle via the communication device.

[0009] During the execution of the remote driving mode by the information processing device, the wearable glasses display the surrounding images of the other vehicle obtained through at least one of direct communication and indirect communication with the other vehicle.

[0010] The second invention further has the following features in the first invention.

[0011] The multi-functional vehicle further includes a driving device, a brake device, and a steering device.

[0012] The driving input device includes an accelerator pedal, a brake pedal, a steering wheel, and a gear shift lever.

[0013] The driving device and the accelerator pedal constitute by-wire drive. The brake device and the brake pedal constitute by-wire braking. The steering device and the steering wheel constitute by-wire steering. The gear shift lever constitutes by-wire shifting.

[0014] The third invention further has the following features in the first or second invention.

[0015] The multi-functional vehicle further includes a driving assistance device and a switch. The driving assistance device assists the driving of the multi-functional vehicle. The switch is operated by the driver of the multi-functional vehicle. The switch causes the driving assistance device to operate.

[0016] In the normal driving mode, the operation instruction information from the switch is output to the driving assistance device. In the remote driving mode, the operation instruction information is sent to the other vehicle via the communication device.

[0017] The fourth invention further has the following features in any one of the first to third inventions.

[0018] The surrounding images include the front image, the left front image, and the right front image of the other vehicle.

[0019] During the execution of the remote driving mode, the wearable glasses generate an image to be displayed on the wearable glasses corresponding to the movement of the driver's head according to the surrounding images.

[0020] The fifth invention is a method for remotely driving the other vehicle using a multifunctional vehicle having an autonomous driving function and a function of remotely driving the other vehicle, and has the following features.

[0021] The method includes:

[0022] A step of setting the driving mode of the multifunctional vehicle to a remote driving mode for driving the other vehicle;

[0023] During the period when the driving mode is set to the remote driving mode, a step of displaying the surrounding images of the other vehicle obtained through at least one of direct communication and indirect communication with the other vehicle on the wearable glasses worn by the driver of the multifunctional vehicle; and

[0024] During the period when the driving mode is set to the remote driving mode, a step of sending control information generated according to the operation information of the driving input device operated by the driver to the other vehicle.

[0025] According to the first or fifth invention, in a multifunctional vehicle in which a typical structure for realizing an autonomous driving function is added with wearable glasses, remote driving of the other vehicle is performed based on the switching of the driving mode. Therefore, remote driving of the other vehicle that utilizes the typical structure of an ordinary vehicle can be performed.

[0026] According to the second invention, since the driving input device of the multifunctional vehicle is composed of a by-wire type input device, the multifunctional vehicle can be prohibited from autonomously driving during the execution of the remote driving mode. Therefore, the safety of the multifunctional vehicle during the execution of the remote driving mode and the safety of the driver of the multifunctional vehicle as a remote operator can be ensured.

[0027] According to the third invention, the work instruction information received during the execution of the remote driving mode is sent to the other vehicle. Therefore, it is possible to prevent the driving assistance device of the multifunctional vehicle from working during the execution of the remote driving mode and transmitting unnecessary information to pedestrians and the like around the vehicle.

[0028] According to the fourth invention, head tracking control is performed using wearable glasses during the execution of the remote driving mode. Therefore, it is possible to help the driver of the multi-functional vehicle, who is the remote operator, grasp the surrounding conditions of other vehicles. Therefore, in remote driving using wearable glasses, the driving safety of other vehicles being remotely driven can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Hereinafter, the features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described with reference to the accompanying drawings, in which the same reference numerals denote the same elements, and:

[0030] Figure 1 FIG. is a diagram illustrating a first example of a remote driving service.

[0031] Figure 2 FIG. is a diagram illustrating a second example of a remote driving service.

[0032] Figure 3 FIG. is a block diagram showing a structural example of a vehicle on the receiving side of a remote driving service.

[0033] Figure 4 FIG. is a block diagram showing a first structural example of a vehicle on the providing side of a remote driving service.

[0034] Figure 5 FIG. is a diagram illustrating an example of generation of image data in head tracking control.

[0035] Figure 6 FIG. is a block diagram showing a second structural example of a vehicle on the providing side of a remote driving service.

[0036] Figure 7 FIG. is a flowchart showing an example of processing executed by an information processing device of a vehicle on the providing side of a remote driving service.

[0037] Figure 8 FIG. is a flowchart showing an example of processing executed by an information processing device of a vehicle on the providing side of a remote driving service. DETAILED DESCRIPTION OF THE INVENTION

[0038] Hereinafter, with reference to the accompanying drawings, a multi-functional vehicle according to an embodiment of the present invention and a remote driving method using the multi-functional vehicle will be described. In addition, the remote driving method according to the embodiment is implemented by computer processing in the multi-functional vehicle according to the embodiment. Further, in each drawing, the same or corresponding parts are denoted by the same reference numerals, and the description thereof is simplified or omitted.

[0039] 1. Remote Driving Service

[0040] The multi-functional vehicle according to the embodiment constitutes a part of a system that provides a remote driving service. Figure 1 FIG. is a diagram illustrating a first example of a remote driving service. InFigure 1 depicts multiple vehicles 1 (1A to 1D). Each of the vehicles 1 is a vehicle on the side providing the remote driving service. In Figure 1 also depicts multiple vehicles 2 (2A to 2D). Each of the vehicles 2 is a vehicle on the side receiving the remote driving service. The vehicles 1 and the vehicles 2 communicate via the network 3. In Figure 1 In the example shown, each of the vehicles 1 and the network 3 constitute a system for providing the remote driving service.

[0041] Each of the vehicles 1 corresponds to the "multi-functional vehicle" of the present application. Each of the vehicles 1 is, for example, a privately-owned vehicle POV. Each of the vehicles 1 may also be a vehicle owned by an operator providing the remote driving service. Each of the vehicles 1 travels by the manual operation of the driver 4 (4a to 4d) sitting in the vehicle. That is, each of the vehicles 1 has an autonomous driving function. The driving mode of the vehicle 1 during autonomous driving is set to the "normal driving mode". Each of the vehicles 1 may also have a function of assisting the manual operation of the driver 4 or an autonomous driving function. In this case, the driving of the vehicle 1 is performed by an in-vehicle driving support or autonomous driving system.

[0042] Each of the drivers 4 remotely drives the vehicle 2 according to their own will. In this case, each of the drivers 4 acts as a remote operator of the vehicle 2. As a specific example, regarding the vehicle 2, vehicles such as taxis and rental cars existing at a remote location are considered. In this case, each of the drivers 4 utilizes the typical structure of the vehicle 1 in which they are riding to remotely drive (manually operate) the vehicle 2. Thus, each of the vehicles 1 has, in addition to the autonomous driving function, a remote driving function.

[0043] Each of the vehicles 2 corresponds to the "other vehicle" of the present application. Each of the vehicles 2 is, for example, a privately-owned vehicle POV. Each of the vehicles 2 may also be a vehicle owned by an operator providing the remote driving service. Each of the vehicles 2 travels by the manual operation of the driver (not shown) sitting in the vehicle. Each of the vehicles 2 may also have a function of assisting the manual operation of the driver or an autonomous driving function. In this case, the driving of the vehicle 2 is performed by an in-vehicle driving support or autonomous driving system. Each of the vehicles 2 has a structure for receiving the manual operation by the remote operator. When the remote operator performs the manual operation, each of the vehicles 2 travels by this manual operation.

[0044] In Figure 1In the example shown, consider a case where a certain vehicle 2 desires to utilize a remote driving service. In this case, the vehicle 2 transmits communication information COM21 to each vehicle of vehicle 1. The communication information COM21 includes, for example, a utilization desire information IUS and a vehicle information IVH. The utilization desire information IUS is information for commissioning remote driving (manual operation by a remote operator). The utilization desire information IUS includes, for example, the ID information (or IP address information) of the vehicle 2, and the specification information of the vehicle 2 (for example, the specification information of the communication device, the specification information of the main equipment such as the power transmission system and the chassis, and the specification information of the driving assistance devices such as the turn signal, the windshield wiper, and the lamp). Examples of the vehicle information IVH will be described later.

[0045] In Figure 1 In the example shown, also consider a case where a certain driver 4 desires the provision of a remote driving service. In this case, the driving mode of the vehicle 1 is set to the "remote driving mode". That is, in the case of providing a remote driving service, the driving mode of the vehicle 1 is switched from the "normal driving mode" to the "remote driving mode". In the remote driving mode, the vehicle 1 automatically or according to an instruction from the driver 4 accesses the network 3 and acquires the communication information COM21. When the specification information of the vehicle 2 is consistent with the specification information of the vehicle 1, the vehicle 1 transmits the communication information COM12 to the vehicle 2. The communication information COM12 includes, for example, a provision desire information ISP and a driving instruction information IIS.

[0046] The provision desire information ISP is information for receiving an order for remote driving. The provision desire information ISP includes, for example, the ID information (or IP address information) of the vehicle 1, and the specification information of the vehicle 1 (for example, the specification information of the communication device, and the specification information of the driving assistance devices such as the turn signal, the windshield wiper, and the lamp). The driving instruction information IIS includes, for example, the control information of the vehicle 2 and the operation information of a switch for switching the operating state of the driving assistance device. The control information is information for performing the remote driving of the vehicle 2. The control information is generated based on the operation information of the vehicle 1 performed by the driver 4 who is a remote operator.

[0047] When the communication information COM12 is transmitted from the vehicle 1 to the vehicle 2, the vehicle 2 identifies the device responsible for its remote driving (in Figure 1 the example shown, the vehicle 1). After that, the vehicle 2 and the vehicle 1 exchange the communication information COM12 and COM21 via the network 3. Thus, the remote driving of the vehicle 2 by the driver 4 of the vehicle 1 is performed.

[0048] Figure 2 is a diagram illustrating a second example of a remote driving service. In Figure 2 it, one vehicle 1 and a remote facility 5 are depicted. InFigure 2 In the example shown, one vehicle 1, a network 3, and a remote facility 5 constitute a system that provides a remote driving service. The remote facility 5 includes a management server 51 and a plurality of remote driving devices 52 (52A to 52C). The remote facility 5 is, for example, a business office that provides a remote driving service. The management server 51 manages the remote driving service. Each of the remote driving devices 52 is assigned to a plurality of remote operators 6 (6a to 6c).

[0049] In Figure 2 In the example shown, consider the case where a certain vehicle 2 desires to use the remote driving service. In this case, the vehicle 2 sends communication information COM25 to the management server 51. The communication information COM25 includes, for example, a usage desire information IUS and vehicle information IVH. That is, the information included in the communication information COM25 is the same as the information included in the Figure 1 communication information COM21 shown.

[0050] In Figure 2 In the example shown, also consider the case where the driver 4 desires the provision of the remote driving service. In this case, the vehicle 1 sends communication information COM15a to the management server 51 according to an instruction from the driver 4. The communication information COM15a includes, for example, a provision desire information ISP. Regarding the provision desire information ISP, it is the same as that described in Figure 1 the above.

[0051] When the management server 51 receives the usage desire information IUS from the vehicle 2, it determines, from among the vehicles 1 waiting for remote driving and the remote driving devices 52, the device responsible for the remote driving of the vehicle 2. For example, when the specification information of the vehicle 2 (specifically, the specification information of the communication device and the driving assistance device) is consistent with the specification information of the vehicle 1, the management server 51 determines the vehicle 1 as the vehicle responsible for the remote driving of the vehicle 2. Then, the management server 51 sends communication information COM51 to the vehicle 1 and communication information COM52 to the vehicle 2. The communication information COM51 includes the information included in the communication information COM25 (that is, the usage desire information IUS and the vehicle information IVH). The communication information COM52 includes the information included in the communication information COM15a (that is, the provision desire information ISP).

[0052] When the communication information COM51 is sent from the management server 51 to the vehicle 1, the vehicle 1 identifies the vehicle 2 for which remote driving is to be performed. On the other hand, when the communication information COM52 is sent from the management server 51 to the vehicle 2, the vehicle 2 identifies the device responsible for its remote driving (in Figure 2In the example shown, vehicle 1). After that, vehicle 2 and vehicle 1 exchange communication information COM25 and COM15b via the management server 51. The communication information COM15b includes, for example, driving instruction information IIS. Regarding the driving instruction information IIS, it is the same as described in Figure 1 . After vehicle 2 and vehicle 1 identify each other, they can also directly exchange communication information without going through the management server 51.

[0053] When the specification information of vehicle 2 is inconsistent with the specification information of vehicle 1, the management server 51 determines one remote driving device 52 as the device responsible for the remote driving of vehicle 2. In this case, the information contained in the communication information COM25 is sent to the remote driving device 52 responsible for the remote driving of vehicle 2.

[0054] 2. Structural example of the vehicle

[0055] 2-1. Structural example of the vehicle (other vehicle) on the side receiving the remote driving service

[0056] Figure 3 is a block diagram showing a structural example common to each vehicle of vehicle 2 shown in Figure 1 . The structural example shown includes a camera 20, a microphone 21, a sensor group 22, a map database (DB) 23, a communication device 24, a driving device 25, a driving assistance device 26, and an information processing device 27. Figure 3 Multiple cameras 20 are provided inside and outside vehicle 2. The camera 20 captures images of the surroundings of vehicle 2. As the surrounding images, the front center image, the right front image, the left front image, the right side image, the left side image, and the rear image are exemplified. The surrounding image data typically consists of moving image data. However, the surrounding image data can also be static image data. The camera 20 sends the surrounding image data as surrounding image information IMG to the information processing device 27.

[0057] Multiple microphones 21 are provided on the outer side surface of vehicle 2. Each microphone 21 has a directivity and collects the surrounding sounds of vehicle 2. As the surrounding sounds, the right front sound, the left front sound, the right rear sound, and the left rear sound are exemplified. The microphones 21 respectively send the surrounding sound data as surrounding sound information SUD to the information processing device 27.

[0058]

[0059] ​The sensor group 22 includes state sensors that detect the state of the vehicle 2. As state sensors, a speed sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor are exemplified. The sensor group 22 further includes a position sensor that acquires the position and traveling direction of the vehicle 2. As the position sensor, a GNSS (Global Navigation Satellite System) sensor is exemplified. The sensor group 22 transmits the state data of the vehicle 2 as state information STS to the information processing device 27. The sensor group 22 further transmits the position data and the traveling direction data as position information LCT to the information processing device 27.

[0060] The sensor group 22 may also include identification sensors other than the camera 20. The identification sensors identify (detect) targets around the vehicle 2 using radio waves or light. As the identification sensors, a millimeter-wave radar and a LIDAR (Laser Imaging Detection and Ranging) are exemplified.

[0061] Map information is stored in the map DB 23. As the map information, information on the position of roads, information on the shape of roads (e.g., the category of curves and straight lines), information on the positions of intersections and structures are exemplified. The map DB 23 is formed in an in-vehicle storage device (e.g., a non-volatile recording medium such as a hard disk or a flash memory). The map DB 23 may also be formed in a computer (e.g., an external server such as the management server 51) capable of communicating with the vehicle 2.

[0062] The communication device 24 performs wireless communication with a base station (not shown) of the network 3. As the communication standard for this wireless communication, communication standards for mobile communications such as 4G, LTE, or 5G are exemplified. As the connection destination of the communication device 24, at least the vehicle 1 is included. In Figure 2 In the example described, as the connection destination of the communication device 24, the remote facility 5 (management server 51) is included. In the communication with the vehicle 1, the communication device 24 transmits the communication information COM21 (or the communication information COM25) received from the information processing device 27 to the vehicle 1 (or the remote facility 5).

[0063] The traveling device 25 performs acceleration, deceleration, and steering operations of the vehicle 2. The traveling device 25 includes, for example, a driving device, a steering device, and a braking device. The driving device drives the tires of the vehicle 2. The steering device steers the tires of the vehicle 2. The braking device applies a braking force to the vehicle 2. Acceleration of the vehicle 2 is performed by controlling the driving device. Deceleration of the vehicle 2 is performed by controlling the braking device. When the driving device is a motor, regenerative braking based on the control of the motor can also be used to brake the vehicle 2. Steering operation of the vehicle 2 is performed by controlling the steering device.

[0064] The driving assistance device 26 is a device that assists the driving of the vehicle 2. Examples of the driving assistance device 26 include a direction indicator, vehicle lights (e.g., headlights, fog lights), a brake light, a windshield wiper, etc. The driving assistance device 26 is manually operated by the driver of the vehicle 2. When the vehicle 2 has a function of supporting the driver's manual operation or an autonomous driving function, the driving assistance device 26 is operated by an in-vehicle driving support or autonomous driving system.

[0065] The information processing device 27 is a computer for processing various information obtained by the vehicle 2. The information processing device 27 includes at least one processor 27a and at least one memory 27b. The processor 27a includes a CPU (Central Processing Unit). The memory 27b is a volatile memory such as a DDR memory, and expands various programs used by the processor 27a and temporarily stores various information. Various information obtained by the vehicle 2 is stored in the memory 27b. As this various information, it includes surrounding image information IMG, surrounding sound information SUD, status information STS, and position information LCT. When remotely driving the vehicle 2, as this various information, it also includes communication information COM12 (or COM25).

[0066] The processor 27a performs various controls related to the driving of the vehicle 2 by executing the programs stored in the memory 27b. The processor 27a also determines whether remote driving is required by executing the programs stored in the memory 27b. When it is determined that remote driving is required, the processor 27a generates vehicle information IVH based on various information stored in the memory 27b. The vehicle information IVH includes, for example, surrounding image information IMG, surrounding sound information SUD, and position information LCT. Status information STS can also be added to the vehicle information IVH.

[0067] When it is determined that remote driving is required, the processor 27a encodes the desired information IUS and the vehicle information IVH and outputs them to the communication device 24. During the encoding process, these information can also be compressed. The encoded information is included in the communication information COM21 (or COM25).

[0068] 2-2. Structural example of a vehicle (multi-functional vehicle) on the side providing the remote driving service

[0069] Figure 4 It shows the Figure 1 block diagram of the first structural example common to each vehicle of the vehicle 1 shown. Figure 4 The shown structural example includes an XBW sensor 10, a sensor group 11, a switch group 12, a map database (DB) 13, a communication device 14, an XBW actuator 15, a speaker 16, a driving assistance device 17, an information processing device 18, and wearable glasses (WG) 19.

[0070] The XBW sensor 10 is a sensor that acquires the operation information IBW of the input device of the XBW type driving device based on the driver 4. As the XBW type driving device, drive-by-wire (DBW), brake-by-wire (BBW), steer-by-wire (SBW), and shift-by-wire (ShBW) are exemplified. The input device of the XBW type driving device corresponds to the "driving input device" of the present application.

[0071] The input device in drive-by-wire is the accelerator pedal. In this case, the XBW sensor 10 is a sensor that detects the depression amount of the accelerator pedal. The input device in brake-by-wire is the brake pedal. In this case, the XBW sensor 10 is a sensor that detects the depression amount of the brake pedal. The input device in steer-by-wire is the steering wheel. In this case, the XBW sensor 10 is a sensor that detects the steering operation angle of the steering wheel. The input device in shift-by-wire is the shift lever. In this case, the XBW sensor 10 is a sensor that detects the shift position. The XBW sensor 10 sends the operation information IBW to the information processing device 27.

[0072] The sensor group 11 is a sensor other than the XBW sensor 10. As an example of the sensor group 11, the same examples as those of the sensor group 22 described in Figure 3 can be cited.

[0073] The switch group 12 includes various switches manually operated by the driver 4. As various switches, an EPB (Electric Parking Brake) switch, a direction indicator switch, a door lock switch, an air conditioner switch, a headlight switch, a brake light switch, a power window switch, a wiper switch, and an ignition switch are exemplified. As various switches, a driving mode switch for switching the driving mode is included.

[0074] The EPB switch switches the operating state of the EPB (ON / OFF). The turn signal switch switches the operating state of the turn signal signal (ON / OFF). The door lock switch switches the locking state of the doors of the vehicle 1 (LOCKED / UNLOCKED). The air conditioner switch switches the operating state of the air conditioner (RUNNING / STOPPED). The light switch switches the operating state of the lights (e.g., headlights, fog lights) (ON / OFF). The power window switch switches the operating state of the windows provided on the doors (OPEN / CLOSED). The brake light switch switches the operating state of the brake lights (ON / OFF) in conjunction with the operation of the brake pedal. The wiper switch switches the operating state of the wipers (ON / OFF). The ignition switch switches the operating state of the power supply circuit of the vehicle (ON / OFF). The driving mode switch switches the driving mode of the vehicle 1 between the normal driving mode and the remote driving mode.

[0075] The switch group 12 transmits information based on the driver's 4 operation to the information processing device 27. The switch group 12 includes switches that switch the operating state of the driving assistance device 17. Examples of such switches include a turn signal switch, a light switch, a brake light switch, and a wiper switch. The information regarding the operation of the switches that switch the operating state of the driving assistance device 17 is hereinafter referred to as "operation instruction information ISW."

[0076] Map information is stored in the map DB 13. Examples of map information include Figure 3 The example of the map information stored in the map DB 23 described in is the same as the example.

[0077] The communication device 14 performs wireless communication with a base station (not shown) of the network 3. Examples of the communication standard for this wireless communication include mobile communication standards such as 4G, LTE, or 5G. The connection destination of the communication device 14 includes at least the vehicle 2. Figure 2 In the example described in , the remote facility 5 (management server 51) is a connection destination of the communication device 14. In communication with the vehicle 2, the communication device 14 transmits the communication information COM12 (or communication information COM15a and COM15b) received from the information processing device 18 to the vehicle 2 (or the remote facility 5).

[0078] The XBW actuator 15 is an XBW actuator that operates based on a workload calculated based on the IBW operating information. An example of an actuator in drive-by-wire is a vehicle drive motor. An example of an actuator in brake-by-wire is a brake motor. An actuator in steer-by-wire includes a motor located on the steering side and a reaction force motor located on the steering wheel side. An example of an actuator in shift-by-wire is an electric hydraulic actuator that operates based on the IBW operating information.

[0079] The speaker 16 reproduces the ambient sound data included in the ambient sound information SUD. The ambient sound data is the recorded sound data of the microphone 21 in each pointing direction. The ambient sound data may also be the data after processing such as wavefront synthesis. The ambient sound data may also be the data after removing the sounds other than the vehicle sounds and the environmental sounds identified through data analysis. The vehicle sounds to be identified are the sounds generated accompanying steering operation, acceleration, or deceleration. As the sound generated accompanying steering operation, the static steering sound of the tire and the frictional sound between the tire and the road surface are exemplified. As the sound generated accompanying acceleration, the rotational sound of the vehicle drive motor is exemplified. As the sound generated accompanying deceleration, the rotational sound of the vehicle drive motor, the frictional sound between the tire and the road surface, etc. are exemplified. As the environmental sounds to be identified, the alarm sounds generated at railroad crossings, signal lights, the alarm sounds emitted by ambulances, the warning sounds emitted by the vehicles around the vehicle 2, etc. are exemplified. The ambient sound data may also be the virtual data of the vehicle sounds and the environmental sounds identified through data analysis.

[0080] The driving assistance device 17 is a device that assists the driving of the vehicle 1. As the driving assistance device 17, the direction indicator, vehicle lights (for example, headlamps, fog lamps), brake lights, windshield wipers, etc. are exemplified. The driving assistance device 17 is manually operated by the driver 4. In the case where the vehicle 1 has a function of supporting the manual operation of the driver 4 or an autonomous driving function, the driving assistance device 17 is operated by the in-vehicle driving support or autonomous driving system.

[0081] The information processing device 18 is a computer that processes various information related to the vehicle control of the vehicle 1. The information processing device 18 is also a computer that processes various information related to the remote driving of the vehicle 2. The information processing device 18 includes at least one processor 18a and at least one memory 18b. The basic structure of the information processing device 18 is the same as the structure of the information processing device 27 described in Figure 3 The various programs and various information used by the processor 18a are stored in the memory 18b. As the various information, the operation information IBW and the work instruction information ISW are included. In the case of performing the remote driving of the vehicle 2, the desired information IUS and the vehicle information IVH (however, excluding the surrounding image information IMG) are stored in the memory 18b.

[0082] The processor 18a decodes the desired use information IUS and the vehicle information IVH and transfers the surrounding image information IMG included in the vehicle information IVH to the WG 19 (transfer processing). The processor 18a also reproduces the surrounding sound information SUD included in the vehicle information IVH using the speaker 16 (reproduction processing). If the desired use information IUS and the vehicle information IVH are compressed, these are decompressed during the decoding process. The processor 18a then transmits the driving instruction information IIS to the vehicle 2 (transmission processing).

[0083] WG19 is a glasses-type or goggles-type terminal worn by the driver 4. Figure 4 In the example shown, WG 19 includes a controller 19a and a display 19b. Controller 19a communicates with information processing device 18 in short-range communication to acquire surrounding image information IMG included in vehicle information IVH. Controller 19a then outputs surrounding image data included in the surrounding image information IMG to display 19b.

[0084] When outputting surrounding image data, the controller 19a generates image data to be output to the display 19b through head tracking control. In head tracking control, for example, an acceleration sensor detects the movement of the driver's head HD. Image data to be output to the display 19b is then generated in response to the movement of the head HD.

[0085] Figure 5 : is a diagram illustrating an example of generating image data in head tracking control. Figure 5 In the example shown, the surrounding images include a front center image IM_CFR, a right front image IM_RF, a left front image IM_LF, a right image IM_RS, and a left image IM_LS. The image data output to the display 19b is composite image data SIM obtained by superimposing overlapping portions of two adjacent types of images.

[0086] During head tracking control, when left-right movement of the head HD is detected, the tilt angle θ of the head HD relative to a reference direction is determined. Then, synthetic image data SIM is generated by offsetting the left-right reference position (e.g., the center position) of the image data output to the display 19b by a distance corresponding to the tilt angle θ. During head tracking control, it is also possible to detect not only left-right movement of the head HD but also up-down movement to generate synthetic image data SIM.

[0087] Figure 6 is shown with Figure 1 The block diagram of the second configuration example common to each vehicle of the vehicle 1 shown. Figure 6 The structural example shown hasFigure 4 Elements common to the illustrated structural examples. Therefore, descriptions of common elements are omitted. In Figure 6 the illustrated structural example, WG19 includes a controller 19a, a display 19b, a speaker 19c, and a communication device 19d.

[0088] The functional structure of the speaker 19c is basically the same as that of the speaker 16. The functional structure of the communication device 19d is basically the same as that of the communication device 14. That is, Figure 6 the illustrated structural example includes the long-distance communication type WG19 with a built-in speaker. According to such a structural example, the communication information COM21 (or COM51) from the vehicle 2 is provided to the information processing device 18 and the controller 19a. Therefore, different from Figure 4 the illustrated structural example, the surrounding image information IMG is directly provided to the controller 19a without passing through the information processing device 18. In addition, the surrounding sound information SUD is also directly provided to the controller 19a, and the surrounding sound data included in the surrounding sound information SUD is reproduced using the speaker 19c.

[0089] In addition, in Figure 6 the illustrated structural example, WG19 may also have a structure for performing short-distance communication with the information processing device 18. In this case, WG19 may also indirectly obtain the surrounding image information IMG and the surrounding sound information SUD via the information processing device 18. On the other hand, the information processing device 18 may also indirectly obtain the utilization desired information IUS and the vehicle information IVH (however, except for the surrounding image information IMG and the surrounding sound information SUD) via WG19. The information processing device 18 may also provide the provided desired information ISP and the driving instruction information IIS to the vehicle 2 via WG19.

[0090] 3. Information Processing Example of the Vehicle (Multi-Functional Vehicle) on the Remote Driving Service Providing Side

[0091] Figure 7 And 8 are flowcharts showing processing examples executed by the information processing device 18 (processor 18a). Figure 7 The illustrated routine is repeatedly executed at a predetermined cycle. During the period when the driving mode of the vehicle 1 is set to the "remote driving mode", the Figure 8 illustrated routine is repeatedly executed at a predetermined cycle.

[0092] In Figure 7In the illustrated routine, first, it is determined whether the ignition switch (IG) is set to the operating state (ON) (step S11). If the determination result in step S11 is negative, the current process ends. If the determination result in step S11 is positive, the driving mode of vehicle 1 is set to the normal driving mode (step S12).

[0093] After the process of step S12, it is determined whether the ignition switch is set to the non-operating state (OFF) (step S13). If the determination result in step S13 is positive, the current process ends. If the determination result in step S13 is negative, it is determined whether a mode change signal is received (step S14). The mode change signal is output when the driver 4 manually operates the driving mode switch.

[0094] If the determination result in step S14 is positive, the driving mode is changed (step S15). In the process of step S15, a change is made from the currently set driving mode (e.g., the normal driving mode) to a driving mode that is not currently set (e.g., the remote driving mode). If the determination result in step S14 is negative, the process of step S13 is performed.

[0095] In Figure 8 In the illustrated routine, first, it is determined whether an information utilization request IUS is received (step S21). If the determination result in step S21 is negative, the current process ends. If the determination result in step S21 is positive, it is determined whether the specification conditions are met (step S22). In the process of step S22, the specification information of vehicle 1 (specifically, the specification information of the communication device and the driving assistance device) is compared with the specification information of vehicle 2 included in the information utilization request IUS. If the two pieces of information are the same, it is determined that the specification conditions are met.

[0096] If the determination result in step S22 is positive, an information supply request ISP is sent (step S23). The destination of the information supply request ISP is vehicle 2 that sent the information utilization request IUS. Vehicle 2 that sent the information utilization request IUS is determined based on the ID information (or IP address information) of vehicle 2.

[0097] After the process of step S23, it is determined whether vehicle information IVH is received (step S24). The process of step S24 is repeatedly executed until a positive determination result is obtained.

[0098] If the result of step S24 is affirmative, remote driving processing is executed (step S25). Remote driving processing includes transferring surrounding image information IMG, reproducing surrounding sound information SUD, and sending driving instruction information IIS. In remote driving processing, transfer, reproduction, and sending are performed in parallel.

[0099] In the transfer process, the surrounding image information IMG included in the vehicle information IVH acquired in the process of step S24 is sent to the WG 19 .

[0100] During the reproduction process, ambient sound data is generated for reproduction by the speaker 16 based on the ambient sound information SUD included in the vehicle information IVH acquired in step S24. During the reproduction process, the ambient sound data included in the ambient sound information SUD may be reproduced directly, or the data may be reproduced after undergoing processing such as wavefront synthesis. During the reproduction process, the ambient sound data included in the ambient sound information SUD may also be analyzed. In this case, the ambient sound data may not be reproduced, and virtual data of the vehicle sound and environmental sound identified as a result of the analysis may be reproduced.

[0101] During the transmission process, for example, a determination is made as to whether operation information IBW or work instruction information ISW has been received. If operation information IBW is determined to have been received, control information is generated based on the operation information IBW. This control information is transmitted as driving instruction information IIS. If work instruction information ISW is determined to have been received, this work instruction information is transmitted as driving instruction information IIS.

[0102] Following the processing of step S25, it is determined whether drive end information IEN has been received (step S26). Drive end information IEN is transmitted from vehicle 2 to vehicle 1, for example, when vehicle 2 receiving the remote driving service wishes to terminate its use. In other examples, drive end information IEN is transmitted from vehicle 2 to vehicle 1 when vehicle 2 arrives at its destination. Furthermore, in other examples, drive end information IEN is generated when driver 4 operates the driving mode switch to terminate the provision of the remote driving service.

[0103] If the result of the determination in step S26 is positive, the current process ends. Otherwise, the process in step S24 is performed. In other words, the process in steps S24 to S26 is repeated until a positive result is obtained in the process in step S26.

[0104] also, Figure 8 The routine shown is equivalent to implementing Figure 1 The following is an example of a process for remote driving services. Figure 2In the case of the processing example of the remote driving service shown, examples include deleting the processing of steps S22 and S23 from the Figure 8 routine shown, and replacing the "reproduction processing" of the ambient sound information SUD in the processing of step S25 with "transfer processing" (the transfer destination is WG19). Therefore, the description of this processing example is omitted.

[0105] 4. Effects

[0106] In the vehicle 1 to which WG19 is added to the typical structure for realizing the autonomous driving function according to the embodiment described above, the remote driving of the vehicle 2 is performed based on the switching of the driving mode. Therefore, it is possible to provide a remote driving service that utilizes the typical structure of an ordinary vehicle.

[0107] In addition, according to the embodiment, since the traveling device of the vehicle 1 is configured by an XBW type traveling device, it is possible to prohibit the vehicle 1 from autonomously traveling during the execution of the remote driving mode. Therefore, it is possible to ensure the safety of the vehicle 1 and the driver 4 as the remote operator during the execution of the remote driving mode. Further, according to the embodiment, the work instruction information ISW received during the execution of the remote driving mode is transmitted to the vehicle 2. Therefore, it is possible to prevent the driving assistance device 17 from operating during the execution of the remote driving mode and transmitting unnecessary information to pedestrians and the like around the vehicle 1. In addition, further according to the embodiment, the head tracking control by the controller 19a is performed. Therefore, it is possible to help the driver 4 as the remote operator grasp the surrounding conditions of the vehicle 2. Therefore, it is possible to improve the driving safety of the remotely driven vehicle 2 in the remote driving using WG19.

Claims

1. A multi-purpose vehicle with a remote driving function, which has the functions of autonomous driving and remote driving of other vehicles, wherein the multi-purpose vehicle is characterized by having: a driving input device operated by a driver of the multi-purpose vehicle; a driving assist device for assisting the driving of the multi-purpose vehicle; an information processing device for processing operation information of the driving input device operated by the driver; wearable glasses worn by the driver, communicating with the other vehicle through at least one of direct communication with the other vehicle and indirect communication via the information processing device; as well as a communication device for communicating with the other vehicle, The information processing device switches between a normal driving mode for driving the multi-purpose vehicle and a remote driving mode for driving the other vehicle. In the normal driving mode, vehicle control of the multi-purpose vehicle is performed using control information generated based on the operation information, In the remote driving mode, the control information is sent to the other vehicle via the communication device. During execution of the remote driving mode by the information processing device, the wearable glasses display an image of the surroundings of the other vehicle acquired through at least one of direct communication and indirect communication with the other vehicle, and reproduce sounds of the surroundings of the other vehicle acquired through at least one of direct communication and indirect communication. Furthermore, when the remote driving mode is set, the information processing device, upon receiving remote driving utilization request information from the other vehicle, determines whether the specification information of the communication device and the driving assistance device of the other vehicle included in the remote driving utilization request information is consistent with the specification information of the multi-purpose vehicle. When determining that the specification information of the communication device and the driving assistance device of the other vehicle matches the specification information of the multi-purpose vehicle, the information processing device generates the control information and transmits the control information to the other vehicle via the communication device.

2. The multi-purpose vehicle according to claim 1, characterized in that The multi-purpose vehicle further comprises a drive device, a brake device and a steering device. The driving input device includes an accelerator pedal, a brake pedal, a steering wheel, and a gear lever. The drive device and the accelerator pedal constitute a drive-by-wire system, the brake device and the brake pedal constitute a brake-by-wire system, the steering device and the steering wheel constitute a steer-by-wire system, and the shift lever constitutes a shift-by-wire system.

3. The multi-functional vehicle according to claim 1 or 2, characterized in that, Also features: a switch operated by the driver of the multi-purpose vehicle to activate the driving assist device; In the normal driving mode, the operation instruction information from the switch is output to the driving assistance device. In the remote driving mode, the work instruction information is sent to the other vehicle via the communication device.

4. The multi-purpose vehicle according to claim 1 or 2, characterized in that: The surrounding images include a front image, a left front image, and a right front image of the other vehicle. The wearable glasses perform head tracking control, wherein during execution of the remote driving mode, the head tracking control generates an image displayed on the wearable glasses in accordance with the driver's head movement based on the surrounding image. The surrounding sounds include vehicle sounds and environmental sounds recognized by analyzing sounds collected by a microphone of the other vehicle, or virtual sounds of the other vehicle.

5. A remote driving method for remotely driving another vehicle using a multifunctional vehicle having both autonomous driving capabilities and remote driving capabilities for the other vehicle, the remote driving method comprising: The step of setting the driving mode of the multi-purpose vehicle to a remote driving mode for driving the other vehicle; The steps of displaying, while the driving mode is set to the remote driving mode, surrounding images of the other vehicle acquired through at least one of direct communication and indirect communication with the other vehicle on wearable glasses worn by a driver of the multi-purpose vehicle, and reproducing surrounding sounds of the other vehicle acquired through at least one of direct communication and indirect communication using the wearable glasses; and The step of transmitting control information generated based on operation information of a driving input device operated by the driver to the other vehicle while the driving mode is set to the remote driving mode, The remote driving method further comprises the step of determining, when receiving remote driving utilization request information from the other vehicle while the remote driving mode is set, whether specification information of the communication device and driving assistance device of the other vehicle included in the remote driving utilization request information is consistent with the specification information of the multi-purpose vehicle; Among them, When it is determined that the specification information of the communication device and the driving assistance device of the other vehicle matches the specification information of the multi-purpose vehicle, the control information is generated and transmitted to the other vehicle.

Images