Vehicle, vehicle control method, and vehicle control program

Abstract

To provide a vehicle including a vehicle control device capable of measuring a current position of a running vehicle and controlling running of the vehicle.SOLUTION: A vehicle V includes: a vehicle control device 1 that controls running of the vehicle based on information about a current position of the vehicle; and an on-vehicle sensor 10 that detects an external environment around the vehicle. The vehicle control device 1 includes an absolute position calculation unit that acquires GNSS information necessary for independent positioning from an artificial satellite SA to calculate an absolute position of the vehicle, a receipt determination unit that determines whether GNSS correction information necessary for relative positioning can be received from a reference station ST, a relative position calculation unit that calculates a relative position of the vehicle by relative positioning, and a vehicle control unit that controls running of the vehicle based on the information about the current position of the vehicle based on the relative position and detection information about the external environment around the vehicle, having been obtained by the on-vehicle sensor 10.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a vehicle, a vehicle control method, and a vehicle control program. [Background technology]

[0002] In recent years, autonomous vehicles equipped with ADAS (Advanced Driver Assistance Systems) have become known that can grasp information about the external environment around the vehicle itself and control the vehicle's driving on behalf of the driver to ensure safety and comfort for the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-164056 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 discloses a remote driving technology that controls the driving of a vehicle by communicating via a network in order to safely stop the vehicle or allow it to continue driving when a malfunction occurs in a commonly known vehicle causing a driving disruption.

[0005] An object of the present invention is to provide a vehicle equipped with a vehicle control device capable of measuring the current position of a traveling vehicle and controlling the traveling of the vehicle, a vehicle control method, and a vehicle control program. [Means for solving the problem]

[0006] The above problem is solved by a vehicle of the present invention, which includes a vehicle control device that controls the traveling of the vehicle based on information on the current position of the vehicle, and an on-board sensor that detects the external environment around the vehicle, and the vehicle control device includes an absolute position calculation unit that acquires GNSS information necessary for single-unit positioning through a GNSS receiver mounted on the vehicle and calculates the absolute position of the vehicle by single-unit positioning, a reception determination unit that determines whether GNSS correction information necessary for relative positioning can be received from an external reference station, a relative position calculation unit that calculates the relative position of the vehicle by relative positioning when it is determined that the GNSS correction information can be received, and a vehicle control unit that controls the traveling of the vehicle based on information on the current position of the vehicle based on the relative position and detection information of the external environment of the vehicle obtained by the on-board sensor.

[0007] In addition, according to the vehicle control method of the present invention, there is provided a vehicle control method using a vehicle equipped with a vehicle control device and an on-board sensor, wherein the vehicle control device controls the traveling of the vehicle based on information on the current position of the vehicle, performs the following steps: acquires GNSS information necessary for single-unit positioning through a GNSS receiver mounted on the vehicle, and calculates the absolute position of the vehicle by single-unit positioning; determines whether GNSS correction information necessary for relative positioning can be received from an external reference station; if it is determined that the GNSS correction information can be received, calculates the relative position of the vehicle by relative positioning; and controls the traveling of the vehicle, wherein the on-board sensor performs a step of detecting the external environment around the vehicle, and in the step of controlling the traveling of the vehicle, the traveling of the vehicle is controlled based on information on the current position of the vehicle based on the relative position and detection information of the external environment of the vehicle obtained by the on-board sensor. In addition, according to the vehicle control program of the present invention, a computer as a vehicle control device that is mounted on a vehicle and controls the driving of the vehicle based on information on the current position of the vehicle is configured to perform the following processes: acquiring GNSS information necessary for single-unit positioning through a GNSS receiver mounted on the vehicle, and calculating the absolute position of the vehicle by single-unit positioning; determining whether GNSS correction information necessary for relative positioning can be received from an external reference station; and if it is determined that the GNSS correction information can be received, calculating the relative position of the vehicle by relative positioning; and controlling the driving of the vehicle based on information on the current position of the vehicle based on the relative position and detection information of the external environment of the vehicle obtained by an on-board sensor mounted on the vehicle that detects the external environment around the vehicle. Effect of the Invention

[0008] According to the vehicle, vehicle control method, and vehicle control program of the present invention, it is possible to measure the current position of a traveling vehicle and control the traveling of the vehicle. [Brief description of the drawings]

[0009] [Figure 1] 1 is a configuration diagram showing an entire vehicle remote control system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hardware configuration of the vehicle remote control system (excluding the control device). [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of an operation device. [Figure 4] FIG. 2 is a diagram illustrating functions of a vehicle control device and an operation device. [Figure 5A] 11A and 11B are explanatory diagrams showing a composite image displayed on a display monitor of an operating device, and are diagrams showing examples of positions and sizes at which image information captured by a plurality of imaging devices is arranged. [Figure 5B] FIG. 11 is an explanatory diagram showing another example of a composite image. [Figure 5C] 11 is an explanatory diagram showing an example of a composite image displayed on a display monitor of an operating device when the vehicle is backing up; FIG. [Figure 6]FIG. 11 is a sequence diagram showing a flow of displaying a composite image. [Figure 7] FIG. 11 is a flow chart showing a process for selecting the resolution of a composite image to be transmitted depending on a communication state. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to FIGS. As shown in FIG. 1, the vehicle remote control system S of this embodiment is a system that realizes "automatic driving" in which the external environment of a traveling vehicle V is grasped, a driving route of the vehicle V is planned on behalf of the driver, and the vehicle V is controlled based on the driving route to drive the vehicle, and "remote driving" in which an operator H outside the vehicle V remotely controls (externally controls) the vehicle V to drive the vehicle, and the vehicle remote control system S is capable of performing a "mode switching process" that switches between the automatic driving mode and the remote driving mode. In addition, there is a manual driving mode (details of which will be described later) in which the driver gets into the vehicle V and drives it, and the mode switching process can switch between the manual driving mode and the automatic driving mode, as well as between the manual driving mode and the remote driving mode. It should be noted that the operator H does not have to be a human being, and may be, for example, an AI (artificial intelligence).

[0011] <<Hardware configuration of vehicle remote control system S>> As shown in Fig. 1, the vehicle remote operation system S is mainly composed of a vehicle control device 1 for a vehicle V and an operation device 50. The vehicle control device 1 is a device mounted on the vehicle V and controls the overall traveling of the vehicle V. The operation device 50 is a device connected to the vehicle control device 1 via a network N (digital communication path) and operates the vehicle V through communication via the network N. Of course, the vehicle control device 1 and the operation device 50 may communicate directly with each other.

[0012] The vehicle V includes an on-vehicle sensor 10, an on-vehicle locator 20, an on-vehicle ECU 30, and an on-vehicle communication device 40. The on-vehicle sensor 10 is a device that detects the external environment around the vehicle V. The on-vehicle locator 20 is a device that receives GNSS signals from an artificial satellite SA and a reference station ST, and measures the current position of the vehicle V. The on-vehicle ECU 30 is a device that controls the steering, acceleration, deceleration, etc. of the vehicle V. The on-vehicle communication device 40 is a device that communicates with an operation device 50 or an external device installed outside the vehicle V. As shown in FIG. 2, a vehicle control device 1, an on-vehicle sensor 10, an on-vehicle locator 20, an on-vehicle ECU 30, and an on-vehicle communication device 40 mounted on a vehicle V are connected via an on-vehicle network (CAN: Controller Area Network). In more detail, the vehicle-mounted locator 20, the vehicle-mounted communication device 40, the imaging device 11 of the vehicle-mounted sensor 10, and the lidar 13 are connected to the vehicle control device 1 so as to be able to communicate serially using a communication interface such as an RS232C connector or a USB (Universal Serial Bus) connector.

[0013] The vehicle control device 1 is a computer (first computer) including a CPU as a data arithmetic and control processing device, a ROM, a RAM and a HDD (SSD) as storage devices, and a communication interface for transmitting and receiving information data via a CAN or the like. In addition to a main program that performs the functions required of a computer, a vehicle control program and a vehicle remote operation program are stored in the storage device of the vehicle control device 1. The functions of the vehicle control device 1 are realized by executing these programs by the CPU.

[0014] In order to perform "automatic driving," the vehicle control device 1 controls the driving of the vehicle V by controlling the on-board ECU 30 (more specifically, the overall ECU 31) based on information on the external environment obtained from the on-board sensor 10 and information on the current location obtained from the on-board locator 20. In addition, in order to perform "remote driving," the vehicle control device 1 is connected to the operation device 50 via the in-vehicle communication device 40 through a network N (digital communication path) including wireless communication lines, etc., and transmits information on the external environment and information on the current location to the operation device 50. In other words, the vehicle control device 1 is newly installed on a vehicle V that already has an "automatic driving function (on-vehicle sensor 10, on-vehicle locator 20, on-vehicle ECU 30)" to enhance the performance of the existing "automatic driving function" and also to give the vehicle V a new "remote driving function." Although details will be described later, the operation device 50 receives information on the external environment and current location necessary for the remote driving function from the vehicle control device 1 and displays the information on the external environment on the display monitor 51, and can also notify information to the operator H of the operation device 50.

[0015] The on-vehicle sensor 10, on-vehicle locator 20, and on-vehicle ECU 30 that realize the "automatic driving function" will be described.

[0016] <Automotive Sensor 10> The on-vehicle sensor 10 is a device that detects moving objects (other vehicles, pedestrians, etc.), various structures, road shapes, etc. (hereinafter, may be referred to as target objects) around the vehicle V as the external environment around the vehicle V. Specifically, as shown in Fig. 2, the on-vehicle sensor 10 is composed of a plurality of imaging devices 11, a plurality of radars 12, and a plurality of lidars 13 installed on the vehicle body. Note that the on-vehicle sensor 10 may further include sensors other than the imaging devices 11, the radars 12, and the lidars 13.

[0017] The imaging device 11 is a photographing camera (wide-angle camera) that captures images of the surroundings of the vehicle V and acquires them as image information, and is, for example, a small digital camera that uses a solid-state imaging element such as a CCD or CMOS. The imaging device 11 creates external image data to execute a "sensing function" for driving control of the vehicle V and a "monitoring function" for the driver (operator H), and transmits the external image data (hereinafter, image information) by serial communication to the vehicle control device 1 connected by a USB cable.

[0018] As shown in FIG. 2, the imaging device 11 is made up of nine imaging devices (first imaging device 11a to ninth imaging device 11i; hereinafter, these may be collectively referred to as imaging device 11) attached at predetermined positions. In this embodiment, the first imaging device 11a, the second imaging device 11b, and the third imaging device 11c capture images of the front, the right diagonally forward, and the left diagonally forward of the vehicle V, and are attached to the inside of the passenger compartment of the front window (windshield) at the front of the roof of the vehicle V. The fourth imaging device 11d images the rear of the vehicle V and is attached to the center of the rear bumper (back bumper) of the vehicle V. The fifth imaging device 11e and the sixth imaging device 11f are attached to the left and right side mirrors of the vehicle V and image the right rear and left rear of the vehicle V. The first imaging device 11a to the sixth imaging device 11f are main cameras that image the main areas around the vehicle V.

[0019] The imaging device 11 also has a seventh imaging device 11g, an eighth imaging device 11h, and a ninth imaging device 11i as sub-cameras. The seventh imaging device 11g is attached to the front bumper of the vehicle V and captures an image in front of the vehicle V. The eighth imaging device 11h and the ninth imaging device 11i are attached around the left and right backlights of the vehicle V and capture an image in the right rear and left rear of the vehicle V. The seventh imaging device 11g, the eighth imaging device 11h, and the ninth imaging device 11i, which are sub-cameras, capture images of blind spots that cannot be captured by the first imaging device 11a to the sixth imaging device 11f of the main camera. As another example of the sub-camera, the seventh imaging device 11g may be attached to an upper portion of the back window (rear window) of the vehicle V and may capture an image of the rear of the vehicle V from that position. The eighth imaging device 11h may be attached to the A-pillar to the C-pillar on the right side of the vehicle V, and the ninth imaging device 11i may be attached to the A-pillar to the C-pillar on the left side.

[0020] As described above, the first imaging device 11a to the ninth imaging device 11i are connected to the vehicle control device 1 via serial communication, and are configured to transmit captured image information to the vehicle control device 1. Note that the number and positions of the imaging devices 11 attached to the vehicle V in this embodiment are merely examples, and the number and positions can be changed depending on the shape or use of the vehicle V. In addition, the vehicle control device 1 may extract the contours of target objects around the vehicle V or extract lane markings (white lines, etc.) on the road by analyzing the image information captured by the imaging device 11.

[0021] The radar 12 is a sensor that performs three-dimensional spatial imaging by transmitting radio waves while continuously changing the irradiation direction, detecting a target object by receiving reflected waves from the target object, and measuring the position and speed of the target object. The vehicle V of this embodiment uses a millimeter wave radar that uses millimeter waves in the 30 to 300 GHz band as radio waves. By using millimeter waves, the radar 12 can detect a target object with high accuracy compared to the imaging device 11 and the lidar 13, even in environmental conditions such as poor visibility at night or in bad weather.

[0022] The radar 12 acquires detection result data (detection signal) of the target object, and transmits the detection result data to the vehicle control device 1 via the CAN. The radar 12 and the vehicle control device 1 may be connected with a USB cable, and the radar 12 may transmit the detection result data to the vehicle control device 1 by serial communication. As shown in FIG. 2, the radar 12 is made up of four radars 12 (a first radar 12a to a fourth radar 12d) attached to the vehicle V at predetermined positions. The first radar 12a and the second radar 12b are attached around the left and right front lights of the vehicle V, and the third radar 12c and the fourth radar 12d are attached around the left and right back lights of the vehicle V. The radar 12 is not particularly limited to a millimeter wave radar, but may be a laser radar, an ultrasonic sensor, or the like.

[0023] The lidar 13 is a remote sensor that measures the distance to a target object by emitting a laser light and receiving the reflected light from the target object, and performs three-dimensional spatial imaging. Compared to the imaging device 11 and the radar 12, the lidar 13 can measure the distance to surrounding target objects in units of a few centimeters. The LIDAR 13 acquires distance measurement data that measures the distance to the target object. The LIDAR 13 is connected to the vehicle control device 1 via a USB cable, and transmits the distance measurement data to the vehicle control device 1 by serial communication. When the LIDAR 13 is connected to the vehicle control device 1 via a CAN, the LIDAR 13 may transmit the distance measurement data to the vehicle control device 1 via the CAN. As shown in FIG. 2, the riders 13 are made up of five riders 13 (a first rider 13a to a fifth rider 13e) attached to the vehicle V at predetermined positions. The first rider 13a and the second rider 13b are attached around the left and right front lights of the vehicle V. The third rider 13c is attached to the back bumper of the vehicle V. The fourth rider 13d and the fifth rider 13e are attached around the left and right tail lights of the vehicle V. The fourth rider 13d and the fifth rider 13e may be attached around the back light of the vehicle V.

[0024] As with the imaging device 11, the number and mounting positions of the radars 12 and lidars 13 are merely examples, and the number and mounting positions can be changed.

[0025] <In-vehicle locator 20> The vehicle-mounted locator 20 measures the current position of the vehicle V by utilizing a satellite positioning system using artificial satellites SA and reference stations ST, and also measures the acceleration and angular velocity of the vehicle V to improve the accuracy of measuring the current position. Specifically, the vehicle-mounted locator 20 is composed of a GNSS receiver 21 that receives GNSS radio waves (Global Navigation Satellite System, GPS radio waves) from multiple artificial satellites SA, and an inertial measurement unit 22 that measures the acceleration and angular velocity of the vehicle V.

[0026] The GNSS receiver 21 is specifically an RTK-GNSS receiver that receives GNSS radio waves from multiple (specifically, four) satellites SA as shown in Fig. 1, and generates "GNSS information" necessary for independent positioning. It also receives "GNSS correction information" necessary for relative positioning from an external reference station ST. The reference station ST is a fixed reference station set at a known point, receives GNSS radio waves from multiple artificial satellites SA, generates “GNSS correction information”, and transmits it to the GNSS receiver 21 of the vehicle-mounted locator 20. "GNSS information" is distance information between multiple artificial satellites SA and the GNSS receiver 21, and "GNSS correction information" is distance information in which the measurement error of the "GNSS information" is corrected by a reference station ST located at a known point receiving GNSS radio waves and communication between the reference station ST and the GNSS receiver 21.

[0027] The inertial measurement unit 22 is also called an IMU (Inertial Measurement Unit) and includes a three-axis gyro sensor (angular velocity meter) and a three-axis acceleration sensor (accelerometer). The inertial measurement unit 22 measures the three-dimensional angular velocity and acceleration of the vehicle V, and transmits information on the acceleration and angular velocity of the vehicle V to the vehicle control device 1. The vehicle control device 1 can measure the current position of the vehicle V with a smaller error range by combining GNSS information (GNSS correction information) received from the GNSS receiver 21 with the angular velocity and acceleration information of the vehicle V received from the inertial measurement device 22.

[0028] <In-vehicle ECU30> The in-vehicle ECU 30 is, for example, an ECU for ADAS, and includes an overall ECU 31 in a higher hierarchy that is connected to the vehicle control device 1 and transmits and receives various data. The in-vehicle ECU 30 also includes a steering wheel ECU 32, an accelerator ECU 33, and a brake ECU 34 as lower hierarchy units that subdivide and control the steering, acceleration, deceleration, and the like of the vehicle V. The steering wheel ECU 32, the accelerator ECU 33, and the brake ECU 34 are each connected to the overall ECU 31 in the higher hierarchy unit. As shown in Fig. 2, the overall ECU 31, the steering wheel ECU 32, the accelerator ECU 33, and the brake ECU 34 form a hierarchical structure. The steering wheel ECU 32 is also called a driving support computer, and the accelerator ECU 33 and the brake ECU 34 are also called a power management control unit.

[0029] The integrated ECU 31 is connected to the vehicle control device 1 and transmits and receives various data to and from the vehicle control device 1. The integrated ECU 31 is also called a driving support computer, and performs the function of comprehensively controlling the steering wheel ECU 32, accelerator ECU 33, and brake ECU 34. It also performs the function of power management (power management for various computers).

[0030] The steering ECU 32, the accelerator ECU 33, and the brake ECU 34 are computers that perform detailed control of the steering, acceleration, deceleration, and the like of the vehicle V.

[0031] The steering wheel ECU 32 is a device that controls the traveling direction of the vehicle V by operating the electric power steering V1 of the vehicle V in accordance with instructions from the integrated ECU 31. The electric power steering V1 includes a steering mechanism that steers the front wheels of the vehicle V. In the case of manual driving, the electric power steering V1 steers the front wheels of the vehicle V in accordance with the steering operation of the steering wheel V1a by the driver.

[0032] The accelerator ECU 33 is a device that mainly controls the acceleration and deceleration of the vehicle V by operating the electric throttle V2 of the vehicle V in accordance with instructions from the integrated ECU 31. The electric throttle V2 includes a drive mechanism that outputs a drive force to rotate the drive wheels of the vehicle V. In the case of manual driving, the electric throttle V2 adjusts the output of the engine in accordance with the accelerator operation of the accelerator pedal V2a by the driver.

[0033] The brake ECU 34 is a device that controls the deceleration and stopping of the vehicle V by operating the electromagnetic brake device V3 of the vehicle V in accordance with instructions from the integrated ECU 31. The electromagnetic brake device V3 is attached to each wheel of the vehicle V and has a mechanism for applying resistance to the rotation of the wheels to slow down or stop the vehicle V. In the case of manual driving, the electromagnetic brake device V3 adjusts the operation of the electromagnetic brake device V3 according to the braking operation of the brake pedal V3a by the driver.

[0034] The number and functions of the individual ECUs connected to the overall ECU 31 are not particularly limited to the steering ECU 32, accelerator ECU 33, and brake ECU 34, and other ECUs may be provided at the same hierarchical level as these ECUs.

[0035] <In-vehicle communication device 40> The in-vehicle communication device 40 and the operation device 50 that realize the "automatic driving function" will be described. The in-vehicle communication device 40 is a device that communicates information with an operation device 50 installed outside the vehicle V and an external server (not shown) via a network N. For example, the in-vehicle communication device 40 transmits image information (composite image) captured by the imaging device 11 of the vehicle control device 1 and position information as information necessary for "remote driving" to the operation device 50, and receives driving operation information for the vehicle V from the operation device 50. The in-vehicle communication device 40 can also communicate with an external server (not shown) to receive, for example, the latest traffic information and weather information. The in-vehicle communication device 40 is composed of three communication devices 41 capable of wireless communication (first communication device 41a to third communication device 41c, hereinafter sometimes collectively referred to as communication devices 41), and each of the three communication devices 41 is capable of communicating with a wireless base station RB (base station).

[0036] The first communication device 41a to the third communication device 41c may each be connected to a different wireless base station RB or may be connected to the same wireless base station RB. As the vehicle V moves, the distance to the currently connected wireless base station RB changes, and the communication state also changes. When the radio wave strength becomes smaller than a predetermined threshold and a band for transmitting image information cannot be secured, the first communication device 41a to the third communication device 41c search for and attempt to connect to another wireless base station RB. When another connectable wireless base station RB is found, the connection is switched. Since a plurality of communication devices are used to connect to the radio base station RB, even if one of the communication devices loses connection to the radio base station RB, the other communication devices are still connected to the radio base station RB, making the connection less likely to be lost compared to the case where only one communication device is connected. In this embodiment, three communication devices 41 (first communication device 41a to third communication device 41c) are used to connect to the radio base station RB, but the number of communication devices 41 is not limited to three and may be two, four or more. That is, the number of wireless communication lines connected by the communication devices 41 is not limited to three and may be two, four or more. Furthermore, when the communication environment is good, communication may be performed using one line. Furthermore, as will be described in more detail later, when communication conditions deteriorate and the bandwidth for transmitting the “composite image” cannot be secured, the in-vehicle communication device 40 reduces the image quality of the “composite image” to be transmitted, i.e., generates a “composite image” with a lower resolution than the “composite image” before transmission, thereby reducing the amount of data and transmitting it.

[0037] Also, the first communication device 41a to the third communication device 41c can connect to the radio base station RB by switching between a long-distance communication line of a fourth generation mobile communication system (4G) and a long-distance communication line of a third generation mobile communication system (3G). Although the 3G communication line (second communication line) can receive signals over a wider range than the 4G communication line (first communication line), the 4G communication line has a higher communication speed or better communication quality. Therefore, the communication device 41 is configured to preferentially connect to the 4G communication line and communicate in an environment with less delay. When the communication device 41 cannot connect to the radio base station RB through the 4G communication line, the communication device 41 connects using the 3G communication line. Note that, when the communication device 41 can connect to a communication line of a fifth generation mobile communication system (5G), which has a higher communication speed and higher communication quality than the 4G line, the communication device 41 may communicate through the 5G communication line.

[0038] The in-vehicle communication device 40 may be configured to be connectable to the network N via wireless communication lines (long distance communication lines) of multiple communication carriers. When the radio wave strength of the wireless communication line in communication in all communication devices becomes less than a predetermined threshold value α, the in-vehicle communication device 40 measures the radio wave strength with the wireless communication line of another communication carrier, and when the measured radio wave strength is equal to or greater than the threshold value α, switches the wireless communication line to be connected. When multiple communication carriers are detected, it is desirable to connect to the communication carrier with the highest radio wave strength, but it may also be possible to connect to the wireless communication line of the communication carrier that was detected first.

[0039] <<Hardware configuration of the operation device 50>> The operation device 50 is a device for providing a remote driving service to a vehicle V having a remote driving function, and is located in a different location from the vehicle V to be remotely operated as shown in FIG. The operation device 50 is connected to the vehicle control device 1 of the vehicle V via a network N, and operates the vehicle V through communication via the network N.

[0040] As shown in FIG. 3, the operation device 50 is a computer (second computer) equipped with a CPU as a data calculation / control processing device, a ROM, RAM, and HDD (or SSD) as memory devices, and a communication interface for communicating via a network N such as the Internet or an intranet. In addition to a main program that performs functions necessary for a computer, a vehicle remote control program, an image information display program, etc. are stored in the storage device of the operation device 50. The functions of the operation device 50 are realized by executing these programs by the CPU.

[0041] The operation device 50 is connected to the communication device 60 via a communication interface. The communication device 60 is a device that communicates information with the vehicle control device 1 of the vehicle V and an external server (not shown) via a network N. The communication device 60 receives information necessary for "remote driving" transmitted from the vehicle control device 1, such as the traveling direction and speed of the vehicle V, image information around the vehicle, and position information of the vehicle V, and transmits driving operation information to the vehicle control device 1. The communication device 60 may also communicate with an external server to receive traffic information and weather information for the road on which the vehicle V is traveling.

[0042] As shown in Figures 1 and 3, the operation device 50 is equipped with a display monitor 51, a navigation display monitor 52, and a speaker 59 as output devices, which are arranged to be positioned in front of the operator H when the operator H is seated in the chair 61. 1 and 3, the operation device 50 includes, as input devices, a steering wheel 53, an accelerator pedal 54, a brake pedal 55, an operation switch 56, a touch panel 57, and a microphone 58. An operator H sits in a chair 61 and remotely operates a traveling vehicle V using the steering wheel 53 etc. while watching an image on a display monitor 51 arranged in front of him.

[0043] <Display monitor 51> The display monitor 51 is a device that outputs visual information for remote driving, and mainly displays information received from the vehicle control device 1. The information received from the vehicle control device 1 includes vehicle information such as CAN data received from the integrated ECU, meter speed, throttle opening, brake depression amount, steering angle, electrically wired shift position, and turn signals, and it is also possible to display this information on the display monitor 51. The display monitor 51 of the operation device 50 is composed of three liquid crystal monitors, as shown in Fig. 1 and Fig. 3. More specifically, it is composed of a first monitor 51a arranged in front of the operator H, a second monitor 51b arranged adjacent to the right side of the first monitor 51a, and a third monitor 51c arranged adjacent to the left side of the first monitor 51a. The second monitor 51b and the third monitor 51c are arranged at an angle so that the display screen faces the operator H so that the operator H can easily see. Each of the first monitor 51a to the third monitor 51c is a liquid crystal monitor having a predetermined resolution, for example, a resolution of 1920 x 1080 (full HD, 2K). The first monitor 51a to the third monitor 51c may be a high-resolution monitor having a resolution of 4K, 8K, etc.

[0044] The display monitor 51 configured with three liquid crystal monitors shown in FIG. 3 is an example, and the display monitor 51 may be configured with one liquid crystal monitor. Also, the display monitor 51 may be configured with two or four or more liquid crystal monitors. Also, the display monitor 51 may be configured with a curved, horizontally elongated liquid crystal monitor. Also, the display monitor 51 may be realized by a mobile terminal such as a smartphone or a tablet terminal. The display monitor 51 may be VR goggles worn on the head of the operator H.

[0045] <Navigation display monitor 52> The navigation display monitor 52 is a device that displays the position of the vehicle V together with map information based on the position information of the vehicle V received from the vehicle control device 1.

[0046] <Steering wheel 53, accelerator pedal 54, brake pedal 55> The steering wheel 53 is an operating unit used to adjust the steering angle (steering amount) of the traveling vehicle V. The accelerator pedal 54 is used to input the accelerator pedal opening degree of the traveling vehicle V. The brake pedal 55 is used to input the brake pedal opening degree of the vehicle V. The input steering amount, accelerator pedal opening degree, and brake pedal opening degree are transmitted to the vehicle control device 1 of the vehicle V by the communication device 60. As described above, the vehicle control device 1 of the vehicle V operates the steering wheel ECU 32, the accelerator ECU 33, and the brake ECU 34 via the integrated ECU of the vehicle V based on the received steering amount, accelerator pedal opening degree, and brake pedal opening degree.

[0047] <Operation switch 56> The operation switch 56 has a role as, for example, a shift lever, and the operator H can press the switch to instruct the vehicle V to move forward or backward. The operator H may also press the operation switch 56 to instruct the vehicle V to start or stop the engine, open or close the door, turn on the horn, turn signals, etc.

[0048] <Touch Panel 57> The touch panel 57 is a device for inputting layout information, which indicates the position and size of image information to be displayed on the display monitor 51, and the like. <Mike 58> The microphone 58 is a device for inputting voice, and the voice signal input by the microphone 58 is transmitted to the vehicle control device 1 by the communication device 60. The voice signal can be emitted, for example, by a speaker 59 of the vehicle V toward an occupant of the vehicle V, thereby transmitting the voice. <Speaker 59> The speaker 59 is an audio device that provides auditory information. The speaker 59 can inform the operator H of audio information received from the vehicle control device 1, such as the voices of the passengers of the vehicle V and sounds around the vehicle V.

[0049] In addition, the operation device 50 is equipped with input devices shaped like a steering wheel 53, accelerator pedal 54, brake pedal 55, etc., so that the operator H can intuitively remotely control the vehicle V, but these input devices may also be a controller for a game console, a keyboard for a PC, or the monitor screen of a mobile terminal capable of touch input. The operator H of the operation device 50 is a human, but the operation may be mainly performed by an AI (artificial intelligence), in which case the human monitors the operation of the AI ​​and the information about the vehicle V displayed on the display monitor 51 or the like.

[0050] <<Functional configuration of vehicle remote control system S>> The vehicle remote control system S will be described from the functional aspect with reference to Fig. 4. The vehicle control device 1 includes a storage unit 100, an external information acquisition unit 101, a position identification unit 107, a communication unit 108 (first communication unit), a vehicle control unit 109, an image processing unit 110, and a traveling direction acquisition unit 111 as components. These functions are realized by the CPU, ROM, RAM, HDD, communication interface, various programs, and the like of the vehicle control device 1.

[0051] The operation device 50 includes a storage unit 500, a communication unit 501, a display unit 502, an input unit 503, a communication control unit 504, and an audio output unit 505 as components. These functions are realized by the CPU, ROM, RAM, HDD, communication interface, various programs, and the like of the operation device 50.

[0052] The functions of the vehicle control device 1 will be described. <Storage section 100> The memory unit 100 stores various data and programs such as information on the direction of travel of the vehicle V, "image information" captured by multiple imaging devices 11, a "composite image" synthesized by the image processing unit 110, information on the direction of travel of the vehicle V, and "layout information."

[0053] <External information acquisition unit 101> The external information acquisition unit 101 acquires information from the on-board sensor 10 attached to the vehicle V, and stores the acquired information in the memory unit 100. More specifically, as the on-board sensor 10, nine imaging devices 11 (first imaging device 11a to ninth imaging device 11i), four radars 12 (first radar 12a to fourth radar 12d), and five LIDARs 13 (first LIDAR 13a to fifth LIDAR 13e) are attached to the body of the vehicle V, as shown in FIG. The external information acquisition unit 101 acquires image information, detection result data of a target object, and distance measurement data to the target object from the on-board sensor 10, and stores them in the memory unit 100. The external information acquisition unit 101 may further acquire "vehicle information" of the vehicle V from the in-vehicle ECU 30. Examples of the "vehicle information" include "steering angle information" obtained from the steering wheel ECU 32, "throttle opening information" obtained from the accelerator ECU 33, and "brake depression amount information" obtained from the brake ECU 34. "Vehicle speed information" may be obtained from a wheel speed sensor.

[0054] <Location specifying section 107> The position identification unit 107 identifies the current position of the vehicle V from the GNSS information received from the GNSS receiver 21 of the in-vehicle locator 20 and information such as the angular velocity and acceleration of the vehicle V received from the inertial measurement unit 22. In addition, when identifying the current position of the vehicle V, the "steering angle information" of the vehicle V may be acquired, and the position may be identified by further combining the "steering angle information." For example, by newly installing a steering angle sensor in the vehicle V, the "steering angle information" can be acquired through the steering angle sensor. In addition, when obtaining "speed information" of the vehicle V, a wheel speed sensor may be newly installed in the vehicle V, and the "speed information" may be obtained through the wheel speed sensor.

[0055] <Communication Division 108> The communication unit 108 (first communication unit) transmits information to the operation device 50. In this embodiment, the communication unit 108 transmits the “synthetic image” generated by the image processing unit 110 and the “position information” of the vehicle V identified by the position identification unit 107 to the operation device 50. The communication unit 108 is realized by the CPU of the vehicle control device 1 and the in-vehicle communication device 40, and the in-vehicle communication device 40 is composed of three communication devices 41 (first communication device 41a to third communication device 41c) capable of wireless communication. Therefore, the communication unit 108 can be connected to the wireless base station RB of the wireless communication line (remote communication line) through three communication lines.

[0056] Each of the three communication devices 41 is connected to a radio base station RB. Since the radio wave strength changes due to the movement of the vehicle V, if the connection cannot be maintained, the communication device 41 connects to another radio base station RB with stronger radio wave strength. Normally, the three communication devices 41 switch the radio base station RB separately. However, when the connections to the base stations are switched simultaneously in all communication lines, the connection to the operation device 50 may be temporarily cut off. For this reason, the communication unit 108 is configured to be capable of processing to change the timing of switching so that the radio base stations RB to which the three communication devices 41 are connected are not switched simultaneously. The number of communication devices 41 provided in the in-vehicle communication device 40 may be two, four or more. When the communication environment is good, the in-vehicle communication device 40 may be configured with one communication device 41.

[0057] <Vehicle control unit 109> The vehicle control unit 109 controls the integrated ECU 31 based on the "image information," "detection result data," and "distance measurement data" acquired by the external information acquisition unit 101 and the "current location information" identified by the location identification unit 107, and performs "automatic driving" of the vehicle V. In addition, the vehicle control unit 109 controls the integrated ECU 31 based on the “driving operation information” of the vehicle V acquired from the operation device 50, and performs “remote driving” of the vehicle V.

[0058] <Progressing direction acquisition unit 111> The travel direction acquisition unit 111 acquires information on the travel direction of the vehicle V, for example, from the position of the shift position of the vehicle V. The travel direction information is information indicating whether the vehicle V is currently traveling forward or backward (at the time when the operator H starts remotely operating the vehicle V, or at the time when the remote operation is in progress). When acquiring the travel direction information, the vehicle V may be in a stopped state. The travel direction information stored in the memory unit 100 is used, for example, when the image processing unit 110 generates a "synthetic image" corresponding to the travel direction of the vehicle V.

[0059] <Image processing unit 110> The image processing unit 110 synthesizes the “image information” acquired from each of the first imaging device 11a to the ninth imaging device 11i by the external information acquisition unit 101 based on the “layout information” stored in the storage unit 100 to generate a “composite image.” The "layout information" defines the position and size of the "image information" acquired from the imaging device 11 when it is displayed on the display monitor 51 of the operation device 50. In other words, it is a setting file that defines the position or size at which the image information acquired by the imaging device 11 is to be placed in the "composite image" when the "composite image" is generated. The "layout information" defines the display mode of the layout that does not create blind spots for the operator H and is easy for the operator H to operate.

[0060] The "layout information" may be stored in a predetermined storage unit of the vehicle V in association with a layout ID (layout identification information). In this case, after switching to the layout display using an operation switch or the like described later, a change operation is further performed on the layout information, and the changed layout ID is transmitted from the operation device 50 (operation switch 56) to the vehicle V. At this time, a composite image is generated in the vehicle V by combining an external image based on the layout information for the changed layout ID, and the composite image is displayed on the display monitor 51, as will be described in detail later. This "layout information" may be stored in an external server (not shown) provided on the network N, or may be transmitted to and received from the external server via a web browser or a dedicated web application.

[0061] The size of the "composite image" generated by the image processing unit 110 is determined by the performance of the display monitor 51 of the operation device 50. In this embodiment, the "composite image" is displayed on three liquid crystal monitors (first monitor 51a to third monitor 51c) of the display monitor 51. When the resolution of each liquid crystal monitor is full HD, the number of pixels of the "composite image" is preferably 5760×1080 pixels. Note that the number of pixels of the "composite image" may be changed to a lower resolution depending on the traveling direction or communication conditions. For example, when driving in reverse and performing remote control using only the central first monitor 51a, the "composite image" is set to be generated at the size of one screen, that is, at a size of 1920 x 1080 pixels. When communication conditions are poor, in order to reduce the amount of data to be transmitted, the "composite image" is generated with a lower resolution than usual, for example 4000 x 800 pixels. Information on the size of the "composite image" generated by image processing unit 110 is, for example, included in the layout information and is pre-stored in storage unit 100 or is input by input unit 503 of operation device 50. Furthermore, like the layout information, information on the size of the "composite image" may be stored in an external server (not shown) and changed by image processing unit 110 via a web browser or a web application.

[0062] When driving the vehicle V, the driver drives while looking at the front of the vehicle V and the rear view using the rearview mirror and side mirrors. Even when the vehicle V is remotely operated, it is desirable that an image of the front and an image of the rear of the vehicle V are displayed simultaneously on the display monitor 51. Therefore, the imaging device 11 mounted on the vehicle V of this embodiment includes a first imaging device 11a to a third imaging device 11c (first direction imaging device) that capture images in the forward direction (first traveling direction), and a fourth imaging device 11d to a sixth imaging device 11f (second direction imaging device) that capture images in the backward direction (second traveling direction) which is different from the forward direction. The "layout information" is configured so that image information of the forward direction captured by the first imaging device 11a to the third imaging device 11c and image information of the backward direction captured by the fourth imaging device 11d to the sixth imaging device 11f are displayed simultaneously.

[0063] Furthermore, the image processing unit 110 generates a "composite image" by superimposing backward image information captured by the fourth imaging device 11d to the sixth imaging device 11f on forward image information captured by the first imaging device 11a to the third imaging device 11c based on the layout information. This is the case when the vehicle V travels forward. For example, when the vehicle V travels backward, the "composite image" is generated by superimposing the forward image on the backward image information. Note that the image information to be superimposed on the forward or backward image information may be image information captured in the left or right direction of the vehicle V.

[0064] Examples of the "composite image" are shown in Figs. 5A to 5C. 5A, the image information FM of the first imaging device 11a capturing an image of the area in front of the vehicle V is displayed on the first monitor 51a in the center. Image information FR of the second imaging device 11b capturing an image of the area diagonally forward to the right of the vehicle V is displayed on the second monitor 51b on the right, and image information FL of the third imaging device 11c capturing an image of the area diagonally forward to the left of the vehicle V is displayed on the third monitor 51c on the left.

[0065] Image information BM of the fourth imaging device 11d capturing an image of the rear of the vehicle V is superimposed on image information FM capturing an image of the front so as to be displayed in a reduced state in the upper part of the first monitor 51a. Image information BR of the fifth imaging device 11e capturing an image of the right rear of the vehicle V is superimposed on image information FR so as to be displayed in a reduced state in the upper left part of the second monitor 51b. Image information BL of the sixth imaging device 11f capturing an image of the left rear of the vehicle V is superimposed on image information FL so as to be displayed in a reduced state in the upper right part of the third monitor 51c.

[0066] Moreover, image information FMs of the seventh imaging device 11g, which is a sub-camera, is superimposed on the image information FM so as to be displayed in a reduced state on the lower side of the first monitor 51a. The image information BRs of the eighth imaging device 11h is superimposed on the image information FR so as to be displayed in a reduced size in the upper right portion of the second monitor 51b. The image information BLs of the ninth imaging device 11i is superimposed on the image information FR so as to be displayed in a reduced state in the upper left portion of the third monitor 51c.

[0067] FIG. 5B shows another example of a "composite image." The display positions of the image information FM, FL, and FR captured by the first imaging device 11a to the third imaging device 11c are the same, but the range for displaying other image information FMs, BM, BL, BR, BLs, and BRs is set to be larger than the "composite image" shown in Figure 5A.

[0068] FIG. 5C is an example of a "synthetic image" generated when reversing. When the travel direction information is rearward, that is, when driving backward, only the first monitor 51a may be used to display the rear of the vehicle V. In this case, the image information BM acquired by the fourth imaging device 11d is arranged to be displayed on the first monitor 51a. The image information FM of the front acquired by the first imaging device 11a is superimposed on the image information BM so as to be displayed in a reduced state on the upper part of the first monitor 51a. When the front image information FM is not necessary, the setting may be such that only the rear image information BM is displayed. In this way, information regarding the size and position for displaying the image information acquired from the imaging device in the "composite image" is set in the layout information.

[0069] The "layout information" is configured to enable the image processing unit 110 to generate different composite images according to the traveling direction of the vehicle V. For example, the layout information includes both a setting for generating a "composite image" when the vehicle V travels forward as shown in Fig. 5A and Fig. 5B, and a setting for generating a "composite image" when the vehicle V travels backward as shown in Fig. 5C. By including the forward and backward settings, the image processing unit 110 can generate a "composite image" according to the traveling direction of the vehicle V, and the display monitor 51 of the operation device 50 displays the "composite image" according to the traveling direction.

[0070] 5A to 5C are merely examples, and for example, vehicle information and the like may be displayed in a part of the display monitor 51, with image information displayed around it. Also, image information BR, BL showing the situations on the left and right sides of the vehicle may be set to be displayed in the center of the display monitor 51 so that the surrounding situation can be confirmed while the vehicle is stopped.

[0071] In this embodiment, the first imaging device 11a, the second imaging device 11b, and the third imaging device 11c, which image the front, capture images so that the bonnet and fenders of the vehicle V are included in the image information. The image processing unit 110 uses the image information including part of the vehicle V to generate a "composite image" in a state in which part of the vehicle V is displayed. By including part of the vehicle V in the "composite image," the operator H can grasp the size and position of the vehicle body, and can remotely operate the vehicle V by using part of the vehicle V in the "composite image" as a landmark.

[0072] By generating the above-mentioned composite video and transmitting the generated composite video data to the operation device 50, the amount of data to be transmitted can be reduced compared to the case of transmitting a plurality of external video data, and the cost of data communication can be reduced. In other words, since the amount of data is small, it is easier to secure the bandwidth required for transmission, and transmission can be performed using a smaller number of lines.

[0073] The function of the operation device 50 will now be described. <Storage section 500> The storage unit 500 stores various data and programs, such as the "synthetic image" received from the vehicle control device 1, information on the traveling direction of the vehicle V, and "layout information" input by the input unit 503. Note that the various data, such as the layout information, may be stored in an external server provided in the network N and transmitted and received by a WEB browser or a WEB application, etc.

[0074] <Communication Unit 501> The communication unit 501 (second communication unit) receives information from the vehicle control device 1. In this embodiment, the communication unit 501 receives a “synthetic image” and “position information” of the vehicle V from the vehicle control device 1. The communication unit 108 is realized by the CPU of the operation device 50 and the communication device 60.

[0075] <Display section 502> The display unit 502 displays the information received by the communication unit 501 on the display monitor 51 or the navigation display monitor 52. The display unit 502 displays the “synthetic image” received from the vehicle control device 1 on the display monitor 51, and displays the position of the vehicle V on the navigation display monitor 52 by superimposing it on the map information.

[0076] <Input section 503> The input unit 503 processes the layout information by the operator H. The input layout information is transmitted to the vehicle control device 1 by the communication unit 501. When there are multiple pieces of layout information and corresponding layout IDs are defined, the layout IDs may be transmitted.

[0077] <Communication control unit 504> The communication control unit 504 establishes a communication path with the communication unit 108 (first communication unit) of the vehicle control device 1 when the communication unit 501 receives information related to the “synthetic image” from the vehicle control device 1. The vehicle remote operation system is composed of a plurality of vehicles V and a plurality of operation devices 50. When remotely operating the vehicle V, one operation device 50 is selected from the plurality of operation devices 50. At that time, a communication path must be maintained for the connection with the operation device 50 connected to the vehicle V until the remote operation is completed, and this communication path is secured as the highest priority communication destination. As a means for securing this communication path, the vehicle V can be connected to a plurality of communication lines using a plurality of communication devices.

[0078] <Audio output unit 505> The audio output unit 505 processes audio information received from the vehicle control device 1. The audio information is output from the speaker 59.

[0079] <<Vehicle remote control method / program>> Next, a vehicle remote control method (vehicle remote control program) executed by the vehicle control device 1 and the operation device 50 will be described with reference to a sequence diagram of FIG. The vehicle remote control method according to this embodiment is executed by a program executed by the CPU (first computer) of the vehicle control device 1 and the CPU (second computer) of the operation device 50. That is, the method is executed by a utility program that consolidates various programs that realize the above-mentioned functions as functional components of the vehicle control device 1 and the operation device 50. The above method and program are executed upon receiving an operation instruction from an operator H of the vehicle remote control system.

[0080] When remote operation is started, the operator H of the operation device 50 inputs "layout information" used to generate a "composite image" to be displayed on the display monitor 51 (layout information input step S101). The "layout information" is input by the operator H via, for example, the touch panel 57. The storage unit 500 temporarily stores the "layout information". The "layout information" may be selected by the operator H from a predetermined form using a plurality of switches at hand. If there is a form that the operator H always uses and that is stored in the storage unit 100 of the vehicle control device 1, the form may be automatically set when the operator H is identified. Also, if a predetermined form is stored in the vehicle control device 1 or the operation device 50, this step may be omitted.

[0081] After the "layout information" is input by the input unit 503, the operation device 50 transmits the "layout information" to the vehicle control device 1 by the communication unit 501 before starting remote driving (layout information transmission step S102). The communication unit 108 of the vehicle control device 1 receives the "layout information" together with an instruction to start remote operation via the network N (layout information reception step S201).

[0082] After receiving the "layout information", the storage unit 100 of the vehicle control device 1 stores the "layout information" (layout information storage step S202). Next, the traveling direction acquisition unit 111 acquires information on the traveling direction of the vehicle V. This is because the image displayed on the display monitor 51 of the operation device 50 differs depending on the traveling direction. The traveling direction is determined based on the position of the shift position of the vehicle V (traveling direction acquisition step S203).

[0083] Next, the image processing unit 110 of the vehicle control device 1 acquires image information captured by the nine imaging devices 11 mounted on the vehicle V by the external information acquisition unit 101 (image information acquisition step S204), and synthesizes the multiple image information based on the layout information stored in the storage unit 100 to generate a composite image (composite image generation step S205). At this time, the layout information for generating the composite image is switched depending on the traveling direction of the vehicle V. For example, when the vehicle V travels forward, a composite image is generated with the images of the first imaging device 11a to the third imaging device 11c capturing the image in front of the vehicle as the main image (see FIG. 5A and FIG. 5B). When the vehicle V travels backward, a composite image is generated with the image captured by the fourth imaging device 11d capturing the image behind the vehicle as the main image (see FIG. 5C). Since the image processing unit 110 of the vehicle control device 1 generates and transmits the composite image, the resolution of the image to be transmitted is constant and the amount of data required for transmission does not increase, making it easier to secure the bandwidth of the communication line required for transmission.

[0084] Next, the vehicle control device 1 transmits the "composite image" generated by the image processing unit 110 to the operation device 50 using the communication unit 108 (composite image transmission step S206). At this time, in order to reduce the data amount of the "composite image", encoding of the "composite image" may be executed. The communication unit 108 transmits the encoded "composite image" to the operation device 50.

[0085] In this case, the vehicle control device 1 may include an encoder that performs encoding using hardware. The encoder encodes the "composite image" according to a video encoding method, for example. Examples of the video encoding method include MPEG (Moving Picture Expert Group)-2, H.264, H.265, VP8, VP9, ​​and AOMedia Video 1 (AV1). The encoder generates encoded video data from the "composite image" by hardware encoding. However, the encoder may generate video data by software encoding.

[0086] Next, at the timing when the communication unit 501 of the operation device 50 receives the "synthetic image" from the vehicle control device 1, the communication control unit 504 establishes a communication path with the communication unit 108 of the vehicle control device 1 (communication path establishment step S103). By establishing the communication path, the communication path is maintained until the remote operation of the vehicle V is completed, and the communication path is secured as the highest priority passage destination. The communication control unit 504 may establish the communication path when the remote operation is started or when the operation device 50 transmits the “layout information” to the vehicle control device 1.

[0087] After the communication unit 501 of the operation device 50 establishes a communication path with the communication unit 108 of the vehicle control device 1, the operation device 50 receives the "composite image" (composite image receiving step S104). The received "composite image" is displayed on the display monitor 51 by the display unit 502 (composite image display step S105). If the received "composite image" is encoded video data, the display unit 502 decodes the video data according to the video encoding method used and displays it. Conventionally, when image information is directly received from a plurality of imaging devices, the operation device 50 arranges the image information on the operation device 50 side and displays it on a display monitor. Since the operation device 50 receives and displays the "composite image" generated by the vehicle control device 1, the number of communication lines required for transmission is small, and the amount of data received can also be reduced. Since the "composite image" generated by the vehicle control device 1 is received, it can be received reliably without any missing images. In addition, since the "composite image" according to the traveling direction is received, the "composite image" can be automatically displayed in an appropriate layout by switching the traveling direction.

[0088] <<Resolution switching>> In the vehicle remote operation system S, the "composite image" transmitted from the vehicle V is displayed on the display monitor 51, allowing the operator H to remotely operate the vehicle V while visually checking the situation around the vehicle. For this reason, the vehicle control device 1 of the vehicle V needs to transmit the "composite image" to the operation device 50 as much as possible. For this reason, the vehicle control device 1 has an adjustment function that changes the resolution of the generated "composite image" so that it can be transmitted even if there is a change in the communication environment. The flow of changing the resolution according to the communication environment will be described with reference to FIG. 7.

[0089] First, the communication unit 108 of the vehicle control device 1 checks the communication state of the three communication devices 41 (the first communication device 41a to the third communication device 41c) of the in-vehicle communication device 40 connected by the first communication line, which is a 4G communication line. Specifically, the radio wave strength of each of the three communication lines is measured. If the radio wave strength of the communication line is greater than a predetermined threshold value α, it is determined that a sufficient bandwidth is secured to transmit the "composite image" (Yes in S302). The communication unit 108 transmits the "composite image" to the operation device 50 as a high-resolution full-page image without reducing it (S303).

[0090] If the radio wave intensity of any of the communication devices 41 is lower than the threshold value α, the connection with the wireless base station RB may be cut off, and communication may not be possible. If the result of S302 is No, it is determined whether the three communication devices 41 are likely to change the wireless base station RB to which they are connected (S304). If the three communication devices 41 are likely to change the wireless base station RB (Yes in S0304), if the three communication devices 41 change the wireless base station RB simultaneously, there is a possibility that a communication path with the operation device 50 may not be secured at that moment. Therefore, the communication unit 108 performs a process of changing the timing of line switching so that the three communication devices 41 do not switch the wireless base station RB simultaneously (S305).

[0091] Next, the vehicle control device 1 measures the number of lines that can be connected using the 4G line (first communication line) (S306). When one communication line is disconnected and communication is possible using two communication lines (Yes in S307), the bandwidth for transmitting the "composite image" is reduced compared to when three communication lines are connected, so a "composite image" with a lower resolution than the "composite image" before the change is generated and transmitted so as to be displayed on three screens (S308).

[0092] The two communication lines are disconnected, and it is determined whether connection is possible in the first round (S309). If communication is possible with only one communication line (Yes in S309), the bandwidth available for transmitting the "composite image" is further reduced, so a "composite image" with a lower resolution than the "composite image" before the change is generated and transmitted so as to be displayed on one screen (S310).

[0093] If a 4G communication line cannot be connected using any of the communication lines (No in S309), an attempt is made to connect using a 3G communication line (S311). If a 3G communication line can be connected (Yes in S311), a single screen image with even higher latency and lower resolution is transmitted (S312). If a 3G communication line cannot be connected (No in S311), a communication path cannot be secured and a "composite image" cannot be transmitted, so image transmission is stopped and remote control is discontinued (S313).

[0094] If the number of 4G lines that can be connected increases from two to three as vehicle V moves, the transmission bandwidth increases, and a higher-resolution "composite image" is transmitted compared to the "composite image" before the change. The same applies when the number of lines increases from one to two or three. Similarly, when switching from a 3G communication line to a 4G communication line, a higher-resolution "composite image" is transmitted.

[0095] In this way, by using multiple wireless communication lines and adjusting the quality of the "composite image" of the surroundings of the vehicle that is transmitted according to the number of connected lines, the vehicle control device 1 of the vehicle V is able to transmit the "composite image" to the operating device 50 as much as possible.

[0096] <Other embodiments> In the above embodiment, as shown in FIG. 2, the vehicle remote control system S includes a vehicle control device 1 and an on-board ECU 30, and newly adds a "remote driving function (vehicle control device 1)" to a vehicle V equipped with an "automatic driving function (on-board ECU 30)", but this can be modified without any particular limitations. For example, the vehicle control device 1 may also have the function of the in-vehicle ECU 30. That is, the vehicle remote control system S may be mainly composed of the vehicle control device 1 (including the function of the in-vehicle ECU 30), the in-vehicle sensor 10, the in-vehicle locator 20, the in-vehicle communication device 40, and the operation device 50 (the in-vehicle ECU 30 may be excluded from the configuration).

[0097] In the above embodiment, as shown in FIG. 2, the vehicle remote control system S includes a vehicle control device 1 and an on-board locator 20, but is not particularly limited, and the vehicle control device 1 may also include the on-board locator 20.

[0098] In the above embodiment, a vehicle remote operation program is stored in a recording medium readable by the vehicle control device 1, and the vehicle control device 1 reads and executes the program to execute processing. Here, the recording medium readable by the vehicle control device 1 refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Alternatively, a dedicated web application may be started using a terminal (mobile terminal) serving as the vehicle control device 1, and the vehicle remote control program may be executed on a web browser.

[0099] In the above embodiment, the vehicle remote control system and the vehicle remote control method according to the present invention have been mainly described. According to the vehicle remote control system S, image information is acquired from each of the multiple imaging devices 11, and the image processing unit 110 of the vehicle control device 1 generates a composite image based on the layout information and transmits it to the operation device 50, thereby reducing the amount of data to be transmitted compared to the case of transmitting all captured image information. Therefore, the bandwidth required for transmitting the composite image is constant, so image information of the surroundings of the vehicle can be transmitted without increasing communication costs. In addition, it becomes easier to secure the bandwidth required for communication, and transmission can be more reliable. In addition, since the composite image synthesized on the vehicle side is transmitted to the operation device 50, there is no loss of image due to a partial disconnection of the communication line. The above embodiment is merely an example for facilitating understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and the present invention naturally includes equivalents thereof. [Explanation of symbols]

[0100] S Vehicle remote control system V Vehicle V1 Electric Power Steering V1a Handle V2 Electric Throttle V2a Accelerator Pedal V3 Electromagnetic Brake Device V3a Brake Pedal 1 Vehicle control device 10. Vehicle Sensors 11 Imaging device 11a to 11i: 1st imaging device to 9th imaging device 12 Radar (millimeter wave radar) 12a~12d Radar 1~4 13 Rider 13a~13e 1st Rider~5th Rider 20 Vehicle Locator 21 GNSS receiver (RTK-GNSS receiver) 22 Inertial Measurement Unit (IMU) 30 Automotive ECU 31 Integrated ECU 32 Steering ECU 33 Accelerator ECU 34 Brake ECU 40 In-vehicle communication device 41, 41a, 41b, 41c communication device, first communication device, second communication device, third communication device 50 Operating device 51 Display Monitor 51a, 51b, 51c 1st monitor, 2nd monitor, 3rd monitor 52 Navigation display monitor 53 Handle 54 Accelerator pedal 55 Brake pedal 56 Operation switch 57 Touch Panel 58 Mike 59 Speaker 60 Communication Equipment 61 Chair 100 Storage section 101 External information acquisition department 107 Location identification part 108 Communications Department 109 Vehicle control unit 110 Image processing section 500 storage section 501 Communications Department 502 Display section 503 Input section 504 Communication control section 505 Audio output section SA satellite ST reference station RB wireless base station

Claims

1. A vehicle control device that controls the traveling of the vehicle based on information on a current position of the vehicle; An on-board sensor that detects an external environment around the vehicle, The vehicle control device includes: an absolute position calculation unit that acquires GNSS information required for single point positioning through a GNSS receiver mounted on the vehicle and calculates an absolute position of the vehicle by single point positioning; a reception determination unit that determines whether GNSS correction information required for relative positioning can be received from an external reference station; a relative position calculation unit that calculates a relative position of the vehicle by relative positioning when it is determined that the GNSS correction information can be received; a vehicle control unit that controls driving of the vehicle based on information on a current position of the vehicle based on the relative position and detection information of an external environment of the vehicle obtained by the on-board sensor.

2. A vehicle control method using a vehicle equipped with a vehicle control device and an on-board sensor, The vehicle control device controls the traveling of the vehicle based on information on a current position of the vehicle, acquiring GNSS information required for single point positioning through a GNSS receiver mounted on the vehicle, and calculating an absolute position of the vehicle through single point positioning; A step of determining whether GNSS correction information required for relative positioning can be received from an external reference station; calculating a relative position of the vehicle by relative positioning when it is determined that the GNSS correction information can be received; and controlling the running of the vehicle. The on-board sensor detects an external environment around the vehicle, A vehicle control method, in which in the step of controlling the driving of the vehicle, the driving of the vehicle is controlled based on information on the current position of the vehicle based on the relative position and detection information of the external environment of the vehicle obtained by the on-board sensor.

3. A computer as a vehicle control device that is mounted on a vehicle and controls the traveling of the vehicle based on information on a current position of the vehicle, A process of acquiring GNSS information required for independent positioning through a GNSS receiver mounted on the vehicle and calculating an absolute position of the vehicle by independent positioning; A process of determining whether GNSS correction information required for relative positioning can be received from an external reference station; A process of calculating a relative position of the vehicle by relative positioning when it is determined that the GNSS correction information can be received; and controlling the driving of the vehicle based on information on the current position of the vehicle based on the relative position and detection information of the external environment of the vehicle obtained by an on-board sensor mounted on the vehicle that detects the external environment around the vehicle.

Images