Vehicles, systems, and programs

Abstract

The present invention provides a vehicle equipped with a vehicle control device capable of measuring the current position of a moving vehicle and controlling the vehicle's movement. [Solution] The vehicle V comprises a vehicle control device 1 that controls the vehicle's movement based on information about the vehicle's current position, and an on-board sensor 10 that detects the external environment around the vehicle. The vehicle control device 1 comprises an absolute position calculation unit that acquires GNSS information necessary for standalone positioning from an artificial satellite SA and calculates the absolute position of the vehicle, a reception determination unit that determines whether or not it can receive GNSS correction information necessary for relative positioning from a reference station ST, a relative position calculation unit that calculates the relative position of the vehicle by relative positioning, and a vehicle control unit that controls the vehicle's movement based on information about the vehicle's current position based on the relative position and detection information of the vehicle's external environment obtained by the on-board sensor 10.

Description

Technical Field

[0001] The present invention relates to a vehicle, a system, and a program.

Background Art

[0002] In recent years, in order to realize the safety and comfort of drivers, vehicles for autonomous driving equipped with an ADAS (Advanced Driving Assistance System) that enables the vehicle itself to grasp information on the surrounding external environment and control the running of the vehicle on behalf of the driver are known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a malfunction occurs in a generally known vehicle and a driving obstacle occurs, Patent Document 1 discloses a remote driving technology for operating the running of the vehicle by communication via a network in order to safely stop or continue running.

[0005] An object of the present invention is to provide a vehicle, a system, and a program capable of ensuring the reliability of communication between a vehicle and an operating device.

Means for Solving the Problems

[0006] The aforementioned problem is solved by the vehicle of the present invention, which is a vehicle connected to an operating device via a network, comprising: a plurality of imaging devices that image the area around the vehicle; an on-board communication device composed of a plurality of communication devices, each of which is connected to a wireless base station via a wireless communication line; and a vehicle control device, wherein the vehicle control device comprises an image processing unit that synthesizes image information acquired from each of the plurality of imaging devices to generate a composite image; and a communication unit that transmits the composite image to the operating device using the on-board communication device, wherein the communication unit measures the radio wave strength of the wireless communication line to which each of the plurality of communication devices is connected, and the vehicle control device changes the resolution of the composite image generated by the image processing unit according to the number of communication lines to which the plurality of communication devices can be connected, based on the radio wave strength. Furthermore, the vehicle comprises a vehicle control device that controls the vehicle's movement based on information about the vehicle's current position, and an on-board sensor that detects the external environment around the vehicle. The vehicle control device comprises an absolute position calculation unit that acquires GNSS information necessary for standalone positioning through a GNSS receiver mounted on the vehicle and calculates the absolute position of the vehicle by standalone positioning, a reception determination unit that determines whether or not it can receive GNSS correction information necessary for relative positioning from an external reference station, a relative position calculation unit that calculates the relative position of the vehicle by relative positioning if it is determined that the GNSS correction information can be received, and a vehicle control unit that controls the vehicle's movement based on information about the vehicle's current position based on the relative position and detection information of the vehicle's external environment obtained by the on-board sensor.

[0007] Furthermore, the present invention provides a system using a vehicle that comprises a plurality of imaging devices, an in-vehicle communication device composed of a plurality of communication devices, and a vehicle control device, and is connected to an operating device via a network, wherein the vehicle control device performs the following processes: acquiring image information from each of the plurality of imaging devices; combining the acquired image information to generate a composite image; transmitting the composite image to the operating device using the in-vehicle communication device; measuring the radio wave strength of the wireless communication line to which each of the plurality of communication devices is connected; and changing the resolution of the composite image according to the number of communication lines to which the plurality of communication devices can be connected, based on the radio wave strength. Furthermore, according to the program of the present invention, a computer acting as a vehicle control device mounted in a vehicle, which is connected to an operating device via a network and includes a plurality of imaging devices and an in-vehicle communication device composed of a plurality of communication devices, is made to execute the following processes: acquiring image information from each of the plurality of imaging devices; combining the acquired image information to generate a composite image; transmitting the composite image to the operating device using the in-vehicle communication device; measuring the radio wave strength of the wireless communication line to which each of the plurality of communication devices is connected; and changing the resolution of the composite image according to the number of communication lines to which the plurality of communication devices can be connected, based on the radio wave strength. Furthermore, the vehicle control method is a vehicle control method using a vehicle equipped with a vehicle control device and an on-board sensor, wherein the vehicle control device, which controls the driving of the vehicle based on information of the vehicle's current position, performs the steps of: acquiring GNSS information necessary for standalone positioning through a GNSS receiver mounted on the vehicle and calculating the absolute position of the vehicle by standalone positioning; determining whether or not it can receive GNSS correction information necessary for relative positioning from an external reference station; calculating the relative position of the vehicle by relative positioning if it is determined that the GNSS correction information can be received; and controlling the driving of the vehicle, wherein the on-board sensor performs the step of detecting the external environment around the vehicle, and in the step of controlling the driving of the vehicle, the driving of the vehicle is controlled based on information of the vehicle's current position based on the relative position and detection information of the external environment of the vehicle obtained by the on-board sensor. Furthermore, the vehicle control program causes a computer, which is mounted on the vehicle and acts as a vehicle control device that controls the vehicle's movement based on information about the vehicle's current position, to execute the following processes: acquiring GNSS information necessary for standalone positioning through a GNSS receiver mounted on the vehicle and calculating the absolute position of the vehicle by standalone positioning; determining whether or not GNSS correction information necessary for relative positioning can be received from an external reference station; if it is determined that the GNSS correction information can be received, calculating the relative position of the vehicle by relative positioning; and controlling the vehicle's movement based on information about the vehicle's current position based on the relative position and detection information of the vehicle's external environment obtained by an on-board sensor mounted on the vehicle that detects the external environment around the vehicle. [Effects of the Invention]

[0008] According to the vehicle, system, and program of the present invention, it is possible to ensure the reliability of communication between the vehicle and the operating device. [Brief explanation of the drawing]

[0009] [Figure 1] This is a configuration diagram showing the entire vehicle remote control system of this embodiment. [Figure 2] This diagram shows the hardware configuration of the vehicle remote control system (excluding the control device). [Figure 3] This diagram shows the hardware configuration of the control device. [Figure 4] This diagram shows the functions of the vehicle control device and operating device. [Figure 5A] This is an explanatory diagram showing a composite image displayed on the control device's display monitor, illustrating examples of the position and size of image information acquired by multiple imaging devices. [Figure 5B] This is an explanatory diagram showing another example of a composite image. [Figure 5C] This is an explanatory diagram showing an example of a composite image displayed on the control panel's display monitor when driving in reverse. [Figure 6] This is a sequence diagram showing the process of displaying a composite image. [Figure 7] This flowchart illustrates the process of selecting the resolution of the composite image to be transmitted based on the communication status. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to Figures 1 to 7. As shown in Figure 1, the vehicle remote control system S of this embodiment is a system that realizes "automatic driving," which involves understanding the external environment of a moving vehicle V, planning the driving path of the vehicle V on behalf of the driver, and controlling the vehicle V based on the driving path, and "remote driving," which involves an operator H outside the vehicle V remotely controlling (external operation) the vehicle V to drive it. In addition, the vehicle remote control system S is capable of performing a "mode switching process" to switch between automatic driving mode and remote driving mode. There is also a manual driving mode in which the driver operates the vehicle V while riding in it (details will be described later), and in the mode switching process described above, it is possible to switch between this manual driving mode and automatic driving mode, as well as between manual driving mode and remote driving mode. Note that Operator H does not have to be a human; for example, it could be an AI (artificial intelligence).

[0011] <<Hardware configuration of the vehicle remote control system S>> As shown in Figure 1, the vehicle remote control system S mainly consists of a vehicle control device 1 and an operating device 50 for the vehicle V. The vehicle control device 1 is mounted on the vehicle V and comprehensively controls the vehicle's movement. The operating device 50 is connected to the vehicle control device 1 via a network N (digital communication channel) and operates the vehicle V through communication via the network N. Of course, the vehicle control device 1 and the operating device 50 may also communicate directly.

[0012] Vehicle V is equipped with an on-board sensor 10, an on-board locator 20, an on-board ECU 30, and an on-board communication device 40. The on-board sensor 10 is a device that detects the external environment around vehicle V. The on-board locator 20 is a device that receives GNSS signals from the artificial satellite SA and the reference station ST and measures the current position of vehicle V. The on-board ECU 30 is a device that controls the steering, acceleration, deceleration, etc., of vehicle V. The on-board communication device 40 is a device that communicates with an operating device 50 or external equipment installed outside of vehicle V. As shown in Figure 2, the vehicle control device 1, onboard sensor 10, onboard locator 20, onboard ECU 30, and onboard communication device 40 mounted on the vehicle V are connected via an onboard network (CAN: Controller Area Network). More specifically, the in-vehicle locator 20, the in-vehicle communication device 40, the imaging device 11 and lidar 13 of the in-vehicle sensor 10 are connected to the vehicle control device 1 via serial communication using communication interfaces such as RS232C connectors and USB (Universal Serial Bus) connectors.

[0013] The vehicle control device 1 is a computer (first computer) equipped with a CPU as a data calculation and control processing device, ROM, RAM and HDD (SSD) as storage devices, and a communication interface for sending and receiving information data via CAN, etc. In the storage device of the vehicle control device 1, in addition to the main program that performs the functions necessary for a computer, a vehicle control program and a vehicle remote operation program are stored. When these programs are executed by the CPU, the functions of the vehicle control device 1 are exerted.

[0014] In order to execute "automatic driving", the vehicle control device 1 controls the in-vehicle ECU 30 (more specifically, the integrated ECU 31) based on the information on the external environment obtained from the in-vehicle sensor 10 and the information on the current position obtained from the in-vehicle locator 20, thereby controlling the running of the vehicle V. In addition, in order to execute "remote driving", the vehicle control device 1 is connected to the operating device 50 through the in-vehicle communication device 40 and via a network N (digital communication path) including a wireless communication line or the like, and transmits the information on the external environment and the information on the current position to the operating device 50. In other words, the vehicle control device 1 is newly installed in the vehicle V that has been pre-installed with the "automatic driving function (in-vehicle sensor 10, in-vehicle locator 20, in-vehicle ECU 30)", thereby enhancing the performance of the existing "automatic driving function" and newly imparting the "remote driving function" to the installed vehicle V. Although details will be described later, the operating device 50 receives the information on the external environment and the information on the current position necessary for the remote driving function from the vehicle control device 1, displays the information on the external environment on the display monitor 51, and can notify the operator H of the operating device 50 of the information and the like.

[0015] The in-vehicle sensor 10, in-vehicle locator 20, and in-vehicle ECU 30 that realize the "automatic driving function" will be described.

[0016] <In-vehicle sensor 10> The on-board sensor 10 is a device that detects the external environment surrounding the vehicle V, including moving objects around the vehicle V (other vehicles, pedestrians, etc.), various structures, road shapes, etc. (hereinafter sometimes referred to as target objects). Specifically, as shown in Figure 2, it consists of multiple imaging devices 11, multiple radars 12, and multiple lidars 13 installed on the vehicle body. The on-board sensor 10 may also have sensors other than the imaging devices 11, radars 12, and lidars 13.

[0017] The imaging device 11 is a wide-angle camera that captures images of the area around the vehicle V and acquires them as image information. For example, it is a small digital camera that uses a solid-state image sensor such as a CCD or CMOS. The imaging device 11 creates external video data to perform a "sensing function" for controlling the vehicle V's movement and a "monitoring function" for the driver (operator H), and transmits the external video data (hereinafter referred to as "image information") to the vehicle control device 1, which is connected via a USB cable, via serial communication.

[0018] As shown in Figure 2, the imaging device 11 consists of nine imaging devices (first imaging device 11a to ninth imaging device 11i, hereinafter sometimes collectively referred to as the imaging device 11) mounted in predetermined positions. In this embodiment, the first imaging device 11a, the second imaging device 11b, and the third imaging device 11c capture images of the front, the right front, and the left front of the vehicle V, and they are mounted on the interior side of the front windshield (front window) at the front of the roof of the vehicle V. The fourth imaging device 11d captures images of the rear of vehicle V and is mounted in the center of the rear bumper (back bumper) of vehicle V. The fifth imaging device 11e and the sixth imaging device 11f are mounted on the side mirrors on the left and right sides of vehicle V and capture images of the right rear and left rear of vehicle V. The first imaging devices 11a to the sixth imaging devices 11f are main cameras that capture images of the main parts around vehicle V.

[0019] Furthermore, the imaging device 11 includes a seventh imaging device 11g, an eighth imaging device 11h, and a ninth imaging device 11i as sub-cameras. The seventh imaging device 11g is mounted on the front bumper of the vehicle V and images the area in front of the vehicle V. The eighth imaging device 11h and the ninth imaging device 11i are mounted around the left and right taillights of the vehicle V and image the area diagonally to the right rear and diagonally to the left rear of the vehicle V. The sub-cameras, the seventh imaging device 11g, the eighth imaging device 11h, and the ninth imaging device 11i, image areas that are blind spots and cannot be captured by the first imaging devices 11a to the sixth imaging devices 11f of the main camera. As an alternative example of a sub-camera, the seventh imaging device 11g may be mounted on the upper part of the rear window of the vehicle V, and imaging the area behind the vehicle V from that position. Alternatively, the eighth imaging device 11h may be mounted on the right side of the vehicle V from the A-pillar to the C-pillar, and the ninth imaging device 11i may be mounted on the left side from the A-pillar to the C-pillar.

[0020] As described above, the first imaging device 11a to the ninth imaging device 11i are connected to the vehicle control device 1 via serial communication and are configured to transmit captured image information to the vehicle control device 1. The number and position of the imaging devices 11 attached to the vehicle V in this embodiment is just an example, and the number and position can be changed depending on the shape or use of the vehicle V. The vehicle control device 1 may also analyze the image information captured by the imaging device 11 to extract the outlines of target objects around the vehicle V, or to extract lane markings (white lines, etc.) on the road.

[0021] Radar 12 is a sensor that performs three-dimensional spatial imaging by emitting radio waves while continuously changing the direction of illumination, detecting an object by receiving reflected waves from the object, and measuring the position and velocity of the object. In this embodiment, vehicle V uses a millimeter-wave radar that uses millimeter waves in the 30-300 GHz band as radio waves. By using millimeter waves, radar 12 can detect an object with higher accuracy even in environmental conditions such as nighttime or bad weather with poor visibility, compared to imaging device 11 and lidar 13.

[0022] The radar 12 acquires the detection result data (detection signal) of the target object and transmits the detection result data to the vehicle control device 1 via CAN. Alternatively, the radar 12 and the vehicle control device 1 may be connected by a USB cable, and the radar 12 may transmit the detection result data to the vehicle control device 1 via serial communication. As shown in Figure 2, the radar 12 consists of four radars 12 (first radar 12a to fourth radar 12d) mounted at predetermined positions on the vehicle V. The first radar 12a and the second radar 12b are mounted around the left and right front lights of the vehicle V, and the third radar 12c and the fourth radar 12d are mounted around the left and right rear lights of the vehicle V. Note that radar 12 is not particularly limited to millimeter-wave radar, and may be laser radar, ultrasonic sensor, or other types of radar.

[0023] The lidar 13 is a remote sensor that measures the distance to an object by irradiating it with laser light and receiving the reflected light from the object, thereby performing three-dimensional spatial imaging. Compared to the imaging device 11 and radar 12, it can measure the distance to surrounding objects in units of several centimeters. The lidar 13 acquires distance measurement data by measuring the distance to the target object. The lidar 13 is connected to the vehicle control unit 1 by a USB cable and transmits the distance measurement data to the vehicle control unit 1 via serial communication. If the lidar 13 is connected to the vehicle control unit 1 via CAN, it may also transmit the distance measurement data to the vehicle control unit 1 via CAN. As shown in Figure 2, the rider 13 consists of five riders 13 (first rider 13a to fifth rider 13e) mounted at predetermined positions on the vehicle V. The first rider 13a and the second rider 13b are mounted around the left and right front lights of vehicle V. The third rider 13c is mounted on the rear bumper of vehicle V. The fourth rider 13d and the fifth rider 13e are mounted around the left and right taillights of vehicle V. The fourth rider 13d and the fifth rider 13e may also be mounted around the rear lights of vehicle V.

[0024] Similar to the imaging device 11, the number and mounting positions of the radar 12 and lidar 13 are examples only, and the number and positions of the units to be mounted can be changed.

[0025] <Vehicle-mounted locator 20> The on-board locator 20 measures the current position of the vehicle V using a satellite positioning system that utilizes the artificial satellite SA and the reference station ST, and also measures the acceleration and angular velocity of the vehicle V in order to improve the accuracy of the current position measurement. Specifically, the vehicle-mounted locator 20 consists of a GNSS receiver 21 that receives GNSS radio waves (Global Navigation Satellite System, GPS radio waves) from multiple artificial satellites SA, and an inertial measuring device 22 that measures the acceleration and angular velocity of the vehicle V.

[0026] Specifically, the GNSS receiver 21 is an RTK-GNSS receiver that receives GNSS radio waves from multiple (specifically four) artificial satellites SA, as shown in Figure 1, and generates "GNSS information" necessary for standalone positioning. It also receives "GNSS correction information" necessary for relative positioning from an external reference station ST. The reference station ST is a fixed reference station set at a known point, which receives GNSS radio waves from multiple satellites SA, generates "GNSS correction information," and transmits it to the GNSS receiver 21 of the vehicle-mounted locator 20. "GNSS information" refers to distance information between multiple artificial satellites SA and the GNSS receiver 21, while "GNSS correction information" refers to distance information obtained by correcting the measurement error of the "GNSS information" through communication between the reference station ST, located at a known point, and the reference station ST.

[0027] The inertial measurement device 22, also known as an IMU (Inertial Measurement Unit), comprises a 3-axis gyro sensor (angular velocity meter) and a 3-axis acceleration sensor (accelerometer). The inertial measurement device 22 measures the three-dimensional angular velocity and acceleration of the vehicle V and transmits the acceleration and angular velocity information of the vehicle V to the vehicle control device 1. The vehicle control device 1 can measure the current position of the vehicle V with a smaller error range by combining the GNSS information (GNSS correction information) received from the GNSS receiver 21 and the angular velocity and acceleration information of the vehicle V received from the inertial measuring device 22.

[0028] <In-vehicle ECU30> The on-board ECU 30 is, for example, an ADAS ECU and is connected to the vehicle control device 1, and includes a higher-level integrated ECU 31 that transmits and receives various data. The on-board ECU 30 also includes a steering wheel ECU 32, an accelerator ECU 33, and a brake ECU 34 as lower-level components that subdivide and control the steering and acceleration / deceleration of the vehicle V. Each of the steering wheel ECU 32, accelerator ECU 33, and brake ECU 34 is connected to the higher-level integrated ECU 31. As shown in Figure 2, a hierarchical structure is formed by the integrated ECU 31, the steering wheel ECU 32, the accelerator ECU 33, and the brake ECU 34. The steering wheel ECU32 is also called the driving support computer, and the accelerator ECU33 and brake ECU34 are also called the power management control units.

[0029] The integrated ECU 31 connects to the vehicle control unit 1 and transmits and receives various data with the vehicle control unit 1. The integrated ECU 31 is also called the driving support computer and performs the function of comprehensively controlling the steering wheel ECU 32, accelerator ECU 33, and brake ECU 34. It also performs power management (power management of various computers).

[0030] The steering ECU32, accelerator ECU33, and brake ECU34 are computers that precisely control the steering, acceleration, and deceleration of the vehicle V.

[0031] The steering wheel ECU 32 is a device that controls the direction of travel of the vehicle V by operating the electric power steering V1 of the vehicle V according to instructions from the overall ECU 31. The electric power steering V1 is equipped with a steering mechanism that steers the front wheels of the vehicle V. In manual driving, the electric power steering V1 steers the front wheels of the vehicle V in accordance with the steering input of the steering wheel V1a by the driver.

[0032] The accelerator ECU33 is a device that primarily controls the acceleration and deceleration of the vehicle V by operating the vehicle V's electric throttle V2 according to instructions from the overall ECU31. The electric throttle V2 is equipped with a drive mechanism that outputs driving force to rotate the drive wheels of vehicle V. In manual operation, the electric throttle V2 adjusts the engine output according to the driver's operation of the accelerator pedal V2a.

[0033] The brake ECU 34 is a device that controls the deceleration and stopping of vehicle V by operating the vehicle V's electromagnetic brake system V3 according to instructions from the general ECU 31. The electromagnetic brake system V3 is attached to each wheel of the vehicle V and has a mechanism that slows down or stops the vehicle V by applying resistance to the rotation of the wheels. In manual operation, the operation of the electromagnetic brake system V3 is adjusted according to the brake operation of the brake pedal V3a by the driver.

[0034] Furthermore, the number and functions of individual ECUs connected to the main ECU 31 are not particularly limited to the steering wheel ECU 32, accelerator ECU 33, and brake ECU 34, and other ECUs may be provided at the same level as these ECUs.

[0035] <In-vehicle communication device 40> This section describes the in-vehicle communication device 40 and operating device 50 that realize the "autonomous driving function". The in-vehicle communication device 40 is a device that communicates information with an operating device 50 and an external server (not shown) installed outside the vehicle V via a network N. For example, it transmits image information (composite image) and location information captured by the imaging device 11 of the vehicle control device 1 to the operating device 50 as information necessary for "remote driving," and receives driving operation information for the vehicle V from the operating device 50. Furthermore, the in-vehicle communication device 40 can communicate with an external server (not shown) to receive, for example, the latest traffic information and weather information. The in-vehicle communication device 40 consists of three wireless communication devices 41 (first communication device 41a to third communication device 41c, hereinafter sometimes collectively referred to as communication device 41), and each of the three communication devices 41 is capable of communicating with a wireless base station RB (base station).

[0036] The first to third communication devices 41a to 41c may each connect to a different radio base station RB, or they may connect to the same radio base station RB. As the vehicle V moves, the distance to the connected radio base station RB changes, and therefore the communication state also changes. If the radio wave strength falls below a predetermined threshold and it becomes impossible to secure bandwidth for transmitting image information, the first to third communication devices 41a to 41c attempt to search for and connect to another radio base station RB. If another connectable radio base station RB is found, the connection is switched. Because multiple communication devices are used to connect to the wireless base station RB, even if one communication device loses its connection to the wireless base station RB, the other communication devices remain connected to the wireless base station RB. Therefore, compared to a system where only one communication device is used, the connection is less likely to be interrupted. In this embodiment, three communication devices 41 (first communication device 41a to third communication device 41c) are used to connect to the wireless base station RB. However, the number of communication devices 41 is not limited to three; it may be two or four or more. In other words, the number of wireless communication lines connected by the communication devices 41 is not limited to three; it may be two or four or more. Furthermore, if the communication environment is good, communication may be done with just one line. Furthermore, as will be explained in more detail later, if the in-vehicle communication device 40 experiences a decline in communication conditions and is unable to secure bandwidth for transmitting the "composite image," it reduces the amount of data transmitted by lowering the image quality of the "composite image" to be transmitted, that is, by generating a "composite image" with lower resolution than the "composite image" before transmission.

[0037] Furthermore, the first to third communication devices 41a to 41c can switch between the long-distance communication lines of the fourth-generation mobile communication system (4G) and the third-generation mobile communication system (3G) to connect with the radio base station RB. Although the 3G communication line (second communication line) can be received over a wider area compared to the 4G communication line (first communication line), the 4G communication line has higher communication speed or superior communication quality. Therefore, the communication device 41 is configured to preferentially connect to the 4G communication line and communicate in an environment with low latency. If the communication device 41 cannot connect to the radio base station RB using the 4G communication line, it will use the 3G communication line to connect. In addition, if the communication device 41 can connect to the fifth-generation mobile communication system (5G) communication line, which is even faster and has higher communication quality than the 4G line, it may communicate using the 5G communication line.

[0038] Furthermore, the in-vehicle communication device 40 may be configured to connect to the network N via wireless communication lines (long-distance communication lines) of multiple telecommunications carriers. In all communication devices, if the radio wave strength of the wireless communication line being used falls below a predetermined threshold α, the in-vehicle communication device 40 measures the radio wave strength of the wireless communication lines of other telecommunications carriers, and switches the wireless communication line to which it is connected if the measured radio wave strength is equal to or greater than the threshold α. If multiple telecommunications carriers are detected, it is desirable to connect to the telecommunications carrier with the highest radio wave strength, but it may also be configured to connect to the wireless communication line of the telecommunications carrier that was detected first.

[0039] <<Hardware configuration of the operating device 50>> The operating device 50 is a device for providing remote driving services to a vehicle V that has a remote driving function, and as shown in Figure 1, it is located in a different place from the vehicle V that is remotely controlled. The operating device 50 is connected to the vehicle control device 1 of the vehicle V via a network N, and operates the vehicle V through communication via the network N.

[0040] As shown in Figure 3, the operating device 50 is a computer (second computer) equipped with a CPU as a data calculation and control processing device, ROM, RAM, and HDD (or SSD) as storage devices, and a communication interface for communicating via a network N such as the Internet or an intranet. The storage device of the operating device 50 stores, in addition to the main program that performs the functions necessary for a computer, a vehicle remote control program and an image information display program, etc. The operating device 50 functions when these programs are executed by the CPU.

[0041] The operating device 50 is connected to the communication device 60 via a communication interface. The communication device 60 is a device that communicates information with the vehicle control device 1 of the vehicle V and an external server (not shown) via a network N. The communication device 60 receives information necessary for "remote driving" transmitted from the vehicle control device 1, such as the direction of travel of the vehicle V, speed, image information of the area around the vehicle, and location information of the vehicle V, and transmits driving operation information to the vehicle control device 1. The communication device 60 may also communicate with the external server to receive traffic information and weather information for the road on which the vehicle V is traveling.

[0042] As shown in Figures 1 and 3, the operating device 50 includes a display monitor 51, a navigation display monitor 52, and a speaker 59 as output devices, which are positioned in front of the operator H when the operator H is seated in the chair 61. As shown in Figures 1 and 3, the operating device 50 is equipped with a steering wheel 53, accelerator pedal 54, brake pedal 55, operation switch 56, touch panel 57, and microphone 58 as input devices. The operator H, while seated in a chair 61, remotely controls the moving vehicle V using the steering wheel 53 and other devices while viewing the image on the display monitor 51 located in front of him.

[0043] <Display Monitor 51> The display monitor 51 is a device that outputs visual information for remote driving, and mainly displays information received from the vehicle control device 1. The information received from the vehicle control device 1 includes CAN data received from the integrated ECU, vehicle information such as speed for the meter, throttle opening, brake pedal depression amount, steering angle, electrically wired shift position, and turn signal information, and it is possible to display this information on the display monitor 51. The display monitor 51 of the operating device 50 consists of three liquid crystal monitors, as shown in Figures 1 and 3. More specifically, it consists of a first monitor 51a positioned in front of operator H, a second monitor 51b positioned adjacent to the right of the first monitor 51a, and a third monitor 51c positioned adjacent to the left of the first monitor 51a. The second monitor 51b and the third monitor 51c are tilted so that their display screens face towards operator H for easier viewing. Each of the first monitor 51a to the third monitor 51c is a liquid crystal monitor with a predetermined resolution, for example, 1920 x 1080 (Full HD, 2K). The first monitor 51a to the third monitor 51c may be high-resolution monitors with resolutions such as 4K or 8K.

[0044] The display monitor 51, consisting of three LCD monitors as shown in Figure 3, is just one example; the display monitor 51 may consist of a single LCD monitor, or two or more LCD monitors. Furthermore, the display monitor 51 may be a curved, horizontally elongated LCD monitor. The display monitor 51 may also be implemented as a mobile device such as a smartphone or tablet. The display monitor 51 may also be a VR goggle worn on the head of operator H.

[0045] <Navigation display monitor 52> The navigation display monitor 52 is a device that displays the position of vehicle V along with map information based on the position information of vehicle V received from the vehicle control device 1.

[0046] <Steering wheel 53, accelerator pedal 54, brake pedal 55> The steering wheel 53 is an operating part used to adjust the steering angle (steering amount) of the moving vehicle V. The accelerator pedal 54 is used to input the accelerator pedal opening degree of the moving vehicle V. The brake pedal 55 is used to input the brake pedal opening degree of the vehicle V. The input steering amount, accelerator pedal opening, and brake pedal opening are transmitted via the communication device 60 to the vehicle control device 1 of the vehicle V. As described above, the vehicle control device 1 of the vehicle V operates the steering ECU 32, accelerator ECU 33, and brake ECU 34 via the vehicle V's integrated ECU based on the received steering amount, accelerator pedal opening, and brake pedal opening.

[0047] <Operation switch 56> The operation switch 56 functions, for example, as a shift lever, and by pressing the switch, operator H can instruct the vehicle V to move forward or in reverse. By pressing the operation switch 56, operator H may also instruct the vehicle V's engine to start or stop, lock or unlock, honk the horn, operate the turn signals, etc.

[0048] <Touch Panel 57> The touch panel 57 is a device for inputting layout information such as the position and size of image information to be displayed on the display monitor 51. <Mike 58> Microphone 58 is a device for inputting sound, and the sound signal input by microphone 58 is transmitted to the vehicle control device 1 by the communication device 60. The sound signal can be transmitted, for example, by the speaker 59 of the vehicle V toward the occupants of the vehicle V. <Speaker 59> Speaker 59 is an acoustic device that provides auditory information. Speaker 59 can inform operator H of audio information received from vehicle control device 1, such as the voices of passengers in vehicle V and sounds around vehicle V.

[0049] Furthermore, the operating device 50 is equipped with input devices that mimic the shapes of a steering wheel 53, accelerator pedal 54, brake pedal 55, etc., so that the operator H can intuitively remotely control the vehicle V. These input devices may be game controllers, PC keyboards, or touch-enabled mobile device monitor screens. The operator H of the control device 50 is a human, but AI (artificial intelligence) may be the primary operator. In that case, the human monitors the AI's actions and the information of the vehicle V displayed on the display monitor 51, etc.

[0050] <<Functional Configuration of Vehicle Remote Control System S>> The vehicle remote control system S will be explained functionally using Figure 4. The vehicle control device 1 consists of a storage unit 100, an external information acquisition unit 101, a location identification unit 107, a communication unit 108 (first communication unit), a vehicle control unit 109, an image processing unit 110, and a direction of travel acquisition unit 111. These functions are realized by the CPU, ROM, RAM, HDD, communication interface, and various programs of the vehicle control device 1.

[0051] The operating device 50 comprises a storage unit 500, a communication unit 501, a display unit 502, an input unit 503, a communication control unit 504, and an audio output unit 505. These functions are realized by the CPU, ROM, RAM, HDD, communication interface, and various programs of the operating device 50.

[0052] The functions of the vehicle control device 1 will be explained below. <Storage section 100> The memory unit 100 stores various data and programs, such as information on the direction of travel of the vehicle V, "image information" captured by the multiple imaging devices 11, "composite images" synthesized by the image processing unit 110, information on the direction of travel of the vehicle V, and "layout information".

[0053] <External information acquisition unit 101> The external information acquisition unit 101 acquires information from the on-board sensors 10 attached to the vehicle V and stores the acquired information in the storage unit 100. More specifically, as shown in Figure 2, the on-board sensors 10 consist of nine imaging devices 11 (first imaging device 11a to ninth imaging device 11i), four radars 12 (first radar 12a to fourth radar 12d), and five lidars 13 (first lidar 13a to fifth lidar 13e) attached to the vehicle body of the vehicle V. The external information acquisition unit 101 acquires image information, target object detection result data, and distance measurement data to the target object from the on-board sensors 10 and stores them in the storage unit 100. The external information acquisition unit 101 may also acquire "vehicle information" of the vehicle V from the on-board ECU 30. Examples of "vehicle information" include "steering angle information" obtained from the steering ECU 32, "throttle opening information" obtained from the accelerator ECU 33, and "brake pedal depression amount information" obtained from the brake ECU 34. "Vehicle speed information" may also be obtained from the wheel speed sensor.

[0054] <Location specifying section 107> The positioning unit 107 determines the current position of the vehicle V from the GNSS information received from the GNSS receiver 21 of the on-board locator 20 and information such as the angular velocity and acceleration of the vehicle V received from the inertial measuring device 22. Furthermore, in determining the current position of vehicle V, it is also possible to acquire "steering angle information" of vehicle V and combine it with other information to determine the position. For example, by newly installing a steering angle sensor on vehicle V, "steering angle information" can be acquired through the steering angle sensor. In addition, to obtain the "speed information" of vehicle V, a wheel speed sensor may be newly installed on vehicle V, and the "speed information" may be obtained through the wheel speed sensor.

[0055] <Communications Department 108> The communication unit 108 (first communication unit) transmits information to the operating device 50. In this embodiment, the communication unit 108 transmits to the operating device 50 the "composite image" generated by the image processing unit 110 and the "location information" of the vehicle V identified by the location identification unit 107. The communication unit 108 is implemented by the CPU of the vehicle control device 1 and the on-board communication device 40, which consists of three wireless communication devices 41 (first communication device 41a to third communication device 41c). Therefore, the communication unit 108 can be connected to the wireless base station RB of the wireless communication line (remote communication line) via three communication lines.

[0056] Each of the three communication devices 41 is connected to a wireless base station RB. As the vehicle V moves, the radio wave strength changes, and if the connection cannot be maintained, it connects to another wireless base station RB with a stronger signal. Normally, the three communication devices 41 switch wireless base stations RB independently. However, if the connection to the base station is switched for all communication lines, and the switches occur simultaneously, the connection to the operating device 50 may be temporarily disconnected. Therefore, the communication unit 108 is configured to change the timing of the switches so that the wireless base stations RB to which the three communication devices 41 are connected are not switched simultaneously. The in-vehicle communication device 40 may consist of two or more communication devices 41. If the communication environment is good, the in-vehicle communication device 40 may consist of a single communication device 41.

[0057] <Vehicle control unit 109> The vehicle control unit 109 controls the integrated ECU 31 based on the "image information," "detection result data," and "distance measurement data" acquired by the external information acquisition unit 101, and the "current location information" identified by the location identification unit 107, and performs "autonomous driving" of the vehicle V. Furthermore, the vehicle control unit 109 controls the integrated ECU 31 based on the "driving operation information" of the vehicle V acquired from the operating device 50, and performs "remote driving" of the vehicle V.

[0058] <Progressing direction acquisition unit 111> The direction of travel acquisition unit 111 acquires information about the direction of travel of vehicle V, for example, from the shift position of vehicle V. The direction of travel information indicates whether vehicle V is traveling forward or backward at the present time (when operator H starts remote control of vehicle V, or while remote control is in progress). When acquiring the direction of travel information, vehicle V may be stationary. The direction of travel information stored in the storage unit 100 is used, for example, when the image processing unit 110 generates a "composite image" corresponding to the direction of travel of vehicle V.

[0059] <Image processing unit 110> The image processing unit 110 combines the "image information" acquired from each of the first imaging devices 11a to the ninth imaging device 11i by the external information acquisition unit 101, based on the "layout information" stored in the storage unit 100, to generate a "composite image". "Layout information" defines the position and size of the "image information" acquired from the imaging device 11 when it is displayed on the display monitor 51 of the operating device 50. In other words, it is a setting file that defines where the image information acquired by the imaging device 11 will be placed or at what size it will be displayed in the "composite image" when the "composite image" is generated. The "layout information" defines a display configuration that does not create blind spots for operator H and is easy for operator H to operate.

[0060] The "layout information" can also be stored in a designated memory unit of the vehicle V, with multiple layout information associated with a layout ID (layout identification information). In this case, after switching to the layout display using an operation switch, etc. (described later), the layout information can be changed, and the changed layout ID will be transmitted from the operation device 50 (operation switch 56) to the vehicle V. As will be described in detail later, at this time, the vehicle V generates a composite image by combining external video based on the layout information corresponding to the changed layout ID, and this composite image is displayed on the display monitor 51. This "layout information" may be stored on an external server (not shown) located on network N, or it may be sent and received with the external server via a web browser or a dedicated web application.

[0061] The size of the "composite image" generated by the image processing unit 110 is determined by the performance of the display monitor 51 of the operating device 50. In this embodiment, the "composite image" is displayed on the three liquid crystal monitors (first monitor 51a to third monitor 51c) of the display monitor 51. If the resolution of each liquid crystal monitor is Full HD, it is preferable to set the number of pixels of the "composite image" to 5760 x 1080 pixels. Note that the number of pixels of the "composite image" may be changed to a lower resolution depending on the direction of travel or communication conditions. For example, when driving in reverse and using only the central first monitor 51a for remote control, the "composite image" is set to be generated at the size of one screen, i.e., 1920 x 1080 pixels. If communication conditions are poor, the "composite image" is generated with a lower resolution than usual, for example, 4000 x 800 pixels, in order to reduce the amount of data transmitted. The size information of the "composite image" generated by the image processing unit 110 is, for example, included in the layout information and is either pre-stored in the storage unit 100 or input via the input unit 503 of the operating device 50. Furthermore, the size information of the "composite image," like the layout information, may be stored on an external server (not shown) and modified by the image processing unit 110 via a web browser or web application.

[0062] When driving vehicle V, the driver drives while observing the area in front of vehicle V and the area behind it using the rearview and side mirrors. Even when remotely controlling vehicle V, it is desirable that images of the area in front of and behind vehicle V be displayed simultaneously on the display monitor 51. Therefore, the imaging device 11 mounted on the vehicle V of this embodiment includes a first imaging device 11a to a third imaging device 11c (first direction imaging device) that images in the forward direction (first direction of travel), and a fourth imaging device 11d to a sixth imaging device 11f (second direction imaging device) that images in a rearward direction (second direction of travel) that is different from the forward direction. The "layout information" is configured to simultaneously display forward-facing image information captured by the first imaging device 11a to the third imaging device 11c and backward-facing image information captured by the fourth imaging device 11d to the sixth imaging device 11f.

[0063] Furthermore, the image processing unit 110 generates a "composite image" by superimposing the rearward-facing image information captured by the fourth imaging unit 11d to the sixth imaging unit 11f onto the forward-facing image information captured by the first imaging unit 11a to the third imaging unit 11c, based on the layout information. This is the case when the vehicle V is moving forward; for example, when the vehicle V is moving backward, the "composite image" is generated by superimposing the forward-facing image onto the rearward-facing image information. Note that the image information superimposed on the forward-facing or rearward-facing image information may be image information captured to the left or right of the vehicle V.

[0064] Examples of "composite images" are shown in Figures 5A to 5C. In the "composite image" shown in Figure 5A, the image information FM from the first imaging device 11a, which captures the area in front of the vehicle V, is displayed on the central first monitor 51a. Additionally, the image information FR from the second imaging device 11b, which captures the area diagonally to the right front of the vehicle V, is displayed on the right-hand second monitor 51b, and the image information FL from the third imaging device 11c, which captures the area diagonally to the left front of the vehicle V, is displayed on the left-hand third monitor 51c.

[0065] Image information BM from the fourth imaging device 11d, which images the rear of vehicle V, is superimposed on image information FM, which images the front of vehicle V, so that it is displayed in a reduced size at the top of the first monitor 51a. Image information BR from the fifth imaging device 11e, which images the right rear of vehicle V, is superimposed on image information FR, so that it is displayed in a reduced size at the top left of the second monitor 51b. Image information BL from the sixth imaging device 11f, which images the left rear of vehicle V, is superimposed on image information FL, so that it is displayed in a reduced size at the top right of the third monitor 51c.

[0066] Furthermore, the image information FMs of the seventh imaging device 11g, which is a sub-camera, are superimposed on the image information FM so that they are displayed in a reduced state below the first monitor 51a. The image information BRs of the eighth imaging device 11h are superimposed on the image information FR so that they are displayed in a reduced size in the upper right corner of the second monitor 51b. The image information BLs of the ninth imaging device 11i are superimposed on the image information FR so that they are displayed in a reduced state in the upper left corner of the third monitor 51c.

[0067] Figure 5B shows another example of a "composite image". The display positions of the image information FM, FL, and FR captured by the first imaging device 11a to the third imaging device 11c are the same, but the display range for the other image information FMs, BM, BL, BR, BLs, and BRs is set to be larger compared to the "composite image" shown in Figure 5A.

[0068] Figure 5C shows an example of a "composite image" generated during reverse driving. When the direction of travel information is to the rear, i.e., when driving in reverse, the rear of the vehicle V may be displayed using only the first monitor 51a. In this case, the image information BM acquired by the fourth imaging device 11d is arranged to be displayed on the first monitor 51a. The forward image information FM acquired by the first imaging device 11a is superimposed on the image information BM so that it is displayed in a reduced state at the top of the first monitor 51a. If the forward image information (FM) is not needed, you may configure the system to display only the rear image information (BM). In this way, within the "composite image," information regarding the size and position of the image information acquired from the imaging device is set in the layout information.

[0069] The "layout information" is configured so that the image processing unit 110 can generate different composite images depending on the direction of travel of the vehicle V. For example, it includes settings for generating a "composite image" when the vehicle V is moving forward, as shown in Figures 5A and 5B, and settings for generating a "composite image" when the vehicle V is moving backward, as shown in Figure 5C. By including settings for forward and backward travel, the image processing unit 110 can generate a "composite image" according to the direction of travel of the vehicle V, and the display monitor 51 of the operating device 50 will display the "composite image" according to the direction of travel.

[0070] Note that the layout of the composite images shown in Figures 5A to 5C is just one example. For example, vehicle information may be displayed on a portion of the display monitor 51, with image information displayed around it. Also, to allow the surrounding situation to be checked while the vehicle is stopped, the image information BR and BL, which show the left and right sides of the vehicle, may be set to be centered on the display monitor 51.

[0071] In this embodiment, the first imaging device 11a, the second imaging device 11b, and the third imaging device 11c, which image the area in front, capture images so that the hood and fenders of the vehicle V are included in the image information. The image processing unit 110 uses the image information which includes a part of the vehicle V to generate a "composite image" which displays a part of the vehicle V. Because a part of the vehicle V is included in the "composite image", the operator H can understand the size and position of the vehicle body and remotely control the vehicle V using the part of the vehicle V in the "composite image" as a guide.

[0072] By generating the composite video described above and transmitting the generated composite video data to the operating device 50, the amount of data transmitted can be reduced compared to transmitting multiple external video data, thereby reducing the cost of data communication. In other words, because the amount of data is small, it becomes easier to secure the bandwidth required for transmission, and it becomes possible to transmit with fewer lines.

[0073] The functions of the operating device 50 will now be explained. <Storage section 500> The storage unit 500 stores various data and programs, such as the "composite image" received from the vehicle control device 1, information on the direction of travel of the vehicle V, and "layout information" input by the input unit 503. Note that various data such as layout information may be stored on an external server located on the network N and transmitted and received via a web browser or web application.

[0074] <Communications Department 501> The communication unit 501 (second communication unit) receives information from the vehicle control device 1. In this embodiment, the communication unit 501 receives a "composite image" and "location information" of the vehicle V from the vehicle control device 1. The communication unit 108 is implemented by the CPU of the operating device 50 and the communication device 60.

[0075] <Display section 502> The display unit 502 displays the information received by the communication unit 501 on the display monitor 51 or the navigation display monitor 52. The display unit 502 displays the "composite image" received from the vehicle control device 1 on the display monitor 51, and displays the position of the vehicle V superimposed on the map information on the navigation display monitor 52.

[0076] <Input section 503> The input unit 503 processes the layout information using operator H. The input layout information is transmitted to the vehicle control device 1 by the communication unit 501. If there are multiple layout entries and corresponding layout IDs are defined, the layout IDs may be transmitted instead.

[0077] <Communication Control Unit 504> When the communication unit 501 receives information related to the "composite image" from the vehicle control device 1, the communication control unit 504 establishes a communication path with the communication unit 108 (first communication unit) of the vehicle control device 1. The vehicle remote control system consists of multiple vehicles V and multiple control devices 50. When remotely controlling a vehicle V, one control device 50 is selected from the multiple control devices 50. At that time, the connection with the control device 50 connected to the vehicle V must be maintained until the remote control is completed, and this communication path is reserved as the highest priority communication destination. As a means of securing this communication path, the vehicle V can connect to multiple communication lines using multiple communication devices.

[0078] <Audio output unit 505> The audio output unit 505 processes the audio information received from the vehicle control device 1. The audio information is output from the speaker 59.

[0079] <<Vehicle Remote Control Method / Program>> Next, the vehicle remote control method (vehicle remote control program) executed by the vehicle control device 1 and the operating device 50 will be explained based on the sequence diagram in Figure 6. In this embodiment, the vehicle remote control method is executed by a program on the CPU (first computer) of the vehicle control device 1 and the CPU (second computer) of the operating device 50. That is, the above method is executed by a utility program that aggregates various programs that realize the above-mentioned functions, as functional components of the vehicle control device 1 and the operating device 50 described above. Furthermore, the above method and program are executed in response to operational instructions from operator H of the vehicle remote control system.

[0080] When remote operation is initiated, operator H of the control device 50 inputs "layout information" used to generate the "composite image" to be displayed on the display monitor 51 (layout information input step S101). The "layout information" is input by operator H, for example, via the touch panel 57. The storage unit 500 temporarily stores the "layout information". The "layout information" may be selected by operator H from a predetermined form using multiple switches at hand. If there is a form that operator H always uses and it is stored in the storage unit 100 of the vehicle control device 1, it may be set automatically when operator H is identified. Also, if a predetermined form is stored in the vehicle control device 1 or the control device 50, this step may be omitted.

[0081] After receiving "layout information" via the input unit 503, the operating device 50 transmits the "layout information" to the vehicle control device 1 via the communication unit 501 before starting remote operation (layout information transmission step S102). The communication unit 108 of the vehicle control device 1 receives the "layout information" along with the instruction to start remote operation via the network N (layout information reception step S201).

[0082] After receiving the "layout information," the storage unit 100 of the vehicle control device 1 stores the "layout information" (layout information storage step S202). Next, the direction of travel acquisition unit 111 acquires information about the direction of travel of the vehicle V. This is because the image displayed on the display monitor 51 of the operating device 50 differs depending on the direction of travel. The direction of travel is determined by the position of the shift position of the vehicle V (direction of travel acquisition step S203).

[0083] Next, the image processing unit 110 of the vehicle control device 1 acquires image information captured from nine imaging devices 11 mounted on the vehicle V by the external information acquisition unit 101 (image information acquisition step S204). Based on the "layout information" stored in the storage unit 100, it combines the multiple image information to generate a "composite image" (composite image generation step S205). At this time, the "layout information" used to generate the "composite image" is switched depending on the direction of travel of the vehicle V. For example, when the vehicle V is moving forward, a "composite image" is generated with the images from the first imaging devices 11a to the third imaging devices 11c, which capture the front of the vehicle, as the main image (see Figures 5A and 5B). When the vehicle V is moving in reverse, a "composite image" is generated so that the image captured by the fourth imaging device 11d, which captures the rear of the vehicle, is the main image (see Figure 5C). Since the image processing unit 110 of the vehicle control device 1 generates and transmits the "composite image", the resolution of the transmitted image remains constant, and the amount of data required for transmission does not increase, making it easier to secure the bandwidth of the communication line required for transmission.

[0084] Next, the vehicle control device 1 transmits the "composite image" generated by the image processing unit 110 to the operating device 50 using the communication unit 108 (composite image transmission step S206). At this time, encoding of the "composite image" may be performed in order to reduce the amount of data of the "composite image". The communication unit 108 transmits the encoded "composite image" to the operating device 50.

[0085] In this case, the vehicle control device 1 may include an encoder that performs encoding in hardware. An encoder, for example, encodes a "composite image" according to a video encoding scheme. Examples of video encoding schemes include MPEG (Moving Picture Expert Group)-2, H.264, H.265, VP8, VP9, ​​and AOMedia Video 1 (AV1). The encoder generates encoded video data from the "composite image" through hardware encoding. However, the encoder may also generate video data through software encoding.

[0086] Next, when the communication unit 501 receives the "composite image" from the vehicle control device 1, the control unit 504 establishes a communication path with the communication unit 108 of the vehicle control device 1 (communication path establishment step S103). By establishing the communication path, the control unit 50 maintains the communication path until the remote control of the vehicle V is completed, and secures the communication path as the highest priority path destination. The communication control unit 504 may establish the communication path when remote operation is initiated, or when the operating device 50 transmits "layout information" to the vehicle control device 1.

[0087] The operating device 50 receives the "composite image" after the communication unit 501 establishes a communication path with the communication unit 108 of the vehicle control device 1 (composite image reception step S104). The received "composite image" is displayed on the display monitor 51 by the display unit 502 (composite image display step S105). If the received "composite image" is encoded video data, the display unit 502 decodes and displays the video data according to the video encoding scheme used. Conventionally, when receiving image information directly from multiple imaging devices, the operating device 50 would arrange the image information on its own and display it on the display monitor. However, since the operating device 50 receives and displays a "composite image" generated by the vehicle control device 1, the number of communication lines required for transmission is reduced, and the amount of data received can also be reduced. Because it receives a "composite image" generated by the vehicle control device 1, it can be received reliably without any parts of the image being missing. Furthermore, because it receives a "composite image" according to the direction of travel, the "composite image" can be displayed in an appropriate layout automatically by switching the direction of travel.

[0088] <<Resolution Switching>> In the vehicle remote control system S, the operator H remotely controls vehicle V while visually confirming the surrounding situation by displaying a "composite image" transmitted from vehicle V on the display monitor 51. Therefore, the vehicle control device 1 of vehicle V needs to transmit the "composite image" to the control device 50 as much as possible. For this reason, the vehicle control device 1 has an adjustment function that allows transmission even if there are changes in the communication environment by changing the resolution of the "composite image" it generates. The process of changing according to the communication environment will be explained using Figure 7.

[0089] First, the communication unit 108 of the vehicle control device 1 checks the communication status of the three communication devices 41 (first communication device 41a to third communication device 41c) of the in-vehicle communication device 40, which are connected via the first communication line, which is a 4G communication line. Specifically, it measures the radio wave strength of each of the three communication lines. If the radio wave strength of the communication lines is greater than a predetermined threshold α, it determines that sufficient bandwidth is secured to transmit the "composite image" (Yes in S302). The communication unit 108 transmits the "composite image" to the operating device 50 as a high-resolution full image without reduction (S303).

[0090] If the radio signal strength of any of the communication devices 41 is lower than the threshold α, the connection with the wireless base station RB may be disconnected, and communication may not be possible. If the answer in S302 is No, it is determined whether the three communication devices 41 may change the wireless base station RB to which they are connected (S304). If the three communication devices 41 may change the wireless base station RB (Yes in S304), the communication unit 108 performs a process to change the timing of the line switching so that the three communication devices 41 do not switch the wireless base station RB at the same time (S305).

[0091] Next, the vehicle control device 1 measures the number of lines that can be connected via the 4G line (first communication line) (S306). If one communication line is disconnected and communication is possible with two communication lines (Yes in S307), the bandwidth available for transmitting the "composite image" is reduced compared to when three communication lines are connected. Therefore, the device generates and transmits a "composite image" with a lower resolution than the original "composite image" to be displayed on three screens (S308).

[0092] The system determines if two communication lines are disconnected and if a connection is possible using only one line (S309). If communication is possible using only one communication line (Yes in S309), the bandwidth available for transmitting the "composite image" is further reduced, so a lower-resolution "composite image" compared to the original "composite image" is generated and transmitted to display on one screen (S310).

[0093] If a 4G connection cannot be established on any communication line (No in S309), an attempt is made to connect using a 3G communication line (S311). If a connection is possible using a 3G communication line (Yes in S311), a single-screen image with higher latency and lower resolution is transmitted (S312). If a connection cannot be established using a 3G communication line (No in S311), the communication path cannot be secured and the "composite image" cannot be transmitted, so image transmission is stopped and remote operation is terminated (S313).

[0094] Furthermore, if the number of 4G connections increases from two to three due to the movement of vehicle V, the available bandwidth will increase, resulting in the transmission of a higher-resolution "composite image" compared to the "composite image" before the change. The same applies when the number of connections increases from one to two or three. Similarly, if the communication line switches from 3G to 4G, a higher-resolution "composite image" will be transmitted.

[0095] In this way, by using multiple wireless communication lines and adjusting the quality of the "composite image" of the vehicle's surroundings according to the number of connected lines, the vehicle control device 1 of the vehicle V can transmit the "composite image" to the operating device 50 as much as possible.

[0096] <Other Embodiments> In the above embodiment, as shown in Figure 2, the vehicle remote control system S comprises a vehicle control device 1 and an on-board ECU 30, and newly adds a "remote driving function (vehicle control device 1)" to a vehicle V equipped with an "autonomous driving function (on-board ECU 30)". However, this can be modified without any particular limitations. For example, the vehicle control device 1 may also include the functions of the on-board ECU 30. That is, the vehicle remote control system S may mainly consist of the vehicle control device 1 (including the functions of the on-board ECU 30), the on-board sensor 10, the on-board locator 20, the on-board communication device 40, and the operating device 50 (the on-board ECU 30 may be excluded from the configuration).

[0097] In the above embodiment, as shown in Figure 2, the vehicle remote control system S comprises a vehicle control device 1 and an on-board locator 20. However, the vehicle control device 1 may also have an on-board locator 20, without being particularly limited.

[0098] In the above embodiment, a vehicle remote control program is stored on a recording medium that the vehicle control device 1 can read, and processing is performed by the vehicle control device 1 reading and executing the program. Here, the recording medium that the vehicle control device 1 can read refers to a magnetic disk, magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, etc. Alternatively, a dedicated web application may be launched using a terminal (mobile device) that serves as the vehicle control device 1, and the vehicle remote control program may be executed on a web browser.

[0099] The above embodiments mainly described the vehicle remote control system and vehicle remote control method according to the present invention. According to the vehicle remote control system S, image information is acquired from each of the multiple imaging devices 11, and the image processing unit 110 of the vehicle control device 1 generates a composite image based on layout information and transmits it to the operating device 50. This reduces the amount of data transmitted compared to transmitting all the captured image information. As a result, the bandwidth required for transmitting the composite image remains constant, allowing for the transmission of image information around the vehicle without increasing communication costs. Furthermore, it becomes easier to secure the necessary bandwidth for communication, enabling more reliable transmission. In addition, since the composite image synthesized on the vehicle side is transmitted to the operating device 50, there is no loss of image due to the disconnection of some communication lines. The above embodiments are merely examples to facilitate understanding of the present invention and do not limit it. The present invention can be modified and improved without departing from its spirit, and of course, equivalents thereof are included. [Explanation of Symbols]

[0100] S Vehicle Remote Control System V Vehicle V1 Electric Power Steering V1a Handle V2 Electric Throttle V2a Accelerator Pedal V3 Electromagnetic Brake System V3a Brake Pedal 1. Vehicle control system 10 In-vehicle sensors 11 Imaging device 11a~11i First Imaging Device~Ninth Imaging Device 12. Radar (Millimeter-wave radar) 12a~12d Radar 1~Radar 4 13 Rider 13a~13e Rider 1~Rider 5 20 In-vehicle locators 21 GNSS receiver (RTK-GNSS receiver) 22 Inertial Measurement Unit (IMU) 30 Automotive ECU 31. Integrated ECU 32 Handle ECU 33 Accelerator ECU 34 Brake ECU 40. In-vehicle communication devices 41, 41a, 41b, 41c communication device, first communication device, second communication device, third communication device 50 Operating device 51 Display Monitor 51a, 51b, 51c: First monitor, second monitor, third monitor 52 Navigation display monitor 53 Handle 54 Accelerator pedal 55 Brake pedal 56 Operating switches 57 Touch panel 58 Mike 59 speakers 60 Communication equipment 61 chairs 100 Storage section 101 External information acquisition department 107 Location identification part 108 Communications Department 109 Vehicle Control Unit 110 Image Processing Unit 500 storage section 501 Communications Department 502 Display section 503 Input section 504 Communication Control Unit 505 Audio output section SA satellite ST reference station RB wireless base station

Claims

1. A vehicle connected to an operating device via a network, Multiple imaging devices for imaging the area around the vehicle, An in-vehicle communication device comprising multiple communication devices, each of which is connected to a wireless base station via a wireless communication line, A vehicle control device is provided, The aforementioned vehicle control device is An image processing unit that synthesizes image information acquired from each of the aforementioned multiple imaging devices to generate a composite image, A communication unit that transmits the composite image to the operating device using the in-vehicle communication device, Equipped with, The communication unit measures the radio wave strength of the wireless communication line to which each of the plurality of communication devices is connected. A vehicle in which the vehicle control device changes the resolution of the composite image generated by the image processing unit according to the number of communication lines to which the plurality of communication devices can be connected, based on the radio wave strength.

2. The aforementioned vehicle control device is When all of the aforementioned communication devices are connectable to the wireless communication line, the image processing unit generates a high-resolution composite image. The vehicle according to claim 1, wherein when the connection between any of the plurality of communication devices and the wireless base station is disconnected, and the number of connectable communication lines decreases, the image processing unit generates a composite image with a lower resolution compared to the composite image before the change.

3. The aforementioned communications unit is If all of the aforementioned communication devices cannot be connected via the first communication line, they will be connected to the wireless base station via a second communication line that has a wider reception range than the first communication line. The vehicle according to claim 1 or 2, wherein when the vehicle control device is connected via the second communication line, the image processing unit generates a composite image with an even lower resolution than when connected via the first communication line.

4. The aforementioned vehicle control device is The vehicle according to any one of claims 1 to 3, wherein when the number of connectable communication lines increases due to the movement of the vehicle, the image processing unit generates a composite image with a higher resolution than the composite image before the change.

5. The aforementioned communications unit is The vehicle according to claim 3, wherein if it is not possible to connect to the wireless base station via either the first communication line or the second communication line, the transmission of the composite image is stopped.

6. A system using a vehicle that includes multiple imaging devices, an in-vehicle communication device consisting of multiple communication devices, and a vehicle control device, and is connected to an operating device via a network, The aforementioned vehicle control device A process for acquiring image information from each of the aforementioned multiple imaging devices, A process to generate a composite image by combining the acquired image information, The process of transmitting the composite image to the operating device using the in-vehicle communication device, A process for measuring the radio wave strength of the wireless communication line to which each of the aforementioned multiple communication devices is connected, Based on the radio wave intensity, the process of changing the resolution of the composite image according to the number of communication lines to which the plurality of communication devices can connect, A system that executes this.

7. A vehicle control system computer mounted in a vehicle, comprising multiple imaging devices and multiple communication devices, is connected to an operating device via a network. A process for acquiring image information from each of the aforementioned multiple imaging devices, A process to generate a composite image by combining the acquired image information, The process of transmitting the composite image to the operating device using the in-vehicle communication device, A process for measuring the radio wave strength of the wireless communication line to which each of the aforementioned multiple communication devices is connected, Based on the radio wave intensity, the process involves changing the resolution of the composite image according to the number of communication lines to which the multiple communication devices can be connected. A program that executes something.

Images