Remote control system for vehicle operation

Abstract

To enhance remote control of vehicle travel.SOLUTION: A remote control system 1 for vehicle travel is configured to periodically transmit a remote control value for controlling travel of a vehicle 2 from a remote control device 4, which is communicable with a plurality of vehicles 2, to the vehicle 2 through communication between the remote control device 4 and the vehicle 2. The vehicle 2 transmits to the remote control device 4 own vehicle information including vehicle outside picked-up images of a plurality of vehicle outside cameras 63 (71-76) provided in the own vehicle and periodically executes travel control through remote control using the remote control value received from the remote control device 4. A travel control ECU 24 of the vehicle 2 determines, during remote control, whether or not a status of the communication between the vehicle 2 and the remote control device 4 or a processing status of the remote control device 4 exerts influences upon the periodical travel control using the remote control value and, in a case where it is determined that the situation exerts influences, selects a part of vehicle outside picked-up images of the plurality of vehicle outside cameras 63 (71-76) provided in the own vehicle in accordance with a travel status of the own vehicle and transmits it to the remote control device 4.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a remote control system for vehicle travel. [Background technology]

[0002] In recent years, technologies for automating the driving of vehicles such as automobiles have been developed. In this case, it is conceivable that location information and images taken outside the vehicle obtained by the vehicle are transmitted to a server device for remote control, and the server device for remote control generates remote control values ​​that can be used to control the vehicle's driving (Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-133498 [Patent Document 2] International Publication No. 2018 / 179392 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the server device for remote control must generate and transmit the above-mentioned remote control values ​​for each of the multiple vehicles periodically, which places a heavy processing load on the server device for remote control. If each vehicle that controls its driving by remote control is unable to continue to receive remote control values ​​periodically at appropriate time intervals, it may become difficult to continue to properly control the driving using the remote control values.

[0005] Furthermore, it is desirable that the vehicle information transmitted from the vehicle to the server device for remote control include images of the exterior of the vehicle taken by a plurality of exterior cameras of the vehicle so that the server device for remote control can appropriately determine the driving conditions of the vehicle and obtain good remote control values. If the server device for remote control cannot obtain images of the exterior of the surroundings from each vehicle, it will be difficult to appropriately determine the driving conditions of each vehicle and to generate remote control values ​​that appropriately correspond to the driving conditions of each vehicle.

[0006] Thus, there is a need for improvements in the remote control of vehicle travel. [Means for solving the problem]

[0007] A remote control system for vehicle driving according to one aspect of the present invention is a remote control system for vehicle driving in which a remote control device capable of communicating with a plurality of vehicles communicates with the vehicle and the remote control device communicates with the vehicle, thereby periodically transmitting remote control values ​​for controlling the driving of the vehicle from the remote control device to the vehicle, the system comprising: a transmission control unit provided in the vehicle and capable of transmitting, from the vehicle to the remote control device, host vehicle information including at least vehicle exterior images captured by a plurality of exterior cameras provided in the vehicle; and a host vehicle driving control unit provided in the vehicle and configured to periodically execute remote driving control by the vehicle using the remote control values ​​periodically received by the vehicle from the remote control device, wherein the transmission control unit of the vehicle, during remote control, By determining whether or not there is an influence on the driving control of the host vehicle driving control unit based on at least a period of driving control using a remote control value executed by the host vehicle driving control unit in the vehicle and a reception interval of a plurality of remote control values ​​from the remote control device, The car Both The system determines whether the communication status with the remote control device or the processing status of the remote control device will affect the periodic driving control using remote control values ​​by the vehicle driving control unit, and if it determines that it will have an effect, selects some of the exterior images taken by multiple exterior cameras installed on the vehicle according to the driving status of the vehicle and transmits them to the remote control device. According to one aspect of the present invention, there is provided a remote control system for vehicle driving, in which a remote control device capable of communicating with a plurality of vehicles communicates with the vehicle, and the remote control device can periodically transmit remote control values ​​for controlling the driving of the vehicle to the vehicle by the vehicle communicating with the remote control device, the system comprising: a transmission control unit provided in the vehicle and capable of transmitting, from the vehicle to the remote control device, host vehicle information including at least images captured outside the vehicle by a plurality of exterior cameras provided in the vehicle; and a host vehicle driving control unit provided in the vehicle and configured to periodically execute remote control driving control by the vehicle using the remote control values ​​periodically received by the vehicle from the remote control device, wherein the plurality of exterior cameras provided in the vehicle capture images of the surroundings of the vehicle, divided into at least the front, right, rear, and left sides of the vehicle, and the transmission control unit of the vehicle is configured to receive, during remote control, information on the communication status between the vehicle and the remote control device, or the processing status of the remote control device, determines whether the periodic driving control using the remote control value by the vehicle driving control unit will be affected, and if it is determined that the driving of the vehicle will be affected when remote control is used to automatically park the vehicle in a parking lot, at least an exterior image of the front side of the vehicle and an exterior image of the side of the parking lot will be selected and transmitted to the remote control device as part of the exterior images taken by the multiple exterior cameras installed on the vehicle while the vehicle is driving until it stops in the aisle in front of the parking position, and at least an exterior image of the rear side of the vehicle and an exterior image of the side of the parking position will be selected and transmitted to the remote control device as part of the exterior images taken by the multiple exterior cameras installed on the vehicle while the vehicle is driving from the stopped position to entering the parking position, and the exterior image to be selected is dynamically changed according to each driving stage from entering the parking position from the aisle in front of the parking position to stopping. A remote control system for vehicle driving according to one aspect of the present invention is a remote control system for vehicle driving in which a remote control device capable of communicating with a plurality of vehicles communicates with the vehicle and the remote control device communicates with the vehicle, thereby periodically transmitting remote control values ​​for controlling the driving of the vehicle from the remote control device to the vehicle, the system comprising: a transmission control unit provided in the vehicle and capable of transmitting from the vehicle to the remote control device host vehicle information including at least vehicle exterior captured images from a plurality of exterior cameras provided in the vehicle; and a host vehicle driving control unit provided in the vehicle and periodically executing remotely controlled driving control by the vehicle using the remote control values ​​periodically received by the vehicle from the remote control device, wherein the plurality of exterior cameras provided in the vehicle capture images of the surroundings of the vehicle divided into at least the front, right front side, right rear side, rear side, left front side, and left rear side of the vehicle, and the transmission control unit of the vehicle monitors the communication status between the vehicle and the remote control device or the processing of the remote control device during remote control. and when it is determined that the operating conditions will have an effect on the periodic driving control using a remote control value by the vehicle driving control unit, and when it is determined that the operating conditions will have an effect when the driving of the vehicle is controlled by remote control so that the vehicle turns right or left at an intersection, while the vehicle is driving on the road before entering the intersection, at least an exterior image taken in front of the vehicle is selected as part of the exterior images taken by a plurality of exterior cameras provided on the vehicle and transmitted to the remote control unit; while the vehicle is driving after starting to turn at the intersection, at least an exterior image taken in front of the vehicle, an exterior image taken in front of the vehicle, an exterior image taken in front of the vehicle, an exterior image taken in front of the vehicle, an exterior image taken in front of the vehicle, an exterior image taken in front of the vehicle, and an exterior image taken on the outside of the right rear side or the left rear side of the vehicle, which is selected as part of the exterior images taken by the plurality of exterior cameras provided on the vehicle, and transmitted to the remote control unit; and the exterior image to be selected is dynamically changed according to each driving stage from the road before entering the intersection to the road after the turn. [Effects of the Invention]

[0008] In the present invention, a vehicle is provided with a transmission control unit that periodically transmits vehicle information, including at least exterior images captured by multiple exterior cameras installed on the vehicle, from a communication device of the vehicle to a remote control device. The transmission control unit determines whether the communication status between the communication device and the remote control device or the processing status of the remote control device affects periodic driving control using remote control values ​​by a vehicle driving control unit of the vehicle. If it determines that the communication status between the communication device and the remote control device or the processing status of the remote control device affects the periodic driving control using remote control values ​​by a vehicle driving control unit of the vehicle, the transmission control unit selects and transmits to the remote control device some of the exterior images captured by the multiple exterior cameras installed on the vehicle in accordance with the driving conditions of the vehicle. This reduces the amount of uplink information transmitted from the vehicle to the remote control device and allows for earlier termination of uplink transmission compared to transmitting vehicle information including all exterior images captured by the multiple exterior cameras installed on the vehicle. The remote control value generation unit of the remote control device can early generate and transmit to the vehicle remote control values ​​that can be used for vehicle driving control based on the early received vehicle information. As a result, the vehicle's own vehicle driving control unit becomes more likely to receive remote control values ​​from the remote control device without delay, and the own vehicle driving control unit becomes able to maintain periodic driving control using the remote control values. Furthermore, the transmission control unit of the vehicle of the present invention does not randomly select exterior images to transmit from the exterior images captured by multiple exterior cameras installed on the vehicle, but selects them based on the vehicle's driving conditions. This allows the remote control value generation unit of the remote control device to generate a meaningful remote control value corresponding to the vehicle's driving conditions even if it receives vehicle information from the vehicle that excludes unselected exterior images. The remote control value generation unit of the remote control device can ensure continuity of remote control values, for example, between a remote control value generated based on vehicle information including all exterior images and a remote control value generated based on subsequently received vehicle information including some of the exterior images. Furthermore, even if the remotely controlled vehicle receives these remote control values ​​consecutively and executes remote driving control, the vehicle can continue driving control without significantly changing its driving. Furthermore, in the present invention, the vehicle communicating with the remote control device, rather than the remote control device, determines the communication status between the vehicle and the remote control device and the impact of the processing status of the remote control device, and selects the outside-vehicle captured image to be sent to the remote control device. As a result, in the present invention, the remote control device basically does not need to perform the processes for these determinations and selections. The remote control device does not need to perform these processes when the processing load of periodically generating remote control values ​​for multiple vehicles is high. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of a remote control system for controlling the running of an automobile according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a hardware configuration diagram of a computer device that can be used as the server device of the remote control device of FIG. [Figure 3] FIG. 3 is a block diagram of a control system that controls the running of the automobile shown in FIG. [Figure 4] FIG. 4 is a timing chart illustrating the flow of basic remote control in the remote control system of FIG. [Figure 5] FIG. 5 is an explanatory diagram of an example of a plurality of exterior cameras provided in the automobile of FIG. [Figure 6] FIG. 6 is a flowchart of remote driving control in the first embodiment by the control system of the automobile of FIG. [Figure 7] FIG. 7 is a flowchart of remote control in the first embodiment by the server device of the remote control device of FIG. [Figure 8] FIG. 8 is an explanatory diagram of the correspondence between the driving conditions of the automobile in FIG. 1 and the selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras. [Figure 9] FIG. 9 is a flowchart of the selection process of the outside-of-vehicle captured image in the second embodiment, which is performed by the control system of the automobile in FIG. [Figure 10]FIG. 10 is a diagram illustrating an example of the correspondence between the driving conditions and driving stages of the automobile in FIG. 1 and the selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras. [Figure 11] FIG. 11 is a diagram illustrating another example of the correspondence between the driving conditions and driving stages of the automobile in FIG. 1 and the selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras. [Figure 12] FIG. 12 is a diagram illustrating another example of the correspondence between the driving conditions and driving stages of the automobile in FIG. 1 and the selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras. [Figure 13] FIG. 13 is a flowchart of remote control in the third embodiment by the server device of the remote control device of FIG. [Figure 14] FIG. 14 is a flowchart of the process of selecting an outside-vehicle captured image in the third embodiment, which is performed by the control system of the automobile shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [First embodiment] FIG. 1 is a configuration diagram of a remote control system 1 for controlling the running of an automobile 2 according to a first embodiment of the present invention. The remote control system 1 in Fig. 1 is for enabling remote control of the driving of automobiles 2, and includes a control system 3 provided in multiple automobiles 2, and a remote control device 4 having a server device 5 and a remote control value generating device 6 for generating remote control values. The multiple automobiles 2 and the server device 5 of the remote control device 4 are connected to each other so as to be able to communicate wirelessly via a communication system 7 having a communication network 8 and multiple base stations 9 arranged along roads 100 along which the automobiles 2 travel. Communication between the multiple automobiles 2 and the remote control device 4 separate from the multiple automobiles 2 enables the remote control device 4 to repeatedly transmit remote control values ​​for controlling the driving of the automobiles 2 to each of the multiple automobiles 2. 1 also shows a GNSS satellite 110 that outputs GNSS (Global Navigation Satellite System) radio waves that can be received by multiple automobiles 2 and server device 5. By receiving radio waves from multiple GNSS satellites 110, automobiles 2 or server device 5 can obtain their respective positions and times in a common positioning system.

[0012] The automobile 2 is an example of a vehicle. Other examples of the automobile 2 include a motorcycle, a cart, and personal mobility. The automobile 2 can travel on a road 100 or the like using the driving force of an engine or a motor, which is a power source, under the driving control of a control system 3 provided in the automobile, can decelerate and stop by operating a braking device, and can change its direction of travel left or right by operating a steering device. The control system 3 of the automobile 2 may basically be capable of performing driving control according to manual driving based on the operation of the passenger of the automobile. The control system 3 may be capable of controlling driving to support manual driving based on detection results from the automobile. The control system 3 may be capable of performing driving control for automatic driving using high-precision map data as well as detection results from the automobile.

[0013] The multiple base stations 9 may be, for example, base stations 9 of a carrier communication network for mobile terminals or the like, or base stations for ITS services or ADAS services for the automobile 2. The base station 9 of the carrier communication network may be, for example, a fifth-generation base station. The base station 9 may be fixedly installed on, for example, a roadside, a road surface, or a building, or may be provided on a moving object such as the automobile 2, a ship, a drone, or an airplane. The base station 9 establishes a wireless communication path for transmitting and receiving information with an AP (access point) communication device of the control system 3 of the automobile 2 located within its radio wave coverage area. When the automobile 2 travels along a road 100 and moves out of the radio wave coverage area, the base station 9 with which the wireless communication path is established switches among the multiple base stations 9. This allows the automobile 2 to continuously establish a wireless communication path while traveling, for example, with multiple base stations 9 lined up along the road 100. A wireless communication path established with a fifth-generation base station can transmit and receive a significantly larger amount of information at higher speeds than a wireless communication path established with a fourth-generation base station 9. Furthermore, fifth-generation base stations can have advanced information processing capabilities and functions for transmitting and receiving information between base stations 9. In V2V communication between automobiles 2, automobiles 2 may transmit and receive information directly with each other, or automobiles 2 may transmit and receive information with each other via a fifth-generation base station. By using a fifth-generation base station, it is expected that high-speed communication with a maximum delay time of about milliseconds will be possible in one-way communication, either uplink or downlink, between the remote control device 4 and each vehicle 2. However, when multiple vehicles 2 communicate with the remote control device 4, it is not easy to achieve communication at the same maximum communication speed for all of the vehicles 2. When the automobile 2 is moving, the base station 9 with which the automobile 2 establishes a communication path switches according to changes in the position of the automobile 2. The hands-over process for switching the base station 9 may take some time.

[0014] The communication network 8 may be configured, for example, by a communication network for a carrier communication network, a communication network for ITS services or ADAS services, or the Internet, which is an open wide-area communication network. The communication network 8 may include a dedicated communication network newly established for the remote control system 1. A communication network dedicated to a carrier communication network or the Internet realizes communication on a best-effort basis. In a best-effort communication network, the communication bandwidth available to each device and the communication transmission delay are not fixed but change dynamically depending on the communication environment. In particular, in a communication network for communication compliant with the TCP / IP protocol, collisions due to asynchronous communication may occur, causing transmission delays due to frame retransmissions, etc. If the handshake process takes a long time, transmission delays due to frame retransmissions are likely to occur.

[0015] FIG. 2 is a hardware configuration diagram of a computer device 10 that can be used as the server device 5 of the remote control device 4 of FIG. The computer device 10 of FIG. 2 includes a server communication device 11, a server GNSS receiver 12, a server timer 13, a server memory 14, a server CPU 15, and a server bus 16 to which these are connected.

[0016] The server communication device 11 is connected to the communication network 8. The server communication device 11 can send and receive information to and from other devices connected to the communication network 8, such as a base station 9 and a control system 3 of the automobile 2. The server GNSS receiver 12 receives radio waves from the GNSS satellites 110 to obtain the current time. The server timer 13 measures the time and duration. The time of the server timer 13 may be calibrated by the current time of the server GNSS receiver 12. The server memory 14 stores programs and data executed by the server CPU 15 . The server CPU 15 reads and executes the program from the server memory 14. In this way, the server device 5 realizes a server control unit. The server CPU 15 as a server control unit manages the overall operation of the server device 5 and the overall control of the remote control system 1. The server CPU 15 manages the running of multiple automobiles 2 that use the remote control system 1, etc. The server CPU 15, for example, manages information received from each of the multiple automobiles 2, controls the generation of remote control values ​​for the automobiles 2 that received the information, and controls the transmission of the remote control values ​​generated for the automobiles 2 that received the information. In this case, the server memory 14 stores information received from the multiple automobiles 2, such as high-precision map data for generating the remote control values. The server CPU 15 also repeatedly receives the latest information from each automobile 2, thereby repeatedly generating and transmitting remote control values ​​for each automobile 2. This allows each automobile 2 to continue traveling in accordance with the remote control values ​​repeatedly generated by the remote control device 4.

[0017] The remote control value generating device 6 may basically be any device that performs the same functions as the cruise control ECU 24 of the control system 3 of the automobile 2 described later, and may use the computer device 10 of FIG. 2 as hardware. In this embodiment, the remote control value generating device 6 for generating the remote control values ​​of each automobile 2 is described as being separate from the server device 5 for managing communication of the remote control devices 4. The remote control value generating device 6 and the server device 5 can also be realized in one computer device 10. Furthermore, the remote control value generating device 6 may basically be provided in a one-to-one correspondence with the multiple automobiles 2 managed by the remote control device 4. However, in reality, it is preferable that one remote control value generating device 6 is provided in a one-to-many correspondence with the multiple automobiles 2. Such a remote control value generating device 6 repeatedly generates remote control values ​​that can be used for driving control of each of the multiple automobiles 2 for each automobile 2. In this case, a remote control value generating device 6 may be provided for each type of automobile 2. It is considered that the driving characteristics and driving control characteristics of automobiles 2 basically differ depending on the type of automobile 2. The multiple remote control value generating devices 6 may be connected in a one-to-many correspondence to a server device 5 that manages communications of the remote control devices 4. In this case, the server device 5 of the remote control device 4 manages communications with the multiple automobiles 2 that generate remote control values ​​using the multiple remote control value generating devices 6.

[0018] FIG. 3 is a configuration diagram of a control system 3 that controls the running of the automobile 2 in FIG. The control system 3 provided in the automobile 2 in Fig. 3 is shown with a plurality of control devices represented by control ECUs (Electronic Control Units) incorporated therein. Similar to the server device 5 in Fig. 2, the control devices may have, in addition to the control ECUs, for example, a memory for recording control programs and data, an input / output port, a timer for measuring time and clock time, and an internal bus to which these are connected (not shown). 3 shows a plurality of control ECUs for the control system 3 of the automobile 2, such as a drive ECU 21 for the drive device, a steering ECU 22 for the steering device, a braking ECU 23 for the braking device, and a cruise control ECU 24. Also shown are a driving operation ECU 25, a detection ECU 26, an AP communication ECU 27, and a V2V communication ECU 28. The control system 3 of the automobile 2 may include other control ECUs not shown.

[0019] The multiple control ECUs are connected to a vehicle network 30, such as a Controller Area Network (CAN) or a Local Interconnect Network (LIN), employed in the automobile 2. The vehicle network 30 may be configured with multiple bus cables 31 that can connect the multiple control ECUs, and a central gateway (CGW) 32 that serves as a relay device to which the multiple bus cables 31 are connected. Each of the multiple control ECUs is assigned a unique ID as identification information. A control ECU basically periodically outputs data to another control ECU. The ID of the control ECU that is the output source and the ID of the control ECU that is the output destination are added to the data. The control ECU monitors the bus cable 31, and if the ID of the output destination is, for example, its own, it acquires the data and executes processing based on the data. The central gateway 32 monitors each of the multiple connected bus cables 31, and when it detects a control ECU connected to a bus cable 31 different from the control ECU that is the output source, it outputs the data to that bus cable 31. Such relay processing by the central gateway 32 allows a plurality of control ECUs to input and output data to and from other control ECUs connected to bus cables 31 different from the bus cables 31 to which they are connected.

[0020] The driving operation ECU 25 is connected to operation members such as a steering wheel 51, a brake pedal 52, an accelerator pedal 53, and a shift lever 54 that allow the user to control the driving of the automobile 2. When an operation member is operated, the driving operation ECU 25 outputs data including whether or not the operation was performed, the amount of operation, etc. to the vehicle network 30. The driving operation ECU 25 may also execute processing related to the operation of the operation member and include the processing results in the data.

[0021] The detection ECU 26 is connected to host vehicle sensors for detecting the driving environment of the vehicle 2, such as a speed sensor 61 for detecting the speed of the vehicle 2, an acceleration sensor 62 for detecting the acceleration of the vehicle 2, and an exterior camera 63 for capturing images outside the vehicle 2. The detection ECU 26 is also connected to a LIDAR 64 for detecting objects outside the vehicle 2 by emitting laser light, an interior camera 65 for capturing images inside the vehicle 2, and a GNSS receiver 66 for detecting the position of the vehicle 2. The exterior camera 63 may be, for example, a stereo camera, a monocular camera, or a 360-degree camera. The GNSS receiver 66 receives radio waves from multiple GNSS satellites 110, similar to those of the server GNSS receiver 12, and obtains the latitude, longitude, and altitude of the vehicle's current position, as well as the current time. This allows the current time of the vehicle 2 to be expected to match the current time measured by the server GNSS receiver 12 of the server device 5 with high accuracy. The detection ECU 26 may output detection information acquired from the host vehicle sensors, processing results based on the detection information, and the like, to the vehicle network 30. For example, the detection ECU 26 may perform recognition processing on pedestrians, traffic lights, other vehicles, the shape of the road 100, etc. outside the vehicle that are included in the image captured outside the vehicle by the exterior camera 63, and output the recognition results to the vehicle network 30. Note that, other than the in-vehicle camera 65, occupant sensors such as an in-vehicle millimeter wave sensor, a seating sensor, and a steering sensor may be connected to the detection ECU 26.

[0022] The AP communication ECU 27, as an AP communication device serving as an AP communication apparatus, establishes a wireless communication link with the base station 9 in the automobile 2. During remote control, the AP communication ECU 27 repeatedly transmits and receives data to and from the server device 5 of the remote control device 4 using the wireless communication link established with the base station 9.

[0023] The V2V communication ECU 28, as a V2V communication device serving as a V2V communication device, executes V2V communication between the automobile 2 and other automobiles. The V2V communication ECU 28 communicates with other automobiles that have established wireless communication links with the base station 9. This enables the V2V communication ECU 28 to repeatedly send and receive data to and from the server device 5 of the remote control device 4 for remote control via the other automobiles.

[0024] A timer 42 and a memory 41 are connected to the cruise control ECU 24. The memory 41 is a computer-readable recording medium that stores programs, data, and the like to be executed by the cruise control ECU 24. The memory 41 may store data for driving assistance such as lane keeping and distance control, high-precision map data for autonomous driving, and the like. The cruise control ECU 24 reads and executes the programs from the memory 41. This allows the cruise control ECU 24 to function as a control unit for controlling the cruise of the automobile 2. The driving control ECU 24, which serves as a control unit for controlling the driving of the automobile 2, acquires information from each part of the control system 3 of the automobile 2 in order to control the driving of the own vehicle.

[0025] For example, when the travel control ECU 24 acquires information on the occupant's manual operation from the driving operation ECU 25, it generates a vehicle control value based on the occupant's manual operation as is, or generates a vehicle control value that has been fine-tuned to assist the occupant's manual operation. During autonomous driving, for example, the cruise control ECU 24 acquires information from the detection ECU 26 and the like, determines the vehicle's position in high-precision map data and the possibility of collision with other vehicles, and generates vehicle control values ​​for autonomous driving. Autonomous driving can be achieved, for example, by vehicle control values ​​such as steering for lane keeping to maintain the lateral position of the vehicle 2 near the center of the lane, and vehicle control values ​​such as acceleration / deceleration to maintain the longitudinal position of the vehicle 2 and ensure a safe inter-vehicle distance. The cruise control ECU 24 then outputs the generated host vehicle control values ​​to the drive ECU 21, the steering ECU 22, and the braking ECU 23 via the vehicle network 30. This allows the cruise control ECU 24, as a host vehicle control value generating unit, to generate host vehicle control values ​​to be used for cruise control of the automobile 2 based on the operation by a passenger or automatic driving of the host vehicle.

[0026] Furthermore, the driving control ECU 24, which serves as a control unit for controlling the driving of the automobile 2, may use the AP communication ECU 27 to communicate with the server device 5 of the remote control device 4 when remotely controlling the driving of the vehicle, and may acquire remote control values ​​from the server device 5. The driving control ECU 24 may use the V2V communication ECU 28 instead of the AP communication ECU 27. The remote control device 4 may generate a remote control value equivalent to the host vehicle control value generated by the cruise control ECU 24 described above by the same process as the generation process performed by the cruise control ECU 24 for autonomous driving. The cruise control ECU 24 outputs the acquired remote control values ​​to the drive ECU 21, the steering ECU 22, and the braking ECU 23 via the vehicle network 30. As a result, the driving control ECU 24 can execute driving control according to the remote control values ​​repeatedly received from the remote control device 4 as a remote driving control unit.

[0027] As a driving controller, the drive ECU 21 controls the operation of the driving force power source such as the engine or motor of the automobile 2 by inputting control values ​​generated or acquired from the driving control ECU 24, and controls the acceleration of the automobile 2 in accordance with the control values. The steering ECU 22, as a driving controller, receives control values ​​generated or acquired from the driving control ECU 24 and controls the operation of a steering force generating unit such as the steering motor 51 of the automobile 2. As a result, the steering ECU 22 controls the driving direction of the automobile 2 in accordance with the control values. The brake ECU 23, as a travel controller, receives control values ​​generated or acquired from the travel control ECU 24 and controls the operation of a braking force generating unit such as a brake pump of the automobile 2. As a result, the brake ECU 23 controls the deceleration of the automobile 2 in accordance with the control values.

[0028] FIG. 4 is a timing chart illustrating the flow of basic remote control in the remote control system 1 of FIG. FIG. 4 shows an example in which a single automobile 2 repeatedly communicates with a remote control device 4 through a communication system 7 including a communication network 8. The communication system 7 may cause delays in the upstream and downstream data. The amount of delay is likely to vary depending on the communication environment. In FIG. 4, time flows from top to bottom. Note that the step numbers in FIG. 4 correspond to the figures described below.

[0029] 4, first, the automobile 2 acquires information about its own vehicle in step ST1, and transmits the information to the remote control device 4 via the communication system 7 in step ST6. The automobile 2 may transmit to the remote control device 4 at least the detection information of its own vehicle sensor, including an image captured outside the vehicle by the exterior camera 63 installed in the automobile 2, and the latest position and time of the automobile 2. After receiving this uplink data from the automobiles 2, the remote control device 4 acquires the latest host vehicle information for the automobiles 2 in step ST22. The remote control device 4 generates and acquires a remote control value using the host vehicle information received from each automobile 2 in steps ST23 and ST24, and transmits the acquired remote control value to the automobiles 2 via the communication system 7 in step ST25. The amount of delay between when the remote control device 4 receives the uplink data from the automobiles 2 and when it transmits the downlink data including the remote control value is likely to vary depending on the remote processing load of the remote control device 4 at that time. After receiving the downlink data from the remote control device 4, the automobile 2 executes driving control using the remote control values ​​in step ST9. The automobile 2 acquires from the remote control device 4 remote control values ​​that can be input to the driving controller in the same way as the host vehicle control values ​​generated by the host vehicle, and executes driving control. The automobile 2 and the remote control device 4 repeat the above-described series of processes. As a result, the automobile 2 receives multiple remote control values ​​repeatedly transmitted from the remote control device 4, and can continuously execute driving control using the remote control values. The automobile 2 executes driving control of the automobile 2 using a driving control cycle corresponding to the reception cycle of the multiple remote control values ​​transmitted from the remote control device 4, thereby realizing driving under remote control.

[0030] In this way, the remote control device 4 can periodically obtain remote control values ​​that can be used to control the driving of the automobile 2 based on the own vehicle information received from the automobile 2 and transmit them to the automobile 2. In addition, each automobile 2 can periodically and repeatedly receive remote control values ​​generated by the remote control device 4 based on the automobile information transmitted by the automobile 2 to the remote control device 4, and periodically perform driving control using the remote control values. Here, the remote control device 4 may generate remote control values ​​that can be used for driving control as they are when received by the automobile 2. Such remote control values ​​include at least a steering amount that can be input to the steering ECU 22 and used to control the steering of the automobile 2. In addition to this, remote control values ​​include acceleration / deceleration control amounts that can be input to the drive ECU 21 and the brake ECU 23 and used to control the acceleration / deceleration of the automobile 2. Furthermore, the remote control value generating device 6 of this embodiment generates, in addition to the control variables for directly controlling the traveling of the automobile 2 itself, remote control values ​​for controlling the lighting state of exterior vehicle lights such as blinkers to enhance the safety of traveling of the automobile 2. Exterior vehicle lights such as blinkers are controlled from an off state to a lighting state or a flashing state when, for example, turning right or left or decelerating and stopping.

[0031] Incidentally, the remote control device 4 used for such remote control can be provided in one-to-one correspondence with the automobiles 2, but it is preferable to provide it in one-to-multiple correspondence with the automobiles 2. In this case, the remote control device 4 must transmit the above-mentioned remote control value to each of the multiple vehicles 2, and periodically and repeatedly for each vehicle 2. This places a heavy processing load on the remote control device 4. As shown by the dashed line in Fig. 4, if processing for other vehicles takes too long, the timing of transmitting subsequent remote control values ​​will be delayed. If each vehicle 2 is unable to continue to receive the remote control values ​​periodically at an appropriate time interval corresponding to the vehicle's own driving control period, it may be unable to maintain appropriate driving control using the remote control values. For example, if the reception interval of the remote control values ​​exceeds the driving control period of the vehicle 2, it will become difficult for the vehicle 2 to continue driving control using the remote control values ​​continuously. As such, there is a demand for improvements in the remote driving control of the automobile 2.

[0032] FIG. 5 is an explanatory diagram of an example of a plurality of exterior cameras 63 provided on the automobile 2 of FIG. FIG. 5 shows the multiple exterior cameras 63 as a front exterior camera 71, a front right exterior camera 72, a rear right exterior camera 73, a front left exterior camera 74, a rear left exterior camera 75, and a rear exterior camera 76. The automobile 2 is provided with a plurality of exterior cameras 63, for example, to capture images of the surrounding environment outside the automobile 2 while the automobile is traveling in an automated or assisted driving mode, or to ensure collision safety while the driver is driving manually.

[0033] The front exterior camera 71 is provided facing forward at the front of the automobile 2. In Fig. 5, the front exterior camera 71 is provided in the passenger compartment of the automobile 2, for example, at the center of the front edge of the roof of the automobile 2. The front exterior camera 71 may capture an image of the area in front of the automobile 2 to generate an exterior captured image FR of the front side of the automobile 2. The right front exterior camera 72 is provided facing forward on the right side of the automobile 2. In FIG. 5, the right front exterior camera 72 is provided in the passenger compartment of the automobile 2, for example, at the right end of the dashboard. The right front exterior camera 72 may capture an image of the range on the right front side of the automobile 2 to generate an exterior captured image RF of the right front side of the automobile 2. The right front exterior camera 72 may also be provided on the right front door or door mirror. The right rear exterior camera 73 is provided facing backward on the right side of the automobile 2. In Fig. 5, the right rear exterior camera 73 is provided, for example, on the right front door or door mirror, which are exterior members that can be opened and closed, of the automobile 2. The right rear exterior camera 73 may capture an image of the range on the right rear side of the automobile 2 to generate an exterior captured image RR of the right rear side of the automobile 2. The left front exterior camera 74 is provided facing forward on the left side of the automobile 2. In FIG. 5, the left front exterior camera 74 is provided in the passenger compartment of the automobile 2, for example, at the left end of the dashboard. The left front exterior camera 74 may capture an image of the left front side of the automobile 2 to generate an exterior captured image LF of the left front side of the automobile 2. The left front exterior camera 74 may also be provided in the left front door or door mirror. The left rear exterior camera 75 is provided facing rearward on the left side of the automobile 2. In Fig. 5, the left rear exterior camera 75 is provided, for example, on the left front door or door mirror, which are exterior members that can be opened and closed, of the automobile 2. The left rear exterior camera 75 may capture an image of the left rear range of the automobile 2 to generate an exterior captured image LR of the left rear side of the automobile 2. The rear exterior camera 76 is provided facing rearward at the rear of the automobile 2. In Fig. 5, the rear exterior camera 76 is provided, for example, in the upper center portion of a rear gate, which is an exterior member that can be opened and closed, on the automobile 2. The rear exterior camera 76 may capture an image of the area behind the automobile 2 to generate an exterior captured image BA of the rear side of the automobile 2. 5 can capture images of the 360-degree surroundings outside the automobile 2 in divided areas. The imaging range of each exterior camera 71-76 overlaps with the imaging range of the other exterior cameras 71-76 that capture adjacent areas. Based on the exterior images captured by the exterior cameras 71-76, the control system 3 of the automobile 2 can predict, for example, a collision or lane departure and execute driving control to avoid such an occurrence. Furthermore, it is desirable that the vehicle information transmitted from the vehicle 2 to the server device 5 for remote control includes all of the exterior images captured by these exterior cameras 71 to 76. This allows the remote control device 4 to obtain exterior images of the entire surroundings of each vehicle 2, determine the driving conditions of each vehicle 2, and generate remote control values ​​appropriate for the driving conditions of each vehicle 2.

[0034] FIG. 6 is a flowchart of the remote travel control in the first embodiment by the control system 3 of the automobile 2 in FIG. For example, the driving control ECU 24 of the control system 3 of the automobile 2 repeatedly executes the remote driving control of FIG. 6 to control the driving of the own vehicle. Note that a control ECU other than the driving control ECU 24 of the control system 3 of the automobile 2, for example, the remote control ECU 29 shown by the dashed line in Fig. 6, may repeatedly execute the remote driving control of Fig. 6. The same applies to the control of each of the following flowcharts.

[0035] Step ST1 is a step for acquiring host vehicle information. The cruise control ECU 24 of the control system 3 of the automobile 2 acquires the host vehicle information detected by the host vehicle. The host vehicle information may include at least an image captured outside the vehicle by the exterior camera 63 installed on the host vehicle, the latest position and time of the host vehicle acquired by the GNSS receiver 66, speed, acceleration, steering amount, and information on the status of the exterior vehicle lights.

[0036] Step ST2 is a step of acquiring a communication status or a remote processing status. During remote control, the cruise control ECU 24 acquires information about the communication status between the communication device of the host vehicle and the remote control device 4, or about the remote processing status of the remote control device 4. As such information, the cruise control ECU 24 may acquire the reception interval of multiple downlink data shown in FIG. 4. When the cruise control ECU 24 receives a remote control value, the cruise control ECU 24 may store and record the received remote control value in the memory 41 together with the reception time measured by the timer 42. In this case, the cruise control ECU 24 can acquire the reception interval of the remote control value from the memory 41.

[0037] In step ST3, the cruise control ECU 24 determines whether the communication status between the host vehicle and the remote control device 4 or the remote processing status of the remote control device 4 may affect the periodic cruise control using the remote control value of the host vehicle. As shown in FIG. 4, the cruise control ECU 24 basically needs to execute cruise control using remote control values ​​at regular cruise control intervals. In this case, the cruise control ECU 24 may determine whether the time difference (excess time) between reception of the remote control values ​​is smaller than a predetermined threshold, based on the cruise control period. If the time difference is smaller than the threshold, the cruise control ECU 24 determines that remote cruise control may be affected and proceeds to step ST5. If the time difference is equal to or greater than the threshold, the cruise control ECU 24 determines that remote cruise control may not be affected and proceeds to step ST4.

[0038] In step ST4, all of the vehicle exterior captured images are selected. The cruise control ECU 24 selects all of the vehicle exterior captured images captured by the plurality of vehicle exterior cameras 63 as images to be transmitted. After that, the cruise control ECU 24 proceeds to step ST6.

[0039] In step ST5, the driving control ECU 24 selects, as images to be transmitted, only a portion of the images taken outside the vehicle by the plurality of outside cameras 63. Then, the driving control ECU 24 proceeds to step ST6.

[0040] Step ST6 is a step of transmitting the vehicle information. The cruise control ECU 24 transmits the acquired vehicle information to the remote control device 4. The cruise control ECU 24 transmits the vehicle information to the remote control device 4 using the communication path established by the AP communication ECU 27 or the communication path established by the V2V communication ECU 28. The vehicle information transmitted from the vehicle 2 is received by the server communication device 11 of the server device 5 of the remote control device 4, for example, via a base station 9, a carrier communication network, or the Internet. The remote control device 4 uses the vehicle information received from each vehicle 2 to generate a remote control value for that vehicle 2 and transmits it to the vehicle 2 that sent it.

[0041] Step ST7 is a step for waiting for reception of a remote control value. The cruise control ECU 24 waits for reception of the remote control value as downlink data from the remote control device 4. The cruise control ECU 24 repeats this process until it receives the remote control value from the remote control device 4. When the AP communication ECU 27 or the V2V communication ECU 28 receives the downlink data of the remote control value transmitted from the remote control device 4 to the host vehicle, the cruise control ECU 24 proceeds to step ST8.

[0042] Step ST8 is a step for recording the reception time. The cruise control ECU 24 executes cruise control using the remote control value received from the remote control device 4. The cruise control ECU 24 outputs the remote control value received for remote control to, for example, the drive ECU 21, the steering ECU 22, and the braking ECU 23. The drive ECU 21, the steering ECU 22, and the braking ECU 23 execute their respective cruise controls using the input remote control value. As a result, the cruise of the automobile 2 is controlled by the remote control value generated by the remote control device 4.

[0043] Step ST9 is a step for executing cruise control based on the remote control value. The cruise control ECU 24 records the reception time. When the cruise control ECU 24 receives the remote control value, it stores and records the received remote control value in the memory 41 together with the reception time measured by the timer 42. The cruise control ECU 24 may also store and record in the memory 41 the position at the time of reception.

[0044] Step ST10 is a step for determining the end of remote control. The cruise control ECU 24 determines whether or not to end cruise control. For example, if the occupant operates an ignition switch (not shown), the cruise control ECU 24 determines to end cruise control and ends this control. If cruise control is not to be ended, the cruise control ECU 24 returns the process to step ST1. The cruise control ECU 24 repeatedly executes, for example, the remote cruise control described above until it determines to end cruise control in step ST10. As a result, the cruise of the automobile 2 continues to be controlled by a plurality of remote control values ​​repeatedly generated by the remote control device 4.

[0045] Next, the process of selecting the outside-of-vehicle captured image in step ST5 will be described in detail.

[0046] Step ST11 is a step of acquiring information about the driving environment of the host vehicle. The driving control ECU 24 acquires information about the driving environment of the host vehicle. The driving control ECU 24 may acquire information about surrounding objects that can be acquired by analyzing an image captured outside the vehicle.

[0047] Step ST12 is a step of acquiring past remote control values ​​as cruise control information. The cruise control ECU 24 acquires information about the cruise control situation. The cruise control ECU 24 may acquire multiple past remote control values ​​recorded in the memory 41, along with information about the time each remote control value was received and the location where each remote control value was received.

[0048] Step ST13 is a step for estimating a future course. The cruise control ECU 24 estimates the immediate future course of the host vehicle. Based on the past multiple remote control values ​​acquired in step ST12, the cruise control ECU 24 may estimate the immediate future course of the host vehicle by estimating that the host vehicle will travel in the direction of extension of those values. In particular, when the cruise control ECU 24 receives from the remote control device 4 a remote control value for flashing exterior vehicle lights along with steering amounts and control amounts for acceleration and deceleration for controlling the actual driving of the automobile 2, the cruise control ECU 24 can easily estimate the course of a right or left turn at an intersection. For example, if the only cruise control information received is deceleration before an intersection, it may be unclear whether the automobile 2 intends to continue straight through the intersection or to turn right or left. When the automobile 2 intends to turn right or left at an intersection, it needs to execute control to flash the turn signal on the side of the turn. By receiving such a remote control value for flashing exterior vehicle lights such as turn signals, the cruise control ECU 24 can accurately estimate whether the automobile 2 intends to continue straight through the intersection or to turn right or left.

[0049] Step ST14 is a step for determining the driving situation. The cruise control ECU 24 determines the driving situation of the vehicle. The automobile 2 not only travels straight along the road 100 at a constant speed, but also changes lanes, merges onto another road 100 in a merging section, branches off onto another road 100, or heads toward an intersection. The cruise control ECU 24 may determine which of these driving situations the driving situation of the vehicle corresponds to based on the information acquired in steps ST11 to ST13.

[0050] Step ST15 is a step of selecting some of the outside-of-vehicle images according to the driving conditions. The driving control ECU 24 selects the outside-of-vehicle images to be transmitted to the remote control device 4 according to the estimated course and driving conditions of the vehicle. The driving control ECU 24 selects only some of the outside-of-vehicle images captured by the multiple outside-vehicle cameras 63 installed on the vehicle as the outside-of-vehicle images to be transmitted. Here, when the cruise control ECU 24 receives a request for vehicle exterior captured images to select for transmission from the server device 5 of the remote control device 4, the cruise control ECU 24 may select some of the images including the requested vehicle exterior captured images. As in a third embodiment described later, the server device 5 of the remote control device 4 can determine for itself which direction or range of vehicle exterior captured images are required and transmit the request to the automobile 2. In this case, the cruise control ECU 24 may select some of the images according to the driving situation, including the vehicle exterior captured images selected and requested by the server device 5 of the remote control device 4. This allows the cruise control ECU 24 to select only a portion of the outside-of-vehicle images captured by the plurality of outside-vehicle cameras 63 provided on the host vehicle, depending on the driving conditions (driving control conditions, driving environment) of the host vehicle. Furthermore, in step ST6, the amount of vehicle information transmitted from the automobile 2 to the remote control device 4 by the cruise control ECU 24 is reduced compared to when all of the vehicle exterior images captured by the multiple exterior cameras 63 installed on the vehicle are included. Transmission of the vehicle information can be completed in a short time.

[0051] In this way, the driving control ECU 24 is provided in the automobile 2 as a transmission control unit, and can periodically transmit to the remote control device 4 vehicle information including at least exterior images captured by a plurality of exterior cameras 63 provided in the vehicle, the vehicle's position and time. The cruise control ECU 24 can select a part of the plurality of outside-of-vehicle images according to at least one of the communication status between the communication device of the automobile 2 and the remote control device 4, the processing status of the remote control device 4, and the driving status of the vehicle (driving control status, driving environment). The cruise control ECU 24 can also transmit the selected part of the outside-of-vehicle images.

[0052] FIG. 7 is a flowchart of the remote control of the server device 5 of the remote control device 4 of FIG. 1 in the first embodiment. The server CPU 15 of the server device 5 of the remote control device 4 repeats the remote control shown in FIG.

[0053] Step ST21 is a step for determining whether or not there is unprocessed received vehicle information. The server CPU 15 of the server device 5 of the remote control device 4 determines whether or not there is unprocessed received vehicle information. If there is no unprocessed vehicle information, the server CPU 15 repeats this process. If there is unprocessed vehicle information, the server CPU 15 proceeds to step ST22 to process it.

[0054] Step ST22 is a step for acquiring the vehicle information. The server CPU 15 acquires the vehicle information of the vehicle 2 that has not yet been processed.

[0055] Step ST23 is a step of instructing the generation of remote control values ​​based on the outside-of-vehicle captured images acquired from each vehicle. The server CPU 15 provides the host vehicle information to the remote control value generation device 6 connected to the server device 5 of the remote control device 4 and instructs it to generate a remote control value. The remote control value generation device 6 generates a remote control value using the provided host vehicle information. The remote control value generation device 6 may generate a remote control value for each vehicle 2 based on the outside-of-vehicle captured image taken by the outside camera 63 included in the host vehicle information of each vehicle 2. The remote control value generation device 6 may generate a remote control value that can be used directly in the vehicle 2 that transmitted the host vehicle information, for example, for lane keeping control or vehicle distance control, by performing processing equivalent to that performed by the cruise control ECU 24 of the vehicle 2 that transmitted the host vehicle information. The remote control value generation device 6 outputs the generated remote control value to the server device 5.

[0056] Step ST24 is a step for acquiring a remote control value. The server CPU 15 acquires the remote control value generated by the remote control value generation device 6 from the remote control value generation device 6. The remote control value generated by the remote control value generation device 6 may include a steering amount that can be input to the steering ECU 22 of the automobile 2 and used to control the steering of the automobile 2. The remote control value may also include an acceleration / deceleration control amount that can be input to the drive ECU 21 or the braking ECU 23 and used to control the acceleration / deceleration of the automobile 2, a remote control value for controlling the lighting state of exterior lights such as blinkers of the automobile 2, and the like.

[0057] Step ST25 is a transmission step. The server CPU 15 transmits the remote control value to the sending vehicle 2 related to the processing. After transmitting the vehicle information in step ST6 of FIG. 6, the control system 3 of the sending vehicle 2 waits to receive the remote control value in step ST7. The control system 3 of the sending vehicle 2 performs remote driving control using the remote control value received from the server device 5 in step ST9. Thereafter, the server CPU 15 returns the process to step ST21. In this way, the server CPU 15 of the server device 5 of the remote control device 4 can repeatedly generate and transmit, for each of the multiple automobiles 2, a remote control value based on the latest host vehicle information of each automobile 2 by repeating the remote control of Fig. 7. The server CPU 15 can continuously generate, for example, a remote control value of the steering amount for lane keeping control, or a remote control value of the acceleration / deceleration amount for vehicle distance control or vehicle speed control.

[0058] FIG. 8 is an explanatory diagram of the correspondence between the driving conditions of the automobile 2 in FIG. 1 and a selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras 63. In FIG. FIG. 8 shows the driving conditions of Case 1 to Case 3.

[0059] Case 1 is a situation in which the automobile 2 is traveling on a straight road 100. In this traveling situation, the cruise control ECU 24 determines in step ST3 that the communication situation or the remote processing situation may affect the cruise control. In step ST5, the cruise control ECU 24 selects only a portion of the outside-of-vehicle images captured by the multiple outside-vehicle cameras 63 installed on the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is traveling on a straight road 100, and in step ST15, selects only the outside-vehicle captured images FR of the front side, which is the traveling direction of the host vehicle, as images to be transmitted.

[0060] Thus, in case 1, the amount of image information included in the vehicle information from the automobile 2 to the remote control device 4 is reduced, and the transmission of uplink data can be completed in a short time. As a result, the remote control device 4 can generate and transmit remote control values ​​to the automobile 2 at a faster timing. In this embodiment, it is expected that the time difference between the reception interval of the remote control values ​​and the cruise control cycle at the automobile 2 will be wider. As a result, the automobile 2 can continue to receive remote control values ​​without delay in the cruise control cycle, even if the communication load between the automobile 2 and the remote control device 4 is high or the processing load of the remote control device 4 is high. The automobile 2 can continue to stably perform remote cruise control. In addition, the automobile 2 can continue to travel under remote control along the straight road 100 on which it is traveling by controlling its travel using remote control values ​​generated by the remote control device 4 based on the externally captured image FR of the front side of the vehicle, which is the direction of travel.

[0061] Case 2 is a situation in which the automobile 2 is about to change lanes from the driving lane to the passing lane 101 on a two-lane road 100. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication conditions or remote processing conditions may affect cruise control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed on the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is about to change lanes. In step ST15, the cruise control ECU 24 selects, as images to be transmitted, only the exterior-captured image FR of the front side, which is the direction of travel of the host vehicle, the exterior-captured image RF of the right front side, which is the side of the lane to be changed to, and the exterior-captured image RR of the right rear side, which is the side of the lane to be changed to.

[0062] In this way, in case 2, the amount of image information included in the vehicle information from the automobile 2 to the remote control device 4 is reduced, and the transmission of uplink data can be completed in a short time. As a result, the remote control device 4 can generate and transmit remote control values ​​to the automobile 2 at a faster timing. In this embodiment, it is expected that the time difference between the reception interval of the remote control values ​​and the cruise control cycle of the automobile 2 will be wider. As a result, the automobile 2 can continue to receive remote control values ​​without delay in the cruise control cycle, even when the communication load between the automobile 2 and the remote control device 4 is high or the processing load of the remote control device 4 is high. The automobile 2 can continue to stably perform cruise control when changing lanes remotely. The automobile 2 controls its travel using remote control values ​​generated by the remote control device 4 based on the outside-vehicle-taken image FR of the front side, which is the direction of travel, the outside-vehicle-taken image RF of the front right side, and the outside-vehicle-taken image RR of the rear right side, which is the side of the lane to be changed to. This allows the automobile 2 to perform remotely controlled travel to change lanes from the travel lane to the passing lane 101. When the automobile 2 changes lanes to merge at a section where two roads 100 merge, the cruise control ECU 24 may select an outside-of-vehicle image in the same pattern as in Case 2 described above. In addition, when the automobile 2 changes lanes to leave the first road 100 and branch off onto the second road 100, the driving control ECU 24 may select an outside-vehicle image using a pattern similar to that of case 2 described above.

[0063] Case 3 is a situation in which the automobile 2 is traveling along a straight road 100 toward an intersection 102. In this traveling situation, the cruise control ECU 24 determines in step ST3 that the communication situation or the remote processing situation may affect the cruise control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed in the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is traveling toward the intersection 102. In step ST15, the cruise control ECU 24 selects only the exterior image FR of the front side, which is the traveling direction of the host vehicle, the exterior image RF of the right front side, and the exterior image LF of the left front side, as images to be transmitted.

[0064] As described above, in case 3, the amount of image information included in the vehicle information from the vehicle 2 to the remote control device 4 is reduced, and the transmission of uplink data can be completed in a short time. Therefore, the remote control device 4 can generate and transmit remote control values ​​at a faster timing. In this embodiment, the time difference between the reception interval of the remote control values ​​and the cruise control period of the vehicle 2 can be expected to increase. As a result, even if the communication load between the vehicle 2 and the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the cruise control period, and can stably continue remote cruise control when passing through the intersection 102. Furthermore, even if the processing load of the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the cruise control period, and can stably continue remote cruise control when passing through the intersection 102. The automobile 2 also controls its travel using remote control values ​​based on the front outside-vehicle captured image FR, the right front outside-vehicle captured image RF, and the left front outside-vehicle captured image LF, which are capable of capturing images of the state of the intersecting road 100 at the intersection 102 ahead in the traveling direction. This allows the automobile 2 to travel under remote control to safely pass through the intersection 102.

[0065] As described above, in this embodiment, the driving control ECU 24 of the automobile 2 serves as a transmission control unit and periodically transmits to the remote control device 4 vehicle information including at least exterior images captured by the multiple exterior cameras 63 installed on the vehicle, the position and time of the vehicle. Furthermore, the cruise control ECU 24 determines whether the communication status between the communication device of the host vehicle and the remote control device 4 or the processing status of the remote control device 4 affects the periodic cruise control using the remote control value of the host vehicle. If it determines that there is an effect, the cruise control ECU 24 selects only a portion of the outside-vehicle images captured by the multiple outside-vehicle cameras 63 installed on the host vehicle according to the driving status of the host vehicle and transmits the selected portion to the remote control device 4. As a result, in this embodiment, the amount of uplink data sent from the automobile 2 to the remote control device 4 can be reduced and the transmission of uplink data can be terminated earlier than when transmitting host vehicle information including all of the exterior images captured by the multiple exterior cameras 63 installed on the host vehicle. Based on the host vehicle information received so as to terminate the transmission earlier, the remote control device 4 can promptly generate and transmit to the automobile 2 remote control values ​​that can be used for driving control of the automobile 2. As a result, the driving control ECU 24 of the control system 3 of the automobile 2, as a remote driving control unit, can more easily receive the remote control values ​​from the remote control device 4 without delay, and can stably continue periodic driving control using the remote control values.

[0066] Moreover, in this embodiment, the cruise control ECU 24 of the control system 3 of the automobile 2 does not randomly select an outside-vehicle image to transmit from the multiple outside-vehicle cameras 63 installed on the automobile, but selects the image in accordance with the driving conditions of the automobile. This allows the remote control device 4 to generate useful remote control values ​​that adequately correspond to the driving conditions of the automobile 2, even when receiving from the automobile 2 host vehicle information from which the remaining portion of the outside-vehicle image has been removed. For example, the cruise control ECU 24 of the control system 3 of the automobile 2 can continuously execute cruise control using remote control values ​​based on the host vehicle information before the remaining portion of the outside-vehicle image has been removed, followed by cruise control using remote control values ​​based on the host vehicle information after the remaining portion of the outside-vehicle image has been removed. Moreover, the cruise control ECU 24 is expected to maintain continuity of control so that the cruise control does not change significantly between them.

[0067] Furthermore, in this embodiment, the automobile 2 determines the communication status between the automobile 2's communication device and the remote control device 4 and the impact on the processing status of the remote control device 4, and selects the outside-of-vehicle captured images to be sent to the remote control device 4. As a result, in this embodiment, there is no need to add processing for these determinations in the remote control device 4. This makes it less likely that the effect of reducing the processing load and communication load on the remote control device 4 by reducing the amount of information about the automobile 2's own vehicle will be lost.

[0068] [Second embodiment] Next, a remote control system 1 for driving an automobile 2 according to a second embodiment of the present invention will be described. The remote control system 1 of this embodiment not only determines the driving status of the vehicle, but also determines the driving stage during the driving status, and dynamically changes the selection of the outside-of-vehicle captured images according to the driving stage. Below, differences from the above-mentioned embodiment will be mainly described.

[0069] FIG. 9 is a flowchart of the process of selecting an outside-vehicle captured image in the second embodiment, which is performed by the control system 3 of the automobile 2 in FIG. For example, the driving control ECU 24 of the control system 3 of the automobile 2 executes the selection process of FIG. 9 in step ST5 of FIG. 6 in order to control the driving of the own vehicle. Steps ST11 to ST14 in Fig. 9 are the same as those in Fig. 6. After step ST14, the cruise control ECU 24 advances the process to step ST31.

[0070] Step ST31 is a step for determining the driving stage. The driving control ECU 24 determines the current driving stage of the vehicle in the driving conditions determined in step ST14. For example, the driving control of the automobile 2 in a driving situation where the automobile is parking or stopping can be divided into a plurality of driving stages, as will be described later. The first stage is a road 100 forward stage in which the automobile travels forward across the road 100 in front of the parking position and stops. The second stage is a road 100 backward stage in which the automobile travels backward from the road 100 toward the parking position. The third stage is a parking position entry and stop stage in which the automobile enters the parking position and stops. In this way, the cruise control ECU 24 determines the current driving stage in the driving situation. Then, the cruise control ECU 24 proceeds to step ST15. In step ST15, the cruise control ECU 24 selects outside-vehicle images to be transmitted to the remote control device 4 according to the determined driving situation and driving stage. The cruise control ECU 24 selects only a portion of the outside-vehicle images captured by the multiple outside-vehicle cameras 63 installed on the vehicle as the outside-vehicle images to be transmitted. Here, when the cruise control ECU 24 receives a request for vehicle exterior captured images to select for transmission from the server device 5 of the remote control device 4, the cruise control ECU 24 may select some of the images including the requested vehicle exterior captured images. As in a third embodiment described later, the server device 5 of the remote control device 4 can determine for itself which direction or range of vehicle exterior captured images are required and transmit the request to the automobile 2. In this case, the cruise control ECU 24 may select some of the images according to the driving situation and driving stage, including the vehicle exterior captured images selected and requested by the server device 5 of the remote control device 4. This eliminates the need for the cruise control ECU 24 to select all of the outside-vehicle images required for each of the multiple driving stages in one driving situation. In this embodiment, the number of outside-vehicle images to be selected can be reduced compared to when an outside-vehicle image is selected for each driving situation.

[0071] FIG. 10 is an explanatory diagram of an example of the correspondence between the driving conditions and driving stages of the automobile 2 in FIG. 1 and the selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras 63. In FIG. FIG. 10 shows, as case 4, a situation in which the automobile 2 is traveling by remote control in order to be automatically parked in a parking lot such as a valet park.

[0072] 10, the first traveling stage, Step 1 in the upper part, is the first traveling stage in this parking traveling situation. In the first traveling stage, the car 2 traveling by remote control travels along the road 100 in front of the parking position 103, passing through the parking position 103, and stops on the road 100 in front of the parking position 103. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or remote processing situation may affect driving control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed on the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is traveling on road 100 in a parking lot. In step ST31, the cruise control ECU 24 determines that the vehicle is in the first driving stage in the driving situation for parking. In step ST15, the cruise control ECU 24 selects only the exterior image FR of the front side, which is the direction of travel of the host vehicle, the exterior image LF of the front left side where the parking position 103 is located, and the exterior image LR of the rear left side where the parking position 103 is located, as images to be transmitted.

[0073] Step 2 in the middle of Figure 10 is the second driving stage in this parking driving situation. In the second driving stage, the car 2, which is driven by remote control, moves backward from a state in which the car is stopped on the road 100 in front of the parking position 103 while steering toward the parking position 103. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or remote processing situation may affect driving control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed on the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is traveling on road 100 in a parking lot. In step ST31, the cruise control ECU 24 determines that the vehicle is in a second driving stage in the driving situation in which parking is required. In step ST15, the cruise control ECU 24 selects only the exterior image BA of the rear side, which is the direction in which the host vehicle is reversing, the exterior image LF of the left front side where the parking position 103 is located, and the exterior image LR of the left rear side where the parking position 103 is located, as images to be transmitted.

[0074] Step 3 in the lower part of Figure 10 is the third driving stage in this parking driving situation. In the third driving stage, the car 2, which is driven by remote control, moves from the road 100 in front of the parking position 103 toward the parking position 103 and stops there. In this driving situation, the driving control ECU 24 determines in step ST3 that the communication situation or the remote processing situation may affect driving control. In step ST5, the driving control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed on the host vehicle. Specifically, in step ST14, the driving control ECU 24 determines that the host vehicle is traveling on road 100 in a parking lot. In step ST31, the driving control ECU 24 determines that the driving situation is the third driving stage in the parking situation. In step ST15, the driving control ECU 24 selects only the exterior image BA of the rear side, which is in the direction in which the host vehicle is reversing, and the exterior image RR of the right rear side, which is opposite the steering direction, as images to be transmitted.

[0075] In this manner, in this embodiment, the driving control ECU 24 dynamically changes the outside-of-vehicle captured image that the automobile 2 selects to transmit for each driving stage in the driving situation during parking. When the vehicle 2 is traveling under remote control for automatic parking in a parking lot, the cruise control ECU 24, which serves as a transmission control unit of the vehicle 2, dynamically changes the selected exterior image according to each stage of travel from the passage in front of the parking position until the vehicle enters the parking position and stops. For example, while the vehicle 2 is traveling until it stops in the passage in front of the parking position, the cruise control ECU 24 selects at least an exterior image of the front side of the vehicle 2 and an exterior image of the side of the parking position. While the vehicle 2 is traveling from the stopped position until it enters the parking position, the cruise control ECU 24 may select at least an exterior image of the rear side of the vehicle 2 and an exterior image of the side of the parking position. In this case, the multiple exterior cameras 63 provided on the vehicle 2 may capture images of the surroundings of the vehicle 2, divided into at least the front, right side, rear side, and left side of the vehicle 2. In this embodiment, the amount of vehicle information can be reduced compared to when all of the outside-vehicle captured images selected from the first to third driving stages are selected at once based solely on the parking situation determination. The transmission of uplink data for transmitting the vehicle information can be completed in a short time. Therefore, the remote control device 4 can generate and transmit remote control values ​​to the vehicle 2 at a faster timing. In this embodiment, the time difference between the reception interval of the remote control values ​​and the driving control period of the vehicle 2 can be expected to be increased. As a result, even when the communication load between the vehicle 2 and the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the driving control period, thereby stably continuing remote driving control. Furthermore, even when the processing load of the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the driving control period, thereby stably continuing remote driving control.

[0076] FIG. 11 is an explanatory diagram of another example of the correspondence between the driving conditions and driving stages of the automobile 2 in FIG. 1 and the selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras 63. In FIG. FIG. 11 shows, as case 5, a situation in which the automobile 2 turns left at an intersection 104 by remote control.

[0077] 11, step 1 in the upper part, which is the first traveling stage, is the first traveling stage in a traveling situation in which the vehicle is turning left. In the first traveling stage, the vehicle 2 traveling by remote control is traveling on a road 100 leading to an intersection 104, toward the intersection 104. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or remote processing situation may affect driving control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed in the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is in a driving situation in which it is making a left turn at the intersection 104. In step ST31, the cruise control ECU 24 determines that the vehicle is in the first driving stage of the driving situation in which it is making a left turn. In step ST15, the cruise control ECU 24 selects, as images to be transmitted, only the exterior-captured image FR of the front side, which is the direction of travel of the host vehicle, the exterior-captured image LF of the front left side of the turn, the exterior-captured image LR of the rear left side of the turn, and the exterior-captured image RF of the front right side of the turn on the opposite side.

[0078] Step 2 in the middle of Fig. 11 is the second travel stage in the travel situation of turning left. In the second travel stage, the car 2 traveling by remote control is traveling in the intersection 104 so as to turn left. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or remote processing situation may affect driving control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed in the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is in a driving situation in which it is making a left turn at an intersection 104. In step ST31, the cruise control ECU 24 determines that the vehicle is in a second driving stage in the driving situation in which it is making a left turn. In step ST15, the cruise control ECU 24 selects, as images to be transmitted, an exterior-captured image FR of the front side, which is the direction of travel of the host vehicle, an exterior-captured image LF of the left front side of the turn, an exterior-captured image RF of the right front side on the opposite side, and an exterior-captured image RR of the right rear side on the outside of the turn.

[0079] Step 3 in the lower part of Figure 11 is the third driving stage in the driving situation of turning left. In the third driving stage, the car 2, which is driven by remote control, has completed a left turn at the intersection 104 and is driving away from the intersection 104. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or remote processing situation may affect driving control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed in the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is in a driving situation where it is making a left turn at an intersection 104. In step ST31, the cruise control ECU 24 determines that the vehicle is in a third driving stage in a driving situation where it is making a left turn. In step ST15, the cruise control ECU 24 selects, as images to be transmitted, only the exterior-captured image FR of the front side, which is the direction of travel of the host vehicle, the exterior-captured image LF of the left front side where the vehicle is turning, and the exterior-captured image RF of the opposite front right side.

[0080] In this manner, in this embodiment, the cruise control ECU 24 dynamically changes the outside-vehicle captured image that the automobile 2 selects to transmit for each driving stage in a driving situation where the automobile 2 is turning left at an intersection 104. When the automobile 2 turns right or left at an intersection, the cruise control ECU 24, which serves as a transmission control unit of the automobile 2, dynamically changes the selected exterior image according to each stage of travel from the road before entering the intersection to the road after the turn. For example, while the automobile 2 is traveling on the road before entering the intersection, the cruise control ECU 24 selects at least an exterior image of the front side of the automobile 2. While the automobile 2 is traveling after starting to turn at an intersection, the cruise control ECU 24 selects at least an exterior image of the front side of the automobile 2, an exterior image of the right front side, an exterior image of the left front side, and an exterior image of the outer side of the turn between the right rear side and the left rear side. In this case, the multiple exterior cameras 63 provided on the automobile 2 need only capture images of the surroundings of the automobile 2 divided into at least the front, right front side, right rear side, rear side, left front side, and left rear side of the automobile 2. In this embodiment, the amount of vehicle information is reduced compared to when all of the outside-vehicle captured images selected from the first to third driving stages are selected collectively based solely on the parking situation determination. The transmission of uplink data for transmitting the vehicle information can be completed in a short time. Therefore, the remote control device 4 can generate and transmit remote control values ​​to the vehicle 2 at a faster timing. In this embodiment, the time difference between the reception interval of the remote control values ​​and the driving control period of the vehicle 2 can be expected to increase. As a result, even when the communication load between the vehicle 2 and the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the driving control period, thereby stably continuing remote driving control. Furthermore, even when the processing load of the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the driving control period, thereby stably continuing remote driving control. When turning right at the intersection 104, the outside-of-vehicle captured image can be selected by reversing the left and right sides from those in FIG.

[0081] FIG. 12 is an explanatory diagram of another example of the correspondence between the driving conditions and driving stages of the automobile 2 in FIG. 1 and the selection of some of the images outside the vehicle captured by the plurality of outside-vehicle cameras 63. In FIG. FIG. 12 shows, as case 5, a situation in which the automobile 2 overtakes the preceding vehicle 105 by remote control.

[0082] 12, step 1 in the upper part, which is the first traveling stage, is the first traveling stage in the traveling situation in which the preceding vehicle 105 is overtaken. In the first traveling stage, the automobile 2 traveling by remote control is traveling by changing lanes from the current traveling lane to the overtaking lane. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or the remote processing situation may affect driving control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed in the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is in a driving situation in which it is overtaking the preceding vehicle 105. In step ST31, the cruise control ECU 24 determines that the vehicle is in the first driving stage in the driving situation in which it is overtaking the preceding vehicle 105. In step ST15, the cruise control ECU 24 selects, as images to be transmitted, only the exterior-captured image FR of the front side, which is in the direction of travel of the host vehicle, the exterior-captured image RF of the right front side, which is on the side of the lane to be changed, and the exterior-captured image RR of the right rear side, which is on the side of the lane to be changed.

[0083] Step 2 in the middle of Fig. 12 is the second driving stage in the driving situation in which the vehicle 2 is overtaking the preceding vehicle 105. In the second driving stage, the vehicle 2, which is driven by remote control, is driving in an adjacent passing lane so as to overtake the preceding vehicle 105. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or the remote processing situation may affect cruise control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed in the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is in a driving situation in which it is overtaking the preceding vehicle 105. In step ST31, the cruise control ECU 24 determines that the vehicle is in a second driving stage in the driving situation in which it is overtaking the preceding vehicle 105. In step ST15, the cruise control ECU 24 selects, as images to be transmitted, only the exterior image FR of the front side, which is the direction of travel of the host vehicle, the exterior image LF of the left front side, which is the side of the original lane, and the exterior image LR of the left rear side, which is the side of the original lane. The preceding vehicle 105 that is about to be overtaken may be captured in at least one of the exterior image LF of the left front side or the exterior image LR of the left rear side.

[0084] Step 3 in the lower part of Fig. 12 is the third driving stage in the driving situation in which the preceding vehicle 105 is overtaken. In the third driving stage, the vehicle 2, which is driven by remote control, is driving by changing lanes from the adjacent overtaking lane back to the original lane. In this driving situation, the cruise control ECU 24 determines in step ST3 that the communication situation or the remote processing situation may affect driving control. In step ST5, the cruise control ECU 24 selects only a portion of the exterior images captured by the multiple exterior cameras 63 installed in the host vehicle. Specifically, in step ST14, the cruise control ECU 24 determines that the host vehicle is in a driving situation in which it is overtaking the preceding vehicle 105. In step ST31, the cruise control ECU 24 determines that the driving situation in which it is overtaking the preceding vehicle 105 is the third driving stage. In step ST15, the cruise control ECU 24 selects only the exterior image FR of the front side, which is the direction of travel of the host vehicle, the exterior image LF of the left front side, which is the side of the original lane, and the exterior image LR of the left rear side, which is the side of the original lane, as images to be transmitted. The overtaken preceding vehicle 105 may be captured in the exterior image LR of the left rear side.

[0085] In this manner, in this embodiment, the cruise control ECU 24 dynamically changes the outside-vehicle captured image that the automobile 2 selects to transmit for each driving stage in a driving situation in which the automobile 2 is overtaking the preceding vehicle 105. When the vehicle 2 is remotely controlled to overtake a preceding vehicle, the cruise control ECU 24, which serves as a transmission control unit of the vehicle 2, dynamically changes the selected exterior-captured image according to each stage of travel, from the current lane to the passing lane and back. For example, when the vehicle 2 is moving from the current lane to the passing lane, the cruise control ECU 24 selects at least an exterior-captured image of the front side of the vehicle 2 and an exterior-captured image of the right front side or the left rear side that is closer to the passing lane. When the vehicle 2 is moving from the passing lane back to the original lane, the cruise control ECU 24 selects at least an exterior-captured image of the front side of the vehicle 2 and an exterior-captured image of the right front side or the left rear side that is closer to the original lane. In this case, the multiple exterior cameras 63 provided on the vehicle 2 may capture images of the surroundings of the vehicle 2, divided into at least the front, right front side, right rear side, rear side, left front side, and left rear side of the vehicle 2. In this embodiment, the amount of vehicle information is reduced compared to when all of the outside-vehicle captured images selected from the first to third driving stages are selected collectively based solely on the parking situation determination. The transmission of uplink data for transmitting the vehicle information can be completed in a short time. Therefore, the remote control device 4 can generate and transmit remote control values ​​to the vehicle 2 at a faster timing. In this embodiment, the time difference between the reception interval of the remote control values ​​and the driving control period of the vehicle 2 can be expected to increase. As a result, even when the communication load between the vehicle 2 and the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the driving control period, thereby stably continuing remote driving control. Furthermore, even when the processing load of the remote control device 4 is high, the vehicle 2 can continue to receive remote control values ​​without delay in the driving control period, thereby stably continuing remote driving control.

[0086] [Third embodiment] Next, a remote control system 1 for controlling the driving of an automobile 2 according to a third embodiment of the present invention will be described. In the above-described embodiment, only each remotely controlled automobile 2 determines its own driving status or each driving stage, and selects some of the images captured outside the automobile according to the determination result. Communication of remote control values ​​may be delayed not only due to the processing load of the remote control device 4, but also due to the communication load between the multiple automobiles 2 and the remote control device 4. In this case, the processing load of the remote control device 4 itself may not be high enough to cause a communication delay of the remote control values. Under such circumstances, the remote control device 4 can execute processes other than the process of generating remote control values ​​for the multiple automobiles 2 it controls. The following mainly describes the differences from the above-described embodiment.

[0087] FIG. 13 is a flowchart of the remote control of the server device 5 of the remote control device 4 of FIG. 1 in the third embodiment. The server CPU 15 of the server device 5 of the remote control device 4 repeats the remote control shown in FIG. The processing from step ST21 to step ST24 is the same as that in Fig. 7. After step ST24, the server CPU 15 advances the processing to step ST41.

[0088] Step ST41 is a step for determining whether or not the additional process is executable. The server CPU 15 determines whether or not the server device 5 of the remote control device 4 is able to execute the additional process. For example, if the number of automobiles 2 managed is small, the processing load of the server device 5 of the remote control device 4 will be small. The server CPU 15 may, for example, count the number of automobiles 2 currently managed, and determine that the additional process is executable if the count value is smaller than a threshold value. Alternatively, for example, the server CPU 15 may determine that the additional process is executable if the number of processing steps per most recent unit time is smaller than a threshold value. Furthermore, the server CPU 15 may determine that the additional process is executable, for example, at predetermined time intervals. If the additional process is executable, the server CPU 15 advances the process to step ST42. If the additional process is not executable, the server CPU 15 advances the process to step ST25.

[0089] Step ST42 is a step for determining other vehicles in the missing direction. The server CPU 15 determines the situation in the direction that is missing in the host vehicle information of the vehicle 2 currently being processed. The vehicle 2 can transmit only some, but not all, of the exterior images captured by the multiple exterior cameras 63 to the server device 5 of the remote control device 4. In this case, the server CPU 15 does not determine the driving situation for the direction that is not included in the exterior images of the host vehicle information of the vehicle 2 currently being processed. For example, if another vehicle approaches from behind the host vehicle while turning left at the intersection 104 in the middle of FIG. 11, the server CPU 15 does not determine the driving situation of the other vehicle approaching from behind. The server CPU 15 may determine the situation in the direction that is missing in the host vehicle information of the vehicle 2 currently being processed based on the vehicle information of the other vehicles, traffic information, control information, etc., other than the host vehicle information of the vehicle 2 currently being processed.

[0090] Step ST43 is a step for determining the influence on the traveling of the vehicle 2. The server CPU 15 determines whether or not the other moving object in the missing direction determined in step ST42 will have an influence on the traveling of the vehicle 2 being processed. For example, when turning left at intersection 104 in the middle of Fig. 11, there is a possibility that the speed of another vehicle approaching from behind the own vehicle is relatively high and the own vehicle will not be able to exit intersection 104 before reaching intersection 104. In this case, the server CPU 15 determines that there is a possibility that this will affect the running of the vehicle 2 being processed, and proceeds to step ST44. On the other hand, there may be cases where the speed of another vehicle approaching from behind the own vehicle is low and the own vehicle can exit the intersection 104 before reaching the intersection 104. The server CPU 15 determines that this will not affect the running of the vehicle 2 currently being processed, and proceeds to step ST25.

[0091] Step ST44 is a step for generating a transmission request for an image in the direction of the approaching vehicle. The server CPU 15 generates a transmission request for an outside-vehicle image capable of capturing an image in the direction of the approaching vehicle. The server CPU 15 generates a transmission request for, for example, a rear outside-vehicle image BA. Thereafter, the server CPU 15 proceeds to step ST25. In this case, in step ST25, the server CPU 15 transmits a request to transmit the image generated in step ST44 together with the acquired remote control value to the sending vehicle 2 related to the processing. After transmitting the vehicle information in step ST6 of Fig. 6, the control system 3 of the sending vehicle 2 waits to receive the remote control value in step ST7. The control system 3 of the sending vehicle 2 performs remote driving control using the remote control value received from the server device 5 in step ST9. Thereafter, the server CPU 15 returns the process to step ST21. In this way, the server CPU 15 of the server device 5 of the remote control device 4 can repeatedly generate and transmit, for each of the multiple automobiles 2, a remote control value based on the latest host vehicle information of each automobile 2 by repeating the remote control of Fig. 13. Furthermore, if there is room for processing, the server CPU 15 may execute the processes from step ST41 to step ST44. The server CPU 15 can execute a determination process for each automobile 2 regarding the direction in which the host vehicle information is lacking, and generate and transmit a request to transmit an outside-vehicle image to enhance safety.

[0092] FIG. 14 is a flowchart of the process of selecting an outside-vehicle captured image by the control system 3 of the automobile 2 in FIG. 1 in the third embodiment. For example, the driving control ECU 24 of the control system 3 of the automobile 2 executes the selection process of FIG. 14 in step ST5 of FIG. 6 in order to control the driving of the own vehicle. Steps ST11 to ST14, step ST31, and step ST15 in Figure 14 are the same as those in Figure 9. After step ST15, the cruise control ECU 24 advances the process to step ST51.

[0093] Step ST51 is a step for determining whether or not there is a request to transmit images. The driving control ECU 24 determines whether or not there is a request to transmit outside-vehicle images from the remote control device 4. If there is a request to transmit images, the driving control ECU 24 proceeds to step ST52. If there is no request to transmit images, the driving control ECU 24 ends the processing of FIG. 14 and proceeds to step ST6 of FIG. 6. In this case, the driving control ECU 24 transmits some of the outside-vehicle images selected in step ST15 to the remote control device 4.

[0094] In step ST52, the cruise control ECU 24 adds the requested outside-vehicle image to the part of the outside-vehicle images selected in step ST15.

[0095] Step ST53 is a step for determining whether the selected image after addition maintains a portion of the entire image. The cruise control ECU 24 determines whether the selected outside-vehicle images after addition maintain a portion of the outside-vehicle images captured by the multiple outside-vehicle cameras 63 of the host vehicle. If the selected outside-vehicle images after addition are not a portion but all of the outside-vehicle images captured by the multiple outside-vehicle cameras 63 of the host vehicle, the cruise control ECU 24 proceeds to step ST54. On the other hand, if the selected portion of the outside-vehicle images remains the same even after addition, the cruise control ECU 24 ends the process of FIG. 14 and proceeds to step ST6 of FIG. 6. In this case, the cruise control ECU 24 transmits the outside-vehicle images requested by the remote control device 4 to the remote control device 4, along with the portion of the outside-vehicle images selected in step ST15.

[0096] In step ST54, the driving control ECU 24 reselects some images for transmission from the images outside the vehicle captured by the plurality of outside cameras 63 of the vehicle. At this time, the cruise control ECU 24 may reselect images to be transmitted so that the images include the requested outside-vehicle captured image and are part of the outside-vehicle captured images captured by the plurality of outside-vehicle cameras 63 of the host vehicle. Thereafter, the driving control ECU 24 ends the processing in Fig. 14 and proceeds to step ST6 in Fig. 6. In this case, in step ST54, the driving control ECU 24 transmits to the remote control device 4 a part of the vehicle outside captured images that have been reselected so as to include the vehicle outside captured image requested by the remote control device 4. In addition, the cruise control ECU 24 may select not only some but all of the outside-of-vehicle images after the addition in step ST53. In this case, the cruise control ECU 24 may exceptionally select all of the outside-of-vehicle images captured by the multiple outside-vehicle cameras 63 of the host vehicle and end the processing in FIG. 14. In this case, the cruise control ECU 24 proceeds to step ST6 in FIG. 6 and exceptionally transmits all of the outside-of-vehicle images captured by the multiple outside-vehicle cameras 63 of the host vehicle to the remote control device 4.

[0097] As described above, in this embodiment, the server CPU 15 of the server device 5 of the remote control device 4 functions as an event determination unit. The server CPU 15 determines whether or not there is an event, such as another vehicle, that may affect the traveling of the vehicle 2 in the direction of the outside-vehicle captured image that is missing in the host vehicle information received from the vehicle 2. The server CPU 15 of the server device 5 then transmits to the vehicle 2 a remote control value generated by the remote control value generation device 6, as well as a request to send the outside-vehicle captured image in the direction where it is determined that there is an event. Furthermore, the driving control ECU 24 of the control system 3 of the automobile 2 may receive a transmission request as a transmission control unit from the remote control device 4. In this case, the driving control ECU 24 can select a portion of the outside-of-vehicle images captured by the multiple outside-vehicle cameras 63 installed in the vehicle, including the outside-of-vehicle image requested by the remote control value generation device 6, and transmit the portion to the remote control value generation device 6. Thereafter, the server CPU 15 of the server device 5 can process the outside-of-vehicle captured image in the direction in which it is determined that an influential event, such as another vehicle, is present in the processing of steps ST21 to ST24 in Fig. 13, and can generate a remote control value. The server CPU 15 can generate a remote control value for remotely controlling the automobile 2 so as to more appropriately correspond to the driving situation of the automobile 2.

[0098] The server CPU 15 of the server device 5 of the remote control device 4 serves as an event determination unit and determines whether or not there is an event that may affect the driving of the vehicle 2 in the direction of the outside-vehicle captured images that are missing in the host vehicle information received from the vehicle 2. The server CPU 15 serves as a remote control value generation unit and can transmit to the vehicle 2 a request to transmit the outside-vehicle captured images in the direction in which the event determination unit determines that there is an event, along with the remote control value generated by the server CPU 15. Furthermore, when receiving a transmission request from the remote control device 4, the driving control ECU 24 serves as a transmission control unit and can select a portion of the outside-vehicle captured images captured by the multiple outside-vehicle cameras 63 installed on the host vehicle, including the outside-vehicle captured image requested by the remote control device 4.

[0099] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible within the scope of the gist of the invention.

[0100] In the above-described embodiment, the remote control device 4 is composed of one server device 5 and one remote control value generating device 6. Alternatively, the server device 5 or the remote control value generating device 6 of the remote control device 4 may be configured in multiple units, for example, by dividing it into predetermined units such as regions or the number of automobiles 2. The server device 5 or the remote control value generating device 6 may also be divided into multiple units based on functions or processing loads. The multiple server devices 5 or the multiple remote control value generating devices 6 may be incorporated into a base station 9 of a fifth-generation communication network 8 and provided in a distributed manner.

[0101] In the above-described embodiment, the multiple exterior cameras 63 installed on the automobile 2 capture images of the 360-degree surroundings of the automobile 2, divided into the front, right front, right rear, rear, left front, and left rear sides of the automobile 2, as shown in FIG. In addition, for example, the multiple exterior cameras 63 installed on the automobile 2 may capture images of the 360-degree surroundings of the automobile 2 in multiple sections, for example, the front, right, rear, and left sides of the automobile 2. [Explanation of symbols]

[0102] 1...remote control system, 2...automobile (vehicle), 3...control system, 4...remote control device, 5...server device, 6...remote control value generation device, 7...communication system, 8...communication network, 9...base station, 10...computer device, 11...server communication device, 12...server GNSS receiver, 13...server timer, 14...server memory, 15...server CPU, 16...server bus, 21...driving ECU, 22...steering ECU, 23...braking ECU, 24...cruising control ECU, 25...driving operation ECU, 26...detection ECU, 27...AP communication ECU, 28...V2V communication ECU, 29...remote control ECU, 30...vehicle network, 31...bus cable, 32...central gateway, 41...memory, 42...timer, 51...steering, 52...brake pedal, 53...accelerator pedal, 54...shift lever, 61...speed sensor, 62...acceleration sensor, 63...exterior camera, 64...LIDAR, 65...in-vehicle camera, 66...GNSS receiver, 71...front exterior camera, 72...right front exterior camera, 73...right rear exterior camera, 74...left front exterior camera, 75...left rear exterior camera, 76...rear exterior camera, 100...road, 101...passing lane, 102, 104...intersection, 103...parking position, 105...vehicle ahead, 110...GNSS satellite

Claims

1. A remote control system for vehicle travel in which a remote control device capable of communicating with a plurality of vehicles communicates with the vehicles, and the remote control device periodically transmits remote control values ​​for controlling the travel of the vehicles to the vehicles, a transmission control unit provided in the vehicle, capable of transmitting vehicle information including at least vehicle exterior images captured by a plurality of vehicle exterior cameras provided in the vehicle from the vehicle to the remote control device; a host vehicle driving control unit provided in the vehicle, the host vehicle periodically performing remote-controlled driving control using a remote control value periodically received by the vehicle from the remote control device; and The transmission control unit of the vehicle During remote control, the system determines whether or not the communication status between the vehicle and the remote control device or the processing status of the remote control device will affect the periodic driving control using the remote control values ​​by the vehicle driving control unit, by determining whether or not the communication status between the vehicle and the remote control device or the processing status of the remote control device will affect the periodic driving control using the remote control values ​​by the vehicle driving control unit, based at least on the period of driving control using the remote control values ​​executed by the vehicle driving control unit in the vehicle and the reception intervals of multiple remote control values ​​from the remote control device; If it is determined that the vehicle has an influence, a part of the images outside the vehicle taken by the plurality of outside cameras installed in the vehicle is selected according to the driving conditions of the vehicle and transmitted to the remote control device. A remote control system for vehicle operation.

2. The remote control device generates remote control values ​​including control values ​​for steering and accelerating / decelerating the vehicle, as well as control values ​​for controlling the lighting state of exterior lights; The transmission control unit of the vehicle estimating a future course of the vehicle under remote control based on the remote control value received from the remote control device; changing the vehicle exterior captured image to be selected from the vehicle exterior captured images taken by the plurality of vehicle exterior cameras provided on the vehicle according to the estimated course; 2. The remote control system for vehicle travel according to claim 1.

3. A remote control system for vehicle driving, in which a remote control device capable of communicating with a plurality of vehicles communicates with the vehicles, and the remote control device periodically transmits remote control values ​​for controlling the driving of the vehicles to the vehicles, a transmission control unit provided in the vehicle, capable of transmitting vehicle information including at least vehicle exterior images captured by a plurality of vehicle exterior cameras provided in the vehicle from the vehicle to the remote control device; a host vehicle driving control unit provided in the vehicle, the host vehicle periodically performing remote-controlled driving control using a remote control value periodically received by the vehicle from the remote control device; and the plurality of exterior cameras provided on the vehicle capture images of the surroundings of the vehicle, dividing the surroundings into at least the front, right, rear, and left sides of the vehicle; The transmission control unit of the vehicle During remote control, it is determined whether or not a communication status between the vehicle and the remote control device or a processing status of the remote control device affects periodic driving control using a remote control value by the host vehicle driving control unit; If it is determined that the vehicle will be affected when its driving is remotely controlled for automatic parking in a parking lot, While the vehicle is traveling until it stops in the passage in front of the parking position, at least an exterior image of the front side of the vehicle and an exterior image of the side of the parking position are selected as part of the exterior images of the vehicle captured by a plurality of exterior cameras installed in the vehicle and transmitted to the remote control device; During the travel of the vehicle from the stopped position to the parking position, at least an outside image of the rear side of the vehicle and an outside image of the side of the parking position are selected as part of the outside images of the vehicle captured by the plurality of outside cameras provided on the vehicle and transmitted to the remote control device, The outside-of-vehicle captured image to be selected is dynamically changed according to each traveling stage from the passage in front of the parking position to entering the parking position and stopping; Remote control system for vehicle operation.

4. A remote control system for vehicle driving, in which a remote control device capable of communicating with a plurality of vehicles communicates with the vehicles, and the remote control device periodically transmits remote control values ​​for controlling the driving of the vehicles to the vehicles, a transmission control unit provided in the vehicle, capable of transmitting vehicle information including at least vehicle exterior images captured by a plurality of vehicle exterior cameras provided in the vehicle from the vehicle to the remote control device; a host vehicle driving control unit provided in the vehicle, the host vehicle periodically performing remote-controlled driving control using a remote control value periodically received by the vehicle from the remote control device; and the plurality of exterior cameras provided on the vehicle capture images of the surroundings of the vehicle, dividing the surroundings into at least a front side, a right front side, a right rear side, a rear side, a left front side, and a left rear side of the vehicle; The transmission control unit of the vehicle During remote control, it is determined whether or not a communication status between the vehicle and the remote control device or a processing status of the remote control device affects periodic driving control using a remote control value by the host vehicle driving control unit; If it is determined that there is an impact when the vehicle is remotely controlled to turn right or left at an intersection, While the vehicle is traveling on the road before entering the road, at least a front outside image of the vehicle is selected as part of the outside images captured by a plurality of outside cameras provided on the vehicle and transmitted to the remote control device; During driving after starting to turn at an intersection, at least an outside image of the front side of the vehicle, an outside image of the right front side of the vehicle, an outside image of the left front side of the vehicle, and an outside image of the right rear side or the left rear side of the vehicle that is on the outside of the turn are selected as part of the outside images captured by the plurality of outside cameras provided on the vehicle and transmitted to the remote control device; The outside-of-vehicle captured image to be selected is dynamically changed according to each driving stage from the road before entering the intersection to proceeding to the road after the turn. A remote control system for vehicle operation.

Images