Remote control system and remote control program

Abstract

To provide a remote instruction system and a remote instruction program that make it possible to show communication quality information to a remote commander in an easily recognizable manner for the remote commander.SOLUTION: A surrounding environment image is shown to a remote commander on the basis of detection data of external vehicle sensors having been transmitted via a plurality of communication networks from a vehicle capable of executing remote support in accordance with remote commands from the remote commander. Integrated information in which multiple pieces of communication quality information are integrated is generated on the basis of the communication quality information of each of the plurality of communication networks. The display data of the surrounding environment image is acquired on the basis of the detection data received from the vehicle, and the surrounding environment image is shown in one or more display regions among a plurality of display regions for the remote commander. One target region is selected from among the plurality of display regions and an image representing the integrated information is shown in the selected target region.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to a remote instruction system and a remote instruction program.

Background Art

[0002] For example, Patent Document 1 describes a communication terminal device that receives camera images captured at a plurality of bases in a remote conference via communication, graphs communication quality data of a plurality of communications, and synthesizes and displays them for each camera image for each base.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a remote instruction system in which a remote commander can execute remote support for a vehicle, a plurality of display areas may be prepared for the remote commander. Display data corresponding to detection data of external sensors of the vehicle can be displayed in the plurality of display areas. Since the display data is received via a plurality of communication networks, communication quality information of the plurality of communication networks may be displayed to the remote commander. In this case, for example, in order to make it easier for the remote commander to concentrate on remote instructions, it is desirable to display the communication quality information to the remote commander in a manner that is easy for the remote commander to recognize.

Means for Solving the Problems

[0005] One aspect of the present invention is a remote instruction system that displays an image of the surrounding environment to a remote commander based on detection data from an external sensor of a vehicle transmitted via a plurality of communication networks from a vehicle capable of performing remote support in response to remote instructions from a remote commander, comprising: an integrated information generation unit that generates integrated information by integrating a plurality of communication quality information based on the communication quality information of each of the plurality of communication networks; and a display control unit that acquires display data of an image of the surrounding environment based on detection data received from the vehicle and displays the image of the surrounding environment in one or more display areas of a plurality of display areas for the remote commander, wherein the display control unit selects one target area from the plurality of display areas and displays an image representing the integrated information in the selected target area.

[0006] According to one aspect of the present invention, a remote instruction system generates integrated information by integrating multiple pieces of communication quality information. A display control unit selects one target area from among multiple display areas, and displays an image representing the integrated information in the selected target area. As a result, the remote commander can recognize the integrated information by looking at a single target area. Therefore, compared to, for example, displaying multiple pieces of communication quality information separately in multiple display areas without integrating them, it is possible to display the communication quality information to the remote commander in a manner that is easier for the remote commander to recognize.

[0007] In one embodiment, the display control unit may select a target area from among the display areas that display surrounding environment images corresponding to some of the detected data when some of the detected data is transmitted from the vehicle. In this case, the display area where the surrounding environment images corresponding to some of the detected data are displayed is likely to be viewed by the remote commander. Since the target area is selected from such a display area, it becomes possible to display communication quality information to the remote commander in a manner that is easier for the remote commander to recognize, compared to, for example, displaying an image representing integrated information in a display area where the surrounding environment images are not displayed.

[0008] In one embodiment, the integrated information generation unit generates a recommended speed, which is the vehicle speed corresponding to the communication quality information, as integrated information, and the display control unit may display an image representing the recommended speed in the target area. In this case, the remote commander can recognize the recommended speed corresponding to the communication quality information by recognizing the integrated information.

[0009] Another aspect of the present invention is a remote instruction program that causes a processor to display an image of the surrounding environment to a remote commander based on detection data from an external sensor of a vehicle transmitted via a plurality of communication networks from a vehicle capable of performing remote support in response to a remote instruction from a remote commander, the program comprising: an integrated information generation unit that generates integrated information by integrating a plurality of communication quality information based on the communication quality information of each of the plurality of communication networks; and a display control unit that acquires display data of an image of the surrounding environment based on detection data received from the vehicle and displays the image of the surrounding environment in one or more of a plurality of display areas for the remote commander, the display control unit selects one target area from the plurality of display areas and displays an image representing the integrated information in the selected target area.

[0010] According to another aspect of the present invention, a remote instruction program generates integrated information by combining multiple pieces of communication quality information. One target area is selected from among multiple display areas, and an image representing the integrated information is displayed in the selected target area. As a result, the remote commander can recognize the integrated information by looking at a single target area. Therefore, compared to, for example, displaying multiple pieces of communication quality information separately in each of the multiple display areas without integrating them, it becomes possible to display the communication quality information to the remote commander in a manner that is easier for the remote commander to recognize. [Effects of the Invention]

[0011] According to the present invention, it is possible to display communication quality information to a remote commander in a manner that is easily recognizable to the remote commander. [Brief explanation of the drawing]

[0012] [Figure 1] This diagram illustrates the overall structure of the remote control system according to the embodiment. [Figure 2] This is a block diagram showing an example of a vehicle configuration. [Figure 3] This is a block diagram showing an example of a relay server hardware configuration. [Figure 4] This is a diagram showing an example of the functional configuration of a relay server. [Figure 5] This is a block diagram showing an example of the hardware configuration of a remote command server. [Figure 6] This block diagram shows an example of the configuration of a remote control device. [Figure 7] This shows an example of multiple display areas, and an example of displaying an image representing integrated information for the target area. [Figure 8] (a) is the first example of an image representing integrated information. (b) is an example of an image when the communication quality deteriorates in Figure 8(a). [Figure 9] This is a second example of an image representing integrated information. [Figure 10] (a) is a third example of an image representing integrated information. (b) is a fourth example of an image representing integrated information. [Figure 11] This is a sequence diagram showing an example of processing in a remote control system. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same or equivalent elements will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0014] [Overview of the remote control system] FIG. 1 is a diagram illustrating an overall view of a remote instruction system according to an embodiment. As shown in FIG. 1, the remote instruction system 100 includes a remote instruction device 1 into which a remote commander R inputs a remote instruction, a relay server 50, and a vehicle 2. The remote instruction server 10 of the remote instruction device 1 is communicably connected to a plurality of vehicles 2 via networks N1, N2 and the relay server 50 as an example. The networks N1, N2 are wireless communication networks. Various information is sent from the vehicle 2 to the remote instruction device 1.

[0015] The remote instruction system 100 executes remote support for the vehicle 2 in response to a remote instruction from the remote commander R. The remote instruction is an instruction of the remote commander R regarding remote support for the vehicle 2. The remote commander R is an operator who performs remote support for the vehicle 2. The remote commander R is deployed, for example, in a remote cockpit provided in a facility or the like away from the vehicle 2. The number of remote commanders R may be one or two or more.

[0016] The vehicle 2 is a vehicle configured to be capable of executing remote support. The vehicle 2 may be configured to be capable of executing automatic driving control. The number of vehicles 2 communicable with the remote instruction system 100 is not particularly limited.

[0017] The remote support includes remote monitoring in which the remote commander R monitors the situation around the vehicle and the situation of the driver, remote support in which the remote commander R provides information and gives instructions to the driver of the vehicle, and remote driving in which the remote commander R gives instructions regarding the automatic driving of the vehicle. For example, the instructions regarding automatic driving include an instruction to proceed the vehicle 2 and an instruction to stop the vehicle 2. The instructions regarding automatic driving may include an instruction to change lanes of the vehicle 2. The instructions regarding automatic driving may include an instruction to avoid an offset with respect to an obstacle ahead, an instruction to overtake a preceding vehicle, an instruction to perform an emergency evacuation, and the like.

[0018] Network N1 may include, for example, networks N1a, N1b, and N1c (multiple communication networks) operated by multiple communication carriers. Network N1a is the wireless communication network of the first communication carrier. Network N1b is the wireless communication network of the second communication carrier. Network N1c is the wireless communication network of the third communication carrier. Networks N1a, N1b, and N1c may each have different communication qualities. The number of communication carriers is not limited to this example.

[0019] In the remote instruction system 100, for example, detection data of the surrounding environment detected by the external sensor 22 of the vehicle 2 is transmitted to the relay server 50 via networks N1a, N1b, and N1c. The relay server 50 obtains display data of the surrounding environment image from the received detection data. The surrounding environment image is an image of the surrounding environment of the vehicle 2 detected by the external sensor 22. The relay server 50 generates integrated information by integrating the communication quality information of each of the networks N1a, N1b, and N1c. The display data of the surrounding environment image and the integrated information are transmitted to the remote instruction server 10 via network N2. The remote instruction server 10 displays an image representing the surrounding environment image and the integrated information to the remote commander R. The remote commander R inputs a remote instruction to the commander interface 3 of the remote instruction device 1 while referring to the surrounding environment image. The remote instruction device 1 transmits the remote instruction to the vehicle 2 via networks N1 and N2. In the vehicle 2, remote support is performed in response to the remote instruction.

[0020] [Vehicle configuration] First, let's describe an example of the configuration of vehicle 2. Figure 2 is a block diagram showing an example of the vehicle's configuration. As shown in Figure 2, vehicle 2, as an example, has a remote driving ECU 30. The remote driving ECU 30 is an electronic control unit that includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The remote driving ECU 30 realizes various functions by, for example, loading a program recorded in ROM into RAM and executing the program loaded into RAM with the CPU. The remote driving ECU 30 may be composed of multiple electronic units. The remote driving ECU 30 is also referred to as, for example, an RDK (Remote Driving Kit).

[0021] The remote driving ECU 30 is connected to a GPS (Global Positioning System) receiver 21, an external sensor 22, an internal sensor 23, a map database 24, an actuator 25, a first communication unit 26, a second communication unit 27, and a third communication unit 28.

[0022] The GPS receiver 21 measures the position of the vehicle 2 (for example, the latitude and longitude of the vehicle 2) by receiving signals from three or more GPS satellites. The GPS receiver 21 transmits the measured position information of the vehicle 2 to the remote driving ECU 30.

[0023] The external sensor 22 is an on-board sensor that detects the surrounding environment around the vehicle 2. The external sensor 22 transmits the detected data to the remote driving ECU 30.

[0024] The external sensor 22 includes at least a camera. The camera is an imaging device that captures images of the surrounding environment of the vehicle 2. The camera is, for example, mounted behind the windshield of the vehicle 2 and captures images of the area in front of the vehicle. The camera may also capture images of the sides and rear of the vehicle 2.

[0025] The external sensor 22 may include a radar sensor. A radar sensor is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects around the vehicle 2. Radar sensors include, for example, radar (millimeter-wave radar) or lidar [LiDAR: Light Detection and Ranging].

[0026] The internal sensor 23 is an on-board sensor that detects the driving state of the vehicle 2. The internal sensor 23 may include a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. Known sensors can be used as the vehicle speed sensor, acceleration sensor, and yaw rate sensor.

[0027] The map database 24 is a database that records map information. The map database 24 is formed in a recording device such as an HDD [Hard Disk Drive] installed in the vehicle 2. The map information includes road location information, road shape information (e.g., curvature information), intersection and junction location information, etc.

[0028] The actuator 25 is a device used for controlling the movement of the vehicle 2 and operates in response to control signals from the remote driving ECU 30. The actuator 25 includes at least a drive actuator, a brake actuator, and a steering actuator. The drive actuator is provided, for example, on an engine or motor as a power source and controls the driving force of the vehicle 2. The brake actuator is provided, for example, on a hydraulic brake system and controls the braking force applied to the wheels of the vehicle 2. The steering actuator is, for example, an assist motor in an electric power steering system and controls the steering torque of the vehicle 2.

[0029] The first communication unit 26, the second communication unit 27, and the third communication unit 28 are communication devices that control wireless communication between the vehicle 2 and the outside. The first communication unit 26 transmits and receives various information with the relay server 50 via the network N1a of the first communication carrier. The second communication unit 27 transmits and receives various information with the relay server 50 via the network N1b of the second communication carrier. The third communication unit 28 transmits and receives various information with the relay server 50 via the network N1c of the third communication carrier.

[0030] Next, an example of the functional configuration of the remote driving ECU 30 will be described. The remote driving ECU 30 includes a vehicle position acquisition unit 31, a surrounding environment recognition unit 32, a driving state recognition unit 33, a display information acquisition unit 34, a display information transmission unit 35, a route generation unit 36, and a remote driving control unit 37.

[0031] The vehicle position acquisition unit 31 acquires the position information (location on the map) of the vehicle 2 based on the position information of the GPS receiver unit 21 and the map information of the map database 24, or by using SLAM (Simultaneous Localization and Mapping) technology.

[0032] The surrounding environment recognition unit 32 recognizes the surrounding environment of the vehicle 2 based on the detection data from the external sensor 22. The surrounding environment may include information used for autonomous driving, such as the relative position, relative speed, and direction of movement of surrounding objects relative to the vehicle 2.

[0033] The driving state recognition unit 33 recognizes the driving state of the vehicle 2 based on the detection results of the internal sensor 23. The driving state includes the vehicle speed of the vehicle 2, the acceleration of the vehicle 2, and the yaw rate of the vehicle 2 (the orientation of the vehicle 2).

[0034] The display information acquisition unit 34 acquires display information for display to the remote commander R based on detection data from the external sensor 22 or calculation results from the remote driving ECU 30. The display information may include information such as the position of the vehicle 2, the destination, the route of the vehicle 2 generated by the route generation unit 36 ​​(described later), and the driving status (e.g., vehicle speed). The display information may also include information outside the operating range [ODD: operation design domain] of the automated driving system. ODD is the range in which the automated driving system operates as designed.

[0035] The display information includes vehicle transmission image data transmitted to the remote commander R to display images of the surrounding environment. The vehicle transmission image data is acquired, for example, based on images (detection data) captured by the camera of vehicle 2. The vehicle transmission image data may include, for example, images of the scenery in front of vehicle 2, an overhead view of vehicle 2, etc. The vehicle transmission image data may also include detection data of images of the side or rear of vehicle 2 captured by the camera of vehicle 2. The display information acquisition unit 34 acquires the vehicle transmission image data based on images captured by the camera of the external sensor 22.

[0036] The vehicle transmitted image data may be a portion of the detection data from the external sensor 22. The display information acquisition unit 34 may set the range of information from the detection data of the external sensor 22 to be transmitted to the relay server 50. For example, the display information acquisition unit 34 may set the range of data from the detection data of the external sensor 22 to be transmitted to the relay server 50 (remote instruction server 10) based on the surrounding environment recognized by the surrounding environment recognition unit 32, the map information of the map database 24, and the path of the vehicle 2. The path here refers to the path before the path corresponding to the remote instruction is generated.

[0037] The display information acquisition unit 34 may determine which of the multiple sensors of the external sensor 22 should have their detection data included in the vehicle transmission image data. For example, based on the recognized surrounding environment, the display information acquisition unit 34 may not include in the vehicle transmission image data the detection data of sensors whose detection range does not include the area around the vehicle 2 that the remote commander R should check.

[0038] The display information acquisition unit 34 may extract portions of the detection data from the external sensor 22 to be transmitted to the relay server 50, based on the map information and the route. Extracting portions of the detection data from the external sensor 22 to be transmitted to the relay server 50 means, for example, cutting out unnecessary parts of the detection data and leaving only the necessary parts (the parts to be transmitted). As an example, the display information acquisition unit 34 can cut out a portion of the image captured by the camera of the external sensor 22 and use it as the portion to be transmitted.

[0039] The display information transmission unit 35 transmits the display information acquired by the display information acquisition unit 34 to the relay server 50. The display information transmission unit 35 transmits vehicle transmission image data in parallel to the relay server 50 via networks N1a, N1b, and N1c, respectively, using the first communication unit 26, the second communication unit 27, and the third communication unit 28.

[0040] The trajectory generation unit 36 ​​generates a trajectory used for the autonomous driving of the vehicle 2. The trajectory generation unit 36 ​​generates the autonomous driving trajectory based on a pre-set target route, map information, the location information of the vehicle 2, the surrounding environment of the vehicle 2, and the driving state of the vehicle 2. The trajectory corresponds to the autonomous driving plan. The method of generating the trajectory in the trajectory generation unit 36 ​​can employ well-known methods related to autonomous driving. The same applies to the content of the trajectory.

[0041] The remote driving control unit 37 performs remote assistance, remote driving, or automated driving of vehicle 2. The remote driving control unit 37 may provide information and instructions to the driver of vehicle 2 in response to remote instructions from the remote commander R. The remote driving control unit 37 may perform automated driving of vehicle 2 based on the surrounding environment of vehicle 2, the driving state of vehicle 2, and the path generated by the path generation unit 36. The remote driving control unit 37 may give instructions regarding automated driving of vehicle 2 in response to remote instructions from the remote commander R. The remote driving control unit 37 can perform remote driving and automated driving of vehicle 2 by transmitting control signals to the actuator 25.

[0042] [Configuration of relay servers] The relay server 50 relays data transmission and reception between the vehicle 2 and the remote control device 1. As an example of the hardware configuration of the relay server 50, a cloud server can be used. As an example of the functional configuration (software configuration) of the relay server 50, the remote control program described in this disclosure can be used.

[0043] Figure 3 is a block diagram showing an example of the hardware configuration of a relay server. As shown in Figure 3, the relay server 50 is configured as a general-purpose computer equipped with a processor 50a, a storage unit 50b, a communication unit 50c, and a user interface 50d. In this case, "user" refers to the user (administrator, etc.) of the relay server 50.

[0044] The processor 50a controls the relay server 50 by running various operating systems. The processor 50a is an arithmetic unit such as a CPU, which includes a control unit, arithmetic unit, registers, etc. The processor 50a manages the storage unit 50b, the communication unit 50c, and the user interface 50d. The storage unit 50b is composed of at least one of memory and storage. Memory is a recording medium such as ROM or RAM. Storage is a recording medium such as HDD.

[0045] The communication unit 50c is a communication device for performing communication via networks N1 and N2. The communication unit 50c can utilize network devices, network controllers, network cards, etc. The user interface 50d is the input / output unit of the relay server 50 for users such as administrators. The user interface 50d includes output devices such as displays and speakers, and input devices such as touch panels.

[0046] Next, the functional configuration of the relay server 50 will be described. Figure 4 is a block diagram showing an example of the functional configuration of the relay server. As shown in Figure 4, the relay server 50 has a relay information receiving unit 51, a relay information arbitration unit 52, an integrated information generation unit 53, and a relay information transmission unit 54.

[0047] The relay information receiving unit 51 receives various information and data transmitted from the vehicle 2 or the remote control device 1. The relay information receiving unit 51 also receives vehicle transmitted image data transmitted from the vehicle 2. For example, the relay information receiving unit 51 receives vehicle transmitted image data transmitted in parallel via networks N1a, N1b, and N1c by the display information transmitting unit 35.

[0048] The relay information receiving unit 51 acquires communication quality information for each of the networks N1a, N1b, and N1c. The communication quality information includes information regarding the communication quality of the communication using the first communication unit 26 and network N1a as the communication path, the communication using the second communication unit 27 and network N1b as the communication path, and the communication using the third communication unit 28 and network N1c as the communication path. The communication quality information includes, for example, the assumed values ​​of the frame rate, bit rate, and number of packets, the actual values ​​of the frame rate, bit rate, and number of packets, and the radio wave strength at the location of vehicle 2 for the received vehicle transmission image data. The relay information receiving unit 51 can acquire communication quality information based, for example, on measured values ​​of the amount of data transmitted and received associated with the communication, or on information provided by each communication carrier.

[0049] The relay information receiving unit 51 obtains, for example, the estimated number of packets for each of the networks N1a, N1b, and N1c. The estimated number of packets is the total number of packets that would be received if all data were relayed without loss or other errors during communication through each of the networks N1a, N1b, and N1c. The estimated number of packets corresponds, for example, to the number of packets of data in the vehicle transmitted image data. Therefore, the number of packets obtained by subtracting the number of constructed packets from the estimated number of packets corresponds to packet loss.

[0050] The relay information receiving unit 51, for example, acquires the signal strength of networks N1a, N1b, and N1c from each communication carrier.

[0051] The relay information receiving unit 51 may also receive remote command data from the remote commander R transmitted from the remote command device 1. For example, the relay information receiving unit 51 receives remote command data from the remote commander R transmitted via the network N2 by the remote command transmission unit 14, which will be described later.

[0052] The relay information arbitration unit 52 arbitrates vehicle transmission image data transmitted in parallel from vehicle 2. For example, the relay information arbitration unit 52 takes three vehicle transmission image data received in parallel via networks N1a, N1b, and N1c, respectively, and performs actions such as deleting duplicate packets and rearranging the packet order to generate one arbitrated image data representing the surrounding environment image. The arbitrated image data is an example of display data for the surrounding environment image. In other words, the relay information arbitration unit 52 functions as part of the display control unit that acquires display data for the surrounding environment image based on detection data received from vehicle 2.

[0053] The relay information arbitration unit 52 obtains, for example, the number of constructed packets for each of the networks N1a, N1b, and N1c. The number of constructed packets is the total number of packets used when generating the arbitration image data. The number of constructed packets is counted for each of the networks N1a, N1b, and N1c.

[0054] The integrated information generation unit 53 generates integrated information by integrating multiple communication quality information based on the communication quality information of each of the networks N1a, N1b, and N1c. The integrated information is an index that expresses the control state (multipath control state) of the remote instruction system 100 using multiple communication paths by integrating multiple communication quality information. For example, the integrated information generation unit 53 generates the average number of constructed packets, which is the average number of constructed packets of vehicle transmission image data received through each of the networks N1a, N1b, and N1c, as integrated information. The integrated information generation unit 53 may also generate the average value of the actual frame rate of the vehicle transmission image data received through each of the networks N1a, N1b, and N1c as integrated information. The integrated information generation unit 53 may also generate the average value of the actual bit rate of the vehicle transmission image data received through each of the networks N1a, N1b, and N1c as integrated information.

[0055] The integrated information generation unit 53 may obtain the number of times a packet was adopted when the relay information arbitration unit 52 generated arbitration image data. The number of adoptions is obtained for each network N1a, N1b, and N1c used for transmission from the vehicle 2. The integrated information generation unit 53 may calculate the contribution of the first, second, and third communication carriers based on the number of times each of the packets of the first, second, and third communication carriers were adopted and the average number of constructed packets.

[0056] The integrated information generation unit 53 may generate a recommended speed as integrated information, which is the vehicle speed of vehicle 2 according to the communication quality information. The recommended speed is a guideline value for the vehicle speed of vehicle 2 when providing remote support to vehicle 2. If the vehicle speed of vehicle 2 is less than or equal to the recommended speed, the image quality of the surrounding environment image affected by the communication quality will be of sufficient quality for the remote commander R to provide remote support under that communication quality. The remote commander R can refer to the recommended speed when giving remote instructions to vehicle 2. The recommended speed may be obtained, for example, according to the actual values ​​of the frame rate and bitrate. The recommended speed may be lower if the actual value of the frame rate is small. The recommended speed may be lower if the actual value of the bitrate is small.

[0057] The relay information transmission unit 54 transmits various information and data calculated by the relay server 50 to the vehicle 2 or the remote control device 1. For example, the relay information transmission unit 54 transmits arbitration image data to the remote control device 1 as display data for the surrounding environment image. As integrated information, the relay information transmission unit 54 transmits to the remote control device 1, for example, the number of constructed packets, the average number of constructed packets, the expected number of packets, the recommended speed, and the respective radio wave strengths of networks N1a, N1b, and N1c. The relay information transmission unit 54 transmits the remote commander R remote instruction data received from the remote control device 1 to the vehicle 2.

[0058] [Configuration of the remote control device] The configuration of the remote control device 1 will be described with reference to the drawings. The remote control device 1 includes a remote control server 10 and a commander interface 3.

[0059] First, the hardware configuration of the remote command server 10 will be described. Figure 5 is a block diagram showing an example of the hardware configuration of the remote command server. As shown in Figure 5, the remote command server 10 is configured as a general-purpose computer equipped with a processor 10a, a storage unit 10b, a communication unit 10c, and a user interface 10d. In this case, "user" refers to the user (administrator, etc.) of the remote command server 10. The processor 10a, storage unit 10b, communication unit 10c, and user interface 10d may have the same hardware configuration as, for example, a processor 50a, a storage unit 50b, a communication unit 50c, and a user interface 50d. The processor 10a controls the remote command server 10 by running various operating systems. The communication unit 10c is a communication device for communication via the network N2. Note that the remote command server 10 does not necessarily need to be installed in a facility with a remote cockpit, etc., and may be mounted on a mobile device such as a vehicle.

[0060] Figure 6 is a block diagram showing an example of the configuration of a remote control device. As shown in Figure 6, the commander interface 3 is the input / output section of the remote control device 1 to the remote commander R. The commander interface 3 has an output section 3a and an instruction input section 3b.

[0061] The output unit 3a is a device that outputs information to the remote commander R for use in remotely instructing the vehicle 2. The output unit 3a includes a plurality of displays (display areas) D1, D2, D3 that output images (see Figure 7). The plurality of displays D1, D2, D3 are multiple display areas for the remote commander R. The output unit 3a may also include a speaker that outputs sound. The speaker may be, for example, a headset speaker worn on the head of the remote commander R. The output unit 3a may also provide information to the remote commander R by vibration, for example, using a vibration actuator provided on the seat.

[0062] Displays D1, D2, and D3 display images of the surrounding environment acquired from vehicle transmission image data. For example, displays D1, D2, and D3 display images of the scenery in front of vehicle 2, captured by the camera of vehicle 2. Displays D1, D2, and D3 may also display images of the side or rear of vehicle 2, captured by the camera of vehicle 2. As will be described later, displays D1, D2, and D3 can display images representing integrated information superimposed on the surrounding environment images.

[0063] The instruction input unit 3b is a device that receives remote instructions from the remote commander R. The instruction input unit 3b may have an operating device of a known configuration, such as an operating lever, operating pedal, or operating button.

[0064] Next, the functional configuration of the remote instruction server 10 will be described. As shown in Figure 6, the remote instruction server 10 includes an operation information receiving unit 11, a display image acquisition unit 12, a display control unit 13, and a remote instruction transmission unit 14.

[0065] The driving information receiving unit 11 receives driving information for the remote commander R to provide remote support to the vehicle 2. The driving information includes display data of surrounding environment images and integrated information. The driving information may also include information related to the autonomous driving of the vehicle 2, such as the path the vehicle 2 will take when it is driving autonomously.

[0066] The driver information receiving unit 11 receives, for example, arbitration image data transmitted from the relay information transmitting unit 54 as display data for the surrounding environment image. The driver information receiving unit 11 also receives integrated information transmitted from the relay information transmitting unit 54. As integrated information, the driver information receiving unit 11 receives, for example, the number of constructed packets, the average number of constructed packets, the expected number of packets, the recommended speed, and the respective radio wave strengths of networks N1a, N1b, and N1c.

[0067] The display image acquisition unit 12 acquires an image to be displayed to the remote commander R based on the received operating information. The display image acquisition unit 12 acquires, for example, an image of the surrounding environment of the received arbitration image data as an image to be displayed to the remote commander R. The display image acquisition unit 12 also acquires, for example, an image representing the integrated information based on the received integrated information. The image representing the integrated information will be described in detail later.

[0068] The display control unit 13 controls the display of images on multiple displays D1, D2, and D3 for the remote commander R. The display control unit 13 causes one or more of the multiple displays D1, D2, and D3 for the remote commander R to display an image of the surrounding environment.

[0069] Figure 7 shows an example of multiple display areas and an example of displaying an image representing integrated information for the target area. As shown in Figure 7, the display control unit 13, for example, when all of the vehicle transmission image data (detection data) from vehicle 2 is transmitted from vehicle 2, displays the surrounding environment image corresponding to all of the vehicle transmission image data on all of the multiple displays D1, D2, and D3 for the remote commander R.

[0070] The display control unit 13 selects one target area from among the multiple displays D1, D2, and D3 and displays an image representing the integrated information in the selected target area. The target area is the display area on which the image representing the integrated information will be displayed. The display control unit 13 selects as the target area a display area from among the multiple displays D1, D2, and D3 that can display the integrated information (communication quality information) in a manner that is easily recognizable by the remote commander R. In the example in Figure 7, the target area can be, for example, display D2, which is located in front of the remote commander R, among the multiple displays D1, D2, and D3. The display control unit 13 may select either display D1 or D3 as the target area depending on the content of the remote support, for example, when the remote commander R's line of sight is directed towards either display D1 or D3, such as when vehicle 2 turns right or left at an intersection or changes lanes.

[0071] The display control unit 13 displays an image representing the integrated information in the selected target area. In the example in Figure 7, the display control unit 13 displays an image representing the integrated information in a portion area D2a of the display D2, which is the selected target area.

[0072] A specific example of an image representing integrated information will be described. Figure 8(a) is the first example of an image representing integrated information. In Figure 8(a), a bar graph 60 is drawn in a partial region D2a, with the longitudinal direction being along the longer side of the display D2. As shown in Figure 8(a), the bar graph 60 has an outer frame 61, a value display 62, and a threshold 63. The outer frame 61 is a rectangle along the longitudinal direction. The value display 62 is a rectangle along the longitudinal direction and is drawn within the outer frame 61, left-aligned to the plane of the figure. The value display 62 has, for example, a different color or pattern from the area within the outer frame 61. Within the outer frame 61, a line segment representing the threshold 63 is drawn so as to extend in the short direction intersecting the longitudinal direction.

[0073] The outer frame 61, value display 62, and threshold 63 are images representing the number of packets, for example, with the left edge of the diagram within the outer frame 61 as the origin. The position within the outer frame 61, the position of the right edge of the value display 62, and the position of the threshold 63 represent larger packet counts as they are located further to the right as they are on the page. The longitudinal dimension of the outer frame 61 represents the expected number of packets. The longitudinal dimension of the value display 62 represents the average number of constructed packets.

[0074] The threshold 63 is a threshold for evaluating the control state (multipath control state) of the remote instruction system 100 using multiple communication paths as a whole. Here, the threshold 63 is set for, for example, the average number of constructed packets represented by the value display 62. The threshold 63 may be a predetermined value set in advance or a variable value.

[0075] As shown in Figure 8(a), if the average number of constructed packets represented by the value display 62 is greater than or equal to the threshold 63, the remote commander R, who has viewed the integrated information, can recognize that the control state of the remote instruction system 100 using multiple communication paths (multipath control state) is in a normal control state. Figure 8(b) is an example of an image when the communication quality deteriorates in Figure 8(a). As shown in Figure 8(b), if the average number of constructed packets represented by the value display 62 is less than the threshold 63, the remote commander R, who has viewed the integrated information, can recognize that the control state of the remote instruction system 100 using multiple communication paths (multipath control state) is not in a normal control state.

[0076] Figure 9 is a second example of an image representing integrated information. In the subregion D2a of Figure 9, in addition to the bar graph 60, multiple bar graphs 64a, 64b, and 64c are drawn, extending along the longer side of the display D2 as the longitudinal direction. The multiple bar graphs 64a, 64b, and 64c represent the communication quality information of networks N1a, N1b, and N1c respectively without averaging.

[0077] The longitudinal dimension of the outer frame 65a of bar graph 64a represents the assumed number of packets in the communication path between the first communication unit 26 and network N1a. The longitudinal dimension of the value display 66a of bar graph 64a represents the number of received packets in the communication path between the first communication unit 26 and network N1a. The longitudinal dimension of the outer frame 65b of bar graph 64b represents the assumed number of packets in the communication path between the second communication unit 27 and network N1b. The longitudinal dimension of the value display 66b of bar graph 64b represents the number of received packets in the communication path between the second communication unit 27 and network N1b. The longitudinal dimension of the outer frame 65c of bar graph 64c represents the assumed number of packets in the communication path between the third communication unit 28 and network N1c. The longitudinal dimension of the value display 66c of bar graph 64c represents the number of received packets in the communication path between the third communication unit 28 and network N1c. In bar graphs 64a, 64b, and 64c, for example, the assumed number of packets corresponds to the size of the vehicle transmission image data transmitted from vehicle 2, and the number of received packets corresponds to the size of the vehicle transmission image data received by the relay server 50. Thresholds 67a, 67b, and 67c are thresholds for individually evaluating the communication quality information of networks N1a, N1b, and N1c. Here, thresholds 67a, 67b, and 67c are set for the number of received packets represented by the value displays 66a, 66b, and 66c, for example.

[0078] In other words, the display control unit 13 may display an unintegrated image, which represents the communication quality information of each of the multiple communication networks without integrating it, on the display D2 (target area) together with an image representing the integrated information. In the example in Figure 9, the multiple bar graphs 64a, 64b, and 64c are unintegrated images that represent the communication quality information of each of the multiple networks N1a, N1b, and N1c without integrating it. By displaying the bar graph 60 representing the integrated information and the multiple bar graphs 64a, 64b, and 64c side by side, the remote commander R can easily recognize the control status of the remote instruction system 100, and the developers or maintenance personnel of the remote instruction system 100 can also check the communication status of each of the networks N1a, N1b, and N1c, as well as the communication terminal status of the first communication unit 26, the second communication unit 27, and the third communication unit 28.

[0079] Figure 10(a) is a third example of an image representing integrated information. In the subregion D2a of Figure 10(a), in addition to the bar graph 60, a carrier display 68 is drawn that extends along the longer side of the display D2 as its longitudinal direction. The carrier display 68 is drawn, for example, alongside the value display 62 of the bar graph 60. The carrier display 68 is a display that shows the breakdown (contribution) of each communication carrier in the average number of constructed packets represented by the value display 62 of the bar graph 60, which is arranged at the top of the page. The carrier display 68 can be, for example, a graph in which the contributions of the first communication carrier 68a, the second communication carrier 68b, and the third communication carrier 68c in the average number of constructed packets are stacked together based on the calculation results of the integrated information generation unit 53. In other words, the display control unit 13 may display a carrier display 68, which is an image representing the contribution of multiple communication carriers to the integrated information for multiple communication carriers that provide multiple communication networks, on the display D2 (target area) together with an image representing the integrated information.

[0080] A radio wave condition indicator 69 may be drawn in the sub-region D2a of Figure 10(a). The radio wave condition indicator 69 is an image representing the radio wave strength of each communication carrier at the location of vehicle 2. The radio wave condition indicator 69 is drawn, for example, below the carrier indicator 68 on the paper. The radio wave condition indicator 69 may represent radio wave strength using, for example, known pictograms. By drawing the radio wave condition indicator 69, it is possible to check, for example, whether vehicle 2 was located at a point with insufficient radio wave strength (a point outside the coverage area) before the display of the surrounding environment image became stuck or disappeared due to communication interruption on any of the multiple displays D1, D2, D3. Note that the radio wave condition indicator 69 may be omitted in Figure 10(a).

[0081] Figure 10(b) is a fourth example of an image representing integrated information. In the sub-region D2a of Figure 10(b), in addition to the bar graph 60, a recommended speed display 70 is drawn, extending along the longer side of the display D2 as its longitudinal direction. The recommended speed display 70 is drawn, for example, alongside the bar graph 60. The recommended speed display 70 includes a recommended speed bar graph 71 drawn with the same outline as the bar graph 60 lined up at the bottom of the page, and a recommended speed value 72 written alongside the recommended speed bar graph 71. In other words, the display control unit 13 causes the recommended speed display 70, which is an image representing the recommended speed, to be displayed on the display D2 (target region).

[0082] The longitudinal dimension of the outer frame 73 of the recommended speed bar graph 71 represents the maximum value of the vehicle speed scale of the value display 74. The maximum value of the vehicle speed scale may be a predetermined fixed value, or it may be variable depending on the vehicle speed recognized by the vehicle 2 or the recommended speed. The longitudinal dimension of the value display 74 of the recommended speed bar graph 71 represents the vehicle speed recognized by the vehicle 2. The threshold 75 is the vehicle speed threshold of the vehicle 2 that represents the recommended speed according to the communication quality information. Here, the threshold 75 is set, for example, based on the calculation result of the integrated information generation unit 53.

[0083] The recommended speed value 72 is an image of a numerical value representing the vehicle speed recognized by vehicle 2 and the recommended speed. In the example in Figure 10(b), a situation is shown where the vehicle speed recognized by vehicle 2 is 60 km / h and the recommended speed is 48 km / h. In this case, the recommended speed value 72 may be displayed as, for example, "60 kph / 48 kph".

[0084] In addition to displaying the information shown in Figures 7 to 10 above, the display control unit 13 may also provide various types of information to the remote commander R regarding remote support.

[0085] The remote instruction transmission unit 14 transmits various information and data related to remote instructions from the remote commander R to the relay server 50. For example, when the remote commander R inputs a remote instruction to the instruction input unit 3b of the commander interface 3, the remote instruction transmission unit 14 transmits the input remote instruction to the vehicle 2.

[0086] [Processing by the remote control system 100] Next, an example of the processing (remote instruction program) of the remote instruction system 100 will be explained with reference to the flowchart in Figure 11. Figure 11 is a sequence diagram showing an example of the processing of the remote instruction system. The processing shown in Figure 11 is executed, for example, while the vehicle 2 is in operation.

[0087] As shown in Figure 11, in the remote control system 100, the remote driving ECU 30 of the vehicle 2 recognizes the surrounding environment of the vehicle 2 in S10 using the surrounding environment recognition unit 32. The surrounding environment recognition unit 32 recognizes the surrounding environment of the vehicle 2, for example, based on detection data from the external sensor 22.

[0088] In S12, the remote driving ECU 30 of vehicle 2 acquires vehicle transmission image data using the display information acquisition unit 34. The display information acquisition unit 34 acquires vehicle transmission image data based on the image captured by the camera of the external sensor 22.

[0089] In S14, the remote driving ECU 30 of vehicle 2 transmits vehicle transmission image data via the display information transmission unit 35. For example, the display information transmission unit 35 transmits vehicle transmission image data to the relay server 50 in parallel via networks N1a, N1b, and N1c, respectively, using the first communication unit 26, the second communication unit 27, and the third communication unit 28.

[0090] In S20, the relay server 50 receives vehicle transmission image data and acquires communication quality information via the relay information receiving unit 51. For example, the relay information receiving unit 51 receives vehicle transmission image data transmitted from vehicle 2 in parallel via networks N1a, N1b, and N1c. The relay information receiving unit 51 acquires communication quality information based, for example, on measured values ​​of the amount of data transmitted and received during communication, or on information provided by each communication carrier.

[0091] In S22, the relay server 50 generates arbitrated image data using the relay information arbitration unit 52. The relay information arbitration unit 52, for example, takes three vehicle transmission image data received in parallel via networks N1a, N1b, and N1c, deletes duplicate packets, rearranges the packet order, and generates one arbitrated image data representing the surrounding environment. The relay information arbitration unit 52, for example, obtains the number of constructed packets for each of networks N1a, N1b, and N1c.

[0092] In S24, the relay server 50 generates integrated information using the integrated information generation unit 53. The integrated information generation unit 53 generates integrated information by integrating multiple pieces of communication quality information based on the communication quality information of networks N1a, N1b, and N1c, for example. The integrated information generation unit 53 generates the average number of constructed packets, which is the average value of the number of constructed packets for networks N1a, N1b, and N1c, as integrated information. The integrated information generation unit 53 may also generate the recommended speed, which is the vehicle speed of vehicle 2 according to the communication quality information, as integrated information.

[0093] In S26, the relay server 50 transmits arbitration image data and integrated information via the relay information transmission unit 54. The relay information transmission unit 54 transmits, for example, the arbitration image data as display data of the surrounding environment image to the remote control device 1. The relay information transmission unit 54 also transmits, for example, the number of constructed packets, the average number of constructed packets, the expected number of packets, the recommended speed, and the radio wave strength of networks N1a, N1b, and N1c to the remote control device 1 as integrated information.

[0094] In S30, the remote instruction server 10 of the remote instruction device 1 receives arbitration image data and integrated information via the operation information receiving unit 11. The operation information receiving unit 11 receives operation information, for example, which includes arbitration image data and integrated information as display data for surrounding environment images.

[0095] In S32, the remote instruction server 10 of the remote instruction device 1 uses the display image acquisition unit 12 to acquire images representing the surrounding environment and integrated information. The display image acquisition unit 12, for example, acquires images representing the surrounding environment of the received arbitration image data. The display image acquisition unit 12, for example, acquires images representing integrated information based on the received integrated information.

[0096] In S34, the remote instruction server 10 of the remote instruction device 1 displays the surrounding environment image in one or more display areas using the display control unit 13. The display control unit 13, for example, causes the surrounding environment image to be displayed on all of the multiple displays D1, D2, and D3 for the remote commander R.

[0097] In S36, the remote instruction server 10 of the remote instruction device 1 causes the display control unit 13 to display an image representing the integrated information in the selected target area. The display control unit 13 selects one target area (for example, display D2) from among multiple displays D1, D2, and D3. The display control unit 13 displays an image representing the integrated information in a sub-area D2a of the selected display D2. After that, the remote instruction system 100 terminates the process shown in Figure 11.

[0098] The remote instruction program causes the processor 50a of the relay server 50 and the processor 10a of the remote instruction server 10 to function (operate) as the integrated information generation unit 53 and the display control unit 13 described above. The remote instruction program is provided, for example, by a non-temporary recording medium such as ROM or semiconductor memory. Alternatively, the remote instruction program may be provided to the relay server 50, which is a cloud server, via communication such as a network.

[0099] As explained above, the remote instruction system 100 and remote instruction program generate integrated information by combining multiple pieces of communication quality information. One display D2 (target area) is selected from among the multiple displays D1, D2, and D3, and an image of a bar graph 60 representing the integrated information is displayed on the selected display D2. As a result, the remote commander R can recognize the integrated information by looking at a single display D2. Therefore, compared to, for example, displaying multiple pieces of communication quality information separately on each of the multiple displays D1, D2, and D3 without integrating them, it is possible to display the communication quality information to the remote commander R in a manner that is easier for the remote commander R to recognize.

[0100] A recommended speed, which is the vehicle speed of vehicle 2 based on the communication quality information, is generated as integrated information, and an image of the recommended speed display 70 representing the recommended speed is displayed on display D2. As a result, the remote commander R can recognize the recommended speed according to the communication quality information by recognizing the integrated information.

[0101] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above.

[0102] In the above embodiment, the case where all of the vehicle transmission image data (detection data) from vehicle 2 is transmitted from vehicle 2 is illustrated, and therefore, surrounding environment images corresponding to all of the vehicle transmission image data are displayed on all of the multiple displays D1, D2, and D3 for the remote commander R, but the embodiment is not limited to this example. The display control unit 13 may select a target area from among the display areas for displaying surrounding environment images corresponding to some of the detection data when only some of the detection data is transmitted from the vehicle. "When some of the detection data is transmitted from the vehicle" includes the case where the display information acquisition unit 34 determines which of the multiple sensors of the external sensor 22 will include the detection data to be included in the vehicle transmission image data, or the display information acquisition unit 34 extracts the portion of the detection data from the external sensor 22 to be transmitted to the relay server 50. Furthermore, "When some of the detection data is transmitted from the vehicle" also includes the case where, for example, some of the vehicle transmission image data accidentally fails to reach the relay server 50 due to the radio wave strength and communication quality of networks N1a, N1b, and N1c.

[0103] For example, if the detection data corresponding to display D2 among displays D1, D2, and D3 in Figure 7 does not become vehicle transmission image data, the surrounding environment image may be displayed only on displays D1 and D3. The display control unit 13 may select a target area from among displays D1 and D3. In such an example, displays D1 and D3, which are display areas where the surrounding environment image corresponding to some of the detection data is displayed, are likely to be visually inspected by the remote commander R. Since the target area is selected from such display areas, it becomes possible to display the communication quality information to the remote commander R in a manner that is easier for the remote commander R to recognize, compared to, for example, displaying an image representing integrated information on display D2, which does not display the surrounding environment image.

[0104] Even if all of the vehicle transmission image data (detection data) from vehicle 2 is transmitted from vehicle 2, it is not necessary to display the surrounding environment image on all of the multiple displays D1, D2, and D3. For example, the surrounding environment image may be displayed only on display D2. In short, the display control unit 13 only needs to display the surrounding environment image in one or more of the multiple display areas.

[0105] In the above embodiment, the recommended speed was generated as integrated information, and an image representing the recommended speed was displayed in the target area. However, the generation and display of the recommended speed may be omitted.

[0106] In the above embodiment, vehicle 2, which has an autonomous driving function, was given as an example of a vehicle capable of performing remote support, but the autonomous driving function is not essential. The vehicle may have a driver assistance function instead of an autonomous driving function, or it may have only a manual driving function. In short, the vehicle just needs to be configured to perform remote support in response to remote instructions from a remote commander.

[0107] In the above embodiment, a relay server 50 was interposed between the vehicle 2 and the remote instruction device 1, but the relay server 50 is not essential. The functions of the relay server 50 may be provided by the vehicle 2 or the remote instruction server 10. In this case, the functions (operations) of the relay information arbitration unit 52 and the integrated information generation unit 53 described above may be realized by executing the remote instruction program on the processor 10a of the remote instruction server 10 instead of the processor 50a of the relay server 50.

[0108] The functions of the remote driving ECU 30 may be realized, for example, by controlling the actuators 25 of the vehicle 2 in response to vehicle control requests from the ADK [Autonomous Driving Kit] connected to the vehicle 2 via a communication interface. [Explanation of Symbols]

[0109] 2...Vehicle, 10a, 50a...Processor, 13...Display control unit, 22...External sensor, 53...Integrated information generation unit, 100...Remote instruction system, D1, D2, D3...Display (display area), D2...Display (target area), R...Remote commander.

Claims

1. A remote instruction system that displays an image of the surrounding environment to a remote commander based on detection data from the vehicle's external sensors transmitted via multiple communication networks from a vehicle capable of performing remote support in response to remote instructions from a remote commander, An integrated information generation unit generates integrated information by integrating the communication quality information of multiple communication networks based on the communication quality information of each of the multiple communication networks, The system includes a display control unit that acquires display data of the surrounding environment image based on the detection data received from the vehicle and displays the surrounding environment image in one or more of the display areas for the remote commander, The display control unit is a remote control system that selects one target area from among a plurality of display areas and displays an image representing the integrated information in the selected target area.

2. The remote instruction system according to Claim 1, wherein the integrated information generation unit generates the following as integrated information: the average number of constructed packets which is the average value of the number of constructed packets of the detection data received through each of the plurality of communication networks, the average value of the actual frame rate of the detection data, or the average value of the actual bit rate of the detection data.

3. The remote command system according to claim 1 or 2, wherein the detection data of the external sensor transmitted from the vehicle does not include detection data of the external sensor that does not include in its detection range the area around the vehicle that the remote commander should verify.

4. The remote instruction system according to claim 1 or 2, wherein the display control unit displays an unintegrated image representing the communication quality information of each of the plurality of communication networks without integrating it, together with the image representing the integrated information, in the target area.

5. The remote instruction system according to claim 1 or 2, wherein the display control unit causes a carrier display, which is an image representing the contribution of a plurality of communication carriers to the integrated information, to be displayed in the target area together with the image representing the integrated information, for a plurality of communication carriers that provide a plurality of communication networks.

6. The remote instruction system according to claim 1 or 2, wherein the display control unit selects one of the multiple display areas to which the remote commander's line of sight is directed.

7. A remote instruction program that causes a processor to display an image of the surrounding environment to a remote commander based on detection data from the vehicle's external sensors transmitted via multiple communication networks from the vehicle, which is capable of performing remote support in response to remote instructions from a remote commander, An integrated information generation unit generates integrated information by integrating the communication quality information of multiple communication networks based on the communication quality information of each of the multiple communication networks, and A display control unit that acquires display data of the surrounding environment image based on the detection data received from the vehicle, and displays the surrounding environment image in one or more of the display areas among a plurality of display areas for the remote commander. The processor is operated as follows: The display control unit is a remote instruction program that selects one target area from among a plurality of display areas and displays an image representing the integrated information in the selected target area.

Images