Remote support system

Abstract

This remote support system helps reduce the burden on users who use remote support devices for remote vehicle support. [Solution] The remote support system comprises a vehicle and a remote support device. The vehicle includes multiple cameras. The remote support device is a device for communicating with the vehicle and providing remote support to the vehicle. At least one of the vehicle and the remote support device includes an image synthesis device that synthesizes multiple camera images captured by the multiple cameras according to predetermined image synthesis rules. The remote support device includes a display that shows the synthesized image synthesized by the image synthesis device and an adjustment device that adjusts the image synthesis rules according to the user operation of the remote support device.

Description

Technical Field

[0001] The present disclosure relates to a technology for remotely supporting a vehicle.

Background Art

[0002] Patent Document 1 discloses a vision device for an autonomous vehicle. The vision device is provided in a remote control unit that remotely controls the autonomous vehicle, performs signal processing for reducing an image signal from a narrow-range imaging device and inserting the reduced image signal at a predetermined position in an image signal from a wide-range imaging device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The vision device described in Patent Document 1 does not have a configuration for an operator who remotely controls an autonomous vehicle to adjust an image displayed on a display unit. Therefore, the technology described in Patent Document 1 has room for improvement in reducing the burden on the operator related to remote control.

[0005] The present disclosure has been made in view of the above problems, and an object thereof is to provide a remote support system that contributes to reducing the burden on a user who uses a remote support device for remote support of a vehicle (e.g., remote driving, remote assistance, remote monitoring).

Means for Solving the Problems

[0006] The remote support system relating to this disclosure comprises a vehicle and a remote support device. The vehicle includes a plurality of cameras. The remote support device is a device for communicating with the vehicle and providing remote support to the vehicle. At least one of the vehicle and the remote support device includes a video synthesis device that synthesizes multiple camera images captured by the plurality of cameras according to predetermined video synthesis rules. The remote support device includes a display that shows the synthesized image synthesized by the video synthesis device and an adjustment device that adjusts the video synthesis rules according to the operation of the user of the remote support device. [Effects of the Invention]

[0007] According to this disclosure, a function can be provided to users who provide remote support for vehicles to adjust the video synthesis rules for the synthesized image displayed on the remote support device's screen. This contributes to reducing the burden on users who use the remote support device. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing an example configuration of the remote support system according to Embodiment 1. [Figure 2] Figure 1 is a conceptual diagram showing an example of image synthesis by the image synthesis unit. [Figure 3] This is a schematic diagram showing an example configuration of a remote support system according to a first modified example of Embodiment 1. [Figure 4] This is a schematic diagram showing an example configuration of a remote support system according to a second modified example of Embodiment 1. [Figure 5] This is a schematic diagram showing an example configuration of a remote support system according to a third modified example of Embodiment 1. [Figure 6] This is a schematic diagram showing an example configuration of a remote support system according to Embodiment 2. [Figure 7] This is a schematic diagram showing an example configuration of a remote support system according to a first modified example of Embodiment 2. [Figure 8] This is a schematic diagram showing an example configuration of a remote support system according to a second modified example of Embodiment 2. [Figure 9] This is a schematic diagram showing an example configuration of a remote support system according to Embodiment 3. [Figure 10] Figure 9 is a conceptual diagram showing an example of image synthesis by the first and second image synthesis units. [Modes for carrying out the invention]

[0009] Embodiments of this disclosure will be described with reference to the attached drawings. Elements common to each drawing are denoted by the same reference numerals, and redundant explanations are omitted or simplified.

[0010] 1. Embodiment 1 1-1. Configuration of the Remote Support System Figure 1 is a schematic diagram showing an example configuration of the remote support system 1 according to Embodiment 1. The remote support system 1 is a system for remote support of a vehicle. Remote support is a concept that includes remote driving, remote assistance, and remote monitoring. As shown in Figure 1, the remote support system 1 includes a vehicle 101 to be remotely supported and a remote cockpit 201. The vehicle 101 and the remote cockpit 201 can communicate with each other via a wireless communication network.

[0011] Vehicle 101 is configured to be remotely supported (e.g., remotely driven) by a user (remote supporter) 1 using a remote cockpit 201. For example, vehicle 101 is an autonomous vehicle. Vehicle 101 includes, for example, a communication device 11, a running device 12, sensors 13, and a control device 14. The communication device 11 communicates wirelessly with the outside of vehicle 101. For example, the communication device 11 communicates wirelessly with the remote cockpit 201.

[0012] The running gear 12 includes a steering gear, a drive gear, and a braking gear. The steering gear steers the wheels of the vehicle 101. The drive gear is a power source that generates the driving force for the vehicle 101 and includes, for example, at least one of an electric motor and an internal combustion engine. The braking gear generates braking force.

[0013] The sensors 13 include, for example, a recognition sensor, a vehicle state sensor, and a position sensor. The recognition sensor recognizes (detects) the surrounding conditions of the vehicle 101. The recognition sensor includes, for example, multiple cameras 15 (151 to 15N: N is an integer of 2 or more) that capture multiple surrounding conditions of the vehicle 101. The vehicle state sensor detects the conditions of the vehicle 101 (e.g., yaw rate, wheel speed, vehicle speed, acceleration, steering angle). The position sensor detects the position and orientation of the vehicle 101.

[0014] The control device 14 controls the running of the vehicle 101. Further, the control device 14 provides the video V captured by the camera 15 to the user U in cooperation with the remote cockpit 201. The control device 14 includes one or more processors 16 (hereinafter simply referred to as the processor 16) and one or more storage devices 17 (hereinafter simply referred to as the storage device 17). The processor 16 executes various processes including processes related to the control of the vehicle 101. Examples of the processor 16 include a general-purpose processor, a special-purpose processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like. The storage device 17 stores various information necessary for various processes. Examples of the storage device 17 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The processor 16 executes a computer program. The computer program is stored in the storage device 17. The computer program may be recorded on a computer-readable recording medium. The functions of the control device 14 are realized by the cooperation of the processor 16 that executes the computer program and the storage device 17.

[0015] The remote cockpit 201 corresponds to an example (cockpit-type terminal) of a "remote support terminal" included in a "remote support device" for the user U to perform remote support for the vehicle 101. The remote cockpit 201 is operated by the user U for remote support of the vehicle 101. The remote cockpit 201 includes a communication device 21, one or more displays 22 (hereinafter simply referred to as the display 22), a video adjustment HMI (Human Machine Interface) 23, and a control device 24.

[0016] The communication device 21 communicates with the vehicle 101. The display 22 displays the video V (V1 to VN: N is an integer of 2 or more) captured by the plurality of cameras 15 mounted on the vehicle 101 (for example, refer to FIG. 2). Although details will be described later, the video adjustment HMI 23 receives an input from the user U. Note that the remote cockpit 201 may include input devices (e.g., steering wheel, accelerator pedal, brake pedal) operated by the user U for remote driving of the vehicle 101.

[0017] The control device 24 controls the remote cockpit 201. Also, the control device 24 provides the video V to the user U via the display 22 in cooperation with the control device 14 of the vehicle 101. The control device 24 includes one or more processors 25 (hereinafter simply referred to as the processor 25) and one or more storage devices 26 (hereinafter simply referred to as the storage device 26). The configuration example of the processor 25 is the same as that of the above-described processor 16. Also, the configuration example of the storage device 26 is the same as that of the above-described storage device 17. The processor 25 executes a computer program. The computer program is stored in the storage device 26. The computer program may be recorded on a computer-readable recording medium. The functions of the control device 24 are realized by the cooperation of the processor 25 that executes the computer program and the storage device 26.

[0018] 1-2. Display of Video V on the Display As described above, in the remote support system 1, the videos V of the plurality of cameras 15 are displayed on the display 22 of the remote cockpit 201. The remote support system 1 includes a "video synthesizing device" that synthesizes a plurality of videos V1 to VN (plural camera videos) captured by the plurality of cameras 15 according to a predetermined video synthesis rule R. In Embodiment 1, the control device 14 of the vehicle 101 includes the video synthesizing device. The display 22 displays the synthesized video Vc synthesized by the video synthesizing device. And the remote cockpit 201 (remote support device) includes an "adjusting device" that adjusts the video synthesis rule R according to the operation of the user U. In FIG. 1, the video adjustment HMI 23 corresponds to the adjusting device.

[0019] Figure 1 shows the functional blocks related to providing video V to user U. The control device 24 of the remote cockpit 201 comprises a control content transmission unit 241 and a video reception unit 242. Meanwhile, the control device 14 of the vehicle 101 comprises a control content reception unit 141, a video synthesis unit 142, and a video transmission unit 143. These functional blocks are implemented in software when a computer program related to providing video V is executed by the processor 25 or 16. The processing of each functional block in Figure 1 will be explained below with reference to Figure 2.

[0020] Figure 2 is a conceptual diagram showing an example of image synthesis by the image synthesis unit 142 shown in Figure 1. In the example shown in Figure 2, the vehicle 101 is equipped with six cameras 15 (front camera 151, right front camera 152, left front camera 153, rear camera 154, right rear camera 155, and left rear camera 156).

[0021] The remote cockpit 201 is equipped with, for example, three displays 22 (221-223). Specifically, the central display 221 displays video V1 (a video of the front of the vehicle 101) from the front camera 151. The right display 222 is located to the right of the central display 221 in the remote cockpit 201 and displays video V2 (a video of the front right of the vehicle 101) from the right front camera 152. The left display 223 is located to the left of the central display 221 and displays video V3 (a video of the front left of the vehicle 101) from the left front camera 153. These videos V1-V3 are panoramic images.

[0022] Furthermore, the central display 221 superimposed and supplementarily displays the image V4 (hereinafter referred to as adjusted image Va4) from the rear camera 154 after the image adjustment described later has been performed by the image synthesis unit 142. That is, the central display 221 displays the composite image Vc1, which is synthesized from image V1 and adjusted image Va4 by the image synthesis unit 142. Similarly, the right display 222 superimposedly displays the image V5 (hereinafter referred to as adjusted image Va5) from the right rear camera 155 after image adjustment. That is, the right display 222 displays the composite image Vc2, which is synthesized from image V2 and adjusted image Va5. The left display 223 superimposedly displays the image V6 (hereinafter referred to as adjusted image Va6) from the left rear camera 156 after image adjustment. That is, the left display 223 displays the composite image Vc3, which is synthesized from image V3 and adjusted image Va6. To add to this, image V4 is the rear view of the vehicle 101 (equivalent to the rearview mirror image). Videos V5 and V6 are images of the right rear and left rear (equivalent to the door mirrors) of vehicle 101, respectively.

[0023] In the remote support system 1, user U can adjust the video synthesis rules R in video synthesis performed by the video synthesis unit 142 by operating the video adjustment HMI 23 (customization function). In the example shown in Figure 2, the video synthesis rules R define how videos V4 to V6 are displayed on displays 221 to 223. The video synthesis rules R relate to, for example, at least one setting for the cropping, resizing, and pasting locations of video V.

[0024] In Figure 1, the video adjustment HMI 23 receives input of the video synthesis rule R from user U who wishes to adjust the video synthesis rule R. In other words, the video synthesis rule R is adjusted by the signal from the video adjustment HMI 23 operated by user U. The video adjustment HMI 23 outputs the content of the video synthesis rule R (video adjustment content) input by user U to the adjustment content transmission unit 241. The adjustment content transmission unit 241 transmits the video adjustment content acquired from the video adjustment HMI 23 to the vehicle 101.

[0025] The adjustment content receiving unit 141 receives video adjustment content from the remote cockpit 201 and outputs the received video adjustment content to the video synthesis unit 142. The video synthesis unit 142 generates a composite video Vc from the video V of each camera 15 according to the video adjustment content acquired from the adjustment content receiving unit 141.

[0026] Specifically, in the example shown in Figure 2, the video compositing unit 142 extracts a portion of the video V4 from the rear camera 154 as video V4_1 according to the video V4 extraction settings included in the video adjustment settings. Next, the video compositing unit 142 resizes the extracted video V4_1 according to the video V4 resize settings to obtain video V4_2. Then, the video compositing unit 142 superimposes the resized video V4_2 onto video V1 as the adjusted video Va4 according to the video V4 paste location settings. This generates the composite video Vc1. The generation of composite videos Vc2 and Vc3, which include videos V5 and V6 respectively, can be performed similarly as shown in Figure 2.

[0027] The video synthesis unit 142 outputs the synthesized video Vc (e.g., Vc1 to Vc3) generated as described above to the video transmission unit 143. The video transmission unit 143 transmits the acquired synthesized video Vc to the remote cockpit 201. The video receiving unit 242 receives the synthesized video Vc from the vehicle 101 and outputs the received synthesized video Vc to the display 22. As a result, the synthesized video Vc is displayed on the display 22, for example, as shown in Figure 2.

[0028] As described above, the remote support system 1 according to Embodiment 1 provides a user U who provides remote support for the vehicle 101 with a function (customization function) to adjust the video synthesis rule R for the synthesized video Vc displayed on the display 22 of the remote cockpit 201. This makes it possible to provide a synthesized video Vc that is easy for each individual user U to view. This contributes to reducing the burden on user U who uses the remote cockpit 201 (remote support device) for remote support.

[0029] 1-3. The first variation Figure 3 is a schematic diagram showing an example of the configuration of the remote support system 2 according to the first modified example of Embodiment 1. The remote support system 2 differs from the remote support system 1 shown in Figure 1 in the configuration of the "remote support device".

[0030] In comparison with the remote cockpit 201 (see Figure 1), the remote cockpit 202 shown in Figure 3 further includes a vehicle selection HMI 27 and a login HMI 28. Furthermore, the control device 24 of the remote cockpit 202 is equipped with a video adjustment content database (video adjustment content DB) 243. The video adjustment content DB 243 stores the content of the video synthesis rule R (video adjustment content) adjusted by the video adjustment HMI 23 (adjustment device) in response to the operation of the user U for each user U. The video adjustment content DB 243 is stored in the storage device 26. Additionally, in the example of the video adjustment content DB 243 shown in Figure 3, the video adjustment content is stored for each user U and for each vehicle 101 that is the target of remote support.

[0031] In the remote support system 2, when user U starts using the remote cockpit 202, user U logs in to the remote cockpit 202 by operating the login HMI 28. For example, user U operates the login HMI 28 and enters their user ID and password. Next, user U operates the vehicle selection HMI 27 and enters the vehicle ID of the vehicle 101 that they will remotely support, that is, selects the vehicle 101 that will be the target of their remote support. In the example shown in Figure 3, the remote cockpit 202 is equipped with three HMIs 23, 27, and 28 individually, but two or three of these HMIs 23, 27, and 28 may be integrated.

[0032] The remote cockpit 202 (processor 25) searches the video adjustment content DB using the user ID and vehicle ID obtained through HMI 28 and 27, and retrieves the video adjustment content corresponding to the obtained user ID and vehicle ID. More specifically, the video adjustment content DB 243 stores the settings of the video V associated with the camera IDs of multiple cameras 15 of the vehicle 101 (e.g., cropping settings, resizing settings, pasting position settings) as video adjustment content. If the video adjustment content corresponding to the entered user ID and vehicle ID is not registered in the video adjustment content DB, a pre-set initial value is used as the video adjustment content.

[0033] In the example shown in Figure 3, the adjustment content transmission unit 241 transmits the video adjustment content obtained from the video adjustment content DB 243 to the vehicle 101, as described above. The processing of the other functional blocks in Figure 3 is the same as in the example shown in Figure 1.

[0034] In the remote cockpit 202 shown in Figure 3, user U can operate the video adjustment HMI 23 as needed to change the video adjustment settings for the selected vehicle 101. The remote cockpit 202 updates the video adjustment settings DB 243 with the information entered by user U through the video adjustment HMI 23. As a result, vehicle 101 generates a composite image Vc according to the updated video adjustment settings, and this composite image Vc is displayed on the display 22 of the remote cockpit 202. User U can repeatedly operate the video adjustment HMI 23 to adjust the way the composite image Vc is displayed (video synthesis rule R) until the desired composite image Vc is obtained.

[0035] As described above, the remote cockpit 202 (remote support device) according to the first modified example is equipped with a function (video adjustment content DB243) to store the video adjustment content for each user U. This makes it possible to save the video adjustment content (i.e., the video adjustment result) set by user U and to reproduce that video adjustment content during the next remote support session. Therefore, a more practical remote support system 2 can be provided.

[0036] 1-4. Second variation Figure 4 is a schematic diagram showing an example configuration of a remote support system 3 according to a second modified example of Embodiment 1. Remote support system 3 differs from remote support system 2 shown in Figure 3 in the configuration of the "remote support device". Specifically, the remote support device in remote support system 3 includes a remote cockpit 203 (remote support terminal) and a relay server 301.

[0037] In comparison with the remote cockpit 202 (see Figure 3), the remote cockpit 203 is equipped only with a video receiver 242 as a functional block.

[0038] The relay server 301 relays communication between the vehicle 101 and the remote cockpit 203. The relay server 301 includes a communication device 31 and a control device 32. The communication device 31 communicates with multiple vehicles 101 that constitute the remote support system 3. The communication device 31 also communicates with multiple remote cockpits 203 that constitute the remote support system 3. The control device 32 provides video V to user U in cooperation with the remote cockpit 203 and the vehicle 101. The control device 32 includes one or more processors 33 (hereinafter simply referred to as processor 33) and one or more storage devices 34 (hereinafter simply referred to as storage devices 34). An example of the configuration of the processor 33 is the same as that of the processor 16 described above. Also, an example of the configuration of the storage device 34 is the same as that of the storage device 17 described above. The processor 33 executes a computer program. The computer program is stored in the storage device 34. The computer program may be recorded on a computer-readable recording medium. The functions of the control unit 32 are realized through the cooperation of the processor 33, which executes the computer program, and the storage device 34.

[0039] The relay server 301 is equipped with an adjustment content transmission unit 321 similar to the adjustment content transmission unit 241 (see Figure 3). The relay server 301 is also equipped with a video relay unit 322 that relays the transmission of the composite video Vc from the video transmission unit 143 of the vehicle 101 to the video receiving unit 242 of the remote cockpit 203.

[0040] Furthermore, the relay server 301 is equipped with a user database 324 and a vehicle database 325, along with a video adjustment content database 323 similar to the video adjustment content database 243 (see Figure 3). These databases 323 to 325 are stored in the storage device 34. The user database 324 stores the user ID of each user U in the remote support system 3. The vehicle database 325 stores the vehicle ID of each vehicle 101 included in the remote support system 3. The relay server 301 (processor 33) obtains the user ID and vehicle ID from the user database 324 and vehicle database 325, respectively, in response to the login HMI 28 and vehicle selection HMI 27 operations performed by the user U. The relay server 301 then obtains the video adjustment content corresponding to the obtained user ID and vehicle ID from the video adjustment content database 323.

[0041] As described above, in the remote support system 3 according to the second modification, the video adjustment content DB323 is stored in the relay server 301 rather than the remote cockpit 203. In other words, the video adjustment content DB323 is provided in a way that allows easy access from multiple remote cockpits 203. As a result, even if user U uses a different remote cockpit 203 when providing remote support for the first time than the remote cockpit 203 used during the previous remote support, user U can smoothly utilize the video adjustment content (i.e., video adjustment results) used in the previous support. Therefore, a remote support system 3 that is more convenient for user U can be provided.

[0042] 1-5. Third variation Figure 5 is a schematic diagram showing an example of the configuration of a remote support system 4 according to a third modified example of Embodiment 1. Remote support system 4 differs from remote support system 3 shown in Figure 4 in the configuration of the "remote support device" and the "relay server".

[0043] The form of the remote cockpit used in a remote support system is not limited to one. Example form EX1 shown in Figure 5 shows a desktop personal computer (PC) based remote cockpit 204. Example form EX2 shows a tablet PC 224 based remote cockpit 204 including a display and a control device 24.

[0044] In comparison with the remote cockpit 203 (see Figure 4), the remote cockpit 204 further includes a cockpit configuration selection HMI 29. In a third modified example, when user U starts using the remote cockpit 204, user U also operates the cockpit configuration selection HMI 29 and inputs a cockpit configuration ID indicating the configuration of the remote cockpit 204 that he / she will use, that is, selects the configuration of the remote cockpit 204.

[0045] In comparison with the relay server 301 (see Figure 4), the relay server 301 shown in Figure 5 includes a video adjustment content DB 326 instead of a video adjustment content DB 323. The video adjustment content DB 326 stores video adjustment content not only for each user ID and vehicle ID, but also for each cockpit configuration ID. The relay server 302 (processor 33) then retrieves the video adjustment content corresponding to the acquired user ID, vehicle ID, and cockpit configuration ID from the video adjustment content DB 326. The remote cockpit 204 may also transmit its own cockpit configuration ID to the relay server 302 without input from user U.

[0046] As described above, in the remote support system 4 according to the third modified example, the video adjustment content DB326 stores video adjustment content for each cockpit configuration ID (i.e., for each configuration of the remote support terminal). By adding the remote cockpit configuration to the search conditions of the video adjustment content DB326 in this way, it becomes possible to set or reproduce the display of the composite video Vc for each remote cockpit configuration. Different remote cockpit configurations may have different display sizes or aspect ratios, for example. Therefore, by changing the way the composite video Vc is displayed according to the remote cockpit configuration, a remote support system 4 that is even more convenient for the user U can be provided. The remote cockpit configuration may also be added to the search conditions of the video adjustment content DB326 used in the second embodiment described later.

[0047] 2. Embodiment 2 In Embodiment 1, the vehicle 101 is equipped with a video synthesis function. In contrast, in Embodiment 2, the remote support device is equipped with a video synthesis function.

[0048] 2-1. Configuration of the Remote Support System Figure 6 is a schematic diagram showing an example configuration of the remote support system 5 according to Embodiment 2. The remote support system 5 includes a vehicle 102 and a remote cockpit 205 (remote support device). The control device 14 of the vehicle 102 includes a video transmission unit 143 (see Figure 1) and a multiplexer (MUX) 144. On the other hand, the control device 24 of the remote cockpit 205 includes a video receiving unit 242 and a video adjustment content DB 243 (see Figure 3), along with a demultiplexer (DEMUX) 244, a video processing unit 245, and a video synthesis unit 246.

[0049] In Figure 6, the MUX144 integrates the signals of multiple video V from multiple cameras 15 of the vehicle 102 into a single video signal and outputs the integrated single video signal to the video transmission unit 143. An example of multiple video V is the aforementioned video V1 to V3, V5, and V6 and video Vx. The multiple cameras 15 may include, for example, an omnidirectional camera that images the area around the vehicle 102. Video Vx is the video from the omnidirectional camera.

[0050] The video receiving unit 242 of the remote cockpit 205 outputs the single video signal received from the video transmitting unit 143 to the DEMUX 244. The DEMUX 244 decomposes the input single video signal into multiple video V signals, one for each of the multiple cameras 15. The DEMUX 244 then outputs video Vx, one of the multiple video V signals, to the video processing unit 245, and directly outputs the remaining video V1-V3, V5, and V6 to the video synthesis unit 246.

[0051] The video processing unit 245 applies a "high-cost video processing P" to the input video Vx. An example of video processing P is the process of unfolding the omnidirectional camera video Vx into a plane to obtain video Vxp, as shown in Figure 6. The video processing unit 245 outputs the acquired video Vxp to the video synthesis unit 246. Similar to the video synthesis unit 142 (see Figure 1), the video synthesis unit 246 generates a composite video Vc according to the video adjustment content from the video adjustment content DB 243 and outputs the generated composite video Vc to the display 22. An example of a composite video Vc generated by the video synthesis unit 246 is the same as composite videos Vc2 to Vc4 shown together with the second video synthesis unit 248 in Figure 10, described later.

[0052] The remote support system 5 according to Embodiment 2 described above also provides a user U who remotely supports the vehicle 102 with a function (customization function) to adjust the video synthesis rule R for the synthesized video Vc displayed on the display 22 of the remote cockpit 205. This provides the same effects as in Embodiment 1.

[0053] Furthermore, the remote cockpit 205 included in the remote support system 5 may be, for example, a desktop PC-based terminal device equipped with a high-performance GPU as the processor 25, which is suitable for executing the aforementioned video processing P. This makes it possible to construct a remote support system 5 with excellent video processing performance while suppressing the increase in video processing costs on the vehicle 102 side.

[0054] 2-2. First variation Figure 7 is a schematic diagram showing an example configuration of the remote support system 6 according to the first modified example of Embodiment 2. The remote support system 6 includes a remote cockpit 206 (remote support terminal) and a relay server 301 as remote support devices together with the vehicle 102. The differences between the remote support system 6 and the remote support system 5 (see Figure 6) are the same as the differences between the remote support system 3 (see Figure 4) and the remote support system 2 (see Figure 3) in Embodiment 1. That is, in Embodiment 2 as well, the video adjustment content DB323 may be provided in the relay server 301 instead of the remote cockpit. This provides the same effect as the second modified example of Embodiment 1. In the remote support system 6, the remote cockpit 206 is equipped with an adjustment content receiving unit 247.

[0055] 2-3. Second Variation Figure 8 is a schematic diagram showing an example configuration of a remote support system 7 according to a second modified example of Embodiment 2. The remote support system 7 includes a remote cockpit 203 (see Figure 4) and a relay server 303 as remote support devices, along with the vehicle 102. The remote support system 7 differs from the remote support system 6 (see Figure 7) in that the relay server 303, rather than the remote cockpit, has a video synthesis function. Specifically, compared to the relay server 301 (see Figure 7), the relay server 303 is equipped with a video receiving unit 327 that receives the aforementioned single video signal from the vehicle 102. Furthermore, the relay server 303 is equipped with a DEMUX 328, a video processing unit 329, and a video synthesis unit 330, similar to the DEMUX 244, video processing unit 245, and video synthesis unit 246 (see Figure 6). In addition, as shown in the example in Figure 8, the relay server 303 has both a video synthesis function and a video processing function.

[0056] According to the second modification described above, the remote support system 7 can be configured while reducing the video processing capability requirements of the remote cockpit 203. Therefore, even when using a tablet PC or smartphone as the remote support terminal, a remote support system 7 equipped with a video synthesis function that involves video processing P with high processing costs can be successfully realized by combining the remote support terminal with a relay server 303 (e.g., a GPU cloud) that has high video processing capability.

[0057] In Figures 6 and 7, the control device 24 of the remote cockpit 205 or 206 (remote support terminal), which includes the video synthesis unit 246, corresponds to the "video synthesis device." In Figure 8, the control device 32 of the relay server 303, which includes the video synthesis unit 330, corresponds to the "video synthesis device." Alternatively, the video synthesis unit may be distributed between the remote support terminal and the relay server.

[0058] 3. Embodiment 3 In Embodiment 3, both the vehicle and the remote support device are equipped with distributed video synthesis functions (first and second video synthesis functions).

[0059] 3-1. Configuration of the Remote Support System Figure 9 is a schematic diagram showing an example configuration of the remote support system 8 according to Embodiment 3. The remote support system 8 includes a vehicle 103 and a remote cockpit 207 (remote support device). Compared with vehicle 102 (see Figure 6), vehicle 103 is additionally equipped with an adjustment content receiving unit 141 (see Figure 3) and a first video synthesis unit 145. On the other hand, compared with remote cockpit 205 (see Figure 6), remote cockpit 207 is additionally equipped with an adjustment content transmitting unit 241 (see Figure 3) and a second video synthesis unit 248 instead of a video synthesis unit 246.

[0060] In Figure 9, the control device 14 of the vehicle 103, which includes the first video synthesis unit 145, corresponds to the "first video synthesis device," and the control device 24 of the remote cockpit 207, which includes the second video synthesis unit 248, corresponds to the "second video synthesis device." However, the remote support device according to Embodiment 3 may include a relay server similar to the relay server 303 (see Figure 8). The control device of the relay server may include functional blocks corresponding to the video processing unit 245 and the second video synthesis unit 248.

[0061] 3-2. First and Second Video Synthesis Functions Figure 10 is a conceptual diagram showing an example of image synthesis performed by the first image synthesis unit 145 and the second image synthesis unit 248 shown in Figure 9.

[0062] The first video synthesis unit 145 of the vehicle 103 receives, for example, videos V1-V3, V5, and V6 (see Figure 2). Similar to the video synthesis unit 142 (see Figure 2), the first video synthesis unit 145 generates synthesized videos Vc2 and Vc3 according to the video synthesis rule R (video adjustment content from video adjustment content DB243) (first synthesis function). Videos V2, V3, V5, and V6, which are the source of synthesized videos Vc2 and Vc3, are examples of "multiple first camera videos" from the video V of the multiple cameras 15 of the vehicle 103 that do not require the high-cost video processing P. The first video synthesis unit 145 outputs the generated synthesized videos Vc2 and Vc3 together with video V1 to the MUX 144. The MUX 144 also receives video Vx (see Figure 6) that requires video processing P (e.g., planar unfolding processing of omnidirectional camera video).

[0063] The DEMUX244 of the remote cockpit 207 decomposes the single video signal from the MUX144 and outputs video V1 and composite videos Vc2 and Vc3 to the second video synthesis unit 248, and outputs video Vx to the video processing unit 245. The video processing unit 245 (for example, a GPU as the processor 25) performs video processing P and outputs the processed video Vxp to the second video synthesis unit 248. The second video synthesis unit 248 generates a composite video Vc4 of videos V1 and Vxp according to the video synthesis rule R (video adjustment content from video adjustment content DB243) (second synthesis function). Then, the second video synthesis unit 248 outputs the generated composite video Vc4 together with composite videos Vc2 and Vc3 to the display 22. The images V1 and Vx that form the basis of the composite image Vc4 correspond to an example of "multiple second camera images" which include the image Vx that has undergone image processing P in the remote support device, among the images V from multiple cameras 15 of the vehicle 103.

[0064] As described above, according to the remote support system 8 of Embodiment 3, the synthesis of multiple first camera images, which does not require the costly video processing P, is performed in the vehicle 103. Then, the synthesis of multiple second camera images, including camera images that require video processing P, is performed in the remote support device (e.g., remote cockpit 207). As a result, unnecessary images are removed during video synthesis by the first video synthesis unit 145 of the vehicle 103 (e.g., cutting out images V5 and V6 when generating adjusted images Va5 and Va6). Therefore, compared to the remote support system 6 (see Figure 6), the amount of video information transmitted by the vehicle 103 is reduced, which is advantageous in terms of communication cost and latency performance. [Explanation of Symbols]

[0065] 1-8 Remote support system, 11, 21, 31 Communication device, 14, 24, 32 Control device, 15 Camera, 22 Display, 101-103 Vehicle, 201-207 Remote cockpit, 301-303 Relay server

Claims

1. A vehicle containing multiple cameras, A remote support device for communicating with the vehicle and providing remote support to the vehicle, A remote support system comprising: At least one of the vehicle and the remote support device includes an image synthesis device that synthesizes multiple camera images captured by the multiple cameras according to predetermined image synthesis rules. The remote support device is A display that shows the composite image synthesized by the aforementioned image synthesis device, An adjustment device that adjusts the video synthesis rules in accordance with the operation of the user of the remote support device, including Remote support system.

2. A remote support system according to claim 1, The remote support device further includes one or more storage devices that store a video adjustment content database for each user of the remote support device, which stores the video adjustment content, which is the content of the video synthesis rules adjusted by the adjustment device in response to the user's operation. Remote support system.

3. A remote support system according to claim 2, The remote support device is A remote support terminal operated by the user for the purpose of the remote support, A relay server that relays the communication between the vehicle and the remote support terminal, Includes, The remote support terminal includes the adjustment device, The relay server includes the one or more storage devices. Remote support system.

4. A remote support system according to any one of claims 1 to 3, The aforementioned video synthesis device is The first video synthesis device included in the aforementioned vehicle, The second video synthesis device included in the remote support device, Includes, The first video synthesis device synthesizes multiple first camera images from among the multiple camera images that do not require high-cost video processing according to the video synthesis rules. The second video synthesis device synthesizes a plurality of second camera images, including the camera image that undergoes the video processing in the remote support device, from among the plurality of camera images, in accordance with the video synthesis rules. Remote support system.

5. A remote support system according to claim 2, The remote support device includes a remote support terminal operated by the user for the purpose of providing remote support. The aforementioned video adjustment database stores the video adjustment details for each type of remote support terminal. Remote support system.

Images