Display control device

Abstract

To improve the visibility of a remote operator with respect to the image display of a display device for remote operation of a moving body when the moving body passes through an intersection.SOLUTION: A display control device is configured to control the image display of a display device for remote operation of a moving body by a remote operator. The display device includes a first screen that displays an intersection image captured by a camera installed at an intersection through which the moving body passes and imaging a traffic situation in an area of the intersection. The display control device includes a sensor that acquires intention information of the remote operator regarding right and left turns of the moving body, and a processor. The processor determines the shape of a course image indicating a planned course of the moving body in accordance with a traveling direction of the moving body in the intersection image and the intention information, and displays the course image in a superimposed manner on the intersection image while associating the course image with the moving body in the intersection image.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a display control device for remotely operating a moving body.

Background Art

[0002] Patent Document 1 discloses a traffic information display system including an information display device mounted on a vehicle. This information display device obtains the direction of the route of the host vehicle in the received captured image from the map information of the selected point and the received camera parameters, and synthesizes information indicating the direction of the route in the captured image and displays it on the screen.

[0003] <-- --> Further, Patent Document 2 discloses a system for remotely instructing an autonomous vehicle. This system sets the range of sensor information transmitted from the vehicle to a remote commander to a limited information range based on the external situation and route of the autonomous vehicle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] For the safety confirmation of a remote operator during the remote operation (remote driving or remote assistance) of a moving body such as an automobile, it is conceivable to display an image of a camera installed at an intersection on the screen of a display device of the remote operator terminal. For such display of an image, it is required that a remote operator who views the image displayed on the display device can easily grasp the traffic situation while instantaneously grasping the route of the moving body in the image.

[0006] However, in the technology described in Patent Document 1, the "information indicating the direction of the path" that is synthesized into the captured image is determined based on map information and camera parameters, and is not determined in a way that promptly reflects the path of the moving object that the driver of the vehicle is currently considering (i.e., the driver's intention regarding the path of the moving object).

[0007] Therefore, even if the technology described in Patent Document 1 could be applied to "remote control of a moving object," the "information indicating the direction of the path" would not be able to appropriately display the path of the moving object according to the intentions of the remote operator when the moving object passes through an intersection. For this reason, the technology described in Patent Document 1 has room for improvement in enabling a remote operator performing "remote control of a moving object" to instantly grasp the path of the moving object in the image and easily understand the traffic situation. The same applies to the technology described in Patent Document 2.

[0008] This disclosure has been made in view of the above-mentioned problems, and aims to provide a display control device that can improve the visibility of the image display on a display device for remote control of a moving object when the moving object passes through an intersection. [Means for solving the problem]

[0009] The display control device according to this disclosure is configured to control the image display of a display device for remote operation of a mobile object by a remote operator. The display device includes a first screen that displays an intersection image, which is an image from a camera installed at an intersection through which the mobile object passes and which captures the traffic conditions in the intersection area. The display control device includes a sensor that acquires the remote operator's intention information regarding the mobile object's right or left turn, and a processor. The processor determines the shape of a path image indicating the mobile object's planned path according to the mobile object's direction of travel within the intersection image and the intention information, and displays the path image superimposed on the intersection image while associating it with the mobile object within the intersection image. [Effects of the Invention]

[0010] According to this disclosure, when a moving object passes through an intersection, the visibility of the image displayed on the display device for remote control of the moving object can be improved for the remote operator. [Brief explanation of the drawing]

[0011] [Figure 1] This figure schematically shows an example configuration of a remote control system to which the display control device according to the embodiment is applied. [Figure 2] This figure is intended to specifically illustrate an example of how an intersection image is displayed according to the embodiment. [Figure 3] This figure illustrates an example of an intersection image display according to the embodiment and specifically explains problem X. [Figure 4] This figure shows a specific example of the display of an intersection image according to the embodiment. [Figure 5] This flowchart shows an example of the processing flow related to image display according to the embodiment. [Figure 6] This is a diagram illustrating additional image processing according to the embodiment. [Modes for carrying out the invention]

[0012] Embodiments of this disclosure will be described below with reference to the attached drawings. In each drawing, elements common to all figures are denoted by the same reference numerals, and redundant explanations are omitted or simplified.

[0013] 1. Configuration of the remote control system Figure 1 is a schematic diagram showing an example of the configuration of a remote control system 1 to which the display control device according to the embodiment is applied. The remote control system 1 comprises a vehicle 10, a remote operator terminal 20, a camera device 30, and a management server 40.

[0014] 1-1. Vehicles Vehicle 10 is an example of a "mobile object" that is the target of "remote operation" according to the present disclosure. More specifically, for example, it is an automobile. Vehicle 10 includes a GNSS (Global Navigation Satellite System) receiver 11, one or more in-vehicle cameras (mobile object cameras) 12, a traveling device 13, a communication device 14, an antenna 15, and a control device 16.

[0015] The GNSS receiver 11 is used to acquire the position and orientation of the vehicle 10. The one or more in-vehicle cameras 12 recognize (detect) the situation around the vehicle 10. The one or more in-vehicle cameras 12 include cameras that image the front of the vehicle 10 (more specifically, for example, the center front, left front, and right front of the vehicle 10, respectively).

[0016] The traveling device 13 includes a driving device, a braking device, and a steering device. The driving device includes, for example, at least one of an electric motor and an internal combustion engine for driving the vehicle 10. The braking device includes a brake actuator for braking the vehicle 10. The steering device includes, for example, an electric motor for steering the wheels.

[0017] The communication device 14 communicates with the outside of the vehicle 10. Specifically, the communication device 14 communicates with the remote operator terminal 20 via, for example, the antenna 15. body Specifically, the communication device 14 communicates with the remote operator terminal 20 via, for example, the antenna 15.

[0018] The control device 16 is a computer that controls the vehicle 10. The control device 16 includes one or more processors and one or more storage devices. The one or more processors execute various processes related to the control of the vehicle 10. The one or more storage devices store various information necessary for the processes by the one or more processors.

[0019] During the remote operation of the vehicle 10, the control device 16 communicates with the remote operator terminal 20 via the communication device 14. The control device 16 transmits vehicle information VCL to the remote operator terminal 20. The vehicle information VCL is information necessary for remote operation by the remote operator and includes information on the surrounding situation of the vehicle 10. The surrounding situation information includes images captured by the in-vehicle camera 12. The surrounding situation information may include sounds detected by the in-vehicle microphone. The surrounding situation information may include object information regarding moving objects around the vehicle 10. The object information indicates the relative position and relative speed of the object with respect to the vehicle 10. Further, the vehicle information VCL may include vehicle state information such as the speed, acceleration, yaw rate, and steering angle of the vehicle 10, as well as vehicle position information.

[0020] Furthermore, the control device 16 receives remote operation information from the remote operator terminal 20. The remote operation information is information regarding remote operation by the remote operator. For example, the remote operation information includes the operation amount by the remote operator. The control device 16 performs vehicle travel control by controlling the travel device 13 according to the received remote operation information.

[0021] 1-2. Remote Operator Terminal The remote operator terminal 20 is a terminal device used when the remote operator remotely operates the vehicle 10. The "remote operation" according to the present disclosure is a concept that includes "remote driving" in which the remote operator drives a moving body from a remote location and "remote support" in which the operation of the moving body is supported from a remote location. In addition, when the remote operation is performed as remote driving, the remote operator terminal 20 corresponds to a remote cockpit.

[0022] The remote operator terminal 20 includes a communication device 21, an antenna 22, a display device 23, a speaker 24, an operation device 25, and a control device 26.

[0023] The communication device 21 communicates (wireless communication) with the vehicle 10 via the antenna 22. The communication device 21 also communicates with the management server 40.

[0024] The display device 23 presents various information to the remote operator by displaying various types of information. This information includes images of the intersection area (intersection images) captured by multiple intersection cameras 31. In other words, the intersection images are images showing the traffic conditions in the area of ​​the intersection into which the vehicle 10 is entering.

[0025] Figure 1 shows an example of the screen configuration of the display device 23 as seen from the remote operator. That is, the right side of the page corresponds to the right side of the screen as seen from the remote operator, and the left side of the page corresponds to the left side of the screen as seen from the remote operator. In this example, the display device 23 has four monitors 231, 232, 233, and 234. Monitors 231, 232, and 233 display images of the front of the vehicle 10 (vehicle front images) captured by multiple on-board cameras 12. More specifically, as an example, the vehicle front images include a front-center image displayed by monitor 231, a left-front image displayed by monitor 232, and a right-front image displayed by monitor 233. As shown in Figure 1, monitor 232 is located to the left of monitor 231, and monitor 233 is located to the right of monitor 231.

[0026] Monitor 234 also displays the intersection image. As shown in Figure 1, Monitor 234 is positioned, for example, above Monitors 231-233. Specific examples of the images displayed on Monitor 234 will be described later with reference to Figures 2-4 and 6. In the example shown in Figure 1, the display device 23 has one monitor (Monitors 231-234) for each image.

[0027] In the example shown in Figure 1, the screen of monitor 234 and the screens of monitors 231-233 correspond to the "first screen" and "second screen" as described in this disclosure, respectively. Alternatively, instead of the example shown in Figure 1, the "display device" as described in this disclosure may have a single monitor that displays both the vehicle forward image and the intersection image. Also, in the example shown in Figure 1, there is one intersection image displayed, but there may be two or more. For example, two intersection images may be arranged side by side above monitors 231-233.

[0028] Speaker 24 presents sound information to the remote operator by outputting sound information. The sound information includes sounds detected by the in-vehicle microphone described above.

[0029] The operating device 25 receives input from a remote operator. For example, the operating device 25 includes remote control components that a remote operator operates when remotely controlling the vehicle 10. These remote control components include an accelerator pedal, a brake pedal, a steering wheel, and a turn signal control. The operating device 25 also includes switches for various other operations related to remote control.

[0030] The control device 26 controls the remote operator terminal 20. The control device 26 includes one or more processors 27 (hereinafter simply referred to as "processor 27") and one or more storage devices 28 (hereinafter simply referred to as "storage devices 28"). The processor 27 performs various processes. For example, the processor 27 includes a CPU (Central Processing Unit). The storage devices 28 store various information necessary for processing by the processor 27. The storage devices 28 are, for example, at least one of volatile memory, non-volatile memory, HDD (Hard Disk Drive), and SSD (Solid State Drive).

[0031] The remote control program is a computer program executed by the processor 27. The execution of the remote control program by the processor 27 enables the functionality of the control device 26. The remote control program is stored in the storage device 28. Alternatively, the remote control program may be recorded on a computer-readable recording medium. The remote control program may also be provided via a network.

[0032] The control device 26 communicates with the vehicle 10 via the communication device 21. The control device 26 receives vehicle information VCL transmitted from the vehicle 10. The control device 26 presents the vehicle information VCL to the remote operator via the display device 23 and speaker 24. For example, the control device 26 displays images captured by the on-board camera 12 on the display device 23. The control device 26 also outputs sounds detected by the on-board microphone through the speaker 24. Based on the vehicle information VCL, the remote operator can recognize the status of the vehicle 10 and the surrounding conditions.

[0033] The remote operator operates the remote control member of the operating device 25. The amount of operation of the remote control member is detected by sensors 29 installed on the remote control member. The control device 26 generates remote control information that reflects the amount of operation of the remote control member by the remote operator. The control device 26 then transmits the remote control information to the vehicle 10 via the communication device 21. More specifically, the sensors 29 include sensors that detect the amount of operation of the accelerator pedal, brake pedal, and steering wheel, respectively. The sensors 29 also include, for example, a sensor that detects the operation position of the turn signal control (e.g., turn signal lever) and a sensor that detects the line of sight (more specifically, for example, the position of the line of sight) of the remote operator looking at monitors 231-233 (i.e., the vehicle front image).

[0034] 1-3. Intersection Camera The intersection camera 31 (hereinafter also simply referred to as "camera 31") is installed at each intersection through which the vehicle 10 passes and captures images of the intersection area (the interior of the intersection and its vicinity). Camera 31 is configured as a camera device 30, for example, which includes camera 31 and a communication device 32. In the examples shown in Figures 2-4 and 6 described later, one camera 31 is installed for each intersection.

[0035] Images of the intersection area captured by each camera 31 (intersection images) are transmitted to the management server 40 via the communication device 32. The management server 40 is configured to communicate with each camera device 30 and manages the intersection images transmitted from each camera device 30. The management server 40 is also configured to communicate with the remote operator terminal 20. When a vehicle 10 approaches an intersection area, the control device 26 of the remote operator terminal 20 requests the management server 40 to transmit images of that intersection area. In response to this request, the management server 40 transmits images of the corresponding intersection area to the remote operator terminal 20.

[0036] Furthermore, communication between the management server 40 and each camera device 30 may be via wired or wireless communication. The same applies to communication between the management server 40 and the remote operator terminal 20. Also, unlike the example shown in Figure 1, the acquisition of intersection images by the remote operator terminal 20 may be performed by direct communication with each camera device 30 without going through the management server 40.

[0037] 2. Controlling image display on the display device In the remote control system 1 described above, the control device 26 of the remote operator terminal 20 controls the image display 23 for remote control of the vehicle 10 (mobile body). For this reason, the control device 26 corresponds to an example of the "display control device" described herein.

[0038] When displaying images from the intersection camera 31 on the display device 23 for safety confirmation by a remote operator, the display of such images requires that the remote operator viewing the images on the display device 23 can instantly grasp the direction of travel Dv of the vehicle 10 in the images and easily understand the traffic situation.

[0039] More specifically, as illustrated in Figure 1, the screen of the display device 23 installed on the remote operator terminal 20 simultaneously displays and mixes images from the on-board camera 12 (vehicle front image) and images from the intersection camera 31 (intersection image). The remote operator is required to check all of these multiple images. According to the remote operator, each time they focus their gaze on an individual image, they perceive the space within the image and, if necessary, mentally transform the space for that spatial perception. Furthermore, the orientation (imaging direction Dc) and installation location of the intersection camera 31 vary depending on the environment of each intersection. Therefore, the intersection image for each intersection is affected by these differences in the orientation and installation location of the intersection camera 31. Consequently, the direction of travel Dv of the vehicle 10 in the intersection image... ( More specifically, it is difficult to instantly grasp the "direction of travel in the image (DVI)" described later, as well as the positional relationship between vehicle 10 and other moving objects such as other vehicles and pedestrians within the intersection area, and to instantly determine the direction of movement of those moving objects (for example, whether or not they are approaching vehicle 10).

[0040] Figures 2 and 3 are diagrams illustrating an example of intersection image display according to the embodiment and problem X in detail. This specific example will be explained using an intersection P1 where one intersection camera 31 is installed, as shown in Figures 2(A) and 3(A).

[0041] The area of ​​intersection P1 captured by camera 31 includes the area located within intersection P1, as well as the first road L1 on which vehicle 10 travels, and the second road L2 which intersects with the first road L1 at intersection P1.

[0042] For more details, first, see the figure. 2(A) shows the traffic situation in the area of ​​intersection P1 when a vehicle 10 traveling along the first road L1 in the direction of travel Dv1 is approaching intersection P1. In this situation, as seen from the vehicle 10, camera 31 is located, for example, on the left side of the road. Camera 31 captures images of the area within intersection P1 as well as the area of ​​the second road L2 located to the right of intersection P1. The image area IA of camera 31 includes a pedestrian 2 crossing the second road L2. Note that the location of camera 31 may be different from the above example; for example, it may be located inside intersection P1.

[0043] Figure 2(B) corresponds to the display of the intersection image on the display device 23 under the conditions described above as shown in Figure 2(A). The intersection image shown in Figure 2(B) is the same image as captured by the camera 31 and displayed directly on the screen of the display device 23. Note that the arrow indicating the direction of travel Dv1 shown in the intersection image in Figure 2(B) is shown for explanatory reference only and is not included in the intersection image. The same applies to the arrow indicating the direction of travel Dv2 in Figure 3(B), which will be described later.

[0044] As shown in the intersection image in Figure 2(B), pedestrian 2, who is located to the right (more specifically, to the right front) of vehicle 10 at the actual intersection P1, can be displayed on the screen of monitor 234 in a way that is similar to how it would appear to a remote operator. Therefore, the intersection image from camera 31 can be displayed in the same way as the vehicle front images displayed on monitors 231 and 233.

[0045] Next, Figure 3(A) shows the traffic situation in the area of ​​intersection P1 when a vehicle 10 traveling along direction Dv2, which is opposite to the direction of travel Dv1 shown in Figure 2(A), is approaching intersection P1. Figure 3(B) corresponds to an example in which the image from camera 31 is displayed directly on the monitor 234 screen, similar to the example shown in Figure 2(B), under the conditions shown in Figure 3(A). The intersection image shown in Figure 3(B) also includes vehicle 10.

[0046] As shown in the image display in Figure 3(B), pedestrian 2, who is located to the left front of vehicle 10 at the actual intersection P1 as shown in Figure 3(A), is displayed to the right from the remote operator's perspective in the intersection image. As a result, when the display device 23 displays the intersection image shown in Figure 3(B) along with the vehicle front image from the onboard camera 12, the position of pedestrian 2 in the intersection image is reversed left to right compared to the position of pedestrian 2 in the vehicle front image. This prevents the remote operator from instantly grasping the direction of travel (especially left to right) of vehicle 10 within the intersection image.

[0047] In view of the above-mentioned problem X, in this embodiment, the control device 26 (processor 27) displays the intersection image on the display device 23 using the following method. Figure 4 is a diagram showing a specific example of the display of the intersection image according to the embodiment. More specifically, Figure 4 shows the display examples (upper, middle, and lower) corresponding to straight-ahead, right-turn, and left-turn of the vehicle 10 for each of the three display examples EX1, EX2, and EX3. Note that the arrows indicating the direction of travel Dvi (Dvi2 to Dvi4) shown in each intersection image in Figure 4 are shown for explanatory reference only and are not included in the intersection image.

[0048] The control device 26 displays a "path image (object) A" indicating the planned path of the vehicle 10 as it passes through the intersection, superimposed on the intersection image while associating it with the vehicle 10 in the intersection image. In other words, the control device 26 superimposes the path image A onto the intersection image captured by the camera 31 using augmented reality (AR) technology.

[0049] Specifically, in this embodiment, the shape of the route image A superimposed on the intersection image is determined based on the direction of travel Dv of the vehicle 10 within the intersection image (hereinafter, for convenience, also referred to as "direction of travel Dvi in ​​the image") and the "intention information II" of the remote operator. In other words, the superimposed route image A is selected from three options: route image As indicating going straight, route image Ar indicating a right turn, and route image Al indicating a left turn, depending on the direction of travel Dvi in ​​the image and the intention information II. More specifically, in each example shown in Figure 4, the route image A uses arrows to indicate the direction of travel.

[0050] Intention Information II is the remote operator's intention information regarding the right or left turn of the vehicle 10, and is acquired using the sensors 29 described above. Intention Information II is, for example, turn signal operation information. This turn signal operation information indicates that the operating position of the turn signal control device operated by the remote operator is the neutral position, the right turn instruction position, or the left turn instruction position. In addition, Intention Information II includes the remote operator's operation information, and the turn signal operation information is an example of the remote operator's operation information.

[0051] Display example EX1 corresponds to the display of an intersection image under the conditions shown in Figure 3(A) above. When vehicle 10 enters intersection P1, if the turn signal control is not operated by the remote operator (i.e., the turn signal control is in the neutral position), the straight-ahead path image As1 is selected. As shown in Figure 4, the direction indicated by the path image As1 is determined to be in line with the direction of travel Dvi2 in the image.

[0052] On the other hand, if the turn signal control is operated by a remote operator when vehicle 10 enters intersection P1, the corresponding path image A is selected. Specifically, if the turn signal control is operated to the left-turn instruction position, the path image Al1 indicating a left turn is selected. Similarly, if the turn signal control is operated to the right-turn instruction position, the path image Ar1 indicating a right turn is selected. Furthermore, as shown in Figure 4, the direction indicated by path image Al1 or Ar1 is also determined to be in line with the direction of travel Dvi2 within the image.

[0053] In addition, as shown in Figure 4, the control device 26 arranges the route images As1, Al1, or Ar1 to be superimposed on the front of the vehicle 10 in the intersection image, based on the information from the intersection image. More specifically, in order to improve visibility for the remote operator, the superimposition of the route image A is performed while changing the angle of the route image A three-dimensionally to match the field of view of the intersection camera 31.

[0054] The superimposition of the path image A in other display examples EX2 and EX3 is also performed based on the same concept as in display example EX1 described above. Display examples EX2 and EX3 target the same intersection image of intersection P2, but differ in the direction of travel Dv of the vehicle 10 relative to intersection P2. Specifically, as shown in Figure 4, in display example EX2, the direction indicated by the path images As2, Al2, or Ar2 is determined so as to be aligned with the direction of travel Dvi3 in the image, and the path image As2, Al2, or Ar2 is selected according to the turn signal operation information. Similarly, in display example EX3, the direction indicated by the path images As3, Al3, or Ar3 is determined so as to be aligned with the direction of travel Dvi4 in the image, and the path image As3, Al3, or Ar3 is selected according to the turn signal operation information.

[0055] Furthermore, instead of the turn signal operation information described above, the intention information II may be information indicating, for example, the amount of steering wheel operation (e.g., rotation angle) performed by a remote operator. Such steering wheel operation information corresponds to another example of remote operator operation information. Specifically, if the steering wheel is in the neutral position when entering an intersection, the path image As indicating going straight is selected. On the other hand, if the steering wheel is steered in the left direction when entering an intersection, the path image Al indicating a left turn is selected. Also, if the steering wheel is steered in the right direction, the path image Ar indicating a right turn is selected.

[0056] Furthermore, intention information II is not necessarily limited to the remote operator's operational information, but may also be, for example, information about the remote operator's gaze (e.g., gaze position information) as they view the vehicle forward images displayed on monitors 231 to 233. Specifically, for example, when entering an intersection, if the remote operator's gaze is directed towards the vehicle forward image on the central monitor 231, the straight-ahead path image As is selected. On the other hand, when entering an intersection, if the remote operator's gaze is directed towards the vehicle forward image on the left monitor 232, the left-turn path image Al is selected. Also, when entering an intersection, if the remote operator's gaze is directed towards the vehicle forward image on the right monitor 233, the right-turn path image Ar is selected.

[0057] Figure 5 is a flowchart showing an example of the processing flow related to image display according to the embodiment. The processing in this flowchart starts when the remote control of the vehicle 10 is started and ends when the remote control is finished.

[0058] In step S100, the control device 26 (processor 27) determines whether or not it is currently displaying an intersection image in the intersection area where the intersection camera 31 is installed. The display of the intersection image begins when the control device 26 determines, for example, that the vehicle 10 is approaching the intersection based on vehicle position information transmitted from the vehicle 10 (i.e., immediately before entering the intersection). If the result of this determination is Yes, the process proceeds to step S102.

[0059] In step S102, the control device 26 determines the direction of travel Dvi of the vehicle 10 within the intersection image based on the imaging direction Dc of the camera 31 and the direction of travel Dv of the vehicle 10. The information of the imaging direction Dc used for this determination is obtained, for example, from the camera device 30 along with the intersection image. The direction of travel Dv used for this determination is calculated, for example, based on vehicle position information.

[0060] Next, in step S104, the control device 26 determines whether or not the turn signal control device has been operated based on the information from the sensors 29. If the result of this determination is Yes (i.e., a right turn instruction operation or a left turn instruction operation has been performed), the process proceeds to step S106; if the result of this determination is No (i.e., no operation has been performed), the process proceeds to step S108. Additionally, in the example where steering wheel operation information is used as intention information II, it is determined whether or not the steering wheel has been operated. In the example where the gaze information of the remote operator is used, it is determined whether or not the gaze of the remote operator has moved to the left or right.

[0061] In step S106, the control device 26 selects either the Ar or Al route image according to the direction of travel Dvi in ​​the image and the turn signal operation information (information indicating a right turn or left turn operation). On the other hand, in step S108, the control device 26 selects the As route image according to the direction of travel Dvi in ​​the image and the turn signal operation information (information indicating going straight).

[0062] Next, in step S110, the control device 26 superimposes the route image A selected in step S106 or S108 onto the intersection image displayed on the display device 23. Then, in step S112, the control device 26 determines, for example, based on vehicle position information, whether or not the vehicle 10 has passed through the intersection (more specifically, the intersection that was the subject of the determination in step S100).

[0063] As a result, while vehicle 10 is not passing through the intersection, the display of the intersection image with the superimposed display of the route image A, processed from step S104 onwards, continues. On the other hand, when vehicle 10 passes through the intersection, the control device 26 proceeds to step S114 and turns off the superimposed display of the route image A in conjunction with the hiding of the intersection image for that intersection.

[0064] As described above, according to this embodiment, a "route image A" showing the planned path of the vehicle 10 when passing through an intersection is displayed superimposed on the intersection image while relating it to the vehicle 10 in the intersection image. The shape of the route image A is determined by the direction of travel Dvi in ​​the image and the intention information I I and The decision is made based on the above. This allows the remote operator's intentions regarding the vehicle 10's right or left turn to be quickly reflected in the superimposed display of the path image A. As a result, the remote operator can intuitively and easily grasp the path of the vehicle 10 within the intersection image. Thus, according to this embodiment, when the vehicle 10 passes through an intersection, the visibility of the image display on the display device 23 for the remote control of the vehicle 10 can be improved for the remote operator.

[0065] Furthermore, as illustrated in Figure 4, the direction of the vehicle 10's path within the intersection image is indicated by an arrow in the path image A, making it easier for the remote operator to focus their attention on the path ahead. In other words, it makes it easier for the remote operator to recognize the direction of the path.

[0066] Furthermore, according to this embodiment, real-time intention information II can be obtained from turn signal operation information, which is the operation information of the remote operator. This is also true when steering wheel operation information is used as the operation information of the remote operator, and when the remote operator's gaze information is used instead of the operation information of the remote operator.

[0067] 3. Additional image processing (horizontal flipping) In response to the problem X already described, the control device 26 may additionally perform the following image processing IP in addition to the superimposed display of the route image A described above. Figure 6 is a diagram illustrating the additional image processing IP according to the embodiment, together with Figures 2 and 3 described above. Note that Figure 6 shows the intersection image at the same time as the intersection image shown in Figure 3(B).

[0068] According to the image processing IP, first, it is determined whether the imaging direction Dc of the camera 31 coincides with or is opposite to the vehicle's direction of travel Dv. More specifically, the direction Dc' (see Figures 2(A) and 3(A)) of the vehicle's direction of travel component of the vector representing the imaging direction Dc is compared with the direction of travel Dv (Dv1 or Dv2). The direction Dc' is calculated based on the imaging direction Dc and the direction of travel Dv.

[0069] As a result, if the imaging direction Dc (direction Dc') coincides with the direction of travel Dv, as shown in the example in Figure 2(A), the control device 26 displays the intersection image acquired from the intersection camera 31 at a predetermined interval on the display device 23 screen without horizontal reversal (see Figure 2(B)).

[0070] On the other hand, as shown in the example in Figure 3(A), when the imaging direction Dc (direction Dc') is opposite to the direction of travel Dv, the control device 26 displays the intersection image on the screen while horizontally inverting the image acquired from the camera 31 (see Figure 6). Additionally, in Figure 6, as an example of a path image A based on intention information II, a path image Ar1' related to a right turn is superimposed on the intersection image. When the intersection image is horizontally inverted by the image processing IP, the path image A is also horizontally inverted, as shown in the path image Ar1' in Figure 6.

[0071] According to the image processing IP described above, when vehicle 10 enters intersection P1 from direction Dv1 (first direction), the intersection image displayed on the screen (see Figure 2(B)) is reversed horizontally when vehicle 10 enters intersection P1 from direction Dv2 (second direction), which is opposite to direction Dv1, as shown in Figure 6.

[0072] As a result, when the direction of travel Dv is Dv1, pedestrian 2, which is displayed on the right side (front right) of vehicle 10 in the vehicle-front image showing the actual traffic situation, can be similarly displayed on the right side of the image in the intersection image, as shown in Figure 2(B). Furthermore, when the direction of travel Dv is Dv2, pedestrian 2, which is displayed on the left side (front left) of vehicle 10 in the vehicle-front image showing the actual traffic situation, can be similarly displayed on the left side of the image in the intersection image, as shown in Figure 6. In other words, regardless of whether the direction of travel Dv is Dv1 or Dv2, there can be no discrepancy between the vehicle-front image (actual situation) and the intersection image regarding which side of vehicle 10 a moving object (e.g., pedestrian 2) in the intersection area is located on. Thus, with the image processing IP, under the circumstances shown in Figure 3(A), the intersection image displayed on the display device 23 can be used like a reflector placed at an actual intersection. Furthermore, by performing image processing IP in addition to superimposing the trajectory image A, the remote operator can more quickly grasp the position (especially the left-right position) of moving objects (e.g., pedestrian 2) in the intersection area relative to the vehicle 10. [Explanation of Symbols]

[0073] 1 Remote control system, 2 Pedestrians, 10 Vehicles, 12 In-vehicle cameras, 20 Remote operator terminals, 23 Display devices, 25 Operating devices, 26 Control devices, 27 Processors, 29 Sensors, 31 Intersection cameras, 231-234 Monitors

Claims

1. A display control device for controlling the image display of a display device for remote operation of a mobile object by a remote operator, The display device includes a first screen that displays an intersection image, which is an image from a camera installed at the intersection through which the moving object passes and which captures the traffic conditions in the area of ​​the intersection. The aforementioned display control device is A sensor that acquires information about the intentions of the remote operator regarding the right or left turn of the moving object, Processor and Equipped with, The aforementioned processor, Based on the direction of travel of the moving object within the aforementioned intersection image and the intent information, the shape of the path image indicating the planned path of the moving object is determined. The aforementioned path image is displayed superimposed on the intersection image while relating it to the moving object in the intersection image. When the moving body enters the intersection from a first direction, the intersection image displayed on the first screen is reversed horizontally when the moving body enters the intersection from a second direction opposite to the first direction. Display control device.

2. The aforementioned intention information is information indicating that the turn signal control device operated by the remote operator is in the neutral position, the right-turn instruction position, or the left-turn instruction position. The display control device according to claim 1.

3. The display device includes a second screen that displays an image from a mobile camera that captures the area in front of the mobile body. The aforementioned intention information is the gaze information of the remote operator viewing the second screen. The display control device according to claim 1.

4. The aforementioned intention information is information indicating the amount of steering wheel operation performed by the remote operator. The display control device according to claim 1.

Images