Display control device and display control system

Abstract

To improve the visibility of a remote operator with respect to the image display of a display device for remote operation of a moving body.SOLUTION: A display control device is configured to control the image display of a display device for remote operation of a moving body. The display device includes one or more screens that display intersection images captured by one or more cameras installed at an intersection through which the moving body passes and imaging a traffic situation in an area of the intersection. The display control device includes a processor. The processor executes image processing of arranging the intersection images on one or more screens in different modes according to a relation between imaging directions of one or more cameras and a traveling direction of the moving body.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a display control device and a display control system for remotely controlling a moving object. [Background technology]

[0002] Patent Document 1 discloses a communication device that acquires information used for remotely controlling a mobile object. The communication device acquires information from road management cameras based on the geographical location of the mobile object. The road management cameras are installed at traffic lights or on the side of the road and capture images of the road.

[0003] Patent Document 2 discloses a traffic information providing system that captures images of road conditions and displays the captured images on an in-vehicle display device. Patent Document 3 discloses a traffic information display system equipped with an information display device mounted on a vehicle. This information display device determines the route direction of the vehicle in the received captured image from map information of a selected point and received camera parameters, and combines information indicating the route direction with the captured image and displays it on the screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-167550 [Patent Document 2] Japanese Patent Application Publication No. 09-180087 [Patent Document 3] Japanese Patent Application Laid-Open No. 2004-021514 Summary of the Invention [Problem to be solved by the invention]

[0005] To ensure the safety of a remote operator when remotely operating (remote driving, remote assistance, or remote monitoring) a moving object such as an automobile, it is conceivable to display images from a camera installed at an intersection on the screen of a display device of a remote operator terminal. When displaying such images, it is required that the remote operator, viewing the image displayed on the display device, can instantly grasp the direction of travel of the moving object in the image and easily grasp the traffic situation.

[0006] However, the technology described in Patent Document 1 is related to the acquisition of images from road management cameras, and does not disclose anything about how to best display the acquired intersection images on the display device of the remote operator terminal from the perspective of improving the visibility of the remote operator. The same is true for the technologies described in Patent Documents 2 and 3.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a display control device and a display control system that can improve the visibility of a remote operator regarding the image display on a display device for remotely controlling a mobile object. [Means for solving the problem]

[0008] A display control device according to the present disclosure is configured to control image display on a display device for remotely controlling a mobile object. The display device is installed at an intersection through which the mobile object passes, and includes one or more screens for displaying intersection images, which are images taken by one or more cameras capturing traffic conditions in the area of ​​the intersection. The display control device includes a processor. The processor executes image processing to arrange the intersection images on one or more screens in different ways depending on the relationship between the imaging direction of the one or more cameras and the traveling direction of the mobile object. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to improve the visibility of the image displayed by a remote operator on a display device for remotely operating a moving object. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of a remote control system to which a display control device according to an embodiment is applied; [Figure 2] FIG. 2 is a diagram showing an example of a screen configuration of the display device shown in FIG. [Figure 3] FIG. 1 is a diagram for explaining image processing IP applied to an intersection P1 where two intersection cameras are installed. [Figure 4] FIG. 10 is a diagram for explaining image processing IP applied to an intersection P1. [Figure 5] FIG. 10 is a diagram for explaining image processing IP applied to an intersection P2 where only one intersection camera is installed. [Figure 6] 10 is a flowchart showing an example of the flow of an image processing IP according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Elements common to the drawings will be designated by the same reference numerals, and redundant explanations will be omitted or simplified.

[0012] 1. Remote control system configuration 1 is a diagram illustrating a schematic configuration example of a remote control system 1 to which a display control device according to an embodiment is applied. The remote control system 1 includes a vehicle 10, a remote operator terminal 20, a camera device 30, and a management server 40.

[0013] 1-1.Vehicles Vehicle 10 is an example of a "mobile body" that is a target of "remote control" according to the present disclosure, and more specifically, is an automobile. Vehicle 10 includes a GNSS (Global Navigation Satellite System) receiver 11, multiple on-board cameras 12, a driving mechanism 13, a communication device 14, an antenna 15, and a control device 16.

[0014] The GNSS receiver 11 is used to acquire the position and orientation of the vehicle 10. The multiple on-board cameras 12 recognize (detect) the situation around the vehicle 10. The multiple on-board cameras 12 include, for example, cameras that capture images of the front center, front left, and front right of the vehicle 10, respectively.

[0015] The traveling device 13 includes a drive device, a braking device, and a steering device. The drive device includes, for example, at least one of an electric motor and an internal combustion engine for driving the vehicle 10. The braking device includes a brake actuator for braking the vehicle 10. The steering device includes, for example, an electric motor for steering the wheels.

[0016] The communication device 14 communicates with the outside of the vehicle 10. Specifically, the communication device 14 communicates with a remote operator terminal 20 via an antenna 15, for example.

[0017] The control device 16 is a computer that controls the vehicle 10. The control device 16 includes one or more processors and one or more storage devices. The one or more processors execute various processes related to the control of the vehicle 10. The one or more storage devices store various information required for processing by the one or more processors.

[0018] During remote operation of the vehicle 10, the control device 16 communicates with the remote operator terminal 20 via the communication device 14. The control device 16 transmits vehicle information VCL to the remote operator terminal 20. The vehicle information VCL is information necessary for remote operation by the remote operator and includes surrounding situation information of the vehicle 10. The surrounding situation information includes images captured by the on-board camera 12. The surrounding situation information may also include sounds detected by the on-board microphone. The surrounding situation information may also include object information regarding moving objects around the vehicle 10. The object information indicates the relative position and relative speed of the object with respect to the vehicle 10. The vehicle information VCL may also include vehicle state information such as the speed, acceleration, yaw rate, and steering angle of the vehicle 10, as well as vehicle position information.

[0019] Furthermore, the control device 16 receives remote operation information from the remote operator terminal 20. The remote operation information is information related to remote operation by the remote operator. For example, the remote operation information includes the amount of operation by the remote operator. The control device 16 controls the traveling device 13 in accordance with the received remote operation information, thereby performing vehicle traveling control.

[0020] 1-2.Remote operator terminal The remote operator terminal 20 is a terminal device used by the remote operator when remotely operating the vehicle 10. The "remote operation" according to the present disclosure is a concept that includes "remote driving" in which the remote operator drives a mobile object from a remote location, "remote assistance" in which the remote operator assists the driving of the mobile object from a remote location, and "remote monitoring" in which the driving of the mobile object is monitored from a remote location. In addition, when remote operation is performed as remote driving, the remote operator terminal 20 corresponds to a remote cockpit.

[0021] The remote operator terminal 20 includes a communication device 21, an antenna 22, a display device 23, a speaker 24, an operating device 25, and a control device 26.

[0022] The communication device 21 communicates (wirelessly communicates) with the vehicle 10 via the antenna 22. The communication device 21 also communicates with the management server 40.

[0023] The display device 23 displays various types of information to present the various types of information to the remote operator. The various types of information include images of the intersection area (intersection images) captured by the multiple intersection cameras 31. In other words, the intersection images are images that show the traffic conditions in the area of ​​the intersection where the vehicle 10 is entering.

[0024] 2 is a diagram showing an example of the screen configuration of the display device 23 shown in Fig. 1, and shows the display device 23 as seen from the remote operator. That is, the right side of the paper corresponds to the right side of the screen as seen from the remote operator, and the left side of the paper corresponds to the left side of the screen as seen from the remote operator.

[0025] 2, the display device 23 has five monitors 231, 232, 233, 234, and 235. The monitors 231, 232, and 233 display images of the front of the vehicle 10 (vehicle front images) captured by the multiple on-board cameras 12. More specifically, as an example, the vehicle front images include a front center image displayed by the monitor 231, a left front image displayed by the monitor 232, and a right front image displayed by the monitor 233. As shown in FIG. 2, the monitor 232 is disposed to the left of the monitor 231, and the monitor 233 is disposed to the right of the monitor 231.

[0026] Furthermore, monitors 234 and 235 display intersection images. As shown in FIG. 2, monitors 234 and 235 are arranged side by side, and are arranged, for example, above monitors 231 to 233. Specific examples of images displayed on monitors 234 and 235 will be described later with reference to FIGS. 3 to 5. In the example shown in FIG. 2, display device 23 has one monitor (monitor 231 to 235) for each image. Alternatively, the "display device" according to the present disclosure may have one monitor that displays both an image ahead of the vehicle and an image of an intersection point.

[0027] The speaker 24 outputs sound information to present the sound information to the remote operator, including the sound detected by the above-mentioned vehicle-mounted microphone.

[0028] The operation device 25 accepts input from a remote operator. For example, the operation device 25 includes remote operation members that the remote operator operates when remotely operating the vehicle 10. The remote operation members include an accelerator pedal, a brake pedal, and a steering wheel. The operation device 25 also includes switches for various other operations related to remote operation.

[0029] The control device 26 controls the remote operator terminal 20. The control device 26 includes one or more processors 27 (hereinafter simply referred to as "processors 27") and one or more storage devices 28 (hereinafter simply referred to as "storage devices 28"). The processor 27 executes various processes. For example, the processor 27 includes a CPU (Central Processing Unit). The storage device 28 stores various information required for processing by the processor 27. The storage device 28 is, for example, at least one of a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), and an SSD (Solid State Drive).

[0030] The remote operation program is a computer program executed by the processor 27. The functions of the control device 26 are realized by the processor 27 executing the remote operation program. The remote operation program is stored in the storage device 28. Alternatively, the remote operation program may be recorded on a computer-readable recording medium. The remote operation program may be provided via a network.

[0031] The control device 26 communicates with the vehicle 10 via the communication device 21. The control device 26 receives vehicle information VCL transmitted from the vehicle 10. The control device 26 presents the vehicle information VCL to the remote operator via the display device 23 and the speaker 24. For example, the control device 26 displays an image captured by the on-board camera 12 on the display device 23. The control device 26 also outputs a sound detected by the on-board microphone from the speaker 24. The remote operator can recognize the state of the vehicle 10 and the surrounding circumstances based on the vehicle information VCL.

[0032] The remote operator operates a remote control member of the operating device 25. The amount of operation of the remote control member is detected by a sensor installed on the remote control member. The control device 26 generates remote operation information that reflects the amount of operation of the remote control member by the remote operator. The control device 26 then transmits the remote operation information to the vehicle 10 via the communication device 21.

[0033] 1-3.Intersection camera A plurality of intersection cameras 31 (hereinafter also simply referred to as "cameras 31") are installed at each intersection through which the vehicle 10 passes, and capture images of the intersection area (the interior of the intersection and its vicinity). Each of the plurality of cameras 31 is configured as a camera device 30 including, for example, a camera 31 and a communication device 32. More specifically, for example, one or more cameras 31 are installed at one intersection. In the example shown in FIG. 3 described below, two cameras 31 are installed at one intersection.

[0034] Images of the intersection area (intersection images) captured by each camera 31 are transmitted to a management server 40 via a communication device 32. The management server 40 is configured to be able to communicate with each camera device 30, and manages the intersection images transmitted from each camera device 30. The management server 40 is also configured to be able to communicate with a remote operator terminal 20. When the vehicle 10 approaches a certain intersection area, the control device 26 of the remote operator terminal 20 requests the management server 40 to transmit an image of the intersection area. In response to the request, the management server 40 transmits an image of the relevant intersection area to the remote operator terminal 20.

[0035] Note that communication between the management server 40 and each camera device 30 may be either wired or wireless. The same applies to communication between the management server 40 and the remote operator terminal 20. Also, unlike the example shown in FIG. 1, the acquisition of intersection images by the remote operator terminal 20 may be performed by direct communication with each camera device 30 without going through the management server 40.

[0036] 2. Control of image display on display device In the above-described remote operation system 1, the control device 26 of the remote operator terminal 20 controls the image display on the display device 23 for remotely operating the vehicle 10 (mobile body). Therefore, the control device 26 corresponds to an example of a "display control device" according to the present disclosure, and the system including the control device 26 and the intersection camera 31 corresponds to an example of a "display control system" according to the present disclosure.

[0037] When an image from the intersection camera 31 is displayed on the display device 23 to ensure the safety of the remote operator, the display of the image is required to allow the remote operator, upon viewing the image displayed on the display device 23, to instantly grasp the direction of travel Dv of the vehicle 10 in the image and easily grasp the traffic situation.

[0038] More specifically, as illustrated in FIG. 2 , the screen of the display device 23 installed in the remote operator terminal 20 simultaneously displays a mixture of an image from the onboard camera 12 (a forward image of the vehicle) and an image from the intersection camera 31 (an intersection image). The remote operator is required to check all of these multiple images. Each time the remote operator focuses his or her gaze on an individual image, he or she recognizes the space within the image and mentally transforms the space as necessary to recognize the space. Furthermore, the orientation (image capture direction Dc) and installation location of the intersection camera 31 vary depending on the environment of each intersection. Therefore, the intersection image for each intersection is affected by the differences in the orientation and installation location of the intersection camera 31. Therefore, it is difficult to instantly determine the traveling direction Dv of the vehicle 10 in the intersection image, while also instantly determining the positional relationship between the vehicle 10 and moving objects, such as other vehicles and pedestrians, in the intersection area, and the moving direction of the moving objects (e.g., whether they are approaching the vehicle 10).

[0039] In view of the above-mentioned problems, in this embodiment, the control device 26 (processor 27) executes the following "image processing IP." According to the image processing IP, the intersection image is arranged on the screen of the display device 23 in a manner that differs depending on the relationship between the imaging direction Dc of the camera 31 and the traveling direction Dv of the vehicle 10 (moving object).

[0040] A specific example of the image processing IP will be described with reference to Figures 3 and 4. This specific example will be described using an intersection P1 where two intersection cameras 31A and 31B (first camera 31A and second camera 31B) are installed as shown in Figures 3(A) and 4(A).

[0041] The area of ​​the intersection P1 captured by the cameras 31A and 31B includes the area located within the intersection P1, as well as a first road L1 on which the vehicle 10 travels and a second road L2 that intersects with the first road L1 at the intersection P1.

[0042] More specifically, FIG. 3A shows the traffic conditions in the area of ​​intersection P1 when vehicle 10 traveling on first road L1 along traveling direction Dv1 approaches intersection P1. In this situation, as viewed from vehicle 10, first camera 31A is located on the side of the road to the right of vehicle 10. First camera 31A captures an area within intersection P1 as well as an area on second road L2 located to the left of intersection P1. The image capture area A1 of this first camera 31A includes another vehicle 2 traveling on second road L2. Note that the installation location of first camera 31A, which captures an area on second road L2 to the left of intersection P1 shown in FIG. 3A, may be, for example, inside intersection P1 or on the side of the road to the left of vehicle 10, instead of the above example.

[0043] 3(A), when viewed from vehicle 10, second camera 31B is located on the side of the road to the left of vehicle 10. Second camera 31B captures an image of the area within intersection P1 as well as the area of ​​second road L2 located to the right of intersection P1. The image capturing area A2 of second camera 31B includes pedestrian 3 crossing second road L2. Note that the installation location of second camera 31B, which captures an image of the area of ​​second road L2 to the right of intersection P1 shown in FIG. 3(A), may be, instead of the above example, inside intersection P1 or on the side of the road to the right of vehicle 10, for example.

[0044] FIG. 3(B) shows an image display pattern PT1 produced by the image processing IP according to the embodiment. The image display pattern PT1 corresponds to the display of an intersection image on the display device 23 under the above-described circumstances shown in FIG. 3(A). The two intersection images shown in FIG. 3(B) correspond to the images captured by the first and second cameras 31A and 31B, respectively, and displayed as is on the screen of the display device 23. Note that the various arrows shown in the intersection image in FIG. 3(B) are shown for explanatory reference and are not included in the intersection image. The same applies to FIGS. 4(B), 4(C), 5(A), and 5(B), which will be described later.

[0045] 3A, the directions Dc1' and Dc2' of the vehicle traveling direction components of the vectors indicating the imaging directions Dc1 and Dc2 of the first and second cameras 31A and 31B, respectively, coincide with the traveling direction Dv1. In this case, in the image processing IP, the control device 26 (processor 27) selects the image display pattern PT1.

[0046] In image display pattern PT1, as shown in FIG. 3(B), the image of first camera 31A (i.e., the intersection camera capturing an area of ​​second road L2 located on the "left side" of intersection P1 as viewed from vehicle 10) is positioned on the "left side" of the screen of display device 23. More specifically, the image of first camera 31A is displayed on the screen of left monitor 234. Meanwhile, the image of second camera 31B (i.e., the intersection camera capturing an area of ​​second road L2 located on the "right side" of intersection P1 as viewed from vehicle 10) is positioned on the "right side" of the screen of display device 23. More specifically, the image of second camera 31B is displayed on the screen of right monitor 235. In this way, according to image display pattern PT1, the image of first camera 31A is positioned to the left of the image of second camera 31B.

[0047] According to the image display pattern PT1, another vehicle 2 that is present on the left side (more specifically, on the left front) of the vehicle 10 at the actual intersection P1 and traveling in a direction away from the vehicle 10 can be displayed on the screen of the left monitor 234 as seen from the remote operator in a manner similar to how it actually appears. Therefore, the intersection image taken by the first camera 31A can be displayed in a manner similar to the image in front of the vehicle displayed on the monitors 231 and 232.

[0048] Furthermore, according to the image display pattern PT1, a pedestrian 3 who is on the right side (more specifically, on the right front) of the vehicle 10 at the actual intersection P1 and who is crossing the second road L2 in a direction approaching the vehicle 10 can be displayed on the screen of the monitor 235 on the right side as seen from the remote operator so as to look similar to how the pedestrian 3 actually looks. Therefore, the intersection image taken by the second camera 31B can be displayed so as to look the same as the vehicle-front image displayed on the monitors 231 and 233.

[0049] FIG. 4B is a diagram for explaining a problem that occurs when the image display pattern PT is not changed according to the traveling direction Dv of the vehicle 10. In FIG.

[0050] Fig. 4(A) shows the traffic situation in the area of ​​intersection P1 when vehicle 10 traveling in direction Dv2 opposite to direction Dv1 shown in Fig. 3(A) approaches intersection P1. Fig. 4(B) corresponds to a comparative example in which, under the situation shown in Fig. 4(A), the image from first camera 31A is displayed as is on the screen of left monitor 234 and the image from second camera 31B is displayed as is on the screen of right monitor 235 without any special consideration, similar to the example shown in Fig. 3(B). Note that Fig. 4(B) shows an intersection image at the same time as the intersection image shown in Fig. 3(B).

[0051] According to the image display shown in FIG. 4(B), another vehicle 2 located to the right front of the vehicle 10 at the actual intersection P1 is displayed on the left side as seen from the remote operator, as shown in FIG. 4(A). Furthermore, a pedestrian 3 located to the left front of the vehicle 10 at the actual intersection P1 is displayed on the right side as seen from the remote operator. More specifically, on the left monitors 232 and 234 (see FIG. 2) of the display device 23, an image of the left front of the vehicle 10 is displayed together with an image of the intersection in an area that is actually located to the right of the vehicle 10. Similarly, on the right monitors 233 and 235, an image of the right front of the vehicle 10 is displayed together with an image of the intersection in an area that is actually located to the left of the vehicle 10. As a result, the remote operator is confused.

[0052] Fig. 4(C) shows an image display pattern PT2 produced by the image processing IP according to the embodiment. In view of the problem described with reference to Fig. 4(B), the image processing IP selects the image display pattern PT2 for displaying an intersection image on the display device 23 under the above-described circumstances shown in Fig. 4(A). Note that Fig. 4(C) shows an intersection image at the same time as the intersection images shown in Figs. 3(B) and 4(B).

[0053] 4A, the directions Dc1' and Dc2' of the vehicle traveling direction components of the vectors indicating the imaging directions Dc1 and Dc2 of the first and second cameras 31A and 31B, respectively, are opposite to the traveling direction Dv2. Therefore, in the image processing IP, the control device 26 (processor 27) selects the image display pattern PT2.

[0054] In image display pattern PT2, as shown in FIG. 4(C), the image from first camera 31A (i.e., the intersection camera capturing an area of ​​second road L2 located on the "right side" of intersection P1 as seen from vehicle 10) is positioned on the "right side" of the screen of display device 23. More specifically, the image from first camera 31A is displayed on the screen of right monitor 235. Meanwhile, the image from second camera 31B (i.e., the intersection camera capturing an area of ​​second road L2 located on the "left side" of intersection P1 as seen from vehicle 10) is positioned on the "left side" of the screen of display device 23. More specifically, the image from second camera 31B is displayed on the screen of left monitor 234.

[0055] As described above, according to image display pattern PT2, the image from first camera 31A is positioned to the right of the image from second camera 31B. Taking into consideration that the relationship between the imaging direction Dc of camera 31 and traveling direction Dv1 and Dv2 differs, an arrangement that is left-right reversed from the arrangement of the two intersection images in image display pattern PT1 is adopted. This allows the two intersection images to be positioned so as to be consistent with the left-right of the image ahead of the vehicle displayed on display device 23.

[0056] However, with only the above-mentioned measures, the left-right direction in each intersection image will be reversed from the actual left-right direction. More specifically, the moving direction of a moving object (e.g., the other vehicle 2 shown in FIG. 4(A)) moving along the second road L2 will be reversed from the actual moving direction and from the moving direction of the other vehicle 2 displayed in the vehicle front image (right-front image). As a result, in the example of the other vehicle 2, there is a possibility that the remote operator will erroneously determine that the other vehicle 2 is approaching the vehicle 10, even though the other vehicle 2 is actually moving away from the vehicle 10.

[0057] Therefore, according to the image display pattern PT2, the images of the first and second cameras 31A and 31B are reversed left and right as shown in Fig. 4(C). As a result, even when the directions Dc1' and Dc2' of the first and second cameras 31A and 31B are opposite to the traveling direction Dv2, the intersection image can be displayed appropriately to prevent confusion of the remote operator.

[0058] Next, Figures 5(A) and 5(B) are diagrams for explaining the image processing IP applied to an intersection P2 where only one intersection camera 31 (for convenience, hereinafter referred to as camera 31C) is installed. The intersection image shown in Figure 5(A) corresponds to the image itself displayed on the screen of the display device 23 as it is captured by the camera 31C. Note that, if two screens for displaying intersection images are provided as in the example of the display device 23 shown in Figure 2, the image captured by the camera 31C is displayed on one of the two screens. Furthermore, the "display device" to which the example shown in Figure 5 is applied may have only one screen for displaying one intersection image.

[0059] 5(A) also shows the traffic conditions in the area of ​​intersection P2 when vehicle 10 traveling in a traveling direction Dv3 approaches intersection P2. In this situation, camera 31C is located on the side of the road to the right of vehicle 10 as seen from vehicle 10. Camera 31C captures images of the area within intersection P2 as well as the areas of first road L3 and second road L4 that intersect at intersection P2. The image (image capture area) of camera 31C includes pedestrian 4 crossing the crosswalk.

[0060] 5(A), the direction Dc3' of the vehicle traveling direction component of the vector indicating the imaging direction Dc3 of camera 31C coincides with the traveling direction Dv3. In this case, according to the image processing IP, the image of camera 31C is displayed on the screen as is, as shown in FIG. 5(A), without being flipped left and right.

[0061] On the other hand, when the vehicle 10 traveling in a direction Dv4 (see FIG. 5A) opposite to the direction of travel Dv3 approaches the intersection P2, the direction Dc3' of the component in the vehicle's traveling direction of the vector indicating the imaging direction Dc3 of the camera 31C faces the direction of travel Dv4. In this case, the image acquired by the camera 31C is displayed on the screen as shown in FIG. 5B by being flipped left and right according to the image processing IP.

[0062] As described above, according to the image processing IP applied to the example shown in Figure 5, even when one camera 31C is installed for one intersection P2, the intersection image is displayed on the screen in different manners depending on the relationship between the imaging direction Dc3 of the camera 31C and the traveling direction Dv.

[0063] More specifically, according to the image processing IP applied to the example shown in Figure 5, similar to the examples shown in Figures 3(B) and 4(C), the intersection image (see Figure 5(A)) displayed on the screen when the vehicle 10 enters the intersection P2 from the direction of travel Dv3 (first direction) is reversed left to right with the intersection image (see Figure 5(B)) displayed on the screen when the vehicle 10 enters the intersection P2 from the direction of travel Dv4 (second direction) opposite to the direction of travel Dv3.

[0064] As a result, when the traveling direction Dv is Dv3, the pedestrian 4 displayed on the left side (left front side) of the vehicle 10 in the vehicle-front image showing the actual traffic situation can be similarly displayed on the left side of the image in the intersection image as shown in FIG. 5(A). When the traveling direction Dv is Dv4, the pedestrian 4 displayed on the right side (right front side) of the vehicle 10 in the vehicle-front image showing the actual traffic situation can be similarly displayed on the left side of the image in the intersection image as shown in FIG. 5(B). In other words, regardless of whether the traveling direction Dv is Dv3 or Dv4, it is possible to prevent any discrepancy between the vehicle-front image (actual situation) and the intersection image regarding which side of the vehicle 10 a moving object (e.g., pedestrian 4) is located in the intersection area. Thus, according to the image processing IP, even in an example in which one camera 31C is installed for one intersection P2, the intersection image can be appropriately displayed so as to prevent confusion of the remote operator caused by differences in the traveling direction Dv relative to the imaging direction Dc3 of the camera 31C.

[0065] 6 is a flowchart showing an example of the flow of image processing IP according to the embodiment. The processing of this flowchart starts when remote control of the vehicle 10 is started, and ends when the remote control is ended.

[0066] In step S100, the control device 26 (processor 27) determines whether the vehicle 10 has approached an intersection where the intersection camera 31 is installed, for example, based on the vehicle position information transmitted from the vehicle 10. If the determination result is Yes, the process proceeds to step S102.

[0067] In step S102, the control device 26 determines whether the imaging direction Dc of the camera 31 coincides with or faces the traveling direction Dv of the vehicle 10. More specifically, the direction Dc' of the vehicle traveling direction component of the vector indicating the imaging direction Dc is compared with the traveling direction Dv. Information on the imaging direction Dc used in this determination is acquired, for example, from the camera device 30 along with an intersection image. The control device 26 then calculates the direction Dc' from the acquired imaging direction Dc. The control device 26 also calculates the traveling direction Dv based on, for example, vehicle position information.

[0068] As a result, if the imaging direction Dc (direction Dc') matches the traveling direction Dv (step S102; Yes), the process proceeds to step S104. In step S104, the control device 26 displays the intersection image acquired from the intersection camera 31 at a predetermined period on the screen of the display device 23 as is without flipping it left to right. More specifically, in an example in which two intersection images from two cameras 31 are displayed on two screens, respectively, two intersection images are displayed as in the image display pattern PT1 shown in FIG. 3(B).

[0069] On the other hand, if the imaging direction Dc (direction Dc') is opposite to the traveling direction Dv (step S102; No), the process proceeds to step S106. In step S106, the control device 26 determines whether or not there are two intersection images to be displayed.

[0070] If there are two intersection images in step S106, the process proceeds to step S108. In step S108, control device 26 displays two intersection images on the screen, as in image display pattern PT2 shown in FIG. 4(C). Specifically, the arrangement of the two intersection images on the screen of display device 23 is left-right reversed from the arrangement of the two intersection images obtained by the process of step S104. In addition, the two intersection images are left-right reversed with respect to the images acquired from camera 31.

[0071] On the other hand, if there is one intersection image to be displayed (S106; No), the process proceeds to step S110. In step S110, the control device 26 displays the intersection image on the screen while flipping the image acquired from the camera 31 left and right.

[0072] In step S112 following the processing of step S104, S108, or S110, the control device 26 determines, for example, based on vehicle position information, whether the vehicle 10 has passed through an intersection (more specifically, the intersection that was the subject of the determination in step S100).

[0073] As a result, while the vehicle 10 is not passing through the above intersection, the intersection image continues to be displayed by the processing of step S104, S108, or S110. On the other hand, once the vehicle 10 has passed the intersection, the control device 26 proceeds to step S114 and hides the intersection image for that intersection. This allows the image displayed on the display device 23 to be the minimum necessary depending on the current traveling position of the vehicle 10. In other words, the remote operator can concentrate on the image ahead of the vehicle until the vehicle 10 approaches the next intersection.

[0074] As described above, according to the image processing IP of this embodiment, the intersection image is arranged on the screen of the display device 23 in different modes depending on the relationship between the imaging direction Dc of the camera 31 and the traveling direction Dv of the vehicle 10 (moving object). This improves the remote operator's visibility of the image display on the display device 23. More specifically, based on the intersection image displayed on the display device 23, the remote operator can intuitively determine the position and moving direction of moving objects present in the intersection area relative to the vehicle 10. [Explanation of symbols]

[0075] 1 Remote operation system, 2 Other vehicles, 3, 4 Pedestrians, 10 Vehicle, 12 In-vehicle camera, 20 Remote operator terminal, 23 Display device, 26 Control device, 27 Processor, 31 Intersection camera, 231-235 Monitor

Claims

1. A display control device that controls image display on a display device for remotely controlling a moving object, the display device includes one or more screens that are installed at an intersection through which the mobile object passes and that display intersection images that are images taken by one or more cameras that capture traffic conditions in the area of ​​the intersection, the display control device includes a processor; The processor executes image processing to arrange the intersection image on one or more screens in different modes depending on the relationship between the imaging direction of the one or more cameras and the traveling direction of the moving object. Display control device.

2. In the image processing, the processor horizontally flips the intersection image displayed on the one or more screens when the moving object enters the intersection from a first direction, with respect to the intersection image displayed on the one or more screens when the moving object enters the intersection from a second direction opposite to the first direction. The display control device according to claim 1 .

3. the one or more cameras include a first and a second camera; the area of ​​the intersection captured by the first and second cameras includes a first road on which the moving object travels and a second road intersecting the first road at the intersection, When the directions of the vehicle traveling direction components of the vectors indicating the imaging directions of the first and second cameras coincide with the traveling direction, in the image processing, the processor positions the image of the first camera capturing an area of ​​the second road located to the left of the intersection when viewed from the moving body to the left of the image of the second camera capturing an area of ​​the second road located to the right of the intersection. The display control device according to claim 1 .

4. the one or more cameras include a first and a second camera; the area of ​​the intersection captured by the first and second cameras includes a first road on which the moving object travels and a second road intersecting the first road at the intersection, When the directions of the vehicle traveling direction components of the vectors indicating the imaging directions of the first and second cameras are opposite to the traveling direction, in the image processing, the processor: The image of the first camera capturing an area of ​​the second road located to the right of the intersection when viewed from the moving body is positioned to the right of the image of the second camera capturing an area of ​​the second road located to the left of the intersection, and the images of the first and second cameras are flipped left and right. The display control device according to claim 1 .

5. A display control system comprising the display control device according to claim 1 and the one or more cameras, The processor hides the intersection image when the moving object passes through the intersection. Display control system.

Images