Display system

Abstract

To facilitate safety confirmation by a remote operator at an intersection while using image information of an intersection camera.SOLUTION: A display system comprises: one or more moving body cameras that capture images in front of a moving body; one or more intersection cameras that are installed at an intersection through which the moving body passes and that image a traffic situation in an area of the intersection; a display device including one or more screens that display one or more moving body camera images of moving body cameras as images for remote operation of the moving body; and one or more processors. The one or more processors execute: determination processing of determining whether or not a moving object is included in an intersection camera image of one or more intersection cameras by analyzing the intersection camera image; and highlighting display processing of applying, to the moving body camera images, highlighting indicating whether the moving object is present in a left or right direction with respect to a traveling direction of the moving body, when the moving object is included in the intersection camera image.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a display system for displaying images for remote control of a moving object.

Background Art

[0002] Patent Document 1 discloses a remote support control device for remotely supporting the driving of an autonomous vehicle. This control device receives vehicle information including at least position information indicating a driving position and a surrounding image of the driving position from the autonomous vehicle. Then, the control device reads accident pattern data related to the above vehicle information from an accident pattern database in which a plurality of past accident cases and traffic situations are associated and confirmed, and generates driving support assistance information by superimposing attention-calling information on the above surrounding image. The generated driving support assistance information is provided to the autonomous vehicle.

[0003] Also, Patent Document 2 discloses a remote control system for an operator to remotely control a vehicle to be operated. This system includes a display device for displaying the vehicle to be operated and obstacles around the vehicle to be operated. The display device displays obstacles sensed by the vehicle to be operated surrounded by a solid-line square frame.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, to supplement safety checks at intersections performed by remote operators who remotely control (remotely drive, remotely assist, or remotely monitor) mobile vehicles and other vehicles, it is conceivable to display images from intersection cameras installed at the intersection on the screen of the remote operator's terminal display device. However, the installation direction and field of view of intersection cameras vary from intersection to intersection. Therefore, even if images from the intersection cameras are displayed on the display device, it may be difficult for the remote operator to instantly grasp the direction of travel of the mobile vehicle within the image, potentially delaying safety checks.

[0006] This disclosure was made in view of the above-mentioned issues, and aims to provide a display system that facilitates safety checks by remote operators at intersections using image information from intersection cameras. [Means for solving the problem]

[0007] The display system according to this disclosure comprises one or more mobile cameras, one or more intersection cameras, a display device, and one or more processors. The one or more mobile cameras capture images of the area in front of the mobile vehicle. The one or more intersection cameras are installed at intersections through which the mobile vehicle passes and capture images of the traffic conditions in the intersection area. The display device includes one or more screens that display the mobile camera images of the one or more mobile cameras as images for remote control of the mobile vehicle. The one or more processors perform a determination process that determines whether or not a moving object is included in the intersection camera images of the one or more intersection cameras by analyzing the intersection camera images, and a highlighting process that, if a moving object is included in the intersection camera images, highlights the mobile camera images to indicate whether the moving object is located to the left or right of the direction of travel of the mobile vehicle. [Effects of the Invention]

[0008] According to this disclosure, it will become easier for remote operators to perform safety checks at intersections by utilizing image information from intersection cameras. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram schematically shows an example configuration of a remote control system to which the display system according to the embodiment is applied. [Figure 2] This figure shows an example of an intersection and an image from an intersection camera. [Figure 3] This figure shows examples A and B of the emphasis according to the embodiment. [Figure 4] This flowchart shows an example of the processing flow for image display in an embodiment that is performed through collaboration between the remote cockpit and the infrastructure. [Figure 5] This flowchart shows an example of the processing flow related to image display in an embodiment executed by a remote operator terminal. [Figure 6] This figure shows an example of an intersection camera image displayed as reference information. [Modes for carrying out the invention]

[0010] 1. Configuration of the remote control system Figure 1 is a schematic diagram showing an example of the configuration of a remote control system 1 to which the display system according to the embodiment is applied. The remote control system 1 comprises a vehicle 10, a remote operator terminal 20, a camera device 30, and a management server 40.

[0011] 1-1. Vehicles Vehicle 10 is an example of a "mobile object" that is the subject of "remote control" as described in this disclosure, and more specifically, is an automobile. Vehicle 10 comprises a GNSS (Global Navigation Satellite System) receiver 11, one or more on-board cameras (mobile cameras) 12, a driving device 13, a communication device 14, an antenna 15, and a control device 16.

[0012] The GNSS receiver 11 is used to acquire the position and orientation of the vehicle 10. One or more onboard cameras 12 recognize (detect) the surrounding environment of the vehicle 10. One or more onboard cameras 12 capture images of the area in front of the vehicle 10. For example, one or more onboard cameras 12 include three cameras that capture images of the front center, left front, and right front of the vehicle 10, respectively.

[0013] The running gear 13 includes a drive system, a braking system, and a steering system. The drive system includes, for example, at least one of an electric motor and an internal combustion engine for driving the vehicle 10. The braking system includes brake actuators for braking the vehicle 10. The steering system includes, for example, an electric motor for steering the wheels.

[0014] The communication device 14 communicates with the outside of the vehicle 10. body Specifically, the communication device 14 communicates with the remote operator terminal 20, for example, via the antenna 15.

[0015] The control device 16 is a computer that controls the vehicle 10. The control device 16 includes one or more processors and one or more storage devices. One or more processors perform various processes related to the control of the vehicle 10. One or more storage devices store various information necessary for the processing performed by the one or more processors.

[0016] During the remote control of vehicle 10, the control device 16 communicates with the remote operator terminal 20 via the communication device 14. The control device 16 transmits vehicle information VCL to the remote operator terminal 20. Vehicle information VCL is information necessary for remote control by the remote operator and includes information about the surrounding conditions of vehicle 10. The surrounding conditions information includes images captured by the onboard camera 12. The surrounding conditions information may also include object information regarding moving objects around vehicle 10.

[0017] Furthermore, the control device 16 receives remote operation information from the remote operator terminal 20. The remote operation information is information regarding remote operation by the remote operator. For example, the remote operation information includes the operation amount by the remote operator. The control device 16 performs vehicle travel control by controlling the travel device 13 according to the received remote operation information.

[0018] 1-2. Remote Operator Terminal The remote operator terminal 20 is a terminal device used when the remote operator remotely operates the vehicle 10. The "remote operation" according to the present disclosure is a concept including "remote driving" in which a remote operator drives a moving body from a remote location, "remote support" in which the operation of the moving body is supported from a remote location, and "remote monitoring" in which the operation of the moving body is monitored from a remote location. In addition, when the remote operation is performed as remote driving, the remote operator terminal 20 corresponds to a remote cockpit.

[0019] The remote operator terminal 20 includes a communication device 21, an antenna 22, a display device 23, a speaker 24, an operation device 25, and a control device 26.

[0020] The communication device 21 communicates (wireless communication) with the vehicle 10 via the antenna 22. Also, the communication device 21 communicates with the management server 40.

[0021] The display device 23 presents various information to the remote operator by displaying the various information. The various information includes, as an image for remote operation of the vehicle 10, an image of the front of the vehicle 10 (vehicle front image) captured by a plurality of in-vehicle cameras 12. include Note that the "vehicle front image" corresponds to an example of the "moving body camera image" according to the present disclosure.

[0022] Figure 1 shows an example of the screen configuration of the display device 23 as seen from the remote operator. That is, the right side of the page corresponds to the right side of the screen as seen from the remote operator, and the left side of the page corresponds to the left side of the screen as seen from the remote operator. In this example, the display device 23 has three monitors 231, 232, and 233. Monitors 231, 232, and 233 display images of the area in front of the vehicle. More specifically, as an example, the images of the area in front of the vehicle include an image of the front center displayed by monitor 231, an image of the left front displayed by monitor 232, and an image of the right front displayed by monitor 233. As shown in Figure 1, monitor 232 is located to the left of monitor 231, and monitor 233 is located to the right of monitor 231.

[0023] In the example shown in Figure 1, the display device 23 has one monitor (monitor 231, 232, or 233) for each image. However, instead of the example shown in Figure 1, the “display device” according to this disclosure may have one monitor that displays multiple vehicle forward-facing images.

[0024] Speaker 24 presents sound information to the remote operator by outputting sound information. The sound information includes, for example, sounds detected by the vehicle's microphone.

[0025] The control device 25 receives input from a remote operator. For example, the control device 25 includes remote control components that the remote operator operates when remotely controlling the vehicle 10. These remote control components include an accelerator pedal, a brake pedal, and a steering wheel. The control device 25 also includes switches for various other operations related to remote control.

[0026] The control device 26 controls the remote operator terminal 20. The control device 26 includes one or more processors 27 (hereinafter simply referred to as "processor 27") and one or more storage devices 28 (hereinafter simply referred to as "storage devices 28"). The processor 27 performs various processes related to remote operation, including control of the display of images on the display device 23. The storage devices 28 store various information necessary for the processing performed by the processor 27.

[0027] The control device 26 communicates with the vehicle 10 via the communication device 21. The control device 26 receives vehicle information VCL transmitted from the vehicle 10. The control device 26 presents the vehicle information VCL to the remote operator via the display device 23 and speaker 24. For example, the control device 26 displays images captured by the on-board camera 12 on the display device 23. The control device 26 also outputs sounds detected by the on-board microphone through the speaker 24. Based on the vehicle information VCL, the remote operator can recognize the status of the vehicle 10 and the surrounding conditions.

[0028] The remote operator operates the remote control component of the operating device 25. The amount of operation of the remote control component is detected by a sensor installed on the remote control component. The control device 26 generates remote control information that reflects the amount of operation of the remote control component by the remote operator. The control device 26 then transmits the remote control information to the vehicle 10 via the communication device 21.

[0029] 1-3. Intersection Camera The intersection camera 31 (hereinafter also simply referred to as "camera 31") is installed at each intersection through which the vehicle 10 passes, and captures images of the intersection area, which is the interior of the intersection and its vicinity. Camera 31 is configured as a camera device 30, which includes, for example, camera 31, a communication device 32, and a control device 33. In the example shown in Figure 2 described later, one camera 31 is installed for each intersection.

[0030] Images of the intersection area captured by each camera 31 (intersection camera images) are transmitted to the management server 40 via the communication device 32. The management server 40 is configured to communicate with each camera device 30 and transmits the intersection images from each camera device 30. camera The system manages images. Furthermore, the management server 40 is configured to communicate with the remote operator terminal 20.

[0031] The control device 33 includes one or more processors 34 (hereinafter simply referred to as "processor 34") and one or more storage devices 35 (hereinafter simply referred to as "storage devices 35"). The processor 34 performs various processing related to the intersection camera image. The storage devices 35 store various information necessary for processing by the processor 34.

[0032] Furthermore, communication between the management server 40 and each camera device 30 may be via wired or wireless communication. The same applies to communication between the management server 40 and the remote operator terminal 20. Also, when the remote operator terminal 20 communicates with the camera devices 30, unlike the example shown in Figure 1, the remote operator terminal 20 may communicate directly with each camera device 30 without going through the management server 40.

[0033] 2. Controlling image display on the display device In this embodiment, the following "determination process PR1" and "highlighting process PR2" are executed in order to facilitate safety checks by a remote operator at the intersection while utilizing the image information from the intersection camera 31.

[0034] In the determination process PR1, the camera image of the intersection that vehicle 10 is scheduled to pass through is analyzed in real time to determine whether or not a moving object is included in the camera image. Then, the highlighting process PR2 is executed if it is determined that a moving object is included in the camera image of the intersection. In the highlighting process PR2, based on the results of the above analysis, highlighting is applied to the camera image of the moving object to indicate whether it is to the left or right of the direction of travel Dv of vehicle 10. A "moving object" can be, for example, another vehicle (such as a car), a pedestrian, or a bicycle. car That is the case.

[0035] 2-1. Highlighting example A In highlighting example A, the "highlighting" mentioned above is a warning mark indicating whether the moving object is located to the left or right of the vehicle 10's direction of travel Dv, and is added to the front-center image (moving object camera image) displayed on the monitor 231 screen (front-center screen).

[0036] Highlighting Example A and Highlighting Example B, described later, will be explained using the intersection CR1 shown in Figure 2(A) as an example. In Figure 2(A), at intersection CR1, the first road R1 on which vehicle 10 is traveling connects to the second road R2. More specifically, Figure 2(A) shows the traffic situation in the area of ​​intersection CR1 when vehicle 10, traveling on the first road R1 along the direction of travel Dv, is approaching intersection CR1. As an example, camera 31 is capturing images of the area within intersection CR1 as well as the area of ​​the second road R2 located to the right of intersection CR1 with respect to the direction of travel Dv. Another vehicle 2 traveling on the second road R2 is present in the area captured by camera 31. Figure 2(A) also shows the pedestrian crossing CW1 and stop lines SL1 and SL2 provided on the second road R2.

[0037] Figure 2(B) shows an intersection camera image captured by the camera 31 shown in Figure 2(A). The intersection camera image shows the traffic conditions in the area of ​​the intersection into which vehicle 10 is entering. The intersection camera image shown in Figure 2(B) includes another vehicle 2, which is an example of a moving object. According to the determination process PR1 of this embodiment, analysis of the intersection camera image by camera 31 determines that another vehicle 2 is included in the intersection camera image. Then, according to the analysis of the intersection camera image by determination process PR1, if another vehicle 2 is included in the intersection camera image, the direction in which the other vehicle 2 is located relative to the direction of travel Dv of vehicle 10 is identified. In the example of the intersection camera image shown in Figure 2(B), the other vehicle 2 is identified as being on the right side relative to the direction of travel Dv.

[0038] Figure 3(A) shows highlighting example A1, which is one of the highlighting examples A. Figure 3(A) shows the vehicle front images on monitors 231-233 when vehicle 10 approaches intersection CR1 shown in Figure 2(A). In highlighting example A1, the highlighting process PR2 of this embodiment adds a warning mark 50 to the front center image displayed on monitor 231. As shown in Figure 3(A), the warning mark 50 indicates whether the other vehicle 2 is located to the left or right of vehicle 10's direction of travel Dv, and is positioned to the right of the front center image as seen by the remote operator. Thus, according to highlighting example A, information about a moving object (other vehicle 2) included in the intersection camera image is displayed to warn of the moving object in the front center image where the moving object is not displayed. Reference numeral 6 indicates a building on the side of the road.

[0039] More specifically, the warning mark 50 is, for example, a mark (still image) that represents the detection of another vehicle 2 located near vehicle 10. Such a warning mark 50 is preferably used when the type of moving object can be identified by analysis by the judgment process PR1. Additionally, in cases where a moving object such as another vehicle 2 is located to the left of the direction of travel Dv, a warning mark that is a horizontally inverted version of the warning mark 50 is placed to the left of the front-center image displayed on the monitor 231.

[0040] Figure 3(B) shows another example of highlighting example A, namely highlighting example A2. This highlighting example A2 differs from highlighting example A1 in its use of a warning mark for highlighting. Specifically, in highlighting example A2, an animated display is used for the warning mark 52. The warning mark 52 is, for example, an arrow indicating the direction in which another vehicle 2 is located, as shown in Figure 3(B). This arrow is displayed as an animated movement in the direction in which the other vehicle 2 is located. In addition, in examples where a moving object such as another vehicle 2 is located to the left of the direction of travel Dv, a warning mark that is a horizontally inverted version of the warning mark 52 is placed to the left of the front-center image displayed on the monitor 231.

[0041] In this example, highlighting example A (A1 and A2) is applied to a display device 23 having three screens: a front-center screen, a left-front screen, and a right-front screen. However, highlighting example A is not limited to the display device 23 having three screens, but can be applied to any display device having at least a front-center screen.

[0042] 2-2. Highlighting example B Figure 3(C) shows one example of highlighting example B. In highlighting example B, the highlighting process PR2 of this embodiment highlights the left front screen or the right front screen corresponding to the direction in which a moving object is located, relative to the vehicle 10's direction of travel Dv. Specifically, in the example of the intersection camera image shown in Figure 2(B), the right front screen of monitor 233 corresponding to the right direction where the other vehicle 2 is located is highlighted. This highlighting is performed, for example, by surrounding the right front screen with a red frame 54, as shown in Figure 3(C).

[0043] In addition, in cases where another moving object such as vehicle 2 is located to the left of the direction of travel Dv, the left front screen is surrounded by a red frame, contrary to the highlighting example B shown in Figure 3(C). Furthermore, the highlighting in highlighting example B may be done using a frame of any color other than red, as long as it attracts the attention of the remote operator. Alternatively, the highlighting may be done, for example, by flashing or blinking the left front screen or the right front screen, which is the screen on the side where the moving object is located, a predetermined number of times.

[0044] Furthermore, in highlighting example B, unlike highlighting example A, the highlighting is not added to the front-center screen, which is the primary screen for remote control. Therefore, it becomes possible to perform highlighting while preventing interference with the image in front of the vehicle.

[0045] 2-3. Examples of image display processing In the remote control system 1 shown in Figure 1, a configuration including an in-vehicle camera 12, an intersection camera 31, a display device 23, and processors 27 and 34 corresponds to an example of the "display system" according to this disclosure. In a display system configured in this way, the judgment process PR1 and the highlighting process PR2 described above may be executed by the cooperation of the processor 27 of the remote operator terminal 20 (remote operator terminal side) and the processor 34 of the camera device 30 (infrastructure side) (see Figures 4(A) and 4(B)). Alternatively, the judgment process PR1 and the highlighting process PR2 may be executed by the processor 27 (remote operator terminal side) (see Figure 5).

[0046] 2-3-1. Example A of execution through collaboration between the remote operator terminal and the infrastructure side. Figures 4(A) and 4(B) are flowcharts illustrating examples of the image display processing flow in an embodiment that is executed collaboratively by the remote operator terminal and the infrastructure side, respectively. The processing in the flowchart on the remote operator terminal side starts when the remote operation of the vehicle 10 begins and ends when the remote operation ends. The processing in the flowchart on the infrastructure side is executed continuously for each camera 31 while each camera 31 at each intersection is in operation.

[0047] In step S100 of Figure 4(A), the processor 27 determines, for example, whether vehicle 10 is approaching the intersection where the intersection camera 31 is installed, based on vehicle position information transmitted from vehicle 10. If the result of this determination is Yes, the process proceeds to step S102.

[0048] In step S102, the processor 27 performs a process to determine the orientation of the vehicle 10 (the vehicle itself) relative to the position of the camera 31 installed at the intersection the vehicle 10 is approaching. Specifically, the processor 27 obtains information on the imaging direction Dc of the camera 31 from the camera device 30, which includes the camera 31. The processor 27 also calculates the direction of travel Dv based on the vehicle position information. Based on this information on the imaging direction Dc and the direction of travel Dv, the processor 27 determines the orientation of the vehicle 10 relative to the installation position of the camera 31 and obtains this information as the orientation of the vehicle itself.

[0049] Next, in step S104, the processor 27 controls the communication device 21 to transmit the vehicle orientation information acquired in step S102, along with the vehicle position information, to the camera 31. Then, in step S106, the processor 27 determines whether or not it has received motion information of the intersection from the camera 31. This motion information is the information transmitted to the remote operator terminal 20 by the processing in step S210 of Figure 4(B).

[0050] In Figure 4(B), in step S200, the processor 34 acquires an intersection camera image from the camera 31. Then, in step S202, the processor 34 analyzes the acquired intersection camera image. Such real-time analysis is performed using a method, for example, as follows, to detect moving objects in the intersection camera image.

[0051] In other words, the analysis is performed, for example, by motion detection. Motion detection allows for the detection of moving objects from the difference between temporally consecutive intersection camera images. Alternatively, the analysis may be performed by image processing using machine learning. Using machine learning makes it possible to identify and classify moving objects such as cars, pedestrians, and bicycles. As a result, it is possible to switch warning marks according to the type of moving object, for example, as in the warning mark 50 described above.

[0052] More specifically, the analysis of the intersection camera image in step S202 may be performed while adjusting, for example, the size and speed of the moving object to be detected, as well as the detection range of the moving object, as follows: That is, moving objects smaller than a predetermined size may be excluded from detection. This can reduce false detections of moving objects. Also, moving objects with a speed below a predetermined value may be excluded from detection. This can also reduce false detections of moving objects.

[0053] Furthermore, the detection range for moving objects may be adjusted as follows, for example. That is, in order to reduce false detections and computational load, the range within the intersection camera image that is subject to detection of moving objects may be changed (adjusted) for each camera 31 according to at least one of the camera's field of view and installation environment. Specifically, in the case of a camera 31 with a field of view that includes a vehicle 10 (own vehicle) in the intersection camera image, the detection range may be adjusted so that the own vehicle and the road on which the vehicle is traveling are excluded. In addition, the detection range may be specified for each camera 31 in order to avoid false detections caused by the presence of wind-swaying objects such as trees, and false detections caused by flickering fluorescent lights. Furthermore, the detection range may be determined based on the vehicle 10's direction of travel Dv information. For example, in the case of an intersection camera image of a camera 31 with a wide field of view, areas that are unnecessary for analysis, such as the area behind the vehicle 10, may be excluded from the detection target.

[0054] Next, in step S204, the processor 34 internally stores the motion information obtained based on the analysis results of step S202. That is, the motion information is stored in the storage device 35.

[0055] Next, in step S206, the processor 34 determines whether or not information regarding the vehicle's position and orientation has been transmitted from the remote operator terminal 20. If no information regarding the vehicle's position and orientation has been transmitted, the processor 34 repeatedly executes the processes from step S200 onward. On the other hand, if such information has been transmitted, the process proceeds to step S208.

[0056] In step S208, the processor 34 performs a process to determine whether or not there is a moving object to be detected in the highlighting of this embodiment, based on the position and orientation information of the vehicle received from the remote operator terminal 20. Specifically, the processor 34 first determines whether or not a moving object is included in the moving object information held internally in step S204. If a moving object is included in the moving object information, the processor 34 determines whether or not the moving object is a target for detection, for example, based on the direction of movement of the moving object. More specifically, the processor 34 may determine whether the moving object is approaching or moving away from the vehicle 10 based on the position and orientation information of the vehicle transmitted from the remote operator terminal 20. The processor 34 may then determine that the moving object is approaching the vehicle 10, and exclude it from detection if it determines that the moving object is moving away from the vehicle 10. This makes it possible to appropriately select moving objects to be highlighted to alert the remote operator.

[0057] Next, in step S210, the processor 34 controls the communication device 32 to transmit the motion information identified by the processing in step S208 to the remote operator terminal 20. Specifically, the processor 34 transmits the motion information identified upon receiving information on the vehicle's position and orientation from the remote operator terminal 20 (S206; Yes). The motion information transmitted in this manner includes, for example, a flag indicating the presence or absence of a motion, as well as the motion's position and direction of movement. Furthermore, if the type of motion (e.g., another vehicle, pedestrian, or bicycle) can be identified by using the machine learning described above, the transmitted motion information may also include the type of motion.

[0058] In Figure 4(A), if the result of the determination in step S106 is Yes, that is, if the motion information transmitted by the processing in step S210 has been received, the process proceeds to step S108. In step S108, the processor 27 determines whether or not a motion object is included in the intersection camera image based on the received motion information. If the result of this determination is Yes, the process proceeds to step S110; if the result of this determination is No, the process proceeds to step S116.

[0059] In step S110, the processor 27 determines, based on the received motion information, whether the motion is located to the left of the vehicle 10's direction of travel Dv (in other words, the vehicle's path). If the result of this determination is Yes, that is, if the motion is located to the left, the process proceeds to step S112. In step S112, the processor 27 displays a warning mark (in other words, a warning alert) positioned to the left of the front center image displayed on the monitor 231. This warning mark is, for example, a horizontally inverted version of warning mark 50 or 52. The process then proceeds to step S116.

[0060] On the other hand, if the result of the determination in step S110 is No, that is, if the moving object is located to the right, the process proceeds to step S114. In step S114, the processor 27 displays a warning mark positioned to the right of the front-center image displayed on the monitor 231. This warning mark is, for example, warning mark 50 or 52. The process then proceeds to step S116.

[0061] Although the process in steps S112 and S114 described above uses highlighting example A, highlighting example B may also be used.

[0062] In step S116, the processor 27 determines, for example, based on vehicle position information, whether or not vehicle 10 has passed through the intersection that was the subject of the determination in step S100. As long as vehicle 10 has not passed through the intersection, the processor 27 repeatedly executes the processes from step S106 onwards. On the other hand, once vehicle 10 has passed through the intersection, the processor 27 proceeds to step S118 and removes the warning mark displayed by the processing in step S112 or S114.

[0063] 2-3-2. Example of execution by the remote operator terminal B Figure 5 is a flowchart showing an example of the processing flow related to image display in an embodiment executed by the remote operator terminal. The differences between the processing in this flowchart and the flowchart shown in Figure 4(A) will be explained below.

[0064] In Figure 5, after step S102, the process proceeds to step S300. In step S300, the processor 27 of the remote operator terminal 20 executes the process of acquiring an intersection camera image from the camera 31 installed at the intersection that was the subject of the determination in step S100.

[0065] Next, in step S302, the processor 27 performs a process to analyze the intersection camera image acquired in step S300. This process in step S302 is the same as the processes performed in steps S202 and S208 in Figure 4(B). That is, the processor 27 analyzes the intersection camera image received from the camera device 30 and uses the vehicle's position and orientation information to detect the presence or absence of moving objects on the vehicle's planned path. After step S302, the process proceeds to step S108.

[0066] In Figure 5, if the result of step S116 is No, the processor 27 repeatedly executes the processing from S300 onward.

[0067] In addition, Execution Example A, which involves collaboration between the remote operator terminal and the infrastructure, has the following advantages compared to Execution Example B, which involves only the remote operator terminal. Specifically, according to Execution Example A, the only information transmitted from the infrastructure to the remote operator terminal is motion information such as a flag indicating the presence or absence of motion (see step S210), eliminating the need to transmit intersection camera images to the remote operator terminal 20. Therefore, according to Execution Example A, the amount of communication can be reduced compared to Execution Example B. Furthermore, since the transmission of intersection camera images is unnecessary, the impact of communication delay can also be reduced.

[0068] In the flowcharts shown in Figures 4(A) and 4(B), the processes in steps S202, S208, and S108 correspond to an example of "Judgment Process PR1" related to this disclosure, and the processes in steps S110-S114 correspond to an example of "Highlighting Process PR2" related to this disclosure. On the other hand, in the flowchart shown in Figure 5, the processes in steps S302 and S108 correspond to an example of "Judgment Process PR1" related to this disclosure, and the processes in steps S110-S114 correspond to an example of "Highlighting Process PR2" related to this disclosure.

[0069] As described above, according to this embodiment, a determination process PR1 is executed to determine whether or not a moving object is included in the intersection camera image by analyzing the intersection camera image. If a moving object is included in the intersection camera image, a highlighting process PR2 is executed to highlight the vehicle front image to indicate whether the moving object is located to the left or right of the vehicle 10's direction of travel Dv. In other words, the moving object information based on the analysis results of the intersection camera image is displayed as a highlight on the vehicle front image. Therefore, without displaying the intersection camera image itself on the display device 23, the remote operator can be made aware that there is a moving object that could obstruct the vehicle 10's movement to the right or left of the intersection ahead of the vehicle 10, based on the moving object information from the intersection camera image. Thus, according to this embodiment, safety checks by a remote operator at an intersection can be easily performed using the image information of the intersection camera 31.

[0070] In addition, the remote operator terminal 20 does not display the intersection camera image itself acquired by the intersection camera 31. By narrowing the display target on the display device 23 in this way, the remote operator only needs to check the main screens for remote operation, namely the front center screen, left front screen, and right front screen that display the image in front of the vehicle. In other words, it is possible to prevent excessive information from being provided to the remote operator. Therefore, in the example of remote driving, the remote operator can concentrate on remotely driving the vehicle 10. Furthermore, the attention-grabbing by highlighting in this embodiment reduces the number of points that the remote operator needs to check. This leads to a reduction in the remote operator's fatigue.

[0071] 3. Example of displaying intersection camera images as reference. In the embodiment described above, the intersection camera image itself is not displayed on the display device 23. However, the intersection camera image may be displayed on the display device 23 as reference information. For example, the intersection camera image may be displayed on a monitor positioned above the monitors 231-233 shown in Figure 1. Furthermore, the display / hide function of such intersection camera images may be selectable by a remote operator.

[0072] Figures 6(A) and 6(B) show examples of intersection camera images displayed as reference information. In the intersection camera image shown in Figure 6(A), another vehicle 3 approaching intersection CR2 is detected as a moving object. In the intersection camera image shown in Figure 6(B), a pedestrian 4 crossing the crosswalk CW2 and another vehicle 5 traveling on a road away from intersection CR2 are detected as moving objects. To make these detected moving objects easier to understand, rectangular frames 60, 62, and 64 surrounding the other vehicle 3, pedestrian 4, and other vehicle 5, respectively, may be added to the intersection camera image. Furthermore, if the type of each moving object can be identified using the machine learning described above, a mark indicating the type of moving object may be added, such as the pedestrian mark 66 in the example of pedestrian 4. [Explanation of Symbols]

[0073] 1 Remote control system, 2, 3, 5 Other vehicles, 4 Pedestrians, 10 Vehicles, 12 On-board cameras, 20 Remote operator terminals, 23 Display devices, 27 Processors, 31 Intersection cameras, 34 Processors, 50, 52 Warning signs, 54 Frames, 231-233 Monitors

Claims

1. One or more mobile camera systems, including three mobile camera systems that capture images of the front center, left front, and right front of the moving object, The camera device is installed at an intersection through which the aforementioned moving object passes, captures images of the traffic conditions in the area of ​​the intersection, and includes one or more intersection cameras, A display device including one or more screens that display mobile camera images of one or more mobile camera cameras as images for remote control of the mobile body, One or more first processors included in the remote operator terminal, One or more second processors included in the camera device, Equipped with, The one or more screens include a front-center screen, a left-front screen, and a right-front screen, which display the front-center, left-front, and right-front mobile camera images, respectively. The one or more first processors and the one or more second processors cooperate to perform a determination process. The aforementioned determination process is, The one or more second processors analyze the intersection camera images of the one or more intersection cameras, and based on the analysis results of the intersection camera images and the position and orientation information of the moving object from the remote operator terminal, they identify a flag indicating the presence or absence of a moving object in the intersection camera image, as well as moving object information including the position and direction of movement of the moving object, and transmit the identified moving object information to the remote operator terminal. The one or more first processors determine whether or not the moving object is included in the intersection camera image based on the moving object information from the one or more second processors, Includes, If the intersection camera image contains the moving object, the one or more first processors perform a highlighting process on the moving object camera image to highlight whether the moving object is located in the left or right direction relative to the direction of travel of the moving object. In the highlighting process, the one or more first processors apply the highlighting to the left front screen or the right front screen corresponding to the direction in which the moving body is located, among the left and right directions with respect to the direction of travel. Display system.

2. In the determination process, the one or more second processors detect the moving object if the moving object is approaching the moving body, and exclude the moving object from detection if the moving object is moving away from the moving body. The display system according to claim 1.

3. In the determination process, the one or more second processors change the range within the intersection camera image that is subject to the determination of whether or not the moving object is included in the intersection camera image, according to at least one of the field of view of the one or more intersection cameras and the installation environment. The display system according to claim 1.

Images