Remote operator terminal, remote operation support method, and remote operation system

Abstract

To provide technology facilitating safety confirmation when a remote operator turns a mobile body to right or left.SOLUTION: A remote operator terminal is used by a remote operator who performs remote operation on a mobile body. The remote operator terminal includes a display device for displaying a rear-side image taken by a rear-side camera mounted on the mobile body. The display device also displays the mobile body's trajectory behind the mobile body superimposed on the rear side image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to remote operation of a mobile object performed by a remote operator. [Background technology]

[0002] Patent Document 1 discloses an image display method when remotely controlling a work machine. A front camera and a rear camera are attached to the work machine. Images captured by the front camera and the rear camera are displayed on a display on the remote control side. At this time, an image captured by either the front camera or the rear camera is displayed larger than an image captured by the other camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-009823 A Summary of the Invention [Problem to be solved by the invention]

[0004] Consider the remote control of a mobile object by a remote operator. When making the mobile object turn right or left, the remote operator checks the situation around the mobile object to ensure that the mobile object does not hit other objects.

[0005] One object of the present disclosure is to provide a technology that enables a remote operator to more easily perform safety checks when making a mobile object turn right or left. [Means for solving the problem]

[0006] The first aspect relates to a remote operator terminal for a remote operator who remotely operates a mobile object. The remote operator terminal includes a display device that displays rear-side images captured by the rear-side cameras mounted on the vehicle. The display device further displays the trajectory of the moving object behind the moving object by superimposing it on the rear side image.

[0007] The second aspect relates to a remote operation support method for displaying, on a display device, information for a remote operator who remotely operates a moving object. The remote operation support method is as follows: Displaying a rear-side image captured by a rear-side camera mounted on the moving object on a display device; A trajectory of the moving object behind the moving object is displayed superimposed on the rear side image. Includes .

[0008] The third aspect relates to a remote control system. The remote control system is One or more processors for controlling remote operation of a moving object; A display device that displays information to a remote operator who remotely operates the mobile object; Equipped with. The one or more processors acquire rear-side images taken by a rear-side camera mounted on the moving object. The one or more processors acquire information on the trajectory of the moving object behind the moving object. The one or more processors display the trajectory of the moving object on the display device, superimposed on the rear-side image. Effect of the Invention

[0009] According to the present disclosure, a display device used for remotely controlling a moving object displays a rear-side image captured by a rear-side camera mounted on the moving object. Furthermore, the display device displays a trajectory of the moving object behind the moving object superimposed on the rear-side image. By viewing the rear-side image on which the trajectory of the moving object is superimposed, a remote operator can more easily check for safety when the moving object turns left or right. This allows the remote operator to remotely control the moving object more safely. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating a configuration example of a remote control system according to an embodiment; [Diagram 2] 1 is a conceptual diagram illustrating an example of a camera mounted on a vehicle according to an embodiment; [Diagram 3] FIG. 2 is a conceptual diagram for explaining an example of a display device of a remote operator terminal according to an embodiment. [Figure 4] FIG. 13 is a conceptual diagram for explaining a scene in which a remote operator turns a vehicle left. [Diagram 5] 10A to 10C are conceptual diagrams for explaining an example of a path display process according to an embodiment. [Figure 6] 10A to 10C are conceptual diagrams for explaining another example of the path display process according to the embodiment. [Figure 7] 13 is a conceptual diagram for explaining still another example of the path display process according to the embodiment. FIG. [Figure 8] FIG. 11 is a conceptual diagram for explaining ON / OFF of a path display process according to the embodiment. [Figure 9] 4 is a block diagram showing an example of a functional configuration related to a path display process according to an embodiment; FIG. [Figure 10] 10 is a flowchart showing a process related to a path display process according to an embodiment. [Figure 11] 1 is a block diagram showing an example of the configuration of a vehicle according to an embodiment; [Figure 12] 2 is a block diagram showing a configuration example of a remote operator terminal according to an embodiment; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0012] 1. Overview of the remote control system Consider the remote control (remote driving) of a moving object. In particular, consider the remote control of a moving object traveling on the ground. Examples of moving objects that can be remotely controlled include vehicles and robots. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots. As an example, in the following explanation, consider the case where the moving object is a vehicle. When generalizing, "vehicle" in the following explanation should be read as "moving object".

[0013] FIG. 1 is a schematic diagram showing a configuration example of a remote operation system 1 according to the present embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is a target of remote operation. The remote operator terminal 200 is a terminal device used when a remote operator O remotely operates the vehicle 100. The remote operator terminal 200 can also be called a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. Typically, the management device 300 is a management server on a cloud. The management server may be composed of multiple servers that perform distributed processing.

[0014] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. The vehicle 100 and the remote operator terminal 200 may also communicate directly with each other without going through the management device 300.

[0015] The vehicle 100 is equipped with various sensors including a camera C. The camera C captures the surroundings of the vehicle 100 and acquires an image (video) IMG showing the situation around the vehicle 100. The sensor detection information SEN includes information obtained by the various sensors. The sensor detection information SEN includes at least the image IMG captured by the camera C. The sensor detection information SEN may include the position and state of the vehicle 100 (e.g., speed, steering angle, etc.). The vehicle 100 transmits the sensor detection information SEN to the remote operator terminal 200.

[0016] The remote operator terminal 200 receives the sensor detection information SEN transmitted from the vehicle 100. The remote operator terminal 200 presents the sensor detection information SEN to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device 220, and displays information such as an image IMG on the display device 220. The remote operator O recognizes the situation around the vehicle 100 by looking at the displayed information, and remotely operates the vehicle 100. In other words, by displaying information for the remote operator O on the display device 220, the remote operation of the vehicle 100 by the remote operator O is assisted.

[0017] The remote operation information OPE is information related to remote operation by a remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The remote operator terminal 200 transmits the remote operation information OPE to the vehicle 100. The vehicle 100 receives the remote operation information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle driving control according to the received remote operation information OPE. In this manner, remote operation of the vehicle 100 is realized.

[0018] 2. Examples of display devices and displayed images FIG. 2 is a conceptual diagram showing an example of a camera C mounted on a vehicle 100. In the example shown in FIG. 2, the vehicle 100 is equipped with a front camera C1, a left front camera C2, a right front camera C3, a left rear camera C4, a right rear camera C5, and a rear camera C6. The front camera C1 is installed to capture an image of the front of the vehicle 100. The left front camera C2 is installed to capture an image of the left front of the vehicle 100. The right front camera C3 is installed to capture an image of the right front of the vehicle 100. The left rear camera C4 is installed to capture an image of the left rear of the vehicle 100. The right rear camera C5 is installed to capture an image of the right rear of the vehicle 100. The rear camera C6 is installed to capture an image of the rear of the vehicle 100.

[0019] FIG. 3 is a conceptual diagram for explaining an example of the display device 220 of the remote operator terminal 200. The display device 220 displays an image IMG captured by a camera C mounted on the vehicle 100. In the example shown in FIG. 3, the display device 220 includes display areas (display units) 221 to 226. The display area 221 displays a front image IMG1 captured by the front camera C1 and showing the situation in front of the vehicle 100. The display area 222 displays a left front image IMG2 captured by the left front camera C2 and showing the situation in the left front of the vehicle 100. The display area 223 displays a right front image IMG3 captured by the right front camera C3 and showing the situation in the right front of the vehicle 100. The display area 224 displays a left rear image IMG4 captured by the left rear camera C4 and showing the situation in the left rear of the vehicle 100. The display area 225 displays a right rear image IMG5 captured by the right rear camera C5 and showing the situation in the right rear of the vehicle 100. In the display area 226, a rear image IMG6 that is captured by the rear camera C6 and shows the situation behind the vehicle 100 is displayed.

[0020] The remote operator terminal 200 may be a tablet type. The display device 220 of the remote operator terminal 200 may be a touch panel. The touch panel may display operation units 231, 232 for the remote operator O to remotely operate the vehicle 100. The remote operator O can remotely operate the vehicle 100 by touching the operation units 231, 232 on the touch panel.

[0021] The left rear camera C4 and the right rear camera C5 illustrated in FIG. 2 can be said to be "rear side cameras" for photographing the rear side of the vehicle 100. Also, the left rear image IMG4 and the right rear image IMG5 illustrated in FIG. 3 can be said to be "rear side images" showing the situation on the rear side of the vehicle 100. The display device 220 displays the rear side images photographed by the rear side cameras. The remote operator O can check the situation on the rear side of the vehicle 100 by looking at the displayed rear side images. This corresponds to the driver in the vehicle 100 checking the situation on the rear side by looking at the side mirrors.

[0022] The rear side image displayed on the display device 220 is useful for, for example, a safety check when the remote operator O turns the vehicle 100 left or right. As an example, FIG. 4 shows a scene in which the remote operator O turns the vehicle 100 left. By looking at the left rear image IMG4 captured by the left rear camera C4, the remote operator O can be careful not to cause the vehicle 100 to hit other objects X (e.g., pedestrians, bicycles). That is, the remote operator O can carefully turn the vehicle 100 left while checking for safety by looking at the left rear image IMG4. In this way, the remote operation of the vehicle 100 by the remote operator O is supported.

[0023] 3.Trajectory display processing It is desirable to enable the remote operator O to more easily perform safety checks when turning the vehicle 100 right or left.

[0024] As an example, it is possible to enlarge the display area (224, 225) in which the rear side images (IMG4, IMG5) are displayed. However, there may be a limit to the size of the display device 220, and it is not always possible to freely enlarge a specific display area. Furthermore, if there is a limit to the size of the display device 220, when a specific display area is enlarged, the other display areas become smaller accordingly. If the other display areas become smaller, there is a risk that it will become difficult for the remote operator O to smoothly perform remote operation.

[0025] Therefore, in this embodiment, a different approach is adopted. More specifically, the display device 220 of the remote operator terminal 200 displays the trajectory TR of the vehicle 100 behind the vehicle 100 by superimposing it on the rear side image (IMG4, IMG5) as necessary. The display of the trajectory TR of the vehicle 100 by superimposing it on the rear side image is hereinafter referred to as "trajectory display processing." The trajectory display processing is realized, for example, by an augmented reality (AR) technology.

[0026] Fig. 5 is a conceptual diagram for explaining an example of the trajectory display process. In the example shown in Fig. 5, the trajectory TR of the vehicle 100 is an area that the vehicle 100 passed through within a certain period of time in the past. The area that the vehicle 100 passed through within a certain period of time in the past is hereinafter referred to as a "passage area."

[0027] The vehicle position PV is the position of a representative point of the vehicle 100. For example, the vehicle position PV is the position of the center point of the vehicle 100. The history of the vehicle position PV in a certain period of time in the past as viewed from the current vehicle position PV can be calculated based on the steering angle and speed of the vehicle 100 in the certain period of time in the past. The vehicle width W is the width of the vehicle 100 and is known information. As shown in FIG. 5, the passing area is an area of ​​the vehicle width W that extends along the history of the vehicle positions PV. When the vehicle position PV is the position of the center point of the vehicle 100, the center line of the passing area coincides with the history of the vehicle positions PV. By combining the history of the vehicle position PV and the vehicle width W of the vehicle 100, it is possible to estimate the passing area in a certain period of time in the past.

[0028] As another example, the position history of each wheel of the vehicle 100 may be calculated from the history of the vehicle position PV. The installation position of each wheel as viewed from the vehicle position PV is known information, and the position history of each wheel can be calculated based on the known information. In this case, the area surrounded by the position history of all the wheels of the vehicle 100 is used as the passing area.

[0029] As yet another example, a history of the vehicle position PV in the absolute coordinate system may be used. The vehicle position PV in the absolute coordinate system is measured by a Global Navigation Satellite System (GNSS) sensor mounted on the vehicle 100. Alternatively, the vehicle position PV in the absolute coordinate system is calculated by a well-known self-localization method. By using the current vehicle position PV, the history of the vehicle position PV in the absolute coordinate system can be converted into a history of the vehicle position PV as viewed from the current vehicle position PV.

[0030] The display device 220 of the remote operator terminal 200 displays the passing area (trajectory TR) of the vehicle 100 superimposed on the rear side image. The installation position, installation direction, and angle of view of the rear side camera mounted on the vehicle 100 are known information. By combining the known information with the passing area seen from the current vehicle position PV, the passing area can be superimposed and displayed on the rear side image. The passing area is superimposed on the rear side image based on, for example, an augmented reality (AR) technique. In the example shown in FIG. 5, the passing area is superimposed on the left rear image IMG4 captured by the left rear camera C4. Then, the left rear image IMG4 with the passing area superimposed is displayed in the display area 224.

[0031] The remote operator O checks for safety by looking at the rear-side image on which the passing area is superimposed. If another object X (e.g., a pedestrian or a bicycle) is captured outside the passing area, the remote operator O must be particularly careful not to cause the vehicle 100 to hit the object X. The remote operator O may stop the vehicle 100. If another object X is detected in the vicinity of the vehicle 100, the remote operator terminal 200 may alert the remote operator O by at least one of display and audio.

[0032] Fig. 6 is a conceptual diagram for explaining another example of the trajectory display process. In the example shown in Fig. 6, the trajectory TR of the vehicle 100 is the boundary line of the above-mentioned passing area. The position of the boundary line of the passing area can be calculated from the position of the passing area. The rest is the same as the example explained in Fig. 5.

[0033] FIG. 7 is a conceptual diagram for explaining yet another example of the trajectory display process. In the example shown in FIG. 7, the vehicle 100 includes a tractor 101 and a trailer 102 towed by the tractor 101. The rear side camera is mounted on the tractor 101. In the example shown in FIG. 7, the trajectory TR of the vehicle 100 is the trajectory TR of the trailer 102. The connection state between the tractor 101 and the trailer 102, the positions of the wheels of the tractor 101 and the trailer 102, and the like are known information. Based on the known information and the vehicle position PV, the trajectory TR of the trailer 102 can be estimated. Then, the trajectory TR of the trailer 102 is superimposed on the rear side image.

[0034] FIG. 8 is a conceptual diagram for explaining ON / OFF of the trajectory display process. The trajectory display process may be selectively executed only when a predetermined trajectory display condition is satisfied. For example, the trajectory display process is performed in conjunction with a left turn or a right turn of the vehicle 100. For this purpose, the steering angle of the vehicle 100, the steering amount by the remote operator O, or the operation state of the blinker of the vehicle 100 is monitored. For example, when the steering angle of the vehicle 100 exceeds a threshold, the trajectory display process is turned ON, and when the steering angle is equal to or less than the threshold, the trajectory display process is turned OFF. As another example, when the steering amount by the remote operator O exceeds a threshold, the trajectory display process is turned ON, and when the steering amount is equal to or less than the threshold, the trajectory display process is turned OFF. As yet another example, when the blinker of the vehicle 100 is in operation, the trajectory display process may be turned ON, and when the blinker is not in operation, the trajectory display process may be turned OFF.

[0035] <Effects> As described above, according to this embodiment, the display device 220 used for remotely controlling the vehicle 100 displays a rear-side image captured by a rear-side camera mounted on the vehicle 100. Furthermore, the display device 220 displays the trajectory TR of the vehicle 100 behind the vehicle 100 superimposed on the rear-side image. By looking at the rear-side image on which the trajectory TR of the vehicle 100 is superimposed, the remote operator O can more easily check for safety when the vehicle 100 turns left or right. For example, when another object X (e.g., a pedestrian, a bicycle) is shown outside the trajectory TR of the vehicle 100, the remote operator O can pay particular attention so that the vehicle 100 does not get the object X caught up in it. This allows the remote operator O to remotely control the vehicle 100 more safely.

[0036] Furthermore, according to this embodiment, it is not necessary to make the display areas (224, 225) in which the rear side images (IMG4, IMG5) are displayed larger than necessary. Therefore, even if the size of the display device 220 is limited, it is not necessary to unnecessarily narrow other display areas. This is preferable from the viewpoint of smooth remote operation.

[0037] The path display process may be selectively executed only when a predetermined path display condition is satisfied. In this case, the path display process is not executed all the time, so that the processing load on the remote control system 1 is reduced.

[0038] 4. Functional configuration example Fig. 9 is a block diagram showing an example of a functional configuration related to the path display processing according to this embodiment. The remote control system 1 includes, as functional blocks, an information acquisition unit 10, a determination unit 20, an image processing unit 30, and a display control unit 40. Fig. 10 is a flowchart showing processing related to the path display processing according to this embodiment. Hereinafter, processing related to the path display device according to this embodiment will be described with reference to Figs. 9 and 10.

[0039] 4-1. Information acquisition process (step S10) In step S10, the information acquisition unit 10 acquires various pieces of information. More specifically, the information acquisition unit 10 acquires an image IMG captured by a camera C mounted on the vehicle 100. The information acquisition unit 10 also acquires vehicle information VCL related to the vehicle 100.

[0040] For example, the vehicle information VCL includes information on the installation position, installation direction, and angle of view of each camera C mounted on the vehicle 100. This information is known information.

[0041] The vehicle information VCL may include specification information of the vehicle 100. The specification information includes the vehicle width W, the installation position of each wheel, the wheelbase, the tread, etc. In the case where the vehicle 100 includes a tractor 101 and a trailer 102 (see FIG. 7), the specification information of the vehicle 100 may include the connection mode between the tractor 101 and the trailer 102, the position of each wheel of the tractor 101 and the trailer 102, etc.

[0042] The vehicle information VCL may include information on the steering angle and speed of the vehicle 100. The steering angle and speed of the vehicle 100 are obtained from the sensor detection information SEN.

[0043] The vehicle information VCL may include information indicating the operation state of a turn signal of the vehicle 100. The operation state of the turn signal is obtained from the sensor detection information SEN.

[0044] The vehicle information VCL may include information on the vehicle position PV in the absolute coordinate system. The vehicle position PV in the absolute coordinate system is measured by a GNSS sensor mounted on the vehicle 100, for example.

[0045] The information acquisition unit 10 further acquires information on the trajectory TR of the vehicle 100 behind the vehicle 100. In the example shown in FIG. 5, the trajectory TR of the vehicle 100 is a passing area that the vehicle 100 passed through within a certain period of time in the past. In the example shown in FIG. 6, the trajectory TR of the vehicle 100 is a boundary line of the passing area. In the example shown in FIG. 7, the trajectory TR of the vehicle 100 is a trajectory TR of the trailer 102. In either case, the information acquisition unit 10 can calculate (estimate) the trajectory TR of the vehicle 100 based on the above-mentioned vehicle information VCL.

[0046] The information acquiring unit 10 may further acquire remote operation information OPE. The remote operation information OPE includes an operation amount (e.g., steering amount) by the remote operator O. The remote operation information OPE may include an operation state of a direction indicator by the remote operator O.

[0047] 4-2. Judgment process (step S20) In step S20, the determination unit 20 determines whether or not a predetermined path display condition is met.

[0048] For example, the predetermined trajectory display condition is that "the steering angle of the vehicle 100 or the steering amount by the remote operator O exceeds a threshold value." The steering angle of the vehicle 100 is included in the above-mentioned vehicle information VCL. The steering amount by the remote operator O is included in the remote operation information OPE. The steering angle of the vehicle 100 may be calculated from the steering amount by the remote operator O. The determination unit 20 can determine whether the steering angle of the vehicle 100 or the steering amount by the remote operator O exceeds a threshold value based on the vehicle information VCL or the remote operation information OPE.

[0049] As another example, the predetermined trajectory display condition is that "the blinker of the vehicle 100 is in operation." The operation state of the blinker of the vehicle 100 is obtained from the above-mentioned vehicle information VCL. Alternatively, the operation state of the turn signal by the remote operator O may be regarded as the operation state of the blinker of the vehicle 100. The determination unit 20 can determine whether or not the blinker of the vehicle 100 is in operation based on the vehicle information VCL or the remote operation information OPE.

[0050] As yet another example, the predetermined path display condition may include the remote operator O turning on the path display function.

[0051] If the predetermined path display condition is met (step S20; Yes), the process proceeds to step S30. On the other hand, if the predetermined path display condition is not met (step S20; No), the process proceeds to step S35.

[0052] 4-3. Image processing (steps S30, S35) In step S30, the image processing unit 30 superimposes the trajectory TR of the vehicle 100 on the rear-side image captured by the rear-side camera. The installation position, installation orientation, and angle of view of the rear-side camera mounted on the vehicle 100 are obtained from the vehicle information VCL. Therefore, the image processing unit 30 can superimpose the trajectory TR of the vehicle 100 on the rear-side image based on the image IMG, the trajectory TR, and the vehicle information VCL. The image processing unit 30 outputs the image IMG including the rear-side image on which the trajectory TR of the vehicle 100 is superimposed to the display control unit 40.

[0053] On the other hand, in step S35, the image processing unit 30 does not superimpose the trajectory TR of the vehicle 100 on the rear side image. The image processing unit 30 outputs the image IMG to the display control unit 40 as it is.

[0054] 4-4. Display control process (step S40) In step S40, the display control unit 40 displays an image IMG on the display device 220 of the remote operator terminal 200. When the trajectory TR of the vehicle 100 is superimposed on the rear-side image, the display control unit 40 displays the rear-side image on which the trajectory TR of the vehicle 100 is superimposed on the display device 220. On the other hand, when the trajectory TR of the vehicle 100 is not superimposed on the rear-side image, the display control unit 40 displays the rear-side image as it is on the display device 220.

[0055] 1, the vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network and can exchange necessary information. Therefore, the information acquisition unit 10 may be included in any of the vehicle 100, the remote operator terminal 200, and the management device 300. Similarly, the determination unit 20 may be included in any of the vehicle 100, the remote operator terminal 200, and the management device 300. Similarly, the image processing unit 30 may be included in any of the vehicle 100, the remote operator terminal 200, and the management device 300. The display control unit 40 is included in the remote operator terminal 200.

[0056] 5. Example of vehicle configuration 5-1. Configuration example 11 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0057] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with a remote operator terminal 200 and a management device 300.

[0058] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera C, a LIDAR (Laser Imaging Detection and Ranging), a radar, etc. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0059] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates a driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates a braking force.

[0060] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processor 160) and one or more storage devices 170 (hereinafter simply referred to as storage device 170). The processor 160 executes various processes. Examples of the processor 160 include a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a field-programmable gate array (FPGA). The storage device 170 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), and a solid state drive (SSD). The control device 150 may include one or more electronic control units (ECUs).

[0061] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 may be realized by cooperation between the processor 160, which executes the vehicle control program PROG1, and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded in a computer-readable recording medium.

[0062] 5-2. Sensor detection information The control device 150 acquires sensor detection information SEN using the sensor group 120. The sensor detection information SEN is stored in the storage device 170. The sensor detection information SEN includes an image IMG, object information, etc. The image IMG is captured by the camera C.

[0063] The object information is information about objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, bicycles, two-wheeled vehicles, other vehicles (preceding vehicles, vehicles running parallel to each other, following vehicles, etc.), white lines, road structures (e.g., curbs, guard rails), poles, traffic lights, signs, etc. The control device 150 can recognize objects around the vehicle 100 by using a recognition sensor. For example, by analyzing an image IMG, it is possible to identify an object and calculate the relative position of the object. In addition, it is also possible to identify an object and obtain the relative position and relative speed of the object based on point cloud information obtained by LIDAR. The object information includes the relative position of the object with respect to the vehicle 100. The object information may further include the relative speed of the object.

[0064] 5-3.Vehicle driving control The control device 150 executes vehicle driving control for controlling the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes the vehicle driving control by controlling the driving device 130 (the steering device, the drive device, and the braking device).

[0065] The control device 150 may perform automatic driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the sensor detection information SEN. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0066] 5-4. Processing related to remote control When remote operation of the vehicle 100 is performed, the control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0067] The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. Typically, the control device 150 transmits an image IMG to the remote operator terminal 200. The control device 150 may transmit object information to the remote operator terminal 200. The control device 150 may transmit vehicle information VCL to the remote operator terminal 200.

[0068] Furthermore, the control device 150 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information related to remote operation by a remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle travel control according to the received remote operation information OPE.

[0069] Furthermore, the control device 150 may have the functions of at least one of the information acquisition unit 10, the determination unit 20, and the image processing unit 30 shown in FIG.

[0070] 6. Example of remote operator terminal configuration 12 is a block diagram showing an example of the configuration of the remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, and a control device 250.

[0071] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0072] The display device 220 displays various information for the remote operator O who performs remote operation. In other words, the display device 220 displays various information, thereby presenting the various information to the remote operator O. The display device 220 may be a touch panel (see FIG. 3).

[0073] The input device 230 is a member that is operated by the remote operator O when remotely operating the vehicle 100. For example, the input device 230 includes remote operation members. The remote operation members include a steering wheel, an accelerator pedal, a brake pedal, a turn signal, and the like.

[0074] The display device 220 and the input device 230 may be an integrated user interface. For example, the display device 220 and the input device 230 may be an integrated touch panel. The touch panel may include display areas (display units) 221 to 226 and operation units 231 and 232 (see FIG. 3).

[0075] The control device 250 controls the remote operator terminal 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as processor 260) and one or more storage devices 270 (hereinafter simply referred to as storage device 270). The processor 260 executes various processes. Examples of the processor 260 include a CPU, a GPU, an ASIC, and an FPGA. The storage device 270 stores various information. Examples of the storage device 270 include a volatile memory, a non-volatile memory, an HDD, and an SSD.

[0076] The remote operation control program PROG2 is a computer program executed by the processor 260. The functions of the control device 250 may be realized by cooperation between the processor 260 executing the remote operation control program PROG2 and the storage device 270. The remote operation control program PROG2 is stored in the storage device 270. Alternatively, the remote operation control program PROG2 may be recorded in a computer-readable recording medium. The remote operation control program PROG2 may be provided via a network.

[0077] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives sensor detection information SEN transmitted from the vehicle 100. The control device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the control device 250 presents the image IMG to the remote operator O by displaying the image IMG on the display device 220. That is, the control device 250 has the function of the display control unit 40 shown in FIG. 9. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the presented information.

[0078] The remote operator O operates the input device 230. The amount of operation of the input device 230 is detected by a sensor installed in the input device 230. The control device 250 generates remote operation information OPE reflecting the amount of operation of the input device 230 by the remote operator O. Then, the control device 250 transmits the remote operation information OPE to the vehicle 100 via the communication device 210.

[0079] Furthermore, the control device 250 may have the functions of at least one of the information acquisition unit 10, the determination unit 20, and the image processing unit 30 shown in FIG. [Explanation of symbols]

[0080] 1 Remote Control System 10 Information acquisition department 20 Judgment section 30 Image processing section 40 Display control unit 100 vehicles 101 Tractor 102 Trailer 200 Remote operator terminal 220 Display device 250 Control device 260 processor 270 Storage device 300 Management device IMG Image OPE Remote Operation Information SEN Sensor detection information TR trajectory VCL vehicle information

Claims

1. A remote operator terminal for a remote operator who remotely operates a moving object, comprising: a display device that displays a rear-side image captured by a rear-side camera mounted on the moving body; The display device further displays a trajectory of the moving object behind the moving object by superimposing it on the rear side image. Remote operator terminal.

2. 2. A remote operator terminal according to claim 1, comprising: When a predetermined condition is satisfied, the display device displays the trajectory of the moving object in a manner superimposed on the rear-side image; When the predetermined condition is not satisfied, the display device displays the rear side image without superimposing the trajectory of the moving object. Remote operator terminal.

3. 3. A remote operator terminal as claimed in claim 2, comprising: The predetermined condition is that the steering angle of the moving object or the steering amount by the remote operator exceeds a threshold value. Remote operator terminal.

4. 3. A remote operator terminal as claimed in claim 2, comprising: The predetermined condition is that the blinker of the moving object is in operation. Remote operator terminal.

5. A remote operator terminal according to any one of claims 1 to 4, comprising: The trajectory of the moving object is an area through which the moving object has passed. Remote operator terminal.

6. A remote operator terminal according to any one of claims 1 to 4, comprising: The trajectory of the moving object is the boundary line of the area through which the moving object has passed. Remote operator terminal.

7. A remote operator terminal according to any one of claims 1 to 4, comprising: The moving body includes a tractor and a trailer towed by the tractor, The trajectory of the moving object includes at least the trajectory of the trailer. Remote operator terminal.

8. A remote operation support method for displaying information for a remote operator who remotely operates a moving object on a display device, comprising: displaying a rear-side image captured by a rear-side camera mounted on the moving body on the display device; displaying a trajectory of the moving object behind the moving object in a manner superimposed on the rear side image; Includes Remote operation support method.

9. One or more processors for controlling remote operation of the mobile object; a display device that displays information for a remote operator who remotely operates the mobile object; Equipped with The one or more processors: A rear-side image is acquired by a rear-side camera mounted on the moving body, Acquire information about a trajectory of the moving object behind the moving object; The trajectory of the moving object is displayed on the display device while being superimposed on the rear side image. It was configured as Remote control system.

Images