Remote operator terminal, image display method, and remote control system

Abstract

To provide a remote operator terminal capable of improving the image quality of displaying multiple images captured by multiple cameras in remote operation of a moving object.SOLUTION: A first image is captured by a camera mounted on the moving object at first designated time. A first viewpoint is a camera viewpoint at a first point of time, while a second viewpoint is a camera viewpoint at a second point of time later than the first point of time. In projection transformation processing, a first image taken from the first viewpoint is converted into a second image taken from the second viewpoint. Multiple second images are obtained by applying projection transformation processing to the multiple first images captured by each of the multiple cameras. The multiple second images are displayed side by side so that the priority second images corresponding to the viewing direction of a remote operator appear in front of the other secondary images among those multiple secondary images.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to remote control of a mobile object by a remote operator, and more particularly to delay compensation in remote control. [Background technology]

[0002] In remote control of a vehicle, images (video) captured by an on-board camera are transmitted from the vehicle to a remote operator terminal and displayed on a display device of the remote operator terminal. At this time, it is desirable to compensate for communication delays between the vehicle and the remote operator terminal. The following technologies are known as those related to image (video) delay compensation.

[0003] Patent Document 1 discloses a remote video output system. The remote video output system includes an autonomous vehicle that transmits video and a remote video output device that receives video from the autonomous vehicle. The remote video output device estimates a change in the viewpoint position of the autonomous vehicle according to the communication delay time from the autonomous vehicle to the remote video output device. The remote video output device then takes into account the change in the viewpoint position of the autonomous vehicle, cuts out a portion of the frame of the received video, and displays the video of the cut-out portion.

[0004] Non-Patent Document 1 discloses a delay compensation technique that uses projective transformation. More specifically, this delay compensation technique performs projective transformation on images received from a vehicle and compensates for the delay by changing the viewpoint according to the amount of vehicle movement equivalent to the delay time. In other words, this delay compensation technique visually compensates for the delay by looking ahead at camera images seen from a viewpoint that is the delay time ahead. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 155159 [Non-patent literature]

[0006] [Non-Patent Document 1] Kodai Matsubara and Manabu Ohmae, "Research on delay compensation of camera images for remotely controlled automobiles using projective transformation," 19th ITS Symposium 2021, 4-A-12, December 2021 Summary of the Invention [Problem to be solved by the invention]

[0007] In remote control of a moving object, it is possible to perform delay compensation for images. However, the above-mentioned conventional technology does not consider delay compensation for multiple images taken by multiple cameras. Further study and improvement are desired regarding delay compensation for multiple images taken by multiple cameras.

[0008] For example, consider a case where multiple images are each subjected to a projective transformation and the multiple transformed images are displayed side by side. In this case, overlapping between adjacent transformed images may occur. If part of one transformed image is hidden by another transformed image, the information that the remote operator can see will be reduced. [Means for solving the problem]

[0009] A first aspect relates to a remote operator terminal used by a remote operator for remote operation of a mobile object. The first image is an image captured at a first timing by a camera mounted on a moving object. The first viewpoint is defined by the combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera that is predicted at a second timing that is later than the first timing. Projective transformation processing transforms a first image seen from a first viewpoint into a second image seen from a second viewpoint. The remote operator terminal includes an information processing device. The information processing device acquires a plurality of first images captured by a plurality of cameras mounted on a moving object at a first timing. The information processing device acquires a plurality of second images by applying a projective transformation process to each of a plurality of first images. The information processing device estimates the gaze direction of the remote operator. The information processing device displays the plurality of second images side by side on the display device of the remote operator terminal so that a priority second image corresponding to the gaze direction among the plurality of second images appears to be closer than the other second images.

[0010] A second aspect relates to a remote operator terminal used by a remote operator for remote operation of a mobile object. The first image is an image captured at a first timing by a camera mounted on a moving object. The first viewpoint is defined by the combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera that is predicted at a second timing that is later than the first timing. Projective transformation processing transforms a first image seen from a first viewpoint into a second image seen from a second viewpoint. The remote operator terminal includes an information processing device. The information processing device acquires a plurality of first images captured by a plurality of cameras mounted on a moving object at a first timing. The information processing device acquires a plurality of second images by applying a projective transformation process to each of a plurality of first images. The information processing device displays the plurality of second images side by side on the display device of the remote operator terminal. The priority second image is one of the plurality of second images that corresponds to the gaze direction of the remote operator. The enlarged second image is the priority second image or a second image displayed next to the priority second image. The enlarged first image is the first image before the projective transformation process, which is the basis of the enlarged second image. The information processing device further adjusts the plurality of first images so that the enlarged first image is larger than the other first images before the projective transformation process.

[0011] A third aspect relates to an image display method for displaying an image for a remote operator when remotely operating a mobile object. The first image is an image captured at a first timing by a camera mounted on a moving object. The first viewpoint is defined by the combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera that is predicted at a second timing that is later than the first timing. Projective transformation processing transforms a first image seen from a first viewpoint into a second image seen from a second viewpoint. The image display method is acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at a first timing; obtaining a plurality of second images by applying a projective transformation process to each of the plurality of first images; estimating a gaze direction of a remote operator; displaying the plurality of second images side by side on a display device of the remote operator terminal so that a priority second image corresponding to the gaze direction among the plurality of second images appears to be closer than the other second images; Includes.

[0012] A fourth aspect relates to a remote control system for remotely controlling a moving object by a remote operator. The first image is an image captured at a first timing by a camera mounted on a moving object. The first viewpoint is defined by the combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera that is predicted at a second timing that is later than the first timing. Projective transformation processing transforms a first image seen from a first viewpoint into a second image seen from a second viewpoint. The remote control system includes one or more processors. The one or more processors acquire a plurality of first images taken by each of a plurality of cameras mounted on the moving object at a first timing. The one or more processors obtain a plurality of second images by applying a projective transformation process to each of the plurality of first images. The one or more processors estimate the gaze direction of the remote operator. The one or more processors display the multiple second images side by side on the display device of the remote operator terminal so that a priority second image among the multiple second images that corresponds to the gaze direction appears to be closer than the other second images. [Effects of the Invention]

[0013] According to the first, third, and fourth aspects, the priority second image corresponding to the gaze direction of the remote operator is displayed in front of the other second images. In other words, the priority second image corresponding to the gaze direction of the remote operator is displayed in the foreground. Therefore, the remote operator can see at least the entire priority second image in the gaze direction. As a result, the accuracy of remote operation is ensured.

[0014] According to the second aspect, the size of the enlarged second image generated from the enlarged first image is increased. This fills in the gaps around the priority second image that correspond to the remote operator's gaze direction. As a result, the appearance is improved and the sense of incongruity felt by the remote operator is reduced. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a remote control system. [Figure 2] FIG. 10 is a conceptual diagram for explaining an overview of delay compensation processing. [Figure 3] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 4] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 5] FIG. 10 is a conceptual diagram for explaining an example of image deformation by projective transformation. [Figure 6] FIG. 10 is a conceptual diagram for explaining a first example of projective transformation processing in the case of multiple cameras. [Figure 7] FIG. 10 is a conceptual diagram for explaining a first example of projective transformation processing in the case of multiple cameras. [Figure 8] FIG. 10 is a conceptual diagram for explaining a second example of projective transformation processing in the case of multiple cameras. [Figure 9] FIG. 10 is a conceptual diagram for explaining a second example of projective transformation processing in the case of multiple cameras. [Figure 10] FIG. 10 is a conceptual diagram for explaining a second example of projective transformation processing in the case of multiple cameras. [Figure 11] FIG. 10 is a conceptual diagram for explaining a third example of projective transformation processing in the case of multiple cameras. [Figure 12] FIG. 10 is a conceptual diagram for explaining a third example of projective transformation processing in the case of multiple cameras. [Figure 13] FIG. 1 is a block diagram showing an example of the configuration of a vehicle. [Figure 14] FIG. 2 is a block diagram showing an example of the configuration of a remote operator terminal. DETAILED DESCRIPTION OF THE INVENTION

[0016] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0017] 1.Remote control system Consider the remote control (remote driving) of a moving object. Examples of moving objects that can be remotely controlled include vehicles, robots, flying objects, etc. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones, etc. As an example, in the following explanation, consider a case where the moving object is a vehicle. When generalizing, "vehicle" in the following explanation should be read as "moving object."

[0018] FIG. 1 is a schematic diagram showing an example of the configuration of a remote operation system 1 according to this embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote operation. The remote operator terminal 200 is a terminal device used by a remote operator O to remotely operate the vehicle 100. The remote operator terminal 200 can also be called a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. Typically, the management device 300 is a management server on the cloud. The management server may be composed of multiple servers that perform distributed processing.

[0019] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 may communicate directly without going through the management device 300.

[0020] The vehicle 100 is equipped with various sensors including a camera CAM. The camera CAM photographs the surroundings of the vehicle 100 and acquires images (video) IMG showing the situation around the vehicle 100. The sensor detection information SEN includes information obtained by the various sensors. The sensor detection information SEN includes at least the images IMG photographed by the camera CAM. The sensor detection information SEN may include the position and status of the vehicle 100 (e.g., speed, steering angle, etc.). The vehicle 100 transmits the sensor detection information SEN to the remote operator terminal 200.

[0021] The remote operator terminal 200 receives the sensor detection information SEN transmitted from the vehicle 100. The remote operator terminal 200 presents the sensor detection information SEN to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device 220, and displays information such as an image IMG on the display device 220. The remote operator O looks at the displayed information, recognizes the situation around the vehicle 100, and remotely controls the vehicle 100. In other words, by displaying information for the remote operator O on the display device 220, the remote operator O's remote operation of the vehicle 100 is assisted.

[0022] The remote operation information OPE is information related to remote operation by a remote operator O. For example, the remote operation information OPE includes the amount of operation by the remote operator O. The remote operator terminal 200 transmits the remote operation information OPE to the vehicle 100. The vehicle 100 receives the remote operation information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle driving control in accordance with the received remote operation information OPE. In this way, remote operation of the vehicle 100 is realized.

[0023] 2. Delay compensation processing using projective transformation Remote operation of the vehicle 100 involves a communication delay between the vehicle 100 and the remote operator terminal 200. The communication delay may destabilize the behavior of the vehicle 100 during remote operation. The communication delay may also result in a decrease in the accuracy of the remote operation of the vehicle 100. Therefore, it is important to perform delay compensation when remotely operating the vehicle 100.

[0024] The remote operation system 1 according to this embodiment performs visual delay compensation for the image IMG displayed on the display device 220, taking communication delays into consideration. In particular, the remote operation system 1 according to this embodiment performs visual delay compensation for the image IMG by using "projective transformation." The subject of the delay compensation process is, for example, the remote operator terminal 200. However, the subject of the delay compensation process is not limited to the remote operator terminal 200. At least a part of the delay compensation process may be performed by the vehicle 100 or the management device 300.

[0025] 2 is a conceptual diagram for explaining an outline of the delay compensation process by the remote operation system 1. The first image IMG1 is an image IMG that is actually captured at a first timing T1 by a camera CAM mounted on the vehicle 100. The first image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 acquires the first image IMG1 after the first timing T1. If it is possible to estimate (predict) an image IMG that will be captured in the future from the first image IMG1, it becomes possible to perform delay compensation.

[0026] The second timing T2 is a target timing for look-ahead and is later than the first timing T1. The difference between the second timing T2 and the first timing T1 corresponds to the "delay compensation time." The remote operation system 1 may set at least a part of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200 as the delay compensation time. The communication delay time between the vehicle 100 and the remote operator terminal 200 can be estimated using well-known techniques. The delay compensation time may be set to the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The delay compensation time may be set to the communication time required for the image IMG to be transmitted from the vehicle 100 and reach the remote operator terminal 200. Alternatively, the delay compensation time may be set to a constant value. In either case, the remote operation system 1 sets the second timing T2 to be later than the first timing T1 by the delay compensation time.

[0027] For convenience, the camera CAM at the first timing T1 will be referred to as the first camera CAM1, and the camera CAM at the second timing T2 will be referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the combination of the position and orientation of the first camera CAM1 at the first timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the combination of the predicted position and orientation of the second camera CAM2 at the second timing T2.

[0028] The remote control system 1 acquires camera information CINF related to the camera CAM mounted on the vehicle 100. The camera information CINF includes installation information and performance information of the camera CAM. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. Because the camera CAM is fixed to the vehicle 100, by using the installation information of the camera CAM, the direction and amount of movement of the vehicle 100 can be converted into the direction and amount of movement of the camera CAM in the camera coordinate system. In other words, the change in the viewpoint of the camera CAM can be estimated based on the installation information of the camera CAM and the direction and amount of movement of the vehicle 100.

[0029] More specifically, the remote control system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 (i.e., the delay compensation time). For example, the remote control system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 based on the speed and steering angle of the vehicle 100 at the first timing T1 and the delay compensation time. Information on the speed and steering angle of the vehicle 100 is obtained from sensor detection information SEN provided by the vehicle 100. Alternatively, the steering angle in the steering operation by the remote operator O may be considered to be the steering angle of the vehicle 100. The vehicle 100 may be assumed to make a steady circular turn. Then, the remote control system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information CINF (installation information) and the amount and direction of movement of the vehicle 100 during the delay compensation time.

[0030] The first image IMG1 can be said to be an image IMG captured from the first viewpoint, i.e., an image IMG viewed from the first viewpoint. An image IMG expected to be captured from the second viewpoint, i.e., an image IMG expected to be seen from the second viewpoint, will hereinafter be referred to as a "second image IMG2." The remote control system 1 converts the first image IMG1 viewed from the first viewpoint into a second image IMG2 viewed from the second viewpoint based on the difference between the first and second viewpoints. In other words, the remote control system 1 predicts (reads ahead) the second image IMG2 viewed from the second viewpoint based on the first image IMG1 viewed from the first viewpoint. Projective transformation is used for this look-ahead.

[0031] FIG. 3 is a conceptual diagram for explaining projective transformation. Projective transformation is performed based on perspective projection transformation. Perspective projection transformation is a rendering technique for rendering an object in three-dimensional space on a two-dimensional plane as seen from the camera CAM. To achieve this, perspective projection transformation projects points in three-dimensional space onto a projection plane P, taking into account the viewpoint of the camera CAM. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane perpendicular to the optical axis of the camera CAM. Note that points in three-dimensional space are defined in a three-dimensional world coordinate system (absolute coordinate system). On the other hand, points projected onto the projection plane P are defined in a two-dimensional image coordinate system.

[0032] For example, N virtual points are virtually set in a three-dimensional world coordinate system. N is an integer equal to or greater than 4. The N virtual points as viewed from a first camera CAM1 (first viewpoint) are projected onto a first projection plane P1 associated with the first camera CAM1 by perspective projection transformation. The N virtual points as viewed from a second camera CAM2 (second viewpoint) are projected onto a second projection plane P2 associated with the second camera CAM2 by perspective projection transformation. The second viewpoint is obtained from the difference between the first viewpoint and the second viewpoint. The image coordinates of the virtual points on the first projection plane P1 as viewed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the virtual points on the second projection plane P2 as viewed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on a comparison of the two, a projection transformation matrix H for converting from the first viewpoint to the second viewpoint is calculated. Then, the projective transformation matrix H is applied to the entire first image IMG1 actually captured by the first camera CAM1, thereby generating a second image IMG2 that is expected to be seen from the second viewpoint.

[0033] As another example, the method described in Non-Patent Document 1 may be used. Specifically, by inverse transformation of the perspective projection transformation, each image coordinate point on the first image IMG1 (projection plane P) is transformed into a world coordinate point in the world coordinate system. Based on the difference between the first viewpoint and the second viewpoint, the world coordinate point as seen from the first viewpoint is transformed into a world coordinate point as seen from the second viewpoint. Then, by perspective projection transformation, the world coordinate point as seen from the second viewpoint is returned onto the projection plane P. In this way, a second image IMG2 that is expected to be seen from the second viewpoint is generated. Note that according to Non-Patent Document 1, it is assumed that the ground surface S is reflected across the entire image IMG, as shown in FIG. 4.

[0034] FIG. 5 is a conceptual diagram for explaining an example of image deformation by projective transformation (see Non-Patent Document 1). The first image IMG1 is the original image before projective transformation. The second image IMG2 is generated by applying projective transformation to the first image IMG1. FIG. 5 shows how the first image IMG1 and the second image IMG2 appear on the screen 222 of the display device 220. For example, when the vehicle 100 travels straight, the second image IMG2 appears to be tilted backward. As another example, when the vehicle turns right, the second image IMG2 appears to be tilted to the left.

[0035] 3. Projection transformation processing for multiple cameras In the following description, "projective transformation processing" refers to delay compensation processing using the projective transformation described in Section 2 above. The projective transformation processing converts a first image IMG1 viewed from a first viewpoint into a second image IMG2 viewed from a second viewpoint based on perspective projection transformation using a projection plane P associated with the camera CAM. The remote operation system 1 can perform delay compensation by applying projective transformation processing to the image IMG captured by the camera CAM. The subject of the projective transformation processing is, for example, the remote operator terminal 200. However, the subject of the projective transformation processing is not limited to the remote operator terminal 200. At least a part of the projective transformation processing may be executed by the vehicle 100 or the management device 300.

[0036] Here, consider a case where multiple cameras CAM are mounted on the vehicle 100. The installation orientations of the multiple cameras CAM in the vehicle coordinate system are different from each other. For example, the multiple cameras CAM include a front camera CAM-F for capturing images of the front, a left camera CAM-L for capturing images of the left front, and a right camera CAM-R for capturing images of the right front. However, the components of the multiple cameras CAM are not limited to this.

[0037] The remote operator terminal 200 acquires a plurality of images IMG captured by each of a plurality of cameras CAM mounted on the vehicle 100. The remote operator terminal 200 then displays the plurality of images IMG on the display device 220. The remote operator terminal 200 may display the plurality of images IMG side by side on one or more screens 222 of the display device 220.

[0038] Hereinafter, the projective transformation process for a plurality of images IMG captured by a plurality of cameras CAM will be considered.

[0039] 3-1. First example 6 is a conceptual diagram for explaining a first example. Projection planes PF, PL, and PR are projection planes P associated with the front camera CAM-F, left camera CAM-L, and right camera CAM-R, respectively. For example, projection plane PF is orthogonal to the optical axis of the front camera CAM-F, projection plane PL is orthogonal to the optical axis of the left camera CAM-L, and projection plane PR is orthogonal to the optical axis of the right camera CAM-R. The projection planes PF, PL, and PR are different from one another.

[0040] The multiple cameras CAM (CAM-F, CAM-L, CAM-R) are installed in different orientations. Therefore, as shown in Fig. 6, when the vehicle 100 moves in a certain direction, the change in the viewpoint of the camera CAM relative to the projection plane P differs between the multiple cameras CAM (CAM-F, CAM-L, CAM-R).

[0041] The remote control system 1 acquires a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R) captured by a plurality of cameras CAM (CAM-F, CAM-L, CAM-R) at a first timing T1. The remote control system 1 acquires a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) by separately applying a projective transformation process to each of the plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). In a first example, a plurality of different projection planes P (PF, PL, PR) are used for each of the plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). That is, the remote control system 1 generates a second image IMG2-F by applying a projective transformation process using the projection plane PF to the first image IMG1-F captured by the front camera CAM-F. The remote control system 1 also generates a second image IMG2-L by applying a projective transformation process using a projection plane PL to a first image IMG1-L captured by the left camera CAM-L. Furthermore, the remote control system 1 generates a second image IMG2-R by applying a projective transformation process using a projection plane PR to a first image IMG1-R captured by the right camera CAM-R.

[0042] Fig. 7 shows an example of displaying a plurality of images IMG in the first example. In the example shown in Fig. 7, the display device 220 includes a plurality of screens 222-F, 222-L, and 222-R. The plurality of screens 222-F, 222-L, and 222-R are arranged adjacent to each other in a row. More specifically, the screens 222-L and 222-R are arranged on either side of the screen 222-F, and the screen 222-F is sandwiched between the screens 222-L and 222-R.

[0043] If projective transformation processing is not performed, the remote operator terminal 200 displays multiple first images IMG1 (IMG1-F, IMG1-L, IMG1-R) on each of multiple screens 222 (222-F, 222-L, 222-R) of the display device 220.

[0044] When the projective transformation process is performed, the remote operator terminal 200 acquires a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) from a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). In the example shown in FIG. 7, the vehicle 100 is turning right. The remote operator terminal 200 then displays the plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) on the display device 220. For example, the remote operator terminal 200 displays the plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) side by side on one or more screens 222 of the display device 220. The remote operator terminal 200 may display multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) side by side on multiple screens 222 (222-F, 222-L, 222-R). In this case, the second image IMG2-F is generally arranged on the screen 222-F, but a portion of the second image IMG2-F may extend onto the screens 222-L and 222-R adjacent to the screen 222-F. Similarly, the second image IMG2-L is generally arranged on the screen 222-L, but a portion of the second image IMG2-L may extend onto the screen 222-F adjacent to the screen 222-L. Similarly, the second image IMG2-R is generally arranged on the screen 222-R, but a portion of the second image IMG2-R may extend onto the screen 222-F adjacent to the screen 222-R.

[0045] 3-2. Second example 7, the shapes of the multiple second images IMG2 do not necessarily match. Therefore, when multiple second images IMG2 are displayed side by side on the display device 220, the boundaries of adjacent second images IMG2 may not match, resulting in "overlap" between the adjacent second images IMG2. If part of a second image IMG2 is hidden by another second image IMG2, the information that the remote operator O can see will be reduced.

[0046] Therefore, the second example proposes a technique that can appropriately present necessary information to the remote operator O. Note that explanations that overlap with the first example above will be omitted as appropriate.

[0047] As an example, consider a situation in which the vehicle 100 is turning right, as shown in FIG. 8. FIG. 9 shows an example of the display of multiple second images IMG2 when turning right. When turning right, the remote operator O mainly gazes at the second image IMG2-R on the right side. Therefore, the remote operation system 1 displays multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) side by side so that the second image IMG2-R on the right side appears closer than the others. In other words, the second image IMG2-R on the right side is displayed in the foreground. Therefore, the remote operator O can see at least the entire second image IMG2-R in the gaze direction. As a result, the accuracy of the remote operation is ensured. Although the entire second images IMG2-F and IMG2-L may not be displayed, the remote operator O is not gazing at the second images IMG2-F and IMG2-L, so the impact on the accuracy of the remote operation is small.

[0048] In general, the second image IMG2 corresponding to the gaze direction of the remote operator O will be referred to as the "priority second image IMG2-PRI" hereinafter. The camera CAM corresponding to the gaze direction of the remote operator O will be referred to as the "priority camera CAM-PRI" hereinafter. The priority second image IMG2-PRI is the second image IMG2 obtained by applying a projective transformation process to the first image IMG1 captured by the priority camera CAM-PRI.

[0049] According to the second example, the remote operation system 1 estimates the gaze direction of the remote operator O. Then, the remote operation system 1 dynamically selects the priority camera CAM-PRI and the priority second image IMG2-PRI in consideration of the gaze direction of the remote operator O. In other words, the remote operation system 1 dynamically switches between the priority camera CAM-PRI and the priority second image IMG2-PRI in conjunction with the gaze direction of the remote operator O.

[0050] FIG. 10 is a conceptual diagram illustrating the correspondence between the gaze direction of the remote operator O and the priority second image IMG2-PRI. When the gaze direction of the remote operator O is within the gaze direction range RNG-F, the remote operation system 1 selects the front camera CAM-F as the priority camera CAM-PRI and the second image IMG2-F as the priority second image IMG2-PRI. When the gaze direction of the remote operator O is within the gaze direction range RNG-L, the remote operation system 1 selects the left camera CAM-L as the priority camera CAM-PRI and the second image IMG2-L as the priority second image IMG2-PRI. When the gaze direction of the remote operator O is within the gaze direction range RNG-R, the remote operation system 1 selects the left camera CAM-L as the priority camera CAM-PRI and the second image IMG2-R as the priority second image IMG2-PRI.

[0051] The gaze direction of the remote operator O is estimated based on, for example, the steering angle of the steering operation by the remote operator O. In this case, a predetermined steering angle range θ-F including straight ahead is associated with the gaze direction range RNG-F. A steering angle range θ-R to the right of the predetermined steering angle range θ-F is associated with the gaze direction range RNG-R. A steering angle range θ-L to the left of the predetermined steering angle range θ-F is associated with the gaze direction range RNG-L. The remote operation system 1 estimates the gaze direction of the remote operator O based on the steering angle of the steering operation by the remote operator O. More specifically, the remote operation system 1 selects a steering angle range θ-X (X=ForLorR) including the steering angle of the steering operation by the remote operator O. Furthermore, the remote operation system 1 estimates that the gaze direction of the remote operator O is included in the gaze direction range RNG-X associated with the selected steering angle range θ-X. Then, the remote control system 1 selects the camera CAM-X corresponding to the gaze direction range RNG-X as the priority camera.

[0052] As another example, the gaze direction of the remote operator O may be estimated from the gaze direction of the remote operator O. More specifically, the gaze direction of the remote operator O is detected by the operator monitor 240 (described later) of the remote operator terminal 200. The detected gaze direction of the remote operator O is considered to be the gaze direction of the remote operator O. The remote operation system 1 selects, as the priority camera, a camera CAM-X corresponding to a gaze direction range RNG-X that includes the gaze direction (gaze direction) of the remote operator O.

[0053] In this way, the remote operation system 1 estimates the gaze direction of the remote operator O and dynamically selects a priority camera and a priority second image IMG2-PRI according to the gaze direction of the remote operator O. Then, the remote operation system 1 displays multiple second images IMG2 side by side so that the priority second image IMG2-PRI appears to be in front of the other second images IMG2. In other words, the priority second image IMG2-PRI corresponding to the gaze direction of the remote operator O is displayed in the foreground. Therefore, the remote operator O can see at least the entire priority second image IMG2-PRI in the gaze direction. As a result, the accuracy of remote operation is ensured.

[0054] 3-3. Third Example 7, the shapes of the multiple second images IMG2 do not necessarily match. Therefore, when multiple second images IMG2 are displayed side by side on the display device 220, the boundaries of adjacent second images IMG2 may not match, and "gaps" may appear between the adjacent second images IMG2. This may cause the remote operator O to feel uncomfortable.

[0055] Therefore, the third example proposes a technique that can suppress the sense of discomfort felt by the remote operator O. Note that explanations that overlap with the first and second examples above will be omitted as appropriate.

[0056] Fig. 11 is a conceptual diagram for explaining a third example. In the example shown in Fig. 11, the second image IMG2-R is the priority second image IMG2-PRI corresponding to the gaze direction of the remote operator O. If it is possible to at least fill the gap between the priority second image IMG2-PRI and the adjacent second image IMG2-F, it is thought that the sense of discomfort felt by the remote operator O can be suppressed. To this end, the size of the second image IMG2-F adjacent to the priority second image IMG2-PRI is set larger than usual. Alternatively, the size of the priority second image IMG2-PRI itself may be set larger than usual.

[0057] In general, the priority second image IMG2-PRI, or the second image IMG2 displayed next to the priority second image IMG2-PRI, will be referred to as the "enlarged second image IMG2-MAG." The first image IMG1, which is the source of the enlarged second image IMG2-MAG, will be referred to as the "enlarged first image IMG1-MAG." The enlarged second image IMG2-MAG is obtained by applying a projective transformation process to the enlarged first image IMG1-MAG. In the example shown in FIG. 11, the first image IMG1-F is the enlarged first image IMG1-MAG, and the second image IMG2-F is the enlarged second image IMG2-MAG.

[0058] Before the projective transformation process, the remote control system 1 adjusts the multiple first images IMG1 (IMG1-F, IMG1-L, IMG1-R) so that the enlarged first image IMG1-MAG is larger than the other first images IMG1. The first image IMG1 to be enlarged can be recognized based on the priority camera CAM-PRI.

[0059] FIG. 12 is a conceptual diagram illustrating an example of a technique for realizing an enlarged first image IMG1-MAG. The image IMG captured by the camera CAM includes a "central region" and a "peripheral region" surrounding the central region. More specifically, the angle of view of the camera CAM includes a "central angle of view" and a "peripheral angle of view" surrounding the central angle of view. For example, the central angle of view is a range corresponding to a horizontal field of view (HFOV) of 90 degrees, and the peripheral angle of view is a range corresponding to a horizontal field of view > 90 degrees. The central region is an image region of the image IMG that corresponds to the central angle of view. On the other hand, the peripheral region is an image region of the image IMG that corresponds to the peripheral angle of view.

[0060] By default, only the central region of the image IMG captured by the camera CAM is used as the first image IMG1. However, for the first image IMG1 to be enlarged, both the central region and the peripheral region are used. That is, the enlarged first image IMG1-MAG includes both the central region and the peripheral region. The remote operation system 1 adjusts the multiple first images IMG1 in advance so that the enlarged first image IMG1-MAG includes both the central region and the peripheral region, and the other first images IMG1 include only the central region.

[0061] After the adjustment, the remote operation system 1 applies a projective transformation process to each of the multiple first images IMG1 (IMG1-F, IMG1-L, IMG1-R) to obtain multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R). At this time, an enlarged second image IMG2-MAG is obtained from the enlarged first image IMG1-MAG. The remote operation system 1 displays the multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) obtained in this manner side by side on the display device 220.

[0062] 11, the size of the enlarged second image IMG2-MAG is increased. This fills the gap around the priority second image IMG2-PRI that corresponds to the gaze direction of the remote operator O. As a result, the "appearance" as seen by the remote operator O is improved, and the sense of discomfort felt by the remote operator O is reduced.

[0063] 3-4. Fourth Example The fourth example is a combination of the second and third examples described above. By combining the second and third examples described above, the effects of both the second and third examples can be obtained.

[0064] 4. Example of vehicle configuration 4-1.Configuration example 13 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0065] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300.

[0066] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera (CAM), a LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0067] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates braking force.

[0068] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processors 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processors 160 perform various processes. Examples of the processors 160 include a general-purpose processor, a specific-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, and / or a combination thereof. The storage device 170 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), a solid-state drive (SSD), etc. The control device 150 may include one or more electronic control units (ECUs). In general terms, the control device 150 can also be called a processing circuitry.

[0069] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 may be realized by cooperation between the processor 160, which executes the vehicle control program PROG1, and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0070] 4-2. Sensor detection information The control device 150 acquires sensor detection information SEN using the sensor group 120. The sensor detection information SEN is stored in the storage device 170. The sensor detection information SEN includes an image IMG, vehicle state information, position information, object information, etc. The image IMG is captured by a camera CAM. The vehicle state information indicates the state of the vehicle 100 (e.g., speed, steering angle, etc.) detected by the vehicle state sensor. The position information indicates the position and orientation of the vehicle 100 detected by the position sensor.

[0071] The object information is information about objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, bicycles, motorcycles, other vehicles (leading vehicles, vehicles running alongside, following vehicles, etc.), white lines, road structures (e.g., curbs, guardrails), poles, traffic lights, signs, etc. The control device 150 can recognize objects around the vehicle 100 by using a recognition sensor. For example, by analyzing an image (IMG), it is possible to identify an object and calculate the relative position of the object. It is also possible to identify an object and obtain the relative position and relative speed of the object based on point cloud information obtained by LIDAR. The object information includes the relative position of the object with respect to the vehicle 100. The object information may further include the relative speed of the object.

[0072] 4-3.Vehicle driving control The control device 150 executes vehicle driving control to control the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes vehicle driving control by controlling the driving device 130 (steering device, drive device, and brake device).

[0073] The control device 150 may perform automatic driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the sensor detection information SEN. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0074] 4-4. Processing related to remote control When the vehicle 100 is remotely operated, the control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0075] The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. Typically, the control device 150 transmits an image IMG to the remote operator terminal 200. The control device 150 may transmit vehicle state information to the remote operator terminal 200. The control device 150 may transmit object information to the remote operator terminal 200.

[0076] Furthermore, the control device 150 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information related to remote operation by the remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle travel control in accordance with the received remote operation information OPE.

[0077] 4-5.Camera information The camera information CINF includes installation information and performance information for each of one or more camera CAMs mounted on the vehicle 100. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. The camera information CINF is stored in the storage device 170. The control device 150 may transmit the camera information CINF to the remote operator terminal 200.

[0078] 5. Example of remote operator terminal configuration 14 is a block diagram showing an example of the configuration of the remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, an operator monitor 240, and an information processing device 250.

[0079] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0080] The display device 220 displays various information for the remote operator O who performs remote operation. In other words, the display device 220 displays various information to present the information to the remote operator O. The display device 220 includes a plurality of screens 222.

[0081] The input device 230 is a member that the remote operator O operates when remotely operating the vehicle 100. For example, the input device 230 includes remote operation members, such as a steering wheel, an accelerator pedal, a brake pedal, and turn signals.

[0082] The operator monitor 240 includes a sensor for monitoring the state of the remote operator O. For example, the operator monitor 240 includes a camera that captures the face and eyes of the remote operator O. The operator monitor 240 extracts images of the face and eyes of the remote operator O by analyzing the image of the remote operator O captured by the camera. The extraction of the face and eye images is performed, for example, by using a machine learning model that has been generated in advance through machine learning. The operator monitor 240 then recognizes the gaze direction of the remote operator O based on the position and rotation angle of the pupils of each eye.

[0083] The information processing device 250 controls the remote operator terminal 200. The information processing device 250 includes one or more processors 260 (hereinafter simply referred to as processors 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processor 260 executes various processes. Examples of the processor 260 include a general-purpose processor, a special-purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, and / or a combination thereof. The storage device 270 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a HDD, an SSD, etc. In general terms, the information processing device 250 can also be called a processing circuitry.

[0084] The remote operation control program PROG2 is a computer program executed by the processor 260. The functions of the information processing device 250 may be realized by cooperation between the processor 260 executing the remote operation control program PROG2 and the storage device 270. The remote operation control program PROG2 is stored in the storage device 270. Alternatively, the remote operation control program PROG2 may be recorded on a computer-readable recording medium. The remote operation control program PROG2 may be provided via a network.

[0085] The information processing device 250 communicates with the vehicle 100 via the communication device 210. The information processing device 250 receives sensor detection information SEN transmitted from the vehicle 100. The information processing device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the information processing device 250 presents an image IMG to the remote operator O by displaying the image IMG on the display device 220. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the presented information.

[0086] The remote operator O operates the input device 230. The amount of operation of the input device 230 is detected by a sensor installed on the input device 230. The information processing device 250 generates remote operation information OPE that reflects the amount of operation of the input device 230 by the remote operator O. Then, the information processing device 250 transmits the remote operation information OPE to the vehicle 100 via the communication device 210.

[0087] The information processing device 250 may receive the camera information CINF transmitted from the vehicle 100. The camera information CINF is stored in the storage device 270.

[0088] The information processing device 250 performs the projective transformation processing (delay compensation processing) described in Sections 2 and 3 above. The speed and steering angle of the vehicle 100 are obtained from the sensor detection information SEN. The steering angle in the steering operation by the remote operator O may be considered as the steering angle of the vehicle 100. Installation information and performance information of each camera CAM mounted on the vehicle 100 are obtained from the camera information CINF. The gaze direction of the remote operator O can be estimated from the amount of steering operation by the remote operator O. The gaze direction of the remote operator O can also be estimated from the line of sight of the remote operator O recognized by the operator monitor 240. The information processing device 250 performs the projective transformation processing (delay compensation processing) described in Sections 2 and 3 above based on this information. [Explanation of symbols]

[0089] 1. Remote control system 100 vehicles 200 Remote Operator Terminal 220 Display device 222 screens 250 Information Processing Equipment 300 Management device P projection plane CAM camera CINF Camera Information IMG image IMG1 1st image IMG2 2nd image

Claims

1. A remote operator terminal used by a remote operator for remote operation of a mobile object, comprising: the first image is an image captured at a first timing by a camera mounted on the moving object, a first viewpoint is defined by a combination of the position and orientation of the camera at the first timing; the second viewpoint is defined by a combination of the position and orientation of the camera predicted at a second timing that is later than the first timing; The projective transformation process transforms the first image viewed from the first viewpoint into a second image viewed from the second viewpoint; the remote operator terminal comprises an information processing device; The information processing device includes: acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at the first timing; obtaining a plurality of second images by applying the projective transformation process to each of the plurality of first images; estimating the gaze direction of the remote operator; The plurality of second images are displayed side by side on the display device of the remote operator terminal so that a priority second image corresponding to the gaze direction among the plurality of second images appears to be closer than the other second images. It was configured as Remote operator terminal.

2. 2. A remote operator terminal according to claim 1, the enlarged second image is the priority second image or a second image displayed next to the priority second image, the enlarged first image is a first image before the projective transformation process, which is a source of the enlarged second image; The information processing device is further configured to adjust the plurality of first images so that the enlarged first image is larger than the other first images before the projective transformation process. Remote operator terminal.

3. 3. A remote operator terminal according to claim 2, comprising: the angle of view of each of the plurality of cameras includes a central angle of view and a peripheral angle of view around the central angle of view, the central region is an image region corresponding to the central angle of view, the peripheral region is an image region corresponding to the peripheral angle of view, The information processing device is configured to adjust the plurality of first images so that the enlarged first image includes both the central region and the peripheral region, and so that the first images other than the enlarged first image include only the central region. Remote operator terminal.

4. A remote operator terminal used by a remote operator for remote operation of a mobile object, comprising: the first image is an image captured at a first timing by a camera mounted on the moving object, a first viewpoint is defined by a combination of the position and orientation of the camera at the first timing; the second viewpoint is defined by a combination of the position and orientation of the camera predicted at a second timing that is later than the first timing; The projective transformation process transforms the first image viewed from the first viewpoint into a second image viewed from the second viewpoint; the remote operator terminal comprises an information processing device; The information processing device includes: acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at the first timing; obtaining a plurality of second images by applying the projective transformation process to each of the plurality of first images; The plurality of second images are displayed side by side on a display device of the remote operator terminal. It is configured as follows: the priority second image is one of the plurality of second images corresponding to the gaze direction of the remote operator, the enlarged second image is the priority second image or a second image displayed next to the priority second image, the enlarged first image is a first image before the projective transformation process, which is a source of the enlarged second image; The information processing device is further configured to adjust the plurality of first images so that the enlarged first image is larger than the other first images before the projective transformation process. Remote operator terminal.

5. 5. A remote operator terminal according to claim 4, comprising: the angle of view of each of the plurality of cameras includes a central angle of view and a peripheral angle of view around the central angle of view, the central region is an image region corresponding to the central angle of view, the peripheral region is an image region corresponding to the peripheral angle of view, The information processing device is configured to adjust the plurality of first images so that the enlarged first image includes both the central region and the peripheral region, and so that the first images other than the enlarged first image include only the central region. Remote operator terminal.

6. A remote operator terminal according to any one of claims 1 to 5, The information processing device is further configured to estimate the gaze direction of the remote operator based on the line of sight of the remote operator or the steering angle of a steering operation by the remote operator. Remote operator terminal.

7. A remote operator terminal according to any one of claims 1 to 5, The projective transformation process is estimating a moving direction and a moving amount of the moving object during a period from the first timing to the second timing; calculating a difference between the first viewpoint and the second viewpoint based on the movement direction and the movement amount of the moving object; converting the first image viewed from the first viewpoint into a second image viewed from the second viewpoint based on the difference between the first viewpoint and the second viewpoint; Contains Remote operator terminal.

8. A remote operator terminal according to any one of claims 1 to 5, The information processing device further comprises: setting at least a part of a communication delay time between the mobile unit and the remote operator terminal as a delay compensation time; The second timing is set to be later than the first timing by the delay compensation time. It was configured as Remote operator terminal.

9. 1. An image display method for displaying an image for a remote operator during remote operation of a mobile object, comprising: the first image is an image captured at a first timing by a camera mounted on the moving object, a first viewpoint is defined by a combination of the position and orientation of the camera at the first timing; the second viewpoint is defined by a combination of the position and orientation of the camera predicted at a second timing that is later than the first timing; The projective transformation process transforms the first image viewed from the first viewpoint into a second image viewed from the second viewpoint; The image display method includes: acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at the first timing; obtaining a plurality of second images by applying the projective transformation process to each of the plurality of first images; estimating the gaze direction of the remote operator; displaying the plurality of second images side by side on a display device of the remote operator terminal so that a priority second image corresponding to the gaze direction among the plurality of second images appears to be closer than the other second images; Contains Image display method.

10. A remote operation system for remotely operating a mobile object by a remote operator, comprising: the first image is an image captured at a first timing by a camera mounted on the moving object, a first viewpoint is defined by a combination of the position and orientation of the camera at the first timing; the second viewpoint is defined by a combination of the position and orientation of the camera predicted at a second timing that is later than the first timing; The projective transformation process transforms the first image viewed from the first viewpoint into a second image viewed from the second viewpoint; the remote control system comprises one or more processors; the one or more processors: acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at the first timing; obtaining a plurality of second images by applying the projective transformation process to each of the plurality of first images; estimating the gaze direction of the remote operator; The plurality of second images are displayed side by side on the display device of the remote operator terminal so that a priority second image corresponding to the gaze direction among the plurality of second images appears to be closer than the other second images. It was configured as Remote control system.

Images