Remote operator terminal, image display method, and remote control system

Abstract

To provide a remote operator terminal capable of improving the image quality of displaying multiple images captured by multiple cameras in remote operation of a movable body.SOLUTION: A first image is captured by a camera mounted on a moving object at first designated time. A first viewpoint is a camera viewpoint at a first point of time, while a second viewpoint is a camera viewpoint at a second point of time later than the first point of time. In projection transformation processing, a first image taken from the first viewpoint is converted into a second image taken from the second viewpoint on the basis of through-view projection conversion using a projection plane corresponding to the camera. When multiple first images are captured by each of the multiple cameras mounted on a moving object, a series of projection transformation processing using a common projection plane is applied to the combination of those multiple first images. The combined second images obtained through the projection transformation processing are displayed on the remote operator terminal.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to remote control of a mobile object by a remote operator, and more particularly to delay compensation in remote control. [Background technology]

[0002] In remote control of a vehicle, images (video) captured by an on-board camera are transmitted from the vehicle to a remote operator terminal and displayed on a display device of the remote operator terminal. At this time, it is desirable to compensate for communication delays between the vehicle and the remote operator terminal. The following technologies are known as those related to image (video) delay compensation.

[0003] Patent Document 1 discloses a remote video output system. The remote video output system includes an autonomous vehicle that transmits video and a remote video output device that receives video from the autonomous vehicle. The remote video output device estimates a change in the viewpoint position of the autonomous vehicle according to the communication delay time from the autonomous vehicle to the remote video output device. The remote video output device then takes into account the change in the viewpoint position of the autonomous vehicle, cuts out a portion of the frame of the received video, and displays the video of the cut-out portion.

[0004] Non-Patent Document 1 discloses a delay compensation technique that uses projective transformation. More specifically, this delay compensation technique performs projective transformation on images received from a vehicle and compensates for the delay by changing the viewpoint according to the amount of vehicle movement equivalent to the delay time. In other words, this delay compensation technique visually compensates for the delay by looking ahead at camera images seen from a viewpoint that is the delay time ahead. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 155159 [Non-patent literature]

[0006] [Non-Patent Document 1] Kodai Matsubara and Manabu Ohmae, "Research on delay compensation of camera images for remotely controlled automobiles using projective transformation," 19th ITS Symposium 2021, 4-A-12, December 2021 Summary of the Invention [Problem to be solved by the invention]

[0007] In remote control of a moving object, it is possible to perform delay compensation for images. However, the above-mentioned conventional technology does not consider delay compensation for multiple images taken by multiple cameras. Further study and improvement are desired regarding delay compensation for multiple images taken by multiple cameras.

[0008] As an example, consider a case where multiple images are each subjected to projective transformation and the multiple transformed images are displayed side by side. In this case, there is a risk that overlaps or gaps will occur between adjacent transformed images, which may cause a sense of incongruity to the remote operator. [Means for solving the problem]

[0009] A first aspect relates to a remote operator terminal used by a remote operator for remote operation of a mobile object. The first image is an image captured at a first timing by a camera mounted on a moving object. The first viewpoint is defined by the combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera that is predicted at a second timing that is later than the first timing. The projective transformation process transforms a first image seen from a first viewpoint into a second image seen from a second viewpoint based on perspective projection transformation using a projection plane associated with the camera. The remote operator terminal includes an information processing device. The information processing device acquires a plurality of first images captured by a plurality of cameras mounted on a moving object at a first timing. The information processing device acquires a combination of a plurality of second images by applying a projective transformation process using a common projection plane to a combination of a plurality of first images. The information processing device displays a combination of the plurality of second images on a display device of the remote operator terminal.

[0010] The second aspect relates to an image display method for displaying an image for a remote operator when remotely operating a mobile object. The first image is an image captured at a first timing by a camera mounted on a moving object. The first viewpoint is defined by the combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera that is predicted at a second timing that is later than the first timing. The projective transformation process transforms a first image seen from a first viewpoint into a second image seen from a second viewpoint based on perspective projection transformation using a projection plane associated with the camera. The image display method is acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at a first timing; acquiring a combination of a plurality of second images by applying a projective transformation process using a common projection plane to a combination of a plurality of first images; displaying the combination of the plurality of second images on a display device of a remote operator terminal used by the remote operator; Includes:

[0011] A third aspect relates to a remote control system for remotely controlling a moving object by a remote operator. The first image is an image captured at a first timing by a camera mounted on a moving object. The first viewpoint is defined by the combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera that is predicted at a second timing that is later than the first timing. The projective transformation process transforms a first image seen from a first viewpoint into a second image seen from a second viewpoint based on perspective projection transformation using a projection plane associated with the camera. The remote control system includes one or more processors. The one or more processors acquire a plurality of first images taken by each of a plurality of cameras mounted on the moving object at a first timing. The one or more processors acquire a combination of a plurality of second images by applying a projective transformation process using a common projection plane to a combination of a plurality of first images. The one or more processors display a combination of the plurality of second images on a display device of a remote operator terminal used by the remote operator. [Effects of the Invention]

[0012] According to the present disclosure, a projective transformation process is performed on a plurality of first images captured by a plurality of cameras by deliberately using a common projection plane instead of different projection planes. The plurality of second images obtained by such a projective transformation process can be displayed side by side without overlaps or gaps. As a result, the appearance of the displayed plurality of second images is improved, and the sense of incongruity felt by the remote operator is reduced. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a remote control system. [Figure 2] FIG. 10 is a conceptual diagram for explaining an overview of delay compensation processing. [Figure 3] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 4] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 5] FIG. 10 is a conceptual diagram for explaining an example of image deformation by projective transformation. [Figure 6] FIG. 10 is a conceptual diagram for explaining a first example of projective transformation processing in the case of multiple cameras. [Figure 7] FIG. 10 is a conceptual diagram for explaining a first example of projective transformation processing in the case of multiple cameras. [Figure 8] FIG. 10 is a conceptual diagram for explaining a second example of projective transformation processing in the case of multiple cameras. [Figure 9] FIG. 10 is a conceptual diagram for explaining a second example of projective transformation processing in the case of multiple cameras. [Figure 10] FIG. 10 is a conceptual diagram for explaining a third example of projective transformation processing in the case of multiple cameras. [Figure 11] FIG. 10 is a conceptual diagram for explaining a third example of projective transformation processing in the case of multiple cameras. [Figure 12] FIG. 10 is a conceptual diagram for explaining a third example of projective transformation processing in the case of multiple cameras. [Figure 13] FIG. 1 is a block diagram showing an example of the configuration of a vehicle. [Figure 14] FIG. 2 is a block diagram showing an example of the configuration of a remote operator terminal. DETAILED DESCRIPTION OF THE INVENTION

[0014] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0015] 1.Remote control system Consider the remote control (remote driving) of a moving object. Examples of moving objects that can be remotely controlled include vehicles, robots, flying objects, etc. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones, etc. As an example, in the following explanation, consider a case where the moving object is a vehicle. When generalizing, "vehicle" in the following explanation should be read as "moving object."

[0016] FIG. 1 is a schematic diagram showing an example of the configuration of a remote operation system 1 according to this embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote operation. The remote operator terminal 200 is a terminal device used by a remote operator O to remotely operate the vehicle 100. The remote operator terminal 200 can also be called a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. Typically, the management device 300 is a management server on the cloud. The management server may be composed of multiple servers that perform distributed processing.

[0017] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 may communicate directly without going through the management device 300.

[0018] The vehicle 100 is equipped with various sensors including a camera CAM. The camera CAM photographs the surroundings of the vehicle 100 and acquires images (video) IMG showing the situation around the vehicle 100. The sensor detection information SEN includes information obtained by the various sensors. The sensor detection information SEN includes at least the images IMG photographed by the camera CAM. The sensor detection information SEN may include the position and status of the vehicle 100 (e.g., speed, steering angle, etc.). The vehicle 100 transmits the sensor detection information SEN to the remote operator terminal 200.

[0019] The remote operator terminal 200 receives the sensor detection information SEN transmitted from the vehicle 100. The remote operator terminal 200 presents the sensor detection information SEN to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device 220, and displays information such as an image IMG on the display device 220. The remote operator O looks at the displayed information, recognizes the situation around the vehicle 100, and remotely controls the vehicle 100. In other words, by displaying information for the remote operator O on the display device 220, the remote operator O's remote operation of the vehicle 100 is assisted.

[0020] The remote operation information OPE is information related to remote operation by a remote operator O. For example, the remote operation information OPE includes the amount of operation by the remote operator O. The remote operator terminal 200 transmits the remote operation information OPE to the vehicle 100. The vehicle 100 receives the remote operation information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle driving control in accordance with the received remote operation information OPE. In this way, remote operation of the vehicle 100 is realized.

[0021] 2. Delay compensation processing using projective transformation Remote operation of the vehicle 100 involves a communication delay between the vehicle 100 and the remote operator terminal 200. The communication delay may destabilize the behavior of the vehicle 100 during remote operation. The communication delay may also result in a decrease in the accuracy of the remote operation of the vehicle 100. Therefore, it is important to perform delay compensation when remotely operating the vehicle 100.

[0022] The remote operation system 1 according to this embodiment performs visual delay compensation for the image IMG displayed on the display device 220, taking communication delays into consideration. In particular, the remote operation system 1 according to this embodiment performs visual delay compensation for the image IMG by using "projective transformation." The subject of the delay compensation process is, for example, the remote operator terminal 200. However, the subject of the delay compensation process is not limited to the remote operator terminal 200. At least a part of the delay compensation process may be performed by the vehicle 100 or the management device 300.

[0023] 2 is a conceptual diagram for explaining an outline of the delay compensation process by the remote operation system 1. The first image IMG1 is an image IMG that is actually captured at a first timing T1 by a camera CAM mounted on the vehicle 100. The first image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 acquires the first image IMG1 after the first timing T1. If it is possible to estimate (predict) an image IMG that will be captured in the future from the first image IMG1, it becomes possible to perform delay compensation.

[0024] The second timing T2 is a target timing for look-ahead and is later than the first timing T1. The difference between the second timing T2 and the first timing T1 corresponds to the "delay compensation time." The remote operation system 1 may set at least a part of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200 as the delay compensation time. The communication delay time between the vehicle 100 and the remote operator terminal 200 can be estimated using well-known techniques. The delay compensation time may be set to the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The delay compensation time may be set to the communication time required for the image IMG to be transmitted from the vehicle 100 and reach the remote operator terminal 200. Alternatively, the delay compensation time may be set to a constant value. In either case, the remote operation system 1 sets the second timing T2 to be later than the first timing T1 by the delay compensation time.

[0025] For convenience, the camera CAM at the first timing T1 will be referred to as the first camera CAM1, and the camera CAM at the second timing T2 will be referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the combination of the position and orientation of the first camera CAM1 at the first timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the combination of the predicted position and orientation of the second camera CAM2 at the second timing T2.

[0026] The remote control system 1 acquires camera information CINF related to the camera CAM mounted on the vehicle 100. The camera information CINF includes installation information and performance information of the camera CAM. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. Because the camera CAM is fixed to the vehicle 100, by using the installation information of the camera CAM, the direction and amount of movement of the vehicle 100 can be converted into the direction and amount of movement of the camera CAM in the camera coordinate system. In other words, the change in the viewpoint of the camera CAM can be estimated based on the installation information of the camera CAM and the direction and amount of movement of the vehicle 100.

[0027] More specifically, the remote control system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 (i.e., the delay compensation time). For example, the remote control system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 based on the speed and steering angle of the vehicle 100 at the first timing T1 and the delay compensation time. Information on the speed and steering angle of the vehicle 100 is obtained from sensor detection information SEN provided by the vehicle 100. Alternatively, the steering angle in the steering operation by the remote operator O may be considered to be the steering angle of the vehicle 100. The vehicle 100 may be assumed to make a steady circular turn. Then, the remote control system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information CINF (installation information) and the amount and direction of movement of the vehicle 100 during the delay compensation time.

[0028] The first image IMG1 can be said to be an image IMG captured from the first viewpoint, i.e., an image IMG viewed from the first viewpoint. An image IMG expected to be captured from the second viewpoint, i.e., an image IMG expected to be seen from the second viewpoint, will hereinafter be referred to as a "second image IMG2." The remote control system 1 converts the first image IMG1 viewed from the first viewpoint into a second image IMG2 viewed from the second viewpoint based on the difference between the first and second viewpoints. In other words, the remote control system 1 predicts (reads ahead) the second image IMG2 viewed from the second viewpoint based on the first image IMG1 viewed from the first viewpoint. Projective transformation is used for this look-ahead.

[0029] FIG. 3 is a conceptual diagram for explaining projective transformation. Projective transformation is performed based on perspective projection transformation. Perspective projection transformation is a rendering technique for rendering an object in three-dimensional space on a two-dimensional plane as seen from the camera CAM. To achieve this, perspective projection transformation projects points in three-dimensional space onto a projection plane P, taking into account the viewpoint of the camera CAM. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane perpendicular to the optical axis of the camera CAM. Note that points in three-dimensional space are defined in a three-dimensional world coordinate system (absolute coordinate system). On the other hand, points projected onto the projection plane P are defined in a two-dimensional image coordinate system.

[0030] For example, N virtual points are virtually set in a three-dimensional world coordinate system. N is an integer equal to or greater than 4. The N virtual points as viewed from a first camera CAM1 (first viewpoint) are projected onto a first projection plane P1 associated with the first camera CAM1 by perspective projection transformation. The N virtual points as viewed from a second camera CAM2 (second viewpoint) are projected onto a second projection plane P2 associated with the second camera CAM2 by perspective projection transformation. The second viewpoint is obtained from the difference between the first viewpoint and the second viewpoint. The image coordinates of the virtual points on the first projection plane P1 as viewed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the virtual points on the second projection plane P2 as viewed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on a comparison of the two, a projection transformation matrix H for converting from the first viewpoint to the second viewpoint is calculated. Then, the projective transformation matrix H is applied to the entire first image IMG1 actually captured by the first camera CAM1, thereby generating a second image IMG2 that is expected to be seen from the second viewpoint.

[0031] As another example, the method described in Non-Patent Document 1 may be used. Specifically, by inverse transformation of the perspective projection transformation, each image coordinate point on the first image IMG1 (projection plane P) is transformed into a world coordinate point in the world coordinate system. Based on the difference between the first viewpoint and the second viewpoint, the world coordinate point as seen from the first viewpoint is transformed into a world coordinate point as seen from the second viewpoint. Then, by perspective projection transformation, the world coordinate point as seen from the second viewpoint is returned onto the projection plane P. In this way, a second image IMG2 that is expected to be seen from the second viewpoint is generated. Note that according to Non-Patent Document 1, it is assumed that the ground surface S is reflected across the entire image IMG, as shown in FIG. 4.

[0032] FIG. 5 is a conceptual diagram for explaining an example of image deformation by projective transformation (see Non-Patent Document 1). The first image IMG1 is the original image before projective transformation. The second image IMG2 is generated by applying projective transformation to the first image IMG1. FIG. 5 shows how the first image IMG1 and the second image IMG2 appear on the screen 222 of the display device 220. For example, when the vehicle 100 travels straight, the second image IMG2 appears to be tilted backward. As another example, when the vehicle turns right, the second image IMG2 appears to be tilted to the left.

[0033] 3. Projection transformation processing for multiple cameras In the following description, "projective transformation processing" refers to delay compensation processing using the projective transformation described in Section 2 above. The projective transformation processing converts a first image IMG1 viewed from a first viewpoint into a second image IMG2 viewed from a second viewpoint based on perspective projection transformation using a projection plane P associated with the camera CAM. The remote operation system 1 can perform delay compensation by applying projective transformation processing to the image IMG captured by the camera CAM. The subject of the projective transformation processing is, for example, the remote operator terminal 200. However, the subject of the projective transformation processing is not limited to the remote operator terminal 200. At least a part of the projective transformation processing may be executed by the vehicle 100 or the management device 300.

[0034] Here, consider a case where multiple cameras CAM are mounted on the vehicle 100. The installation orientations of the multiple cameras CAM in the vehicle coordinate system are different from each other. For example, the multiple cameras CAM include a front camera CAM-F for capturing images of the front, a left camera CAM-L for capturing images of the left front, and a right camera CAM-R for capturing images of the right front. However, the components of the multiple cameras CAM are not limited to this.

[0035] The remote operator terminal 200 acquires a plurality of images IMG captured by each of a plurality of cameras CAM mounted on the vehicle 100. The remote operator terminal 200 then displays the plurality of images IMG on the display device 220. The remote operator terminal 200 may display the plurality of images IMG side by side on one or more screens 222 of the display device 220.

[0036] Hereinafter, the projective transformation process for a plurality of images IMG captured by a plurality of cameras CAM will be considered.

[0037] 3-1. First example 6 is a conceptual diagram for explaining a first example. Projection planes PF, PL, and PR are projection planes P associated with the front camera CAM-F, left camera CAM-L, and right camera CAM-R, respectively. For example, projection plane PF is orthogonal to the optical axis of the front camera CAM-F, projection plane PL is orthogonal to the optical axis of the left camera CAM-L, and projection plane PR is orthogonal to the optical axis of the right camera CAM-R. The projection planes PF, PL, and PR are different from one another.

[0038] The multiple cameras CAM (CAM-F, CAM-L, CAM-R) are installed in different orientations. Therefore, as shown in Fig. 6, when the vehicle 100 moves in a certain direction, the change in the viewpoint of the camera CAM relative to the projection plane P differs between the multiple cameras CAM (CAM-F, CAM-L, CAM-R).

[0039] The remote control system 1 acquires a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R) captured by a plurality of cameras CAM (CAM-F, CAM-L, CAM-R) at a first timing T1. The remote control system 1 acquires a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) by applying a projective transformation process to each of the plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). In a first example, a plurality of different projection planes P (PF, PL, PR) are used for each of the plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). That is, the remote control system 1 generates a second image IMG2-F by applying a projective transformation process using the projection plane PF to the first image IMG1-F captured by the front camera CAM-F. The remote control system 1 also generates a second image IMG2-L by applying a projective transformation process using a projection plane PL to a first image IMG1-L captured by the left camera CAM-L. Furthermore, the remote control system 1 generates a second image IMG2-R by applying a projective transformation process using a projection plane PR to a first image IMG1-R captured by the right camera CAM-R.

[0040] Fig. 7 shows an example of displaying a plurality of images IMG in the first example. In the example shown in Fig. 7, the display device 220 includes a plurality of screens 222-F, 222-L, and 222-R. The plurality of screens 222-F, 222-L, and 222-R are arranged adjacent to each other in a row. More specifically, the screens 222-L and 222-R are arranged on either side of the screen 222-F, and the screen 222-F is sandwiched between the screens 222-L and 222-R.

[0041] If projective transformation processing is not performed, the remote operator terminal 200 displays multiple first images IMG1 (IMG1-F, IMG1-L, IMG1-R) on each of multiple screens 222 (222-F, 222-L, 222-R) of the display device 220.

[0042] When the projective transformation process is performed, the remote operator terminal 200 acquires a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) from a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). In the example shown in FIG. 7, the vehicle 100 is turning right. The remote operator terminal 200 then displays the plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) on the display device 220. For example, the remote operator terminal 200 displays the plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) side by side on one or more screens 222 of the display device 220. The remote operator terminal 200 may display multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) side by side on multiple screens 222 (222-F, 222-L, 222-R). In this case, the second image IMG2-F is generally arranged on the screen 222-F, but a portion of the second image IMG2-F may extend onto the screens 222-L and 222-R adjacent to the screen 222-F. Similarly, the second image IMG2-L is generally arranged on the screen 222-L, but a portion of the second image IMG2-L may extend onto the screen 222-F adjacent to the screen 222-L. Similarly, the second image IMG2-R is generally arranged on the screen 222-R, but a portion of the second image IMG2-R may extend onto the screen 222-F adjacent to the screen 222-R.

[0043] 3-2. Second example In the first example described above, in the projective transformation process, different projection planes P are used for the multiple first images IMG1. As a result, the shapes of the multiple second images IMG2 obtained do not necessarily match. Therefore, when the multiple second images IMG2 are displayed side by side on the display device 220, the boundaries of adjacent second images IMG2 may not match, and overlaps or gaps may occur between the adjacent second images IMG2. This may deteriorate the appearance of the multiple second images IMG2 and cause the remote operator O to feel uncomfortable.

[0044] Therefore, the second example proposes a technique that can suppress the sense of discomfort felt by the remote operator O. Note that explanations that overlap with the first example above will be omitted as appropriate.

[0045] 8 is a conceptual diagram for explaining the second example. In the second example, in the projective transformation process, the same projection plane P is used for the multiple first images IMG1, rather than different projection planes P. Hereinafter, this same projection plane P will be referred to as a "common projection plane P-COM."

[0046] For example, the common projection plane P-COM is any one of the projection planes PF, PL, and PR. In this case, the remote control system 1 selects one of the multiple cameras CAM (CAM-F, CAM-L, CAM-R) as the reference camera. Then, the remote control system 1 sets the projection plane P associated with the reference camera as the common projection plane P-COM. In the example shown in FIG. 8, the projection plane PF associated with the front camera CAM-F is set as the common projection plane P-COM.

[0047] The remote control system 1 applies a projective transformation process using a common projection plane P-COM (reference camera) to a combination of a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R) to obtain a combination of a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R). This is equivalent to assuming that all cameras CAM are facing in the same direction as the reference camera.

[0048] For example, the remote control system 1 may concatenate multiple first images IMG1 (IMG1-F, IMG1-L, IMG1-R) in advance to generate a single large first image IMG1 (see FIG. 9). The first images IMG1-L and IMG1-R are arranged on either side of the first image IMG1-F, and the first image IMG1-F is sandwiched between the first images IMG1-L and IMG1-R. The remote control system 1 then applies a projective transformation process using a common projection plane P-COM to the single large first image IMG1 to obtain a single large second image IMG2. The single large second image IMG2 corresponds to a combination of multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R).

[0049] As another example, the remote operation system 1 may acquire multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) by applying a projective transformation process using a common projection plane P-COM to each of multiple first images IMG1 (IMG1-F, IMG1-L, IMG1-R). The remote operation system 1 then concatenates the multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) to acquire one large second image IMG2. The second images IMG2-L and IMG2-R are arranged on either side of the second image IMG2-F, and the second image IMG2-F is sandwiched between the second images IMG2-L and IMG2-R. One large second image IMG2 corresponds to a combination of the multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R).

[0050] FIG. 9 shows a display example of multiple images IMG in the second example. As in FIG. 7, the vehicle 100 is turning right. When the projective transformation process is performed, the remote operator terminal 200 displays a combination of multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) on the display device 220. For example, the remote operator terminal 200 displays multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) side by side on one or more screens 222 of the display device 220. The remote operator terminal 200 may also display multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) side by side on multiple screens 222 (222-F, 222-L, 222-R). As shown in FIG. 9, when multiple second images IMG2 are displayed side by side on the display device 220, the boundaries of adjacent second images IMG2 coincide with each other. Therefore, overlaps and gaps between adjacent second images IMG2 are suppressed. In other words, it is possible to display a plurality of second images IMG2 side by side without overlaps or gaps.

[0051] As described above, according to the second example, the projective transformation process is performed on the multiple first images IMG1 captured by the multiple cameras CAM by deliberately using a common projection plane P-COM instead of different projection planes P. The multiple second images IMG2 obtained by such projective transformation process can be displayed side by side without overlaps or gaps. As a result, the appearance of the multiple displayed second images IMG2 is improved, and the sense of incongruity felt by the remote operator O is reduced.

[0052] 3-3. Third Example In the second example described above, there are some areas where the accuracy of the projective transformation process is not necessarily high. For example, in the example shown in FIG. 8, the projection plane PF associated with the front camera CAM-F is used as the common projection plane P-COM. For the first image IMG1-F captured by the front camera CAM-F, the common projection plane P-COM is the same as the original projection plane PF, so the accuracy of the projective transformation process is high. However, for the first image IMG1-L captured by the left camera CAM-L, a common projection plane P-COM different from the original projection plane PL is used, so the accuracy of the projective transformation process is not necessarily high. Similarly, for the first image IMG1-R captured by the right camera CAM-R, a common projection plane P-COM different from the original projection plane PR is used, so the accuracy of the projective transformation process is not necessarily high.

[0053] If the accuracy of the projective transformation process decreases, the accuracy of the second image IMG2 generated by the projective transformation process also decreases. For example, when turning right, the remote operator O is likely to focus mainly on the second image IMG2-R on the right side, but if the accuracy of the second image IMG2-R is low, the accuracy of the remote operation may also decrease.

[0054] However, conversely, as long as the accuracy of the second image IMG2 that the remote operator O gazes at is at least high, the accuracy of the remote operation is ensured even if the accuracy of the other second images IMG2 is low. Based on this viewpoint, in the third example, the common projection plane P-COM is dynamically set in consideration of the gaze direction of the remote operator O. In other words, the common projection plane P-COM is dynamically switched in conjunction with the gaze direction of the remote operator O.

[0055] 10 is a conceptual diagram illustrating the correspondence between the gaze direction of the remote operator O and the common projection plane P-COM. When the gaze direction of the remote operator O is within the gaze direction range RNG-F, the remote operation system 1 selects the front camera CAM-F as the reference camera and the projection plane PF as the common projection plane P-COM. When the gaze direction of the remote operator O is within the gaze direction range RNG-L, the remote operation system 1 selects the left camera CAM-L as the reference camera and the projection plane PL as the common projection plane P-COM. When the gaze direction of the remote operator O is within the gaze direction range RNG-R, the remote operation system 1 selects the right camera CAM-R as the reference camera and sets the projection plane PR as the common projection plane P-COM.

[0056] The gaze direction of the remote operator O is estimated based on, for example, the steering angle of the steering operation by the remote operator O. In this case, a predetermined steering angle range θ-F including straight ahead is associated with the gaze direction range RNG-F. A steering angle range θ-R to the right of the predetermined steering angle range θ-F is associated with the gaze direction range RNG-R. A steering angle range θ-L to the left of the predetermined steering angle range θ-F is associated with the gaze direction range RNG-L. The remote operation system 1 estimates the gaze direction of the remote operator O based on the steering angle of the steering operation by the remote operator O. More specifically, the remote operation system 1 selects a steering angle range θ-X (X=ForLorR) including the steering angle of the steering operation by the remote operator O. Furthermore, the remote operation system 1 estimates that the gaze direction of the remote operator O is included in the gaze direction range RNG-X associated with the selected steering angle range θ-X. Then, the remote control system 1 selects the camera CAM-X corresponding to the gaze direction range RNG-X as the reference camera.

[0057] As another example, the gaze direction of the remote operator O may be estimated from the gaze direction of the remote operator O. More specifically, the gaze direction of the remote operator O is detected by the operator monitor 240 (described later) of the remote operator terminal 200. The detected gaze direction of the remote operator O is considered to be the gaze direction of the remote operator O. The remote operation system 1 selects, as the reference camera, a camera CAM-X corresponding to a gaze direction range RNG-X that includes the gaze direction (gaze direction) of the remote operator O.

[0058] In this way, the remote operation system 1 estimates the gaze direction of the remote operator O and dynamically selects one of the multiple cameras CAM that corresponds to the gaze direction as the reference camera. Then, the remote operation system 1 sets the projection plane P associated with the selected reference camera as the common projection plane P-COM.

[0059] 11 shows the common projection plane P-COM when turning right. When turning right, the gaze direction of the remote operator O is included in the gaze direction range RNG-R. Therefore, the right camera CAM-R is selected as the reference camera, and the projection plane PR is set as the common projection plane P-COM.

[0060] FIG. 12 shows an example of displaying multiple images IMG in the example shown in FIG. 11. When turning right, the remote operator O mainly looks at the second image IMG2-R on the right side. Because the projection plane PR associated with the right camera CAM-R is used as the common projection plane P-COM, the accuracy of the second image IMG2-R is at least high. Because the accuracy of the second image IMG2-R that the remote operator O looks at is at least high, the accuracy of remote operation is ensured even if the accuracy of the other second images IMG2 is low.

[0061] As described above, according to the third example, the common projection plane P-COM is dynamically set in consideration of the gaze direction of the remote operator O. This ensures at least the accuracy of the second image IMG2 that the remote operator O gazes at. As a result, the accuracy of remote operation is also ensured.

[0062] 4. Example of vehicle configuration 4-1.Configuration example 13 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0063] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300.

[0064] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera (CAM), a LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0065] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates braking force.

[0066] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processors 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processors 160 perform various processes. Examples of the processors 160 include a general-purpose processor, a specific-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, and / or a combination thereof. The storage device 170 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), a solid-state drive (SSD), etc. The control device 150 may include one or more electronic control units (ECUs). In general terms, the control device 150 can also be called a processing circuitry.

[0067] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 may be realized by cooperation between the processor 160, which executes the vehicle control program PROG1, and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0068] 4-2. Sensor detection information The control device 150 acquires sensor detection information SEN using the sensor group 120. The sensor detection information SEN is stored in the storage device 170. The sensor detection information SEN includes an image IMG, vehicle state information, position information, object information, etc. The image IMG is captured by a camera CAM. The vehicle state information indicates the state of the vehicle 100 (e.g., speed, steering angle, etc.) detected by the vehicle state sensor. The position information indicates the position and orientation of the vehicle 100 detected by the position sensor.

[0069] The object information is information about objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, bicycles, motorcycles, other vehicles (leading vehicles, vehicles running alongside, following vehicles, etc.), white lines, road structures (e.g., curbs, guardrails), poles, traffic lights, signs, etc. The control device 150 can recognize objects around the vehicle 100 by using a recognition sensor. For example, by analyzing an image (IMG), it is possible to identify an object and calculate the relative position of the object. It is also possible to identify an object and obtain the relative position and relative speed of the object based on point cloud information obtained by LIDAR. The object information includes the relative position of the object with respect to the vehicle 100. The object information may further include the relative speed of the object.

[0070] 4-3.Vehicle driving control The control device 150 executes vehicle driving control to control the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes vehicle driving control by controlling the driving device 130 (steering device, drive device, and brake device).

[0071] The control device 150 may perform automatic driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the sensor detection information SEN. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0072] 4-4. Processing related to remote control When the vehicle 100 is remotely operated, the control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0073] The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. Typically, the control device 150 transmits an image IMG to the remote operator terminal 200. The control device 150 may transmit vehicle state information to the remote operator terminal 200. The control device 150 may transmit object information to the remote operator terminal 200.

[0074] Furthermore, the control device 150 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information related to remote operation by the remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle travel control in accordance with the received remote operation information OPE.

[0075] 4-5.Camera information The camera information CINF includes installation information and performance information for each of one or more camera CAMs mounted on the vehicle 100. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. The camera information CINF is stored in the storage device 170. The control device 150 may transmit the camera information CINF to the remote operator terminal 200.

[0076] 5. Example of remote operator terminal configuration 14 is a block diagram showing an example of the configuration of the remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, an operator monitor 240, and an information processing device 250.

[0077] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0078] The display device 220 displays various information for the remote operator O who performs remote operation. In other words, the display device 220 displays various information to present the information to the remote operator O. The display device 220 includes a plurality of screens 222.

[0079] The input device 230 is a member that the remote operator O operates when remotely operating the vehicle 100. For example, the input device 230 includes remote operation members, such as a steering wheel, an accelerator pedal, a brake pedal, and turn signals.

[0080] The operator monitor 240 includes a sensor for monitoring the state of the remote operator O. For example, the operator monitor 240 includes a camera that captures the face and eyes of the remote operator O. The operator monitor 240 extracts images of the face and eyes of the remote operator O by analyzing the image of the remote operator O captured by the camera. The extraction of the face and eye images is performed, for example, by using a machine learning model that has been generated in advance through machine learning. The operator monitor 240 then recognizes the gaze direction of the remote operator O based on the position and rotation angle of the pupils of each eye.

[0081] The information processing device 250 controls the remote operator terminal 200. The information processing device 250 includes one or more processors 260 (hereinafter simply referred to as processors 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processor 260 executes various processes. Examples of the processor 260 include a general-purpose processor, a special-purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, and / or a combination thereof. The storage device 270 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a HDD, an SSD, etc. In general terms, the information processing device 250 can also be called a processing circuitry.

[0082] The remote operation control program PROG2 is a computer program executed by the processor 260. The functions of the information processing device 250 may be realized by cooperation between the processor 260 executing the remote operation control program PROG2 and the storage device 270. The remote operation control program PROG2 is stored in the storage device 270. Alternatively, the remote operation control program PROG2 may be recorded on a computer-readable recording medium. The remote operation control program PROG2 may be provided via a network.

[0083] The information processing device 250 communicates with the vehicle 100 via the communication device 210. The information processing device 250 receives sensor detection information SEN transmitted from the vehicle 100. The information processing device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the information processing device 250 presents an image IMG to the remote operator O by displaying the image IMG on the display device 220. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the presented information.

[0084] The remote operator O operates the input device 230. The amount of operation of the input device 230 is detected by a sensor installed on the input device 230. The information processing device 250 generates remote operation information OPE that reflects the amount of operation of the input device 230 by the remote operator O. Then, the information processing device 250 transmits the remote operation information OPE to the vehicle 100 via the communication device 210.

[0085] The information processing device 250 may receive the camera information CINF transmitted from the vehicle 100. The camera information CINF is stored in the storage device 270.

[0086] The information processing device 250 performs the projective transformation processing (delay compensation processing) described in Sections 2 and 3 above. The speed and steering angle of the vehicle 100 are obtained from the sensor detection information SEN. The steering angle in the steering operation by the remote operator O may be considered as the steering angle of the vehicle 100. Installation information and performance information of each camera CAM mounted on the vehicle 100 are obtained from the camera information CINF. The gaze direction of the remote operator O can be estimated from the amount of steering operation by the remote operator O. The gaze direction of the remote operator O can also be estimated from the line of sight of the remote operator O recognized by the operator monitor 240. The information processing device 250 performs the projective transformation processing (delay compensation processing) described in Sections 2 and 3 above based on this information. [Explanation of symbols]

[0087] 1. Remote control system 100 vehicles 200 Remote Operator Terminal 220 Display device 222 screens 250 Information Processing Equipment 300 Management device P projection plane CAM camera CINF Camera Information IMG image IMG1 1st image IMG2 2nd image

Claims

1. A remote operator terminal used by a remote operator for remote operation of a mobile object, comprising: the first image is an image captured at a first timing by a camera mounted on the moving object, a first viewpoint is defined by a combination of the position and orientation of the camera at the first timing; the second viewpoint is defined by a combination of the position and orientation of the camera predicted at a second timing that is later than the first timing; the projection transformation process transforms the first image seen from the first viewpoint into a second image seen from the second viewpoint based on a perspective projection transformation using a projection plane associated with the camera; the remote operator terminal comprises an information processing device; The information processing device includes: acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at the first timing; obtaining a combination of a plurality of second images by applying the projective transformation process using a common projection plane to the combination of the plurality of first images; Displaying the combination of the plurality of second images on a display device of the remote operator terminal. It was configured as Remote operator terminal.

2. 2. A remote operator terminal according to claim 1, The information processing device further comprises: selecting one of the plurality of cameras as a reference camera; The projection plane associated with the reference camera is set as the common projection plane. It was configured as Remote operator terminal.

3. 3. A remote operator terminal according to claim 2, comprising: The information processing device further comprises: estimating the gaze direction of the remote operator; Dynamically selecting one of the plurality of cameras corresponding to the gaze direction as the reference camera. It was configured as Remote operator terminal.

4. 4. A remote operator terminal according to claim 3, The information processing device is further configured to estimate the gaze direction of the remote operator based on the line of sight of the remote operator or the steering angle of a steering operation by the remote operator. Remote operator terminal.

5. A remote operator terminal according to any one of claims 1 to 4, comprising: The information processing device is further configured to display the plurality of second images side by side on one or more screens of the display device. Remote operator terminal.

6. A remote operator terminal according to any one of claims 1 to 4, comprising: The projective transformation process is estimating a moving direction and a moving amount of the moving object during a period from the first timing to the second timing; calculating a difference between the first viewpoint and the second viewpoint based on the movement direction and the movement amount of the moving object; converting the first image viewed from the first viewpoint into a second image viewed from the second viewpoint based on the difference between the first viewpoint and the second viewpoint; Contains Remote operator terminal.

7. A remote operator terminal according to any one of claims 1 to 4, comprising: The information processing device further comprises: setting at least a part of a communication delay time between the mobile unit and the remote operator terminal as a delay compensation time; The second timing is set to be later than the first timing by the delay compensation time. It was configured as Remote operator terminal.

8. 1. An image display method for displaying an image for a remote operator during remote operation of a mobile object, comprising: the first image is an image captured at a first timing by a camera mounted on the moving object, a first viewpoint is defined by a combination of the position and orientation of the camera at the first timing; the second viewpoint is defined by a combination of the position and orientation of the camera predicted at a second timing that is later than the first timing; the projection transformation process transforms the first image seen from the first viewpoint into a second image seen from the second viewpoint based on a perspective projection transformation using a projection plane associated with the camera; The image display method includes: acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at the first timing; acquiring a combination of a plurality of second images by applying the projective transformation process using a common projection plane to the combination of the plurality of first images; displaying said combination of said plurality of second images on a display device of a remote operator terminal used by said remote operator; Contains Image display method.

9. A remote operation system for remotely operating a mobile object by a remote operator, comprising: the first image is an image captured at a first timing by a camera mounted on the moving object, a first viewpoint is defined by a combination of the position and orientation of the camera at the first timing; the second viewpoint is defined by a combination of the position and orientation of the camera predicted at a second timing that is later than the first timing; the projection transformation process transforms the first image seen from the first viewpoint into a second image seen from the second viewpoint based on a perspective projection transformation using a projection plane associated with the camera; the remote control system comprises one or more processors; the one or more processors: acquiring a plurality of first images captured by a plurality of cameras mounted on the moving object at the first timing; obtaining a combination of a plurality of second images by applying the projective transformation process using a common projection plane to the combination of the plurality of first images; Displaying the combination of the plurality of second images on a display device of a remote operator terminal used by the remote operator. It was configured as Remote control system.

Images