Remote operator terminal, image display method, and remote operation system

Abstract

The purpose of the present invention is to improve the appearance when displaying a plurality of images captured by a plurality of cameras during remote operation of a moving body. The first image is captured at a first timing by a camera mounted on a moving body. The first viewpoint is a camera viewpoint at a first timing, and the second viewpoint is a camera viewpoint at a second timing later than the first timing. The projective conversion process converts a first image observed from a first viewpoint into a second image observed from a second viewpoint in accordance with a perspective projection conversion using a projection plane associated with the camera. When a plurality of first images are captured by each of a plurality of cameras mounted on a moving body, projection conversion processing using a common projection surface is applied to a combination of the plurality of first images. A combination of a plurality of second images obtained by such projective transformation processing is displayed on a remote operator terminal.

Description

Technical Field

[0001] The present disclosure relates to remote operation of a mobile object by a remote operator, and more particularly to delay compensation in remote operation. Background Art

[0002] During vehicle remote operation, images captured by an onboard camera are transmitted from the vehicle to a remote operator terminal and displayed on the terminal's display. In this case, it is desirable to compensate for communication delays between the vehicle and the remote operator terminal. The following technologies are known as techniques for compensating for image delays.

[0003] Patent Document 1 discloses a remote video output system. The remote video output system includes an autonomous vehicle that transmits video and a remote video output device that receives video from the autonomous vehicle. The remote video output device estimates changes in the autonomous vehicle's viewpoint position based on the communication delay between the autonomous vehicle and the remote video output device. Furthermore, the remote video output device cuts out a portion of the received video frame, taking into account the change in the autonomous vehicle's viewpoint position, and displays the image of the cutout portion.

[0004] Non-Patent Document 1 discloses a delay compensation technique using projection transformation. More specifically, this delay compensation technique performs a projection transformation on images received from a vehicle, changing the viewpoint based on the amount of vehicle movement corresponding to the delay time, thereby compensating for the delay. In other words, this delay compensation technique visually compensates for the delay by anticipating the camera image as it would be viewed from a viewpoint that is ahead by the delay time. Prior art literature Patent Literature

[0005] Patent Document 1: International Publication No. 2018 / 155159 Non-patent literature

[0006] Non-Patent Literature 1: Matsubara Kodai and Ohmae Manabu, "Study on Delay Compensation of Camera Images for Remote Operation of Automobiles Based on Projection Transformation," 19th ITS Symposium 2021, 4-A-12, December 2021 Summary of the Invention Technical problem to be solved by the invention

[0007] Considering the need to compensate for image delay during remote operation of a mobile object. However, the aforementioned prior art has not yet explored delay compensation for multiple images captured by multiple cameras. Further research and improvements are desired regarding delay compensation for multiple images captured by multiple cameras.

[0008] As an example, consider the case where multiple images are projectively transformed and then displayed side by side. In this case, there is a risk of overlap or gaps between adjacent transformed images, which can cause a sense of discomfort to the remote operator. Means for solving technical problems

[0009] A first aspect relates to a remote operator terminal used by a remote operator for remote operation of a mobile object. The first image is an image captured by a camera mounted on the moving object at a first timing. The first viewpoint is defined by a combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera expected at a second timing later than the first timing. The projective transformation process transforms a first image observed from a first viewpoint into a second image observed from a second viewpoint according to a perspective projection transformation using a projection plane associated with a camera. The remote operator terminal includes an information processing device. The information processing device acquires a plurality of first images captured by a plurality of cameras mounted on the moving object at a first timing. The information processing device obtains a combination of a plurality of second images by applying a projection transformation process using a common projection plane to the combination of a plurality of first images. The information processing device displays the combination of the plurality of second images on the display device of the remote operator terminal.

[0010] A second aspect relates to an image display method for displaying an image for a remote operator during remote operation of a mobile object. The first image is an image captured by a camera mounted on the moving object at a first timing. The first viewpoint is defined by a combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera expected at a second timing later than the first timing. The projective transformation process transforms a first image observed from a first viewpoint into a second image observed from a second viewpoint according to a perspective projection transformation using a projection plane associated with a camera. Image display methods include: acquiring a plurality of first images captured by a plurality of cameras mounted on the mobile object at a first timing; Obtaining a combination of a plurality of second images by applying a projective transformation process using a common projection plane to the combination of a plurality of first images; and The combination of the plurality of second images is displayed on a display device of a remote operator terminal used by the remote operator.

[0011] A third aspect relates to a remote operation system for remote operation of a mobile object by a remote operator. The first image is an image captured by a camera mounted on the moving object at a first timing. The first viewpoint is defined by a combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera expected at a second timing later than the first timing. The projective transformation process transforms a first image observed from a first viewpoint into a second image observed from a second viewpoint according to a perspective projection transformation using a projection plane associated with a camera. The remote operating system has one or more processors. The one or more processors acquire a plurality of first images captured by a plurality of cameras mounted on the mobile object at a first timing. The one or more processors obtain a combination of a plurality of second images by applying a projective transformation process using a common projection plane to the combination of the plurality of first images. The one or more processors display the combination of the plurality of second images on a display device of a teleoperator terminal used by the teleoperator. Effects of the Invention

[0012] According to the present disclosure, multiple first images captured by multiple cameras are projected onto a common projection surface, rather than using separate projection surfaces. This projected transformation allows multiple second images to be displayed in a row without overlap or gaps. This improves the appearance of the displayed second images and reduces the perceived discomfort experienced by the remote operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram showing a configuration example of a remote operation system. Figure 2 This is a conceptual diagram for explaining an outline of delay compensation processing. Figure 3 This is a conceptual diagram used to explain projective transformation based on perspective projection transformation. Figure 4 This is a conceptual diagram used to explain projective transformation based on perspective projection transformation. Figure 5 This is a conceptual diagram for explaining an example of image deformation caused by projective transformation. Figure 6 This is a conceptual diagram for explaining a first example of the projection transformation process in the case of multiple cameras. Figure 7 This is a conceptual diagram for explaining a first example of the projection transformation process in the case of multiple cameras. Figure 8 This is a conceptual diagram for explaining a second example of the projection transformation process in the case of multiple cameras. Figure 9 This is a conceptual diagram for explaining a second example of the projection transformation process in the case of multiple cameras. Figure 10 This is a conceptual diagram for explaining a second example of the projection transformation process in the case of multiple cameras. Figure 11 This is a conceptual diagram for explaining a third example of the projection transformation process in the case of multiple cameras. Figure 12 This is a conceptual diagram for explaining a third example of the projection transformation process in the case of multiple cameras. Figure 13 It is a block diagram showing an example of the configuration of a vehicle. Figure 14 This is a block diagram showing a configuration example of a remote operator terminal. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0015] 1. Remote operating system Consider remote operation (remote driving) of a mobile object. Examples of mobile objects to be remotely operated include vehicles, robots, and flying objects. Vehicles can be autonomous vehicles or vehicles driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones. As an example, in the following description, consider a case where the mobile object is a vehicle. For generalization, the term "vehicle" in the following description is replaced with "mobile object."

[0016] Figure 1 This is a schematic diagram showing an example of the configuration of a remote operation system 1 according to the present embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote operation. The remote operator terminal 200 is a terminal device used by a remote operator O to remotely operate the vehicle 100. The remote operator terminal 200 may also be referred to as a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. Typically, the management device 300 is a management server on the cloud. The management server may also be composed of multiple servers that perform distributed processing.

[0017] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via the communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 can communicate directly without intermediary of the management device 300.

[0018] Vehicle 100 is equipped with various sensors, including a camera CAM. Camera CAM captures the surroundings of vehicle 100, acquiring images (videos) IMG showing the surrounding conditions of vehicle 100. Sensor detection information SEN includes information obtained by various sensors. Sensor detection information SEN includes at least the images IMG captured by camera CAM. Sensor detection information SEN may also include the position and status of vehicle 100 (e.g., speed, steering angle, etc.). Vehicle 100 transmits sensor detection information SEN to remote operator terminal 200.

[0019] The remote operator terminal 200 receives sensor detection information SEN transmitted from the vehicle 100. The remote operator terminal 200 presents the sensor detection information SEN to the remote operator O. Specifically, the remote operator terminal 200 includes a display device 220, which displays information such as an image IMG on the display device 220. The remote operator O views the displayed information, recognizes the conditions surrounding the vehicle 100, and remotely operates the vehicle 100. In other words, by displaying information specific to the remote operator O on the display device 220, the remote operator O is assisted in remotely operating the vehicle 100.

[0020] Remote operation information OPE is information related to remote operations performed by remote operator O. For example, remote operation information OPE includes the amount of operation performed by remote operator O. Remote operator terminal 200 transmits remote operation information OPE to vehicle 100. Vehicle 100 receives remote operation information OPE transmitted from remote operator terminal 200. Vehicle 100 controls vehicle travel according to the received remote operation information OPE. This enables remote operation of vehicle 100.

[0021] 2. Delay compensation processing using projection transformation Remote operation of vehicle 100 involves communication delays between vehicle 100 and remote operator terminal 200. This communication delay can destabilize the behavior of vehicle 100 during remote operation. Furthermore, communication delays can reduce the accuracy of remote operation of vehicle 100. Therefore, delay compensation is crucial during remote operation of vehicle 100.

[0022] The remote operation system 1 according to this embodiment takes into account communication delays and performs visual delay compensation on the image IMG displayed on the display device 220. Specifically, the remote operation system 1 according to this embodiment utilizes "projection transformation" to perform visual delay compensation on the image IMG. The delay compensation process is primarily performed by, for example, the remote operator terminal 200. However, the process is not limited to the remote operator terminal 200. At least a portion of the delay compensation process may also be performed by the vehicle 100 or the management device 300.

[0023] Figure 2 This is a conceptual diagram for explaining the outline of delay compensation processing performed by the remote operation system 1. A first image IMG1 is an image IMG actually captured by a camera CAM mounted on a vehicle 100 at a first timing T1. The first image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 obtains the first image IMG1 after the first timing T1. If it is possible to estimate (forecast) images IMG to be captured in the future based on this first image IMG1, delay compensation can be performed.

[0024] The second timing T2 is the timing that becomes the foreseeable target, and is a timing that is later than the first timing T1. The difference between the second timing T2 and the first timing T1 is equivalent to the "delay compensation time". The remote operation system 1 may also set at least a part of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200 as the delay compensation time. The communication delay time between the vehicle 100 and the remote operator terminal 200 can be estimated by a known technique. The delay compensation time may also be set as the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The delay compensation time may also be set as the communication time from when the image IMG is sent from the vehicle 100 to when it arrives at the remote operator terminal 200. Alternatively, the delay compensation time may also be set to a fixed value. In either case, the remote operation system 1 sets the second timing T2 to be the delay compensation time later than the first timing T1.

[0025] For convenience, the camera CAM at the first timing T1 is referred to as the first camera CAM1, and the camera CAM at the second timing T2 is referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the position and orientation of the first camera CAM1 at the first timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the position and orientation of the second camera CAM2 expected at the second timing T2.

[0026] The remote operation system 1 obtains camera information CINF related to the camera CAM mounted on the vehicle 100. The camera information CINF includes camera CAM installation information and performance information. The installation information indicates the camera CAM's installation position and orientation in the vehicle coordinate system. The performance information indicates the camera CAM's focal length, field of view, and other information. Since the camera CAM is fixed to the vehicle 100, the camera CAM installation information can be used to convert the vehicle 100's movement direction and amount into the camera CAM's movement direction and amount in the camera coordinate system. In other words, based on the camera CAM installation information and the vehicle 100's movement direction and amount, changes in the camera CAM's viewpoint can be estimated.

[0027] More specifically, the remote operation system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 (i.e., the delay compensation time). For example, the remote operation system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 based on the speed and steering angle of the vehicle 100 at the first timing T1 and the delay compensation time. Information on the speed and steering angle of the vehicle 100 is obtained from the sensor detection information SEN provided by the vehicle 100. Alternatively, the steering angle in the steering operation performed by the remote operator O can also be regarded as the steering angle of the vehicle 100. It can also be assumed that the vehicle 100 performs a stable circular turn. In addition, the remote operation system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information CINF (setting information) and the movement amount and direction of movement of the vehicle 100 during the delay compensation time.

[0028] The first image IMG1 can be said to be an image IMG captured from the first viewpoint, that is, an image IMG observed from the first viewpoint. Hereinafter, the image IMG expected to be captured from the second viewpoint, that is, the image IMG expected to be observed from the second viewpoint, will be referred to as the "second image IMG2." The remote operation system 1 transforms the first image IMG1 observed from the first viewpoint into the second image IMG2 observed from the second viewpoint based on the difference between the first viewpoint and the second viewpoint. In other words, the remote operation system 1 anticipates (foresees) the second image IMG2 observed from the second viewpoint based on the first image IMG1 observed from the first viewpoint. Projective transformation is used in this foresight.

[0029] Figure 3This is a conceptual diagram for explaining the projective transformation. The projective transformation is performed based on the perspective projection transformation. The perspective projection transformation is a drawing method for depicting an object in a three-dimensional space on a two-dimensional plane as viewed from the camera CAM. Therefore, the perspective projection transformation takes into account the viewpoint of the camera CAM and projects the points in the three-dimensional space onto the projection plane P. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane that is orthogonal to the optical axis of the camera CAM. In addition, the points in the three-dimensional space are defined in the three-dimensional world coordinate system (absolute coordinate system). On the other hand, the points projected onto the projection plane P are defined in the two-dimensional image coordinate system.

[0030] For example, N imaginary points are hypothetically set in a three-dimensional world coordinate system. N is an integer greater than or equal to 4. The N imaginary points when observed from the first camera CAM1 (first viewpoint) are projected onto the first projection plane P1 associated with the first camera CAM1 through perspective projection transformation. In addition, the N imaginary points when observed from the second camera CAM2 (second viewpoint) are projected onto the second projection plane P2 associated with the second camera CAM2 through perspective projection transformation. The second viewpoint is obtained based on the difference between the first viewpoint and the second viewpoint. The image coordinates of the imaginary point on the first projection plane P1 when observed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the imaginary point on the second projection plane P2 when observed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on the comparison between the two, the projective transformation matrix H for transforming from the first viewpoint to the second viewpoint is calculated. Then, the projective transformation matrix H is applied to the entire first image IMG1 actually captured by the first camera CAM1, thereby generating a second image IMG2 that is expected to be observed from a second viewpoint.

[0031] As another example, the method described in non-patent document 1 can also be used. Specifically, each image coordinate point on the first image IMG1 (projection surface P) is transformed into a world coordinate point in the world coordinate system through the inverse transformation of the perspective projection transformation. According to the difference between the first viewpoint and the second viewpoint, the world coordinate point observed from the first viewpoint is transformed into the world coordinate point observed from the second viewpoint. Then, through the perspective projection transformation, the world coordinate point observed from the second viewpoint is returned to the projection surface P. Thus, a second image IMG2 that is expected to be observed from the second viewpoint is generated. In addition, according to non-patent document 1, as Figure 4 As shown, it is assumed that the ground surface S is projected on the entire image IMG.

[0032] Figure 5 This is a conceptual diagram for explaining an example of image deformation caused by projective transformation (see Non-Patent Document 1). The first image IMG1 is an original image before projective transformation. The second image IMG2 is generated by applying projective transformation to the first image IMG1. Figure 5 2 shows how the first image IMG1 and the second image IMG2 appear on the screen 222 of the display device 220. For example, when the vehicle 100 is traveling straight, the second image IMG2 appears to be tilted inward. As another example, when turning right, the second image IMG2 appears to be tilted to the left.

[0033] 3. Projection transformation processing in the case of multiple cameras In the following description, "projection transformation processing" refers to the delay compensation processing using the projection transformation described in Section 2 above. The projection transformation processing transforms the first image IMG1 observed from the first viewpoint into the second image IMG2 observed from the second viewpoint based on a perspective projection transformation using the projection plane P associated with the camera CAM. The remote control system 1 can perform delay compensation by applying the projection transformation processing to the image IMG captured by the camera CAM. The subject of the projection transformation processing is, for example, the remote operator terminal 200. However, the subject of the projection transformation processing is not limited to the remote operator terminal 200. At least a portion of the projection transformation processing can also be performed by the vehicle 100 or the management device 300.

[0034] Here, consider a case where multiple cameras CAM are mounted on vehicle 100. The multiple cameras CAM are positioned in different directions within the vehicle coordinate system. For example, the multiple cameras CAM include a front camera CAM-F for capturing images of the front, a left camera CAM-L for capturing images of the left front, and a right camera CAM-R for capturing images of the right front. However, the components of the multiple cameras CAM are not limited to these.

[0035] The remote operator terminal 200 acquires a plurality of images IMG captured by a plurality of cameras CAM mounted on the vehicle 100. The remote operator terminal 200 then displays the plurality of images IMG on the display device 220. The remote operator terminal 200 may also display the plurality of images IMG in a row on one or more screens 222 of the display device 220.

[0036] Next, a projective transformation process for a plurality of images IMG captured by a plurality of cameras CAM will be considered.

[0037] 3-1. The first example Figure 6 This is a conceptual diagram used to illustrate the first example. Projection planes PF, PL, and PR are projection planes P associated with the front camera CAM-F, the left camera CAM-L, and the right camera CAM-R, respectively. For example, projection plane PF is perpendicular to the optical axis of the front camera CAM-F, projection plane PL is perpendicular to the optical axis of the left camera CAM-L, and projection plane PR is perpendicular to the optical axis of the right camera CAM-R. Projection planes PF, PL, and PR are distinct from one another.

[0038] The orientations of multiple CAM cameras (CAM-F, CAM-L, CAM-R) are different. Figure 6 As shown, the change of the viewpoint of the camera CAM relative to the projection plane P when the vehicle 100 moves in a certain direction is different among the plurality of cameras CAM (CAM-F, CAM-L, CAM-R).

[0039] The remote operation system 1 obtains a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R) captured by a plurality of cameras CAM (CAM-F, CAM-L, CAM-R) at a first timing T1. The remote operation system 1 obtains a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) by applying a projection transformation process to each of the plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). In the first example, a plurality of different projection planes P (PF, PL, PR) are used for each of the plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). That is, the remote operation system 1 generates the second image IMG2-F by applying a projection transformation process using the projection plane PF to the first image IMG1-F captured by the front camera CAM-F. Furthermore, the remote operation system 1 generates the second image IMG2-L by applying a projection transformation process using the projection plane PL to the first image IMG1-L captured by the left camera CAM-L. Furthermore, the remote operation system 1 generates a second image IMG2-R by applying a projection transformation process using the projection plane PR to the first image IMG1-R captured by the right camera CAM-R.

[0040] Figure 7 FIG. 2 shows an example of displaying a plurality of images IMG in the first example. Figure 7 In the example shown, display device 220 includes multiple screens 222-F, 222-L, and 222-R. Screens 222-F, 222-L, and 222-R are arranged adjacent to each other in a row. More specifically, screens 222-L and 222-R are positioned on either side of screen 222-F, with screen 222-F sandwiched between them.

[0041] Assuming that no projection transformation processing is performed, the remote operator terminal 200 displays the plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R) on the plurality of screens 222 (222-F, 222-L, 222-R) of the display device 220.

[0042] When performing the projection transformation process, the remote operator terminal 200 obtains a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) based on a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R). Figure 7 In the example shown, the vehicle 100 is turning right. Furthermore, the remote operator terminal 200 displays multiple second images IMG2 (IMG2-F, IMG2-L, and IMG2-R) on the display device 220. For example, the remote operator terminal 200 displays the multiple second images IMG2 (IMG2-F, IMG2-L, and IMG2-R) in a row on one or more screens 222 of the display device 220. Alternatively, the remote operator terminal 200 may display the multiple second images IMG2 (IMG2-F, IMG2-L, and IMG2-R) in a row on multiple screens 222 (222-F, 222-L, and 222-R). In this case, the second image IMG2-F is generally located on screen 222-F, but a portion of the second image IMG2-F may extend beyond screens 222-L and 222-R adjacent to screen 222-F. Similarly, the second image IMG2-L is generally arranged on the screen 222-L, but a portion of the second image IMG2-L may extend beyond the screen 222-F adjacent to the screen 222-L. Similarly, the second image IMG2-R is generally arranged on the screen 222-R, but a portion of the second image IMG2-R may extend beyond the screen 222-F adjacent to the screen 222-R.

[0043] 3-2. Second Example In the first example described above, different projection planes P are used for each of the multiple first images IMG1 during the projection transformation process. As a result, the shapes of the multiple second images IMG2 obtained may not necessarily be consistent. Therefore, when the multiple second images IMG2 are displayed side by side on the display device 220, the boundaries between adjacent second images IMG2 may not align, resulting in "overlaps" or "gaps" between adjacent second images IMG2. This can degrade the appearance of the multiple second images IMG2 and create a sense of disharmony for the remote operator O.

[0044] Therefore, the second example proposes a technique that can suppress the sense of discomfort of the remote operator O. In addition, descriptions that overlap with those of the first example are omitted as appropriate.

[0045] Figure 8 This is a conceptual diagram for explaining the second example. In the second example, in the projection transformation process, the plurality of first images IMG1 are not projected on mutually different projection planes P, but on the same projection plane P. Hereinafter, this same projection plane P is referred to as a "common projection plane P-COM."

[0046] For example, the common projection surface P-COM is one of the projection surfaces PF, PL, and PR. In this case, the remote operation system 1 selects one of the multiple cameras CAM (CAM-F, CAM-L, and CAM-R) as the reference camera. Then, the remote operation system 1 sets the projection surface P associated with the reference camera as the common projection surface P-COM. Figure 8 In the example shown, the projection plane PF associated with the front camera CAM-F is set as the common projection plane P-COM.

[0047] The remote operation system 1 applies a projection transformation process using a common projection plane P-COM (reference camera) to a combination of a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R) to obtain a combination of a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R). This is equivalent to assuming that all cameras CAM are facing the same direction as the reference camera.

[0048] For example, the remote operation system 1 pre-links a plurality of first images IMG1 (IMG1-F, IMG1-L, IMG1-R) to generate a single large first image IMG1 (see Figure 9 ). First images IMG1-L and IMG1-R are arranged on both sides of the first image IMG1-F, and the first image IMG1-F is sandwiched between the first images IMG1-L and IMG1-R. Then, the remote operation system 1 obtains a large second image IMG2 by applying a projection transformation process using a common projection surface P-COM to a large first image IMG1. A large second image IMG2 is equivalent to a combination of multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R).

[0049] As another example, the remote operation system 1 may obtain multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) by applying a projection transformation process using a common projection surface P-COM to multiple first images IMG1 (IMG1-F, IMG1-L, IMG1-R). Then, the remote operation system 1 connects the multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R) to obtain a single large second image IMG2. Second images IMG2-L and IMG2-R are arranged on both sides of second image IMG2-F, with second image IMG2-F sandwiched between second images IMG2-L and IMG2-R. One large second image IMG2 is equivalent to a combination of multiple second images IMG2 (IMG2-F, IMG2-L, IMG2-R).

[0050] Figure 9FIG. 2 shows a display example of a plurality of images IMG in the second example. Figure 7 Similarly, the vehicle 100 is turning right. When performing the projection transformation process, the remote operator terminal 200 displays a combination of the plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) on the display device 220. For example, the remote operator terminal 200 arranges and displays the plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) on one or more screens 222 of the display device 220. The remote operator terminal 200 may also arrange and display the plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R) on a plurality of screens 222 (222-F, 222-L, 222-R). Figure 9 As shown, when multiple second images IMG2 are displayed in a row on display device 220, the boundaries between adjacent second images IMG2 are aligned. Therefore, overlaps and gaps between adjacent second images IMG2 are minimized. In other words, multiple second images IMG2 can be displayed in a row without overlaps or gaps.

[0051] As described above, according to the second example, multiple first images IMG1 captured by multiple cameras CAM are projected using a common projection plane P-COM, rather than using different projection planes P. The multiple second images IMG2 obtained through this projection transformation can be displayed in a row without overlap or gaps. As a result, the appearance of the displayed multiple second images IMG2 is improved, minimizing the sense of discomfort felt by the remote operator O.

[0052] 3-3. The third example In the second example above, there may be parts where the accuracy of the projection transformation process is not necessarily high. Figure 8 In the example shown, the projection plane PF associated with the front camera CAM-F is used as the common projection plane P-COM. For the first image IMG1-F captured by the front camera CAM-F, the common projection plane P-COM is identical to the original projection plane PF, resulting in high accuracy in the projection transformation process. However, for the first image IMG1-L captured by the left camera CAM-L, the common projection plane P-COM, which differs from the original projection plane PL, is used, so the projection transformation process may not be as accurate. Similarly, for the first image IMG1-R captured by the right camera CAM-R, the common projection plane P-COM, which differs from the original projection plane PR, is used, so the projection transformation process may not be as accurate.

[0053] If the accuracy of the projective transformation processing is low, the accuracy of the second image IMG2 generated by the projective transformation will also be low. For example, when turning right, the remote operator O primarily looks at the second image IMG2-R on the right side. However, if the accuracy of this second image IMG2-R is low, the accuracy of the remote operation may also be reduced.

[0054] However, conversely, as long as the accuracy of at least the second image IMG2 that the remote operator O is looking at is high, remote operation accuracy can be maintained even if the accuracy of other second images IMG2 is low. Based on this perspective, in the third example, the common projection plane P-COM is dynamically set based on the gaze direction of the remote operator O. In other words, the common projection plane P-COM is dynamically switched in conjunction with the gaze direction of the remote operator O.

[0055] Figure 10 This is a conceptual diagram illustrating the relationship between the gaze direction of the remote operator O and the common projection plane P-COM. When the gaze direction of the remote operator O is within the gaze direction range RNG-F, the remote operation system 1 selects the front camera CAM-F as the reference camera and the projection plane PF as the common projection plane P-COM. When the gaze direction of the remote operator O is within the gaze direction range RNG-L, the remote operation system 1 selects the left camera CAM-L as the reference camera and the projection plane PL as the common projection plane P-COM. When the gaze direction of the remote operator O is within the gaze direction range RNG-R, the remote operation system 1 selects the right camera CAM-R as the reference camera and sets the projection plane PR as the common projection plane P-COM.

[0056] The gaze direction of the remote operator O is estimated, for example, based on the steering angle of the steering operation performed by the remote operator O. In this case, a predetermined steering angle range θ-F, including straight driving, is associated with the gaze direction range RNG-F. Furthermore, a steering angle range θ-R to the right of the predetermined steering angle range θ-F is associated with the gaze direction range RNG-R. Furthermore, a steering angle range θ-L to the left of the predetermined steering angle range θ-F is associated with the gaze direction range RNG-L. The remote operation system 1 estimates the gaze direction of the remote operator O based on the steering angle of the steering operation performed by the remote operator O. More specifically, the remote operation system 1 selects a steering angle range θ-X (X = F, L, or R) that includes the steering angle of the steering operation performed by the remote operator O. Furthermore, the remote operation system 1 estimates that the gaze direction of the remote operator O is within the gaze direction range RNG-X associated with the selected steering angle range θ-X. The remote operation system 1 then selects the camera CAM-X corresponding to the gaze direction range RNG-X as the reference camera.

[0057] As another example, the gaze direction of the remote operator O can also be estimated based on the gaze direction of the remote operator O. More specifically, the gaze direction of the remote operator O is detected by the operator monitor 240 (described later) of the remote operator terminal 200. The detected gaze direction of the remote operator O is regarded as the gaze direction of the remote operator O. The remote operation system 1 selects the camera CAM-X corresponding to the gaze direction range RNG-X that includes the gaze direction (gaze direction) of the remote operator O as the reference camera.

[0058] In this way, the remote operation system 1 estimates the gaze direction of the remote operator O and dynamically selects one of the multiple cameras CAM corresponding to the gaze direction as the reference camera. The remote operation system 1 then sets the projection plane P associated with the selected reference camera as the common projection plane P-COM.

[0059] Figure 11 The common projection plane P-COM is shown when turning right. When turning right, the gaze direction of the teleoperator O is included in the gaze direction range RNG-R. Therefore, the right camera CAM-R is selected as the reference camera, and the projection plane PR is set to the common projection plane P-COM.

[0060] Figure 12 Shown Figure 11 The example shown shows the display of multiple images IMG. When turning right, the teleoperator O primarily focuses on the right-side second image IMG2-R. Because the projection plane PR associated with the right camera CAM-R is used as the common projection plane P-COM, at least the second image IMG2-R has high accuracy. Because at least the second image IMG2-R, which the teleoperator O is focusing on, has high accuracy, remote operation accuracy can be maintained even if the accuracy of other second images IMG2 is low.

[0061] As described above, according to the third example, the common projection plane P-COM is dynamically set in consideration of the gaze direction of the remote operator O. This ensures at least the accuracy of the second image IMG2 that the remote operator O is looking at. As a result, the accuracy of the remote operation is also ensured.

[0062] 4. Example of vehicle configuration 4-1. Example of configuration Figure 13 1 is a block diagram showing a configuration example of a vehicle 100 . The vehicle 100 includes a communication device 110 , a sensor group 120 , a travel device 130 , and a control device 150 .

[0063] The communication device 110 communicates with the outside of the vehicle 100 . For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300 .

[0064] Sensor group 120 includes an identification sensor, a vehicle state sensor, a position sensor, and the like. The identification sensor identifies (detects) the surrounding conditions of vehicle 100. Examples of identification sensors include a CAM camera, LIDAR (Laser Imaging Detection and Ranging), and radar. The vehicle state sensor detects the state of vehicle 100. Vehicle state sensors include speed sensors, acceleration sensors, yaw rate sensors, and steering angle sensors. The position sensor detects the position and orientation of vehicle 100. For example, the position sensor includes a GNSS sensor.

[0065] The travel device 130 includes a steering system, a drive system, and a brake system. The steering system steers the wheels. For example, the steering system includes an Electric Power Steering (EPS) system. The drive system is a power source that generates driving force. Examples of drive systems include an engine, an electric motor, and an in-wheel motor. The brake system generates braking force.

[0066] Control device 150 is a computer that controls vehicle 100. Control device 150 includes one or more processors 160 (hereinafter referred to as processor 160) and one or more storage devices 170 (hereinafter referred to as storage device 170). Processor 160 performs various processes. Examples of processor 160 include a general-purpose processor, a special-purpose processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), an integrated circuit, and / or a combination thereof. Storage device 170 stores various information. Examples of storage device 170 include volatile memory, non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. Control device 150 may also include one or more ECUs (Electronic Control Units). For generalization, control device 150 may also be referred to as a processing circuit.

[0067] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 can be realized by the cooperation between the processor 160 executing the vehicle control program PROG1 and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 can be recorded on a computer-readable recording medium.

[0068] 4-2. Sensor detection information The control device 150 uses the sensor group 120 to obtain sensor detection information SEN. The sensor detection information SEN is stored in the storage device 170. The sensor detection information SEN includes an image IMG, vehicle state information, position information, object information, and the like. The image IMG is captured by the camera CAM. The vehicle state information indicates the state of the vehicle 100 (e.g., speed, steering angle, etc.) detected by the vehicle state sensor. The position information indicates the position and orientation of the vehicle 100 detected by the position sensor.

[0069] Object information is information related to objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, bicycles, two-wheeled vehicles, other vehicles (preceding vehicles, parallel vehicles, following vehicles, etc.), white lines, road structures (for example, curbs, guardrails), poles, traffic lights, signs, etc. The control device 150 can identify objects around the vehicle 100 by using an identification sensor. For example, by analyzing the image IMG, it is possible to identify the object and calculate the relative position of the object. In addition, based on the point cloud information obtained by LIDAR, it is also possible to identify the object and obtain the relative position and relative speed of the object. The object information includes the relative position of the object relative to the vehicle 100. The object information may also include the relative speed of the object.

[0070] 4-3. Vehicle driving control The control device 150 performs vehicle travel control for controlling the travel of the vehicle 100. Vehicle travel control includes steering control, drive control, and brake control. The control device 150 performs vehicle travel control by controlling the travel device 130 (steering device, drive device, and brake device).

[0071] The control device 150 can also perform autonomous driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, based on the sensor detection information SEN, the control device 150 generates a target trajectory for the vehicle 100 to follow the driving plan. The target trajectory includes a target position and a target speed. The control device 150 then controls vehicle driving so that the vehicle 100 follows the target trajectory.

[0072] 4-4. Processing related to remote operation When remote operation of the vehicle 100 is performed, the control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0073] The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. Typically, the control device 150 transmits the image IMG to the remote operator terminal 200. The control device 150 may also transmit vehicle status information to the remote operator terminal 200. The control device 150 may also transmit object information to the remote operator terminal 200.

[0074] Furthermore, the control device 150 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information related to remote operation performed by the remote operator O. For example, the remote operation information OPE includes the amount of operation performed by the remote operator O. The control device 150 controls vehicle travel according to the received remote operation information OPE.

[0075] 4-5. Camera Information Camera information CINF includes installation information and performance information for each of one or more cameras CAM installed in vehicle 100. The installation information indicates the installation position and orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, field of view, and other information of the camera CAM. The camera information CINF is stored in storage device 170. Control device 150 may also transmit camera information CINF to remote operator terminal 200.

[0076] 5. Example of Remote Operator Terminal Configuration Figure 14 2 is a block diagram showing a configuration example of the remote operator terminal 200 . The remote operator terminal 200 includes a communication device 210 , a display device 220 , an input device 230 , an operator monitor 240 , and an information processing device 250 .

[0077] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0078] The display device 220 displays various information for the remote operator O who is performing remote operation. In other words, the display device 220 presents various information to the remote operator O by displaying the various information. The display device 220 includes a plurality of screens 222 .

[0079] The input device 230 is a member operated by the remote operator O when remotely operating the vehicle 100. For example, the input device 230 includes remote operating members. The remote operating members include a steering wheel, an accelerator pedal, a brake pedal, a direction indicator, and the like.

[0080] The operator monitor 240 includes sensors for monitoring the status of the remote operator O. For example, the operator monitor 240 includes a camera that captures the face and eyes of the remote operator O. The operator monitor 240 analyzes the image of the remote operator O captured by the camera to extract images of the remote operator O's face and eyes. This extraction of the face and eyes is performed, for example, using a machine learning model pre-generated through machine learning. Furthermore, the operator monitor 240 identifies the gaze direction of the remote operator O based on the position and rotation angle of each eye's pupil.

[0081] The information processing device 250 controls the remote operator terminal 200. The information processing device 250 includes one or more processors 260 (hereinafter referred to as processors 260) and one or more storage devices 270 (hereinafter referred to as storage devices 270). The processor 260 performs various processes. Examples of the processor 260 include a general-purpose processor, a special-purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, and / or a combination thereof. The storage device 270 stores various information. Examples of the storage device 170 include volatile memory, non-volatile memory, an HDD, an SSD, and the like. For generalization, the information processing device 250 may also be referred to as a processing circuit.

[0082] The remote operation control program PROG2 is a computer program executed by the processor 260. The functions of the information processing device 250 can be realized by the cooperation between the processor 260 executing the remote operation control program PROG2 and the storage device 270. The remote operation control program PROG2 is stored in the storage device 270. Alternatively, the remote operation control program PROG2 can be recorded on a computer-readable recording medium. The remote operation control program PROG2 can also be provided via a network.

[0083] The information processing device 250 communicates with the vehicle 100 via the communication device 210. The information processing device 250 receives the sensor detection information SEN transmitted from the vehicle 100. The information processing device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the information processing device 250 presents the image IMG to the remote operator O by displaying it on the display device 220. The remote operator O can understand the status of the vehicle 100 and the surrounding conditions based on the presented information.

[0084] Remote operator O operates input device 230. The amount of operation on input device 230 is detected by a sensor provided on input device 230. Information processing device 250 generates remote operation information OPE reflecting the amount of operation on input device 230 by remote operator O. Information processing device 250 then transmits this remote operation information OPE to vehicle 100 via communication device 210.

[0085] The information processing device 250 may also receive camera information CINF transmitted from the vehicle 100 . The camera information CINF is stored in the storage device 270 .

[0086] The information processing device 250 performs the projection transformation processing (delay compensation processing) described in Sections 2 and 3 above. The speed and steering angle of the vehicle 100 are obtained from the sensor detection information SEN. The steering angle during the steering operation performed by the remote operator O can also be regarded as the steering angle of the vehicle 100. The installation information and performance information of each camera CAM mounted on the vehicle 100 are obtained from the camera information CINF. The gaze direction of the remote operator O can be estimated based on the steering operation amount of the remote operator O. In addition, the gaze direction of the remote operator O can also be estimated based on the line of sight of the remote operator O recognized by the operator monitor 240. Based on this information, the information processing device 250 performs the projection transformation processing (delay compensation processing) described in Sections 2 and 3 above. [Explanation of Reference Numerals]

[0087] 1 Remote Operation System 100 Vehicle 200 Remote Operator Terminal 220 display device 222 pictures 250 Information Processing Device 300 Management Device P projection surface CAM Camera CINF Camera Information IMG images IMG1 First image IMG2 Second image

Claims

1. A remote operator terminal used by a remote operator for remote operation of a mobile object. The first image is an image captured by a camera mounted on the mobile object at a first timing. The first viewpoint is defined by a combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera expected at a second timing later than the first timing. Projection transformation processing transforms the first image observed from the first viewpoint into a second image observed from the second viewpoint according to a perspective projection transformation using a projection plane associated with the camera, The remote operator terminal includes an information processing device, The information processing device is configured as follows: acquiring a plurality of first images captured by a plurality of cameras mounted on the mobile object at the first timing, A combination of a plurality of second images is obtained by applying the projective transformation process using a common projection plane to the combination of the plurality of first images, The combination of the plurality of second images is displayed on a display device of the remote operator terminal.

2. The remote operator terminal according to claim 1, wherein: The information processing device is further configured to: selecting one of the plurality of cameras as a reference camera, The projection plane associated with the reference camera is set as the common projection plane.

3. The remote operator terminal according to claim 2, wherein: The information processing device is further configured to: inferring the remote operator's gaze direction, A camera corresponding to the gaze direction among the plurality of cameras is dynamically selected as the reference camera.

4. The remote operator terminal according to claim 3, wherein: The information processing device is further configured to estimate the gaze direction of the remote operator based on a line of sight direction of the remote operator or a steering angle of a steering operation performed by the remote operator.

5. The remote operator terminal according to any one of claims 1 to 4, wherein: The information processing device is further configured to: arrange and display the plurality of second images on one or more screens of the display device.

6. The remote operator terminal according to any one of claims 1 to 4, wherein: The projective transformation process includes: estimating a moving direction and a moving amount of the moving object during a period from the first timing to the second timing; calculating a difference between the first viewpoint and the second viewpoint based on the moving direction and the moving amount of the moving object; and The first image observed from the first viewpoint is transformed into a second image observed from the second viewpoint according to the difference between the first viewpoint and the second viewpoint.

7. The remote operator terminal according to any one of claims 1 to 4, wherein: The information processing device is further configured to: setting at least a portion of a communication delay time between the mobile object and the remote operator terminal as a delay compensation time, The second timing is set to be later than the first timing by the delay compensation time.

8. An image display method for displaying an image to a remote operator during remote operation of a mobile object. The first image is an image captured by a camera mounted on the mobile object at a first timing. The first viewpoint is defined by a combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera expected at a second timing later than the first timing. Projection transformation processing transforms the first image observed from the first viewpoint into a second image observed from the second viewpoint according to a perspective projection transformation using a projection plane associated with the camera, The image display method comprises: acquiring a plurality of first images captured by a plurality of cameras mounted on the mobile object at the first timing; Obtaining a combination of a plurality of second images by applying the projective transformation process using a common projection plane to the combination of the plurality of first images; as well as The combination of the plurality of second images is displayed on a display device of a remote operator terminal used by the remote operator.

9. A remote operation system for a remote operator to remotely operate a mobile object. The first image is an image captured by a camera mounted on the mobile object at a first timing. The first viewpoint is defined by a combination of the position and orientation of the camera at the first timing. The second viewpoint is defined by a combination of the position and orientation of the camera expected at a second timing later than the first timing. Projection transformation processing transforms the first image observed from the first viewpoint into a second image observed from the second viewpoint according to a perspective projection transformation using a projection plane associated with the camera, The remote operating system has one or more processors. The one or more processors are configured to: acquiring a plurality of first images captured by a plurality of cameras mounted on the mobile object at the first timing, A combination of a plurality of second images is obtained by applying the projective transformation process using a common projection plane to the combination of the plurality of first images, The combination of the plurality of second images is displayed on a display device of a remote operator terminal used by the remote operator.

Images