Teleoperation system

Abstract

A remote operation system acquires an image set including a plurality of images obtained by using one or more cameras mounted on a mobile body. The remote operation system performs an image adjustment process of acquiring an adjusted image set by adjusting at least one image, and displays the adjusted image set on a remote operation person terminal. A first plane is a plane orthogonal to an optical axis of a camera corresponding to a first image. A second plane is a plane orthogonal to an optical axis of a camera corresponding to a second image. In the image adjustment process, the first image is arranged on the first plane, and the second image is arranged on the second plane. Furthermore, the image adjustment process acquires the adjusted image set by observing the arranged first image and second image from a common viewpoint, and projecting the first image and the second image onto a common plane.

Description

Technical Field

[0001] This invention relates to a remote operation of a mobile body based on a remote operator. Background Technology

[0002] Japanese Patent Application Publication No. 2021-067977 discloses a technique for expanding the field of view by connecting multiple images captured by multiple cameras. Summary of the Invention

[0003] Consider remote operation of a mobile device based on a remote operator. Images (videos) captured by cameras mounted on the mobile device are presented to the remote operator. To expand the field of view, multiple images captured by one or more cameras (multiple cameras or fisheye cameras) mounted on the mobile device can be presented to the remote operator. For example, multiple images can be presented adjacent to each other. However, since the multiple images are captured from different directions, simply arranging them adjacently may create an unnatural appearance at the boundaries between the images. For example, lines such as white lines on a road surface may appear unnaturally bent at the boundaries between images.

[0004] One object of the present invention is to provide a technique that can suppress the awkwardness of presenting multiple images obtained by one or more cameras (multiple cameras or fisheye cameras) mounted on a mobile body to a remote operator.

[0005] One aspect of this invention relates to a remote operating system for remote operation of a mobile body operated by a remote operator. The remote operating system has one or more processors.

[0006] One or more processors acquire an image set comprising multiple images obtained by using one or more cameras mounted on a mobile body. One or more processors perform image adjustment processing to acquire an adjusted image set by adjusting at least one of the multiple images. One or more processors display the adjusted image set on a remote operator terminal used by a remote operator.

[0007] Multiple images include a first image and a second image. A first plane is a plane orthogonal to the optical axis of the camera corresponding to the first image. A second plane is a plane orthogonal to the optical axis of the camera corresponding to the second image. In the image adjustment process, the first image is positioned on the first plane, and the second image is positioned on the second plane. Furthermore, the image adjustment process obtains an adjusted image set by observing the positioned first and second images from a common viewpoint and projecting the first and second images onto the common plane.

[0008] According to the present invention, image adjustment processing is performed on an image set obtained by using one or more cameras mounted on a mobile body. In the image adjustment processing, a first image is positioned on a first plane, and a second image is positioned on a second plane. Furthermore, the image adjustment processing obtains an adjusted image set by observing the positioned first and second images from a common viewpoint and projecting the first and second images onto a common plane. The adjusted image set is then displayed on a remote operator's terminal. This image adjustment processing, for example, can suppress the bending of lines such as white lines at the boundaries between images. That is, it can suppress the sense of disharmony perceived by the remote operator. Attached Figure Description

[0009] Hereinafter, with reference to the accompanying drawings, the features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described, in which the same reference numerals denote the same elements.

[0010] Figure 1 This is a schematic diagram illustrating the structure of a remote operating system.

[0011] Figure 2 This is a conceptual diagram used to illustrate multiple cameras and image adjustment processing.

[0012] Figure 3A This is a concept diagram used to illustrate an example of image adjustment processing.

[0013] Figure 3B This is a concept diagram used to illustrate an example of image adjustment processing.

[0014] Figure 4A This is a concept diagram used to illustrate an example of image adjustment processing.

[0015] Figure 4B This is a concept diagram used to illustrate an example of image adjustment processing.

[0016] Figure 5 This is a block diagram representing an example of a functional structure related to image adjustment processing.

[0017] Figure 6 This is a block diagram representing a variation. Detailed Implementation

[0018] 1. Overview of Remote Operating Systems

[0019] Consider the remote operation (remote driving) of a mobile body. Examples of mobile bodies that can be remotely operated include vehicles, robots, and flying bodies. Vehicles can be autonomous vehicles or vehicles driven by a human. Examples of robots include logistics robots and operational robots. Examples of flying bodies include drones. As an example, the following description considers the case where the mobile body is a vehicle. For generalization, "vehicle" will be replaced with "mobile body" in the following description.

[0020] Figure 1 This is a schematic diagram illustrating an example of the structure of the remote operating system 1 according to this embodiment. The remote operating system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the object of remote operation. The remote operator terminal 200 is a terminal device used by a remote operator to remotely operate the vehicle 100. The management device 300 manages the remote operating system 1. Typically, the management device 300 is a cloud-based management server. The management server can be composed of multiple servers performing distributed processing.

[0021] Vehicle 100, remote operator terminal 200, and management device 300 can communicate with each other via a communication network. Vehicle 100 and remote operator terminal 200 can communicate with each other via management device 300. Furthermore, vehicle 100 and remote operator terminal 200 can communicate directly without going through management device 300.

[0022] 1-1. Example of vehicle structure

[0023] The vehicle 100 is equipped with a communication device 110, a sensor group 120, a driving device 130, and a control device 150.

[0024] The communication device 110 communicates with the remote operator terminal 200 or the management device 300.

[0025] Sensor group 120 includes identification sensors, vehicle status sensors, position sensors, etc. The identification sensors identify (detect) the surrounding conditions of vehicle 100. Examples of identification sensors include cameras (CAMs), LiDAR, radar, etc. The camera (CAM) captures images (videos) (IMGs) representing the surrounding conditions of vehicle 100. The vehicle status sensors detect the state of vehicle 100. Vehicle status sensors include speed sensors, acceleration sensors, yaw rate sensors, steering angle sensors, etc. The position sensors detect the position and orientation of vehicle 100. For example, position sensors include GNSS sensors.

[0026] The driving device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is the power source that generates driving force. Examples of drive devices include an engine, an electric motor, and a hub motor. The braking device generates braking force.

[0027] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors and one or more storage devices. The processor performs various processes. Examples of processors include CPUs, GPUs, ASICs, FPGAs, etc. A processor can also be called a circuit or processing circuitry. The storage devices store various information. Examples of storage devices include volatile memory, non-volatile memory, HDDs, SSDs, etc. The functions of the control device 150 can be achieved through the cooperation of the processor executing the control program and the storage devices. The control program is stored in the storage devices. The control program can be recorded on a computer-readable recording medium.

[0028] The control device 150 uses the sensor array 120 to acquire sensor detection information (SEN). The SEN includes an image (IMG), vehicle status information, position information, and object information. The IMG is captured by a camera (CAM). The vehicle status information indicates the state of the vehicle 100 detected by the vehicle status sensor (e.g., speed, steering angle). The position information indicates the position and orientation of the vehicle 100 detected by the position sensor. The object information is information related to objects around the vehicle 100 (e.g., pedestrians, other vehicles, road structures, signals, signs, etc.). The control device 150 can identify objects around the vehicle 100 using an identification sensor. The object information includes the relative position or relative speed of the object relative to the vehicle 100.

[0029] The control device 150 performs vehicle driving control to control the movement of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 performs vehicle driving control by controlling the driving devices 130 (steering device, drive device, and braking device).

[0030] The control unit 150 can perform autonomous driving control based on the sensor detection information (SEN). More specifically, the control unit 150 generates a driving plan for the vehicle 100 based on the sensor detection information (SEN). Furthermore, the control unit 150 generates a target trajectory required for the vehicle 100 to travel according to the driving plan based on the sensor detection information (SEN). The target trajectory includes a target position and a target speed. Then, the control unit 150 performs vehicle driving control to make the vehicle 100 follow the target trajectory.

[0031] During remote operation of vehicle 100, control device 150 communicates with remote operator terminal 200 via communication device 110. Control device 150 sends at least a portion of sensor detection information (SEN) to remote operator terminal 200. The sensor detection information (SEN) sent to remote operator terminal 200 includes at least an image (IMG) captured by camera (CAM). Furthermore, control device 150 receives remote operation information (OPE) from remote operator terminal 200. Remote operation information (OPE) reflects information based on the amount of operation performed by remote operator O. Control device 150 performs vehicle driving control according to the received remote operation information (OPE).

[0032] 1-2. Example of the structure of a remote operator terminal

[0033] The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, and a control device 250.

[0034] The communication device 210 communicates with the vehicle 100 or the management device 300.

[0035] Display device 220 displays various information to a remote operator O who is performing remote operations. In other words, display device 220 prompts various information to the remote operator O by displaying various information. Examples of display device 220 include a monitor or a touch panel.

[0036] Input device 230 includes components operated by a remote operator O when remotely operating vehicle 100. For example, input device 230 includes remote operating components. Remote operating components include a steering wheel, accelerator pedal, brake pedal, direction indicator, etc.

[0037] Control device 250 is a computer that controls remote operator terminal 200. Control device 250 includes one or more processors and one or more storage devices. The processor performs various processes. Examples of processors include CPU, GPU, ASIC, FPGA, etc. A processor can also be called a circuit or processing circuitry. The storage device stores various information. Examples of storage devices include volatile memory, non-volatile memory, HDD, SSD, etc. The functions of control device 250 can be achieved through the cooperation of the processor executing the control program and the storage device. The control program is stored in the storage device. The control program can be recorded on a computer-readable recording medium.

[0038] During remote operation of vehicle 100, control device 250 communicates with vehicle 100 via communication device 210. Control device 250 receives sensor detection information (SEN) sent from vehicle 100. Control device 250 prompts remote operator O with the necessary information from the received sensor detection information (SEN). For example, control device 250 prompts remote operator O by displaying an image (IMG) on display device 220. Remote operator O can then identify the status of vehicle 100 or the surrounding environment based on the prompted information.

[0039] The remote operator O operates the input device 230. The operation amount of the input device 230 is detected by a sensor installed on the input device 230. The control device 250 generates remote operation information OPE reflecting the operation amount (steering operation amount, throttle operation amount, brake operation amount) of the input device 230 based on the remote operator O. Then, the control device 250 transmits the remote operation information OPE to the vehicle 100 via the communication device 210. In this way, remote operation of the vehicle 100 is realized.

[0040] 2. Image adjustment processing for multiple images

[0041] 2-1. Overview

[0042] like Figure 2 As shown, consider the case where multiple cameras (CAMs) are mounted on a vehicle 100. The orientations of each of the multiple camera CAMs in the vehicle coordinate system are different, i.e., their respective line-of-sight directions. For example, the multiple camera CAMs include a front camera CAM-F for capturing images of the front, a left camera CAM-L for capturing images of the left front, and a right camera CAM-R for capturing images of the right front. The fields of view (capture range) of the front camera CAM-F and the left camera CAM-L are adjacent, and the fields of view (capture range) of the front camera CAM-F and the right camera CAM-R are also adjacent.

[0043] The front image IMG-F, left image IMG-L, and right image IMG-R are image IMGs captured by the front camera CAM-F, left camera CAM-L, and right camera CAM-R, respectively. These three images are collectively referred to as the "image set." The remote operator terminal 200 acquires the image set from the vehicle 100 and displays it on the display device 220. By displaying the image set, which includes multiple image IMGs, the field of view of the remote operator O is expanded. This improves the safety of remote operation.

[0044] exist Figure 2 In the example shown, display device 220 includes display areas 222-F, 222-L, and 222-R. The front image IMG-F is displayed in display area 222-F. The left image IMG-L is displayed in display area 222-L. The right image IMG-R is displayed in display area 222-R. Display areas 222-L, 222-F, and 222-R are arranged in a horizontal row. Display areas 222-L, 222-F, and 222-R can be connected horizontally. Display area 222-F is sandwiched between display areas 222-L and 222-R. Display areas 222-L and 222-F are adjacent. Display areas 222-R and 222-F are adjacent. Display areas 222-L, 222-F, and 222-R have the same height. Because the front image IMG-F, the left image IMG-L, and the right image IMG-R are displayed adjacent to each other, the field of view of the remote operator O is further expanded. This further enhances the security of remote operations.

[0045] Additionally, display area 222 can also be referred to as a "screen". Display areas 222-L, 222-F, and 222-R can be connected to form a large screen. Display area 222 (screen) can be flat or curved.

[0046] The shooting directions of multiple camera CAMs (CAM-F, CAM-L, CAM-R) are different, which corresponds to the shooting directions of multiple image IMGs (IMG-F, IMG-L, IMG-R). Therefore, in the case of simply arranging multiple image IMGs adjacent to each other, such as... Figure 2 As shown, white lines and other lines on the road surface may appear unnaturally bent at the boundaries between images. That is, this may create a sense of disharmony at the boundaries between images. To make it easier for the remote operator O to remotely operate the vehicle 100, it is preferable to suppress this disharmony.

[0047] Therefore, in order to suppress the abrupt changes at the boundaries between images, the remote operating system 1 of this embodiment adjusts (corrects) at least one of the multiple image IMGs included in the image set. This process is hereinafter referred to as "image adjustment processing". Figure 2 In the example shown, to reduce the bending of the white lines at the boundaries between images, the left image IMG-L and the right image IMG-R are adjusted (corrected). The adjusted left image IMG-L and right image IMG-R are referred to as adjusted left image IMG-LX and adjusted right image IMG-RX, respectively. The front image IMG-F, adjusted left image IMG-LX, and adjusted right image IMG-RX are collectively referred to as the "adjusted image set". That is, the remote operating system 1 according to this embodiment obtains the adjusted image set by performing image adjustment processing on the image set. Then, the remote operating system 1 displays the adjusted image set on the display device 220 of the remote operator terminal 200. Through this image adjustment processing, the sense of incongruity felt by the remote operator O can be suppressed. As a result, it is easier for the remote operator O to remotely operate the vehicle 100.

[0048] 2-2. Specific Examples of Image Adjustment Processing

[0049] The following is for reference. Figure 3A , Figure 3B , Figure 4A and Figure 4B Specific examples of image adjustment processing will be explained.

[0050] like Figure 3A As shown, it is assumed that the viewpoints of multiple camera CAMs (CAM-F, CAM-L, CAM-R) are consistent. Furthermore, it is assumed that the fields of view of the multiple camera CAMs (CAM-F, CAM-L, CAM-R) are adjacent and continuous in the horizontal direction. The fields of view of the multiple camera CAMs do not overlap. In other words, the shooting ranges of the multiple image IMGs (IMG-F, IMG-L, IMG-R) are adjacent and continuous in the horizontal direction. The shooting ranges of the multiple image IMGs do not overlap. Moreover, even if the actual structure deviates slightly from these assumptions, a certain degree of effect can still be obtained.

[0051] Next, refer to Figure 3B This section explains the multiple imaginary displays DS (DS-F, DS-L, DS-R) used in image adjustment processing. The frontal image (IMG-F) captured by the front camera CAM-F is displayed on the imaginary front display DS-F. The left image (IMG-L) captured by the left camera CAM-L is displayed on the imaginary left display DS-L. The right image (IMG-R) captured by the right camera CAM-R is displayed on the imaginary right display DS-R.

[0052] More specifically, the hypothetical front display DS-F is positioned on plane PL-F. Plane PL-F is orthogonal to the optical axis AX-F of the front camera CAM-F, which captured the frontal image IMG-F. The hypothetical left display DS-L is positioned on plane PL-L. Plane PL-L is orthogonal to the optical axis AX-L of the left camera CAM-L, which captured the left image IMG-L. The hypothetical right display DS-R is positioned on plane PL-R. Plane PL-R is orthogonal to the optical axis AX-R of the right camera CAM-R, which captured the right image IMG-R. Since the directions of the optical axes AX-F, AX-L, and AX-R are different, the planes PL-F, PL-L, and PL-R are also different.

[0053] The front display DS-F and the left display DS-L are arranged adjacent and continuous in the horizontal direction. Similarly, the front display DS-F and the right display DS-R are arranged adjacent and continuous in the horizontal direction. Point C is the viewpoint corresponding to the multiple cameras (CAM). At point C, the optical axes AX-F, AX-L, and AX-R intersect. When viewed from point C, the horizontal field of view (HFOV) of the front display DS-F coincides with the HFOV of the front camera CAM-F. Here, the HFOV of the front display DS-F refers to the angle between the left and right ends of the front display DS-F when viewed from point C. Likewise, when viewed from point C, the HFOV of the left display DS-L coincides with the HFOV of the left camera CAM-L, and the HFOV of the right display DS-R coincides with the HFOV of the right camera CAM-R. The optical axis AX-F of the front camera CAM-F passes through the center of the front display DS-F. The optical axis AX-L of the left camera CAM-L passes through the center of the left display DS-L. The optical axis AX-R of the right camera CAM-R passes through the center of the right display DS-R.

[0054] exist Figure 3B In this context, "Hc" represents the HFOV of the front camera CAM-F, which is the HFOV of the front display DS-F. Furthermore, "Hs" represents the HFOV of the left camera CAM-L, which is the HFOV of the left display DS-L. The angle θ between the front display DS-F and the left display DS-L, i.e., the angle θ between plane PL-F and plane PL-L, is expressed by the following equation (1).

[0055] Equation (1): θ=Hc / 2+Hs / s

[0056] The angle formed by the front display DS-F and the right display DS-R, that is, the angle formed by plane PL-F and plane PL-R, is also the same.

[0057] Next, regarding Figure 4A and Figure 4B The following explanation is provided. Viewpoint D is the hypothetical viewpoint of the remote operator O as envisioned in the image adjustment process. Viewpoint D does not necessarily need to coincide with the actual viewpoint of the remote operator O. The relative positional relationship between viewpoint D and multiple hypothetical displays DS (DS-F, DS-L, DS-R) is pre-designed (set) by the designer, taking into account the setup information of multiple camera CAMs or the ease of remote operation when viewed from viewpoint D. The relative positional relationship between viewpoint D and multiple hypothetical displays DS (DS-F, DS-L, DS-R) can be pre-designed (determined) according to the vehicle model. For example, consider a coordinate system with the position of viewpoint D as the origin. The X-axis is the forward direction, the Y-axis is the horizontal direction (yaw direction), and the Z-axis is the height direction. Typically, viewpoint D is located on the optical axis AX-F of the front camera CAM-F. The configuration of the multiple imaginary displays DS (DS-F, DS-L, DS-R) in this coordinate system takes into account the setting information of multiple cameras CAM or the ease of remote operation when viewed from viewpoint D, and is pre-designed (set) by the designer.

[0058] And, as Figure 4B As shown, the height of viewpoint D is consistent with the height of the vanishing point in the front image IMG-F displayed on the front display DS-F. Furthermore, the height of viewpoint D is consistent with the height of the vanishing point in the left image IMG-L displayed on the left display DS-L. And, the height of viewpoint D is consistent with the height of the vanishing point in the right image IMG-R displayed on the right display DS-R. That is, in the image adjustment process, the height of viewpoint D is set in a manner consistent with the height of the vanishing point in each image displayed on each of the imaginary displays DS. Additionally, as... Figure 4B As shown, the heights of the multiple hypothetical displays DS (DS-F, DS-L, DS-R) are identical.

[0059] In image adjustment processing, the front image IMG-F is displayed on the front display DS-F corresponding to the front camera CAM-F. That is, the front image IMG-F is positioned on the plane PL-F corresponding to the front camera CAM-F. Furthermore, the left image IMG-L is displayed on the left display DS-L corresponding to the left camera CAM-L. That is, the left image IMG-L is positioned on the plane PL-L corresponding to the left camera CAM-L. Furthermore, the right image IMG-R is displayed on the right display DS-R corresponding to the right camera CAM-R. That is, the right image IMG-R is positioned on the plane PL-R corresponding to the right camera CAM-R. The front image IMG-F on plane PL-F (front display DS-F) and the left image IMG-L on plane PL-L (left display DS-L) are positioned adjacent to each other. Similarly, the front image IMG-F on plane PL-F (front display DS-F) and the right image IMG-R on plane PL-R (right display DS-R) are positioned adjacent to each other.

[0060] Then, the image adjustment process observes multiple image IMGs (IMG-F, IMG-L, IMG-R) configured from a common viewpoint D and projects these multiple image IMGs (IMG-F, IMG-L, IMG-R) onto a "common plane." The image IMG adjusted through this projection process is the adjusted image. That is, the image adjustment process obtains the adjusted image set by observing the image set from the common viewpoint D and projecting it onto a common plane.

[0061] exist Figure 4A In the example shown, the "common plane" is plane PL-F (first plane), which contains the front image IMG-F (first image). The image adjustment process generates an adjusted left image IMG-LX by projecting the left image IMG-L (second image) on plane PL-L (second plane) onto plane PL-F (first plane) from viewpoint D. At this time, the angle θ between plane PL-F (first plane) and plane PL-L (second plane) is given by the above equation (1). The image adjustment process projects the left image IMG-L (second image) on plane PL-L (second plane) onto plane PL-F (first plane) according to the angle θ given by the above equation (1). The projection transformation of the image is a known technique. The same applies to the right image IMG-R. The image adjustment process generates an adjusted right image IMG-RX by projecting the right image IMG-R (second image) on plane PL-R (second plane) onto plane PL-F (first plane) from viewpoint D. The adjusted image set includes the front image IMG-F, the adjusted left image IMG-LX, and the adjusted right image IMG-RX.

[0062] Furthermore, the common plane is not limited to plane PL-F. The common plane can be arbitrary. The common plane can be dynamically changed in response to the gaze direction of the remote operator O. The gaze direction of the remote operator O can be inferred, for example, from the steering information of the remote operator O or the travel direction of the vehicle 100. For example, if the remote operator O's steering direction is to the left and the steering angle is greater than or equal to a predetermined value, it can be inferred that the remote operator O is facing to the left, and the common plane can be set as the left-hand plane PL-L. More generally, the plane PL corresponding to the gaze direction of the remote operator O can be set as the common plane.

[0063] 2-3. Effects

[0064] As explained above, according to this embodiment, image adjustment processing is performed on an image set including multiple images (IMGs). In the image adjustment processing, a first image is positioned on a first plane, and a second image is positioned on a second plane. Furthermore, the image adjustment processing obtains the adjusted image set by observing the positioned first and second images from a common viewpoint D and projecting the first and second images onto the common plane. Then, the adjusted image set is displayed on the remote operator terminal 200. Through this image adjustment processing, for example, bending of lines such as white lines at the boundaries between images can be suppressed (see reference). Figure 2 That is, it reduces the sense of incongruity felt by the remote operator O when dealing with multiple image IMGs. As a result, it makes it easier for the remote operator O to remotely operate vehicle 100.

[0065] In addition, several assumptions were made in the above description, but even if the actual structure deviates slightly from these assumptions, a certain degree of effect can still be achieved.

[0066] 3. Functional Structure Example

[0067] Figure 5 This illustrates a functional structure related to image adjustment processing. The remote operating system 1 includes an image acquisition unit 10, an information management unit 40, an image adjustment processing unit 50, and an image display unit 60. These functional blocks are included in the vehicle 100 or the remote operator terminal 200.

[0068] The image acquisition unit 10 acquires multiple image IMGs (IMG-F, IMG-L, IMG-R) captured simultaneously by multiple cameras CAM (CAM-F, CAM-L, CAM-R) mounted on the vehicle 100. The image set includes multiple image IMGs (IMG-F, IMG-L, IMG-R). The image acquisition unit 10 is included in the vehicle 100.

[0069] The Information Management Department 40 manages and maintains the Image Adjustment Information INF. The Image Adjustment Information INF contains information required for image adjustment processing. For example, the Image Adjustment Information INF includes information on the field of view (horizontal and vertical) of each camera CAM mounted on the vehicle 100. The field of view of each camera CAM is determined by its hardware structure or performance. Furthermore, the Image Adjustment Information INF includes information on the relative positional relationship between the viewpoint D and multiple virtual displays DS (DS-F, DS-L, DS-R). As described above, this relative positional relationship is pre-designed (set) by the designer, taking into account the setup information (set position and orientation) of the multiple camera CAMs or the ease of remote operation when viewed from viewpoint D. This relative positional relationship can be pre-set according to the vehicle model. Moreover, the Image Adjustment Information INF includes information on the coordinates (especially the height) of the vanishing point on each image IMG. The coordinates of the vanishing point are determined based on the setup information (set position and orientation) of each camera CAM. Furthermore, the Image Adjustment Information may include information on the resolution of each camera CAM.

[0070] Information management unit 40 is included, for example, in vehicle 100. When image adjustment information INF is set according to vehicle model, information management unit 40 is preferably included in vehicle 100. However, information management unit 40 may also be excluded from vehicle 100. Information management unit 40 may be included in remote operator terminal 200 or management device 300.

[0071] The image adjustment processing unit 50 acquires an image set from the image acquisition unit 10. Furthermore, the image adjustment processing unit 50 acquires image adjustment information INF from the information management unit 40. Then, the image adjustment processing unit 50 performs image adjustment processing on the image set based on the image adjustment information INF, thereby acquiring an adjusted image set. The image adjustment processing is as described in Section 2 above. Additionally, the image adjustment processing unit 50 may be included in the vehicle 100 or in the remote operator terminal 200.

[0072] The image display unit 60 is included in the remote operator terminal 200. The image display unit 60 acquires an adjusted image set from the image adjustment processing unit 50. Then, the image display unit 60 displays the adjusted image set on the display device 220 of the remote operator terminal 200.

[0073] 4. Variations

[0074] Figure 6 This is a block diagram illustrating a modified example. The vehicle 100 is equipped with a fisheye camera CAM-X. The image acquisition unit 10 acquires an image IMG-X captured by the fisheye camera CAM-X.

[0075] The remote operating system 1 includes an image segmentation unit 30. The image segmentation unit 30 segments multiple image IMGs from an image IMG-X captured by a fisheye camera CAM-X. At this time, the multiple image IMGs are segmented in a manner where their shooting ranges are adjacent and continuous. For example, the field of view of the fisheye camera CAM-X is sometimes 180 degrees. In this case, the portion of the image corresponding to 90 degrees in front is segmented as the front image IMG-F, the portion corresponding to 45 degrees on the left is segmented as the left image IMG-L, and the portion corresponding to 45 degrees on the right is segmented as the right image IMG-R. The image set includes multiple image IMGs (IMG-F, IMG-L, IMG-R). Furthermore, the method of segmenting a portion of an image from an image captured by a fisheye camera is a known technique.

[0076] In addition to the information mentioned above, the image adjustment information INF managed by the Information Management Department 40 also includes information such as the optical axis center coordinates and projection method related to the fisheye camera CAM-X.

[0077] The image adjustment processing unit 50 acquires an image set from the image acquisition unit 10. Furthermore, the image adjustment processing unit 50 acquires image adjustment information INF from the information management unit 40. Then, the image adjustment processing unit 50 performs image adjustment processing on the image set based on the image adjustment information INF, thereby acquiring an adjusted image set. The image adjustment processing is as described in Section 2 above. Additionally, the image adjustment processing unit 50 may be included in the vehicle 100 or in the remote operator terminal 200.

[0078] When using the fisheye camera CAM-X, the line-of-sight height can also be changed according to instructions from the remote operator O. Therefore, the remote operating system 1 may further include a line-of-sight height indicator 70. The remote operator O uses the input device 230 to specify the line-of-sight height. The line-of-sight height indicator 70 acquires the line-of-sight height information specified by the remote operator O and transmits this information to the image adjustment processing unit 50.

[0079] The image adjustment processing unit 50 recalculates the coordinates (especially the height) of the vanishing point on the image IMG according to the line-of-sight height specified by the remote operator O. As a method for obtaining the coordinates of the vanishing point on the image IMG, the following methods can be considered. For example, the image adjustment processing unit 50 extracts feature points in the image IMG before and after the line-of-sight change, and uses the movement of the feature points as the movement of the vanishing point coordinates. As another example, the image adjustment processing unit 50 can use lane detection technology to detect multiple white lines on the road, and identify the intersection of the detected multiple white lines as the vanishing point. The image adjustment processing unit 50 performs image adjustment processing based on the recalculated coordinates of the vanishing point.

[0080] As explained above, the image adjustment processing described in this embodiment can also be applied to multiple image IMGs obtained using a fisheye camera CAM-X. The same effect as described above can also be achieved through this variation.

Claims

1. A remote operating system for remote operation of a mobile body operated by a remote operator, the remote operating system being characterized by comprising: One or more processors, The one or more processors are configured as follows: Acquire an image set comprising multiple images obtained by using one or more cameras mounted on the mobile body; Perform image adjustment processing to obtain an adjusted image set by adjusting at least one of the plurality of images; and The adjusted image set is displayed on the remote operator's terminal used by the remote operator. The plurality of images includes image 1 and image 2. The first plane is a plane orthogonal to the optical axis of the camera corresponding to the first image. The second plane is a plane orthogonal to the optical axis of the camera corresponding to the second image. The image adjustment process includes the following steps: The first image is positioned on the first plane, and the second image is positioned on the second plane; and The adjusted image set is obtained by observing the configured first image and the second image from a common viewpoint and projecting the first image and the second image onto a common plane.

2. The remote operating system according to claim 1, characterized in that, The relative positional relationship between the common viewpoint and the first and second planes is predetermined. The one or more processors are further configured to acquire image adjustment information representing the relative positional relationship, and perform the image adjustment processing based on the image adjustment information.

3. The remote operating system according to claim 1, characterized in that, The multiple images are obtained by capturing images separately by multiple cameras with adjacent fields of view or by segmenting images captured by fisheye cameras.

4. The remote operating system according to any one of claims 1 to 3, characterized in that, The shooting range of the first image is adjacent to the shooting range of the second image. The common plane is the first plane on which the first image is disposed. The image adjustment process includes the following steps: The first image on the first plane and the second image on the second plane are configured to be adjacent; A second adjusted image is generated by projecting the second image on the second plane onto the first plane; and Obtain the set of adjusted images that includes the first image and the second adjusted image.

5. The remote operating system according to claim 4, characterized in that, The image adjustment process includes the following steps: The second adjusted image is generated by projecting the second image on the second plane onto the first plane based on the angle between the first plane and the second plane.

Images