Vehicle surrounding image display device in remote operation system of vehicle

Abstract

To suppress loss and distortion in images as much as possible and provide higher-quality images when displaying an image of the surroundings of a vehicle captured by an onboard camera 2 with compensation for communication delay of the image from the vehicle to an operation console on a display 21 of the remote operation console 20 from the vehicle in a remote operation system for the vehicle 1.SOLUTION: A vehicle surrounding image display device of a remote operation system of a vehicle inputs an image captured by an onboard camera and movement information of the vehicle transmitted from the vehicle being a remote operation target into a model configured according to a machine learning algorithm that, upon receiving the input of any one high-precision map image, outputs a high-precision map image at a location moved from the image-capturing location, and displays, on a display of an operation console, an image estimated to be captured by the onboard camera at a time point after the elapse of the communication time required for communication of the output image data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a remote operation system for vehicles such as automobiles, and more particularly to an apparatus for displaying an image around a vehicle on a remote operation desk in a vehicle remote operation system.

Background Art

[0002] With the development of communication technology, a remote operation system for remotely operating a vehicle has been developed. In a vehicle remote operation system, usually, an image of the periphery (especially the front) of the vehicle taken by an in-vehicle camera is displayed on a display on an operation desk located remotely from the vehicle, and the operator executes the driving operation of the vehicle while viewing the image. In this regard, in the image data communication between the vehicle and the operation desk, a delay occurs from when the image is taken by the in-vehicle camera until it reaches the operation desk and is displayed on the display. Therefore, the image displayed on the display of the operation desk at a certain point in time may be significantly delayed compared to the image being taken by the in-vehicle camera at that time. On the other hand, since the vehicle is also moving during the communication of the image data, the situation appearing in the image on the display of the operation desk at a certain point in time may be significantly different from the situation around the vehicle at that time, and such a discrepancy can lead to misrecognition of the situation and operational errors. Therefore, techniques for attempting to compensate for the delay during image data communication have been proposed. For example, in Patent Document 1, in a configuration where a video of the front view of a vehicle taken by an in-vehicle television camera is transmitted by wireless communication to a driver's seat away from the vehicle and displayed on a television monitor there, together with the image, the driving information of the vehicle is also transmitted to the driver's seat, and an image processing device at the driver's seat calculates the distance the vehicle moves and the changing yaw angle during the delay time of video transmission (from the driving information) from the driving trajectory of the vehicle, estimates the true position and direction of the vehicle in the image that arrives with a delay of the delay time, and displays a virtual image of the vehicle at the true position of the vehicle on the television monitor to display a virtual image without apparent delay.

Prior Art Documents

Patent Documents

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-28495 [Summary of the Invention] [Problems to be Solved by the Invention]

[0004] When compensating for the delay in image data communication from a vehicle to a remote operation console as described above, if an attempt is made to estimate an image at another point in the future from a later time using only the image at a certain point in time, defects or distortions may occur in the estimated image, resulting in a decrease in image quality. For example, when estimating an image after the elapse of the communication time from only the image received from the vehicle (received image) using driving information, parts that are not shown in the received image may be missing, or unnatural distortions may occur when converting the orientation, position, length, etc. of the image. In such a case, it may be difficult to perform remote operation with sufficient accuracy.

[0005] In view of the above circumstances, the main problem of the present invention is to suppress as much as possible the defects and distortions in the displayed image and provide a higher-quality image when displaying an image of the periphery of the vehicle taken by an in-vehicle camera on the display of a remote operation console remote from the vehicle in a vehicle remote operation system, while compensating for the delay in image data communication from the vehicle to the operation console.

[0006] By the way, as part of the technology for the practical implementation of autonomous driving, a technology has been developed to widely collect high-precision image data (and its position information data) around vehicles obtained by a large number of vehicles equipped with sensors (such as cameras) for detecting the situation around the vehicle while driving on various roads through a communication network, accumulate them in a database, and construct a high-precision road map (high-precision map). In such a database, each of the images collected for the high-precision map (images for high-precision map) is accumulated together with the position information on the road where the vehicle that obtained each image was traveling. If an arbitrary position is specified, the data of the image for high-precision map corresponding to that position can be obtained from the database. Therefore, it is also possible to selectively extract, for example, the image for high-precision map at a certain point and the image for high-precision map at a point moved by an arbitrary distance from that point. That is, in the database of the high-precision map, images for high-precision map between any two points obtained by moving along an arbitrary road are available.

[0007] When it is possible to use the high-precision map image between two points along an arbitrary road from the database of the high-precision map as described above, using the data of the high-precision map image at an arbitrary point as input data and the data of the high-precision map image at a point moved by an arbitrary distance from that point as correct answer data, learning data is prepared along each road. Using these learning data, according to the algorithm of an arbitrary machine learning model, when an image for a high-precision map at a certain point and an arbitrary distance from that point are input, it is possible to construct a model trained to output an image for a high-precision map at a point at that arbitrary distance from that point. That is, by using the high-precision map images widely collected in advance from a large number of vehicles traveling various roads in the database as learning data and using the model trained as described above, (on the learned road) if an image of the vehicle surroundings obtained by the in-vehicle camera of a vehicle at an arbitrary point and the moving distance from there are input, an image of the vehicle surroundings of the in-vehicle camera that is estimated to be obtained at a point where the vehicle has moved the above-mentioned moving distance from such a point can be obtained. Here, the moving distance is given by the vehicle speed of the vehicle × moving time. If such a moving time is set to the data communication time (delay time) of the image in the remote operation system of the vehicle described above, according to the model trained as described above, on the operation console, from the image of the vehicle surroundings obtained by the in-vehicle camera of the remotely operated vehicle and the vehicle speed (and data communication time), an image of the vehicle surroundings of the in-vehicle camera that is estimated to be obtained by the in-vehicle camera at the time when the image is received on the operation console side can be obtained. Thereby, it becomes possible to compensate for the delay in the communication of the image and display the surrounding image of the remotely operated vehicle on the display of the operation console. Such knowledge is utilized in the present invention.

Means for Solving the Problems

[0008] According to the present invention, the above problem is a vehicle surrounding image display device in a vehicle remote operation system, Data receiving means for receiving the image data of the vehicle surrounding captured image captured by the in-vehicle camera of the vehicle targeted for remote operation and the movement information of the vehicle through wireless communication technology; Image estimation means for generating an estimated vehicle surrounding image estimated to be captured by the in-vehicle camera at a time point after the elapse of the communication time required for the communication of the image data from the vehicle to the data receiving means from the time point when the vehicle surrounding captured image was captured, based on the image data and the movement information received by the data receiving means; Display means for displaying the estimated vehicle surrounding image; comprising; The image estimation means has a machine learning model configured according to a machine learning algorithm using a number of learning data with the data of any one high-precision map image collected in the high-precision map image database as input data and the data of another high-precision map image at a point moved from the point where the one high-precision map image was captured as correct data. When the distance between the capture point of the one high-precision map image and the capture point of the another high-precision map image is specified and the data of the one high-precision map image is input, it is learned to output the data of the another high-precision map image. The apparatus is configured such that the image output from the machine learning model by inputting the image data of the vehicle surrounding captured image and the movement information received by the data receiving means into the machine learning model is the estimated vehicle surrounding image. is achieved by.

[0009] In the above configuration, the "remote vehicle operation system" is, as already described, a system for remotely operating a vehicle. Here, the driver (operator) of the vehicle gives instructions at an operation console placed away from the vehicle, and these instructions are sent to the vehicle through any wireless communication means to operate the vehicle. The "vehicle surrounding image display device" according to the present invention is a device that, in such a remote vehicle operation system, displays, as an image, the situation captured by a camera from the vehicle on a display for the driver to check the situation around the vehicle, particularly the situation in the traveling direction of the vehicle. It may be incorporated as a part of the remote operation system. "Image data" is the data obtained by digitizing the image captured by an in-vehicle camera. Since such image data generally has a large data volume and the communication time from the vehicle to the operation console is long, the image reaching the operation console will be significantly delayed compared to the image actually being captured by the in-vehicle camera. "Movement information" is data used to determine the moving distance of the vehicle, and may be data representing vehicle speed, position, moving distance, etc. Note that for these data, the communication time from the vehicle to the operation console is very short, and delay is hardly a problem in the device of the present invention. The "data receiving means" may be a communication means using ordinary techniques used in this field. The "image estimation means" is, as described above, using the data of the "vehicle surrounding captured image", which is the image of the vehicle surrounding actually captured by the in-vehicle camera of the vehicle targeted for remote operation, and its movement information, outputs a "vehicle surrounding estimated image" that is estimated to be captured by the in-vehicle camera of the vehicle at the time when the data of the "vehicle surrounding captured image" reaches the operation console or later. The "image estimation means" may be realized by a computer device operating according to a program. The "display means" may be a display for displaying images commonly used in this field. Also, the "image database for high-precision map", as already described, is a database that widely collects "images for high-precision map", which are images of the vehicle surrounding sequentially captured along the driving route when a large number of vehicles travel in various regions for constructing a high-precision map. Such a database may be placed, for example, in a cloud network and be accessible in a timely manner.The "machine learning model" may be a model configured using any machine learning algorithm capable of generating an image for any input image using deep learning technology or the like.

[0010] According to the above configuration of the present invention, in the remote operation of a vehicle during travel, when the distance between the shooting point of one high-precision map image and the shooting point of another high-precision map image is specified and the data of one high-precision map image is input, using a machine learning model trained to output the data of another high-precision map image, from the image actually captured by the vehicle's camera (vehicle surrounding captured image), an image (vehicle surrounding captured image) that is supposed to be captured by the in-vehicle camera at any point in time (such as the point in time when the image is displayed on the display, the point in time when the driver refers to the display to execute an operation, etc.) after the time when the image data reaches the operation console is generated. That is, the vehicle surrounding estimated image can be regarded as an image that compensates for the delay in the communication time of the image data of the vehicle surrounding captured image, and the image output from the machine learning model is generated using the high-precision map image at the point where it is estimated that the vehicle surrounding estimated image is obtained in the remotely operated vehicle. Therefore, the parts not shown in the vehicle surrounding captured image are interpolated, there is no unnatural image movement, and the defects and distortions in the displayed image are suppressed as much as possible, and it is expected to be a higher-quality image (compared to the case of using only the vehicle surrounding captured image).

[0011] In the above configuration, the movement information input to the machine learning model for generating the vehicle surrounding estimated image may be in any form. Specifically, when using the vehicle speed of the remotely operated vehicle as the movement information, the distance obtained by multiplying the vehicle speed by the communication time to the operation console of the image data may be specified in the machine learning model. As the communication time of the image data, a general average value may be used, or an instantaneous value may be specified. Also, as the movement information, the position information of the vehicle obtained from the navigation device may be used, and the distance between the position of the vehicle at the time of receiving the vehicle surrounding captured image and the position of the vehicle at the time point traced back by the communication time from there may be specified in the machine learning model.

[0012] Incidentally, in the data communication of the image from the vehicle to the operation console, loss of image data may occur, so there may be cases where the captured image of the vehicle surroundings received at the operation console has missing or unclear parts. Regarding this point, as described above, when using the high-precision map images of various locations, a machine learning model (high-resolution learning model) can also be constructed to compensate and convert an image with missing or unclear parts at various locations into an image without missing or unclear parts. Specifically, the high-resolution learning model uses one high-precision map image as the correct data, and a large number of learning data with images in which missing or unclear parts are generated in various ways in the one high-precision map image as the input, and performs learning processing so as to generate the high-precision map image from the image in which missing or unclear parts are generated therein. Therefore, in the device of the present invention, further, using the above high-precision map image as learning data, and using a high-resolution model constructed to generate an image without missing or unclear parts from an image with missing or unclear parts due to loss in data communication, means for performing high-resolution processing on the captured image of the vehicle surroundings or the estimated image of the vehicle surroundings may be provided.

[0013] In the above configuration, the high-precision map image database is sequentially updated, and the above machine learning model for generating the estimated image of the vehicle surroundings from the captured image of the vehicle surroundings is also sequentially updated, and the accuracy may be improved.

[0014] The image estimation means of the present invention described above may be installed in the computer device of the operation console, or may be installed in a server computer separate from the operation console. In the former case, the data of the captured image of the vehicle surroundings and the movement information are directly transmitted to the operation console, where the estimated image of the vehicle surroundings is generated. In the latter case, they are transmitted to another server computer, where the estimated image of the vehicle surroundings is generated, and further, the generated estimated image of the vehicle surroundings is transmitted to the operation console.

Effect of the Invention

[0015] Thus, according to the present invention, in a remote operation system for a vehicle, a vehicle surrounding image with communication delay compensation of image data based on a vehicle surrounding captured image and movement information by an in-vehicle camera of a vehicle to be remotely operated by a machine learning model constructed using a high-precision map image is expected to have less loss and distortion and be of higher quality (compared to the case of using only the vehicle surrounding captured image), and it is expected that such high-quality images will be displayed on an operation desk, making it possible.

[0016] Other objects and advantages of the present invention will become apparent from the following description of the preferred embodiments of the present invention.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Explanation of Reference Numerals

[0018] 1…Vehicle, 2…Camera, 3…Communicator, 4…Vehicle speed detector, 20…Operator console, 21…, Display, 22…Communicator, 30…Server, 31…Communicator

Best Mode for Carrying Out the Invention

[0019] Configuration of the vehicle remote operation system The image display device according to this embodiment is a device that, in a remote operation system for remotely operating and driving a vehicle 1 from a remote operator console 20 as schematically depicted in FIG. 1, transmits the image data of an image of the surrounding of the vehicle (image of the surrounding of the vehicle taken by the in-vehicle camera 2 of the vehicle 1 to be remotely operated) to the operator console 20, compensates for the delay in the transmission time of the image data from the vehicle 1 to the operator console 20 in the image of the surrounding of the vehicle taken, and displays the compensated image (estimated image of the surrounding of the vehicle) on the display 21 of the operator console 20. Communication for remotely operating the vehicle 1 may be performed directly between the communicator 3 of the vehicle 1 and the communicator 22 of the operator console 20 or via the communicator 31 of a server (relay server) 30 using a normal mode of wireless communication technology (for communication between the server 30 and the operator console 20, a faster wired communication technology may be used). In the case of this embodiment, for delay compensation of the image of the surrounding of the vehicle taken, movement information (vehicle speed information, position information) of the vehicle 1 obtained by a vehicle speed detector 4 or the like of the vehicle 1 is transmitted to the operator console 20 or the server 30 through the communicator 3. The operation of the system in the operation of the vehicle 1 in the remote operation system may be in a normal mode.

[0020] In one aspect of the configuration related to the operation of the image display device in the above remote operation system, referring to FIG. 2(A), in the vehicle 1 to be remotely operated, the image data of the vehicle surrounding captured image captured by a camera that captures the surrounding of the vehicle (especially the traveling direction of the vehicle), and the vehicle speed detected in an arbitrary manner from the wheel speed detected by the wheel speed sensor (or the position information obtained from a navigation device (not shown), etc.) are transmitted from the communicator to the communicator of the operation console. At the operation console terminal, the image data of the vehicle surrounding captured image received from the vehicle and information such as the vehicle speed are given to the image processing unit, and there, using the model for image processing prepared by the server described later and downloaded to the model storage unit, as shown in FIG. 2(B), a process for compensating the delay in the time required for transmitting the image data (delay compensation processing unit) or further a process for compensating the loss during transmission of the image data (high-resolution processing unit) is executed, and the processed image (estimated vehicle surrounding image) is displayed on a display visible to the operator of the vehicle. At the server, the image processing model used in the image processing unit of the operation console terminal is generated (generation server), stored in the database, and can be downloaded to the operation console terminal in a timely manner. In the generation of the image processing model, the high-precision map image as described above is appropriately extracted from its database and used as learning data. As the high-precision map image, vehicle surrounding images captured by in-vehicle cameras in many vehicles traveling on various roads, etc., may be widely collected together with the information on the shooting positions to a high-precision map image collection server. Although not shown, images obtained from the vehicle to be remotely operated may also be used as the high-precision map image.

[0021] Configuration of image processing (a) Overview As described above, in a remote operation system of a vehicle, when an image of the vehicle surroundings captured by the vehicle to be remotely operated for operating the vehicle on the operation console is transmitted from the vehicle to the operation console and displayed on a display on the operation console, since it takes a significant amount of time to transmit the image data (and other processes) from the vehicle to the operation console, the time when the image captured by the vehicle can be displayed on the display of the operation console is significantly delayed from the time of image capture. On the other hand, since the vehicle can also move during the time from the time of image capture to the time of display of such an image, if the image captured by the vehicle is directly displayed on the display of the operation console, there may be a difference between the situation that should be visible from the vehicle at that time and the situation displayed on the display, and this deviation in the situation can lead to misrecognition of the situation and operational errors. Therefore, when displaying an image of the vehicle surroundings on the display of the operation console, it is advantageous if the above delay can be compensated for the image captured by the vehicle. Also, in the transmission (and other processes) of image data from the vehicle to the operation console, part of the data may be lost, so if the image received at the operation console is directly displayed, the image may become unclear or partial defects may occur, and this can also lead to misrecognition of the situation and operational errors. Therefore, it is advantageous if the above image loss can be compensated when displaying an image of the vehicle surroundings captured by the vehicle on the display of the operation console. When compensating these images, as already mentioned, it is preferable to suppress as much as possible the defects and distortions in the displayed image and provide a higher-quality image.

[0022] Therefore, in the present embodiment, as one aspect, using an image (image for high-precision map) stored for a high-precision map as learning data, when an image is input, a model that outputs or generates an image estimated to be taken at a point moved by an arbitrary distance from the point of that image is prepared by machine learning, and using that model, delay compensation such as the communication time is performed on the image taken by the vehicle. Further, as another aspect, using an image for high-precision map as learning data, when an input image with a certain defect is input, a model that outputs an image with the defect compensated is prepared by machine learning, and using that model, defect compensation is performed on the image received at the operation console.

[0023] (b) Delay Compensation In the learning stage of a model (delay compensation learning model) that compensates for the delay such as the communication time of image data from the vehicle to the operation console (or the delay time from the shooting time to the display time), from the high-precision map image database, arbitrary vehicle surrounding images before and after moving at various set distances are extracted, and a large number of learning data are prepared with the image before moving as input data and the image after moving as correct answer data. According to an arbitrary machine learning algorithm, a model is prepared that is learned to output an image after moving the specified distance for the input of the image before moving in a state where various distances are specified. Specifically, for example, as shown in FIG. 3(A), as the input image, one image as shown on the left in the figure is set, and another image as shown on the right in the figure obtained by moving Xm from there is set as the correct answer data, and the parameters of the model are adjusted so that when one image is input, an image after moving Xm is output (generated). Then, by executing this learning process using images obtained at various roads and various moving distances, when an arbitrary image is input by specifying the moving distance, a model is configured that outputs (generates) an image estimated to be obtained at the position moved by the specified distance from the input image.

[0024] In the stage of executing the image processing according to the above model in the remote operation of the vehicle, when the delay compensation unit incorporating the delay compensation learning model receives the data of the image (image of the vehicle surroundings) taken by the camera from the vehicle to be remotely operated and the vehicle speed value, the moving distance X [m] of the vehicle during that time is calculated by multiplying the vehicle speed value V [km / h] by the time Δt [s] required for image communication etc. (see Fig. 3(A)), and by specifying such a moving distance, the image of the vehicle surroundings is input to the delay compensation learning model. Then, the model generates and outputs an image (estimated image of the vehicle surroundings) estimated to be taken at a point moved by the moving distance from the shooting point of the image of the vehicle surroundings. Since this output image is estimated to match the image taken from the current position of the vehicle, as a result, the image compensated for delay is displayed on the display.

[0025] In the case of the above delay compensation model, since it is generated using the high-precision map image at the point where the estimated image of the vehicle surroundings is presumed to be obtained, it is expected that the parts not shown in the image of the vehicle surroundings are also interpolated with the high-precision map image corresponding to the point of the estimated image of the vehicle surroundings, there is no unnatural image movement, and the defects and distortions in the displayed image are suppressed as much as possible, and it is expected to be a higher-quality image (compared to the case of using only the image of the vehicle surroundings).

[0026] In the above configuration, the time required for image communication etc. may be an actual value or an average value. Instead of the vehicle speed value of the vehicle to be remotely operated, the moving distance may be obtained from the position information or the moving distance information of the vehicle.

[0027] (c) Higher resolution In the learning stage of the model (high-resolution deep learning model) that compensates for the loss of the image transmitted from the vehicle and received on the operation console, any image in the high-precision map image database is used as the correct data as it is, and a large number of training data are prepared with the image obtained by randomly deleting the data of that image as the input data. According to an arbitrary machine learning algorithm, when the image with the deletion is input, a model is prepared that is trained to output the image before the deletion. Specifically, for example, as shown in Fig. 3(B), one image shown on the right in the figure is set as the correct data, and an image obtained by deleting the data of the right image and reducing its resolution as shown on the left in the figure is set as the input image. The parameters of the model are adjusted so that when the left image is input, the right image is output (generated). Then, by executing the learning process using images obtained at various locations, when an arbitrary image is input and there is a loss in the input image, a model is configured to output an image in a state where the loss is compensated (i.e., an image with increased resolution).

[0028] In the stage of performing image processing by the above model in the remote operation of the vehicle, when the image processed by the above delay compensation unit is input to the high-resolution deep learning model, an image with increased resolution is output. Since it is estimated that when there is a loss in the image data, the output image is in a state where the lost part is compensated, the image before the loss will be displayed on the display accordingly.

[0029] In the above high-resolution model, a process of adding color to a black-and-white image may be executed simultaneously. In that case, learning is executed in the same manner as above using, as the training data, an image obtained by converting the high-precision map image into a black-and-white image as the input data and the color image as the correct data.

[0030] As described above, it is expected that more accurate remote operation will be possible by performing delay compensation on the image captured by the vehicle and / or by displaying the image with increased resolution on the display of the operation console.

[0031] Execution location of image processing As shown in FIG. 4(A), the above-described image processing unit may be provided in a server that relays communication between the vehicle and the operation console. In particular, when the data processing ability of the operation console is not so high, image compensation may be executed by the server, and the compensated image may be transmitted to the operation console. (If the server and the operation console are connected by wired communication, it is expected that data communication can be achieved at a higher speed.)

[0032] Whether the image processing is executed on the operation console or the server may be determined according to the flowchart of FIG. 4(B). Referring to the figure, specifically, (i) when the model can be downloaded to the operation console terminal and the image processing can be performed within the frame interval of the camera, or (ii) when the image processing cannot be performed within the frame interval of the camera, but a part of the image processing can be reduced and the image processing can be performed within the frame interval of the camera, a part of the image processing may be reduced and the image processing may be executed on the operation console terminal. On the other hand, (iii) when the model cannot be downloaded to the operation console terminal, or (iv) when the image processing cannot be performed within the frame interval of the camera and a part of the image processing cannot be reduced, the image processing may be executed by the server.

[0033] The above description has been made in relation to the embodiments of the present invention. However, many modifications and changes are easily possible for those skilled in the art, and the present invention is not limited to only the embodiments illustrated above, and it will be apparent that the present invention can be applied to various devices without departing from the concept of the present invention.

Claims

【Claim 1】 A vehicle peripheral image display device in a vehicle remote operation system, comprising: data receiving means for receiving image data of a vehicle peripheral captured image captured by an in-vehicle camera of the vehicle and movement information of the vehicle, which are transmitted through a wireless communication technology from the vehicle that is the target of remote operation; image estimation means for generating a vehicle peripheral estimated image estimated to be captured by the in-vehicle camera at a time point after the elapse of the communication time required for the communication of the image data from the vehicle to the data receiving means, from the time point when the vehicle peripheral captured image was captured, based on the image data and the movement information received by the data receiving means; display means for displaying the vehicle peripheral estimated image; and the image estimation means is a machine learning model configured according to a machine learning algorithm using a number of learning data with data of any one high-precision map image collected in a high-precision map image database as input data and data of another high-precision map image at a point moved from the point where the one high-precision map image was captured as correct data. When the distance between the capture point of the one high-precision map image and the capture point of the another high-precision map image is specified and the data of the one high-precision map image is input, it is learned to output the data of the another high-precision map image. The device is configured such that an image output from the machine learning model by inputting the image data and the movement information of the vehicle peripheral captured image received by the data receiving means into the machine learning model is the vehicle peripheral estimated image.

Images