Vehicle surrounding moving image transmission device in vehicle remote control system

Abstract

To transmit moving images captured by an on-board camera from a vehicle to a control console with the highest possible quality and without interruption, in a vehicle remote control system.SOLUTION: A vehicle surrounding moving image transmission device uses a machine learning model that outputs the required bandwidth of a converted moving image when a moving image captured by an on-board camera is converted into a moving image with image quality set by an image quality setting value, based on an input of the image quality setting value for the moving image captured by the on-board camera around the vehicle, which is transmitted from the vehicle to a control console via wireless communication and the vehicle's motion state index value, and searches for an image quality setting value that makes the required bandwidth less than or equal to an available bandwidth and as large as possible, converts the moving image by the on-board camera into a moving image with the image quality set by the searched image quality setting value, and transmits it to the control console.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a remote control system for a vehicle such as an automobile, and more particularly to a device for transmitting moving images of the surroundings captured by an on-board camera from the vehicle to a remote control table in a vehicle remote control system. [Background technology]

[0002] Advances in communication technology have led to the development of remote control systems for remotely driving and operating vehicles. In a vehicle remote control system, video images of the vehicle's surroundings (particularly the front) captured by an on-board camera are typically transmitted from the vehicle to a remote control console and displayed on a display on the console, allowing the operator to drive the vehicle while viewing the transmitted video images. In such vehicle remote control technology, for example, Patent Document 1 proposes that communication between an on-board remote control device and a control center be performed using a wide-area communication system and a short-range communication system, so that even if communication via one wireless communication system becomes unavailable, communication can still be performed via the other wireless communication system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2004-295360 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described vehicle remote control system, in order to achieve more accurate remote driving of the vehicle, it is preferable to be able to transmit video captured by an onboard camera from the vehicle to the cockpit with as high quality and without interruption as possible. To achieve this, the transmission bandwidth required for uninterrupted video transmission (hereinafter referred to as the "required bandwidth") must be equal to or less than the available bandwidth for wireless communication between the vehicle and the cockpit. In this regard, the required bandwidth for video captured by an onboard camera is determined by the quality (e.g., FPS, pixel size, compression rate, color depth, number of colors) set for the video, as well as the amount of image change between consecutive images (hereinafter referred to as the "image change amount"). The amount of image change depends on changes in the situation around the vehicle captured by the onboard camera, which in turn depends on the vehicle's motion state (e.g., whether the vehicle is moving or turning, its speed, acceleration, yaw rate, etc.). In other words, the required bandwidth for video from an onboard camera is determined by the set quality of the video and the vehicle's motion state. Therefore, if the required bandwidth for video from an onboard camera can be estimated based on the set quality of the video and the momentary motion state of the vehicle, and the set quality of the video can be adjusted so that the estimated required bandwidth is equal to or less than the available bandwidth at that time but as large as possible, it will be possible to transmit video from an onboard camera with as high quality as possible and without interruption. Estimation of the required bandwidth for video to be transmitted based on the set quality of the video and the motion state of the vehicle can be achieved using a model configured to output the required bandwidth for video in response to the input of the set quality of the video and the motion state of the vehicle, according to any machine learning algorithm. The available bandwidth can be detected in the vehicle using any method.

[0005] In view of the above circumstances, a main object of the present invention is to enable a vehicle remote control system to transmit video images captured by an on-board camera from the vehicle to an operation console with as high quality as possible and without interruption. [Means for solving the problem]

[0006] According to the present invention, the above problem is solved by a vehicle surroundings moving image transmission device in a vehicle remote control system, a required bandwidth determination means for outputting a required bandwidth for a converted moving image when the moving image captured by the on-board camera is converted into a moving image of image quality set by the image quality setting value, based on an input of an image quality setting value for the moving image captured by the on-board camera around the vehicle and transmitted from the vehicle to an operation console that remotely operates the vehicle via wireless communication and a motion state index value of the vehicle; an available bandwidth detection means for detecting an available bandwidth in wireless communication from the vehicle to the console; an image quality setting value search means for inputting various image quality setting values ​​to the necessary bandwidth determination means, causing the necessary bandwidth determination means to output the necessary bandwidth, and searching for an image quality setting value that makes the necessary bandwidth equal to or smaller than the available bandwidth and as large as possible; a moving image conversion means for converting the moving image captured by the vehicle-mounted camera into a moving image of image quality set by the image quality setting value searched by the image quality setting value search means; an image transmission means for transmitting the moving image obtained by the conversion by the moving image conversion means to the operation console; Including, The required bandwidth determination means is configured to use, as input data, a vehicle motion state index value when a pre-collected video of the periphery of the vehicle is obtained and an arbitrary image quality setting value, and to output, according to a machine learning algorithm, the required bandwidth of the converted video when the pre-collected video is converted into a video of image quality set at the arbitrary image quality setting value, using a large amount of learning data in which the required bandwidth of the converted video when the pre-collected video is converted into a video of image quality set at the arbitrary image quality setting value. This is achieved by:

[0007] In the above-described configuration of the present invention, the "vehicle remote control system" refers to a system for remotely driving and operating a vehicle, as described above. In this system, the driver (operator) of the vehicle issues commands from a console located away from the vehicle, which are then transmitted to the vehicle via any wireless communication means, thereby operating the vehicle. The "vehicle surroundings video transmission device" of the present invention is an in-vehicle device for transmitting video images captured by a camera mounted on the vehicle to the driver's console so that the driver can check the conditions around the vehicle, particularly the conditions in the vehicle's direction of travel. The "image quality setting value" refers to parameters for setting the quality of the video, such as frames per second (FPS), pixel size, compression rate, number of colors, and color depth. The term "one image quality setting value" refers to one combination of parameters for setting the quality of the video. Furthermore, converting video captured by an onboard camera into video with a quality set by a picture quality setting value means converting the video captured by the onboard camera into a video so that each parameter that determined the original quality of the video becomes the corresponding parameter set by the picture quality setting value. For example, if the pixel count of the video captured by the onboard camera is 1920 x 1080 and the pixel count of the picture quality setting value is 1280 x 720, the 1920 x 1080 captured video will be converted into a 1280 x 720 video during readjustment using the picture quality setting value. The "vehicle motion state index value" may be any index value that represents the vehicle's motion state, and specifically may be vehicle speed, acceleration, yaw rate, etc. detected by the vehicle.

[0008] In the configuration of the present invention, the required bandwidth determination means is configured to, when a vehicle motion state index value and an image quality setting value are input, output a required bandwidth required for transmitting a moving image determined in accordance with the image change amount corresponding to the motion state index value and the image quality setting value using a machine learning algorithm, as described above. Here, any of the methods listed below can be used as the machine learning algorithm.

[0009] In the above-described configuration of the device of the present invention, when transmitting video captured by an onboard camera from a vehicle to an operation console via wireless communication, a picture quality setting value, which is a parameter for setting the quality of the video, is searched for taking into consideration the vehicle's motion state so that the transmission bandwidth (required bandwidth) that achieves uninterrupted video transmission is maximized within the available bandwidth, and the video captured by the onboard camera is converted into a video of the quality set by the searched parameter, and the reconverted video is transmitted. With this configuration, the required bandwidth for the video is always equal to or less than the available bandwidth regardless of the vehicle's motion state, allowing for uninterrupted video transmission. Furthermore, since the transmission bandwidth is maximized within the available bandwidth, video of the highest possible picture quality is transmitted to the operation console, which is expected to enable more accurate driving operations.

[0010] In the above configuration, if the vehicle-mounted cameras are provided at the front, sides, and rear of the vehicle, the image quality setting values ​​may be determined separately for each camera since the amount of image change differs for each camera.

[0011] In the above configuration, the available bandwidth detection means can detect the available bandwidth using various methods, such as a passive method that uses estimation and prediction from information passively obtained by the in-vehicle communication device (RSSI, current communication volume, change in communication bandwidth), an inquiry method that uses a communication quality map on an external server, or an active method that performs test communication for measurement to an external terminal. [Effects of the Invention]

[0012] Thus, in the vehicle surroundings video transmission device of the vehicle remote control system of the present invention, focusing on the fact that the amount of image change per unit time in the video captured by the onboard camera (i.e., the required bandwidth for uninterrupted transmission of the video) depends on the vehicle's motion state, when transmitting the video captured by the onboard camera from the vehicle to the control console, the video quality of the transmitted video is adjusted taking into account the vehicle's motion state so that the required bandwidth is maximized within the available bandwidth, thereby enabling the video captured by the onboard camera to be transmitted from the vehicle to the control console with the highest possible quality and without interruption, even if the vehicle's motion state changes in various ways. With this configuration, the video captured by the onboard camera is displayed uninterruptedly on the control console display, which is expected to make remote control of the vehicle easier and more accurate.

[0013] Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the invention. [Brief explanation of the drawings]

[0014] [Figure 1] Fig. 1(A) is a schematic diagram of a vehicle remote control system to which this embodiment is applied. Fig. 1(B) is a block diagram showing the configuration of a vehicle periphery video transmission device according to this embodiment. Fig. 1(C) is a schematic diagram of a vehicle showing the capture range of video images transmitted by the vehicle periphery video transmission device according to this embodiment. [Figure 2] FIG. 2 shows an example of the change in the amount of image change per unit time (one frame) of a moving image captured around a vehicle (top) and an example of the image transmitted at that time (bottom). [Figure 3] FIG. 3 is a flowchart showing the process of determining the image quality setting value in the vehicle surroundings moving image transmission device according to this embodiment. [Explanation of symbols]

[0015] 1...vehicle, 2...camera, 3...vehicle surroundings moving image transmission device, 4...communication device, 20...operation console, 21..., display, 22...communication device, 30...server, 31...communication device BEST MODE FOR CARRYING OUT THE INVENTION

[0016] Vehicle remote control system configuration The vehicle surroundings video image transmission device according to this embodiment is a device 3 that transmits data of video images of the vehicle surroundings captured by an on-board camera 2 of the vehicle 1 to be remotely operated to the remote operation console 20 in a remote operation system in which the vehicle 1 is remotely operated from the remote operation console 20, as shown schematically in FIG. 1(A). Communication for remotely operating the vehicle 1 may be performed directly between the communicator 4 of the vehicle 1 and the communicator 22 of the operation console 20, or may be performed using a conventional wireless communication technique via a network or a relay server (not shown) (communication between the server and the operation console 20 may use a higher-speed wired communication technique). Operation of the system for operating the vehicle 1 in the remote operation system may be conventional.

[0017] Configuration of vehicle surroundings video transmission device In short, the vehicle surroundings video transmission device of this embodiment in the above-mentioned remote control system determines image parameters (image quality setting values) that determine the quality of the video to be transmitted so that, when transmitting video captured by the vehicle-mounted camera 2 of the vehicle 1 to the remote control console 20, the video is transmitted without interruption and with the highest possible quality, so that the amount of data (communication volume) sent per unit time of the transmitted video is maximized and not greater than the available bandwidth from the communicator 4 of the vehicle 1 to the communicator 22 of the control console 20, i.e., so that the transmission bandwidth (required bandwidth) required to transmit the video without interruption on the display 21 of the control console 20 is maximized and not greater than the available bandwidth, converts the video captured by the vehicle-mounted camera 2 into video of the image quality set by the determined image quality setting values, and then converts the converted video data into data for transmission, and the data is transmitted from the communicator 4 of the vehicle 1 to the communicator 22 of the control console 20. In order to achieve the above operation, the configuration of the vehicle surroundings video transmission device according to this embodiment includes, as shown in Figure 1(B), a video converter that converts the video images captured by the vehicle-mounted camera and an on-board communication device that transmits the image data to the console side, as well as a communication quality estimator, a required bandwidth determiner, and a required bandwidth searcher.

[0018] More specifically, the communication quality estimator is configured to detect the available bandwidth from the communication device 4 of the vehicle 1 to the communication device 22 of the console 20. Such detection of the available bandwidth can be achieved by various methods. Specifically, for example, the available bandwidth may be estimated or predicted from information passively obtained by the in-vehicle communication device, such as RSSI, current communication volume, and changes in communication bandwidth (passive method). Alternatively, the available bandwidth can be detected by an inquiry method using a communication quality map on an external server or an active method in which test communication for measurement is performed to an external terminal.

[0019] The required bandwidth determiner is configured to receive various index values ​​representing the vehicle's motion state (such as the vehicle speed calculated from the detection values ​​of the wheel speed sensors, the acceleration detected by the acceleration sensor, and the yaw rate detected by the gyro sensor) and image quality setting values ​​(such as FPS, pixel size, compression rate, and color depth), and to output the required bandwidth (i.e., the transmission speed required to complete the transmission of data required to generate the next image between consecutive frames) for seamlessly transmitting the converted video when video images captured by a camera on the vehicle in the input motion state are converted into video images of a quality set by the input image quality setting values. In this regard, as explained in the Summary of the Invention section, in the case of video images captured by an on-board camera, the amount of change in the image between consecutive images varies depending on the vehicle's motion state, so the required bandwidth will vary depending on the vehicle's motion state as well as the image quality setting values ​​that determine the quality of the video images. Specifically, as illustrated in Figure 2, for example, when a vehicle is stopped, the image in the captured video ((a) in Figure 2) hardly moves, so the amount of image change per unit time is small and the required bandwidth is small, but as the vehicle speed increases or when turning, the displacement of the images in the captured video ((b) to (d) in Figure 2) becomes greater, so the amount of image change per unit time increases and the required bandwidth increases. Therefore, as described above, the required bandwidth determiner in this embodiment uses not only the image quality setting value that determines the quality of the transmitted image but also the vehicle motion state index value as described above as input variables for determining the required bandwidth for the transmitted image.

[0020] As already mentioned, the required bandwidth determiner may be configured using an arbitrary machine learning algorithm. Specifically, as a machine learning model that outputs the required bandwidth for a video to be transmitted, a machine learning model may be prepared that uses, as training data, a vehicle motion state index value when a large number of pre-collected videos of the vehicle's surroundings are obtained and an arbitrary image quality setting value, and a large number of data sets, in which the pre-collected videos are converted into videos with image quality set at the arbitrary image quality setting value, as correct answer data, and that is trained according to an arbitrary machine learning algorithm to output the required bandwidth for a converted video when a vehicle motion state index value when a video of the vehicle's surroundings is obtained and a certain image quality setting value are input, and the required bandwidth determiner may be configured, using the machine learning model, to output the required bandwidth for a converted video when a video captured in the input vehicle motion state is converted into a video with image quality set at the input image quality setting value, when the vehicle motion state index value and the image quality setting value are input. As machine learning algorithms, decision trees, k-nearest neighbors, random forests, support vector machines, neural networks, etc. may be used.

[0021] The necessary bandwidth searcher is configured to search for an image quality setting value that converts video images captured by the camera in the vehicle's current motion state into video images with the maximum necessary bandwidth that is equal to or less than the available bandwidth, with reference to the available bandwidth detected by the communication quality estimator. Specifically, as shown in Fig. 3 described below, the necessary bandwidth searcher sequentially changes the image quality setting value while providing it to the necessary bandwidth determiner, which calculates the necessary bandwidth corresponding to the image quality setting value and the vehicle's current motion state index value, and determines whether the calculated necessary bandwidth is equal to or less than the available bandwidth, repeating this process until the necessary bandwidth becomes equal to or less than the available bandwidth. In this way, the necessary bandwidth searcher detects an image quality setting value that converts video images captured by the camera in the vehicle's current motion state into video images with the maximum necessary bandwidth that is equal to or less than the available bandwidth.

[0022] When the required bandwidth searcher determines the image quality setting value that provides the maximum required bandwidth below the available bandwidth, the video converter converts the video captured by the camera into a video with the image quality set to the determined image quality setting value, and then converts it into data to be transmitted by the in-vehicle communication device, and the converted image data is transmitted from the in-vehicle communication device to the operation console. The video converter and the in-vehicle communication device may have configurations commonly used in this field.

[0023] In addition, when multiple cameras are installed on one vehicle to capture images of the surroundings behind the vehicle R and the surroundings to the sides of the vehicle S in addition to the surroundings in front of the vehicle F, as shown in Fig. 1(C), and the moving images are transmitted to the console, the changes in the images captured by each camera will be different, and therefore the required bandwidth may differ even if the same image quality is set. Therefore, when multiple cameras are installed on one vehicle, the image quality setting value may be determined separately for each camera.

[0024] Search for the maximum required bandwidth within the available bandwidth - Device operation The search for image quality setting values ​​that convert video images captured by the camera in the device of this embodiment into video images with the maximum required bandwidth within the available bandwidth may be performed repeatedly while the vehicle is operating using the process illustrated in Figure 3.

[0025] Referring to FIG. 3, first, the communication quality estimator detects the available bandwidth Ab (step 1), and the initial value of the image quality setting value is acquired (step 2). Here, the image quality setting value may be the image quality setting value of a moving image captured by a camera. Then, the required bandwidth determiner determines the required bandwidth Rb using the vehicle's driving state index value at that time and the initial value of the image quality setting value (step 3), and determines whether the determined required bandwidth Rb is greater than the available bandwidth Ab (step 4). If the required bandwidth Rb is greater than the available bandwidth Ab, the number of colors in the image quality setting value is reduced to reduce the required bandwidth Rb (step 5), and the required bandwidth determiner recalculates the required bandwidth Rb (step 6). If the required bandwidth Rb is still greater than the available bandwidth Ab (step 7), the number of pixels in the image quality setting value is reduced (step 8), and the required bandwidth determiner recalculates the required bandwidth Rb (step 9). If the required bandwidth Rb is still greater than the available bandwidth Ab (step 10), the FPS of the image quality setting is further reduced (step 11), and the required bandwidth Rb is again calculated by the required bandwidth determiner (step 12). If the required bandwidth Rb is still greater than the available bandwidth Ab (step 13), the processing from step 5 onwards may be repeated. On the other hand, if the required bandwidth Rb is less than the available bandwidth Ab in any of steps 4, 7, 10 or 13, the required bandwidth for the video of the image quality set by the current image quality setting will be the largest required bandwidth below the available bandwidth, and so the current image quality setting is determined as the image quality setting to be used in the video converter (step 14). Note that in the above, the number of colors, number of pixels and FPS may each be reduced by an appropriately set amount.

[0026] In the above-described processing configuration for searching for the required bandwidth, if the initial value of the image quality setting value acquired in step 2 is always set to the image quality setting value of the video captured by the camera, when the vehicle's surroundings change significantly, such as when the vehicle is traveling at high speed, and the required bandwidth tends to be larger than the available bandwidth, a long search process will be required each time to find an image quality setting value that reduces the image quality until the required bandwidth is equal to or less than the available bandwidth. On the other hand, since the vehicle's motion state may not change much over a short period of time, the image quality setting value in a certain processing cycle may not change much from the image quality setting value in the processing cycles before and after it. In such cases, the search process time can be shortened by adopting the image quality setting value of the previous processing cycle as the initial value of the image quality setting value acquired in step 2. Furthermore, if it is possible to search for an image quality setting value that increases the image quality compared to the previous value, it is possible to search for an image quality setting value that maximizes the required bandwidth within a range equal to or less than the available bandwidth and transmits video with higher image quality. Therefore, in this embodiment, the initial value of the image quality setting value obtained in step 2 is set to the previous value, and in step 4, if required bandwidth Rb<available bandwidth Ab, a process may be executed to search for an image quality setting value that improves image quality in order to obtain an image quality setting value that gives the largest possible required bandwidth Rb that is less than the available bandwidth Ab.

[0027] 3, if required bandwidth Rb<available bandwidth Ab, the following steps are executed in order: reducing the FPS of the image quality setting (step 15), calculating the required bandwidth Rb (step 16), determining whether required bandwidth Rb>available bandwidth Ab (step 17), increasing the number of pixels (step 18), calculating the required bandwidth Rb (step 19), determining whether required bandwidth Rb>available bandwidth Ab (step 20), increasing the number of colors (step 21), calculating the required bandwidth Rb (step 22), and determining whether required bandwidth Rb>available bandwidth Ab (step 23). If required bandwidth Rb>available bandwidth Ab holds true during this process, the previous setting may be determined as the image quality setting used in the video converter (steps 24, 14). Note that in the above, the number of colors, number of pixels, and FPS may each be increased by an appropriately set increase amount.

[0028] The above-described process of searching for image quality setting values ​​in FIG. 3 may be executed at an interval that is set appropriately between the camera's image capture frame interval and an interval that is shorter than the change in the vehicle state.

[0029] Thus, according to the above-described embodiment, by adjusting the image quality of the moving images transmitted to the control console according to the vehicle's motion state, the required bandwidth for the moving images is always equal to or less than the available bandwidth and is at its maximum, making it possible to transmit the moving images captured by the onboard camera from the vehicle to the control console with the highest possible quality and without interruption.

[0030] The above description has been made in relation to the embodiments of the present invention, but it will be apparent that many modifications and changes will be readily apparent to those skilled in the art, and the present invention is not limited to the above-described exemplary embodiments, but can be applied to various devices without departing from the concept of the present invention.

Claims

[Claim 1] A vehicle surroundings moving image transmission device in a vehicle remote control system, a required bandwidth determination means for outputting a required bandwidth for a converted moving image when the moving image captured by the on-board camera is converted into a moving image of image quality set by the image quality setting value, based on an input of an image quality setting value for the moving image captured by the on-board camera around the vehicle and transmitted from the vehicle to an operation console that remotely operates the vehicle via wireless communication and a motion state index value of the vehicle; an available bandwidth detection means for detecting an available bandwidth in wireless communication from the vehicle to the console; an image quality setting value search means for inputting various image quality setting values ​​to the necessary bandwidth determination means, causing the necessary bandwidth determination means to output the necessary bandwidth, and searching for an image quality setting value that makes the necessary bandwidth equal to or smaller than the available bandwidth and as large as possible; a moving image conversion means for converting the moving image captured by the vehicle-mounted camera into a moving image of image quality set by the image quality setting value searched by the image quality setting value search means; an image transmission means for transmitting the moving image obtained by the conversion by the moving image conversion means to the operation console; Including, The required bandwidth determination means is configured to use as input data the vehicle's motion state index value when a previously collected video of the vehicle's surroundings is obtained and an arbitrary image quality setting value, and to use a large amount of learning data in which the required bandwidth of the converted video when the previously collected video is converted into a video of image quality set at the arbitrary image quality setting value as correct answer data, and when the vehicle's motion state index value when a previously collected video of the vehicle's surroundings is obtained and one image quality setting value are input according to a machine learning algorithm, output the required bandwidth of the converted video when the video of the vehicle's surroundings is converted into a video of image quality set at the one image quality setting value.

Images