Automatic driving control device, remote assistance device and automatic driving control system

Abstract

The present invention realizes low-delay image transmission and low-delay remote control by optimally controlling the bit rate of each transmission image from an automatic driving control device to a remote control device according to a frequency band. The automatic driving control apparatus calculates an estimated bit rate corresponding to a frequency band available to transmit images to the remote control apparatus, calculates an image priority for each image captured by the plurality of cameras, and transmits only high-priority images determined to be transmittable at a low latency not greater than the estimated bit rate. The remote support apparatus switches an image to be displayed on the display unit to one of a camera captured image, a composite image of the camera captured image and a 3DCG image, and the 3DCG image, in accordance with an image received from the automatic driving control apparatus, and displays the one image.

Description

Automatic driving control device, remote assistance device and automatic driving control system Technical Field

[0001] This disclosure relates to an autonomous driving control device, a remote assistance device, and an autonomous driving control system. Specifically, this disclosure relates to an autonomous driving control device, a remote assistance device, and an autonomous driving control system in which an autonomous vehicle transmits images captured by a camera to an external driving assistance device, and an operator on the driving assistance device side examines the images to perform vehicle control. Background Technology

[0002] In recent years, the development of technologies related to autonomous driving has been rapid. This development includes so-called Automated Driving Systems (ADS) and Advanced Driver Assistance Systems (ADAS).

[0003] In autonomous driving, cameras and laser-based sensors installed in vehicles (cars) are used to detect obstacles and to use the detected information for safe driving. Sensors such as cameras detect objects around the vehicle, such as oncoming vehicles, pedestrians, or guardrails, and the vehicle's autonomous driving actuators detect safe driving routes to avoid collisions with such objects in order to perform autonomous driving.

[0004] SAE (Society of Automotive Engineers), the organization that establishes automotive technology standards, has defined the following levels as international standards for autonomous driving levels.

[0005] Level 0 = No driving automation

[0006] Level 1 = Driver Assistance

[0007] Level 2 = Partial Driving Automation

[0008] Level 3 = Conditional driving automation

[0009] Level 4 = High level of driving automation

[0010] Level 5 = Fully Automated Driving

[0011] Currently, autonomous vehicles are reaching Level 4, where almost no human intervention is required, and Level 5, where fully autonomous driving has been achieved.

[0012] However, even vehicles capable of Level 4 or 5 autonomous driving encounter complex situations that cannot be handled by the vehicle alone. Specifically, for example, in situations such as a collision caused by the faulty driving of another vehicle, an accident such as contact, or driving onto a sidewalk due to an accident, an autonomous vehicle may become unable to continue driving independently in some cases. In order to perform appropriate driving control in such situations, intervention from an external operator is required.

[0013] When external control is applied, the autonomous vehicle transmits images captured by cameras installed on the vehicle to a remote assistance center via a communication network. The operator at the remote assistance center examines the images displayed on the screen to determine the optimal control and sends the determined control information to the vehicle to execute the vehicle's driving control.

[0014] In situations requiring remote assistance from an external operator, wireless communication between the vehicle and the remote assistance center is crucial. Images captured by the vehicle's cameras are transmitted to the remote assistance center via wireless communication.

[0015] However, compared to wired communication, wireless communication is more prone to quality degradation. For example, it is susceptible to problems such as communication delays and packet loss.

[0016] For autonomous vehicles to operate safely, near real-time control is required. For example, when there is a delay in transmitting images captured by the camera, operator control is delayed; as a result, there is also the possibility of new accidents occurring.

[0017] Furthermore, if high-precision images cannot be transmitted due to deteriorated communication quality, the images displayed on the remote assistance center's screen will become unclear. In such cases, the operator's monitoring burden increases, operator fatigue accumulates, and operational errors are likely to occur.

[0018] As a technique to address the communication latency problem of image data, a configuration is conceivable in which CG images with less data than those captured by a camera are sent to perform remote collaboration.

[0019] For example, Patent Document 1 (JP-2007-323481-A) discloses a configuration in which images captured by a camera are converted into CG images and the CG images are sent.

[0020] However, if such CG conversion is performed on the vehicle side, processing time is required for the CG conversion; as a result, real-time data transmission becomes difficult, and delays in operator control that cannot be avoided arise.

[0021] Reference List

[0022] Patent documents

[0023] Patent Document 1: JP-2007-323481-A Summary of the Invention

[0024] Technical issues

[0025] This disclosure addresses, for example, the aforementioned problems, and its object is to provide an autonomous driving control device, a remote assistance device, and an autonomous driving control system that enables reliable remote collaboration of autonomous vehicles with short latency.

[0026] Solution to the problem

[0027] A first aspect of this disclosure provides an autonomous driving control device, the autonomous driving control device comprising:

[0028] The communication unit transmits images captured by multiple cameras mounted on the vehicle to the remote assistance device, and receives autonomous driving control information from the remote assistance device; and

[0029] The data processing unit executes the transmission control of each of the plurality of camera-captured images, wherein...

[0030] The data processing unit performs the following:

[0031] The processing involves calculating an estimated bit rate equivalent to the bandwidth that can be used to send images to the remote control device;

[0032] Processing to calculate the image priority of each of the multiple camera-captured images; and

[0033] The image selection process selects images captured by the plurality of cameras in descending order of image priority and sends the selected images that are evaluated to be able to be sent to the remote control device at the estimated bit rate or lower, with a delay equal to or less than a predetermined value.

[0034] Furthermore, a second aspect of this disclosure provides a remote assistance device, the remote assistance device having:

[0035] The communication unit receives images captured by multiple cameras from the autonomous driving control unit, which are captured by multiple cameras installed on the vehicle; and

[0036] The data processing unit performs display control to display the images captured by the plurality of cameras on the display unit, wherein,

[0037] The data processing unit switches the image to be displayed on the display unit to any one of the following based on the image received from the automatic driving control device:

[0038] (a) A camera-captured image received from the autonomous driving control device;

[0039] (b) A composite image of camera-captured images and 3DCG images received from the autonomous driving control device; or

[0040] (c) 3DCG image.

[0041] Furthermore, a third aspect of this disclosure provides an autonomous driving control system, which includes an autonomous driving control device and a remote assistance device, wherein,

[0042] The automatic driving control device performs:

[0043] The processing involves calculating an estimated bit rate equivalent to the bandwidth that can be used to send images to the remote control device;

[0044] The processing of calculating the image priority of each camera capture image among multiple camera capture images captured by multiple cameras mounted on a vehicle; and

[0045] The image selection process selects images captured by the plurality of cameras in descending order of image priority, and transmits the selected images that are evaluated as capable of being transmitted to the remote control device at the estimated bit rate or lower, with a delay equal to or less than a predetermined value.

[0046] The remote assistance device:

[0047] The process of switching the image to be displayed on the display unit based on the image received from the automatic driving control device is performed to any of the following:

[0048] (a) A camera-captured image received from the autonomous driving control device;

[0049] (b) A composite image of camera-captured images and 3DCG images received from the autonomous driving control device; or

[0050] (c) 3DCG images, and

[0051] The system sends control information generated based on the inspection results of the displayed image to the autonomous driving control device.

[0052] Other objects, features, and advantages of this disclosure will become apparent from the more detailed description based on the embodiments and accompanying drawings of the invention, which will be mentioned later. Note that a system in this specification refers to a logical set configuration of multiple devices. The system is not limited to those systems in which the various constituent devices are housed in a single housing.

[0053] According to the configuration of the embodiments of this disclosure, by optimally controlling the bit rate of each image sent from the automatic driving control device to the remote control device based on the bandwidth, image transmission with short latency and remote control with short latency are achieved.

[0054] Specifically, for example, the automatic driving control device calculates an estimated bit rate equivalent to the bandwidth available for sending images to the remote control device, calculates the image priority of each camera-captured image among multiple camera-captured images, and only sends high-priority images that are evaluated as being able to be sent with a short delay at the estimated bit rate or a lower bit rate. The remote assistance device switches the image to be displayed on the display unit based on the image received from the automatic driving control device to any of the following: a camera-captured image, a composite image of a camera-captured image and a 3DCG image, or a 3DCG image.

[0055] According to this configuration, by optimally controlling the bit rate of each image sent from the autonomous driving control unit to the remote control unit based on the bandwidth, image transmission with short latency and remote control with short latency are achieved.

[0056] Note that the beneficial effects described in this specification are illustrative only and not limiting, and additional beneficial effects may exist. Attached Figure Description

[0057] Figure 1 is a diagram illustrating an example of the automatic driving control system of this disclosure.

[0058] Figure 2 is a diagram illustrating the configuration examples of the autonomous driving control device for an autonomous vehicle and the remote assistance device for a remote assistance center.

[0059] Figure 3 is a diagram illustrating an example of sensor installation in an autonomous vehicle.

[0060] Figure 4 is a diagram illustrating an example of an image (view) captured using a camera in an autonomous vehicle.

[0061] Figure 5 is a diagram illustrating the configuration of the data processing unit of the autonomous driving control device in an autonomous vehicle and the processes to be performed.

[0062] Figure 6 is a diagram illustrating the configuration of the data processing unit of the remote assistance device in the remote assistance center and the processes to be performed.

[0063] Figure 7 is a diagram illustrating a specific example of a 3DCG image generated by the map control unit of the data processing unit of the automatic driving control device.

[0064] Figure 8 is a diagram illustrating the configuration for processing data to be sent from the automatic driving control unit to the remote assistance unit.

[0065] Figure 9 is a flowchart illustrating the sequence of data transmission control processing performed by the data processing unit of the automatic driving control device.

[0066] Figure 10 is a flowchart illustrating the sequence of data transmission control processing performed by the data processing unit of the automatic driving control device.

[0067] Figure 11 is a diagram illustrating the definitions of the various parameters used in the flowcharts shown in Figures 9 and 10.

[0068] Figure 12 illustrates the bit rate (estimated bit rate b) available for data transmission. e ) and the target bit rate b, which is the bit rate used for actual data transmission. target A graph showing the transition over time.

[0069] Figure 13 is a diagram illustrating a specific example of the process for determining the allocation of transmission bit rates for each captured image (video track) 1 to n.

[0070] Figure 14 is a flowchart illustrating the details of the priority-based sorting of camera-captured images and the priority-based selection of transmitted images performed by the data processing unit of the autonomous driving control device of an autonomous vehicle.

[0071] Figure 15 is a diagram illustrating the definition of the parameters used in the flowchart shown in Figure 14.

[0072] Figure 16 is a diagram illustrating a specific example of image sorting processing based on image priority.

[0073] Figure 17 is a specific example of a gaze heatmap used to illustrate the degree of concentration of the operator's gaze at the remote assistance center.

[0074] Figure 18 is a diagram illustrating an example of the analysis process used to explain (a) the number of surrounding objects and (b) the number of high-risk objects.

[0075] Figure 19 is a diagram illustrating the object risk level calculation processing technique using neural networks.

[0076] Figure 20 is a diagram illustrating a specific example of image selective transmission processing based on image priority.

[0077] Figure 21 is a diagram illustrating a specific example of image selective transmission processing based on image priority.

[0078] Figure 22 is a diagram illustrating a specific example of the processing in which 3D maps are used to generate and display 3DCG images observed from the position and orientation of cameras mounted on autonomous vehicles.

[0079] Figure 23 is a flowchart illustrating a sequence of communication control processes performed by an autonomous driving control unit to explain a large amount of metadata, such as images of objects.

[0080] Figure 24 is a diagram illustrating a specific example of image segmentation processing.

[0081] Figure 25 is a diagram illustrating an example of sorted data in which the extracted objects are ordered in descending order of risk level from 1 to 4.

[0082] Figure 26 is a diagram illustrating specific examples of different compression processes corresponding to the risk level of an object.

[0083] Figure 27 is a flowchart illustrating a specific example of compression and filtering processes (sending object selection processes) corresponding to the risk level of an object.

[0084] Figure 28 is a flowchart illustrating a specific example of compression and filtering processes (send object selection processes) corresponding to the risk level of an object.

[0085] Figure 29 is a diagram illustrating the definitions of the various parameters used in the flowcharts shown in Figures 27 and 28.

[0086] Figure 30 is an example illustrating how a remote assistance device uses data sent by the autonomous driving control device of an autonomous vehicle to display an image on a display unit.

[0087] Figure 31 is a diagram illustrating the 3DCG image opacity, which is an adjustment parameter for the synthesis ratio of the 3DCG image and the received image.

[0088] Figure 32 is a diagram illustrating an example of composite processing of a 3DCG image and a received image corresponding to a setting value for the opacity of the 3DCG image.

[0089] Figure 33 is a diagram illustrating an example of composite processing of a 3DCG image and a received image corresponding to a setting value for the opacity of the 3DCG image.

[0090] Figure 34 is a flowchart illustrating the 3DCG image opacity control sequence executed by the data processing unit of the remote assistance device.

[0091] Figure 35 is a flowchart illustrating the 3DCG image opacity control sequence executed by the data processing unit of the remote assistance device.

[0092] Figure 36 is a diagram illustrating the definitions of the parameters described in the flowcharts shown in Figures 34 and 35.

[0093] Figure 37 is a diagram illustrating the object display processing performed by the display control unit of the data processing unit of the remote assistance device, which corresponds to the risk level of the object.

[0094] Figure 38 is a diagram illustrating a specific example of object display processing performed by the display control unit of the data processing unit of the remote assistance device, which corresponds to the risk level of the object.

[0095] Figure 39 is a diagram illustrating an example of approximation processing performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40.

[0096] Figure 40 is a diagram illustrating an example of interpolation processing performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40.

[0097] Figure 41 is a diagram illustrating the configuration for processing data to be sent from an automatic driving control unit to a remote assistance unit.

[0098] Figure 42 is a diagram illustrating a configuration example of an autonomous driving control unit of an autonomous vehicle sending camera-captured images to multiple remote assistance devices.

[0099] Figure 43 is a diagram illustrating a configuration example in which multiple autonomous vehicles a to c send camera-captured images, captured by cameras mounted on each vehicle, to the same single remote assistance device 4.

[0100] Figure 44 is a diagram illustrating an example of the hardware configuration of the automatic driving control device and remote assistance device of this disclosure. Detailed Implementation

[0101] The details of the automatic driving control device, remote assistance device, and automatic driving control system of this disclosure are illustrated below with reference to the accompanying drawings. Note that the description is provided with reference to the following items.

[0102] 1. Overview of the automatic driving control device, remote assistance device, and automatic driving control system disclosed herein

[0103] 2. Configuration of automatic driving control device and remote assistance device

[0104] 3. Regarding the processing of data transmission from the automatic driving control unit to the remote assistance unit.

[0105] 4. Regarding the data transmission control processing performed by the automatic driving control unit

[0106] 5. Details regarding the processing of selectively sending camera-captured images based on priority.

[0107] 6. Details regarding the processing of sending objects detected in images that were never sent.

[0108] 7. Regarding processing performed by remote assistance devices

[0109] 8. Details regarding the processing of composite (overlay) camera-captured images and 3DCG images performed by the remote assistance device.

[0110] 9. Details regarding object display control processing performed by the remote assistance device.

[0111] 10. Regarding other embodiments

[0112] 11. Hardware configuration examples for automatic driving control devices and remote assistance devices

[0113] 12. Summary of the configuration of this disclosure

[0114] [1. Overview of the automatic driving control device, remote assistance device, and automatic driving control system disclosed herein]

[0115] First, an overview of the automatic driving control device, remote assistance device, and automatic driving control system disclosed herein will be provided.

[0116] Figure 1 is a diagram illustrating an example of the automatic driving control system of this disclosure.

[0117] The autonomous vehicle 10 is a vehicle that operates autonomously. The autonomous driving control device 20 is installed on the autonomous vehicle 10.

[0118] The autonomous driving control unit 20 performs autonomous driving using information obtained by detections made by sensors such as cameras and LiDAR (laser imaging detection and ranging) installed on the autonomous driving vehicle 10.

[0119] Note that LiDAR is an object detection sensor that uses lasers.

[0120] However, the autonomous vehicle 10 encounters complex situations that it cannot handle alone. For example, in cases of accidents caused by the erroneous driving of another vehicle or driving onto a sidewalk, and further in cases of driving through road construction zones, landslides from the shoulder, unexpected obstacles, or other accidents, the autonomous vehicle becomes unable to continue driving independently. In such situations, the autonomous driving control unit 20 performs autonomous driving by receiving control information from an external operator.

[0121] The autonomous driving control device 20 communicates with the external remote assistance center 30 via a wireless communication base station arranged along the road, and sends camera-captured images, LiDAR detection information, etc. to the remote assistance center 30.

[0122] In the remote assistance center 30, there is an operator 50 who provides driving assistance to the autonomous vehicle 10, and a remote assistance device 40 including a display unit is arranged in the remote assistance center 30.

[0123] The remote assistance device 40 enables the display unit to show camera-captured images sent from the autonomous driving control device 20 of the autonomous vehicle 10.

[0124] The operator 50 analyzes the image displayed on the display of the remote assistance device 40, determines the optimal driving mode for the autonomous vehicle 10, and sends autonomous driving control information to the autonomous driving control device 20 of the autonomous vehicle 10 to enable the autonomous vehicle 10 to drive according to the determined driving mode.

[0125] The autonomous driving control device 20 of the autonomous vehicle 10 receives autonomous driving control information from the remote assistance device 40 and drives according to the received information.

[0126] In this way, the autonomous driving control unit 20 of the autonomous vehicle 10 performs autonomous driving control using information obtained by detections made by sensors such as cameras and LiDAR installed on the autonomous vehicle 10, and also uses autonomous driving control information received from the remote assistance device 40 to drive.

[0127] Communication is performed between the autonomous driving control device 20 of the autonomous vehicle 10 and the remote assistance device 40 of the remote assistance center 30 via wireless communication.

[0128] However, as mentioned earlier, wireless communication is more difficult to maintain communication quality compared to wired communication. Specifically, for example, it is prone to problems such as communication delays and packet loss.

[0129] For autonomous vehicles to operate safely, near real-time control is required. However, if operator control is delayed due to delays in the transmission of images or other data, there is a possibility of accidents.

[0130] In the system disclosed herein, the autonomous driving control device 20 of the autonomous vehicle 10 monitors the wireless communication network, calculates information about the bandwidth available for communication, and performs various data transmission control processes, such as data transmission rate adjustment processing, data transmission selection processing, or data transmission compression processing, based on the calculated available bandwidth.

[0131] The remote assistance device 40 of the remote assistance center 30 receives data transmitted from the autonomous driving control device 20 of the autonomous vehicle 10 and displays the received data, such as camera-captured images, on the display unit. However, a scenario is also envisioned where some camera-captured images are not transmitted. In such a case, the remote assistance device 40 performs processing such as data interpolation and data restoration using a pre-reserved 3D map CG image to generate a more complete image of the vehicle's surroundings and displays that image on the display unit.

[0132] Furthermore, in cases where the autonomous driving control unit 20 of the autonomous vehicle 10 cannot receive camera-captured images or the camera-captured images are too unclear, or in other cases, the remote assistance device 40 performs the following processing: using the information about objects such as oncoming vehicles included in the metadata received from the autonomous driving control unit 20, it overlays images of objects such as vehicles around the autonomous vehicle 10 onto the 3DCG image.

[0133] The remote assistance device 40 performs such processing to generate display data that more reliably reproduces the current surroundings of the autonomous vehicle 10, and causes the display unit to display the display data.

[0134] The operator 50 at the remote assistance center 30 examines the camera-captured images, 3DCG images, or composite images of these images displayed on the display of the remote assistance device 40, analyzes the situation around the autonomous vehicle 10, determines the optimal driving control that should be performed by the autonomous vehicle 10, generates autonomous driving control information based on the determined information, and sends the determined information to the autonomous vehicle 10.

[0135] By performing such processing, even in the event of communication delays or packet loss in wireless communication, the system of this disclosure allows the display unit of the remote assistance device 40 to display a highly accurate image of the surroundings of the autonomous vehicle 10.

[0136] As a result, the operator 50 can reliably send the correct autonomous driving control information to the autonomous vehicle 10 with a short delay.

[0137] [2. Regarding the configuration of the automatic driving control device and the remote assistance device]

[0138] Next, the configuration of the automatic driving control device and the remote assistance device will be explained.

[0139] Figure 2 is a diagram showing an example configuration of the autonomous driving control device 20 of the autonomous vehicle 10 and an example configuration of the remote assistance device 40 of the remote assistance center 30.

[0140] As shown in Figure 2, the autonomous driving control device 20 of the autonomous vehicle 10 includes sensors (cameras, LiDAR, etc.) 21, a data processing unit 22, an autonomous driving control unit (ADS (Autonomous Driving System)) 23, and a communication unit 24.

[0141] The remote assistance device 40 of the remote assistance center 30 includes a communication unit 41, a data processing unit 42, a display unit 43, an input unit 44, and a 3D map storage unit 45.

[0142] First, the configuration of the autonomous driving control device 20 of the autonomous vehicle 10 will be explained.

[0143] Sensors (cameras, LiDAR, etc.) 21 include sensors such as cameras and LiDARs installed on the autonomous vehicle 10. An example of sensor installation is illustrated with reference to Figure 3.

[0144] In the example shown in Figure 3, the camera includes the following six cameras.

[0145] Forward-facing camera 21FC

[0146] Right-view camera 21RC

[0147] Right rearview camera 21RBC

[0148] Rearview camera 21BC

[0149] Left rearview camera 21LBC

[0150] Left-view camera 21LC

[0151] These six cameras continuously capture images (moving images) in all directions and input the captured images into the data processing unit 22.

[0152] In addition, LiDAR (LiDAR) as an object sensor using lasers includes two LiDARs: forward LiDAR21FL and backward LiDAR 21BL.

[0153] These two LiDARs acquire object detection information in various directions (forward and backward) and input the acquired information into the data processing unit 22.

[0154] Note that the sensor installation example shown in Figure 3 is an example, and different sensor installation configurations can be used, such as a configuration with more cameras, a configuration with sensors other than LiDAR, etc.

[0155] Figure 4 shows an example of images (views) captured using the six cameras shown in Figure 3. Figure 4 shows the following images (views) captured side by side using the six individual cameras while the autonomous vehicle 10 is driving on the road.

[0156] (LBv) Left rear view (image captured by the left rear view camera)

[0157] (Bv) Rear view (image captured by the rear-view camera)

[0158] (RBv) Right rear view (image captured by the right rear-view camera)

[0159] (Lv) Left side view (image captured by the left-side camera)

[0160] (Fv) Front view (image captured by the front-view camera)

[0161] (Rv) Right side view (image captured by the right-view camera)

[0162] These images are input to the data processing unit 22 of the automatic driving control device 20 shown in FIG2, and are used for automatic driving control executed by the automatic driving control unit (ADS) 23. In addition, these images are also wirelessly transmitted to the remote assistance device 40 of the remote assistance center 30 via the communication unit 24.

[0163] It should be noted that in some cases, due to the state of the wireless communication network (e.g., due to the bandwidth available for wireless communication), it is difficult to transmit the data of all six images with a short delay (equal to or less than a predetermined delay). The data processing unit 22 of the automatic driving control device 20 of this disclosure monitors the wireless communication network, calculates information about the bandwidth available for communication, and performs various data transmission control processes based on the calculated available bandwidth, such as transmission bit rate adjustment processing for each camera-captured image in the multiple camera-captured images, transmission data selection processing, and transmission data compression processing, etc.

[0164] A specific example of this process will be mentioned later.

[0165] Returning to Figure 2, let's continue to explain the configuration of the automatic driving control device 20.

[0166] The data processing unit 22 receives sensor detection information from the sensor 21, such as from the camera and LiDAR, and inputs the input sensor detection information (such as images captured by the camera) to the automatic driving control unit (ADS) 23.

[0167] Based on the sensor detection information input from the data processing unit 22, the automatic driving control unit (ADS) 23 determines the safe driving route, driving speed, etc. of the automatic driving vehicle 10, and performs automatic driving.

[0168] Note that the Automated Driving Control Unit (ADS) 23 also uses automated driving control information input from the driving assistance device 40 via the communication unit 24 to perform driving control.

[0169] Next, the configuration of the remote assistance device 40 of the remote assistance center 30 will be described. As shown in FIG2, the remote assistance device 40 includes a communication unit 41, a data processing unit 42, a display unit 43, an input unit 44, and a 3D map storage unit 45.

[0170] The communication unit 41 communicates with the automatic driving control device 20 of the automatic driving vehicle 10.

[0171] The communication unit 41 receives images captured by the camera from the autopilot control unit 20. In addition, the communication unit 41 also receives metadata, including information about objects detected by the camera, LiDAR, etc.

[0172] The received information is input into the data processing unit 42.

[0173] The data processing unit 42 analyzes the camera-captured images and metadata received from the automatic driving control device 20, and generates image data to be output to (displayed on) the display unit 43.

[0174] For example, if the camera-captured image received from the automatic driving control unit 20 is a complete image in six directions as illustrated with reference to FIG4, these six images are output to the display unit 43.

[0175] Operator 50 observes the image displayed on display unit 43, checks the current surroundings of autonomous vehicle 10, inputs optimal control information (autonomous driving control information) via input unit 44, and sends the optimal control information to autonomous vehicle 10.

[0176] The autonomous driving control unit 20 of the autonomous vehicle 10 receives autonomous driving control information sent by the remote assistance device 40 of the remote assistance center 30 via the communication unit 24, and inputs the autonomous driving control information into the autonomous driving control unit (ADS) 23. The autonomous driving control unit (ADS) 23 executes autonomous driving based on the input autonomous driving control information. That is, it performs driving corresponding to the driving control determined by the operator 50.

[0177] However, as mentioned earlier, in some cases, depending on the state of the communication network, it is not possible to receive complete camera capture images from six directions from the autonomous driving control unit 20.

[0178] In this case, the data processing unit 42 of the remote assistance device 40 outputs the 3DCG image acquired from the 3D map storage unit 45 to the display unit 43. Alternatively, a composite image is generated from the unclear camera capture image and the 3DCG image received from the automatic driving control device 20, and then output to the display unit 43.

[0179] The data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 shown in FIG2 to generate and display 3DCG images observed from the position and orientation of the camera installed on the autonomous vehicle 10.

[0180] Note that although the 3D map storage unit 45 is included in the remote assistance device 40 in the example shown in Figure 2, the 3D map can be obtained from an external server.

[0181] Note that the 3DCG image generated from the 3D map stored in the 3D map storage unit 45 does not include real-time surrounding objects such as oncoming vehicles and pedestrians that exist around the autonomous vehicle 10.

[0182] Therefore, the data processing unit 42 performs processing in which it analyzes metadata received from the autonomous driving control unit 20, including information about surrounding objects (such as oncoming vehicles), and displays images of objects around the autonomous driving vehicle 10 on a 3DCG image, etc.

[0183] By performing such processing, display data that more reliably reproduces the current surroundings of the autonomous vehicle 10 is generated and displayed on the display unit.

[0184] The operator 50 at the remote assistance center 30 checks the camera-captured images, 3DCG images, or composite images of these images displayed on the display unit 43 of the remote assistance device 40, as well as images of objects around the autonomous vehicle 10.

[0185] Operator 50 views the image displayed on display unit 43, analyzes the situation around autonomous vehicle 10, determines the best driving control that should be performed by autonomous vehicle 10, generates autonomous driving control information based on the determined information, and sends the autonomous driving control information to autonomous vehicle 10.

[0186] The automated driving control unit (ADS) 23 of the automated driving control device 20 of the automated driving vehicle 10 executes vehicle driving control based on the automated driving control information received from the remote assistance device 40. That is, the automated driving vehicle 10 performs driving corresponding to the control mode determined by the operator 50.

[0187] Next, referring to FIG5, the configuration of the data processing unit 22 of the autonomous driving control device 20 of the autonomous vehicle 10 and the details of the processing to be performed by the data processing unit 22 will be described.

[0188] Figure 5 is a diagram showing the detailed configuration of the data processing unit 22 of the automatic driving control device 20.

[0189] As shown in Figure 5, the data processing unit 22 of the automatic driving control device 20 has the following constituent elements.

[0190] Risk Classification Department (Risk Classification Manager) 111

[0191] Communication Network Monitoring Department (Network Monitoring Manager) 112

[0192] Data Analysis Department (Data Manager) 113

[0193] Communication Control Department (Data Transmission Manager) 114

[0194] The data processing unit 22 of the automatic driving control device 20 has these constituent elements.

[0195] For example, the risk level classification unit 111 determines the priority of the images captured by the six cameras illustrated in Figure 3, i.e., the priority of the images sent to the remote assistance center. Furthermore, the risk level classification unit 111 analyzes the risk level of various surrounding objects (such as vehicles, pedestrians, and buildings) captured in the images. Specifically, the risk level classification unit 111 performs risk assessments such as the collision risk of each object relative to the autonomous vehicle 10.

[0196] As shown in Figure 5, the risk level classification section 111 has the following modules.

[0197] Image segmentation module (Image Segmentation Module) 111a

[0198] Object Extraction Module (Object Extraction Module) 111b

[0199] Sensor detection data analysis module (raw sensor data processing module) 111c

[0200] Image Priority Analysis Module (Video Priority Classification Module) 111d

[0201] Extracting Object Risk Level Analysis Module (Extracting Object Risk Level Classification Module) 111e

[0202] Risk Classification Section 111 contains these modules.

[0203] The image segmentation module 111a segments the image (i.e., the image captured by the camera) into regions.

[0204] The object extraction module 111b extracts objects from each region obtained through region segmentation, such as various objects like vehicles, bicycles, pedestrians and other structures.

[0205] The sensor detection data analysis module 111c uses sensor detection data such as LiDAR to analyze the distance, direction of movement, and speed of objects extracted by the object extraction module 111b.

[0206] Specifically, the sensor detection data analysis module 111c is a module that analyzes the data (raw data) detected by various sensors (e.g., cameras, LiDAR, other radars and sonars) installed on the autonomous vehicle 10 when these sensors are installed.

[0207] In addition to the image frames captured by the camera, the output of the sensor detection data analysis module 111c also includes data such as the coordinates of the detected object relative to the position of the autonomous vehicle 10, which serves as the origin, and the 3D bounding box, which serves as the three-dimensional bounding box around the object.

[0208] The output of the sensor detection data analysis module 111c is used as input data for other modules. For example, each image frame captured by the camera is input to the image segmentation module 111a, and image region segmentation processing is performed on it.

[0209] In addition, the coordinates and 3D bounding boxes of the detected objects are input into the object risk level analysis module 111e, which will be mentioned later, and are used for object risk level determination processing at the object risk level analysis module 111e.

[0210] The image priority analysis module 111d performs processes such as determining the image priority (i.e., the priority of images sent to the remote assistance center) for each camera (e.g., the six cameras illustrated with reference to Figure 3 or Figure 4). For example, this priority assessment is performed based on factors such as the number of objects captured in each image and the risk of each object.

[0211] The specific image priority evaluation process will be discussed later.

[0212] The object extraction risk level analysis module 111e analyzes the risk level of objects extracted from each segmented region in each image by the object extraction module 111b, and determines the risk level corresponding to each object. Specifically, the object extraction risk level analysis module 111e assesses hazards such as the risk of collision relative to the autonomous vehicle 10, and determines the risk level corresponding to the assessed risk.

[0213] Note that the risk level of each detected object determined by the extracted object risk level analysis module 111e is sent as metadata to the remote assistance device 40 on the remote assistance center 30 side.

[0214] The remote assistance device 40 performs a process that determines whether to output the object CG image to the display unit 43 based on the risk level of each detected object received as metadata.

[0215] Details and specific examples of risk level analysis and control processing, as well as the corresponding object display control processing, will be discussed later.

[0216] The Risk Classification Department 111 uses these modules to perform processes such as determining the priority of images sent to the remote assistance center and analyzing the risk levels of various surrounding objects captured in the images, such as vehicles, pedestrians, and buildings. Specific examples and details of these processes will be mentioned later.

[0217] The communication network monitoring unit 112 monitors the wireless communication network between the autonomous driving control device 20 of the autonomous vehicle 10 and the remote assistance device 40 of the remote assistance center 30. Specifically, the communication network monitoring unit 112 performs processing such as estimating the bandwidth (bit rate) available for communication, jitter estimation processing, and estimating the frame rate of each transmitted image.

[0218] As shown in Figure 5, the communication network monitoring unit 112 has the following modules.

[0219] Bit rate estimation module (bit rate estimation module) 112a

[0220] Frame rate estimation module (Frame Rate Estimation Module) 112b

[0221] Jitter estimation module (Jitter estimation module) 112c

[0222] The communication network monitoring unit 112 has these modules.

[0223] The communication network monitoring unit 112 uses these modules to perform the above-mentioned processing.

[0224] The data analysis unit 113 performs the process of deciding the allocation of the transmission bit rate for each of a plurality of images to be sent from the autonomous driving control device 20 of the autonomous vehicle 10 to the remote assistance device 40 of the remote assistance center 30. Specifically, for example, when six images corresponding to the six cameras illustrated earlier with reference to FIG3 or FIG4 are to be sent to the remote assistance device 40, the data analysis unit 113 determines the transmission bit rate for each of these images.

[0225] If, based on the network monitoring results of the communication network monitoring unit 112, it is determined that sufficient communication bandwidth cannot be guaranteed, the data analysis unit 113 further performs processing such as selecting images to be sent from the automatic driving control device 20 to the remote assistance device 40, compressing object image data such as vehicles captured in the images, and selecting object data to be considered as the sending target.

[0226] As shown in Figure 5, the data analysis unit 113 has the following modules.

[0227] Bit rate allocation module (bit rate allocation module) 113a

[0228] Target bit rate control module (Target bit rate control module) 113b

[0229] Image analysis module (video usage control module) 113c

[0230] Object data compression module (Object data compression module) 113d

[0231] Object data filtering module (Object data filtering module) 113e

[0232] The data analysis department 113 has these modules.

[0233] The data analysis department 113 uses these modules to perform the above processing.

[0234] The communication control unit 114 performs control over the wireless communication between the autonomous driving control device 20 of the autonomous vehicle 10 and the remote assistance device 40 of the remote assistance center 30. Specifically, the communication control unit 114 performs various communication control processes required for performing wireless communication, such as the selection and control of the frequency used and the allocation of communication resources.

[0235] As shown in Figure 5, the communication control unit 114 has the following modules.

[0236] Frequency Selection Module (Frequency Selection Module) 114a

[0237] Communication data frequency control module (data transmission frequency control module) 114b

[0238] Communication resource allocation module (radio resource allocation module) 114c

[0239] The communication control unit 114 has these modules.

[0240] The communication control unit 114 uses these modules to perform the above-mentioned processing.

[0241] In this way, the data processing unit 22 of the automatic driving control device 20 has the following components: risk level classification unit 111, communication network monitoring unit 112, data analysis unit 113 and communication control unit 114, and uses these components to perform the processing of data to be sent to the remote assistance device 40 to the remote assistance center 30.

[0242] That is, based on the priority order of the transmitted data and the network status, the data processing unit 22 performs control over allocating transmission bandwidth (bit rate) to the video track (which is the transmission track that utilizes the images captured by each camera) and enabling / disabling the video track, and also achieves stable image transmission even when the available bandwidth is narrow.

[0243] Furthermore, by pre-setting a limited degree of fluctuation in the bit rate to be used for image transmission, the data processing unit 22 achieves stable image transmission even in the event of sudden fluctuations in available bandwidth. Additionally, in cases where available bandwidth is narrow and untransmitted images are set to not be transmitted (disabled video tracks), the data processing unit 22 performs processes such as generating and transmitting images of objects detected in the untransmitted images, such as oncoming vehicles, and generating and transmitting compressed object images.

[0244] Details and specific examples of these processes will be discussed later.

[0245] Next, referring to FIG6, the configuration of the data processing unit 42 of the remote assistance device 40 of the remote assistance center 30 and the details of the processing to be performed by the data processing unit 42 will be described.

[0246] Figure 6 is a diagram showing the detailed configuration of the data processing unit 42 of the remote assistance device 40.

[0247] As shown in FIG6, the data processing unit 42 of the remote assistance device 40 has the following constituent elements.

[0248] Map Control Department (Map Manager) 121

[0249] Display Control Unit (Visual Manager) 122

[0250] Communication Control Department (Data Transmission Manager) 123

[0251] In cases where the camera-captured images included in the data transmitted by the autonomous driving control unit 20 of the autonomous vehicle 10 are insufficient, such as when a camera-captured image in a specific direction is not transmitted, when a low-precision image is received due to a low image transmission bit rate, or in other cases, the map control unit 121 performs processing such as acquiring a 3D map stored in the 3D map storage unit 42 and generating a 3DCG image that matches the current camera position and orientation of the autonomous vehicle 10.

[0252] Figure 7 shows a specific example of a 3DCG image generated by the map control unit 121.

[0253] Figure 7 shows examples of 3DCG images corresponding to the six images illustrated earlier with reference to Figure 4.

[0254] That is, Figure 7 shows an example of 3DCG images observed from the positions and orientations of the six cameras shown in Figure 3, which have captured images (views) using cameras. Figure 7 shows the following six 3DCG images side by side.

[0255] (LB-3DCG) Left rear view 3DCG image (CG corresponding to the image captured by the left rear view camera)

[0256] (B-3DCG) Rear view 3DCG image (CG corresponding to the image captured by the rear-view camera)

[0257] (RB-3DCG) Right rear view 3DCG image (CG corresponding to the image captured by the right rear view camera)

[0258] (L-3DCG) Left side view 3DCG image (CG corresponding to the image captured by the left-view camera)

[0259] (F-3DCG) Front view 3DCG image (CG corresponding to the image captured by the front-view camera)

[0260] (R-3DCG) Right-side view 3DCG image (CG corresponding to the image captured by the right-side camera)

[0261] In this way, the map control unit 121 uses the 3D map stored in the 3D map storage unit 42 to perform processing such as generating a 3DCG image that matches the current camera position and orientation of the autonomous vehicle 10.

[0262] As shown in Figure 6, the map control unit 121 has the following modules.

[0263] 3D Map Update Module (3D Map Update Module) 121a

[0264] 3DCG Image Acquisition Module (3DCG View Capture Module) 121b

[0265] The map control unit 121 has these various modules.

[0266] The map control unit 121 uses these modules to perform the above-mentioned processing.

[0267] Note that details and specific examples of the processing to be performed by the map control unit 121 will be mentioned later.

[0268] The display control unit 122 performs control over the image to be displayed on the display unit 43. That is, the display control unit 122 performs processing to control the display of the camera-captured image received from the autonomous driving control device 20 of the autonomous driving vehicle 10.

[0269] In addition, if the camera-captured image received from the autonomous driving control unit 20 is insufficient image data, the display control unit 122 performs processing to synthesize the camera-captured image received from the autonomous driving control unit 20 and the 3DCG image generated by the map control unit 121, such as data interpolation or data restoration, and outputs the resulting image to the display unit 43.

[0270] In addition, the display control unit 122 performs processing such as displaying images of objects around the autonomous vehicle 10 on a 3DCG image using metadata received from the autonomous driving control unit 20, including information about surrounding objects (such as oncoming vehicles).

[0271] As shown in Figure 6, the display control unit 122 has the following modules.

[0272] Image overlay control module (view overlay control module) 122a

[0273] Data interpolation / extrapolation processing module (data extrapolation / interpolation module) 122b

[0274] Object Image Visualization Module (Object Visualization Module) 112c

[0275] Image Opacity Control Module (View Opacity Control Module) 121d

[0276] The display control unit 122 has these modules.

[0277] The display control unit 122 uses these modules to perform the above-mentioned processing.

[0278] Note that details and specific examples of the processing to be performed by the display control unit 122 will be mentioned later.

[0279] The communication control unit 123 performs control over the wireless communication between the autonomous driving control device 20 of the autonomous vehicle 10 and the remote assistance device 40 of the remote assistance center 30. Specifically, the communication control unit 123 performs various communication control processes required for performing wireless communication, such as the selection and control of the frequency used and the allocation of communication resources.

[0280] As shown in Figure 6, the communication control section 123 has the following modules.

[0281] Frequency Selection Module (Frequency Selection Module) 123a

[0282] Communication data frequency control module (data transmission frequency control module) 123b

[0283] Communication resource allocation module (radio resource allocation module) 123c

[0284] The communication control unit 123 has these modules.

[0285] The communication control unit 123 uses these modules to perform the above-mentioned processing.

[0286] In this way, the data processing unit 42 of the remote assistance device 40 has the following components: map control unit 121, display control unit 122 and communication control unit 123, and uses these components to perform various processing on the data received from the autonomous driving control unit 20 of the autonomous vehicle 10.

[0287] That is, the data processing unit 42 of the remote assistance device 40 controls the data displayed on the display unit 43 by the display control unit 122. Specifically, as described above, the data processing unit 42 generates a 3DCG image from the camera viewpoint of the autonomous vehicle 10, and overlays the 3DCG image onto the image captured by the camera as needed. In addition, the data processing unit 42 performs processes such as overlaying images of objects such as oncoming vehicles onto the 3DCG image.

[0288] Details and specific examples of these processes will be discussed later.

[0289] [3. Regarding the processing of data transmission from the automatic driving control unit to the remote assistance unit]

[0290] Next, we will explain the process of sending data from the autonomous driving control unit to the remote assistance unit.

[0291] Figure 8 is a diagram illustrating the configuration for processing data to be sent from the automatic driving control unit to the remote assistance unit.

[0292] The left side of Figure 8 shows the autonomous driving control device 20 of the autonomous vehicle 10, and the right side of Figure 8 shows the remote assistance device 40 of the remote assistance center 30.

[0293] The autonomous driving control unit 20 of the autonomous vehicle 10 inputs the sensor acquisition values ​​of the sensor 21 (including the camera and LiDAR) to the data processing unit 22, specifically the images captured by the camera, object detection information detected by the LiDAR, etc.

[0294] The data processing unit 22 includes a risk level classification unit 111, a communication network monitoring unit 112, a data analysis unit 113, and a communication control unit 114, as described earlier with reference to Figure 5.

[0295] The data processing unit 22 uses these components to analyze images captured by the camera and object detection information detected by LiDAR. Additionally, the data processing unit 22 estimates the bandwidth (bit rate) available for data transmission based on monitoring information about the wireless communication network between the autonomous driving control device 20 and the remote assistance device 40 of the remote assistance center 30.

[0296] Furthermore, the data processing unit 22 performs bit rate control on the transmitted data based on the estimated available bandwidth (bit rate). Specifically, the data processing unit 22 performs processing such as determining the transmission bit rate of each of the six images corresponding to the six cameras described earlier with reference to Figures 3 and 4, and transmits each image to the remote assistance device 40 via the communication unit 24 at the determined bit rate corresponding to the image.

[0297] Figure 8 illustrates multiple video tracks 1 to n and metadata transmission paths in the communication path between the autonomous driving control device 20 and the remote assistance device 40 of the remote assistance center 30.

[0298] For example, the multiple video tracks 1 to n shown in the communication path are the transmission video tracks for six individual images corresponding to the six cameras illustrated earlier with reference to Figures 3 and 4. In the case of transmitting six images, n = 6.

[0299] Multiple video tracks 1 to n are transmitted using a single stream established between the autopilot control unit 20 and the remote assistance unit 40 of the remote assistance center 30. That is, six images corresponding to six cameras are transmitted via a multi-track video stream, in which multiple videos are transmitted using a single stream. The bitrate of these six images can be changed at any time according to the available bandwidth. For example, allocating more communication bandwidth (bitrate) to high-priority images can be performed based on image priority.

[0300] Additionally, in situations where the available communication bandwidth for data transmission is limited, filtering is sometimes performed to selectively send only high-priority images and not all six camera-captured images. These processes will be explained later.

[0301] In addition to the video track transmission path, Figure 8 also shows the metadata transmission path. The metadata includes information about objects detected from the camera-captured images and LiDAR sensor information, such as surrounding vehicles, pedestrians, and buildings.

[0302] Specifically, the metadata includes the coordinates of the object relative to the subject vehicle as the origin, the size of the bounding box of the three-dimensional box surrounding the object, a label indicating the object type (vehicle, pedestrian, etc.), and an image of the object.

[0303] The data processing unit 22 generates this metadata using point cloud information such as camera-captured images or LiDAR detection values. Note that in configurations that include other sensors such as radar, metadata is generated that also uses detection values ​​from these sensors.

[0304] Note that, regarding metadata, the autonomous driving control device 20 also performs communication control to change the metadata to be sent at any time according to the communication bandwidth (bit rate) available for metadata communication between the autonomous driving control device 20 and the remote assistance device 40.

[0305] Metadata includes data that must be sent regardless of fluctuations in available communication bandwidth, and data that should be stopped when available bandwidth is limited.

[0306] For example, metadata such as object coordinates, bounding box dimensions, and labels used to identify the location and size of objects (such as vehicles and pedestrians around the autonomous vehicle 10) is always sent.

[0307] In contrast, for large amounts of data (such as images of objects), if the bandwidth of the communication path that can be guaranteed for metadata communication is narrow, the transmission of large amounts of data can be stopped, the large amounts of data can be transmitted as compressed data, or filtering can be performed to selectively transmit data to the target based on the risk level of each object.

[0308] Details of these processes will be discussed later.

[0309] As shown in Figure 8, the camera-captured images (video tracks) and metadata sent from the automatic driving control device 20 are received by the communication unit 41 of the remote assistance device 40 of the remote assistance center 30 and input into the data processing unit 42.

[0310] The data processing unit 42 analyzes the camera-captured images and metadata received from the automatic driving control device 20, and generates image data to be output to (displayed on) the display unit 43.

[0311] For example, if the camera-captured image received from the automatic driving control unit 20 is a complete image in six directions as illustrated with reference to FIG4, these six images are output to the display unit 43.

[0312] However, as mentioned earlier, in some cases, depending on the state of the communication network, it is not possible to receive complete camera capture images from six directions from the autonomous driving control unit 20.

[0313] In this case, the data processing unit 42 of the remote assistance device 40 uses the 3DCG image obtained from the 3D map storage unit 45 to interpolate the insufficient area in the camera-captured image, and outputs the resulting image to the display unit 43.

[0314] The data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 shown in FIG2 to generate and display 3DCG images observed from the position and orientation of the camera installed on the autonomous vehicle 10.

[0315] In addition, the data processing unit 42 performs processing in which it analyzes metadata received from the autonomous driving control unit 20, including information about surrounding objects (such as oncoming vehicles), and displays images of objects around the autonomous driving vehicle 10 on a 3DCG image.

[0316] By performing such processing, display data that more reliably reproduces the current surroundings of the autonomous vehicle 10 is generated and displayed on the display unit.

[0317] Operator 50 observes the image displayed on display unit 43, checks the current surroundings of autonomous vehicle 10, inputs optimal control information (autonomous driving control information) via input unit 44, and sends the optimal control information to autonomous vehicle 10.

[0318] [4. Regarding the data transmission control processing performed by the automatic driving control unit]

[0319] Next, the data transmission control processing performed by the automatic driving control unit will be explained.

[0320] As illustrated with reference to Figure 8, in addition to the images captured by the camera, the autonomous driving control device 20 of the autonomous vehicle 10 also sends metadata, including object information and other data, to the remote assistance device 40 of the remote assistance center 30.

[0321] When performing data transmission processing, the data processing unit 22 of the automatic driving control device 20 estimates the bandwidth (bit rate) available for data transmission and performs bit rate control on each transmitted image based on the estimated available bandwidth (bit rate). Furthermore, it performs processes such as selecting images to be transmitted, selecting and compressing object images to be transmitted as metadata, etc.

[0322] The details of these processes performed by the data processing unit 22 of the automatic driving control device 20 will be explained below.

[0323] The flowcharts shown in Figures 9 and 10 are flowcharts illustrating the sequence of data transmission control processing performed by the data processing unit 22 of the automatic driving control device 20.

[0324] Note that the processing corresponding to the flowcharts shown in FIG9 and subsequent figures can be executed by the data processing unit 22 of the automatic driving control device 20 according to the program stored in the storage unit of the automatic driving control device 20. The data processing unit 22 has a processor with program execution capabilities, such as a CPU, and is capable of executing processing as a program execution process executed by the processor.

[0325] The following text will describe the processing at each step in turn.

[0326] Note that the processes at steps S101 to S125 in the flowcharts shown in Figures 9 and 10 are processes that are repeatedly executed during the execution period of the process of sending data from the autonomous driving control device 20 of the autonomous vehicle 10 to the remote assistance device 40 of the remote assistance center 30.

[0327] (Step S101)

[0328] First, in step S101, the data processing unit 22 of the autonomous driving control device 20 calculates the bit rate (estimated bit rate b) that can be used to send data from the autonomous driving control device 20 of the autonomous vehicle 10 to the remote assistance device 40 of the remote assistance center 30. e ).

[0329] Calculate the bit rate available for data transmission (estimate bit rate b) e This processing is performed by the communication network monitoring unit 112 of the data processing unit 22.

[0330] Note the bit rate available for data transmission (estimated bit rate b). e It fluctuates constantly.

[0331] As described above, the processes at steps S101 to S125 in the flowcharts shown in Figures 9 and 10 are repeatedly executed during the execution period of the process of sending data to the remote assistance device 40 of the remote assistance center 30, and the processes executed at steps S102 and subsequent steps are based on the available bandwidth calculated at step S101 (i.e., the estimated bit rate b). e The value is different.

[0332] Note that in the flowcharts shown in Figures 9 and 10, terms such as the estimated bit rate b are used. e The definitions of the various parameters are as follows, as shown in the table in Figure 11.

[0333] b e Estimated bit rate (the bit rate available for communication that is equivalent to the available bandwidth calculated based on monitoring results of the wireless communication network).

[0334] B = {b} min b max}: The target bit rate b, set as the actual data transmission bit rate. target Scope

[0335] b min : The target bit rate b, predefined as the actual data transmission bit rate target minimum bit rate

[0336] b max : The target bit rate b, predefined as the actual data transmission bit rate target Maximum bit rate

[0337] b target : Target bit rate (target value) used for actual data transmission

[0338] b(disabled) allocated The communication bit rate required to maintain communication with communication partners (such as Keep Alive).

[0339] (Step S102)

[0340] Next, at step S102, the estimated bit rate b calculated at step S101 is compared. e and the predefined maximum bit rate b max That is, assess whether the formula described below is satisfied.

[0341]

[0342] Under the condition that the formula described above is satisfied, that is, at an estimated bit rate b equivalent to the available bandwidth... eEqual to or greater than the predefined maximum bit rate b max If the value is "yes", the evaluation at step S102 is "yes" and the process proceeds to step S103.

[0343] On the other hand, in estimating the bit rate b e Less than the predefined maximum bit rate b max If the value is "no", the evaluation at step S102 is "no" and the process proceeds to step S104.

[0344] Maximum bit rate b max It is the target bit rate b, which is to be used for the actual data transmission. target The maximum allowable value, and this value is predefined.

[0345] On the other hand, the target bit rate b is the bit rate to be used for actual data transmission. target The minimum permissible value is the minimum bit rate b. min And this value is also predefined.

[0346] Refer to Figure 12 to illustrate the maximum bit rate b. max and minimum bit rate b min .

[0347] Figure 12 shows a graph with time (seconds) on the horizontal axis and bandwidth (bit rate (bps) on the vertical axis.

[0348] The lines represented by dashed lines indicate the bit rate available for data transmission (estimated bit rate b). e (A line representing the passage of time.)

[0349] As can be understood from the dashed line shown in Figure 12, the bit rate available for data transmission (estimated bit rate b) e It fluctuates significantly over time.

[0350] On the other hand, the line represented by the solid line represents the target bit rate b, which is the target bit rate to be used for actual data transmission. target A timeline.

[0351] As shown in Figure 12, the target bit rate b is the target bit rate to be used for actual data transmission. target The time progression line is set at the maximum bit rate b max and minimum bit rate b min between.

[0352] These maximum bit rates b max and minimum bit rate b min The range of target bit rates, B = [b, ..., b], is predefined as the range to be used for actual data transmission. min bmax The maximum and minimum values ​​of ].

[0353] minimum bit rate b min It is the minimum bit rate at which all camera-captured images 1 to n can be sent to the remote assistance device 40 without causing significant delay. This is achieved by using a bit rate equal to or greater than the minimum bit rate b. min The bit rate is high enough to send all camera-captured images to the remote assistance device 40, and control from the remote assistance device 40 can be received without delay.

[0354] It should be noted that even at the bit rate available for data transmission (estimated bit rate b) e Under conditions where the bandwidth is large and sufficient, that is, even at a bit rate (estimated bit rate b) available for data transmission... e The maximum bit rate b shown in Figure 12 is located at... max In the case of bandwidth region A at or above the specified level, the target bit rate b used for actual data transmission is... target The maximum value is also set to the maximum bit rate b. max .

[0355] The reason for implementing such bit rate control is to ensure that even at the estimated bit rate b representing the available bandwidth... e It can achieve stable data transmission even under sudden changes.

[0356] The estimated bit rate b, represented by the dashed line in Figure 12, indicates the available bandwidth. e The passage of time is also understandable; in some cases, depending on the communication environment, a sudden drop in the estimated bit rate can occur.

[0357] For example, if the target bit rate b is used as the bit rate for actual data transmission target Along the line represented by the dashed line in Figure 12, the estimated bit rate of the available bandwidth is b. e The matching line is used to set the target bit rate b for data transmission. target Unexpectedly, the bit rate may suddenly decrease. If the actual data transmission bit rate suddenly decreases unexpectedly, packet loss and other issues may occur, leading to a so-called video freeze, where the displayed image on the remote assistance device side unexpectedly freezes. To avoid this, the target bit rate b is adjusted. target Maintain control within a certain range.

[0358] That is, stable data transmission is achieved by performing bit rate control to ensure that the target bit rate b, which is the bit rate used for actual data transmission, is within the specified range. target At the predefined maximum bit rate b max and minimum bit rate b min Changes between them.

[0359] However, in estimating the bit rate b e In situations where there is a small amount of bandwidth and insufficient transmission bandwidth, i.e., at the bit rate available for data transmission (estimated bit rate b) e The minimum bit rate b shown in Figure 12 is located at... min In the following bandwidth region C, if all camera-captured images 1 to n are sent, transmission delays may occur, and the reception of control information from the remote assistance device 40 may also be undesirably delayed.

[0360] To avoid this situation, at the bit rate available for data transmission (estimated bit rate b) e (at the minimum bit rate b shown in Figure 12) min In the case of bandwidth region C below, the transmission bit rate is adjusted by performing a process (filtering) that selectively transmits images without transmitting all camera-captured images 1 to n.

[0361] This process is equivalent to the process in the flowchart shown in Figure 9 from (step S104 = No) through (step S108) to (step S111 and subsequent steps).

[0362] (Step S103)

[0363] In step S102, if the evaluation satisfies the formula described below,

[0364]

[0365] That is, at an estimated bit rate b that is assessed as being equivalent to the available bandwidth. e Equal to or greater than the predefined maximum bit rate b max The value (estimated bit rate b) e In the case where the bandwidth is in region A as shown in Figure 12, at step S103, the target bit rate b, which is the bit rate used for actual data transmission, is... target The value is set to the predefined maximum bit rate b max .

[0366] Right now,

[0367] Target bit rate b target =b max

[0368] As described above, the target bit rate b is the bit rate used for actual data transmission. target The value is set to the predefined maximum bit rate b max .

[0369] (Step S104)

[0370] On the other hand, if the evaluation at step S102 determines that the formula described below is not satisfied,...

[0371]

[0372] That is, at an estimated bit rate b that is assessed as being equivalent to the available bandwidth. e Less than the predefined maximum bit rate b max The value (estimated bit rate b) e In the case where the bandwidth is in bandwidth region B or bandwidth region C as shown in Figure 12, at step S104, the estimated bit rate b calculated at step S101 is compared. e and the predefined minimum bit rate b min That is, assess whether the formula described below is satisfied.

[0373]

[0374] Under the condition that the formula described above is satisfied, that is, when the estimated bit rate b is evaluated as being equivalent to the available bandwidth... e Equal to or greater than the predefined minimum bit rate b min The value (estimated bit rate b) e In the case of being in bandwidth region B as shown in Figure 12, the evaluation at step S104 is "yes" and the process proceeds to step S105.

[0375] On the other hand, in the evaluation of the estimated bit rate b e Less than the predefined minimum bit rate b min The value (estimated bit rate b) e In the case of being in the bandwidth region C shown in Figure 12, the evaluation at step S104 is "No", and the process proceeds to step S108.

[0376] (Step S105)

[0377] If the evaluation at step S104 is satisfied with the formula described below,

[0378]

[0379] That is, at an estimated bit rate b that is assessed as being equivalent to the available bandwidth. e Equal to or greater than the predefined minimum bit rate b min The value (estimated bit rate b) e In the case where the bandwidth is in region B as shown in Figure 12, at step S105, the target bit rate b is set as the bit rate for actual data transmission.target The value is set to an estimated bit rate b, which is equivalent to the available bandwidth estimated at step S101. e .

[0380] Right now,

[0381] Target bit rate b target =b e .

[0382] It should be noted that here, the estimated bit rate b, which is equivalent to the available bandwidth, is... e It satisfies the formula described below.

[0383]

[0384] That is, estimate the bit rate b e Located in bandwidth region B shown in Figure 12, and with a target bit rate b target =Change within bandwidth region B shown in Figure 12.

[0385] By doing so, the target bit rate b target By varying the bandwidth region B shown in Figure 12, a minimum bit rate b was achieved that was equal to or greater than the minimum bit rate b. min Data is transmitted at a bit rate.

[0386] As mentioned earlier, the minimum bit rate b min It is the bit rate at which all camera-captured images can be sent to the remote assistance device 40 without causing significant delay.

[0387] By using this method, the target bit rate b for data transmission is... target By setting the bandwidth area B shown in Figure 12 and sending all camera-captured images to the remote assistance device 40, control from the remote assistance device 40 can be received without delay.

[0388] (Step S106)

[0389] At step S103 or step S105, the target bit rate b, which is the bit rate used for actual data transmission, is determined. target If the value is , then the processing at step S106 is executed.

[0390] That is, at the target bit rate b, which is the bit rate used for actual data transmission target The value has

[0391] The target bit rate b is determined at step S103. target =b max or

[0392] The target bit rate b is determined at step S105. target =b e In this case,

[0393] Perform the processing at step S106.

[0394] Note that the target bit rate b determined in step S103 or step S105 target (That is, the target bit rate b, which is the bit rate used for actual data transmission) target The value of ) satisfies the formula described below.

[0395]

[0396] That is, the target bit rate b, which is the bit rate used for data transmission. target The value is set within bandwidth region B shown in Figure 12. Target bit rate b target =Change within bandwidth region B shown in Figure 12.

[0397] By target bit rate b target When set within bandwidth zone B, all camera-captured images can be sent to the remote assistance device 40 without causing significant delay.

[0398] In step S106, the allocation of communication bit rates for the captured images (video tracks) 1 to n of each camera 1 to n is determined based on the resolution (number of pixels) of each image.

[0399] The total communication bit rate of the captured images (video tracks) from cameras 1 to n is the target bit rate b. target Target bit rate b target Each captured image (video track) 1 to n is assigned a bit rate, and the transmission bit rate of each captured image (video track) 1 to n is determined.

[0400] The data processing unit 22 of the automatic driving control device 20 calculates the bit rate of each captured image (video track) 1 to n based on the resolution of each image 1 to n and the target bit rate.

[0401] This processing is performed by the data analysis unit 113 of the data processing unit 22 shown in Figure 5.

[0402] Refer to Figure 13 for a specific example of the process for determining the allocation of transmission bit rates for each captured image (video track) 1 to n.

[0403] Figure 13 shows the resolution (number of pixels) of images captured by six cameras mounted on the autonomous vehicle 10, as illustrated earlier with reference to Figures 3 and 4.

[0404] The resolutions of the following six individual images are shown.

[0405] (a) Left rear-view captured image (track ID = video track 1) resolution = w1 x h1

[0406] (b) Right rear-view captured image (track ID = video track 2) resolution = w2 x h2

[0407] (c) Rear-view captured image (track ID = video track 3) resolution = w3 x h3

[0408] (d) Left-view captured image (track ID = video track 4) resolution = w4 x h4

[0409] (e) Right-view captured image (track ID = video track 5) resolution = w5 x h5

[0410] (f) Forward-looking captured image (track ID = video track 6) resolution = w6 x h6

[0411] Note that wi is equivalent to the number of pixels in the horizontal direction of image i, and hi is equivalent to the number of pixels in the vertical direction of image i.

[0412] wiXhi is equivalent to the resolution (number of pixels) of image i.

[0413] The bit rate used to send all these images is defined as the target bit rate b. target Let N be the total number of images (video tracks) and i be the camera ID. Note that in the example shown in Figure 13, N = 6 and i = 1 to 6.

[0414] At this point, the bit rate (b(i)) allocated to the i-th image (video track) can be calculated using the following formula (Formula 1). allocated ).

[0415] [Formula 1]

[0416] ...(Formula 1)

[0417] in,

[0418] ...(Formula 1a)

[0419] ...(Formula 1b)

[0420] Notice,

[0421] r i These are the multiplication coefficients for the i-th video track, and can be obtained by adjusting the target bit rate b.target Multiplied by the multiplication coefficient r of the i-th image (video track) i To calculate the transmission bit rate (b(i)) of the i-th image (video track). allocated ).

[0422] Note that (Formula 1a) is used to calculate the target bit rate b. target The formula for the multiplication coefficient ri.

[0423] (Formula 1b) is a formula that shows that the total value of the multiplication coefficients ri corresponding to each of all images (video tracks) becomes 1.

[0424] As expressed by (Equation 1a), the target bit rate b is determined based on the product of wi and hi for each image, wiXhi (i.e., the resolution commensurate with the number of pixels). target The multiplication coefficient ri. That is, the transmission bit rate (b(i)) for each image (video track) i is determined proportionally to the resolution (number of pixels) of each image. allocated The allocation of ).

[0425] In this way, at step S106, the data processing unit 22 of the automatic driving control device 20 determines the allocation of communication bit rates for the captured images (video tracks) 1 to n of each camera 1 to n based on the resolution (number of pixels) of each image.

[0426] The total communication bit rate of the captured images (video tracks) from cameras 1 to n is the target bit rate b. target Target bit rate b target Each captured image (video track) 1 to n is assigned a bit rate, and the transmission bit rate of each captured image (video track) 1 to n is determined.

[0427] (Step S107)

[0428] Next, at step S107, the data processing unit 22 of the automatic driving control device 20 sends each captured image (video track 1 to n) of each camera 1 to n to the remote assistance device 40 according to the communication bit rate allocated to each image (video track) at step S106.

[0429] The total bitrate of the transmission bitrates for each captured image (video track) 1 to n used for data transmission is the target bitrate b. target And the target bit rate b target Within the bandwidth region B shown in Figure 12, that is, at the minimum bit rate b min and maximum bit rate b max Within the range. That is, it satisfies the formula described below.

[0430]

[0431] In this way, the target bit rate b is achieved by making the total bit rate of the transmission bit rates of each captured image (video track) 1 to n. target By changing the bandwidth region B shown in Figure 12, it becomes possible to send all camera-captured images 1 to n to the remote assistance device 40 without causing significant delay.

[0432] Note that after the processing at step S107 is completed, the process returns to step S101 and repeats the processing of step S101 and subsequent steps.

[0433] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0434] (Step S108)

[0435] On the other hand, the evaluation at step S104 is deemed not to meet the estimated bit rate b calculated in step S101. e and the predefined minimum bit rate b min Comparison / Evaluation Formula

[0436]

[0437] In the case that the estimated bit rate b is evaluated... e Less than the predefined minimum bit rate b min The value (estimated bit rate b) e In the case of being located in the bandwidth region C shown in Figure 12, the process proceeds to step S108.

[0438] At step S108, the target bit rate b target Set to minimum bit rate b min .

[0439] b target =b min

[0440] As mentioned above, the target bit rate b target Set to minimum bit rate b min .

[0441] However, the evaluation at the execution stage of step S108 is "no" as the evaluation at step S104, that is, the formula described below is not satisfied.

[0442]

[0443] And the estimated bit rate b available for data transmission obtained from network monitoring results. e Less than minimum bit rate b min That is, estimating the bit rate b. e Below the minimum bit rate b shown in Figure 12 min In the bandwidth region C.

[0444] In this state, even if the target bit rate b target Set to minimum bit rate b min The bit rate available for actual data transmission is also set to be less than the minimum bit rate b. min Estimated bit rate b e .

[0445] In this situation, if all camera-captured images 1 to n are sent, there is a possibility of transmission delays, and the control of the remote assistance device 40 may be undesirably delayed.

[0446] Therefore, in such a case, that is, at the bit rate available for data transmission (estimated bit rate b) e (Below the minimum bit rate b shown in Figure 12) min In the case of bandwidth region C, the transmission bit rate is adjusted by performing processes such as selectively transmitting images without transmitting all camera-captured images 1 to n.

[0447] This process is the process at step S111 and subsequent steps in the flowchart shown in Figure 10.

[0448] (Step S111)

[0449] The bit rate assessed as usable for data transmission (estimated bit rate b) e Less than the minimum bit rate b min In the case that the estimated bit rate b is evaluated... e Below the minimum bit rate b shown in Figure 12 min In the case of bandwidth region C, firstly, at step S111, the data processing unit 22 of the automatic driving control device 20 performs processing to classify each captured image (video track) 1 to n of each camera 1 to n into sent image (enabled video track) or unsent image (disabled video track) according to image priority.

[0450] As mentioned earlier, at the bit rate available for data transmission (estimated bit rate b) e (Below the minimum bit rate b shown in Figure 12) minIn the case of bandwidth region C, if all camera-captured images 1 to n are sent, communication delays may occur, and control information from the remote assistance device 40 may not be received in a timely manner.

[0451] To avoid this situation, in step S111, a process for selecting and sending images (filtering process) is performed so that data can be sent with very little delay even at low bit rates.

[0452] Note that details on the processing of categorizing images into those sent (video track enabled) and those not sent (video track disabled) based on image priority will be discussed later.

[0453] Here, firstly, as the bit rate available for data transmission (estimated bit rate b) e (Below the minimum bit rate b shown in Figure 12) min The summary of the processing performed by the data processing unit 22 of the automatic driving control device 20 in the case of bandwidth region C is explained, and the processing at steps S112 to S114 and steps S121 to S125 shown in FIG10 are described.

[0454] Note that the processing at steps S112 to S114 shown in Figure 10 is a processing sequence for the camera-captured images, and the processing at steps S121 to S125 is a processing sequence related to metadata such as objects detected from the camera-captured images and LiDAR detection information.

[0455] The processing at steps S112 to S114 and steps S121 to S125 shown in Figure 10 are executed in parallel at the data processing unit 22 of the automatic driving control device 20.

[0456] (Step S112)

[0457] In step S112, the bit rate b (disabled) for maintaining communication is set for the communication track of each image. allocated For example, the communication track for each image is set with the bit rate b (disabled) configured as described below to maintain communication. allocated .

[0458] b(disabled) allocated =30000bps

[0459] This involves allocating bandwidth to maintain connectivity for each track even when video transmission on the communication track used for each image is disabled. For example, in the case of WebRTC, the allocated bitrate is 30 kbps. Note that when some images are classified as untransmitted and the video track used to transmit these images is disabled, the bitrate is reallocated to other enabled video tracks, i.e., the transmission video tracks that transmit the target images. Thus, image transmission on the video tracks used to transmit high-priority images is performed at a high bitrate.

[0460] (Step S113)

[0461] Next, in step S113, the data processing unit 22 of the automatic driving control device 20 performs a process of reallocating the communication bit rate only for the images selected as images to be sent (video track enabled) in the classification process in step S111.

[0462] In step S111, the data processing unit 22 performs a process of classifying each captured image (video track) 1 to n of each camera 1 to n into transmitted images (enabled video track) or untransmitted images (disabled video track) according to image priority.

[0463] In step S113, the data processing unit 22 performs a process of allocating communication bit rates only to the images selected for transmission (video track enabled) in step S111.

[0464] Note that the total communication bit rate allocated here is the bit rate (estimated bit rate b) available for data transmission obtained by the communication network monitoring unit 112 of the data processing unit 22 of the automatic driving control device 20. e ), and is the estimated bit rate b obtained at step S101. e .

[0465] Estimated bit rate b e Below the minimum bit rate b shown in Figure 12 min In the bandwidth region C, and it is a low bit rate that makes it difficult to send all camera-captured images with a short delay, but it is a bit rate that allows sending only the images selected according to priority at step S111 (enabled video track) with a short delay.

[0466] Note that details of the specific bit rate allocation process will be explained later.

[0467] (Step S114)

[0468] Next, at step S114, the data processing unit 22 of the automatic driving control device 20 performs the processing of sending each sent image (enabled video track) selected according to priority at step S111 based on the communication bit rate reassigned to the sent image (enabled video track) at step S113.

[0469] Since the images transmitted here are not all images captured by the cameras, but only a selection of images chosen for transmission based on priority at step S111, even if the available bandwidth (bit rate) for transmission is below the minimum bit rate b shown in Figure 12, min In the case of bandwidth region C, it becomes possible to send images without causing significant delay, and to receive control information from remote assistance device 40 in a timely manner.

[0470] Note that after the processing at step S114 is completed, the process returns to step S101 and repeats the processing at step S101 and subsequent steps.

[0471] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0472] Next, the processing at steps S121 to S125 will be explained.

[0473] The processing at steps S121 to S125 shown in Figure 10 is a sequence of transmission control processes related to metadata such as objects detected from camera-captured images and LiDAR detection information.

[0474] As mentioned above, in step S111, processing is performed to classify images into those sent (video track enabled) and those not sent (video track disabled) based on image priority, and the processing of sending low-priority images is stopped undesirably.

[0475] However, in some cases, objects that may collide with or come into contact with the autonomous vehicle 10 (such as oncoming vehicles and pedestrians), i.e., high-risk objects, are also included in the low-priority images that are selected as not to be sent.

[0476] Steps S121 to S125 are a sequence of processes for performing high-risk objects included in unsent images (disabled video tracks) in this manner and sending object information about the extracted objects as metadata separated from the images to the remote control device 40.

[0477] Note that the details of the object extraction process, risk level assessment process, object filtering process, etc., performed in steps S121 to S125 will be explained in more detail later. Here, only an outline of the processing sequence is given.

[0478] (Step S121)

[0479] First, in step S121, the data processing unit 22 of the autonomous driving control device 20 extracts objects (vehicles, pedestrians, buildings, etc.) around the autonomous driving vehicle 10 from the unsent images (disabled video track).

[0480] The target image for object extraction is the unsent image (disabled video track) from the sent images (enabled video track) and unsent images (disabled video track) classified according to image priority in step S111. That is, object extraction processing is performed on images whose image data was not sent.

[0481] (Step S122)

[0482] Next, at step S122, the data processing unit 22 of the automatic driving control device 20 assesses the risk level of each object extracted from the unsent image at step S121.

[0483] For example, objects with a high risk of collision are assessed as high risk.

[0484] The specific handling will be discussed later.

[0485] (Step S123)

[0486] Next, in step S123, the data processing unit 22 of the automatic driving control device 20 compresses each extraction object using a different compression mode (using different compression algorithms) based on the risk level of each extraction object assessed in step S122.

[0487] (Step S124)

[0488] Next, in step S124, the data processing unit 22 of the automatic driving control device 20 performs a filtering process on the extracted and compressed object image data.

[0489] The filtering process is the process of selecting compressed object image data to be sent, and it is a process of classifying compressed object image data that should be considered as the target of transmission and compressed object image data that should not be sent.

[0490] Filtering is performed when the bit rate required to send data for the compressed object is greater than the available bandwidth.

[0491] It should be noted that the object coordinates representing the object's location, the bounding box coordinates representing the three-dimensional bounding box region surrounding the object, and the label representing the object type (vehicle, person, etc.) are all included in the metadata independently of the object's risk level and are sent to the remote assistance device 40.

[0492] Details of the filtering process will be mentioned later.

[0493] (Step S125)

[0494] Next, in step S125, the data processing unit 22 of the automatic driving control device 20 includes the compressed object image data that is finally selected as the sending target as the result of the filtering process in step S124 in the metadata, and sends the metadata to the remote assistance device 40.

[0495] By executing steps S121 to S125, if a high-risk object exists in an image that is not expected to be sent (video track disabled) at step S111, the image of the high-risk object is sent as constituent data of metadata to the remote assistance device 40.

[0496] The remote assistance device 40 performs the following processing: analyzes the metadata received from the automatic driving control device 20 to extract object compressed image data, decodes the object compressed image data, and overlays object data such as vehicles and pedestrians onto the display image (e.g., a 3DCG image) on the display unit 43.

[0497] Through this process, the operator at the remote assistance center 30 can inspect high-risk objects included in the unsent images and become able to perform the process of generating appropriate control information to prevent the autonomous vehicle 10 from colliding with the high-risk objects and sending the control information to the autonomous driving control unit 20.

[0498] Note that after the processing at step S125 is completed, the process returns to step S101 and repeats the processing at step S101 and subsequent steps.

[0499] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0500] [5. Details regarding the processing of selectively sending camera-captured images based on priority]

[0501] Next, we will explain the details of the processing for selectively sending camera-captured images based on priority.

[0502] The flowchart shown in Figure 14 is a flowchart explaining the details of the priority-based camera captured image sorting process and priority-based image transmission selection process performed by the data processing unit 22 of the autonomous driving control device 20 of the autonomous vehicle 10.

[0503] Note that the flowchart shown in Figure 14 is a detailed flowchart of the processes at steps S111 to S114 in the flowchart described earlier with reference to Figure 10.

[0504] That is, at step S108 in the flowchart shown in Figure 9, the bit rate (estimated bit rate b) is evaluated as usable for data transmission. e Less than the minimum bit rate b min In this case, processing is performed. Specifically, in the case of an evaluation to estimate the bit rate b... e Below the minimum bit rate b shown in Figure 12 min In the case of bandwidth region C, the processing is performed.

[0505] The details of the processing at steps S201 to S209 in the flowchart shown in Figure 14 will be described in turn below.

[0506] Note that, as described in the table shown in Figure 15, the parameters used in the flowchart shown in Figure 14 are defined as follows.

[0507] M: Total number of enabled images (images sent)

[0508] s(M) required Total required bitrate for all enabled images (sending images)

[0509] b(j) required : The bit rate required to send the j-th image

[0510] j: The index in the image list sorted by priority (descending order) (the index j of the highest priority image is 1).

[0511] (Step S201)

[0512] First, the estimated bit rate (b) is assessed as usable for data transmission. e Less than the minimum bit rate b min In the case that the estimated bit rate b is evaluated... e Below the minimum bit rate b shown in Figure 12 min In the case of bandwidth region C, at step S201, the data processing unit 22 of the automatic driving control device 20 performs processing to sort the captured images (video tracks) 1 to n of each camera 1 to n according to image priority.

[0513] Note that this processing is performed by the image priority analysis module 111d of the risk level classification unit 111 of the data processing unit 22 of the automatic driving control device 20, which was described earlier with reference to FIG5.

[0514] As mentioned earlier, at the bit rate available for data transmission (estimated bit rate b) e (Below the minimum bit rate b shown in Figure 12) min In the case of bandwidth region C, if all camera-captured images 1 to n are sent, communication delays may occur, and control information from the remote assistance device 40 may not be received in a timely manner.

[0515] The flowchart shown in Figure 14 is a processing flowchart for avoiding such a situation, and is a flowchart for enabling data transmission with short latency even at low bit rates by performing processes such as selective image transmission.

[0516] Refer to Figure 16 and subsequent figures to illustrate a specific example of image sorting processing based on image priority.

[0517] Figure 16 is a diagram illustrating the image analysis parameters used in the process of determining the priority of six images corresponding to the six cameras described earlier with reference to Figures 3 and 4.

[0518] That is, Figure 16 is a diagram illustrating the elements used in the process of determining the priority of the six images (front view image, rear view image, right rear view image, left rear view image, right view image and left view image) of the autonomous vehicle 10.

[0519] For example, as shown in Figure 16, image priority is calculated using each of the following elements (analysis parameters (f = 1 to n)) obtained through image analysis processing. That is,

[0520] (a) Number of surrounding objects (f=1) (=Descending order of the number of surrounding objects detected in the image)

[0521] (b) Number of high-risk objects (f=2) (=Descending order of the number of high-risk objects detected in the image)

[0522] (c) Remote assistance tasks (f=3) (=Descending order of importance of generating autopilot control information at the remote assistance device)

[0523] (d) Gaze (f=4) (=Descending order of the operator's gaze at the remote assistance device)

[0524] Note that the types of objects to be counted as either (a) the number of surrounding objects or (b) the number of high-risk objects are predefined. For example, the types of objects to be considered as detection targets (such as nearby vehicles, pedestrians, bicycles, motorcycles, and roadside structures (guardrails, traffic lights, etc.)) are predefined.

[0525] Additionally, the four elements (analysis parameters (f = 1 to n) parameters to be applied to image priority calculation shown in Figure 16 are examples. Image priority calculation processing can be performed using only some of these elements (analysis parameters), or it can be performed using another element (analysis parameter).

[0526] First, the image priority analysis module 111d of the risk level classification unit 111 of the data processing unit 22 of the autonomous driving control device 20 calculates a score (safety score S) corresponding to each of the above elements (each analysis parameter) according to a predetermined algorithm. f (P)).

[0527] Note that the scores for (a) the number of surrounding objects (f=1) and (b) the number of high-risk objects (f=2) of the above-mentioned elements (each analysis parameter (f=1 to n)) are calculated only by analyzing the images captured by the camera. On the other hand, the scores for (c) the remote assistance task (f=3) and (d) the gaze (f=4) are calculated by analyzing the information received from the remote assistance device 40.

[0528] For example, when calculating the score for (d) gaze (f=4), analytical information about the gaze heatmap representing the degree of concentration of the operator's gaze on the remote assistance center 30 side is received to perform the score calculation.

[0529] Figure 17 is a specific example of a gaze heatmap showing the degree of focus of the operator's gaze on the remote assistance center 30 side. Bright circular areas in each image of this figure represent areas of high operator gaze concentration.

[0530] The data processing unit 22 of the automatic driving control device 20 calculates the score (safety score S) corresponding to each of the above elements (a) to (d) (analysis parameters (f = 1 to 4)) according to the defined score calculation algorithm. f (P)), and then further apply the following calculation formula (Formula 2) to calculate the final priority value P for each image. * .

[0531] [Formula 2]

[0532] …(Formula 2)

[0533] Note that in the above (Formula 2),

[0534] s f (P) is the safety score corresponding to each element (analysis parameter: each of elements f1 to f4 shown in Figure 16).

[0535] f is the index of the element (analysis parameter) that affects image priority, and

[0536] F represents the total number of elements (analysis parameters).

[0537] Referring to Figure 18, an example of a process is given for analyzing (a) the number of surrounding objects and (b) the number of high-risk objects as elements (analysis parameters) to be used in the image priority calculation process according to the above (Formula 2).

[0538] Figure 18 is a diagram illustrating objects detected from six images corresponding to the six cameras described earlier with reference to Figures 3 and 4. That is, Figure 18 is a diagram illustrating objects detected from six images (front view image, rear view image, right rear view image, left rear view image, right view image, and left view image) of the autonomous vehicle 10.

[0539] The rectangle shown in each image is an object detection box, and the detected objects include surrounding vehicles, pedestrians, etc.

[0540] (a) A score (safety score) is calculated based on the number of objects detected in each image, representing the number of objects in the surrounding area.

[0541] (b) The score of the number of high-risk objects (safety score) is calculated by calculating the risk of each object detected in each image, and the score (safety score) is calculated in the order of the sum of the risks of the objects detected in each image.

[0542] A specific example of how to calculate the risk for each object is provided.

[0543] For example, object risk calculation processing can be performed as a process that classifies each object into one of multiple risk levels 1 to n.

[0544] As a technique for calculating the risk level of an object, for example, RSS-based (responsibility-sensitive security) processing can be used.

[0545] RSS is a technique for evaluating the appropriateness of a pair of objects detected from an image with an autonomous vehicle 10, which is the subject vehicle.

[0546] The risk level classification unit 111 of the data processing unit 22 of the autonomous driving control device 20 shown in Figure 5 has an RSS module and receives an object list including information about objects (road agents) in the surrounding environment of the subject vehicle.

[0547] The RSS module generates a description of the object and the subject vehicle pair for each object, as well as an attribute called "situation".

[0548] Next, the RSS module performs a corresponding RSS security check for each situation, and calculates the risk level of each object as a result of the security check.

[0549] Another method that can be used as a technique for calculating the risk level of an object is to use a neural network technique as shown in Figure 19.

[0550] In this case, the risk level analysis module 111e of the risk level classification unit 111 of the data processing unit 22 of the autonomous driving control device 20 shown in FIG. 5 has a neural network as shown in FIG. 19.

[0551] The input data for the neural network in the object risk level analysis module 111e consists of the type, size, direction of movement, and distance from the subject vehicle of the detected object in each image, as well as network status, weather, and road conditions. The output is the risk level of the object.

[0552] Note that past accident data of autonomous vehicles can be used as training data for neural networks. For example, the US NHTSA (National Highway Traffic Safety Administration) publishes accident reports of autonomous vehicles, and this data can be used to train and process neural networks for risk level calculations.

[0553] In this way, techniques such as RSS (responsibility-sensitive safety) and neural networks can be used as processing techniques for calculating the risk level of an object.

[0554] Note that, regardless of the processing method applied, the risk level corresponding to each object calculated by the object risk level analysis module 111e is calculated as the level corresponding to the predefined level definition.

[0555] As a classification of levels, for example, a classification can be used based on the analysis results of past accident data at the current location of the autonomous vehicle 10, as shown below.

[0556] Risk Level 1 = Serious Accident

[0557] Risk Level 2 = Moderate Accidents Expected

[0558] Risk Level 3 = Minor Accident

[0559] Risk Level 4 = No harm reported

[0560] The risk level analysis module 111e of the risk level classification unit 111 of the data processing unit 22 of the autonomous driving control device 20 calculates the risk level for each object detected from each camera-captured image shown in Figure 17, which was described earlier, using the aforementioned RSS or neural network.

[0561] The risk level of each object calculated by the object risk level analysis module 111e is input to the image priority analysis module 111d of the risk level classification department 111, and the image priority analysis module 111d also calculates the priority (P) of each image using a safety score corresponding to other factors (analysis parameters) according to (Formula 2) described earlier. * ).

[0562] At step S201 in the flowchart shown in Figure 14, the image priority of each captured image (video track) 1 to n of each camera 1 to n is calculated in this way and the images are sorted according to the calculated image priority.

[0563] (Step S202)

[0564] Next, in step S202, the data processing unit 22 of the automatic driving control device 20 calculates the bit rate (required bit rate s(M)) required to send the M (initial value M = total number of images) images (video enabled) sorted according to priority in step S201. required ).

[0565] Required bit rate s(M) required It is determined to be the bit rate at which M (initial value M = total number of images) images (video enabled) ordered by priority are sent to the remote control device 40 with a delay equal to or less than a predetermined value.

[0566] Note that the initial value of M is the total number of images captured by the cameras. For example, in the example illustrated earlier with reference to Figures 3 and 4, since there are six cameras mounted on the autonomous vehicle 10 and six images are captured, M = 6.

[0567] The data processing unit 22 of the automatic driving control device 20 calculates the bit rate (required bit rate s(M)) required to send M images according to the following formula (Formula 3). required ).

[0568] [Formula 3]

[0569] …(Formula 3)

[0570] in,

[0571]

[0572] Note that in the above formula (Formula 3),

[0573] F(*) is the bit rate required for each image-by-image segment, which is equal to b(j). required Frame rate under certain conditions, and

[0574] f target It is the target frame rate used to ensure the smoothness of the image reproduced on the remote device 40 side.

[0575] Note the bit rate b(j) required per image. required It is the bit rate required to send an image with priority index j to the remote control device 40 with a delay equal to or less than a predetermined value.

[0576] The delay amount equal to or less than the predetermined value is allowed to be at the target frame rate (f) of the remote control device 40. target The range of delay in reproducing the received image.

[0577] The required bit rate b(j) per image for sending an image with priority index j to the remote control device 40 with a delay equal to or less than a predetermined value. required The minimum value is appropriately used to maintain a smooth video display on the remote assistance device 40 side, such as a bit rate that can achieve an image reproduction frame rate of more than 30kbps.

[0578] (Step S203)

[0579] Next, in step S203, the data processing unit 22 of the automatic driving control device 20 executes the calculation in step S202 of the bit rate (required bit rate s(M)) required to send M images. required ) and estimated bit rate b e The comparison process.

[0580] Note the estimated bit rate b e The current available bit rate (estimated bit rate b) for data transmission is obtained by the communication network monitoring unit 112 of the data processing unit 22 of the automatic driving control device 20. e This is the estimated bit rate b obtained at step S101 in the flowchart illustrated earlier with reference to Figure 9. e .

[0581] At step S203, evaluate whether the formula described below is satisfied.

[0582]

[0583] That is, the required bit rate (required bit rate s(M)) calculated at step S202 for sending M images is evaluated. required Is it equal to or less than the estimated bit rate b currently available for communication? e .

[0584] If the evaluation satisfies the formula described below...

[0585]

[0586] The process proceeds to step S204.

[0587] On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S205.

[0588] (Step S204)

[0589] If the evaluation at step S203 satisfies the evaluation formula described below, then...

[0590]

[0591] The process proceeds to step S204.

[0592] Satisfying the evaluation formula described above is equivalent to evaluating the bit rate (required bit rate s(M)) required to send M images, as calculated at step S202. required The bit rate b is equal to or less than the estimated bit rate currently available for communication. e .

[0593] That is, this means that all M images calculated in step S202 can be sent.

[0594] In this case, at step S204, the data processing unit 22 of the automatic driving control device 20 sends all M images calculated at step S202.

[0595] Note that after the processing at step S204 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0596] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0597] (Step S205)

[0598] On the other hand, if the evaluation at step S203 fails to meet the evaluation formula described below,...

[0599]

[0600] The process proceeds to step S205.

[0601] The failure to satisfy the evaluation formula described above means that the bit rate required to send M images (required bit rate s(M)) calculated at step S202 is not met. required (b) is greater than the estimated bit rate currently available for communication. e .

[0602] That is, this means that, in the case of sending all M images calculated in step S202, the estimated bit rate b currently available for communication is... e This may cause delays, etc.

[0603] In this case, the data processing unit 22 of the automatic driving control device 20 performs the processing in step S205 and subsequent steps.

[0604] First, in step S205, the data processing unit 22 of the automatic driving control device 20 disables (disables) the Mth image with the lowest priority according to the image priority-based sorting generated in step S201, that is, sets the Mth image as an unsent image that stops being sent.

[0605] (Step S206)

[0606] Next, in step S206, the data processing unit 22 of the automatic driving control device 20 performs the process of deciding to allocate bit rates to each of the remaining transmitted images ((M-1) images) except for the disabled images (untransmitted images).

[0607] This process is equivalent to the process at step S113 in the flowchart shown in Figure 10, which was described earlier.

[0608] The bit rate b(i) that is reassigned to each transmitted image ((M-1) images) other than the disabled images (unsent images) is calculated according to the following formula (Formula 4). allocated .

[0609] …(Formula 4)

[0610] Note that in the formula described above (Formula 4),

[0611] r i These are the multiplication coefficients of the i-th image, and

[0612] n disabled This is the number of disabled images (the number of images that were not sent).

[0613] Note that, as explained earlier (Equation 1a), the multiplication coefficient r of the i-th image... i The resolution is determined by the product of wi and hi for each image, wiXhi, which is equivalent to the number of pixels.

[0614] (Step S207)

[0615] Next, in step S207, the data processing unit 22 of the automatic driving control device 20 performs the process of updating the value of the variable M, which represents the number of target images to be processed, to M = M-1, that is, subtracting 1 from the variable M.

[0616] This is the process of excluding disabled images, i.e., images that are set as not to be sent, from the processing target.

[0617] (Step S208)

[0618] Next, at step S208, the data processing unit 22 of the automatic driving control device 20 evaluates whether the value of the variable M representing the number of target images to be processed is equal to or greater than 1, that is, whether it satisfies the formula described below.

[0619]

[0620] If the formula described above is satisfied, that is, if the number of target images M is equal to or greater than 1, the process returns to step S202, and the processing at step S202 and subsequent steps is repeated. That is, for the updated number of images M, the process is repeated as described with the estimated bit rate b. e Processing such as comparisons and bit rate reallocation.

[0621] If, after repeating these processes, the value of the number of target images M becomes less than 1, that is, if the evaluation at step S208 determines that the formula described below is not satisfied,

[0622]

[0623] The process proceeds to step S209.

[0624] (Step S209)

[0625] If the evaluation at step S208 determines that the formula described below is not satisfied, then...

[0626]

[0627] The process proceeds to step S209.

[0628] The case where the formula described above is not satisfied is the case where M=0, that is, the number of target images to be processed becomes 0.

[0629] That is, this means that it is possible to transmit data at a bit rate equal to or less than the bit rate available for current data transmission (i.e., the estimated bit rate b). e The number of images transmitted at the bit rate of 0 is 0. In this case, at step S209, the data processing unit 22 of the automatic driving control device 20 disables (disables) all images. That is, all images are set to not be transmitted.

[0630] Note that after the processing at step S209 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0631] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0632] Note that, as previously stated, the flowchart shown in Figure 14 corresponds to a detailed flowchart of the processing at steps S111 to S114 in the flowchart illustrated earlier with reference to Figure 10, and is the bit rate (estimated bit rate b) evaluated at step S108 in the flowchart shown in Figure 9 that is suitable for data transmission. e Less than the minimum bit rate b min The processing performed under these circumstances. Specifically, in the case of evaluating as an estimated bit rate b e Below the minimum bit rate b shown in Figure 12 min These processes are performed in the case of bandwidth region C.

[0633] That is, these processes are performed when it is difficult to transmit all camera-captured images with a short delay (equal to or less than a predetermined delay). In such cases, the data processing unit 22 of the automatic driving control device 20 performs these processes according to the flowchart shown in FIG14. That is, it selects images to be transmitted according to priority, reallocates the transmission bit rate to each selected image, and transmits the selected images.

[0634] By performing such processing, the selected image can be sent to the remote assistance device 40 without delay, and the remote control of the operator 50 can be performed in a timely manner without delay.

[0635] Note that for low-priority unsent images, a process is performed to send information about objects such as vehicles extracted from the unsent images as metadata to the remote assistance device 40. That is, this is the process at steps S121 to S125 in the flowchart shown in FIG10.

[0636] The details of the process are described in the following paragraphs.

[0637] Referring to Figure 20, a specific example of image priority-based selective image transmission processing performed according to the flowchart shown in Figure 14 is illustrated.

[0638] The upper part of Figure 20 shows six images captured using six cameras mounted on the autonomous vehicle 10, as illustrated earlier with reference to Figures 3 and 4. These six images are the following camera capture images (views).

[0639] (LBv) Left rear view (image captured by the left rear view camera)

[0640] (Bv) Rear view (image captured by the rear-view camera)

[0641] (RBv) Right rear view (image captured by the right rear-view camera)

[0642] (Lv) Left side view (image captured by the left-side camera)

[0643] (Fv) Front view (image captured by the front-view camera)

[0644] (Rv) Right side view (image captured by the right-view camera)

[0645] In step S201 of the flowchart shown in Figure 14, the data processing unit 22 of the automatic driving control device 20 sorts the images according to image priority.

[0646] That is, as explained earlier with reference to Figure 16, the data processing unit 22 calculates the priority of the image based on factors (analysis parameters) such as (a) the number of surrounding objects, (b) the number of high-risk objects, (c) remote assistance tasks and (d) gaze, and sorts the images in descending order of the calculated priority.

[0647] An example of the image sorting results based on priority order is shown in the table in the lower left corner of Figure 20.

[0648] In this example, the images are sorted in descending order of priority as follows.

[0649] Priority 1 = (Fv) Front view (image captured by the front-view camera)

[0650] Priority 2 = (Bv) Rear View (Image captured by the rear-view camera)

[0651] Priority 3 = (RBv) Right Rear View (Image captured by the right rear-view camera)

[0652] Priority 4 = (LBv) Left Rear View (Image captured by the left rear-view camera)

[0653] Priority 5 = (Rv) Right-side view (image captured by the right-side camera)

[0654] Priority 6 = (Lv) Left side view (image captured by the left-side camera)

[0655] In step S202 and subsequent steps of the flowchart shown in Figure 14, the data processing unit 22 of the automatic driving control device 20 calculates the bit rate (required bit rate s(M)) required to send six images sorted according to priority. required ).

[0656] The required bit rate (bit rate s(M)) for sending six images required ) is the bit rate b(j) allocated to each of the six individual images. required The total.

[0657] The table in the lower left corner of Figure 20 shows the bit rate b(j) allocated to each image. required .

[0658] Furthermore, based on the bit rate b(j) allocated to each image... required The table in the lower right corner of Figure 20 shows the total bit rate (required bit rate for each image combination unit) for each image combination unit, including one or more images selected from high-priority images.

[0659] The total bit rate of each image combination unit (i.e., the required bit rate of each image combination unit) is equivalent to the required bit rate for enabling the image combination selected in descending order of image priority to be sent to the remote control device 40 with a short delay (equal to or less than a predetermined delay amount).

[0660] S(1) required It is the total bit rate when only selectively transmitting the (Fv) front view with priority = 1.

[0661] S(2) required The total bit rate is the case where a (Fv) front view with priority = 1 and a (Bv) back view with priority = 2 are selectively transmitted.

[0662] S(3) required This represents the total bit rate when selectively transmitting images with priorities 1 to 3.

[0663] S(4) requiredThis represents the total bit rate when selectively transmitting images with priorities 1 to 4.

[0664] S(5) required This represents the total bit rate when selectively transmitting images with priorities 1 to 5, and

[0665] S(6) required This represents the total bit rate when selectively transmitting all images with priorities 1 to 6.

[0666] In steps S202 to S209 of the flowchart in Figure 14, the total bit rate S(6) is first evaluated when selectively transmitting all images with priorities 1 to 6. required Is it less than the estimated bit rate b? e That is, at step S203, it is evaluated whether the formula described below is satisfied.

[0667]

[0668] If the formula described above is satisfied, the process proceeds to step S204, and all images with priorities 1 to 6 are selectively sent.

[0669] On the other hand, if the formula described above is not satisfied, step S205 and subsequent steps are performed, as well as the process of excluding the lowest priority image (i.e., the left-side view with priority 6, Lv) from the transmission target is performed.

[0670] Subsequently, the total bit rate S(5) was evaluated when images with priorities 1 to 5 were selectively transmitted, excluding the left-side view with priority 6 (Lv) which had been set not to be transmitted. required Is it less than the estimated bit rate b? e That is, at step S203, it is evaluated whether the formula described below is satisfied.

[0671]

[0672] If the formula described above is satisfied, the process proceeds to step S204, and images with priorities 1 to 5 are selectively sent.

[0673] This process is repeated, and the removal of low-priority images from the transmission target is performed sequentially, thereby increasing the total bit rate s(M) in the case of selectively transmitting high-priority images. required Less than the estimated bit rate b e In this case, the process of sending M images is executed.

[0674] However, ultimately, the total bit rate S(1) is achieved when only one highest priority image is selectively sent. required Not less than the estimated bit rate b e In this case, no image is sent.

[0675] This process is equivalent to step S208 in the flowchart shown in Figure 14, that is, the process of evaluating "No" using the evaluation formula described below.

[0676]

[0677] And at step S209, all image processing is disabled.

[0678] In this way, by performing the processing according to the flowchart shown in Figure 14, only the bit rate that can be equal to or less than the currently available bit rate for data transmission (estimated bit rate b) is selected based on priority. e The image is transmitted at a bit rate of 40 and with a delay equal to or less than a predetermined value, and then transmitted to the remote assistance device 40.

[0679] That is, by performing a process that compares the bit rate required to send images selected in priority order with the estimated bit rate while removing low-priority images, it becomes possible to send only high-priority images to the remote assistance device 40 with a short delay (equal to or less than a predetermined delay amount).

[0680] As a result, the autonomous vehicle 10 can receive control information from the remote assistance device 40 without delay and perform safe driving.

[0681] Note that because images are sent selectively in order of priority during this process, low-priority images are not sent to the remote assistance device 40.

[0682] Refer to Figure 21 for a specific example.

[0683] The top left corner of Figure 21 shows the result of sorting the six images by priority and the bit rate b(j) allocated to each of the six images. required .

[0684] The top right corner shows the selection of one or more bit rates b(j) from the high-priority images to be allocated to each image. required The total bit rate under the given conditions.

[0685] For example, assuming that as a result of processing according to the flowchart shown in Figure 14, the total bit rate S(5) is evaluated in the case of selectively transmitting images with priorities 1 to 5. required Equal to or greater than the estimated bit rate be And it is evaluated as the total bit rate S(4) in the case of selectively sending images with priorities 1 to 4. required Less than the estimated bit rate b e That is, it satisfies the formula described below.

[0686]

[0687] In this case, the data processing unit 22 of the automatic driving control device 20 only sends images with priorities 1 to 4 to the remote control device 40.

[0688] When only images with priorities 1 to 4 that have been sent to the remote control device 40 are displayed on the display unit 43 of the remote control device 40 at this time, the displayed image becomes the image shown in the lower part of FIG21.

[0689] Right now,

[0690] Priority 1 = (Fv) Front view (image captured by the front-view camera)

[0691] Priority 2 = (Bv) Rear View (Image captured by the rear-view camera)

[0692] Priority 3 = (RBv) Right Rear View (Image captured by the right rear-view camera)

[0693] Priority 4 = (LBv) Left Rear View (Image captured by the left rear-view camera)

[0694] These images with priorities 1 to 4 can be displayed.

[0695] However,

[0696] Priority 5 = (Rv) Right-side view (image captured by the right-side camera)

[0697] Priority 6 = (Lv) Left side view (image captured by the left-side camera)

[0698] Images with priorities of 5 to 6 cannot be displayed.

[0699] In this state, the operator 50 at the remote assistance center 30 struggles to determine the precise control actions that should be taken. As a result, the correct control information cannot be sent to the autonomous vehicle 10.

[0700] To prevent such a situation, the data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 shown in FIG2 to generate and display a 3DCG image observed from the position and orientation of the camera installed on the autonomous vehicle 10.

[0701] A specific example is shown in Figure 22.

[0702] The example shown in Figure 22 is such that, in the display area of ​​the following two low-priority images not sent from the automatic driving control unit 20, i.e.,

[0703] Priority 5 = (Rv) Right-side view (image captured by the right-side camera)

[0704] Priority 6 = (Lv) Left side view (left-view camera captures image),

[0705] Generate and display 3DCG images observed from the positions and orientations of the two cameras.

[0706] The data processing unit 42 of the remote assistance device 40 further performs such processing, in which it analyzes metadata received from the autonomous driving control device 20, including information about surrounding objects (such as oncoming vehicles), and displays images of objects around the autonomous driving vehicle 10 on a 3DCG image.

[0707] By performing such processing, display data that more reliably reproduces the current surroundings of the autonomous vehicle 10 is generated and displayed on the display unit.

[0708] This processing allows the operator 50 to check the current surroundings of the autonomous vehicle 10 based on the camera-captured images, CG images, and object images displayed on the display unit 43, and to send optimal control information (autonomous driving control information) to the autonomous vehicle 10.

[0709] The autonomous driving control unit 20 generates metadata, including information about surrounding objects (such as oncoming vehicles) included in the unsent images, and sends the metadata to the remote assistance unit 40.

[0710] The details of this process will be explained below.

[0711] [6. Details regarding the processing of objects detected in images that have never been sent]

[0712] Next, the details of processing objects detected in images that have never been sent will be explained.

[0713] As illustrated earlier with reference to Figure 8, in addition to the communication path that transmits multiple video tracks via a single stream, a metadata communication path is also established between the autopilot control device 20 and the remote assistance device 40.

[0714] Metadata includes objects detected from camera-captured images and LiDAR sensor information, such as information about objects (like surrounding vehicles, pedestrians, and buildings).

[0715] As previously stated, the autonomous driving control device 20 performs the following communication control: it changes the metadata to be sent at any time according to the communication bandwidth (bit rate) available for metadata communication between the autonomous driving control device 20 and the remote assistance device 40.

[0716] For example, object coordinates, bounding box size, and labels indicating the object type are always sent to identify the location and size of objects (such as vehicles and pedestrians around the autonomous vehicle 10).

[0717] However, regarding large amounts of data (such as images of objects), if it is possible to ensure that the communication path bandwidth used for metadata communication is narrow, then processing such as stopping the transmission, sending large amounts of data as compressed data, and selectively sending data to the target based on the risk level of each object can be performed.

[0718] Note that the data processing unit 22 of the autonomous driving control unit 20 assesses the risk level of the object and sends the assessment result as metadata to the remote assistance device 40.

[0719] The flowchart shown in Figure 23 is a flowchart illustrating the processing sequence of communication control for large amounts of metadata (such as images of objects) performed by the autonomous driving control unit 20.

[0720] Specifically, this flowchart illustrates the details of the process for sending objects detected in an image that has never been sent.

[0721] Note that the flowchart shown in Figure 23 corresponds to a detailed flowchart of the processes at steps S121 to S125 in the flowchart described earlier with reference to Figure 10.

[0722] That is, at step S108 in the flowchart shown in Figure 9, the bit rate (estimated bit rate b) is evaluated as usable for data transmission. e Less than the minimum bit rate b min In this case, these processes are performed. Specifically, in the case of evaluating the estimated bit rate b... e Below the minimum bit rate b shown in Figure 12 min These processes are performed in the case of bandwidth region C.

[0723] The details of the processing at steps S301 to S309 in the flowchart shown in Figure 23 will be described in turn below.

[0724] (Step S301)

[0725] First, the estimated bit rate (b) is assessed as usable for data transmission. e Less than the minimum bit rate bmin In the case that the estimated bit rate b is evaluated... e Below the minimum bit rate b shown in Figure 12 min In the case of bandwidth region C, at step S301, the data processing unit 22 of the automatic driving control device 20 evaluates whether there are untransmitted images (disabled (disabled) videos), which are images excluded from the transmission target.

[0726] As illustrated earlier in the flowchart shown in Figure 14, at the bit rate available for data transmission (estimated bit rate b) e In cases where it is impossible to send all low-bitrate images captured by the cameras, low-priority images are excluded from the transmission target as unsent images.

[0727] At step S301, it is evaluated whether an image was not sent during the processing according to the flowchart shown in FIG14.

[0728] If no image is sent, step S302 and subsequent steps are not executed, and the process ends and returns to step S101. That is, a new estimated bit rate b is obtained. e And perform the operation with the newly estimated bit rate b obtained. e Appropriate measures should be taken.

[0729] On the other hand, if the assessment indicates that no image has been sent, the processing at step S302 and subsequent steps is performed. That is, the processing of extracting objects such as vehicles from the unsent images and sending the objects as metadata to the remote assistance device 40 is performed.

[0730] (Step S302)

[0731] If it is determined in step S301 that there is an unsent image, in step S302, the data processing unit 22 of the automatic driving control device 20 selects an image frame included in the unsent image.

[0732] (Step S303)

[0733] Next, in step S303, the data processing unit 22 performs image segmentation processing on one image frame included in the unsent image.

[0734] For example, image segmentation is semantic segmentation, and it involves dividing an image into pixel regions based on the type of object each pixel belongs to, using pixels as the unit of measurement. For instance, by setting different colors according to the types of various objects (such as vehicles, people, and trees) included in the image, information that can identify the object type of each pixel can be generated.

[0735] Refer to Figure 24 for a specific example of image segmentation processing.

[0736] Figure 24(a) shows an image frame captured by a camera mounted on the autonomous vehicle 10. The result of performing image segmentation processing on this image frame is the image segmentation result shown in Figure 24(b).

[0737] As shown in Figure 24(b), when performing image segmentation processing on the camera-captured image shown in Figure 24(a), different colors are set according to the object type to which each pixel in the camera-captured image belongs. For example, by setting different colors (such as red for vehicles, blue for people, and green for trees) according to the type of object included in the image, distinguishing information about the object type of each pixel region can be generated.

[0738] At step S303, image segmentation processing is performed on an image frame included in an unsent image, and data is generated in which objects included in the image can be distinguished.

[0739] (Step S304)

[0740] Next, at step S304, using the result of the image segmentation process at step S303, the data processing unit 22 performs a process to extract objects included in an image frame included in an unsent image.

[0741] This process corresponds to the process at step S121 in the flowchart shown in Figure 10.

[0742] Note that the target object type is predefined. For example, objects that affect the driving of the autonomous vehicle 10 (such as vehicles and people) are predefined as extraction targets, and the image segmentation processing results are used to extract objects corresponding to the target object type.

[0743] Refer to Figure 24 for a specific example of this process.

[0744] The object extraction result in (c) of Figure 24 shows the objects extracted from the camera-captured image in (a) using the image segmentation processing result shown in (b) of Figure 24.

[0745] In the example shown in Figure 24, the extracted objects include vehicles, pedestrians, bicycles, etc.

[0746] Note that this object extraction processing example is an example, and in addition to the example that only considers vehicles and people as extraction targets, such processing can also be performed, in which traffic lights, signs, poles, etc. on the road are set as extraction target objects and these objects are extracted.

[0747] (Step S305)

[0748] Next, at step S305, the data processing unit 22 calculates the risk level of each object extracted in step S304 and generates sorted data in which the extracted objects are arranged in descending order of risk.

[0749] This process corresponds to the process at step S122 in the flowchart shown in FIG10, which was described earlier. As illustrated by the process at step S122 in the flowchart shown in FIG10, for example, objects with a high risk of collision are assessed as high risk.

[0750] Additionally, in the image priority assessment process described earlier with reference to Figure 16, the risk level of objects in the image is calculated.

[0751] The calculation process for the risk level of each object calculated in step S305 is also performed in a similar manner to the calculation process for the risk level of each object to be used in the image priority calculation process.

[0752] That is, specifically, as explained earlier, for example, RSS-based (responsibility-sensitive security) processing can be used.

[0753] RSS is a technique for evaluating the appropriateness of a pair of objects detected from an image with an autonomous vehicle 10, which is the subject vehicle.

[0754] Alternatively, a technique using neural networks, as described earlier with reference to Figure 19, can be applied. The input data for the neural network consists of the type, size, direction of movement, and distance from the subject vehicle of the detected objects in each image, as well as network status, weather, and road conditions. The output is the risk level of the object.

[0755] At step S305, for example, these techniques are applied to calculate the risk level of each object extracted from the image at step S304, and to generate sorted data in which the extracted objects are arranged in descending order of risk.

[0756] For example, as shown in Figure 25(d), sorted data is generated in which the extracted objects are sorted in descending order of risk level from risk level 1 to risk level 4.

[0757] Note that, similar to the risk levels used for image priority calculation, the risk level corresponding to an object is output as the level corresponding to the predefined level definition.

[0758] As a classification of levels, for example, a classification can be used based on the analysis results of past accident data at the current location of the autonomous vehicle 10, as shown below.

[0759] Risk Level 1 = Serious Accident

[0760] Risk Level 2 = Moderate Accidents Expected

[0761] Risk Level 3 = Minor Accident

[0762] Risk Level 4 = No harm reported

[0763] (Step S306)

[0764] Next, in step S306, the data processing unit 22 of the automatic driving control device 20 calculates the bit rate that can be used for metadata transmission.

[0765] As illustrated earlier with reference to Figure 8, in addition to the communication path that transmits multiple video tracks via a single stream, a metadata communication path is also established between the autopilot control device 20 and the remote assistance device 40.

[0766] The bandwidth available for metadata communication also fluctuates with network conditions, similar to that of image communication paths. Specifically, the bandwidth available for metadata communication paths is related to the estimated bit rate b illustrated earlier with reference to Figure 12. e Similarly, it fluctuates at any time.

[0767] At step S306, the communication network monitoring unit 112 of the data processing unit 22 of the automatic driving control device 20 shown in FIG5 calculates the bit rate available for metadata transmission.

[0768] (Step S307)

[0769] Next, at step S307, the data processing unit 22 of the automatic driving control device 20 compresses each extraction object using a different compression mode (using different compression algorithms) based on the risk level of each extraction object assessed at step S305.

[0770] Refer to Figure 26 for specific examples of different compression processes performed based on the risk level of the object.

[0771] As shown in Figure 26, objects with risk level 1 (the highest risk level) are compressed using the compression algorithm c corresponding to risk level 1. RL1 The compression process converts it into C. RL1 Compressed images.

[0772] Objects with risk level 2, which is the second highest risk level, are compressed using the compression algorithm c corresponding to risk level 2. RL2 The compression process converts it into C. RL2 Compressed images.

[0773] Objects with risk level 3, which is the second highest risk level, are compressed using the compression algorithm c corresponding to risk level 3.RL3 The compression process converts it into C. RL3 Compressed images.

[0774] Objects with risk level 4, which is the lowest risk level, are compressed using the compression algorithm c corresponding to risk level 4. RL4 The compression process converts it into C. RL4 Compressed images.

[0775] Note that c RL1 c RL2 c RL3 and c RL4 These represent the compression rates from risk level 1 to risk level 4, and have the following relationship with the compression rates corresponding to each risk level.

[0776]

[0777] That is, the higher the risk level, the lower the compression ratio; and the lower the risk level, the higher the compression ratio. This means that for objects with a high risk level, a compressed image is generated that closely approximates the original object image by reducing the compression ratio. On the other hand, for objects with a low risk level, a compressed image is generated that yields an abstracted reconstructed image by increasing the compression ratio.

[0778] Note that the flowcharts shown in Figures 27 and 28 will be used later to illustrate specific examples of the compression and filtering processes (sending object image selection processes) performed in steps S306 to S308, which correspond to the risk level of the object.

[0779] (Step S308)

[0780] Next, in step S308, the data processing unit 22 of the automatic driving control device 20 performs a filtering process on the extracted and compressed object image data.

[0781] The filtering process is the process of selecting compressed object image data to be sent, and it is a process of classifying compressed object image data that should be considered as the target of transmission and compressed object image data that should not be sent.

[0782] It should be noted that filtering should be performed when the bit rate required for sending data to the object to be compressed is greater than the available bandwidth.

[0783] The details of the process will be mentioned later.

[0784] (Step S309)

[0785] Next, in step S309, the data processing unit 22 of the automatic driving control device 20 will send the compressed object image data of the target to the remote assistance device 40 as metadata, which is the result of the filtering process in step S308.

[0786] By executing the process according to the flowchart shown in Figure 23, when an object such as an oncoming vehicle is detected in an image that is set to not send (video track disabled), information about the object is sent as metadata to the remote assistance device 40.

[0787] Next, referring to the flowcharts shown in Figures 27 and 28, we will describe the detailed sequence of processes at steps S306 to S308 in the flowchart in Figure 23, namely, compression processing and filtering processing (sending object image selection processing) corresponding to the risk level of the object.

[0788] Note that the definitions of the various parameters used in the flowcharts shown in Figures 27 and 28 are as follows, as described in the table shown in Figure 29.

[0789] X required : Total bit rate required to send images of all extracted objects

[0790] x a : The bandwidth (bit rate) that can be used to send the image of the object (object image).

[0791] R: The risk level of the object, R∈[1, 2, 3, 4]

[0792] X(1) required Total bitrate required to send images of extracted objects at risk level 1.

[0793] X(2) required Total bitrate required to send images of extracted objects for risk levels 1 and 2.

[0794] X(3) required Total bitrate required to send images of extracted objects for risk levels 1, 2, and 3.

[0795] X(4) required : Total bit rate required to send all extracted object images (X(4)) required =X required )

[0796] The following sections describe the processing at each step shown in the flowcharts of each of Figures 27 to 28.

[0797] (Step S311)

[0798] First, in step S311, the data processing unit 22 of the automatic driving control device 20 performs processing to calculate the following two types of bit rates.

[0799] (1) The bit rate required to send all extracted objects (bit rate X for sending all objects) required )

[0800] (2) The bit rate of the bandwidth available for sending object images in the metadata communication path (bit rate x for sending object images) a )

[0801] Note that the extracted object is the object extracted at step S304 in the flowchart shown in Figure 23 (= step S121 in the flowchart shown in Figure 10), and is the object extracted from an image that has never been sent.

[0802] The types of objects to be extracted (vehicles, pedestrians, etc.) are predefined.

[0803] In addition, the bit rate X used to send all objects required (The bit rate required to send all extracted objects) is determined to be the bit rate that enables all extracted objects to be sent to the remote assistance device 40 without delay.

[0804] (Step S312)

[0805] Next, in step S312, the data processing unit 22 of the automatic driving control device 20 performs a process that compares the following two bit rates calculated in step S311, namely,

[0806] Bit rate X used to send all objects required ,as well as

[0807] Bit rate x available for object image transmission a .

[0808] Evaluate whether the formula described below is satisfied.

[0809]

[0810] That is, at the bit rate X evaluated for sending all objects required The bit rate x that can be used to send object images is equal to or less than a In this case, the process proceeds to step S313.

[0811] On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S314.

[0812] (Step S313)

[0813] If the evaluation at step S312 satisfies the evaluation formula described below, then...

[0814]

[0815] The process proceeds to step S313.

[0816] The evaluation formula described above implies that the bit rate X used to send all objects is... required The bit rate x that can be used to send object images is equal to or less than a .

[0817] That is, this means that all object images can be sent with a short delay (equal to or less than a predetermined delay) within the currently available communication bandwidth (bit rate).

[0818] In this case, at step S313, the data processing unit 22 of the automatic driving control device 20 sends the entire object image data of the object extracted from the unsent image to the remote assistance device 40 as is, without compression.

[0819] Note that after the processing at step S313 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0820] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0821] (Step S314)

[0822] On the other hand, if the evaluation at step S312 does not satisfy the evaluation formula described below, then...

[0823]

[0824] The process proceeds to step S314.

[0825] An evaluation formula that does not meet the above description means that the bit rate X used to send all objects is... required Not equal to or less than the bit rate x available for sending object images a .

[0826] This means that it is difficult to send all object images with a short delay (equal to or less than a predetermined delay) within the currently available communication bandwidth (bit rate).

[0827] That is, this means that, assuming all extracted object images are sent as is, the current bit rate x available for sending object image data is... a This may cause delays, etc.

[0828] In this case, the data processing unit 22 of the automatic driving control device 20 performs the processing in step S314 and subsequent steps.

[0829] First, in step S314, the data processing unit 22 of the automatic driving control device 20 sorts the objects in ascending order of risk level (1 to 4) and uses a compression mode corresponding to the risk level (using compression algorithm c). RL1 To c RL4 Use this to compress and extract objects.

[0830] Compression mode (compression algorithm c) RL1 To c RL4 (This is the compression algorithm corresponding to the risk level, as explained earlier with reference to Figure 26.)

[0831] At step S314, firstly, only the object image data of risk level 4, which is the lowest risk level, is compressed, and the process proceeds to step S315.

[0832] (Step S315)

[0833] Next, in step S315, the data processing unit 22 of the automatic driving control device 20 performs a process to compare two bit rates: the total bit rate for sending the compressed object image data of the low-risk object compressed in step S314 and the uncompressed high-risk object data not compressed in step S314, that is,

[0834] Bit rate X used to send all objects required ,as well as

[0835] Bit rate available for communication x a .

[0836] Evaluate whether the formula described below is satisfied.

[0837]

[0838] That is, at the bit rate X evaluated for sending all objects required Equal to or less than the bit rate x available for communication a In this case, the process proceeds to step S317.

[0839] On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S318.

[0840] (Step S317)

[0841] If the evaluation at step S316 satisfies the evaluation formula described below, then...

[0842]

[0843] The process proceeds to step S317.

[0844] Satisfying the evaluation formula described above means that the total bit rate X used to send all objects is the compressed object image data of the low-risk objects compressed in step S314 and the uncompressed object data of the high-risk objects not compressed in step S314. required Equal to or less than the bit rate x available for communication a .

[0845] That is, this means that all object images (at least some of which are compressed object images) can be sent with a short delay (equal to or less than a predetermined delay) under the current available communication bandwidth (bit rate).

[0846] In this case, at step S317, the data processing unit 22 of the automatic driving control device 20 sends all object images (at least some of which are compressed object images) extracted from the never-sent images to the remote assistance device 40.

[0847] Note that after the processing at step S317 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0848] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0849] (Step S318)

[0850] On the other hand, if the evaluation at step S316 fails to satisfy the evaluation formula described below,...

[0851]

[0852] The process proceeds to step S318.

[0853] The evaluation formula described above does not satisfy the requirement that the bit rate X used to send the object image (where at least some of it is a compressed object image) extracted from the never-sent image is not satisfied. required It is not equal to or less than the bit rate xa that can be used for communication.

[0854] This means that it is difficult to send all object images (at least some of which are compressed object images) with a short delay (equal to or less than a predetermined delay) within the currently available communication bandwidth (bit rate).

[0855] That is, this means that when sending the entire image data of the extracted object (of which at least a portion is a compressed object image), the currently available bitrate x a This may cause delays, etc.

[0856] In this case, the data processing unit 22 of the automatic driving control device 20 performs the processing in step S318 and subsequent steps.

[0857] In this case, at step S318, the data processing unit 22 of the automatic driving control device 20 evaluates whether there is an uncompressed object image that has not undergone compression processing.

[0858] In step S314, as described earlier, the compression is performed in ascending order of the object's risk level (1 to 4) using a compression mode corresponding to the risk level (utilizing compression algorithm c). RL1 To c RL4 This is used to compress and extract object processing.

[0859] Initially, the processing of objects with compression risk level 4 is performed, and the processing at steps S315 and subsequent steps is executed. At this point in time, if the evaluation formula at step S316 is satisfied, i.e., the formula described below, is true.

[0860]

[0861] The process proceeds to step S317, and the processing of sending compressed image data of objects with risk level 4 and uncompressed image data of objects with risk levels 1 to 3 is performed.

[0862] However, after processing objects with compression risk level 4, if the evaluation formula of step S316, i.e., the formula described below, is not satisfied,

[0863]

[0864] The evaluation at step S316 is "No", and the evaluation of "Does uncompressed data exist?" at step S318 is "Yes".

[0865] In this case, the process returns to step S314, and then the processing of objects with compression risk level 3 is performed.

[0866] In this way, at step S314, the processing of compressing high-risk level objects is performed sequentially.

[0867] Finally, at step S314, the highest risk level object, i.e., risk level 1 object, is compressed, and if the evaluation formula of step S316, i.e., the formula described below, is not satisfied at this point in time,

[0868]

[0869] The assessment concluded that even if all objects were converted into compressed image data, it would be difficult to send all objects of risk levels 1 to 4 with short latency using the currently available communication bandwidth (bit rate).

[0870] In this case, the evaluation at step S316 is "No", and the evaluation of "Does uncompressed data exist?" at step S318 is also "No".

[0871] In this case, the processing at step S321 and subsequent steps is performed.

[0872] In step S321 and subsequent steps, a process is performed to select objects to be sent from the automatic driving control device 20 to the remote assistance device 40. That is, instead of setting all extracted objects as sending targets, a selection process (filtering) is performed on some of the extracted objects that are set as images not to be sent.

[0873] (Step S321)

[0874] If the evaluation at step S318 is "No", that is, if it is evaluated that even if all objects of risk level 1 to 4 are converted into compressed image data, it is difficult to transmit them with short latency under the currently available communication bandwidth (bit rate), then the processing at step S321 is performed.

[0875] At step S321, the data processing unit of the automatic driving control device 20 calculates the required bit rate for sending each of the following combinations of compressed object images.

[0876] * The bit rate X(1) required to send compressed object image data of risk level 1. required

[0877] * The bit rate X(2) required to send compressed object image data for risk levels 1 and 2. required

[0878] * The bit rate X(3) required to send compressed object image data of risk levels 1 to 3 required

[0879] Calculate these individual bit rates.

[0880] (Step S322)

[0881] Next, in step S322, the data processing unit of the automatic driving control device 20 performs a comparison of the two bit rates.

[0882] * The bit rate X(3) required for sending compressed object image data of risk levels 1 to 3, calculated at step S321. required The bit rate required to send compressed object image data of risk levels 1 to 3, and

[0883] * Available bit rate x a It is the bit rate that can be used for current communication.

[0884] That is, assess whether the formula described below is satisfied.

[0885]

[0886] The evaluation assumes that the formula is satisfied, namely, the bit rate X(3) required for sending compressed object image data of risk levels 1 to 3. required Equal to or less than the available bit rate x a In this case, the process proceeds to step S323.

[0887] On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S324.

[0888] (Step S323)

[0889] If the evaluation at step S322 is satisfied with the formula described below,

[0890]

[0891] That is, the bit rate X(3) required to send compressed object image data of risk levels 1 to 3. required Equal to or less than the available bit rate x a In this case, the process proceeds to step S323.

[0892] In this case, at step S323, the data processing unit 22 of the automatic driving control device 20 only sends objects with risk levels 1 to 3 from the extracted objects as compressed data (compressed image data). Note that objects with risk level 4 are set as objects not sent.

[0893] Note that after the processing at step S323 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0894] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0895] (Step S324)

[0896] On the other hand, if the evaluation at step S322 determines that the formula described below is not satisfied,...

[0897]

[0898] That is, the bit rate X(3) required for transmitting compressed object image data of risk levels 1 to 3 as assessed. required Not equal to or less than the available bit rate x a In this case, the process proceeds to step S324.

[0899] In step S324, the data processing unit of the automatic driving control device 20 performs a comparison of two bit rates.

[0900] * The bit rate X(2) required for sending compressed object image data of risk levels 1 to 2, calculated at step S321. required The bit rate required to send compressed object image data of risk levels 1 to 2, and

[0901] * Available bit rate x a , which is the bit rate available for current communication.

[0902] That is, assess whether the formula described below is satisfied.

[0903]

[0904] The evaluation assumes that the formula is satisfied, namely, the bit rate X(2) required for sending compressed object image data of risk levels 1 to 2. required Equal to or less than the available bit rate x a In this case, the process proceeds to step S325.

[0905] On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S326.

[0906] (Step S325)

[0907] If the evaluation at step S324 is satisfied with the formula described below, then...

[0908]

[0909] That is, the bit rate X(2) required for transmitting compressed object image data of risk levels 1 to 2 as assessed. required Equal to or less than the available bit rate x a In this case, the process proceeds to step S325.

[0910] In this case, at step S325, the data processing unit 22 of the automatic driving control device 20 only sends objects with risk levels 1 to 2 from the extracted objects as compressed data (compressed image data). Note that objects with risk levels 3 and 4 are set as objects not sent.

[0911] Note that after the processing at step S325 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0912] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0913] (Step S326)

[0914] On the other hand, if the evaluation at step S324 determines that the formula described below is not satisfied,...

[0915]

[0916] That is, the bit rate X(2) required for transmitting compressed object image data of risk levels 1 to 2 as assessed. required Not equal to or less than the available bit rate x a In this case, the process proceeds to step S326.

[0917] In step S326, the data processing unit of the automatic driving control device 20 performs a comparison of two bit rates.

[0918] * The bit rate X(1) for transmitting compressed object image data of risk level 1, calculated at step S321. required The bit rate required to send compressed object image data of risk level 1, and

[0919] * Available bit rate x a , which is the bit rate available for current communication.

[0920] That is, assess whether the formula described below is satisfied.

[0921]

[0922] The evaluation assumes that the formula is satisfied, namely, the bit rate X(1) required for transmitting compressed object image data of risk level 1. required Equal to or less than the available bit rate x a In this case, the process proceeds to step S327.

[0923] On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S328.

[0924] (Step S327)

[0925] If the evaluation at step S326 is satisfied with the formula described below,

[0926]

[0927] That is, the bit rate X(1) required for transmitting compressed object image data of risk level 1 as assessed. required Equal to or less than the available bit rate x a In this case, the process proceeds to step S327.

[0928] In this case, at step S327, the data processing unit 22 of the automatic driving control device 20 only sends objects of risk level 1 from the extracted objects as compressed data (compressed image data). Note that objects of risk levels 2 to 4 are set as objects not sent.

[0929] Note that after the processing at step S327 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0930] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0931] (Step S328)

[0932] On the other hand, if the evaluation at step S326 determines that the formula described below is not satisfied...

[0933]

[0934] That is, the bit rate X(1) required for transmitting compressed object image data of risk level 1 as assessed. required Not equal to or less than the available bit rate x aIn this case, the process proceeds to step S328.

[0935] In this case, at step S328, the data processing unit 22 of the autonomous driving control device 20 notifies the autonomous driving control device 20 of the autonomous driving vehicle 10 that remote assistance is not possible.

[0936] That is, the data processing unit 22 notified that remote assistance could not be provided because it was impossible to check the objects in the unsent images.

[0937] Note that, for example, the autonomous driving control unit 20 of the autonomous vehicle 10, which has received such a notification, responds by performing an emergency stop procedure for the autonomous vehicle.

[0938] Note that after the processing at step S328 is completed, the process returns to step S101 in the flowchart shown in Figure 9, and the processing at step S101 and subsequent steps is repeated.

[0939] That is, to obtain a new estimated bit rate b e And repeat the process with the newly obtained estimated bit rate b. e Appropriate measures should be taken.

[0940] As can be understood from the flowcharts shown in Figures 27 and 28, the data processing unit 22 of the autonomous driving control device 20 of the autonomous vehicle 10 performs transmission data control processing to reliably transmit information about objects detected in images that have never been transmitted (video track disabled) to the remote assistance device 40.

[0941] Specifically, as a means to reliably communicate with short latency within the currently available communication bandwidth (bit rate x) a The data processing unit 22 performs compression processing and data selection (filtering) processing corresponding to the risk level of the object, and sends the information to the remote assistance device 40.

[0942] [7. Regarding processing performed by remote assistance devices]

[0943] Next, the processing performed by the remote assistance device will be explained.

[0944] The remote assistance device 40 receives camera-captured images and metadata, including object images and object location information, sent by the autonomous driving control device 20 of the autonomous vehicle 10. Using this received information, the display unit 42 of the remote assistance device 40 displays an image representing the surroundings of the autonomous vehicle.

[0945] Figure 30 is a diagram showing an example of an image displayed on the display unit 22 by the remote assistance device 40 using data transmitted from the autonomous driving control device 20 of the autonomous vehicle 10.

[0946] In the example shown in Figure 30, the following four camera-captured images are sent from the autonomous driving control unit 20 to the remote assistance unit 40 according to image priority.

[0947] Priority 1 = (Fv) Front view (image captured by the front-view camera)

[0948] Priority 2 = (Bv) Rear View (Image captured by the rear-view camera)

[0949] Priority 3 = (RBv) Right Rear View (Image captured by the right rear-view camera)

[0950] Priority 4 = (LBv) Left Rear View (Image captured by the left rear-view camera)

[0951] The automatic driving control unit 20 does not send the following two low-priority images, namely,

[0952] Priority 5 = (Rv) Right-side view (image captured by the right-side camera)

[0953] Priority 6 = (Lv) Left view (left view camera captures image).

[0954] The remote assistance device 40 can output four high-priority images sent from the autopilot control device 20 to the display unit 43, and cause the display unit 43 to display these images.

[0955] However, the remote assistance device 40 is unable to make the display unit 43 display images captured by the two low-priority cameras that were not sent from the autopilot control device 20.

[0956] For the image display area of ​​two low-priority images not sent from the autonomous driving control device 20, the data processing unit 42 of the remote assistance device 40 performs the process of generating and displaying 3DCG images observed from the positions and orientations of the two cameras.

[0957] In addition, the data processing unit 42 of the remote assistance device 40 performs processing in which it analyzes metadata received from the autonomous driving control device 20, including information about surrounding objects (such as oncoming vehicles), and displays images of objects around the autonomous driving vehicle 10 on a 3DCG image.

[0958] By performing such processing, display data that more reliably reproduces the current surroundings of the autonomous vehicle 10 is generated and displayed on the display unit.

[0959] This processing allows the operator 50 to check the current surroundings of the autonomous vehicle 10 based on the camera-captured images, CG images, and object images displayed on the display unit 43, and to send optimal control information (autonomous driving control information) to the autonomous vehicle 10.

[0960] Note that regarding images that were not received from the autopilot control unit 20, the data processing unit 42 of the remote assistance unit 40 performs the process of generating and displaying 3DCG images observed from the position and orientation of the image capture camera.

[0961] Furthermore, in cases where the received image is unclear, the data processing unit 42 of the remote assistance device 40 performs the following processing: generating a 3DCG image observed from the position and orientation of the image capture camera that can receive the image from the automatic driving control device 20; generating a composite image obtained by compositing (overlaying) the generated 3DCG image and the received image; and displaying the composite image.

[0962] The communication path between the automatic driving control device 20 and the remote assistance device 40 may experience sudden fluctuations in available bandwidth, congestion, etc. In such cases, communication packet loss, delays, etc. occur; as a result, in some situations, the data received by the remote assistance device 40 becomes an unclear image. In such cases, the data processing unit 42 of the remote assistance device 40 performs the process of generating a composite image obtained by combining (overlaying) the 3DCG image and the received image, and displays the composite image.

[0963] Note that the 3D map stored in the 3D map storage unit 45 of the remote assistance device 40 shown in Figure 2 is used to generate the 3DCG image. Note that, as mentioned earlier, the 3D map can be obtained from an external server.

[0964] For example, a 3D map is a map that has been created in advance using instruments such as LiDAR.

[0965] As a method for generating 3DCG images from a specific viewpoint (e.g., a specific camera position) using 3D maps, one approach is to utilize a VPS (Vision Positioning System). In a VPS, a database recording the relationships between image feature points and their positions is used to enable the generation of 3DCG images from a specific viewpoint (e.g., a specific camera position).

[0966] Techniques such as Structure from Motion (SfM) used to estimate 3D shape using feature points of an image can also be used to obtain the entire space as a point cloud.

[0967] When generating a 3DCG image using a VPS, firstly, camera images are acquired from the vehicle and compared with a database. If an image cannot be acquired, the image immediately preceding the previously acquired frame is used. Based on this image, the position and orientation of the autonomous vehicle 10 are acquired, and point cloud data of surrounding roads, buildings, etc., is further acquired. Next, the polygonal shapes of various objects are reconstructed based on the acquired point cloud data, and then unwanted parts are removed. Through this process, a 3DCG image observed from a specific viewpoint (i.e., a specific camera position) can be created. Note that the point cloud data can be acquired from an external server or stored in the remote assistance device 40. Furthermore, the process of creating a 3DCG image observed from a specific camera position using a 3D map requires polygonal shape reconstruction and unwanted part removal, and these processes can be performed independently by the data processing unit 42 of the remote assistance device 40 or in collaboration with an external data processing server.

[0968] [8. Details regarding the processing of composite (overlay) camera-captured images and 3DCG images performed by a remote assistance device]

[0969] Next, the details of the processing of composite (overlay) camera captured images and 3DCG images performed by the data processing unit 42 of the remote assistance device 40 will be explained.

[0970] As explained earlier, the data processing unit 42 of the remote assistance device 40 not only performs the process of generating a 3DCG image observed from the position and orientation of the image capture camera that cannot receive images from the autopilot control device 20 and displaying the 3DCG image, but also performs the process of generating a 3DCG image observed from the position and orientation of the image capture camera that can receive images, synthesizing (overlaying) the generated 3DCG image and the received image, and displaying the synthesized image on the display unit 43 when the received image is unclear or in other cases.

[0971] In the processing of compositing 3DCG images and the received image, the data processing unit 42 of the remote assistance device 40 performs processing that uses the opacity of the 3DCG image as a parameter for adjusting the compositing ratio of the 3DCG image and the received image.

[0972] Referring to Figure 31 and subsequent figures, the 3DCG image opacity is explained as an adjustment parameter for the synthesis ratio of the 3DCG image and the received image.

[0973] In Figure 31, the following images are shown in sequence, starting from the top row.

[0974] (a) 3DCG image

[0975] (b) Camera captures images

[0976] (c) Composite (overlay) image (= Display output image)

[0977] (a) The 3DCG image is a 3DCG image generated by the map control unit 121 of the data processing unit 42 of the remote assistance device 40, and is a 3DCG image generated using a 3D map stored in the 3D map storage unit 45. This 3DCG image is a 3DCG image viewed from the same camera position and orientation as the image captured by the camera in (b).

[0978] (b) The camera-captured image is a camera-captured image received from the autonomous driving control unit 20 of the autonomous vehicle 10. (b) The camera-captured image is an image captured using a camera mounted on the autonomous vehicle 10.

[0979] (c) The composite (overlay) image (=display output image) is an image generated by the display control unit 122 of the data processing unit 42 of the remote assistance device 40. The display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs a process (overlay process) to combine (a) a 3DCG image and (b) a camera-captured image to generate a composite image. The generated composite image is output to the display unit 43 of the remote assistance device 40 and observed by the operator 50.

[0980] Regarding the 3DCG image opacity, which is an adjustment parameter for the composite ratio of the 3DCG image and the received image, Figure 31 shows an example of composite processing set according to two different parameters.

[0981] An example of compositing with the 3DCG image opacity set to 0, and

[0982] An example of compositing processing with the 3DCG image opacity set to 1.

[0983] An example of compositing a 3DCG image with an opacity of 0 is an example in which the 3DCG image is set to be a completely transparent image, essentially no 3DCG image is used, and only the "(b) camera captured image" received from the autonomous vehicle 10 is used to generate the composite image.

[0984] The compositing process in which the opacity of the 3DCG image is set to 0 is an example of compositing process performed by the data processing unit 42 of the remote assistance device 40 after it has been evaluated that the sharpness of the "(b) camera-captured image" received from the autonomous vehicle 10 is sufficient.

[0985] On the other hand, an example of compositing a 3DCG image with an opacity of 1 is an example in which the 3DCG image is made completely opaque, that is, only the 3DCG image generated from the 3D map is used, and the "(b) camera captured image" received from the autonomous vehicle 10 is not used at all to generate the composite image.

[0986] The compositing process in which the opacity of the 3DCG image is set to 1 is an example of compositing process performed by the data processing unit 42 of the remote assistance device 40 when it does not receive “(b) camera captured image” from the autonomous vehicle 10.

[0987] Alternatively, similar processing may be performed if the image captured by camera (b) received from autonomous vehicle 10 is obviously unclear.

[0988] The 3DCG image opacity, which serves as an adjustment parameter for the composite ratio of the 3DCG image and the received image, can be adjusted as needed between 0 and 1. The display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs image compositing processing while controlling the value of the 3DCG image opacity based on the sharpness of the image captured by the camera (b) received from the autonomous vehicle 10.

[0989] Figure 32 shows an example of image synthesis processing based on the following three different types of parameter settings.

[0990] Figure 32 shows an example of the synthesis process set according to the following three different parameters:

[0991] An example of compositing processing with the 3DCG image opacity set to 0.25;

[0992] An example of compositing with the 3DCG image opacity set to 0.5; and

[0993] An example of compositing with the 3DCG image opacity set to 0.75.

[0994] As the resolution of the "(b) camera-captured image" received by the data processing unit 42 of the remote assistance device 40 from the autonomous vehicle 10 decreases, the value set for the 3DCG image opacity is increased. That is, the output ratio of the 3DCG image in the composite image is set higher.

[0995] In the example shown in Figure 32, the sharpness of the “(b) camera captured image” received from the autonomous vehicle 10 is highest at the left end (3DCG image opacity = 0.25), i.e., sharp, and lowest at the right end (3DCG image opacity = 0.75), i.e., blurry.

[0996] In the three composite processing examples shown in Figure 32, the composite ratio of the 3DCG image in the composite image is the highest in the composite processing example at the right end (3DCG image opacity = 0.75), and the composite ratio of the 3DCG image in the composite image is the lowest in the composite processing example at the left end (3DCG image opacity = 0.25).

[0997] In this way, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs image compositing processing while controlling the value of the 3DCG image opacity based on the sharpness of the image captured by the camera (b) received from the autonomous vehicle 10.

[0998] Referring to Figure 33, an example of 3DCG image opacity control processing performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40 is illustrated.

[0999] The example shown in Figure 33 is a processing example of determining the value of 3DCG image opacity based on the transmission bit rate assigned to each camera-captured image received from the autonomous vehicle 10.

[1000] Figure 33 shows five examples of opacity toggling processing.

[1001] 3DCG image opacity = 0 (= completely transparent (only the image captured by the camera is displayed))

[1002] 3DCG image opacity = 0.25

[1003] 3DCG image opacity = 0.5

[1004] 3DCG image opacity = 0.75

[1005] 3DCG image opacity = 1 (= completely opaque (only 3DCG image is displayed))

[1006] The example shown in Figure 33 illustrates the transmission bit rate (b(j)) allocated to a camera-captured image (with image priority j (image priority index = j)) received from the autonomous vehicle 10. allocated An example of processing to determine the value of opacity in a 3DCG image.

[1007] Specifically, the transmission bit rate (b(j)) allocated to the camera-captured images received from the autonomous vehicle 10... allocated As the value of increases, the opacity of the 3DCG image decreases, and the output ratio of the camera-captured image in the composite image to be displayed on the display unit 43 of the remote assistance device 40 increases. On the other hand, in the processing example, the execution control causes the transmission bit rate (b(j)) allocated to the camera-captured image to increase. allocated The opacity of the 3DCG image decreases, the opacity of the 3DCG image increases, and the output ratio of the 3DCG image in the composite image to be displayed increases.

[1008] When receiving a camera-captured image (image priority index = j) from the autonomous vehicle 10, the data processing unit 42 of the remote assistance device 40 allocates a transmission bit rate (b(j)) to each camera-captured image. allocated It is received as metadata or image-related attribute information, and this information is used to determine the opacity value (0 to 1) of the 3DCG image to be composited with the received camera-captured image (image priority index = j).

[1009] Figure 33 shows 3DCG images with five different opacity values. That is, 3DCG images are those whose opacity is set to 0, 0.25, 0.5, 0.75, and 1.

[1010] On the right side of the five 3DCG images shown in Figure 33, the transmit bit rate (b(j)) allocated to the image is indicated. allocated The vertical axis (transmit bit rate axis) of the bit rate axis. The lower end of the transmit bit rate axis is 0, that is,

[1011] b(j) allocated =0.

[1012] Additionally, the threshold for switching the opacity of the 3DCG images is shown at each boundary of the five 3DCG images.

[1013] The data processing unit 42 of the remote assistance device 40 allocates a transmission bit rate (b(j)) to each camera-captured image received from the autonomous vehicle 10. allocatedThe image is compared with each threshold to determine the opacity value of the 3DCG image to be superimposed on the camera-captured image, the image compositing process is performed, and the composite image is displayed on the display unit 43.

[1014] Note that the threshold setting example shown in Figure 33 is an example. More thresholds can be set, and more detailed control over the 3DCG image opacity can be performed.

[1015] The setting at the lower end of the transmit bit rate axis shown on the right side of the 3DCG image, that is,

[1016] b(j) allocated =0

[1017] A value of 0 indicates a transmission bit rate. This is equivalent to no images captured by the camera being transmitted from the autonomous vehicle 10; that is, the corresponding image is an untransmitted image.

[1018] In b(j) allocated When the value of the transmission bit rate is 0, that is, when the value of the transmission bit rate is 0, the remote assistance device 40 cannot receive the camera-captured image from the autonomous vehicle 10, and therefore the data processing unit 42 of the remote assistance device 40 performs the process of setting the opacity of the 3DCG image to 1 and displaying only the 3DCG image.

[1019] Not only in cases where the camera cannot capture images, but also in cases where the camera can capture images, if the received image has significantly low clarity or significant delay, the data processing unit 42 of the remote assistance device 40 also performs the process of setting the opacity of the 3DCG image to 1 and displaying only the 3DCG image.

[1020] As a determining indicator used for this, the transmission bit rate (b(j)) allocated to the image is used. allocated ).

[1021] In the example shown in Figure 33, at the transmission bit rate (b(j)) allocated If the condition falls within the following range, that is, if the conditions described below are met,

[1022]

[1023] The data processing unit 42 of the remote assistance device 40 performs the process of setting the opacity of the 3DCG image to 1 and displaying only the 3DCG image.

[1024] Notice,

[1025] b(j)required It is the transmission bit rate required to transmit an image with image priority j with a delay equal to or less than a predetermined value, and

[1026] b buffer It is the bit rate used to prevent image freezing during display switching between images captured using a camera and 3DCG images captured without a camera, and it is also the bit rate used to ensure the time for receiving image buffer processing.

[1027] Additionally, at the transmission bit rate (b(j)) allocated Under the following conditions, that is, when the following conditions are met,

[1028]

[1029] The data processing unit 42 of the remote assistance device 40 performs the following processing: generating a composite image by overlaying a 3DCG image with the opacity of the 3DCG image set to 0.75 onto a camera-captured image with image priority j, and displaying the composite image.

[1030] Note that b(j) overlayed It is the minimum bit rate required to use the image captured by the camera with image priority j as a composite image to be output to the display of the remote assistance device 40, and it is a predefined value.

[1031] Additionally, at the transmission bit rate (b(j)) allocated Under the following conditions, that is, when the following conditions are met,

[1032]

[1033] The data processing unit 42 of the remote assistance device 40 performs the following processing: generating a composite image by overlaying a 3DCG image with the opacity of the 3DCG image set to 0.5 onto a camera-captured image with image priority j, and displaying the composite image.

[1034] Additionally, at the transmission bit rate (b(j)) allocated Under the following conditions, that is, when the following conditions are met,

[1035]

[1036] The data processing unit 42 of the remote assistance device 40 performs the following processing: generating a composite image by overlaying a 3DCG image with the opacity of the 3DCG image set to 0.25 onto a camera-captured image with image priority j, and displaying the composite image.

[1037] Furthermore, at the transmission bit rate (b(j)) allocated Under the following conditions, that is, when the following conditions are met,

[1038]

[1039] The data processing unit 42 of the remote assistance device 40 performs the process of setting the opacity of the 3DCG image to 0 and displaying the camera-captured image with image priority j as is.

[1040] In this manner, the data processing unit 42 of the remote assistance device 40 performs the following processing: based on the transmission bit rate (b(j)) allocated to the camera-captured images received from the autonomous vehicle 10. allocated The value of the 3DCG image opacity is determined by the value of the 3DCG image opacity, a composite image is generated by overlaying the 3DCG image with the determined opacity onto the image captured by the camera received from the autonomous vehicle 10, and the composite image is output to the display unit 43.

[1041] Control is performed so that the transmission bit rate (b(j)) allocated to the camera for capturing images is adjusted accordingly. allocated The opacity of the 3DCG image is increased while the output ratio of the camera-captured image received from the autonomous vehicle 10 in the image to be displayed on the display unit 43 is increased, and the transmission bit rate (b(j)) is increased. allocated This reduces and increases the opacity of the 3DCG image and increases the output ratio of the 3DCG image in the displayed image.

[1042] The 3DCG image opacity control sequence executed by the data processing unit 42 of the remote assistance device 40, as illustrated in Figures 34 and 35, will be explained with reference to the flowcharts shown in Figures 33.

[1043] The parameters described in the flowcharts shown in Figures 34 and 35 are defined as follows, as shown in the table in Figure 36.

[1044] j: Index of the image list sorted in descending order of priority by the images sent by the autonomous vehicle.

[1045] b(j)allocated : The transmission bit rate assigned to the image with the j-th image priority.

[1046] b(j) required : The required transmission bit rate for transmitting an image with the j-th image priority at a delay equal to or less than a predetermined value.

[1047] b(j) overlayed : The minimum bit rate (predefined value) required to use the image with the j-th image priority as the composite image.

[1048] b buffer : The bit rate required to prevent image freezing during display switching between images captured using a camera and 3DCG images captured without a camera (= the bit rate used to ensure the time for receiving image buffer processing) (predefined value).

[1049] B allocated The transmission bit rate for each image (M images) sent from the autonomous vehicle.

[1050] (B allocated ={b(1)} allocated b(2) allocated ,…,b(M) allocated})

[1051] O target The target value to be applied to the opacity of the 3DCG image generated from the composite image.

[1052] O current The current setting for the opacity of the 3DCG image generated from the composite image.

[1053] The following sections will describe the processing at each step shown in the flowchart in Figure 34.

[1054] Note that the flowchart shown in Figure 34 is the process performed at the receiving time of each of the multiple camera-captured images (e.g., images corresponding to the six cameras illustrated earlier with reference to Figures 3 and 4) received by the remote assistance device 40 from the autonomous vehicle 10.

[1055] Note the image priority index = j and the transmission bit rate (b(j)). allocated The information has been assigned to each received image, and together with the image, the remote assistance device 40 receives this information as metadata or image-related attribute information, and uses this information to perform processing according to the flowchart shown in FIG34.

[1056] (Step S501)

[1057] First, at step S501, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 receives the transmission bit rate (B) of each of the M images (i.e., M images having image priorities represented by image priority indices j = 1 to M) received from the autonomous vehicle 10. allocated ).

[1058] Note that, for example, M images are images corresponding to the six cameras illustrated with reference to Figures 3 and 4, and in the example using the six cameras illustrated with reference to Figures 3 and 4, M = 6.

[1059] As mentioned earlier, the image priority index = j and the transmission bit rate (b(j)) allocated The information has been assigned to each received image, and together with the image, the remote assistance device 40 receives this information as metadata or image-related attribute information.

[1060] In step S501, the data processing unit 42 of the remote assistance device 40 receives, via the communication unit 41, the transmission bit rate (B) allocated to each of the M images having image priority indices j = 1 to M. allocated ), that is, the transmission bit rate (B) allocated to each image. allocated ) information,

[1061] B allocated ={b(1)} allocated b(2) allocated ,…,b(M) allocated}

[1062] (Step S502)

[1063] Step S502 is the initial setting process for parameter j.

[1064] In step S502, the data processing unit 42 of the remote assistance device 40 sets the value of the image priority index j to the initial value j = 1.

[1065] This means selecting the image with the highest image priority and initially processing it.

[1066] (Step S503)

[1067] Next, in step S503, the data processing unit 42 of the remote assistance device 40 evaluates whether the value of the image priority index j is greater than the total number of images M, that is, whether it satisfies the formula described below.

[1068] j > M.

[1069] If the formula described above is satisfied, the processing of the M images that should be processed has been completed, and therefore the processing according to the flowchart has ended.

[1070] On the other hand, in cases where the formula described above is not satisfied, that is, in In the case of priority indexes The target image is processed in step S504 and subsequent steps.

[1071] (Step S504)

[1072] Next, in step S504, the data processing unit 42 of the remote assistance device 40 evaluates the data with priority index. Does the target image being processed satisfy the following formula?

[1073] Evaluate whether the evaluation formula described below is satisfied.

[1074]

[1075] The evaluation formula described above is used to evaluate the transmission bit rate (b(j)) assigned to an image with image priority index j. allocated Is b(j) equal to or greater than b(j)? overlayed The formula.

[1076] Note that, as mentioned earlier, b(j) overlayed It is the minimum bit rate required to use the image captured by the camera with image priority j as a composite image to be output to the display of the remote assistance device 40, and it is a predefined value.

[1077] If the evaluation formula described above is satisfied, the process proceeds to step S505. On the other hand, if the evaluation formula described above is not satisfied, the process proceeds to step S506.

[1078] (Step S505)

[1079] If the evaluation at step S504 satisfies the evaluation formula described below...

[1080]

[1081] In step S505, the data processing unit 42 of the remote assistance device 40 performs a process of setting the opacity of the 3DCG image to 0 (= completely transparent) and causing the display unit 43 to display the camera-captured image with image priority j as is.

[1082] This processing is equivalent to setting the highest opacity (Opacity) to 0 (completely transparent) in a 3DCG image with five different opacity values ​​illustrated in Figure 33. That is, this is the processing of outputting the camera-captured image to the display unit 43 as is without using the 3DCG image.

[1083] The processing is determined based on the following: the transmission bit rate (b(j)) allocated to the camera-captured images with image priority j received from the autonomous vehicle 10. allocated It has a sufficiently high bit rate and can output clear camera-captured images.

[1084] After the processing at step S505, the process proceeds to step S510, where the value of the image priority index j is updated (j = j + 1), and the processing at steps S503 and subsequent steps is performed on the image with the updated image priority index j.

[1085] (Step S506)

[1086] On the other hand, if the evaluation at step S504 fails to meet the evaluation formula described below,...

[1087]

[1088] In step S506, the data processing unit 42 of the remote assistance device 40 evaluates whether the following formula is satisfied.

[1089] Evaluate whether the evaluation formula described below is satisfied.

[1090]

[1091] The evaluation formula described above is used to evaluate the transmission bit rate (b(j)) assigned to an image with image priority index j. allocated Is (b(j)) equal to or greater than (b(j))? required +b buffer The formula for ).

[1092] As illustrated earlier with reference to Figure 33,

[1093] b(j) required The transmission bit rate required to transmit an image with image priority j is equal to or less than a predetermined delay value.

[1094] b bufferIt is the bit rate required to prevent image freezing during display switching between images captured using a camera and 3DCG images captured without a camera, and it is also the bit rate used to ensure the time for receiving image buffer processing.

[1095] If, at step S506, the evaluation does not satisfy the evaluation formula described below, then...

[1096]

[1097] The process proceeds to step S507.

[1098] On the other hand, if the evaluation meets the evaluation formula, the process proceeds to step S509.

[1099] (Step S507)

[1100] If, at step S506, the evaluation does not satisfy the evaluation formula described below, then...

[1101]

[1102] The process proceeds to step S507.

[1103] Step S507 involves evaluating the transmission bit rate (b(j)) assigned to the image with image priority index j. allocated () is less than (b(j)) required +b buffer The processing performed under the condition of ).

[1104] In this case, the data processing unit 42 of the remote assistance device 40 sets the 3DCG image opacity to [value missing].

[1105] Opacity = 1 (completely opaque)

[1106] This is equivalent to processing a 3DCG image with the lowest opacity (Opacity) of 1 (completely opaque) among the five different opacity values ​​illustrated earlier with reference to Figure 33. That is, the image is not captured using a camera, and only the 3DCG image is output to the display unit 43.

[1107] The processing is determined based on the following: the transmission bit rate (b(j)) allocated to the camera-captured images with image priority j received from the autonomous vehicle 10. allocated The bit rate is zero or extremely low, and it cannot output clear camera-captured images.

[1108] (Step S508)

[1109] After the process of outputting only the 3DCG image to the display unit 43 in step S507, in step S508, the data processing unit 42 of the remote assistance device 40 performs the processing of rendering objects on the 3DCG image displayed on the display unit 43.

[1110] As explained earlier, the 3DCG image generated from the 3D map stored in the 3D map storage unit 45 does not include real-time surrounding objects such as oncoming vehicles and pedestrians that exist around the autonomous vehicle 10.

[1111] For this reason, the data processing unit 42 of the remote assistance device 40 performs processing in which it analyzes metadata received from the autonomous driving control device 20, including information about surrounding objects (such as oncoming vehicles), and displays images of objects around the autonomous driving vehicle 10 on a 3DCG image.

[1112] By performing such processing, display data that more reliably reproduces the current surroundings of the autonomous vehicle 10 is generated and displayed on the display unit.

[1113] Note that the details of this process will be explained later.

[1114] After the processing at step S508, the process proceeds to step S510, where the value of the image priority index j is updated (j = j + 1), and step S503 and subsequent steps are performed on the image with the updated image priority index j.

[1115] (Step S509)

[1116] On the other hand, if the evaluation at step S506 satisfies the evaluation formula described below...

[1117]

[1118] The process proceeds to step S509.

[1119] Step S509 is a process performed when the evaluation does not meet the evaluation formula at step S504.

[1120]

[1121] Furthermore, the evaluation is performed to satisfy the evaluation formula at step S506.

[1122]

[1123] That is, the transmission bit rate (b(j)) allocated to the image with image priority index j. allocated If the value is within the following range, proceed to step S509.

[1124]

[1125] This is equivalent to using the settings of the three images in the middle row (i.e., opacity = 0.25 to 0.75) in a 3DCG image with five different opacity values ​​as illustrated earlier with reference to Figure 33.

[1126] In this case, as explained earlier with reference to FIG33, the data processing unit 42 of the remote assistance device 40 executes the transmission bit rate (b(j)) to be allocated to the image having image priority index j. allocated The data processing unit 42 performs a process that compares the value of the data with each threshold shown in Figure 33, and performs a process that calculates the opacity (Opacity) of the 3DCG image (=0 to 1). In addition, the data processing unit 42 performs the following processes: generating a composite image that overlays the 3DCG image with the calculated opacity (Opacity) onto the image captured by the camera, and outputting the composite image to the display unit.

[1127] Note that after the processing at step S509, the process proceeds to step S510, where the value of the image priority index j is updated (j = j + 1), and the processing at steps S503 and subsequent steps is performed on the image with the updated image priority index j.

[1128] Next, referring to the flowchart shown in FIG35, a detailed sequence of processing control for changes in the opacity of the 3DCG image when the image frame of the composite image (i.e., the composite image of the camera-captured image and the 3DCG image) displayed on the display unit 43 of the remote assistance device 40 is updated will be described.

[1129] The composite image of the camera-captured image and the 3DCG image displayed on the display unit 43 of the remote assistance device 40 is a moving image, and the display control unit 122 of the data processing unit 42 of the remote assistance device 40 calculates the 3DCG image opacity in each image frame according to the processing sequence described with reference to FIG34.

[1130] However, for example, if the opacity of a 3DCG image changes significantly between consecutive image frames, and a composite image with the calculated opacity is generated and output to the display unit 43 as is, a sudden change occurs in the displayed image, and the image becomes undesirably difficult to observe.

[1131] The flowchart shown in Figure 35 is a flowchart illustrating the control sequence used to prevent such a situation. Specifically, the flowchart illustrates the controls used to smoothly change the value of the 3DCG image opacity and output a more easily observable composite image.

[1132] The following describes the processing at each step of the flowchart shown in Figure 35.

[1133] (Step S521)

[1134] First, in step S521, the data processing unit 42 of the remote assistance device 40 acquires the target value O of the 3DCG image opacity. target .

[1135] Target value O for 3DCG image opacity target The calculation is performed according to the flowchart shown in Figure 34, which was described earlier.

[1136] (Steps S522 to S523)

[1137] Next, in step S522, the data processing unit 42 of the remote assistance device 40 checks whether a 3DCG image opacity change animation is being performed in the composite image output on the display unit 43.

[1138] The 3DCG image opacity change animation is an animation of the smooth change of 3DCG image opacity performed according to the flowchart shown in Figure 35.

[1139] In the case where the animation of the 3DCG image opacity changing smoothly has already been executed, at step S523, the animation is stopped to prevent redundant execution of the process, and the process then proceeds to step S524.

[1140] Note that animation stabilization processing can be performed in parallel with the processing at step S522 and subsequent steps. Animation stabilization processing targets the opacity of the 3DCG image at a value of O. targetThe processing prevents display flickering when the value fluctuates around the threshold. As specific examples, the following processing provides advantageous effects: processing that calculates a moving average and applies the moving average result while taking into account the time transition of the calculated opacity, or processing that provides hysteresis and changes the threshold instead of setting the threshold shown in Figure 33 to a fixed value, etc.

[1141] (Step S524)

[1142] Next, in step S524, the data processing unit 42 of the remote assistance device 40 evaluates the target value O of the 3DCG image opacity. target The opacity O of the 3DCG image at the current time point in the composite image currently being output on display unit 43 current Is the difference between them less than the threshold? That is, assess whether the formula described below is satisfied.

[1143]

[1144] If the evaluation finds that the formula described above is satisfied, the process proceeds to step S525. On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S526.

[1145] (Step S525)

[1146] If the evaluation at step S524 is satisfied with the formula described below, then...

[1147]

[1148] In step S525, the data processing unit 42 of the remote assistance device 40 sets the opacity of the 3DCG image to the target value O. target A composite image is generated and output to the display unit 43.

[1149] In this case, the change in 3DCG image opacity decreases (below the threshold). The output image shows reduced changes and no flickering is perceived.

[1150] (Step S526)

[1151] If, at step S524, the evaluation indicates that the formula described below is not satisfied, then...

[1152]

[1153] In step S526, the data processing unit 42 of the remote assistance device 40 calculates the time parameter T according to the following formula.

[1154]

[1155] Note that T full It is a predefined time as the time required to perform a seamless transition when the opacity of a 3DCG image changes from 0 to 1.

[1156] The time parameter T calculated according to the formula described above is equivalent to the current opacity O of the 3DCG image. current Transform into the target value O without causing discomfort. target Required transition time.

[1157] (Step S527)

[1158] Next, in step S527, the data processing unit 42 of the remote assistance device 40 acquires the current time t.

[1159] (Step S528)

[1160] Next, in step S528, the data processing unit 42 of the remote assistance device 40 calculates the start time (t) of the 3DCG image opacity change animation according to the following calculation formula. start Time elapsed up to the current time t .

[1161]

[1162] (Step S529)

[1163] Next, in step S529, the data processing unit 42 of the remote assistance device 40 calculates the new 3DCG image opacity O that should be set at the current time t according to the following calculation formula. new .

[1164]

[1165] The new 3DCG image opacity O is calculated based on the formula described above. new The opacity is set as follows: the opacity of the 3DCG image at the current time t is set to the new 3DCG image opacity O. new In the case of an animation starting at the opacity change of a 3DCG image (t... startThe resulting changes in opacity are planetized, enabling the output of composite images that do not cause discomfort.

[1166] (Step S530)

[1167] Next, in step S530, the data processing unit 42 of the remote assistance device 40 evaluates the target value O of the 3DCG image opacity. target The new 3DCG image opacity O calculated at step S529 new Is the difference between them less than the threshold? That is, assess whether the formula described below is satisfied.

[1168]

[1169] If the evaluation finds that the formula described above is satisfied, the process proceeds to step S531. On the other hand, if the evaluation finds that the formula described above is not satisfied, the process proceeds to step S532.

[1170] (Step S531)

[1171] If, at step S530, the evaluation is satisfied with the formula described below, then...

[1172]

[1173] In step S531, the data processing unit 42 of the remote assistance device 40 sets the opacity of the 3DCG image to the target value O. new A composite image is generated and output to the display unit 43.

[1174] In this case, the change in 3DCG image opacity decreases (below the threshold). The output image shows reduced changes and no flickering is perceived.

[1175] (Step S532)

[1176] On the other hand, if the evaluation at step S530 determines that the formula described below is not satisfied...

[1177]

[1178] At step S532, the data processing unit 42 of the remote assistance device 40 waits until the next image frame is received from the autonomous vehicle 10. After waiting, the data processing unit 42 returns to step S527 and performs the processing at step S527 and subsequent steps as 3DCG image opacity control processing for the received new image frame.

[1179] By performing the processing according to the flowchart shown in Figure 35, the change in the opacity value of the 3DCG image becomes smoother, and a more easily observable composite image can be output.

[1180] [9. Details regarding object display control processing performed by the remote assistance device]

[1181] Next, details of the object display control processing performed by the data processing unit 42 of the remote assistance device 40 will be explained.

[1182] As can be understood from the above description, the data processing unit 42 of the remote assistance device 40 causes the display unit 43 to display any one of the camera-captured image, the composite image of the camera-captured image and the 3DCG image, or the 3DCG image received from the autonomous vehicle 10.

[1183] When using 3DCG images as display images, the data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 to generate and display 3DCG images observed from the position and orientation of the camera installed on the autonomous vehicle 10.

[1184] However, the 3DCG image generated from the 3D map stored in the 3D map storage unit 45 does not include real-time surrounding objects such as oncoming vehicles and pedestrians that exist around the autonomous vehicle 10.

[1185] For this reason, the data processing unit 42 of the remote assistance device 40 performs processing in which it analyzes metadata received from the autonomous driving control device 20, including information about surrounding objects (such as oncoming vehicles), and overlays images of objects around the autonomous driving vehicle 10 onto a 3DCG image.

[1186] By performing such processing, display data that more reliably reproduces the current surroundings of the autonomous vehicle 10 can be generated and displayed on the display unit.

[1187] The details of the object display control processing performed by the data processing unit 42 of the remote assistance device 40 will be described below.

[1188] Note that the object display control processing described below is performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40 shown in FIG6.

[1189] As illustrated earlier with reference to Figures 23 to 29, the autonomous driving control device 20 generates metadata including information about surrounding objects (such as oncoming vehicles) included in the unsent image and sends the metadata to the remote assistance device 40.

[1190] Note that, for example, the data processing unit 22 of the autonomous driving control unit 20 continuously sends object coordinates, bounding box sizes, labels, etc., for identifying the position and size of objects (such as vehicles and pedestrians around the autonomous driving vehicle 10). It should be noted that for large amounts of data such as object images, when the bandwidth of the communication path used for metadata communication can be ensured to be narrow, processes such as stopping transmission, sending large amounts of data as compressed data, and selectively sending data based on the risk level of each object are performed. Note that the data processing unit 22 of the autonomous driving control unit 20 assesses the risk level of the object and also sends the assessment result as metadata to the remote assistance device 40.

[1191] The display control unit 122 of the data processing unit 42 of the remote assistance device 40 extracts object image data, object compressed image data, etc. from the metadata sent by the data processing unit 22 of the automatic driving control device 20, generates an object image to be output to the display unit 43 of the remote assistance device 40, and outputs the object image.

[1192] The display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs processing to generate display data that varies according to the risk level of the object.

[1193] Referring to Figure 37, the object display processing corresponding to the risk level of the object is explained by the display control unit 122 of the data processing unit 42 of the remote assistance device 40.

[1194] Figure 37 shows an example of object display processing corresponding to the risk level of the object, performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40.

[1195] Note that there are four object risk levels, from 1 to 4, with object risk level 1 being the highest risk level and object risk level 4 being the lowest risk level.

[1196] As shown in Figure 37, the data processing unit 42 of the remote assistance device 40 performs the following object display processing according to the risk level of the object.

[1197] For objects with an object risk level of 1, perform 3D object image display processing using uncompressed object image data (i.e., uncompressed object image data).

[1198] For objects with an object risk level of 2, perform 3D object image display processing using compressed object image data with a low compression ratio.

[1199] For objects with an object risk level of 3, perform 3D object image display processing using compressed object image data with a high compression ratio.

[1200] For objects with an object risk level of 4, perform processing to display the bounding box around the object area and the label indicating the object type.

[1201] As explained earlier with reference to Figures 23 to 29, the automatic driving control device 20 is configured to send object image data with different compression rates depending on the risk level of surrounding objects such as oncoming vehicles, and the data processing unit 42 of the remote assistance device 40 uses the data received from the automatic driving control device 20 to perform object display processing corresponding to the object risk level as shown in Figure 37.

[1202] Figure 38 shows a specific example of object display processing corresponding to the risk level of the object, performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40.

[1203] Figure 38 shows the specific object image table instructions corresponding to the object risk levels 1 to 4 illustrated with reference to Figure 37.

[1204] Figure 38(a) shows an example of object display with object risk level = 1, and also shows an example of 3D object image display using uncompressed object image data (i.e., uncompressed object image data).

[1205] (b) shows an example of object display with an object risk level of 2, and shows an example of 3D object image display using compressed object image data with a low compression ratio.

[1206] (c) shows an example of object display with an object risk level of 3, and shows an example of 3D object image display using compressed object image data with a high compression ratio.

[1207] (d) shows an example of an object display with an object risk level of 4, and shows an example of a display of the bounding box around the object area and a label indicating the object type.

[1208] In this way, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs the processing to generate display data that varies according to the risk level of the object.

[1209] Next, the object image intra-interpolation processing and object image extra-interpolation processing performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40 as object image interpolation processing will be described.

[1210] As illustrated earlier with reference to Figures 23 to 29, the autonomous driving control device 20 generates metadata, including information about surrounding objects such as oncoming vehicles, and sends the metadata to the remote assistance device 40. However, in some cases, where the bandwidth of the communication path used for metadata communication can be ensured to be narrow, a transmission halt process is performed for large amounts of data, such as images of objects. Furthermore, in some cases, filtering is performed to exclude images of low-risk objects from the transmission target.

[1211] Regarding object images that cannot be received from the autopilot control unit 20 in this manner, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs interpolation processing on the object image using the object image corresponding to the previous image frame and the subsequent image frame.

[1212] The interpolation processing performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40 includes the following two processing modes.

[1213] (1) Extrapolation processing, wherein, by taking the previous image frame (at time t) n-1 The image of the object (at time t) is estimated to generate an unreceiveable image frame (at time t) by estimating the object image (at time t) and the previous image frame. n The image of the object (image frame at the location)

[1214] (2) Interpolation processing, wherein, by interpolating from the previous image frame (at time t) n-1 Image frames at and before time t) and subsequent image frames (at time t) n+1 The object image (at time t and subsequent image frames) is estimated to generate an unreceiveable image frame (at time t). n The image of the object (image frame at the location)

[1215] Refer to Figures 39 and 40 for specific examples of interpolation and extrapolation processing.

[1216] Figure 39 is a diagram illustrating a specific example of extrapolation processing performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40.

[1217] The inability to receive image frames of the object image occurred at time t. nImage frames at that location. Able to receive previous image frames (at time t). n-1 and time t n-2 The image of the object (image frame at the location).

[1218] The display control unit 122 of the data processing unit 42 of the remote assistance device 40 transmits data from a previous image frame (at time t). n-1 and time t n-2 The image of the object in the image frame at time t is estimated to generate an unreceiveable image at time t. n The object image of the image frame at that location. The display control unit 122 of the data processing unit 42 of the remote assistance device 40 executes the operation from the previous image frame (at time t). n-1 and time t n-2 The image of the object in the image frame at time t is used to estimate the time to time t. n The position and size of the object displayed on the image frame at time t n The process of rendering the object on the image frame at that location.

[1219] Figure 40 is a diagram illustrating a specific example of interpolation processing performed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40.

[1220] Interpolation processing is a process designed to improve the accuracy of the extrapolation processing illustrated with reference to Figure 39. That is, it is a process that improves the accuracy of the extrapolated image by further correcting the interpolated image generated by the extrapolation processing performed in the past using object images received at previous and subsequent times.

[1221] As shown at the top of Figure 40, assume that at time t n The image of the image frame at that location is an interpolated image generated through extrapolation processing illustrated in Figure 39.

[1222] Able to receive previous image frames (at time t) n-1 Image frames at and before time t) and subsequent image frames (at time t) n+1 The object image (the image frame at the time and the subsequent image frames).

[1223] The display control unit 122 of the data processing unit 42 of the remote assistance device 40 uses the previous image frame (at time t) n-1 The object image at time t) and subsequent image frames (at time t) n+1 The image of the object at time t (image frame) n The extrapolated image is then further corrected (interpolated). By performing this correction (interpolated) process, the accuracy of the extrapolated image can be improved.

[1224] For example, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 improves the accuracy of the interpolated image by: [the method described in the original text, which is incomplete and requires further context to translate accurately.] n-1 The object image at time t) and subsequent image frames (at time t) n+1 Image estimation of the object image at time t (image frame at time t) n The position and size of the extrapolated image at time t are further corrected. n The position and size of the extrapolated image at that location.

[1225] In this way, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs processing to re-correct the object image generated by the extrapolation processing estimate using the object image corresponding to the previous image frame and the subsequent image frame.

[1226] Through these processes, the operator at the remote assistance center 30 can reliably inspect objects such as oncoming vehicles based on the displayed images, and can generate appropriate control information to prevent the autonomous vehicle 10 from colliding with the objects, and send it to the autonomous driving control unit 20.

[1227] [10. Regarding other embodiments]

[1228] Next, other embodiments will be described.

[1229] In the above embodiments, for example, the autonomous driving control device 20 of the autonomous vehicle 10 is configured to use a single stream set between the autonomous driving control device 20 and the remote assistance device 40 of the remote assistance center 30 to send video tracks 1 to n corresponding to n images of the plurality of cameras (n cameras) illustrated earlier with reference to Figures 3 and 4, as illustrated with reference to Figure 8.

[1230] That is, multiple images are transmitted via multitrack video streaming, for example, six images corresponding to six cameras, using a single stream to transmit multiple videos. In this configuration, the bitrate of the multiple images to be transmitted varies constantly based on the available bandwidth. For example, in this configuration, a larger communication bandwidth (bitrate) is allocated to higher-priority images based on image priority.

[1231] Instead of performing a multi-track video streaming transmission that uses a single stream to send multiple videos, as shown in Figure 41, a configuration can also be adopted in which n images corresponding to multiple cameras (n cameras) are sent together as a single video track.

[1232] It should be noted that in the configuration shown in Figure 41, it is not possible to control the transmission bit rate of each of the n individual images corresponding to multiple cameras (n cameras), and therefore a process of uniformly adjusting the overall transmission bit rate of the n images is performed.

[1233] That is, this is a configuration that executes control to change the transmission bit rate of a single video track that co-transmits n images based on the available bandwidth.

[1234] Furthermore, although the above embodiment describes a configuration in which the autonomous driving control device 20 of an autonomous vehicle 10 sends images captured by a camera to a remote assistance device 40, the configuration shown in FIG42 is also possible, for example.

[1235] In the configuration shown in Figure 42, the autonomous driving control unit 20 of an autonomous vehicle 10 sends images captured by a camera to multiple remote assistance devices 40a to 40c.

[1236] The image controlled according to the above embodiment, namely the composite image of the camera-captured image and the 3DCG image, is displayed on the display unit of the remote assistance device 40a to 40c.

[1237] The images displayed on the display units of each remote assistance device 40a to 40c are checked by multiple operators a to c. Each operator a to c detects obstacles, etc., generates control information, and sends the control information to the autonomous vehicle 10.

[1238] By adopting this configuration, the omission of obstacles and other obstacles is reduced, the accuracy of image inspection is improved, and safer control can be implemented.

[1239] Alternatively, the configuration shown in Figure 43 is also acceptable.

[1240] In the configuration shown in Figure 43, each of the multiple autonomous vehicles a to c captures images using cameras mounted on the vehicle and sends them to the same single remote assistance device 40. An operator 50 on the remote assistance device 40 examines the images sent from each vehicle and sends control information to each vehicle.

[1241] By adopting this configuration, the number of operators can be reduced by 50.

[1242] [11. Hardware configuration examples for automatic driving control devices and remote assistance devices]

[1243] Next, an example of the hardware configuration of the automatic driving control device 20 and the remote assistance device 40 of this disclosure will be described with reference to FIG44.

[1244] The CPU (Central Processing Unit) 301 serves as a data processing unit that performs various types of processing according to programs stored in the ROM (Read-Only Memory) 302 or the storage unit 308. For example, the CPU 301 performs processing according to the sequence described in the above embodiments. Programs, data, etc., to be executed by the CPU 301 are stored in the RAM (Random Access Memory) 303. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304.

[1245] The CPU 301 is connected to the input / output interface 305 via the bus 304, and the input / output interface 305 is connected to the input section 306 and the output section 307. The input section includes various types of switches, touch panels, microphones, and user input sections, etc., and the output section includes displays, speakers, etc.

[1246] Note that in the case of the autonomous driving control device 20, the input unit 306 receives sensor acquisition information (camera captured images, object detection information, object distance information, etc.) from various types of sensors 321 such as cameras and LiDAR.

[1247] In addition, in the case of the automatic driving control device 20, the output unit 307 also outputs driving information for the vehicle's drive unit 322.

[1248] For example, CPU 301 receives instructions, status data, etc. from input unit 306, performs various types of processing, and outputs the processing results to output unit 307.

[1249] For example, the storage unit 308 connected to the input / output interface 305 includes a hard disk and stores programs to be executed by the CPU 301 and various types of data. The communication unit 309 serves as a transmitting / receiving unit for data communication via networks such as the Internet and local area networks, and communicates with external devices.

[1250] In addition to the CPU, a GPU (Graphics Processing Unit) may also be included as a dedicated processing unit for image information input from a camera, etc.

[1251] The driver 310 connected to the input / output interface 305 drives removable media 311 such as a disk, optical disk, magneto-optical disk, or semiconductor memory (such as a memory card) and performs data recording or reading.

[1252] [12. Summary of the configuration according to this disclosure]

[1253] The embodiments according to this disclosure have been described in detail above with reference to specific examples. However, it will be apparent to those skilled in the art that modifications or substitutions to the embodiments can be conceived without departing from the spirit of this disclosure. That is, the invention is disclosed in an exemplary manner and should not be interpreted in a limited manner. The claims portion should be considered in order to determine the spirit of this disclosure.

[1254] Note that the technology disclosed in this specification can be configured as follows.

[1255] (1) An automatic driving control device, comprising:

[1256] The communication unit transmits images captured by multiple cameras mounted on the vehicle to the remote assistance device, and receives autonomous driving control information from the remote assistance device; and

[1257] The data processing unit executes the transmission control of each of the plurality of camera-captured images, wherein...

[1258] The data processing unit performs the following:

[1259] The processing involves calculating an estimated bit rate equivalent to the bandwidth that can be used to send images to the remote control device;

[1260] Processing to calculate the image priority of each of the multiple camera-captured images; and

[1261] The image selection process selects images captured by the plurality of cameras in descending order of image priority and sends the selected images that are evaluated to be able to be sent to the remote control device at the estimated bit rate or lower, with a delay equal to or less than a predetermined value.

[1262] (2) The automatic driving control device according to (1), wherein,

[1263] If the estimated bit rate is equal to or greater than the required bit rate for transmitting all the plurality of camera-captured images to the remote control device with a delay equal to or less than a predetermined value, the data processing unit performs transmission processing for all the plurality of camera-captured images without performing the transmission image selection processing.

[1264] If the estimated bit rate is less than the required bit rate determined to enable all the multiple camera-captured images to be sent to the remote control device with a delay equal to or less than a predetermined value, the data processing unit calculates the required bit rate for each image combination unit selected from the multiple camera-captured images in descending order of image priority, and sends the image combination unit having the largest required bit rate among the required bit rates calculated for the image combination unit and equal to or less than the estimated bit rate.

[1265] (3) The automatic driving control device according to (1) or (2), wherein,

[1266] The data processing unit determines the allocation of the transmission bit rate for each of the plurality of camera-captured images to be transmitted to the remote control device in proportion to the resolution of the number of pixels in each camera-captured image selected as the transmission target.

[1267] (4) The automatic driving control device according to any one of (1) to (3), wherein,

[1268] The data processing unit considers at least one of the following to calculate the image priority of each of the plurality of camera-captured images:

[1269] (a) The number of objects detected in each of the images captured from the plurality of cameras; or

[1270] (b) The number of high-risk objects detected in each of the images captured from the plurality of cameras.

[1271] (5) The automatic driving control device according to any one of (1) to (4), wherein,

[1272] The data processing unit:

[1273] Perform transmit bit rate control to set a target bit rate between a predefined minimum bit rate and a maximum bit rate, said target bit rate being the transmit bit rate to be applied when all of the plurality of camera-captured images are to be transmitted.

[1274] If the estimated bit rate is equal to or greater than the maximum bit rate, the target bit rate is set to the maximum bit rate.

[1275] (6) The automatic driving control device according to any one of (1) to (5), wherein,

[1276] The data processing unit sends the object image of the object detected in the image captured by the camera to the remote assistance device.

[1277] (7) The automatic driving control device according to (6), wherein,

[1278] The data processing unit performs the process of detecting objects in unsent images excluded from the transmission target in the transmitted image selection process, and sends the object image of the object detected in the unsent images to the remote assistance device.

[1279] (8) The automatic driving control device according to (6) or (7), wherein,

[1280] When the bit rate used for transmitting object images, which is the bandwidth available for transmitting object images, is equal to or greater than the bit rate required to transmit all object images to the remote control device with a delay equal to or less than a predetermined value, the data processing unit performs the processing of transmitting all object images of the detected object, and

[1281] If the bit rate used to send the object image is less than the required bit rate for sending all the object images to the remote control device with a delay equal to or less than a predetermined value, the data processing unit generates a compressed object image obtained by compressing at least some of the object images, and sends the compressed object image.

[1282] (9) The automatic driving control device according to any one of (6) to (8), wherein,

[1283] The data processing unit assesses the risk level of the detected objects from the image and selectively sends object images of high-risk objects.

[1284] (10) The automatic driving control device according to any one of (6) to (9), wherein,

[1285] The data processing unit:

[1286] Configured to perform compression processing by applying different compression algorithms based on the risk level of the detected objects in the image, and

[1287] Compressed object images are generated and sent by applying a compression algorithm that decreases the compression ratio as the object's risk level increases and increases the compression ratio as the object's risk level decreases.

[1288] (11) The automatic driving control device according to any one of (6) to (10), wherein,

[1289] The data processing unit performs object detection processing by performing image segmentation processing on the image.

[1290] (12) The automatic driving control device according to any one of (1) to (11), wherein,

[1291] The communication unit performs image transmission by using a multi-track video stream technique that transmits multiple camera-captured images in a single stream.

[1292] (13) A remote assistance device, comprising:

[1293] The communication unit receives images captured by multiple cameras from the autonomous driving control unit, which are captured by multiple cameras installed on the vehicle; and

[1294] The data processing unit performs display control to display the images captured by the plurality of cameras on the display unit, wherein,

[1295] The data processing unit switches the image to be displayed on the display unit to any one of the following based on the image received from the automatic driving control device:

[1296] (a) A camera-captured image received from the autonomous driving control device;

[1297] (b) A composite image of camera-captured images and 3DCG images received from the autonomous driving control device; or

[1298] (c) 3DCG image.

[1299] (14) The remote assistance device according to (13), wherein,

[1300] The data processing unit:

[1301] If the image received from the autonomous driving control device has high resolution, the image to be displayed on the display unit is set to be the camera-captured image received from the autonomous driving control device.

[1302] In the absence of an image received from the automatic driving control device, the image to be displayed on the display unit is set to a 3DCG image.

[1303] (15) The remote assistance device according to (13) or (14), wherein,

[1304] The data processing unit controls the opacity of the 3DCG image in the composite image of the camera-captured image and the 3DCG image to generate each image as any one of (a) to (c), and causes the display unit to display the image.

[1305] (16) The remote assistance device according to (15), wherein,

[1306] The data processing unit controls the opacity of the 3DCG image based on the value of the transmission bit rate of each of the plurality of camera-captured images determined by the autonomous driving control device.

[1307] (17) The remote assistance device according to any one of (13) to (16), wherein,

[1308] The data processing unit displays the object image received from the automatic driving control device on the 3DCG image displayed on the display unit.

[1309] (18) The remote assistance device according to (17), wherein,

[1310] The data processing unit displays the object image in a display mode that varies according to the object's risk level.

[1311] (19) The remote assistance device according to any one of (13) to (18), wherein,

[1312] Regarding image frames from which an object image cannot be received from the automatic driving control device, the data processing unit displays an object image generated by interpolation processing of the object image using previous and subsequent image frames.

[1313] (20) An automatic driving control system, comprising an automatic driving control device and a remote assistance device, wherein,

[1314] The automatic driving control device performs:

[1315] The processing involves calculating an estimated bit rate equivalent to the bandwidth that can be used to send images to the remote control device;

[1316] The processing of calculating the image priority of each camera capture image among multiple camera capture images captured by multiple cameras mounted on a vehicle; and

[1317] The image selection process selects images captured by the plurality of cameras in descending order of image priority, and transmits the selected images that are evaluated as capable of being transmitted to the remote control device at the estimated bit rate or lower, with a delay equal to or less than a predetermined value.

[1318] The remote assistance device:

[1319] The process of switching the image to be displayed on the display unit based on the image received from the automatic driving control device is performed to any of the following:

[1320] (a) A camera-captured image received from the autonomous driving control device;

[1321] (b) A composite image of camera-captured images and 3DCG images received from the autonomous driving control device; or

[1322] (c) 3DCG images, and

[1323] The system sends control information generated based on the inspection results of the displayed image to the autonomous driving control device.

[1324] (21) An image transmission control method executed at an automatic driving control device, wherein,

[1325] The automatic driving control device has:

[1326] The communication unit transmits images captured by multiple cameras mounted on the vehicle to the remote assistance device, and receives autonomous driving control information from the remote assistance device; and

[1327] The data processing unit executes the transmission control of each of the plurality of camera-captured images, and

[1328] The data processing unit performs the following:

[1329] The processing involves calculating an estimated bit rate equivalent to the bandwidth that can be used to send images to the remote control device;

[1330] Processing to calculate the image priority of each of the multiple camera-captured images; and

[1331] The image selection process selects images captured by the plurality of cameras in descending order of image priority and sends the selected images that are evaluated to be able to be sent to the remote control device at the estimated bit rate or lower, with a delay equal to or less than a predetermined value.

[1332] (22) An image display control method executed at a remote assistance device, wherein,

[1333] The remote assistance device has:

[1334] The communication unit receives images captured by multiple cameras from the autonomous driving control unit, which are captured by multiple cameras installed on the vehicle; and

[1335] The data processing unit performs display control to display the images captured by the plurality of cameras on the display unit, and

[1336] The data processing unit switches the image to be displayed on the display unit to any one of the following based on the image received from the automatic driving control device:

[1337] (a) A camera-captured image received from the autonomous driving control device;

[1338] (b) A composite image of camera-captured images and 3DCG images received from the autonomous driving control device; or

[1339] (c) 3DCG image.

[1340] (23) A program that causes image transmission control processing to be performed at an automatic driving control device, wherein,

[1341] The automatic driving control device has:

[1342] The communication unit transmits images captured by multiple cameras mounted on the vehicle to the remote assistance device, and receives autonomous driving control information from the remote assistance device; and

[1343] The data processing unit executes the transmission control of each of the plurality of camera-captured images, and

[1344] The program causes the data processing unit to execute:

[1345] The processing involves calculating an estimated bit rate equivalent to the bandwidth that can be used to send images to the remote control device;

[1346] Processing to calculate the image priority of each of the multiple camera-captured images; and

[1347] The image selection process selects images captured by the plurality of cameras in descending order of image priority and sends the selected images that are evaluated to be able to be sent to the remote control device at the estimated bit rate or lower, with a delay equal to or less than a predetermined value.

[1348] (24) A program that causes image display control processing to be performed at a remote assistance device, wherein,

[1349] The remote assistance device has:

[1350] The communication unit receives images captured by multiple cameras from the autonomous driving control unit, which are captured by multiple cameras installed on the vehicle; and

[1351] The data processing unit performs display control to display the images captured by the plurality of cameras on the display unit, and

[1352] The program causes the data processing unit to switch the image to be displayed on the display unit to any one of the following based on the image received from the automatic driving control device:

[1353] (a) A camera-captured image received from the autonomous driving control device;

[1354] (b) A composite image of camera-captured images and 3DCG images received from the autonomous driving control device; or

[1355] (c) 3DCG image.

[1356] Furthermore, the series of processes described in the manual can be executed via hardware, software, or a combination of hardware and software. When processing is performed via software, a program containing the processing sequence can be installed in the memory of a computer built into dedicated hardware and executed thereon, or the program can be installed in and executed on a general-purpose computer capable of performing various types of processing. For example, the program can be pre-recorded on a recording medium. Besides being installed on a computer from a recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard drive.

[1357] Note that the various types of processing described in this specification can be executed not only sequentially as described, but also in parallel or individually as needed or according to the processing capabilities of the device performing the processing. Furthermore, the system in this specification refers to a logical set configuration of multiple devices. Although in some cases the individual constituent devices are located in a single housing, the system is not limited to a system where the individual constituent devices are located in a single housing. Industrial Applicability

[1358] As described above, according to the configuration of the embodiments of this disclosure, by performing optimal control on the bit rate of each image sent from the autonomous driving control device to the remote control device based on the bandwidth, image transmission with low latency and remote control with low latency are achieved.

[1359] Specifically, for example, the automatic driving control device calculates an estimated bit rate equivalent to the bandwidth available for sending images to the remote control device, calculates the image priority of each camera-captured image among multiple camera-captured images, and only sends high-priority images that are evaluated as being able to be sent with a short delay at the estimated bit rate or a lower bit rate. Based on the image received from the automatic driving control device, the remote assistance device switches the image to be displayed on the display unit to any one of the following: a camera-captured image, a composite image of a camera-captured image and a 3DCG image, or a 3DCG image.

[1360] According to this configuration, by performing optimal control on the bit rate of each image sent from the autonomous driving control unit to the remote control unit based on the bandwidth, image transmission with low latency and remote control with low latency are achieved.

[1361] Reference tag list

[1362] 10 vehicles

[1363] 20 Automatic driving control device

[1364] 21 Sensors

[1365] 22 Data Processing Department

[1366] 23. Automatic Automated Driving Control (ADS)

[1367] 24 Ministry of Communications

[1368] 30 Remote Assistance Center

[1369] 40 Remote assistance devices

[1370] 41 Ministry of Communications

[1371] 42 Data Processing Department

[1372] 43 Display Section

[1373] 44 Input Section

[1374] 45 3D Map Storage Department

[1375] 50 operators

[1376] 111 Risk Level Classification Department

[1377] 112 Communications Network Monitoring Department

[1378] 113 Data Analysis Department

[1379] 114 Communications Control Department

[1380] 121 Map Control Department

[1381] 122 Display Control Unit

[1382] 123 Communications Control Department

[1383] 301 CPU

[1384] 302 ROM

[1385] 303 RAM

[1386] 304 bus

[1387] 305 Input / Output Interface

[1388] 306 Input Section

[1389] 307 Output Section

[1390] 308 Storage Unit

[1391] 309 Ministry of Communications

[1392] 310 drive

[1393] 311 Removable media

[1394] 321 sensor

[1395] 322 Drive Unit

Claims

1. An automatic driving control device, comprising: The communication unit sends multiple camera-captured images captured by multiple cameras installed on the vehicle to the remote assistance device, and receives autonomous driving control information from the remote assistance device. The system includes a data processing unit that performs transmission control for each of the plurality of camera-captured images, wherein the data processing unit performs: processing to calculate an estimated bit rate equivalent to the bandwidth available for transmitting images to the remote control device; processing to calculate the image priority of each of the plurality of camera-captured images; and transmission image selection processing that selects the plurality of camera-captured images in descending order of image priority, and transmits the selected images that are evaluated as capable of being transmitted to the remote control device at the estimated bit rate or a lower bit rate, with a delay equal to or less than a predetermined value.

2. The automatic driving control device according to claim 1, wherein, If the estimated bit rate is equal to or greater than the required bit rate that enables all the plurality of camera-captured images to be transmitted to the remote control device with a delay equal to or less than a predetermined value, the data processing unit performs transmission processing on all the plurality of camera-captured images without performing the transmission image selection processing. If the estimated bit rate is less than the required bit rate that enables all the plurality of camera-captured images to be transmitted to the remote control device with a delay equal to or less than a predetermined value, the data processing unit calculates the required bit rate for each image combination unit selected from the plurality of camera-captured images in descending order of image priority, and transmits the image combination unit having the largest required bit rate among the required bit rates calculated for the image combination unit and equal to or less than the estimated bit rate.

3. The automatic driving control device according to claim 1, wherein, The data processing unit determines the allocation of the transmission bit rate for each of the plurality of camera-captured images to be transmitted to the remote control device in proportion to the resolution of the number of pixels in each camera-captured image selected as the transmission target.

4. The automatic driving control device according to claim 1, wherein, The data processing unit considers at least one of the following to calculate the image priority of each of the plurality of camera-captured images: (a) the number of objects detected in each of the plurality of camera-captured images; Or (b) the number of high-risk objects detected in each of the images captured from the plurality of cameras.

5. The automatic driving control device according to claim 1, wherein, The data processing unit performs transmission bit rate control to set a target bit rate between a predefined minimum bit rate and a maximum bit rate, the target bit rate being the transmission bit rate to be applied when all the multiple camera-captured images are to be transmitted, and sets the target bit rate to the maximum bit rate if the estimated bit rate is equal to or greater than the maximum bit rate.

6. The automatic driving control device according to claim 1, wherein, The data processing unit sends the object image of the object detected in the image captured by the camera to the remote assistance device.

7. The automatic driving control device according to claim 6, wherein, The data processing unit performs the process of detecting objects in unsent images excluded from the transmission target in the transmitted image selection process, and sends the object image of the object detected in the unsent images to the remote assistance device.

8. The automatic driving control device according to claim 6, wherein, When the bit rate for transmitting object images, which is the bandwidth available for transmitting object images, is equal to or greater than the required bit rate for transmitting all object images to the remote control device with a delay equal to or less than a predetermined value, the data processing unit performs the processing of transmitting all object images of the detected object; and when the bit rate for transmitting object images is less than the required bit rate for transmitting all object images to the remote control device with a delay equal to or less than a predetermined value, the data processing unit generates a compressed object image obtained by compressing at least some of the object images, and transmits the compressed object image.

9. The automatic driving control device according to claim 6, wherein, The data processing unit assesses the risk level of the detected objects from the image and selectively sends object images of high-risk objects.

10. The automatic driving control device according to claim 6, wherein, The data processing unit is configured to perform compression processing by applying different compression algorithms based on the risk level of the detected object from the image, and to generate and send a compressed object image by applying a compression algorithm that sets the compression ratio to decrease as the risk level of the object increases and sets the compression ratio to increase as the risk level of the object decreases.

11. The automatic driving control device according to claim 6, wherein, The data processing unit performs object detection processing by performing image segmentation processing on the image.

12. The automatic driving control device according to claim 1, wherein, The communication unit performs image transmission by using a multi-track video stream technique that transmits multiple camera-captured images in a single stream.

13. A remote assistance device, comprising: The communication unit receives images captured by multiple cameras from the autonomous driving control device using multiple cameras installed on the vehicle; The system also includes a data processing unit that performs display control to display the plurality of camera-captured images on a display unit, wherein the data processing unit switches the image to be displayed on the display unit to any one of the following based on the image received from the autonomous driving control device: (a) a camera-captured image received from the autonomous driving control device; (b) a composite image of a camera-captured image and a 3DCG image received from the autonomous driving control device; or (c) a 3DCG image.

14. The remote assistance device according to claim 13, wherein, The data processing unit: when the image received from the autonomous driving control device has high resolution, sets the image to be displayed on the display unit to be a camera-captured image received from the autonomous driving control device; and when no image is received from the autonomous driving control device, sets the image to be displayed on the display unit to be a 3DCG image.

15. The remote assistance device according to claim 13, wherein, The data processing unit controls the opacity of the 3DCG image in the composite image of the camera-captured image and the 3DCG image to generate each image as any one of (a) to (c), and causes the display unit to display the image.

16. The remote assistance device according to claim 15, wherein, The data processing unit controls the opacity of the 3DCG image based on the value of the transmission bit rate of each of the plurality of camera-captured images determined by the autonomous driving control device.

17. The remote assistance device according to claim 13, wherein, The data processing unit displays the object image received from the automatic driving control device on the 3DCG image displayed on the display unit.

18. The remote assistance device according to claim 17, wherein, The data processing unit displays the object image in a display mode that varies according to the object's risk level.

19. The remote assistance device according to claim 13, wherein, Regarding image frames from which an object image cannot be received from the automatic driving control device, the data processing unit displays an object image generated by interpolation processing of the object image using previous and subsequent image frames.

20. An automatic driving control system, comprising an automatic driving control device and a remote assistance device, wherein, The automatic driving control device performs the following processing: calculating an estimated bit rate equivalent to the bandwidth that can be used to send images to the remote control device; The processing of calculating the image priority of each camera capture image among multiple camera capture images captured by multiple cameras mounted on a vehicle; The remote assistance device performs a process of selecting images from the plurality of cameras in descending order of image priority, and sending selected images that are evaluated as capable of being sent to the remote control device at the estimated bit rate or lower, with a delay equal to or less than a predetermined value. The remote assistance device performs a process of switching the image to be displayed on the display unit based on the image received from the autonomous driving control device to any of the following: (a) a camera-captured image received from the autonomous driving control device; (b) a composite image of a camera-captured image and a 3DCG image received from the autonomous driving control device; or (c) a 3DCG image. It also sends control information generated based on the inspection results of the displayed image to the autonomous driving control device.

Images