Automatic driving control device, remote support device, and automatic driving control system

Abstract

The present invention achieves low-delay image transmission and low-delay remote control by optimally controlling, in accordance with a band, the bit rate of each transmission image from an automatic driving control device to a remote control device. The automatic driving control device calculates an estimated bit rate that corresponds to a band which is available for image transmission to the remote control device, calculates an image priority for each image captured by a plurality of cameras, and transmits only a high-priority image for which low-delay transmission at not more than the estimated bit rate is determined to be possible. A remote support device switches, in accordance with an image received from the automatic driving control device, an image to be displayed on a display unit to one of a camera-captured image, a composite image of the camera-captured image and a 3D CG image, and the 3D CG image, and displays said one image.

Description

Autonomous driving control device, remote assistance device, and autonomous driving control system

[0001] The present disclosure relates to an autonomous driving control device, a remote assistance device, and an autonomous driving control system. Specifically, the present disclosure relates to an autonomous driving control device, a remote assistance device, and an autonomous driving control system in which an autonomous vehicle transmits images captured by a camera to an external driving assistance device, and an operator of the driving assistance device checks the images and controls the vehicle.

[0002] Recently, technological development related to automated driving has progressed rapidly, such as the so-called automated driving system (ADS) and advanced driver assistance system (ADAS).

[0003] In autonomous driving, cameras and laser-based sensors installed in vehicles (automobiles) are used to detect obstacles and to achieve safe driving using the detected information. Cameras and other sensors detect various objects around the vehicle, such as oncoming vehicles, pedestrians, and guardrails, and the vehicle's autonomous driving execution unit determines a safe driving route that avoids collisions with these objects and drives the vehicle autonomously.

[0004] The Society of Automotive Engineers (SAE), an organization that sets standards for automotive technology, has established the following levels as international standards for automated driving: Level 0 = No driving automation Level 1 = Driving assistance Level 2 = Partial driving automation Level 3 = Conditional driving automation Level 4 = High driving automation Level 5 = Full driving automation

[0005] Currently, autonomous vehicles are approaching Level 4, which requires almost no human operation, and Level 5, which is fully autonomous. However, even vehicles capable of Level 4 or 5 autonomous driving may encounter complex situations that the vehicle cannot solve on its own. Specifically, if an accident occurs, such as a collision or contact caused by erroneous driving by another vehicle, or if the vehicle runs onto a sidewalk due to an accident, the autonomous vehicle may be unable to continue driving on its own. In order to perform appropriate driving control in such situations, control by an external operator is required.

[0006] When controlled externally, the autonomous vehicle transmits images captured by a camera mounted on the vehicle to a remote support center via a communication network. An operator at the remote support center checks the images displayed on the display, determines the optimal control, and transmits the determined control information to the vehicle to control its driving.

[0007] When remote support is provided by an external operator, wireless communication between the vehicle and the remote support center is essential. Images captured by the vehicle's camera are sent to the remote support center via wireless communication. However, wireless communication is more likely to result in poorer communication quality than wired communication. For example, there are problems such as communication delays and packet loss.

[0008] Near-real-time control is necessary to ensure the safe operation of autonomous vehicles. For example, if there is a delay in transmitting camera images, the operator's control may be delayed, which could result in new accidents.

[0009] Furthermore, if a drop in communication quality prevents high-resolution images from being transmitted, the image displayed on the display unit at the remote support center becomes unclear. In such cases, the operator's monitoring burden increases, and operator fatigue builds up, increasing the likelihood of operational errors.

[0010] One possible approach to solving the problem of image data communication delays is to provide remote support by transmitting CG images, which have a smaller data volume than camera-captured images. For example, Patent Document 1 (JP 2007-323481 A) discloses a configuration in which camera-captured images are converted into CG images and transmitted. However, if such CG conversion processing is performed on the vehicle side, processing time is required for the CG conversion, which makes it difficult to transmit data in real time, resulting in the problem of unavoidable delays in operator control.

[0011] Japanese Patent Application Laid-Open No. 2007-323481

[0012] The present disclosure aims to solve, for example, the above-mentioned problems, and to provide an autonomous driving control device, a remote assistance device, and an autonomous driving control system that enable remote assistance to autonomous vehicles to be performed reliably with low latency.

[0013] A first aspect of the present disclosure is an autonomous driving control device having a communication unit that transmits multiple camera images taken by multiple cameras mounted on a vehicle to a remote support device and receives autonomous driving control information from the remote support device, and a data processing unit that executes transmission control for each of the multiple camera images, wherein the data processing unit executes a process of calculating an estimated bit rate equivalent to the bandwidth available for transmitting images to the remote control device, a process of calculating an image priority for each of the multiple camera images, and a transmission image selection process that selects the multiple camera images in descending order of image priority and transmits selected images that are determined to be transmittable to the remote control device at or below the estimated bit rate and with a delay amount below a predetermined value.

[0014] Furthermore, a second aspect of the present disclosure is a remote assistance device that has a communication unit that receives multiple camera images taken by multiple cameras mounted on a vehicle from an autonomous driving control device, and a data processing unit that performs display control to display the multiple camera images on a display unit, wherein the data processing unit switches and displays an image to be displayed on the display unit in response to the image received from the autonomous driving control device between: (a) a camera image received from the autonomous driving control device, or (b) a composite image of a camera image received from the autonomous driving control device and a 3DCG image, or (c) a 3DCG image, or

[0015] Furthermore, a third aspect of the present disclosure is an autonomous driving control system having an autonomous driving control device and a remote assistance device, wherein the autonomous driving control device performs a process of calculating an estimated bit rate equivalent to a bandwidth available for transmitting images to the remote control device, a process of calculating an image priority for each of a plurality of camera images taken by a plurality of cameras mounted on a vehicle, and a transmission image selection process of selecting the plurality of camera images in descending order of image priority and transmitting selected images that are determined to be transmittable to the remote control device at or below the estimated bit rate and with an amount of delay not greater than a predetermined value, and the remote assistance device performs a process of switching and displaying an image to be displayed on a display unit according to an image received from the autonomous driving control device to any of the images (a) to (c) above, (a) a camera image received from the autonomous driving control device, or (b) a composite image of a camera image and a 3DCG image received from the autonomous driving control device, or (c) a 3DCG image, and transmits control information generated from the confirmation result of the displayed image to the autonomous driving control device.

[0016] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present invention and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0017] According to one embodiment of the present disclosure, low-latency image transmission and low-latency remote control are achieved by optimally controlling the bit rate of each image transmitted from the autonomous driving control device to the remote control device according to the bandwidth. Specifically, for example, the autonomous driving control device calculates an estimated bit rate corresponding to the bandwidth available for image transmission to the remote control device, calculates image priority for each of multiple camera-captured images, and transmits only high-priority images determined to be capable of low-latency transmission at or below the estimated bit rate. The remote assistance device switches the image displayed on the display unit between a camera-captured image, a composite image of a camera-captured image and a 3DCG image, or a 3DCG image, depending on the image received from the autonomous driving control device. This configuration achieves low-latency image transmission and low-latency remote control by optimally controlling the bit rate of each image transmitted from the autonomous driving control device to the remote control device according to the bandwidth. Note that the effects described in this specification are merely exemplary and are not limiting, and additional effects may also be present.

[0018] FIG. 11 is a diagram illustrating an example of an autonomous driving control system according to the present disclosure. FIG. 12 is a diagram illustrating an example of the configuration of an autonomous driving control device of an autonomous driving vehicle and an example of the configuration of a remote support device of a remote support center. FIG. 13 is a diagram illustrating an example of mounting a sensor on an autonomous driving vehicle. FIG. 14 is a diagram illustrating an example of an image (view) captured by a camera of an autonomous driving vehicle. FIG. 15 is a diagram illustrating the configuration of a data processing unit of an autonomous driving control device of an autonomous driving vehicle and the processing executed by it. FIG. 16 is a diagram illustrating the configuration of a data processing unit of a remote support device of a remote support center and the processing executed by it. FIG. 17 is a diagram illustrating a specific example of a 3DCG image generated by a map control unit of a data processing unit of the autonomous driving control device. FIG. 18 is a diagram illustrating the configuration of a data transmission processing from an autonomous driving control device to a remote support device. FIG. 19 is a diagram illustrating a flowchart illustrating the sequence of a transmission data control processing executed by a data processing unit of the autonomous driving control device. FIG. 19 is a diagram illustrating the definition of each parameter used in the flowcharts shown in FIGS. 9 and 10. ｅ) and the target bit rate b, which is the bit rate used for actual data transmission. ｔａｒｇｅｔ14. A diagram illustrating the time progression of. A diagram illustrating a specific example of transmission bit rate allocation processing for each of captured images (video tracks) 1 to n. A diagram illustrating a flowchart illustrating the details of the arrangement processing of camera-captured images according to priority and the transmission image selection processing according to priority, which are executed by the data processing unit of the automatic driving control device of the automatic driving vehicle. A diagram illustrating the definition of parameters used in the flowchart shown in FIG. 14. A diagram illustrating a specific example of image arrangement processing according to image priority. A diagram illustrating a specific example of a gaze heat map showing the degree of gaze concentration of the operator at the remote support center. A diagram illustrating an example of analysis processing of (a) the number of surrounding objects and (b) the number of high-risk objects. A diagram illustrating a method for calculating the risk level of an object using a neural network. A diagram illustrating a specific example of image selection and transmission processing based on image priority. A diagram illustrating a specific example of image selection and transmission processing based on image priority. A diagram illustrating a specific example of processing to generate and display 3DCG images observed from the position and direction of a camera mounted on an automatic driving vehicle using a 3D map. A diagram illustrating a flowchart illustrating the sequence of communication control processing for large-volume metadata, such as images of objects, executed by the automatic driving control device. A diagram illustrating a specific example of image segmentation processing. FIG. 28 is a diagram illustrating an example of array data in which extracted objects are arranged in order of decreasing risk from risk level 1 to risk level 4. FIG. 29 is a diagram illustrating specific examples of different compression processes according to the risk level of an object. FIG. 30 is a diagram illustrating a flowchart illustrating a specific example of compression process according to the risk level of an object and filtering process (processing for selecting a transmission object). FIG. 31 is a diagram illustrating a flowchart illustrating a specific example of compression process according to the risk level of an object and filtering process (processing for selecting a transmission object). FIG. 32 is a diagram illustrating the definition of each parameter used in the flowcharts shown in FIGS. 27 and 28. FIG. 33 is a diagram illustrating an example of an image displayed on a display unit by a remote assistance device using transmission data from an autonomous driving control device of an autonomous driving vehicle.34 and 35. FIG. 35 is a diagram illustrating the opacity of a 3DCG image, which is an adjustment parameter for the composition ratio between the 3DCG image and a received image. FIG. 36 is a diagram illustrating an example of composition processing between a 3DCG image and a received image according to a setting value of the opacity of the 3DCG image. FIG. 37 is a diagram illustrating an example of composition processing between a 3DCG image and a received image according to a setting value of the opacity of the 3DCG image. FIG. 38 is a diagram illustrating a flowchart for controlling the opacity of a 3DCG image executed by the data processing unit of the remote support device. FIG. 39 is a diagram illustrating definitions of parameters described in the flowcharts shown in FIGS. 34 and 35. FIG. 39 is a diagram illustrating object display processing according to the risk level of an object executed by the display control unit of the data processing unit of the remote support device. FIG. 39 is a diagram illustrating a specific example of object display processing according to the risk level of an object executed by the display control unit of the data processing unit of the remote support device. FIG. 39 is a diagram illustrating an example of interpolation processing executed by the display control unit 122 of the data processing unit 42 of the remote support device. FIG. 1 is a diagram illustrating an example of interpolation processing executed by a display control unit 122 of a data processing unit 42 of a remote assistance device 40. FIG. 2 is a diagram illustrating a data transmission processing configuration from an autonomous driving control device to a remote assistance device. FIG. 3 is a diagram illustrating an example configuration in which an autonomous driving control device of one autonomous driving vehicle transmits camera-captured images to multiple remote assistance devices. FIG. 4 is a diagram illustrating an example configuration in which multiple autonomous driving vehicles a to c each transmit camera-captured images by a camera mounted on each vehicle to the same single remote assistance device 4. FIG. 5 is a diagram illustrating an example hardware configuration of an autonomous driving control device and a remote assistance device of the present disclosure.

[0019] The following describes in detail the autonomous driving control device, remote assistance device, and autonomous driving control system of the present disclosure with reference to the drawings. The description will be made according to the following items: 1. Overview of the autonomous driving control device, remote assistance device, and autonomous driving control system of the present disclosure 2. Configurations of the autonomous driving control device and remote assistance device 3. Data transmission process from the autonomous driving control device to the remote assistance device 4. Transmission data control process executed by the autonomous driving control device 5. Details of the process for selecting and transmitting camera-captured images according to priority 6. Details of the transmission process of objects detected from non-transmitted images 7. Process executed by the remote assistance device 8. Details of the process for combining (superimposing) camera-captured images and 3DCG images executed by the remote assistance device 9. Details of the object display control process executed by the remote assistance device 10. Other embodiments 11. Hardware configuration examples of the autonomous driving control device and the remote assistance device 12. Summary of the configuration of the present disclosure

[0020] [1. Overview of the Autonomous Driving Control Device, Remote Support Device, and Autonomous Driving Control System Disclosed Herein] First, an overview of the autonomous driving control device, remote support device, and autonomous driving control system disclosed herein will be described.

[0021] 1 is a diagram illustrating an example of an autonomous driving control system according to the present disclosure. The autonomous driving vehicle 10 is a vehicle that travels autonomously. The autonomous driving vehicle 10 is equipped with an autonomous driving control device 20.

[0022] The autonomous driving control device 20 performs autonomous driving using detection information from sensors such as cameras and LiDAR (Laser Imaging Detection and Ranging) provided in the autonomous driving vehicle 10. Note that LiDAR is an object detection sensor that uses laser light.

[0023] However, there are cases where the autonomous vehicle 10 encounters a complex situation that it cannot solve on its own. For example, in cases where an accident occurs due to erroneous driving by another vehicle, the vehicle runs onto a sidewalk, the vehicle is traveling through a road construction area, there is a landslide on the road shoulder, an unexpected obstacle is encountered, an accident occurs, or the vehicle is unable to continue traveling on its own. In such cases, the autonomous driving control device 20 inputs control information from an external operator and performs autonomous driving.

[0024] The autonomous driving control device 20 communicates with an external remote support center 30 via a wireless communication base station installed on a road or the like, and transmits images captured by the camera, LiDAR detection information, etc. to the remote support center 30.

[0025] The remote assistance center 30 has an operator 50 who provides driving assistance to the autonomously driven vehicle 10, and also has a remote assistance device 40 equipped with a display unit. The remote assistance device 40 displays, on the display unit, images captured by a camera and transmitted from the autonomous driving control device 20 of the autonomously driven vehicle 10.

[0026] The operator 50 analyzes the image displayed on the display unit of the remote assistance device 40, determines the optimal driving mode for the autonomous vehicle 10, and transmits autonomous driving control information to the autonomous driving control device 20 of the autonomous vehicle 10 to cause the vehicle to drive in accordance with the determined driving mode.

[0027] The autonomous driving control device 20 of the autonomous driving vehicle 10 receives the autonomous driving control information from the remote assistance device 40 and drives the vehicle in accordance with the received information.

[0028] In this way, the autonomous driving control device 20 of the autonomous driving vehicle 10 performs autonomous driving control using detection information from sensors such as cameras and LiDAR equipped on the autonomous driving vehicle 10, and also drives using autonomous driving control information received from the remote assistance device 40.

[0029] Communication between the autonomous driving control device 20 of the autonomous vehicle 10 and the remote assistance device 40 of the remote assistance center 30 is performed by wireless communication. However, as described above, it is more difficult to maintain communication quality with wireless communication than with wired communication. Specifically, there are problems such as a high likelihood of communication delays and communication packet loss. To safely drive an autonomous vehicle, near-real-time control is required, and delays in operator control due to delays in image transmission, etc., could result in an accident.

[0030] In the system disclosed herein, the autonomous driving control device 20 of the autonomous driving vehicle 10 monitors the wireless communication network, calculates bandwidth information available for communication, and performs various transmission data control processes such as adjusting the transmission bit rate, selecting transmission data, and compressing transmission data according to the calculated available bandwidth.

[0031] The remote assistance device 40 in the remote assistance center 30 receives data transmitted from the autonomous driving control device 20 of the autonomously driven vehicle 10 and displays the received data, for example, camera images, on a display unit. However, it is conceivable that some camera images may not be transmitted. In such cases, the remote assistance device 40 performs processes such as data interpolation and data restoration using pre-stored 3D map CG images to generate a more complete image of the vehicle's surroundings and display it on the display unit.

[0032] Furthermore, if the remote assistance device 40 cannot receive camera images from the autonomous driving control device 20 of the autonomous driving vehicle 10, or if the images are excessively blurry, it performs a process of superimposing and displaying object images of vehicles and other objects around the autonomous driving vehicle 10 on a 3DCG image using object information such as oncoming vehicles contained in the metadata received from the autonomous driving control device 20.

[0033] The remote assistance device 40 executes such processing to generate display data that more accurately reproduces the current surrounding conditions of the autonomously driven vehicle 10 and displays the data on the display unit.

[0034] An operator 50 at the remote support center 30 checks the camera images, 3DCG images, or a composite image of these displayed on the display unit of the remote support device 40, analyzes the situation around the autonomously driven vehicle 10, determines the optimal driving control that the autonomously driven vehicle 10 should perform, generates autonomous driving control information in accordance with the determined information, and transmits it to the autonomously driven vehicle 10.

[0035] By performing such processing, the system disclosed herein can display a highly accurate image of the surroundings of the autonomously driven vehicle 10 on the display unit of the remote assistance device 40, even when communication delays or communication packet loss occur in wireless communication. As a result, the operator 50 can reliably transmit correct autonomous driving control information to the autonomously driven vehicle 10 with low latency.

[0036] 2. Configuration of the Automatic Driving Control Device and the Remote Support Device Next, the configurations of the automatic driving control device and the remote support device will be described.

[0037] FIG. 2 is a diagram showing an example of the configuration of the automatic driving control device 20 of the automatic driving vehicle 10 and an example of the configuration of the remote assistance device 40 of the remote assistance center 30.

[0038] As shown in FIG. 2 , the autonomous driving control device 20 of the autonomous driving vehicle 10 includes a sensor (camera, LiDAR, etc.) 21, a data processing unit 22, an autonomous driving control unit (ADS (Automated Driving System)) 23, and a communication unit 24.

[0039] The remote support device 40 of the remote support center 30 includes a communication unit 41 , a data processing unit 42 , a display unit 43 , an input unit 44 , and a 3D map storage unit 45 .

[0040] First, the configuration of the autonomous driving control device 20 of the autonomous driving vehicle 10 will be described. The sensor (camera, LIDAR (LiDAR), etc.) 21 is a sensor such as a camera or LiDAR mounted on the autonomous driving vehicle 10. An example of how the sensors are mounted will be described with reference to FIG. 3. In the example shown in FIG. 3, the cameras are configured with the following six cameras: a forward-facing camera 21FC, a right-facing camera 21RC, a right-rearward-facing camera 21RBC, a rearward-facing camera 21BC, a left-rearward-facing camera 21LBC, and a left-facing camera 21LC. These six cameras continuously capture images (moving images) in their respective directions and input the captured images to the data processing unit 22.

[0041] Furthermore, the LiDAR (lidar), which is an object sensor using laser light, is composed of two LiDARs: a forward LiDAR 21FL and a rearward LiDAR 21BL. These two LiDARs acquire object detection information in each direction (forward and rearward), respectively, and input the acquired information to the data processing unit 22. Note that the sensor mounting example shown in Figure 3 is just one example, and a sensor mounting configuration different from the configuration shown in this example, such as a configuration in which more cameras are mounted or a configuration in which sensors other than LiDAR are mounted, may also be used.

[0042] Figure 4 shows an example of images (views) taken by the six cameras shown in Figure 3. Figure 4 shows the following images (views) taken by each of the six cameras while the autonomously driven vehicle 10 is traveling on a road: (LBv) Left back view (image taken by the left rearward camera) (Bv) Back view (image taken by the rearward camera) (RBv) Right back view (image taken by the right rearward camera) (Lv) Left side view (image taken by the left direction camera) (Fv) Front view (image taken by the frontward camera) (Rv) Right side view (image taken by the right direction camera)

[0043] These images are input to the data processing unit 22 of the automatic driving control device 20 shown in Figure 2 and are used for automatic driving control performed by the automatic driving control unit (ADS) 23. Furthermore, these images are also transmitted to the remote support device 40 of the remote support center 30 by wireless communication via the communication unit 24.

[0044] However, depending on the wireless communication network conditions, for example, the bandwidth available for wireless communication, it may be difficult to transmit all six pieces of image data with low latency (a delay amount equal to or less than a predetermined value). The data processing unit 22 of the autonomous driving control device 20 of the present disclosure monitors the wireless communication network, calculates bandwidth information available for communication, and performs various transmission data control processes, such as adjusting the transmission bit rate of each of the images captured by the multiple cameras, selecting the transmission data, and compressing the transmission data, according to the calculated available bandwidth. Specific examples of this process will be described later.

[0045] Returning to Figure 2, we will continue to explain the configuration of the autonomous driving control device 20. The data processing unit 22 receives sensor detection information from the sensor 21, such as a camera or LiDAR, and inputs the input sensor detection information, such as an image captured by the camera, to the autonomous driving control unit (ADS) 23.

[0046] The autonomous driving control unit (ADS) 23 performs autonomous driving by determining a safe driving route, driving speed, etc. for the autonomous driving vehicle 10 based on the sensor detection information input from the data processing unit 22. The autonomous driving control unit (ADS) 23 also performs driving control by utilizing autonomous driving control information from the driving assistance device 40 input via the communication unit 24.

[0047] Next, a description will be given of the configuration of the remote support device 40 of the remote support center 30. As shown in FIG.

[0048] The communication unit 41 communicates with the autonomous driving control device 20 of the autonomous driving vehicle 10. The communication unit 41 receives camera-captured images from the autonomous driving control device 20. The communication unit 41 also receives metadata including object information detected by the camera, LiDAR, etc. The received information is input to the data processing unit 42.

[0049] The data processing unit 42 analyzes the camera-captured images and metadata received from the automatic driving control device 20, and generates image data to be output (displayed) on the display unit 43. For example, if the camera-captured images received from the automatic driving control device 20 are complete six-directional images as described with reference to FIG.

[0050] The operator 50 observes the image displayed on the display unit 43 to check the current surrounding conditions of the autonomous vehicle 10, inputs optimal control information (autonomous driving control information) via the input unit 44, and transmits it to the autonomous vehicle 10.

[0051] The automatic driving control device 20 of the automatically driven vehicle 10 receives the automatic driving control information transmitted by the remote support device 40 of the remote support center 30 via the communication unit 24, and inputs it to the automatic driving control unit (ADS) 23. The automatic driving control unit (ADS) 23 executes automatic driving in accordance with the input automatic driving control information. In other words, the vehicle travels in accordance with the driving control determined by the operator 50.

[0052] However, as mentioned above, depending on the state of the communication network, it may not be possible to receive complete six-directional camera images from the automatic driving control device 20.

[0053] In such a case, the data processing unit 42 of the remote assistance device 40 outputs the 3DCG image acquired from the 3D map storage unit 45 to the display unit 43. Alternatively, the data processing unit 42 generates a composite image of the blurred camera image received from the automatic driving control device 20 and the 3DCG image, and outputs the composite image to the display unit 43.

[0054] The data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 shown in Figure 2 to generate and display a 3DCG image observed from the position and direction of the camera attached to the autonomously driven vehicle 10.

[0055] In the example shown in FIG. 2, the 3D map storage unit 45 is configured within the remote support device 40, but the 3D map may be acquired from an external server.

[0056] It should be noted that the 3DCG images generated from the 3D maps stored in the 3D map storage unit 45 do not include real-time surrounding objects such as oncoming vehicles and pedestrians that exist around the autonomous vehicle 10.

[0057] For this reason, the data processing unit 42 analyzes metadata including information on surrounding objects such as oncoming vehicles received from the automatic driving control device 20, and performs processing such as displaying images of objects around the automatically driven vehicle 10 on a 3DCG image. By performing such processing, display data that more accurately reproduces the current surrounding situation of the automatically driven vehicle 10 is generated and displayed on the display unit.

[0058] An operator 50 at the remote assistance center 30 checks the camera image, the 3DCG image, or a composite image of these, and also images of objects around the autonomously driven vehicle 10, etc., displayed on the display unit 43 of the remote assistance device 40. The operator 50 looks at the images displayed on the display unit 43, analyzes the situation around the autonomously driven vehicle 10, determines the optimal driving control that should be executed by the autonomously driven vehicle 10, generates autonomous driving control information in accordance with the determined information, and transmits it to the autonomously driven vehicle 10.

[0059] The autonomous driving control unit (ADS) 23 of the autonomous driving control device 20 of the autonomous driving vehicle 10 controls the driving of the vehicle in accordance with the autonomous driving control information received from the remote assistance device 40. In other words, the autonomous driving vehicle 10 drives in accordance with the control mode determined by the operator 50.

[0060] Next, the configuration of the data processing unit 22 of the autonomous driving control device 20 of the autonomous driving vehicle 10 and the processing executed therein will be described in detail with reference to FIG. 5 .

[0061] Fig. 5 is a diagram showing a detailed configuration of the data processing unit 22 of the automatic driving control device 20. As shown in Fig. 5, the data processing unit 22 of the automatic driving control device 20 has the following components: a risk level classification unit (Risk Level Classification Manager) 111, a communication network monitoring unit (Network Monitoring Manager) 112, a data analysis unit (Data Manager) 113, and a communication control unit (Data Transmission Manager) 114. The data processing unit 22 of the automatic driving control device 20 has these components.

[0062] The risk level classification unit 111 determines the priority of images from the six cameras described with reference to Fig. 3, i.e., the priority of image transmission to the remote support center. Furthermore, it analyzes the risk levels of various objects captured in the images, such as surrounding vehicles, pedestrians, and buildings. Specifically, it executes processing to determine the risk of collision between each object and the autonomously driven vehicle 10.

[0063] 5, the risk level classification unit 111 includes the following modules: an image segmentation module 111a, an object extraction module 111b, a raw sensor data processing module 111c, a video priority classification module 111d, and an extracted object risk level classification module 111e.

[0064] The image segmentation module 111a segments an image, i.e., an image captured by a camera, into regions. The object extraction module 111b extracts various objects, such as vehicles, bicycles, pedestrians, and other structures, from each segmented region.

[0065] The sensor detection data analysis module 111c uses sensor detection data such as LiDAR to analyze the distance to the object extracted by the object extraction module 111b, the direction of movement, the speed, and the like.

[0066] Specifically, when various sensors such as cameras, LiDAR, and other sensors such as radar and sonar are mounted on the autonomously driven vehicle 10, the sensor detection data analysis module 111c analyzes data (raw data) detected by these sensors. The output of the sensor detection data analysis module 111c is composed of data such as image frames captured by the cameras, coordinates of detected objects with the position of the autonomously driven vehicle 10 as the origin, and 3D bounding boxes that are three-dimensional boxes that surround the objects.

[0067] The output of the sensor detection data analysis module 111c is used as input data for other modules. For example, each image frame captured by the camera is input to the image segmentation module 111a, which performs image region segmentation processing. The coordinates and 3D bounding boxes of detected objects are input to the extracted object risk level analysis module 111e (described later) and used in the extracted object risk level analysis module 111e's process for determining the risk level of the object.

[0068] The image priority analysis module 111d performs processing such as determining the priority of images from each camera, for example, the six cameras described with reference to Figures 3 and 4, that is, the priority of image transmission to the remote support center. This priority determination is performed, for example, based on the number of objects captured in each image, the risk of each object, etc. Specific image priority determination processing will be described later.

[0069] The extracted object risk level analysis module 111e analyzes the risk levels of the objects extracted from each divided region of each image by the object extraction module 111b and determines the risk level for each object. Specifically, the extracted object risk level analysis module 111e determines the risk of collision, etc., for the autonomously driven vehicle 10, and determines the risk level according to the determined risk.

[0070] The risk level for each detected object determined by the extracted object risk level analysis module 111e is transmitted as metadata to the remote support device 40 at the remote support center 30. The remote support device 40 performs processing to determine whether or not to output an object CG image to the display unit 43, depending on the risk level for each detected object received as metadata. Details and specific processing examples of this risk level analysis processing and object display control processing depending on the risk level will be described later.

[0071] The risk level classification unit 111 uses these modules to determine the priority of image transmission to the remote support center and to analyze the risk levels of various objects captured in the images, such as surrounding vehicles, pedestrians, buildings, etc. Specific examples and details of these processes will be described later.

[0072] The communication network monitoring unit 112 monitors the wireless communication network between the autonomous driving control device 20 of the autonomous driving vehicle 10 and the remote support device 40 of the remote support center 30. Specifically, it performs processes such as estimating the bandwidth (bit rate) available for communication, estimating jitter, and estimating the frame rate of each transmitted image.

[0073] 5, the communication network monitoring unit 112 includes the following modules: a bit rate estimation module 112a, a frame rate estimation module 112b, and a jitter estimation module 112c. The communication network monitoring unit 112 uses these modules to perform the above-described processing.

[0074] The data analysis unit 113 executes a process of allocating the transmission bit rate of each of a plurality of images transmitted from the autonomous driving control device 20 of the autonomous driving vehicle 10 to the remote assistance device 40 of the remote assistance center 30. Specifically, for example, when transmitting six images corresponding to the six cameras described above with reference to Figures 3 and 4 to the remote assistance device 40, the data analysis unit 113 determines the transmission bit rate of each image.

[0075] Furthermore, if the data analysis unit 113 determines, based on the network monitoring results of the communication network monitoring unit 112, that sufficient communication bandwidth cannot be secured, it also performs processing such as selecting images to be sent from the automatic driving control device 20 to the remote assistance device 40, compressing object image data of vehicles and the like captured in the image, and selecting object data to be sent.

[0076] 5, the data analysis unit 113 includes the following modules: a bitrate allocation module 113a, a target bitrate control module 113b, a video usage control module 113c, an object data compression module 113d, and an object data filtering module 113e. The data analysis unit 113 uses these modules to perform the above-described processing.

[0077] The communication control unit 114 controls wireless communication between the autonomous driving control device 20 of the autonomous driving vehicle 10 and the remote assistance device 40 of the remote assistance center 30. Specifically, the communication control unit 114 performs various communication control processes required to execute wireless communication, such as selection and control of frequencies to be used and allocation of communication resources.

[0078] 5, the communication control unit 114 has the following modules: a frequency selection module 114a, a data transmission frequency control module 114b, and a radio resource allocation module 114c. The communication control unit 114 uses these modules to perform the above-described processing.

[0079] In this way, the data processing unit 22 of the automatic driving control device 20 has components such as a risk level classification unit 111, a communication network monitoring unit 112, a data analysis unit 113, and a communication control unit 114, and uses these components to perform control processing of data transmitted to the remote support device 40 of the remote support center 30.

[0080] In other words, depending on the priority of the transmitted data and the state of the network, the system allocates transmission bandwidth (bit rate) to the video track, which is the transmission track for images captured by each camera, and controls the activation and deactivation of the video track, thereby achieving stable image transmission even when the available bandwidth becomes narrow.

[0081] Furthermore, by limiting and setting the fluctuation range of the bit rate used for image transmission in advance, stable image transmission is achieved even if the available bandwidth fluctuates suddenly. Also, when the available bandwidth is narrow and non-transmitted images that will not be transmitted (disabled video tracks) are set, images of objects such as oncoming vehicles detected from the non-transmitted images are generated and transmitted, or compressed object images are generated and transmitted. Details of these processes and specific processing examples will be described later.

[0082] Next, the configuration of the data processing unit 42 of the remote support device 40 in the remote support center 30 and the processing executed therein will be described in detail with reference to FIG.

[0083] 6 is a diagram showing a detailed configuration of the data processing unit 42 of the remote support device 40. As shown in FIG. 6, the data processing unit 42 of the remote support device 40 has the following components: a map control unit (Map Manager) 121, a display control unit (Visualization Manager) 122, a communication control unit (Data Transmission Manager) 123,

[0084] When the camera images that make up the transmission data of the autonomous driving control device 20 of the autonomous driving vehicle 10 are insufficient, for example, when camera images in a specific direction are not transmitted, or when the image transmission bit rate is low and low-accuracy images are received, the map control unit 121 acquires the 3D map stored in the 3D map storage unit 42 and performs processing such as generating a 3DCG image that matches the current camera position and direction of the autonomous driving vehicle 10.

[0085] FIG. 7 shows a specific example of a 3DCG image generated by the map control unit 121. FIG. 7 is an example of a 3DCG image corresponding to the six images previously described with reference to FIG. 4. That is, it is an example of a 3DCG image observed from the position and direction of the cameras that captured the images (views) captured by the six cameras shown in FIG. 3. FIG. 7 shows the following six 3DCG images arranged side by side: (LB-3DCG) Left back view 3DCG image (CG corresponding to image captured by left rear camera) (B-3DCG) Back view 3DCG image (CG corresponding to image captured by rear camera) (RB-3DCG) Right back view 3DCG image (CG corresponding to image captured by right rear camera) (L-3DCG) Left side view 3DCG image (CG corresponding to image captured by left camera) (F-3DCG) Front view 3DCG image (CG corresponding to image captured by front camera) (R-3DCG) Right side view 3DCG image (CG corresponding to image captured by right camera)

[0086] The map control unit 121 uses the 3D map stored in the 3D map storage unit 42 in this manner to perform processes such as generating a 3DCG image that matches the current camera position and direction of the autonomously driven vehicle 10.

[0087] As shown in Fig. 6, the map control unit 121 has the following modules: a 3D map update module (3D Map Update Module) 121a, and a 3DCG view capture module (3DCG view capture module) 121b. The map control unit 121 executes the above-mentioned processing using these modules. Details and specific examples of the processing executed by the map control unit 121 will be described later.

[0088] The display control unit 122 controls the images to be displayed on the display unit 43. That is, the display control unit 122 controls the display of the camera-captured images received from the autonomous driving control device 20 of the autonomous driving vehicle 10.

[0089] Furthermore, when the camera-captured image received from the automatic driving control device 20 contains insufficient image data, the display control unit 122 performs a synthesis process, such as data interpolation or data restoration, between the camera-captured image received from the automatic driving control device 20 and the 3DCG image generated by the map control unit 121, and outputs the result to the display unit 43. Furthermore, using metadata consisting of information about surrounding objects such as oncoming vehicles received from the automatic driving control device 20, the display control unit 122 performs a process such as displaying images of objects around the automatic driving vehicle 10 on the 3DCG image.

[0090] As shown in FIG. 6 , the display control unit 122 includes the following modules: an image overlay control module (View Overlap Control Module) 122a, a data extrapolation / interpolation processing module (Data Extrapolation / Interpolation Module) 122b, an object image visualization module (Objects Visualization Module) 121c, and an image transparency control module (View Transparency Control Module) 121d. The display control unit 122 executes the above-described processing using these modules. Details and specific examples of the processing executed by the display control unit 122 will be described later.

[0091] The communication control unit 123 controls wireless communication between the autonomous driving control device 20 of the autonomous driving vehicle 10 and the remote assistance device 40 of the remote assistance center 30. Specifically, the communication control unit 123 performs various communication control processes required to execute wireless communication, such as selection and control of frequencies to be used and allocation of communication resources.

[0092] 6 , the communication control unit 123 includes the following modules: a frequency selection module 123a, a data transmission frequency control module 123b, and a radio resource allocation module 123c. The communication control unit 123 uses these modules to perform the above-described processing.

[0093] In this way, the data processing unit 42 of the remote assistance device 40 has components such as a map control unit 121, a display control unit 122, and a communication control unit 123, and uses these components to perform various processes on data received from the autonomous driving control device 20 of the autonomous driving vehicle 10.

[0094] That is, the data processing unit 42 of the remote assistance device 40 controls the display data for the display unit 43 via the display control unit 122. Specifically, as described above, a 3DCG image from the viewpoint of the camera of the autonomously driven vehicle 10 is generated, and the 3DCG image is superimposed on the image captured by the camera as needed. Furthermore, processing is also performed to superimpose object images of oncoming vehicles and the like on the 3DCG. Details of these processes and specific processing examples will be described later.

[0095] 3. Data Transmission Process from the Automatic Driving Control Device to the Remote Support Device Next, a data transmission process from the automatic driving control device to the remote support device will be described.

[0096] 8 is a diagram illustrating a configuration for processing data transmission from an autonomous driving control device to a remote assistance device. The autonomous driving control device 20 of the autonomous driving vehicle 10 is shown on the left side of Fig. 8, and the remote assistance device 40 of the remote assistance center 30 is shown on the right side.

[0097] The autonomous driving control device 20 of the autonomous driving vehicle 10 inputs sensor acquired values ​​from a sensor 21 configured with a camera or LiDAR, specifically, images captured by the camera and object detection information by the LiDAR, to a data processing unit 22. The data processing unit 22 has a risk level classification unit 111, a communication network monitoring unit 112, a data analysis unit 113, and a communication control unit 114, which were previously described with reference to FIG. 5 .

[0098] The data processing unit 22 uses these components to analyze images captured by the camera and object detection information obtained by LiDAR. It also estimates the bandwidth (bit rate) available for data transmission based on monitoring information of the wireless communication network between the autonomous driving control device 20 and the remote support device 40 in the remote support center 30.

[0099] Furthermore, the data processing unit 22 controls the bit rate of the transmission data in accordance with the estimated available bandwidth (bit rate). Specifically, the data processing unit 22 performs processing such as determining the transmission bit rate for each of the six images corresponding to the six cameras, as previously described with reference to Figures 3 and 4, and transmits each image to the remote assistance device 40 via the communication unit 24 at the determined bit rate corresponding to each image.

[0100] 8 shows multiple video tracks 1 to n and a metadata transmission path within the communication path between the automatic driving control device 20 and the remote support device 40 of the remote support center 30. The multiple video tracks 1 to n shown within the communication path are, for example, video tracks for transmitting six images corresponding to the six cameras previously described with reference to FIGS. 3 and 4. When six images are transmitted, n=6.

[0101] Multiple video tracks 1 to n are transmitted using a single stream set up between the remote support device 40 and the remote support center 30. That is, six images corresponding to six cameras are transmitted using multi-track video streaming, which transmits multiple videos in a single stream. The bit rates of these six images are changed as needed according to the available bandwidth. For example, a larger communication bandwidth (bit rate) is allocated to a higher priority image according to the image priority.

[0102] Furthermore, if the communication bandwidth available for data transmission is narrow, a filtering process may be performed in which only images with high priority are selected and transmitted, rather than transmitting all of the images captured by the six cameras. These processes will be explained later.

[0103] 8 shows the transmission path of the video track as well as the transmission path of the metadata, which includes information about objects detected from camera images and LiDAR sensor information, such as surrounding vehicles, pedestrians, and buildings.

[0104] Specifically, the metadata includes the coordinates of the object with the host vehicle as the origin, the size of the bounding box, which is a three-dimensional box surrounding the object, a label indicating the object type (vehicle, pedestrian, etc.), an image of the object, etc. These metadata are generated by the data processing unit 22 using point cloud information, which is an image captured by a camera and LiDAR detection values. In a configuration equipped with other sensors such as radar, metadata is also generated using these sensor detection values.

[0105] Regarding metadata, the automatic driving control device 20 also performs communication control to change the metadata to be transmitted as needed depending on the communication bandwidth (bit rate) available for metadata communication between the automatic driving control device 20 and the remote assistance device 40.

[0106] The metadata includes data that is constantly transmitted regardless of fluctuations in the available communication bandwidth, and data that stops transmission when the available bandwidth is narrow. For example, the coordinates of objects (objects such as vehicles and pedestrians around the autonomous vehicle 10) used to identify their positions and sizes, as well as the size and label of their bounding boxes, are examples of metadata that is constantly transmitted.

[0107] On the other hand, for large-volume data such as object images, if the bandwidth of the communication path that can be secured for metadata communication is narrow, the transmission will be stopped, or the data will be transmitted as compressed data, or filtering processing will be performed to select the data to be transmitted depending on the risk level of each object. Details of these processing will be described later.

[0108] As shown in Figure 8, the camera images (video track) and metadata transmitted from the automatic driving control device 20 are received by the communication unit 41 of the remote support device 40 in the remote support center 30 and input into the data processing unit 42.

[0109] The data processing unit 42 analyzes the camera-captured images and metadata received from the automatic driving control device 20, and generates image data to be output (displayed) on the display unit 43. For example, if the camera-captured images received from the automatic driving control device 20 are complete six-directional images as described with reference to FIG.

[0110] However, as mentioned above, depending on the state of the communication network, it may not be possible to receive complete six-directional camera images from the autonomous driving control device 20. In such cases, the data processing unit 42 of the remote assistance device 40 interpolates the insufficient areas of the camera images using 3DCG images acquired from the 3D map storage unit 45 and outputs the interpolated images to the display unit 43. The data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 shown in FIG. 2 to generate and display 3DCG images observed from the positions and directions of the cameras mounted on the autonomous driving vehicle 10.

[0111] Furthermore, the data processing unit 42 analyzes metadata including information on surrounding objects such as oncoming vehicles received from the automatic driving control device 20, and performs processing such as displaying images of objects around the automatically driven vehicle 10 on a 3DCG image. By performing such processing, display data that more accurately reproduces the current surrounding situation of the automatically driven vehicle 10 is generated and displayed on the display unit.

[0112] The operator 50 observes the image displayed on the display unit 43 to check the current surrounding conditions of the autonomous vehicle 10, inputs optimal control information (autonomous driving control information) via the input unit 44, and transmits it to the autonomous vehicle 10.

[0113] 4. Transmission Data Control Processing Executed by the Automatic Driving Control Device Next, the transmission data control processing executed by the automatic driving control device will be described.

[0114] As described with reference to Figure 8, the autonomous driving control device 20 of the autonomous driving vehicle 10 transmits not only camera-captured images but also metadata consisting of object information and the like to the remote assistance device 40 of the remote assistance center 30.

[0115] When performing this data transmission process, the data processing unit 22 of the autonomous driving control device 20 estimates the bandwidth (bit rate) available for data transmission and performs bit rate control for each image to be transmitted according to the estimated available bandwidth (bit rate). Furthermore, the data processing unit 22 performs processes such as selecting images to be transmitted and selecting and compressing object images to be transmitted as metadata. Details of these processes performed by the data processing unit 22 of the autonomous driving control device 20 are described below.

[0116] 9 and 10 are flowcharts illustrating the sequence of transmission data control processing executed by the data processing unit 22 of the automatic driving control device 20. Note that the processing according to the flowcharts shown in Figure 9 and subsequent figures can be executed by the data processing unit 22 of the automatic driving control device 20 in accordance with a program stored in a storage unit of the automatic driving control device 20. The data processing unit 22 has a processor such as a CPU with a program execution function, and can be performed as program execution processing by the processor.

[0117] 9 and 10, the processes of steps S101 to S125 are repeatedly executed during the execution period of the data transmission process from the autonomous driving control device 20 of the autonomous driving vehicle 10 to the remote assistance device 40 of the remote assistance center 30.

[0118] (Step S101) First, in step S101, the data processing unit 22 of the automatic driving control device 20 calculates a bit rate (estimated bit rate b ｅ ) is calculated.

[0119] The bit rate available for this data transmission (estimated bit rate b ｅ The process of calculating the distance .gtoreq..times ...

[0120] The bit rate available for data transmission (estimated bit rate b ｅ 9 and 10 are repeatedly executed during the execution period of the data transmission process of the remote support center 30 to the remote support device 40. The available bandwidth calculated in step S101, i.e., the estimated bit rate b ｅ The processing executed in step S102 and subsequent steps will differ depending on the value of .

[0121] The estimated bit rate b used in the flowcharts shown in FIGS. ｅ The definitions of the parameters, such as a, b, and b, are as follows, as shown in the table in FIG. ｅ : Estimated bit rate (communication bit rate equivalent to the available bandwidth calculated from the results of monitoring the wireless communication network) B = {b ｍｉｎ , b ｍａｘ}: Target bit rate b to be set as the actual data transmission bit rate ｔａｒｇｅｔ Range b ｍｉｎ: A target bit rate b that is predetermined as the actual data transmission bit rate ｔａｒｇｅｔ Minimum bit rate of b ｍａｘ : A target bit rate b that is predetermined as the actual data transmission bit rate ｔａｒｇｅｔ Maximum bit rate of b ｔａｒｇｅｔ : Target bit rate (target value) used for actual data transmission b (disabled) ａｌｌｏｃａｔｅｄ : Communication bit rate required to maintain communication with the communication partner, such as Keep Alive

[0122] (Step S102) Next, in step S102, the estimated bit rate b calculated in step S101 is ｅ and a predetermined maximum bit rate b ｍａｘ Compare with b ｅ ≧b ｍａｘ It is determined whether the above formula is true.

[0123] If the above formula is satisfied, that is, the estimated bit rate b corresponding to the available bandwidth ｅ is a predetermined maximum bit rate b ｍａｘ If the estimated bit rate b is equal to or greater than the value of b, the determination in step S102 becomes Yes, and the process proceeds to step S103. ｅ is a predetermined maximum bit rate b ｍａｘ If the value is less than the value of (1), the determination in step S102 becomes No, and the process proceeds to step S104.

[0124] Maximum bit rate b ｍａｘ is the target bit rate b, which is the bit rate used for actual data transmission. ｔａｒｇｅｔ On the other hand, the target bit rate b ｔａｒｇｅｔ The minimum allowable value of is the minimum bit rate b ｍｉｎ and this value is also predefined.

[0125] With reference to FIG. 12, the maximum bit rate b ｍａｘ and minimum bit rate b ｍｉｎFIG. 12 shows a graph in which the horizontal axis represents time (sec) and the vertical axis represents bandwidth (bit rate (bps)). The dotted line represents the bit rate available for data transmission (estimated bit rate b ｅ 12. As can be seen from the dotted line in FIG. 12, the bit rate available for data transmission (estimated bit rate b ｅ ) fluctuates significantly over time.

[0126] On the other hand, the solid line indicates the target bit rate b, which is the target bit rate used for actual data transmission. ｔａｒｇｅｔ As shown in FIG. 12, the target bit rate b ｔａｒｇｅｔ The line showing the time transition of the maximum bit rate b ｍａｘ and minimum bit rate b ｍｉｎ It is set between.

[0127] These maximum bit rates b ｍａｘ and minimum bit rate b ｍｉｎ is the target bit rate range B = [b ｍｉｎ , b ｍａｘ The minimum bit rate b ｍｉｎ is the minimum bit rate at which all of the camera-captured images 1 to n can be transmitted to the remote support device 40 without causing a large delay. ｍｉｎ By transmitting all of the images captured by the camera to the remote support device 40 at the above bit rate, it is possible to receive control from the remote support device 40 without delay.

[0128] However, the bit rate available for data transmission (estimated bit rate b ｅ ) is large and there is a margin of bandwidth, that is, the bit rate available for data transmission (estimated bit rate b ｅ ) is the maximum bit rate b ｍａｘ Even if the data is in the above-mentioned band area A, the target bit rate b used for actual data transmission is ｔａｒｇｅｔThe maximum value of the maximum bit rate b ｍａｘ Let's say.

[0129] The reason for performing such bit rate control is that the estimated bit rate b ｅ This is to realize stable data transmission even when there is a sudden change in the available bandwidth. ｅ As can be seen from the time transition of , a sudden drop in the estimated bit rate may occur depending on the communication environment.

[0130] For example, the target bit rate b ｔａｒｇｅｔ The estimated bit rate b ｅ When the line is set to match the target bit rate b ｔａｒｇｅｔ If the actual data transmission bit rate is suddenly reduced, the loss of transmission packets may occur, causing the display image on the remote support device to freeze, a phenomenon known as video freeze. In order to avoid this situation, the target bit rate b ｔａｒｇｅｔ Control is performed to maintain the value within a certain range.

[0131] That is, the target bit rate b ｔａｒｇｅｔ is set to a predefined maximum bit rate b ｍａｘ and minimum bit rate b ｍｉｎ Stable data transmission is achieved by controlling the bit rate so that it transitions between

[0132] However, the estimated bit rate b ｅ is small and there is no margin in the transmission band, that is, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ If the bandwidth is in the bandwidth region C below 100 Mbps, transmitting all of the camera-captured images 1 to n may cause transmission delays, and the reception of control information from the remote support device 40 may also be delayed.

[0133] To avoid this situation, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ If the bandwidth is in the bandwidth region C less than 100 Mbps, the transmission bit rate is adjusted by performing a process of selecting the images to be transmitted (filtering process) without transmitting all of the camera-captured images 1 to n. This process corresponds to the process from (step S104=No) to (step S108) to (step S111 and subsequent steps) in the flow shown in FIG.

[0134] (Step S103) In step S102, ｅ ≧b ｍａｘ If it is determined that the above formula is true, that is, the estimated bit rate b corresponding to the available bandwidth is ｅ is a predetermined maximum bit rate b ｍａｘ is equal to or greater than the estimated bit rate b ｅ 12), in step S103, the target bit rate b ｔａｒｇｅｔ The maximum bit rate b ｍａｘ Set to.

[0135] That is, target bit rate b ｔａｒｇｅｔ = b ｍａｘ As described above, the target bit rate b ｔａｒｇｅｔ The maximum bit rate b ｍａｘ Set to.

[0136] (Step S104) On the other hand, in step S102, b ｅ ≧b ｍａｘ If it is determined that the above formula does not hold, that is, if the estimated bit rate b corresponding to the available bandwidth is ｅ is a predetermined maximum bit rate b ｍａｘ (estimated bit rate b ｅ 12), in step S104, the estimated bit rate b calculated in step S101 is ｅand a predetermined minimum bit rate b ｍｉｎ Compare with b ｅ ≧b ｍｉｎ It is determined whether the above formula is true.

[0137] If the above formula is satisfied, that is, the estimated bit rate b corresponding to the available bandwidth ｅ is a predetermined minimum bit rate b ｍｉｎ is equal to or greater than the estimated bit rate b ｅ 12), the determination in step S104 is Yes, and the process proceeds to step S105. ｅ is a predetermined minimum bit rate b ｍｉｎ (estimated bit rate b ｅ 12), the determination in step S104 is No, and the process proceeds to step S108.

[0138] (Step S105) In step S104, ｅ ≧b ｍｉｎ If it is determined that the above formula is true, that is, the estimated bit rate b corresponding to the available bandwidth is ｅ is a predetermined minimum bit rate b ｍｉｎ is equal to or greater than the estimated bit rate b ｅ 12), in step S105, the target bit rate b ｔａｒｇｅｔ is set to the estimated bit rate b ｅ Set to.

[0139] That is, target bit rate b ｔａｒｇｅｔ = b ｅ Here, the estimated bit rate b corresponding to the available bandwidth is ｅ is b ｍａｘ ≧b ｅ ≧b ｍａｘ The above formula is satisfied. That is, the estimated bit rate b ｅ is in the band region B shown in FIG. 12, and the target bit rate bｔａｒｇｅｔ = will transition within the band region B shown in FIG.

[0140] In this way, the target bit rate b ｔａｒｇｅｔ By shifting within the band region B shown in FIG. 12, the minimum bit rate b ｍｉｎ As mentioned above, data transmission at the minimum bit rate b ｍｉｎ is the bit rate at which all of the images captured by the camera can be transmitted to the remote support device 40 without causing a large delay.

[0141] In this way, the target bit rate b used for data transmission ｔａｒｇｅｔ 12 within the bandwidth region B shown in FIG. 12 and transmit all of the images captured by the camera to the remote support device 40, it is possible to receive control from the remote support device 40 without delay.

[0142] (Step S106) In step S103 or step S105, the target bit rate b, which is the bit rate used for actual data transmission, is set. ｔａｒｇｅｔ If the value is determined, the process of step S106 is executed.

[0143] That is, in step S103, the target bit rate b ｔａｒｇｅｔ = b ｍａｘ Alternatively, in step S105, the target bit rate b ｔａｒｇｅｔ = b ｅ By using any of these processes, the target bit rate b, which is the bit rate used for actual data transmission, is obtained. ｔａｒｇｅｔ If the value is determined, the process of step S106 is executed.

[0144] The target bit rate b determined in step S103 or step S105 ｔａｒｇｅｔ , i.e., the target bit rate b ｔａｒｇｅｔ The value of b ｍａｘ ≧b ｔａｒｇｅｔ ≧b ｍｉｎ The above formula is satisfied. That is, the target bit rate b ｔａｒｇｅｔThe value of the target bit rate b is set in the band region B shown in FIG. ｔａｒｇｅｔ The target bit rate b = will be shifted within the band area B shown in FIG. ｔａｒｇｅｔ By setting the above, it becomes possible to transmit all of the images captured by the camera to the remote support device 40 without causing a large delay.

[0145] In step S106, the communication bit rate of the images (video tracks) 1 to n captured by each of the cameras 1 to n is allocated according to the resolution (number of pixels) of each image. The sum of the communication bit rates of the images (video tracks) 1 to n captured by each of the cameras 1 to n is equal to the target bit rate b ｔａｒｇｅｔ and this target bit rate b ｔａｒｇｅｔ is allocated to each of the captured images (video tracks) 1 to n to determine the transmission bit rate for each of the captured images (video tracks) 1 to n.

[0146] The data processing unit 22 of the automatic driving control device 20 calculates the bit rate of each of the captured images (video tracks) 1 to n based on the resolution and target bit rate of each of the images 1 to n. This process is executed by the data analysis unit 113 of the data processing unit 22 shown in FIG. 5.

[0147] A specific example of the transmission bit rate allocation process for each of the captured images (video tracks) 1 to n will be described with reference to FIG.

[0148] FIG. 13 shows the resolutions (number of pixels) of images captured by the six cameras mounted on the autonomously driven vehicle 10 described above with reference to FIGS. 3 and 4. The resolutions of the following six images are shown: (a) Left rearward captured image (track ID = video track 1) Resolution = w1 × h1 (b) Right rearward captured image (track ID = video track 2) Resolution = w2 × h2 (c) Rearward captured image (track ID = video track 3) Resolution = w3 × h3 (d) Leftward captured image (track ID = video track 4) Resolution = w4 × h4 (e) Rightward captured image (track ID = video track 5) Resolution = w5 × h5 (f) Forward captured image (track ID = video track 6) Resolution = w6 × h6 Note that wi corresponds to the number of horizontal pixels of image i, and hi corresponds to the number of vertical pixels of image i. wi × hi corresponds to the resolution (number of pixels) of image i.

[0149] The target bit rate for transmitting all these images is b ｔａｒｇｅｔ Let N be the total number of images (video tracks) and i be the camera ID. In the example of FIG. 13, N=6 and i=1 to 6. In this case, the bit rate (b(i)) assigned as the transmission bit rate for the ith image (video track) is ａｌｌｏｃａｔｅｄ ) can be calculated according to the following formula (Formula 1):

[0150]

[0151] In addition, r ｉ is the multiplication coefficient of the i-th video track, and the transmission bit rate of the i-th image (video track) (b(i) ａｌｌｏｃａｔｅｄ ) is the target bit rate b ｔａｒｇｅｔ , the multiplication coefficient r of the i-th image (video track) ｉ It can be calculated by multiplying

[0152] Note that (Equation 1a) is a function of the target bit rate b ｔａｒｇｅｔ (Equation 1b) is an equation for calculating the multiplication coefficient ri to be applied to all images (video tracks). Equation 1b shows that the sum of the multiplication coefficients ri corresponding to all images (video tracks) is 1.

[0153] As shown in (Equation 1a), the target bit rate b ｔａｒｇｅｔ The multiplication coefficient ri applied to each image is determined according to the product wi×hi of wi and hi of each image, that is, the resolution corresponding to the number of pixels. That is, the transmission bit rate (b(i)) of each image (video track) i is ａｌｌｏｃａｔｅｄ ) is allocated in proportion to the resolution (number of pixels) of each image.

[0154] In this way, in step S106, the data processing unit 22 of the automatic driving control device 20 allocates the communication bit rate of the captured images (video tracks) 1 to n of each camera 1 to n in accordance with the resolution (number of pixels) of each image. If the sum of the communication bit rates of the captured images (video tracks) 1 to n of each camera 1 to n is equal to the target bit rate b ｔａｒｇｅｔ and this target bit rate b ｔａｒｇｅｔ is allocated to each of the captured images (video tracks) 1 to n to determine the transmission bit rate for each of the captured images (video tracks) 1 to n.

[0155] (Step S107) Next, in step S107, the data processing unit 22 of the automatic driving control device 20 transmits each of the captured images (video tracks) 1 to n of each camera 1 to n to the remote assistance device 40 in accordance with the communication bit rate allocated to each image (video track) in step S106.

[0156] The total bit rate of the transmission bit rates of each captured image (video track) 1 to n used in this data transmission is the target bit rate b ｔａｒｇｅｔ and this target bit rate b ｔａｒｇｅｔ is the range of the band region B shown in FIG. 12, that is, the minimum bit rate b ｍｉｎ and maximum bit rate b ｍａｘ That is, b ｍａｘ ≧b ｔａｒｇｅｔ ≧b ｍｉｎ The above formula is satisfied.

[0157] In this way, the target bit rate b ｔａｒｇｅｔBy shifting the frequency band within the bandwidth region B shown in FIG. 12, it becomes possible to transmit all of the camera-captured images 1 to n to the remote support device 40 without causing a large delay.

[0158] After the process of step S107 is completed, the process returns to step S101, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0159] (Step S108) On the other hand, in the determination step of step S104, the estimated bit rate b calculated in step S101 is ｅ and a predetermined minimum bit rate b ｍｉｎ Comparison formula with b ｅ ≧b ｍｉｎ If it is determined that the above formula does not hold, that is, if the estimated bit rate b ｅ is a predetermined minimum bit rate b ｍｉｎ (estimated bit rate b ｅ If it is determined that the signal is in the band region C shown in FIG. 12, the process proceeds to step S108.

[0160] In step S108, the target bit rate b ｔａｒｇｅｔ , the minimum bit rate b ｍｉｎ Set to b ｔａｒｇｅｔ = b ｍｉｎ As mentioned above, the target bit rate b ｔａｒｇｅｔ , the minimum bit rate b ｍｉｎ Set to.

[0161] However, at the execution stage of step S108, the determination of step S104 is No, that is, ｅ ≧b ｍｉｎ If it is determined that the above formula does not hold, and the estimated bit rate b available for data transmission obtained from the network monitoring results is ｅ is the minimum bit rate b ｍｉｎ That is, the estimated bit rate b ｅ is the minimum bit rate b shown in FIG. ｍｉｎIt is in a band region C below.

[0162] In this situation, the target bit rate b ｔａｒｇｅｔ , the minimum bit rate b ｍｉｎ Even if you set it to , the bit rate that can actually be used for data transmission is the minimum bit rate b ｍｉｎ Estimated bit rate less than b ｅ In this case, if all of the camera-captured images 1 to n are transmitted, a transmission delay or the like may occur, which may result in a delay in the control of the remote support device 40.

[0163] Therefore, in such a case, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ If the bandwidth is in the bandwidth region C less than 100 Mbps, the transmission bit rate is adjusted by performing a process such as selecting the images to be transmitted without transmitting all of the camera-captured images 1 to n. This process corresponds to step S111 and subsequent steps in the flow shown in FIG.

[0164] (Step S111) The bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ If it is determined that the estimated bit rate is less than b ｅ is the minimum bit rate b ｍｉｎ If it is determined that the image is in a bandwidth region C less than this, the data processing unit 22 of the automatic driving control device 20 first performs a process in step S111 to classify each of the captured images (video tracks) 1 to n of each camera 1 to n into transmitted images (valid video tracks) and non-transmitted images (invalid video tracks) according to the image priority.

[0165] As mentioned above, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎIf the bit rate is in a bandwidth region C less than 100 Mbps, transmitting all of the camera-captured images 1 to n is likely to cause a communication delay, making it impossible to timely receive control information from the remote support device 40. To avoid this situation, in step S111, a process for selecting the images to be transmitted (filtering process) or the like is performed to enable data transmission with little delay even at a low bit rate.

[0166] The details of the classification process into transmission images (valid video tracks) and non-transmission images (invalid video tracks) according to the image priority will be described later. First, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ When the signal is in a band region C less than 1000 kB, the processing executed by the data processing unit 22 of the automatic driving control device 20 will be outlined below, with the processing of steps S112 to S114 and the processing of steps S121 to S125 shown in FIG. 10 being described below.

[0167] 10 is a processing sequence for the camera-captured image, and the processing of steps S121 to S125 is a processing sequence for the metadata of the camera-captured image and the objects detected from the LiDAR detection information. The processing of steps S112 to S114 and the processing of steps S121 to S125 shown in FIG. 10 are executed in parallel by the data processing unit 22 of the autonomous driving control device 20.

[0168] (Step S112) In step S112, a communication maintaining bit rate b (disabled) is set for the communication track of each image. ａｌｌｏｃａｔｅｄ For example, b(disabled) ａｌｌｏｃａｔｅｄ = 30000 bps The communication maintenance bit rate b (disabled) set above ａｌｌｏｃａｔｅｄ is set for the communication track of each image.

[0169] This is a process that allocates a certain bandwidth to maintain the connection of each track even when video transmission using the communication track for each image is disabled. For example, when WebRTC is used, the allocated bit rate is 30 kbps. If some images are not transmitted and the video track for transmitting those images is disabled, the bit rate is reallocated to other valid video tracks, i.e., the video track for transmitting the image to be transmitted. As a result, the video track for transmitting high-priority images transmits images at a high bit rate.

[0170] (Step S113) Next, in step S113, the data processing unit 22 of the automatic driving control device 20 executes a process of reallocating the communication bit rate only for the images selected as transmission images (valid video tracks) by the classification process in step S111.

[0171] In step S111, the images (video tracks) 1 to n captured by the cameras 1 to n are classified into transmission images (valid video tracks) and non-transmission images (invalid video tracks) according to image priority.

[0172] In step S113, a process of allocating a communication bit rate is executed only for the image selected as the image to be transmitted (valid video track) in step S111.

[0173] The total communication bit rate allocated here is the bit rate available for data transmission acquired by the communication network monitoring unit 112 of the data processing unit 22 of the automatic driving control device 20 (estimated bit rate b ｅ ), and the estimated bit rate b ｅ This estimated bit rate b ｅ is the minimum bit rate b ｍｉｎ This is a low bit rate that makes it difficult to transmit all camera-captured images with low latency, but the setting allows transmission with low latency for only the images to be transmitted (valid video tracks) selected according to priority in step S111. Note that specific details of the bit rate allocation process will be explained later.

[0174] (Step S114) Next, in step S114, the data processing unit 22 of the automatic driving control device 20 executes a process of transmitting each of the transmission images (valid video tracks) selected according to priority in step S111 in accordance with the communication bit rate reallocated to these transmission images (valid video tracks) in step S113.

[0175] The images to be transmitted here are not all the images captured by the camera, but only some of the images selected in accordance with the priority in step S111. Therefore, the bandwidth (bit rate) available for transmission is the minimum bit rate b ｍｉｎ Even if the signal is in a band region C less than 100 MHz, transmission can be performed without causing a large delay, and control information from the image support device 40 can be received in a timely manner.

[0176] After the process of step S114 is completed, the process returns to step S101, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0177] Next, the processing of steps S121 to S125 will be described. The processing of steps S121 to S125 shown in Fig. 10 is a sequence of transmission control processing related to metadata of objects detected from camera images and LiDAR detection information.

[0178] As described above, in step S111, a process is performed to classify images into transmission images (valid video tracks) and non-transmission images (invalid video tracks) according to image priority, and the transmission process of images with low priority is stopped.

[0179] However, even images with a low priority and selected as non-transmitted images may contain objects that may collide or come into contact with the autonomous vehicle 10, such as oncoming vehicles or pedestrians, i.e., high-risk objects.

[0180] Steps S121 to S125 are a sequence for extracting high-risk objects contained in the non-transmitted image (invalid video track) and transmitting object information about the extracted objects to the remote control device 40 as metadata separate from the image.

[0181] The object extraction process, risk level determination process, object filtering process, etc. executed in steps S121 to S125 will be described in more detail later. Here, only an outline of the process sequence will be described.

[0182] (Step S121) First, in step S121, the data processing unit 22 of the automatic driving control device 20 extracts objects (vehicles, pedestrians, buildings, etc.) around the automatic driving vehicle 10 from the non-transmitted image (invalid video track).

[0183] The images from which objects are extracted are the transmitted images (valid video tracks) classified according to image priority in step S111, and the non-transmitted images (invalid video tracks) among the non-transmitted images (invalid video tracks). In other words, object extraction processing is performed from images from which image data is not transmitted.

[0184] (Step S122) Next, in step S122, the data processing unit 22 of the automatic driving control device 20 determines the risk level of each of the objects extracted from the non-transmitted image in step S121.

[0185] For example, an object with a high risk of collision is determined to be a high risk. This process will be described in detail later.

[0186] (Step S123) Next, in step S123, the data processing unit 22 of the automatic driving control device 20 compresses each of the extracted objects in a different compression mode (different compression algorithm) according to the risk level of each of the extracted objects determined in step S122.

[0187] (Step S124) Next, in step S124, the data processing unit 22 of the automatic driving control device 20 executes a filtering process on the extracted and compressed object image data.

[0188] The filtering process is a process for selecting compressed object image data to be transmitted, and classifies compressed object image data into those to be transmitted and those not to be transmitted. This filtering process is performed when the bit rate required for transmitting compressed object data is greater than the available bandwidth.

[0189] However, the object coordinates indicating the position of the object, the coordinates of the bounding box indicating the three-dimensional box area surrounding the object, and the label indicating the object type (vehicle, person, etc.) are all included in the metadata regardless of the risk level of the object and transmitted to the remote assistance device 40. Details of this filtering process will be described later.

[0190] (Step S125) Next, in step S125, the data processing unit 22 of the automatic driving control device 20 includes the compressed object image data finally selected as the transmission target as a result of the filtering process in step S124 in metadata and transmits the same to the remote assistance device 40.

[0191] By executing steps S121 to S125, if there is a high-risk object in the image that was determined to be a non-transmitted image (invalid video track) in step S111, the image of this high-risk object will be transmitted to the remote support device 40 as metadata constituent data.

[0192] The remote assistance device 40 analyzes the metadata received from the autonomous driving control device 20, extracts and decodes the object compressed image data, and performs processing to display the image displayed on the display unit 43, for example, a 3DCG image, by superimposing object data of vehicles, pedestrians, etc. This processing enables the operator at the remote assistance center 30 to check for high-risk objects included in the non-transmitted image, and to generate appropriate control information to prevent the autonomous driving vehicle 10 from colliding with the high-risk object, and to transmit the information to the autonomous driving control device 20.

[0193] After the process of step S125 is completed, the process returns to step S101, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0194] 5. Details of the Process of Selecting and Sending Camera-Captured Images According to Priority Next, details of the process of selecting and sending camera-captured images according to priority will be described.

[0195] The flowchart shown in Figure 14 is a flowchart that explains the details of the camera image arrangement process according to priority performed by the data processing unit 22 of the autonomous driving control device 20 of the autonomous driving vehicle 10, and the transmission image selection process according to priority.

[0196] 14 corresponds to the detailed flow of the processing in steps S111 to S114 of the flow previously described with reference to FIG. 10. That is, in step S108 of the flow shown in FIG. 9, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ Specifically, this is a process to be executed when it is determined that the estimated bit rate b ｅ is the minimum bit rate b ｍｉｎ This is executed when it is determined that the signal is in a band region C less than 100 MHz.

[0197] The details of the processing in steps S201 to S209 of the flow shown in Fig. 14 will be explained below. The definitions of the parameters used in the flow shown in Fig. 14 are as follows, as shown in the table in Fig. 15: M: total number of valid images (transmitted images) s(M) ｒｅｑｕｉｒｅｄ : the sum of the required bit rates of all valid images (transmitted images) b(j) ｒｅｑｕｉｒｅｄ : the bit rate required to transmit the jth image j: the index of the list of images sorted by priority (descending order) (the highest priority image is j=1)

[0198] (Step S201) First, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ If it is determined that the estimated bit rate is less than b ｅ is the minimum bit rate b ｍｉｎ If it is determined that the image is in a bandwidth region C less than this, the data processing unit 22 of the automatic driving control device 20 performs a process in step S201 of arranging the captured images (video tracks) 1 to n of each camera 1 to n according to image priority.

[0199] This process is executed by the image priority analysis module 111d of the risk level classification unit 111 of the data processing unit 22 of the automatic driving control device 20 described above with reference to FIG.

[0200] As explained above, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ If the bandwidth is in a bandwidth region C less than 100 Mbps, transmitting all of the camera-captured images 1 to n is likely to cause a communication delay, making it impossible to timely receive control information from the remote support device 40. The flow shown in Fig. 14 is a processing flow for avoiding such a situation, and is a flow for enabling low-delay data transmission even at a low bit rate by performing processing such as selecting the image to be transmitted.

[0201] A specific example of image arrangement processing according to image priority will be described with reference to Figure 16 and subsequent figures. Figure 16 is a diagram illustrating image analysis parameters used in the process of determining the priority of six images corresponding to the six cameras previously described with reference to Figures 3 and 4. That is, it is a diagram illustrating elements used in the process of determining the priority of these six images: a forward image, a rearward image, a right-rearward image, a left-rearward image, a right-direction image, and a left-direction image of the autonomously driven vehicle 10.

[0202] As shown in FIG. 16, for example, the image priority is calculated using the following elements (analysis parameters (f=1 to n)) obtained by image analysis processing. That is, (a) Number of surrounding objects (f=1) (=in descending order of the number of surrounding objects detected from the image) (b) Number of high-risk objects (f=2) (=in descending order of the number of high-risk objects detected from the image) (c) Remote assistance task (f=3) (=in descending order of importance for the generation of autonomous driving control information in the remote assistance device) (d) Line of sight (f=4) (=in descending order of the number of operator's lines of sight in the remote assistance device)

[0203] The types of objects to be counted as the above (a) number of surrounding objects and (b) number of high-risk objects are specified in advance. For example, the types of objects to be detected are specified in advance, such as surrounding vehicles, pedestrians, bicycles, motorbikes, and structures near the road (guardrails, traffic lights, etc.).

[0204] Furthermore, the four elements (analysis parameters (f = 1 to n)) parameters applied to the image priority calculation shown in Figure 16 are just an example, and it is also possible to perform image priority calculation processing using only some of these elements (analysis parameters), or image priority calculation processing using still other elements (analysis parameters).

[0205] The image priority analysis module 111d of the risk level classification unit 111 of the data processing unit 22 of the automatic driving control device 20 first calculates a score (safety score S) corresponding to each of the above elements (each analysis parameter) according to a predetermined algorithm. ｆ (P)) is calculated.

[0206] Among the above elements (analysis parameters (f=1 to n)), the scores for (a) the number of surrounding objects (f=1) and (b) the number of high-risk objects (f=2) are calculated by analyzing only the images captured by the camera. On the other hand, the scores for (c) the remote assistance task (f=3) and (d) the line of sight (f=4) are calculated by analyzing the information received from the remote assistance device 40.

[0207] For example, when calculating the score for (d) gaze (f=4), the score is calculated by receiving analysis information of a gaze heat map indicating the degree of gaze concentration of the operator at the remote support center 30. Fig. 17 is a diagram showing a specific example of a gaze heat map indicating the degree of gaze concentration of the operator at the remote support center 30. Bright circular areas in each image in the figure indicate areas where the operator's gaze is highly concentrated.

[0208] The data processing unit 22 of the automatic driving control device 20 calculates the score (safety score S) corresponding to each of the above elements (a) to (d) (analysis parameters (f=1 to 4)). ｆ (P)) is calculated according to a specified score calculation algorithm, and then the following calculation formula (Formula 2) is applied to obtain the final priority value P for each image. ＊ Calculate.

[0209]

[0210] In the above formula (2), s ｆ (P) is the safety score corresponding to each element (analysis parameter: each element f1 to f4 shown in Figure 16), f is the index of the element (analysis parameter) that affects the image priority, and F is the total number of elements (analysis parameters).

[0211] An example of the analysis process of (a) the number of peripheral objects and (b) the number of high-risk objects, which are elements (analysis parameters) used in the image priority calculation process according to the above (Equation 2), will be described with reference to FIG.

[0212] 18 is a diagram illustrating objects detected from six images corresponding to the six cameras described above with reference to FIGS. 3 and 4. That is, the diagram illustrates objects detected from six images: a forward image, a rearward image, a right-rearward image, a left-rearward image, a right-ward image, and a left-ward image of the autonomously driven vehicle 10. The rectangular frames shown in each image are object detection frames, and detected objects include surrounding vehicles, pedestrians, etc.

[0213] (a) The score for the number of surrounding objects (safety score) is calculated according to the number of objects detected from each image. (b) The score for the number of high-risk objects (safety score) is calculated by calculating the risk of each object detected from each image and then calculating the score (safety score) in order of the sum of the risks of the objects detected from each image.

[0214] A specific example of the risk calculation process for each object will be described below. The risk calculation process for an object can be executed as a process of classifying each object into a plurality of risk levels 1 to n, for example.

[0215] As a method for calculating the risk level of an object, for example, processing based on Responsibility Sensitive Safety (RSS) can be used. RSS is a method for evaluating the suitability of a pair of an object detected from an image and the autonomously driven vehicle 10, which is the vehicle itself.

[0216] The extracted object risk level analysis module 111e of the risk level classification unit 111 of the data processing unit 22 of the automatic driving control device 20 shown in Figure 5 has an RSS module and inputs an object list containing information about objects (road agents) in the surrounding environment of the vehicle.

[0217] For each object, the RSS module generates a description of the object-vehicle pair and an attribute called a "situation." Next, the RSS module executes a corresponding RSS safety check for each situation, and calculates the risk level of each object as a result of this safety check.

[0218] Another method that can be used as a method for calculating the risk level of an object is a method that uses a neural network as shown in Fig. 19. In this case, the extracted object risk level analysis module 111e of the risk level classification unit 111 of the data processing unit 22 of the autonomous driving control device 20 shown in Fig. 5 has a neural network as shown in Fig. 19.

[0219] The input data for the neural network in the extracted object risk level analysis module 111e includes the type, size, moving direction, distance from the vehicle, network status, weather, road conditions, etc. of the object detected from each image. The output is the risk level for the object.

[0220] The neural network's learning data can be based on past accident data involving autonomous driving assistance vehicles. For example, the National Highway Traffic Safety Administration (NHTSA) in the United States publishes accident reports involving autonomous driving assistance vehicles, and a neural network for calculating risk levels can be generated through learning processing using such data.

[0221] In this way, as a method for calculating the risk level of an object, for example, RSS (Responsibility Sensitive Safety) or a method using a neural network can be used.

[0222] Regardless of which process is applied, the risk level for each object calculated by the extracted object risk level analysis module 111e will be calculated as a level according to a predefined level definition. For example, the level definition can be the following level classification that uses the analysis results of past accident situation data at the current location of the autonomously driven vehicle 10: Risk level 1 = serious accident occurred Risk level 2 = medium accident occurred Risk level 3 = minor accident occurred Risk level 4 = no injuries reported

[0223] The extracted object risk level analysis module 111e of the risk level classification unit 111 of the data processing unit 22 of the automatic driving control device 20 calculates the risk level for each object detected from the images captured by each camera shown in Figure 17 described above using the above-mentioned RSS or neural network.

[0224] The risk level of each object calculated by the extracted object risk level analysis module 111e is input to the image priority analysis module 111d of the risk level classification unit 111, and the image priority analysis module 111d calculates the priority (P ＊ ) is calculated.

[0225] In step S201 of the flowchart shown in FIG. 14, the image priority of each of the images (video tracks) 1 to n captured by each camera 1 to n is calculated, and the images are arranged according to the calculated image priority.

[0226] (Step S202) Next, in step S202, the data processing unit 22 of the automatic driving control device 20 calculates the bit rate (required bit rate s(M)) required for transmitting M images (enabled videos) (initial value M=total number of images) arranged according to the priority in step S201. ｒｅｑｕｉｒｅｄ ) is calculated. ｒｅｑｕｉｒｅｄ is a bit rate at which it is determined that M images (enabled videos) arranged in order of priority (initial value M=total number of images) can be transmitted to the remote control device 40 with a delay of a predetermined value or less.

[0227] The initial value of M is the total number of images captured by the cameras. For example, in the example previously shown in Figures 3 and 4, the number of cameras mounted on the autonomous vehicle 10 is 6, and the number of captured images is also 6, so M = 6.

[0228] The data processing unit 22 of the automatic driving control device 20 calculates the bit rate required for transmitting M images (required bit rate s(M) ｒｅｑｕｉｒｅｄ ) is calculated according to the following formula (Formula 3).

[0229]

[0230] In the above formula (formula 3), F(*) is the image-corresponding required bit rate allocated to each image b(j). ｒｅｑｕｉｒｅｄ The frame rate when ｔａｒｇｅｔ is the target frame rate to ensure smoothness of the image reproduced on the remote device 40 side, and .times. ...

[0231] The required bit rate for the image is b(j). ｒｅｑｕｉｒｅｄ is the bit rate required to transmit one image of priority index j to the remote control device 40 with a delay of a predetermined value or less. The delay of a predetermined value or less is the bit rate required for the remote control device 40 to transmit one image of priority index j with a delay of a predetermined value or less when the remote control device 40 transmits the image at a target frame rate (f ｔａｒｇｅｔ ) corresponds to the range of delay amount in which the received image can be reproduced. The required bit rate b(j) for transmitting the image of priority index j to the remote control device 40 with a delay amount equal to or less than a predetermined value. ｒｅｑｕｉｒｅｄ The minimum value is preferably set to a bit rate that can realize an image reproduction frame rate for maintaining smooth video display on the remote support device 40 side, for example, an image reproduction frame rate exceeding 30 kbps.

[0232] (Step S203) Next, in step S203, the data processing unit 22 of the automatic driving control device 20 calculates the bit rate (required bit rate s(M)) required for transmitting the M number of images calculated in step S202. ｒｅｑｕｉｒｅｄ ) and the estimated bit rate b ｅ A comparison process is performed with the above.

[0233] The estimated bit rate b ｅ is the bit rate currently available for data transmission (estimated bit rate b ｅ ) which is the estimated bit rate b obtained in step S101 of the flow described above with reference to FIG. ｅ is.

[0234] In step S203, b ｅ ≧s(M) ｒｅｑｕｉｒｅｄIt is determined whether the above formula is true or not. That is, the bit rate required for transmitting M images calculated in step S202 (required bit rate s(M) ｒｅｑｕｉｒｅｄ ) is the estimated bit rate b currently available for communication. ｅ Determine whether it is equal to or less than the above.

[0235] b ｅ ≧s(M) ｒｅｑｕｉｒｅｄ If it is determined that the above formula is true, the process proceeds to step S204. On the other hand, if it is determined that the above formula is false, the process proceeds to step S205.

[0236] (Step S204) In step S203, ｅ ≧s(M) ｒｅｑｕｉｒｅｄ If it is determined that the above determination formula is satisfied, the process proceeds to step S204.

[0237] The validity of the above determination formula is determined by the bit rate required for transmitting M images (required bit rate s(M)) calculated in step S202. ｒｅｑｕｉｒｅｄ ) is the estimated bit rate b currently available for communication. ｅ This corresponds to determining that the number of images is equal to or less than the number of images calculated in step S202. This means that it is possible to transmit all of the M images calculated in step S202. In this case, in step S204, the data processing unit 22 of the automatic driving control device 20 transmits all of the M images calculated in step S202.

[0238] After the process of step S204 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0239] (Step S205) On the other hand, in step S203, ｅ ≧s(M) ｒｅｑｕｉｒｅｄ If it is determined that the above determination formula does not hold, the process proceeds to step S205.

[0240] The failure of the above determination formula is due to the bit rate (required bit rate s(M)) required for transmitting M images calculated in step S202. ｒｅｑｕｉｒｅｄ ) is the estimated bit rate b currently available for communication. ｅ That is, the estimated bit rate b available for communication at the moment is ｅ means that there is a high possibility that a delay or the like will occur if all of the M images calculated in step S202 are transmitted. In this case, the data processing unit 22 of the automatic driving control device 20 executes the processes from step S205 onwards.

[0241] First, in step S205, the data processing unit 22 of the automatic driving control device 20 disables the Mth image with the lowest priority, i.e., sets it as a non-transmitting image whose transmission will be stopped, according to the image priority arrangement generated in step S201.

[0242] (Step S206) Next, in step S206, the data processing unit 22 of the automatic driving control device 20 executes a process of allocating a bit rate to each of the remaining transmission images ((M-1) images) excluding the invalidated images (non-transmission images).

[0243] This process corresponds to the process in step S113 in the flow shown in Fig. 10. The bit rate b(i) reallocated to each of the transmission images ((M-1) images) excluding the invalidated images (non-transmission images) is ａｌｌｏｃａｔｅｄ is calculated by the following formula (Formula 4).

[0244] b(i) ａｌｌｏｃａｔｅｄ =r ｉ (b ｔａｒｇｅｔ -n ｄｉｓａｂｌｅｄ ×b (disabled) ａｌｌｏｃａｔｅｄ ) ...(Formula 4)

[0245] In the above formula (Formula 4), r ｉ is the multiplication coefficient of the i-th image, n ｄｉｓａｂｌｅｄ is the number of invalidated images (number of non-transmitted images). Note that the multiplication coefficient r of the i-th image ｉAs shown in the above-described (Equation 1a), is determined according to the product wi×hi of wi and hi of each image, that is, the resolution corresponding to the number of pixels.

[0246] (Step S207) Next, in step S207, the data processing unit 22 of the automatic driving control device 20 executes an update process of the variable M, which indicates the number of images to be processed, to M = M - 1, i.e., to decrement the value of the variable M by 1. This is a process of excluding invalidated images, i.e., images that have been set as non-transmitting images, from the images to be processed.

[0247] (Step S208) Next, in step S208, the data processing unit 22 of the automatic driving control device 20 determines whether the value of the variable M indicating the number of images to be processed is 1 or greater, i.e., whether the above formula: M≧1 holds.

[0248] If the above formula is satisfied, that is, if the number of images to be processed M is 1 or more, the process returns to step S202 and the processes from step S202 onward are repeated. That is, the estimated bit rate b for the updated number of images M is ｅ The process of comparing the received signal with the received signal, reallocating the bit rate, etc. is repeated.

[0249] These processes are repeated, and when the value of the number of images to be processed M becomes less than 1, that is, when it is determined in step S208 that the above formula M≧1 does not hold, the process proceeds to step S209.

[0250] (Step S209) If it is determined in step S208 that the above formula M≧1 does not hold, the process proceeds to step S209. If the above formula does not hold, M=0, that is, the number of images to be processed is 0.

[0251] This means that the bit rate currently available for data transmission, i.e., the estimated bit rate b ｅ This means that the number of images that can be transmitted below is 0. In this case, the data processing unit 22 of the automatic driving control device 20 disables all images in step S209. That is, all images are set as non-transmitting images.

[0252] After the process of step S209 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0253] As mentioned above, the flowchart shown in FIG. 14 corresponds to the detailed flow of the processing in steps S111 to S114 of the flow previously described with reference to FIG. 10, and in step S108 of the flow shown in FIG. 9, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ Specifically, this is a process to be executed when it is determined that the estimated bit rate b ｅ is the minimum bit rate b shown in FIG. ｍｉｎ This is executed when it is determined that the signal is in a band region C less than 100 MHz.

[0254] In other words, this is processing that is executed when it is difficult to transmit all camera-captured images with low latency (a delay amount equal to or less than a predetermined value). In such a case, the data processing unit 22 of the automatic driving control device 20 executes processing according to the flowchart shown in Fig. 14. That is, images to be transmitted are selected according to priority, and the transmission bit rate is reallocated to each of the selected images to be transmitted and then transmitted.

[0255] By carrying out such processing, at least the selected image can be transmitted to the remote support device 40 without delay, and remote control by the operator 50 can be performed in a timely manner without delay.

[0256] For non-transmitted images with low priority, a process is executed in which information about objects such as vehicles extracted from the non-transmitted images is transmitted as metadata to the remote assistance device 40. That is, this process corresponds to steps S121 to S125 in the flow shown in Fig. 10. This process will be described in detail later.

[0257] A specific example of the image selection and transmission process based on the priority of the images, which is executed according to the flowchart shown in FIG. 14, will be described with reference to FIG.

[0258] The upper part of Fig. 20 shows six images captured by six cameras mounted on the autonomously driven vehicle 10 previously described with reference to Figs. 3 and 4. The images (views) are those captured by the following cameras: (LBv) Left back view (image captured by the left rearward camera) (Bv) Back view (image captured by the rearward camera) (RBv) Right back view (image captured by the right rearward camera) (Lv) Left side view (image captured by the left direction camera) (Fv) Front view (image captured by the frontward camera) (Rv) Right side view (image captured by the right direction camera)

[0259] The data processing unit 22 of the autonomous driving control device 20 arranges the images according to the image priority in step S201 of the flow shown in Fig. 14. That is, as previously described with reference to Fig. 16 etc., the data processing unit 22 calculates the priority of the intermediate images based on elements (analysis parameters) such as (a) the number of peripheral objects, (b) the number of high-risk objects, (c) the remote assistance task, and (d) the line of sight, and arranges the images in descending order of the calculated priority.

[0260] An example of the image arrangement results in this order of priority is shown as a table in the lower left of Fig. 20. In this example, the images are arranged as follows, in descending order of priority: Priority 1 = (Fv) front view (image taken by the front camera) Priority 2 = (Bv) back view (image taken by the rear camera) Priority 3 = (RBv) right back view (image taken by the right rear camera) Priority 4 = (LBv) left back view (image taken by the left rear camera) Priority 5 = (Rv) right side view (image taken by the right camera) Priority 6 = (Lv) left side view (image taken by the left camera)

[0261] In step S202 and subsequent steps of the flow shown in FIG. 14, the data processing unit 22 of the automatic driving control device 20 calculates the bit rate (required bit rate s(M)) required for transmitting the six images arranged according to the priority. ｒｅｑｕｉｒｅｄ ) is calculated.

[0262] The bit rate required to transmit six images (required bit rate s (M) ｒｅｑｕｉｒｅｄ ) is the bit rate b(j) assigned to each of the six images ｒｅｑｕｉｒｅｄ The table at the bottom left of Figure 20 shows the bit rate b(j) assigned to each image. ｒｅｑｕｉｒｅｄ This shows:

[0263] Furthermore, the table at the bottom right of FIG. 20 shows the bit rate b(j) assigned to each image. ｒｅｑｕｉｒｅｄ The table shows the total bit rate (required bit rate per image combination) for a combination of one or more images selected from the highest priority images.

[0264] The total bit rate per image combination, i.e., the required bit rate per image combination, corresponds to the required bit rate for enabling a combination of images selected in order of highest image priority to be transmitted to the remote control device 40 with low delay (a delay amount less than a specified value).

[0265] S (1) ｒｅｑｕｉｒｅｄ is the total bit rate when only the (Fv) front view with priority=1 is selected and transmitted. ｒｅｑｕｉｒｅｄ is the total bit rate when the (Fv) front view with priority = 1 and the (Bv) back view with priority = 2 are selectively transmitted. ｒｅｑｕｉｒｅｄ is the total bit rate when images with priority levels 1 to 3 are selected and transmitted, and S(4) ｒｅｑｕｉｒｅｄ is the total bit rate when images with priority levels 1 to 4 are selected and transmitted, S(5) ｒｅｑｕｉｒｅｄ is the total bit rate when images with priority levels 1 to 5 are selected and transmitted, S(6) ｒｅｑｕｉｒｅｄ indicates the total bit rate when all images with priority levels 1 to 6 are selected and transmitted.

[0266] In steps S202 to S209 of the flow in FIG. 14, first, the total bit rate S(6) when all images with priorities 1 to 6 are selected and transmitted is calculated. ｒｅｑｕｉｒｅｄ is the estimated bit rate b ｅ It is determined whether or not the value is less than b. ｅ ≧S(6) ｒｅｑｕｉｒｅｄIn step S203, it is determined whether the above formula is true. If the above formula is true, the process proceeds to step S204, where all images with priorities 1 to 6 are selected and transmitted.

[0267] On the other hand, if the above formula does not hold, step S205 and subsequent steps are executed to remove the lowest priority image, i.e., the priority 6=(Lv) left side view, from the images to be transmitted. After that, the total bit rate S(5) when selecting and transmitting images of priority 1 to 5 other than the priority 6=(Lv) left side view that has been selected as the non-transmitted image is ｒｅｑｕｉｒｅｄ is the estimated bit rate b ｅ It is determined whether or not the value is less than b. ｅ ≧S(5) ｒｅｑｕｉｒｅｄ In step S203, it is determined whether the above formula is true. If the above formula is true, the process proceeds to step S204, where images with priorities 1 to 5 are selected and transmitted.

[0268] This process is repeated, and images with a low priority are removed from the transmission target, while images with a high priority are selected and transmitted. The total bit rate s(M) is ｒｅｑｕｉｒｅｄ is the estimated bit rate b ｅ If the number of images is less than M, the process of transmitting the M images will be performed.

[0269] However, ultimately, the total bit rate S(1) when selecting and transmitting one image with the highest priority is ｒｅｑｕｉｒｅｄ is the estimated bit rate b ｅ If M≧1 is not less than M, then transmission of all images is not executed. This process corresponds to step S208 in the flow shown in FIG. 14, i.e., the process of invalidating all images in step S209 when the above determination formula M≧1 is determined to be No.

[0270] In this way, by executing the process according to the flow shown in FIG. 14, the bit rate currently available for data transmission (estimated bit rate b ｅ ) or less and with a delay of a predetermined value or less, only images that can be transmitted are selected in accordance with the priority and transmitted to the remote support device 40.

[0271] That is, by performing a comparison process between the bit rate required for transmitting images selected in order of priority and the estimated bit rate while removing images starting with the lowest priority, it becomes possible to transmit only high-priority images to the remote assistance device 40 with low delay (a delay amount equal to or less than a predetermined value). As a result, the autonomously driven vehicle 10 can receive control information from the remote assistance device 40 without delay and travel safely.

[0272] In this process, images to be transmitted are selected and transmitted in order of priority, so images with low priority are not transmitted to the remote support device 40. A specific example of this will be described with reference to FIG.

[0273] In the upper left corner of FIG. 21, the six image arrangement results in order of priority and the bit rate b(j) assigned to each of the six images are shown. ｒｅｑｕｉｒｅｄ The upper right corner shows the bit rate b(j) assigned to each image. ｒｅｑｕｉｒｅｄ The total bit rate is shown when one or more high-priority images are selected.

[0274] For example, as a result of the processing according to the flow shown in FIG. 14, the total bit rate S(5) when images with priorities 1 to 5 are selected and transmitted is ｒｅｑｕｉｒｅｄ is the estimated bit rate b ｅ The total bit rate S(4) for selecting and transmitting images with priorities 1 to 4 is determined to be equal to or greater than the above. ｒｅｑｕｉｒｅｄ is the estimated bit rate b ｅ It is assumed that it is determined that the value is less than b. ｅ ≧S(4) ｒｅｑｕｉｒｅｄ It is assumed that the above formula is established.

[0275] In this case, the data processing unit 22 of the automatic driving control device 20 transmits only the images with priorities 1 to 4 to the remote control device 40. In this case, if only the images with priorities 1 to 4 transmitted to the remote control device 40 are displayed on the display unit 43 of the remote control device 40, the displayed image will be as shown in the lower part of FIG.

[0276] That is, Priority 1 = (Fv) Front view (image taken by the front camera) Priority 2 = (Bv) Back view (image taken by the rear camera) Priority 3 = (RBv) Right back view (image taken by the right rear camera) Priority 4 = (LBv) Left back view (image taken by the left rear camera) Images with priorities 1 to 4 can be displayed.

[0277] However, Priority 5 = (Rv) Right side view (image taken by camera in the right direction) Priority 6 = (Lv) Left side view (image taken by camera in the left direction) Images with priorities 5 and 6 cannot be displayed.

[0278] In such a state, it becomes difficult for the operator 50 at the remote assistance center 30 to determine the correct control processing to be performed. As a result, a situation occurs in which correct control information cannot be transmitted to the autonomously driven vehicle 10. To prevent such a situation, the data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 shown in FIG. 2 to generate and display a 3DCG image observed from the position and direction of the camera mounted on the autonomously driven vehicle 10. A specific example is shown in FIG. 22.

[0279] The example shown in Figure 22 is an example in which 3DCG images observed from the positions and directions of the two cameras are generated and displayed in the display area of ​​the images with priorities 5 and 6, which are the following two low-priority images that were not transmitted from the automatic driving control device 20: Priority 5 = (Rv) Right side view (image taken by the camera to the right) Priority 6 = (Lv) Left side view (image taken by the camera to the left).

[0280] The data processing unit 42 of the remote assistance device 40 further analyzes metadata including information on surrounding objects such as oncoming vehicles received from the automatic driving control device 20, and performs processing such as displaying images of objects around the automatically driven vehicle 10 on a 3DCG image. By performing such processing, display data that more accurately reproduces the current surrounding situation of the automatically driven vehicle 10 is generated and displayed on the display unit.

[0281] Through this processing, the operator 50 can check the current surrounding conditions of the autonomous vehicle 10 from the camera images, CG images, and object images displayed on the display unit 43, and can transmit optimal control information (autonomous driving control information) to the autonomous vehicle 10.

[0282] The automatic driving control device 20 generates metadata including information about surrounding objects such as oncoming vehicles included in the non-transmitted image, and transmits the metadata to the remote assistance device 40. Details of this process will be described below.

[0283] 6. Details of Transmission Processing of Objects Detected from Non-Transmitting Images Next, details of transmission processing of objects detected from non-transmitting images will be described.

[0284] 8, in addition to a communication path for transmitting multiple video tracks in a single stream, a communication path for metadata is set between the autonomous driving control device 20 and the remote assistance device 40. The metadata includes information about objects detected from camera images and LiDAR sensor information, such as surrounding vehicles, pedestrians, and buildings.

[0285] As described above, the automatic driving control device 20 performs communication control to change the metadata to be transmitted as needed depending on the communication bandwidth (bit rate) available for metadata communication between the automatic driving control device 20 and the remote assistance device 40.

[0286] For example, object coordinates, bounding box size, and labels indicating object type, which are used to identify the position and size of objects (objects such as vehicles and pedestrians around the autonomous vehicle 10), are always transmitted.

[0287] However, for large-volume data such as object images, if the bandwidth of the communication path that can be secured for metadata communication is narrow, processing such as stopping transmission, transmitting compressed data, or filtering to select transmission targets according to the risk level of each object is performed. The risk level of the object is determined by the data processing unit 22 of the autonomous driving control device 20, and this determination result is also transmitted to the remote assistance device 40 as metadata.

[0288] The flowchart shown in Fig. 23 explains the sequence of communication control processing for large-volume metadata such as images of objects, executed by the autonomous driving control device 20. Specifically, this flowchart explains the details of the transmission processing for objects detected from non-transmitted images.

[0289] 23 corresponds to the detailed flow of the processing in steps S121 to S125 of the flow previously described with reference to FIG. 10. That is, in step S108 of the flow shown in FIG. 9, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ Specifically, this is a process to be executed when it is determined that the estimated bit rate b ｅ is the minimum bit rate b shown in FIG. ｍｉｎ This is executed when it is determined that the signal is in a band region C less than 100 MHz.

[0290] The details of the processes in steps S301 to S309 of the flow shown in FIG. 23 will be explained below in order.

[0291] (Step S301) First, the bit rate available for data transmission (estimated bit rate b ｅ ) is the minimum bit rate b ｍｉｎ If it is determined that the estimated bit rate is less than b ｅ is the minimum bit rate b shown in FIG. ｍｉｎIf it is determined that the image is in a bandwidth region C less than this, the data processing unit 22 of the automatic driving control device 20 determines in step S301 whether or not there is a non-transmitted image (disabled video) that is an image excluded from the transmission target.

[0292] As explained above with reference to the flow shown in FIG. 14, the bit rate available for data transmission (estimated bit rate b ｅ ) is a low bit rate that cannot transmit all of the camera-captured images, images with low priority are excluded from the transmission targets as images not to be transmitted. In step S301, it is determined whether or not images not to be transmitted have been set in the processing according to the flow shown in FIG.

[0293] If no non-transmitted image is set, the process is terminated without executing the processes from step S302 onward, and the process returns to step S101. ｅ and obtain the new estimated bit rate b ｅ On the other hand, if it is determined that the non-transmitted image is set, the process from step S302 onward is executed. That is, the process extracts an object such as a vehicle from the non-transmitted image and transmits it to the remote assistance device 40 as metadata.

[0294] (Step S302) If it is determined in step S301 that there is an image that will not be transmitted, the data processing unit 22 of the automatic driving control device 20 selects one image frame that constitutes the image that will not be transmitted in step S302.

[0295] (Step S303) Next, in step S303, the data processing unit 22 executes image segmentation processing on one image frame that constitutes the non-transmitted image.

[0296] Image segmentation processing, for example, is semantic segmentation processing, which is processing for dividing a pixel region into pixels that make up an image according to the type of object each pixel belongs to. For example, by setting different colors according to the type of various objects contained in the image, such as vehicles, people, trees, etc., it is possible to generate information that makes it possible to identify the object type of each pixel.

[0297] A specific example of image segmentation processing will be described with reference to Fig. 24. Fig. 24(a) shows one image frame of an image captured by one camera mounted on the autonomously driven vehicle 10. The result of performing image segmentation processing on this image frame is the image segmentation result shown in Fig. 24(b).

[0298] As shown in Fig. 24(b), when image segmentation processing is performed on the camera-captured image shown in Fig. 24(a), a different color is set depending on the type of object to which each pixel constituting the camera-captured image belongs. For example, by setting different colors depending on the type of object contained in the image, such as red for vehicles, blue for people, and green for trees, it is possible to generate information on the object type for each pixel region.

[0299] In step S303, an image segmentation process is performed on one image frame that constitutes the non-transmitted image to generate discernible data for objects contained in the image.

[0300] (Step S304) Next, in step S304, the data processing unit 22 executes a process of extracting an object included in one image frame constituting the non-transmitted image, using the result of the image segmentation process in step S303. This process corresponds to the process in step S121 in the flowchart shown in FIG.

[0301] The types of objects to be extracted are specified in advance. For example, objects that affect the running of the autonomously driven vehicle 10, such as vehicles and people, are specified in advance as objects to be extracted, and objects corresponding to the object types to be extracted are extracted using the results of the image segmentation process.

[0302] A specific example of this process will be described with reference to Fig. 24. The object extraction result (c) in Fig. 24 is an object extracted from the camera image (a) using the image segmentation process result shown in Fig. 24(b). In the example shown in Fig. 24, the extracted objects are composed of vehicles, pedestrians, bicycles, etc.

[0303] Note that this example of object extraction processing is just one example, and in addition to examples in which only vehicles and people are extracted, it is also possible to set traffic lights, signs, poles, etc. on the road as objects to be extracted and perform processing to extract these objects.

[0304] (Step S305) Next, in step S305, the data processing unit 22 calculates the risk level of each of the objects extracted in step S304, and generates array data in which the extracted objects are arranged in descending order of risk.

[0305] This process corresponds to the process of step S122 in the flow shown in Fig. 10. As described in the process of step S122 in the flow shown in Fig. 10, for example, an object with a high collision risk is determined to be a high risk.

[0306] The risk levels of objects in an image are also calculated during the image priority determination process described above with reference to Fig. 16. The object-by-object risk level calculation process performed in step S305 is also executed as the same process as the object risk level calculation process used in the image priority calculation process.

[0307] Specifically, as described above, for example, processing based on Responsibility Sensitive Safety (RSS) can be used. RSS is a method for evaluating the suitability of a pair of an object detected from an image and the autonomously driven vehicle 10, which is the vehicle itself.

[0308] Alternatively, the method using the neural network described above with reference to Fig. 19 can also be applied. The input data for the neural network includes the type, size, direction of movement, distance from the vehicle, network status, weather, road conditions, etc. of the object detected from each image. The output is a risk level for the object.

[0309] In step S305, for example, these methods are applied to calculate the risk level of each object extracted from the image in step S304, and generate array data in which the extracted objects are arranged in order of decreasing risk. For example, as shown in Figure 25(d), array data is generated in which the extracted objects are arranged in order of decreasing risk from risk level 1 to risk level 4.

[0310] Similar to the risk levels used in the image priority calculation process, the risk levels corresponding to objects are output as levels according to predefined level definitions. For example, the level definitions can be defined as follows, using the results of analyzing past accident situation data at the current location of the autonomously driven vehicle 10: Risk level 1 = serious accident occurred Risk level 2 = medium accident occurred Risk level 3 = minor accident occurred Risk level 4 = no reported injuries

[0311] (Step S306) Next, in step S306, the data processing unit 22 of the automatic driving control device 20 calculates a bit rate that can be used for transmitting metadata.

[0312] As explained above with reference to FIG. 8, in addition to a communication path for transmitting multiple video tracks in a single stream, a communication path for metadata is set between the automatic driving control device 20 and the remote assistance device 40. The bandwidth available for the communication path for metadata also fluctuates as needed depending on the network conditions, similar to the communication path for images. Specifically, the estimated bit rate b ｅ Similarly, it changes from time to time.

[0313] In step S306, the communication network monitoring unit 112 of the data processing unit 22 of the automatic driving control device 20 shown in FIG. 5 calculates a bit rate that can be used for transmitting metadata.

[0314] (Step S307) Next, in step S307, the data processing unit 22 of the automatic driving control device 20 compresses each of the extracted objects in a different compression mode (different compression algorithm) according to the risk level of each of the extracted objects determined in step S305.

[0315] A specific example of different compression processes according to the risk level of an object will be described with reference to Fig. 26. As shown in Fig. 26, an object with the highest risk level, risk level 1, is compressed using compression algorithm c corresponding to risk level 1. ＲＬ１ By applying the compression process, ＲＬ１ The next highest risk level object is risk level 2, which is converted into a compressed image using the compression algorithm c for risk level 2. ＲＬ２ By applying the compression process, ＲＬ２ The next highest risk level object is risk level 3, which is converted into a compressed image using the compression algorithm c for risk level 3. ＲＬ３ By applying the compression process, ＲＬ３ The object with the lowest risk level, risk level 4, is converted into a compressed image using the compression algorithm c for risk level 4. ＲＬ４ By applying the compression process, ＲＬ４ It is converted into a compressed image.

[0316] In addition, c ＲＬ１ , c ＲＬ２ , cＲＬ３ , c ＲＬ４ indicate the compression rates for risk levels 1 to 4, respectively, and the compression rates corresponding to each risk level have the following relationship: ＲＬ１ <c ＲＬ２ <c ＲＬ３ <c ＲＬ４ In other words, the higher the risk level, the lower the compression rate, and the lower the risk level, the higher the compression rate. This means that for objects with a high risk level, the compression rate is reduced to generate a compressed image that will produce a restored image close to the original object image. On the other hand, for objects with a low risk level, the compression rate is increased to generate a compressed image that will produce an abstract restored image.

[0317] Specific examples of the compression process according to the risk level of the object and the filtering process (processing for selecting an object image to be transmitted) executed in steps S306 to S308 will be described in detail later using the flowcharts shown in FIGS. 27 and 28.

[0318] (Step S308) Next, in step S308, the data processing unit 22 of the automatic driving control device 20 executes a filtering process on the extracted and compressed object image data. The filtering process is a process of selecting compressed object image data to be transmitted, and is a process of classifying compressed object image data to be transmitted and compressed object image data not to be transmitted.

[0319] However, this filtering process is performed when the bit rate required for transmitting the compressed object data is greater than the available bandwidth, and this process will be described in detail later.

[0320] (Step S309) Next, in step S309, the data processing unit 22 of the automatic driving control device 20 transmits the compressed object image data finally selected as the transmission target as a result of the filtering process in step S308 to the remote assistance device 40 as metadata.

[0321] By executing the processing according to the flow shown in Figure 23, if an object such as an oncoming vehicle is detected in an image that has been designated as a non-transmitted image (invalid video track), information about this object will be transmitted to the remote assistance device 40 as metadata.

[0322] Next, with reference to the flowcharts shown in Figures 27 and 28, we will explain the detailed sequence of the processing of steps S306 to S308 of the flow in Figure 23, i.e., the compression processing according to the risk level of the object and the filtering processing (selection processing of the transmission object image).

[0323] The definitions of the parameters used in the flowcharts shown in FIGS. 27 and 28 are as follows, as shown in the table in FIG. 29: ｒｅｑｕｉｒｅｄ : total bitrate required to transmit images of all extracted objects x ａ : Bandwidth (bit rate) available for transmitting the image of the object (object image) R: Risk level of the object R∈[1, 2, 3, 4] X(1) ｒｅｑｕｉｒｅｄ : Total bit rate required to transmit extracted object images of risk level 1 × (2) ｒｅｑｕｉｒｅｄ : The total bit rate required to transmit extracted object images at risk levels 1 and 2 x (3) ｒｅｑｕｉｒｅｄ : The total bit rate required for transmitting extracted object images of risk levels 1, 2, and 3 × (4) ｒｅｑｕｉｒｅｄ : The sum of the bit rates required to transmit all extracted object images (X(4) ｒｅｑｕｉｒｅｄ =X ｒｅｑｕｉｒｅｄ )

[0324] The processing of each step shown in the flowcharts of Figures 27 and 28 will be explained below in order. (Step S311) First, in step S311, the data processing unit 22 of the automatic driving control device 20 executes the following two types of bit rate calculation processing. (1) The bit rate required to transmit all extracted objects (all object transmission bit rate X ｒｅｑｕｉｒｅｄ(2) Bit rate (object image transmission bit rate x ) that is a bandwidth available for transmitting object images in the metadata communication path ａ )

[0325] The extracted object is an object extracted in step S304 of the flow shown in FIG. 23 (=step S121 of the flow shown in FIG. 10), and is an object extracted from a non-transmitted image. The type of object to be extracted (vehicle, pedestrian, etc.) is specified in advance. Also, the bit rate required to transmit all extracted objects is the all-object transmission bit rate X ｒｅｑｕｉｒｅｄ is the bit rate at which it is determined that all extracted objects can be transmitted to the remote support device 40 without delay.

[0326] (Step S312) Next, in step S312, the data processing unit 22 of the automatic driving control device 20 calculates the following two bit rates calculated in step S311, i.e., the all-object transmission bit rate X ｒｅｑｕｉｒｅｄ , the bit rate available for transmitting object images x ａ , a comparison process is performed between these two bit rates.

[0327] X ｒｅｑｕｉｒｅｄ ≦x ａ It is determined whether the above formula is true or not. That is, the bit rate for transmitting all objects X ｒｅｑｕｉｒｅｄ but 、 Available bit rate for object image transmission x ａ If it is determined that the above formula is not true, the process proceeds to step S314.

[0328] (Step S313) In step S312, X ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above determination formula is satisfied, the process proceeds to step S313.

[0329] The above formula is satisfied when the bit rate for transmitting all objects X ｒｅｑｕｉｒｅｄ but 、Available bit rate for object image transmission x ａ This means that all object images can be transmitted with low latency (a delay equal to or less than a predetermined value) using the current available communication bandwidth (bit rate). In this case, in step S313, the data processing unit 22 of the automatic driving control device 20 transmits all object image data of objects extracted from the non-transmitted images to the remote assistance device 40 as is, without compressing them.

[0330] After the process of step S313 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0331] (Step S314) On the other hand, in step S312, X ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above determination formula does not hold, the process proceeds to step S314.

[0332] If the above formula is not satisfied, the bit rate for transmitting all objects X ｒｅｑｕｉｒｅｄ but 、 Available bit rate for object image transmission x ａ This means that it is difficult to transmit all object images with low delay (delay amount less than a predetermined value) using the currently available communication band (bit rate). In other words, the bit rate x currently available for transmitting object image data is ａ means that there is a high possibility that delays or the like will occur if all of the extracted object images are transmitted as they are. In this case, the data processing unit 22 of the automatic driving control device 20 executes the processes from step S314 onwards.

[0333] First, in step S314, the data processing unit 22 of the automatic driving control device 20 compresses the extracted objects in ascending order of the object risk level (1 to 4) in accordance with the compression mode (compression algorithm cＲＬ１ ~c ＲＬ４ ) to compress it.

[0334] Compression mode (compression algorithm c ＲＬ１ ~c ＲＬ４ ) is the compression algorithm corresponding to the risk level described above with reference to FIG.

[0335] In step S314, first, only the object image data at risk level 4, which is the lowest risk level, is compressed, and the process proceeds to step S315.

[0336] (Step S315) Next, in step S315, the data processing unit 22 of the automatic driving control device 20 determines a bit rate for transmitting all of the compressed object image data of the objects with low risk levels compressed in step S314 and the uncompressed object data with high risk levels that was not compressed in step S314, i.e., the all-object transmission bit rate X ｒｅｑｕｉｒｅｄ and the available bit rate for communication x ａ , a comparison process is performed between these two bit rates.

[0337] X ｒｅｑｕｉｒｅｄ ≦x ａ It is determined whether the above formula is true or not. That is, the bit rate for transmitting all objects X ｒｅｑｕｉｒｅｄ but 、 Available bit rate for communication x ａ If it is determined that the above formula is not true, the process proceeds to step S318.

[0338] (Step S317) In step S316, X ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above determination formula is satisfied, the process proceeds to step S317.

[0339] The determination of whether the above-mentioned formula is satisfied is determined by whether the all-object transmission bit rate Xｒｅｑｕｉｒｅｄ but 、 Available bit rate for communication x ａ This means that:

[0340] This means that all object images (at least some of which are compressed object images) can be transmitted with low latency (a delay equal to or less than a predetermined value) using the currently available communication bandwidth (bit rate). In this case, in step S317, the data processing unit 22 of the automatic driving control device 20 transmits all of the object images (at least some of which are compressed object images) extracted from the non-transmitted images to the remote assistance device 40.

[0341] After the process of step S317 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0342] (Step S318) On the other hand, in step S316, X ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above determination formula does not hold, the process proceeds to step S318.

[0343] The failure of the above-mentioned criterion formula is due to the fact that the bit rate X for transmitting the object image (at least a part of which is a compressed object image) for transmitting the object image extracted from the non-transmitted image is ｒｅｑｕｉｒｅｄ but 、 Available bit rate for communication x ａ This means that it is difficult to transmit all object images (at least some of which are compressed object images) with low delay (delay amount equal to or less than a predetermined value) using the currently available communication bandwidth (bit rate).

[0344] That is, the currently available bit rate x ａmeans that there is a high possibility that delays or the like will occur if all of the image data of the extracted objects (at least some of which are compressed object images) are transmitted. In this case, the data processing unit 22 of the automatic driving control device 20 executes the processes from step S318 onwards.

[0345] In this case, the data processing unit 22 of the automatic driving control device 20 determines in step S318 whether there are any uncompressed object images for which compression processing has not been performed. In step S314 described above, the extracted objects are compressed in order of decreasing object risk level (1 to 4) using a compression mode (compression algorithm c) according to the risk level. ＲＬ１ ~c ＲＬ４ ) to perform the compression process.

[0346] First, compression processing is performed on objects with a risk level of 4, and the processing from step S315 onward is performed. At this point, the determination formula of step S316, i.e., X ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above formula is true, the process proceeds to step S317, where the transmission process of the compressed image data of the object at risk level 4 and the uncompressed image data of the object at risk levels 1 to 3 is executed.

[0347] However, after the compression process of the object of risk level 4 is performed, the determination formula of step S316, i.e., X ｒｅｑｕｉｒｅｄ ≦x ａ If the above formula is not satisfied, the determination in step S316 is No, and the determination in step S318 is Yes for "Is there uncompressed data?" In this case, the process returns to step S314, and compression processing is performed on the next object with risk level 3.

[0348] In this way, in step S314, compression processing is performed sequentially for objects with higher risk levels.

[0349] Finally, in step S314, the compression process is performed on the object with the highest risk level, i.e., the object with risk level 1. At this point, the determination formula in step S316, i.e., ｒｅｑｕｉｒｅｄ ≦xａ If the above formula does not hold, it is determined that even if all objects at risk levels 1 to 4 are converted into compressed image data, it will be difficult to transmit them with low delay using the current available communication band (bit rate).

[0350] In this case, the determination in step S316 is No, and the determination in step S318 of "Is there uncompressed data?" is also No. In this case, the processing from step S321 onwards is executed. In step S321 onwards, a process of selecting objects to be transmitted from the automatic driving control device 20 to the remote assistance device 40 is executed, that is, a transmission object image selection process (filtering) is executed in which not all extracted objects are to be transmitted, but some are not to be transmitted.

[0351] (Step S321) If the determination in step S318 is No, that is, if it is determined that even if all objects of risk levels 1 to 4 are converted into compressed image data, low-latency transmission processing is difficult with the current communication bandwidth (bit rate), the processing of step S321 is executed.

[0352] In step S321, the data processing unit of the automatic driving control device 20 calculates the bit rate required for transmitting the compressed object image for each of the following combinations: *Bit rate X(1) required for transmitting compressed object image data of risk level 1 ｒｅｑｕｉｒｅｄ *Bit rate X(2) required for transmitting compressed object image data of risk levels 1 and 2 ｒｅｑｕｉｒｅｄ *Bit rate required for transmitting compressed object image data for risk levels 1 to 3 X (3) ｒｅｑｕｉｒｅｄ Each of these bit rates is calculated.

[0353] (Step S322) Next, in step S322, the data processing unit of the automatic driving control device 20 calculates the bit rate X(3) required for transmitting compressed object image data of risk levels 1 to 3 calculated in step S321. ｒｅｑｕｉｒｅｄ The bit rate required for transmitting the compressed object image data of risk levels 1 to 3 and the available bit rate x, which is the bit rate available for current communication, areａ , a comparison process is performed between these two bit rates.

[0354] That is, X(3) ｒｅｑｕｉｒｅｄ ≦x ａ It is determined whether the above formula is true. If it is determined that the formula is true, that is, the bit rate X(3) required for transmitting compressed object image data of risk levels 1 to 3 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the above formula is not true, the process proceeds to step S324.

[0355] (Step S323) In step S322, X(3) ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above formula is true, that is, the bit rate X(3) required for transmitting compressed object image data of risk levels 1 to 3 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the number is equal to or less than the number, the process proceeds to step S323.

[0356] In this case, in step S323, the data processing unit 22 of the automatic driving control device 20 transmits only the objects of risk levels 1 to 3 among the extracted objects as compressed image data. Note that objects of risk level 4 are not transmitted.

[0357] After the process of step S323 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0358] (Step S324) On the other hand, in step S322, X(3) ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above formula does not hold, that is, the bit rate X(3) required for transmitting compressed object image data of risk levels 1 to 3 is ｒｅｑｕｉｒｅｄis the available bit rate x ａ If it is determined that the value is not equal to or greater than the predetermined value, the process proceeds to step S324.

[0359] In step S324, the data processing unit of the automatic driving control device 20 calculates the bit rate X(2) required for transmitting compressed object image data of risk levels 1 to 2 calculated in step S321. ｒｅｑｕｉｒｅｄ The bit rate required for transmitting the compressed object image data of risk level 1 to 2 and the available bit rate x, which is the bit rate available for current communication, are ａ , a comparison process is performed between these two bit rates.

[0360] That is, X(2) ｒｅｑｕｉｒｅｄ ≦x ａ It is determined whether the above formula is true. If it is determined that the formula is true, that is, the bit rate X(2) required for transmitting compressed object image data of risk levels 1 to 2 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the above formula is not true, the process proceeds to step S326.

[0361] (Step S325) In step S324, X(2) ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above formula is true, that is, the bit rate X(2) required for transmitting compressed object image data of risk levels 1 to 2 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the number is equal to or less than the predetermined number, the process proceeds to step S325.

[0362] In this case, in step S325, the data processing unit 22 of the automatic driving control device 20 transmits only the extracted objects with risk levels 1 and 2 as compressed image data. Note that objects with risk levels 3 and 4 are not transmitted.

[0363] After the process of step S325 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0364] (Step S326) On the other hand, in step S324, X(2) ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above formula does not hold, that is, if the bit rate X(2) required for transmitting compressed object image data of risk levels 1 to 2 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the value is not equal to or less than the predetermined value, the process proceeds to step S326.

[0365] In step S326, the data processing unit of the automatic driving control device 20 calculates the bit rate X(1) required for transmitting compressed object image data of risk level 1 calculated in step S321. ｒｅｑｕｉｒｅｄ The bit rate required for transmitting this risk level 1 compressed object image data and the available bit rate x, which is the bit rate available for current communication, are ａ , a comparison process is performed between these two bit rates.

[0366] That is, X(1) ｒｅｑｕｉｒｅｄ ≦x ａ It is determined whether the above formula is true. If it is determined that the formula is true, that is, the bit rate X(1) required for transmitting compressed object image data of risk level 1 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the above formula is not true, the process proceeds to step S328.

[0367] (Step S327) In step S326, X(1) ｒｅｑｕｉｒｅｄ ≦x ａIf it is determined that the above formula is true, that is, the bit rate X(1) required for transmitting compressed object image data of risk level 1 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the number is equal to or less than the number, the process proceeds to step S327.

[0368] In this case, in step S327, the data processing unit 22 of the automatic driving control device 20 transmits only the objects of risk level 1 among the extracted objects as compressed image data. Note that objects of risk levels 2 to 4 are not transmitted.

[0369] After the process of step S327 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0370] (Step S328) On the other hand, in step S326, X(1) ｒｅｑｕｉｒｅｄ ≦x ａ If it is determined that the above formula does not hold, that is, if the bit rate X(1) required for transmitting compressed object image data of risk level 1 is ｒｅｑｕｉｒｅｄ is the available bit rate x ａ If it is determined that the value is not equal to or greater than this, the process proceeds to step S328.

[0371] In this case, in step S328, the data processing unit 22 of the autonomous driving control device 20 notifies the autonomous driving control device 20 of the autonomous driving vehicle 10 that remote assistance is not possible. In other words, the data processing unit 22 notifies the autonomous driving control device 20 of the autonomous driving vehicle 10 that remote assistance is not possible because the object in the non-transmitted image cannot be confirmed at all. Upon receiving such a notification, the autonomous driving control device 20 of the autonomous driving vehicle 10 will take action such as emergency stop processing of the autonomous driving vehicle.

[0372] After the process of step S328 is completed, the process returns to step S101 in the flow shown in FIG. 9, and the processes from step S101 onward are repeated. That is, the new estimated bit rate b ｅ and obtain the new estimated bit rate b ｅ Repeatedly execute the process according to the

[0373] As can be understood from the explanation of the flowcharts shown in Figures 27 and 28, the data processing unit 22 of the autonomous driving control device 20 of the autonomous driving vehicle 10 performs control processing of the transmission data to reliably transmit information about objects detected from non-transmitted images (invalid video tracks) to the remote assistance device 40.

[0374] Specifically, information about an object is transmitted using the currently available communication bandwidth (bit rate x ａ ) and performs compression processing according to the risk level of the object and selection (filtering) of the data to be transmitted, as processing to ensure transmission with low latency, before transmitting the data to the remote support device 40.

[0375] 7. Processing Executed by the Remote Support Device Next, processing executed by the remote support device will be described.

[0376] The remote assistance device 40 receives camera images transmitted by the autonomous driving control device 20 of the autonomous driving vehicle 10, as well as metadata consisting of object images, object position information, etc., and uses this received information to display an image showing the surrounding situation of the autonomous driving vehicle on the display unit 42 of the remote assistance device 40.

[0377] Fig. 30 is a diagram showing an example of an image that the remote assistance device 40 displays on the display unit 22 using data transmitted from the automatic driving control device 20 of the automatic driving vehicle 10. The example shown in Fig. 30 shows an example in which the following four camera images are transmitted from the automatic driving control device 20 to the remote assistance device 40 in accordance with image priority: Priority 1 = (Fv) front view (image captured by the front camera) Priority 2 = (Bv) back view (image captured by the rear camera) Priority 3 = (RBv) right back view (image captured by the right rear camera) Priority 4 = (LBv) left back view (image captured by the left rear camera)

[0378] The automatic driving control device 20 does not transmit the following two low priority images: Priority 5 = (Rv) Right side view (image taken by the right-direction camera) Priority 6 = (Lv) Left side view (image taken by the left-direction camera). These images with priorities 5 and 6 are not transmitted.

[0379] The remote assistance device 40 can output and display the four high-priority images transmitted from the automatic driving control device 20 on the display unit 43. However, the remote assistance device 40 cannot display the camera-captured images of the two low-priority images that were not transmitted from the automatic driving control device 20.

[0380] The data processing unit 42 of the remote assistance device 40 performs processing to generate and display 3DCG images observed from the positions and directions of these two cameras in the display area of ​​these two low-priority images that were not transmitted from the automatic driving control device 20.

[0381] Furthermore, the data processing unit 42 of the remote assistance device 40 analyzes metadata including information on surrounding objects such as oncoming vehicles received from the automatic driving control device 20, and performs processing such as displaying images of objects around the automatic driving vehicle 10 on a 3DCG image. By performing such processing, display data that more accurately reproduces the current surrounding situation of the automatic driving vehicle 10 is generated and displayed on the display unit.

[0382] Through this processing, the operator 50 can check the current surrounding conditions of the autonomous vehicle 10 from the camera images, CG images, and object images displayed on the display unit 43, and can transmit optimal control information (autonomous driving control information) to the autonomous vehicle 10.

[0383] In addition, for images that could not be received from the automatic driving control device 20, the data processing unit 42 of the remote assistance device 40 performs processing to generate and display 3DCG images observed from the position and direction of the camera that captured the images.

[0384] Furthermore, the data processing unit 42 of the remote assistance device 40 generates a 3DCG image observed from the position and direction of the camera that captured the image, even if the image received from the automatic driving control device 20 is unclear, and performs processing to generate and display a composite image by combining (superimposing) the generated 3DCG image with the received image.

[0385] The communication path between the automatic driving control device 20 and the remote support device 40 may be subject to sudden fluctuations in available bandwidth or congestion, which may result in loss or delay of communication packets, resulting in a blurred image in the data received by the remote support device 40. In such cases, the data processing unit 42 of the remote support device 40 performs processing to generate and display a composite image by combining (superimposing) the 3DCG image and the received image.

[0386] The 3DCG image is generated using a 3D map stored in the 3D map storage unit 45 in the remote support device 40 shown in Fig. 2. However, as described above, the 3D map may be acquired from an external server.

[0387] The 3D map is a map created in advance using equipment such as LiDAR. A method for generating a 3DCG image from a specific viewpoint, for example, a specific camera position, using a 3D map can be, for example, a method using a Visual Positioning System (VPS). The VPS uses a database that records the relationship between image feature points and positions, making it possible to generate a 3DCG image from a specific viewpoint, for example, a specific camera position.

[0388] It is also possible to acquire the entire space as a point cloud using a technique for estimating a 3D shape using feature points of an image (Structure from Motion, SfM).

[0389] When generating a 3DCG image using the VPS, a camera image is first acquired from the vehicle and compared with a database. If a camera image cannot be acquired, the image acquired one frame before is used. Based on this image, point cloud data is acquired, including the position and orientation of the autonomous vehicle 10, as well as surrounding roads and buildings. Next, polygon shapes of various objects are reconstructed based on the acquired point cloud data, and unnecessary portions are then deleted. This process allows the creation of a 3DCG image observed from a specific viewpoint, i.e., a specific camera position. Note that the point cloud data may be acquired from an external server or may be stored within the remote assistance device 40. Furthermore, the process of creating a 3DCG image observed from a specific camera position using a 3D map requires polygon shape reconstruction processing and unnecessary portion deletion processing. These processes may be performed independently by the data processing unit 42 of the remote assistance device 40 or in cooperation with an external data processing server.

[0390] [8. Details of the Combining (Superimposing) Process of Camera-Captured Image and 3DCG Image Executed by the Remote Support Device] Next, details of the combining (superimposing) process of the camera-captured image and 3DCG image executed by the data processing unit 42 of the remote support device 40 will be described.

[0391] As explained above, the data processing unit 42 of the remote assistance device 40 not only processes images that could not be received from the automatic driving control device 20 by generating and displaying a 3DCG image observed from the position and direction of the camera that captured the image, but also generates a 3DCG image observed from the position and direction of the camera that captured the image for images that were received, if the received image is not clear, and then combines (superimposes) the generated 3DCG image with the received image and displays it on the display unit 43.

[0392] In the process of combining the 3DCG image and the received image, the data processing unit 42 of the remote assistance device 40 executes processing using the opacity of the 3DCG image as a parameter for adjusting the combination ratio between the 3DCG image and the received image.

[0393] The opacity of a 3DCG image, which is an adjustment parameter for the composition ratio between a 3DCG image and a received image, will be described with reference to FIG. 31 and subsequent figures.

[0394] In Fig. 31, the following images are shown from the top: (a) 3DCG image, (b) camera-captured image, and (c) composite (superimposed) image (= display output image).

[0395] The 3DCG image (a) is a 3DCG image generated by the map control unit 121 of the data processing unit 42 of the remote support device 40, and is a 3DCG image generated using a 3D map stored in the 3D map storage unit 45. This 3DCG image is a 3DCG image viewed from the same camera position and direction as the camera-captured image (b).

[0396] (b) The camera-captured image is a camera-captured image received from the automatic driving control device 20 of the automatically driven vehicle 10. The image is captured by a camera mounted on the automatically driven vehicle 10.

[0397] (c) The composite (superimposed) image (=display unit output image) is an image generated by the display control unit 122 of the data processing unit 42 of the remote support device 40. The display control unit 122 of the data processing unit 42 of the remote support device 40 generates the composite image by performing a synthesis process (superimposition process) of (a) the 3DCG image and (b) the image captured by the camera. The generated composite image is output to the display unit 43 of the remote support device 40 and is observed by the operator 50.

[0398] Figure 31 shows an example of synthesis processing when the opacity of the 3DCG image, which is an adjustment parameter for the synthesis ratio between the 3DCG image and the received image, is set to 0, and an example of synthesis processing when the opacity of the 3DCG image is set to 1, according to two different parameter settings.

[0399] The example of synthesis processing where the opacity (Opacity) of the 3DCG image is 0 is an example in which the 3DCG image is set to a completely transparent image, and in which the 3DCG image is not actually used, and a synthetic image is generated using only the "(b) camera-captured image" received from the autonomously driving vehicle 10.

[0400] The compositing process in which the opacity of this 3DCG image is set to 0 is an example of a compositing process that is executed when the data processing unit 42 of the remote assistance device 40 determines that the clarity of the ``(b) camera-captured image'' received from the autonomously driving vehicle 10 is sufficient.

[0401] On the other hand, the example of synthesis processing where the opacity (Opacity) of the 3DCG image is 1 is an example in which the 3DCG image is a completely opaque image, that is, a synthetic image is generated using only the 3DCG image generated from the 3D map, and without using at all the "(b) camera-captured image" received from the autonomously driving vehicle 10.

[0402] This compositing process in which the opacity of the 3DCG image is set to 1 is an example of compositing process that is executed when the data processing unit 42 of the remote assistance device 40 does not receive the "(b) camera-captured image" from the autonomously driven vehicle 10. Alternatively, the same process is executed when the (b) camera-captured image received from the autonomously driven vehicle 10 is significantly unclear.

[0403] The opacity of the 3DCG image, which is an adjustment parameter for the composition ratio between the 3DCG image and the received image, can be adjusted arbitrarily between 0 and 1. The display control unit 122 of the data processing unit 42 of the remote assistance device 40 controls the value of the opacity of the 3DCG image in accordance with the clarity of the camera-captured image (b) received from the autonomously driving vehicle 10, and executes the image composition process.

[0404] 32 shows examples of image synthesis processing based on the following three different parameter settings: an example of synthesis processing when the opacity (Opacity) of the 3DCG image is set to 0.25; an example of synthesis processing when the opacity (Opacity) of the 3DCG image is set to 0.5; and an example of synthesis processing when the opacity (Opacity) of the 3DCG image is set to 0.75. Examples of synthesis processing according to these three different parameter settings are shown.

[0405] The lower the clarity of the "(b) camera-captured image" that the data processing unit 42 of the remote assistance device 40 receives from the autonomously driving vehicle 10, the larger the opacity of the 3DCG image is set to. In other words, the output ratio of the 3DCG image in the composite image is set to be large.

[0406] In the example shown in Figure 32, the clarity of the "(b) camera-captured image" received from the autonomous vehicle 10 is highest, i.e., clearest, at the left end (opacity of the 3DCG image = 0.25), and lowest, i.e., unclear, at the right end (opacity of the 3DCG image = 0.75).

[0407] Of the three synthesis processing examples shown in Figure 32, the synthesis processing example at the far right (opacity of the 3DCG image = 0.75) has the highest synthesis ratio of the 3DCG image in the synthesized image, and the synthesis processing example at the far left (opacity of the 3DCG image = 0.25) has the lowest synthesis ratio of the 3DCG image in the synthesized image.

[0408] In this way, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 controls the opacity value of the 3DCG image according to the clarity of the camera-captured image (b) received from the autonomously driving vehicle 10, and performs image synthesis processing.

[0409] An example of the opacity control process of a 3DCG image executed by the display control unit 122 of the data processing unit 42 of the remote support device 40 will be described with reference to FIG.

[0410] The example shown in Figure 33 is a processing example in which the opacity value of a 3DCG image is determined according to the transmission bit rate assigned to each camera-captured image received from the autonomously driven vehicle 10.

[0411] Fig. 33 shows the following five opacity switching processing examples: 3DCG image opacity = 0 (= completely transparent (only the image captured by the camera is displayed)) 3DCG image opacity = 0.25, 3DCG image opacity = 0.5, 3DCG image opacity = 0.75, 3DCG image opacity = 1 (= completely opaque (only the 3DCG image is displayed))

[0412] The example shown in FIG. 33 is a transmission bit rate (b(j)) assigned to one camera-captured image (image with image priority j (image priority index = j)) received from the autonomously driven vehicle 10. ａｌｌｏｃａｔｅｄ ) is a processing example in which the opacity value of a 3DCG image is determined according to the

[0413] Specifically, the transmission bit rate (b(j)) allocated to the camera-captured image received from the autonomously driven vehicle 10 is ａｌｌｏｃａｔｅｄ The higher the value of b(j), the lower the opacity of the 3DCG image, and the higher the output ratio of the camera-captured image in the composite image displayed on the display unit 43 of the remote support device 40. On the other hand, the transmission bit rate (b(j)) allocated to the camera-captured image is ａｌｌｏｃａｔｅｄ ) is lower, the opacity of the 3DCG image is increased, thereby increasing the output ratio of the 3DCG image of the composite image to be displayed.

[0414] When receiving camera-captured images (image priority index=j) from the autonomously driven vehicle 10, the data processing unit 42 of the remote assistance device 40 also calculates the transmission bit rate (b(j)) assigned to each camera-captured image. ａｌｌｏｃａｔｅｄ) is also received as metadata or attribute information attached to the image, and this information is used to determine the opacity value (0 to 1) of the 3DCG image to be composited with the received camera-captured image (image priority index = j).

[0415] 33 shows 3DCG images with five different opacities. That is, the opacity of the 3DCG images is set to 0, 0.25, 0.5, 0.75, and 1.

[0416] On the right side of the five 3DCG images shown in FIG. 33, the transmission bit rate (b(j)) assigned to the image is displayed. ａｌｌｏｃａｔｅｄ ) is shown. The lower end of this transmission bit rate axis is 0, i.e., b(j) ａｌｌｏｃａｔｅｄ =0.

[0417] Furthermore, a threshold value for switching the opacity of the 3DCG image is indicated at each boundary of the five 3DCG images. The data processing unit 42 of the remote assistance device 40 calculates the transmission bit rate (b(j)) assigned to each camera-captured image received from the autonomously driven vehicle 10. ａｌｌｏｃａｔｅｄ ) with each threshold value to determine the opacity value of the 3DCG image to be superimposed on the image captured by the camera, and then perform image synthesis processing to display the result on the display unit 43.

[0418] The example of threshold setting shown in FIG. 33 is just an example, and a configuration may be adopted in which more threshold values ​​are set to perform more detailed control of the opacity of the 3DCG image.

[0419] The lower end of the transmission bit rate axis shown on the right side of the 3DCG image, i.e., b(j) ａｌｌｏｃａｔｅｄ =0 This setting indicates that the value of the transmission bit rate is 0. This corresponds to the fact that no camera-captured images are being transmitted from the autonomously driven vehicle 10, i.e., that the images are not being transmitted.

[0420] b(j) ａｌｌｏｃａｔｅｄ= 0, i.e., if the value of the transmission bit rate is 0, the remote assistance device 40 cannot receive camera-captured images from the autonomously driving vehicle 10, so the data processing unit 42 of the remote assistance device 40 sets the opacity of the 3DCG image to 1 and performs processing to display only the 3DCG image.

[0421] The data processing unit 42 of the remote support device 40 sets the opacity of the 3DCG image to 1 and performs processing to display only the 3DCG image not only when it is unable to receive the camera-captured image, but also when it is able to receive the camera-captured image but the clarity of the received image is significantly low or there is a large delay. As a judgment index for this, the transmission bit rate (b(j)) allocated to the image is used. ａｌｌｏｃａｔｅｄ ) is used.

[0422] In the example shown in FIG. 33, the transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ ) is in the following range: 0≦b(j) ａｌｌｏｃａｔｅｄ < b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ If the above conditions are met, the data processing unit 42 of the remote support device 40 sets the opacity of the 3DCG image to 1 and performs processing to display only the 3DCG image.

[0423] Note that b(j) ｒｅｑｕｉｒｅｄ is the transmission bit rate required to transmit an image with image priority j with a delay not greater than a predetermined value, and b ｂｕｆｆｅｒ is the bit rate required to prevent image freezing when switching between an image using a camera-captured image and a 3DCG image not using a camera-captured image, and is the bit rate required to ensure time for buffering the received image.

[0424] Also, the transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ ) is in the range: ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ ≦b(j) ａｌｌｏｃａｔｅｄ <(2b(j) ｏｖｅｒｌａｙｅｄ +(b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ)) / 3 If the above conditions are met, the data processing unit 42 of the remote support device 40 performs processing to generate and display a composite image in which a 3DCG image with an opacity (Opacity) of 0.75 is superimposed on an image captured by a camera with image priority j.

[0425] Note that b(j) ｏｖｅｒｌａｙｅｄ is the minimum bit rate required to use an image captured by a camera with image priority j as a composite image to be output to the display unit of the remote support device 40, and is a predetermined value.

[0426] Also, the transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ ) is in the range: (2b(j) ｏｖｅｒｌａｙｅｄ +(b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ )) / 3≦b(j) ａｌｌｏｃａｔｅｄ <(b(j) ｏｖｅｒｌａｙｅｄ +2(b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ )) / 3 If the above conditions are met, the data processing unit 42 of the remote support device 40 performs processing to generate and display a composite image in which a 3DCG image with an opacity (Opacity) of 0.5 is superimposed on an image captured by a camera with image priority j.

[0427] Also, the transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ ) is in the following range, i.e., (b(j) ｏｖｅｒｌａｙｅｄ +2(b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ )) / 3≦b(j) ａｌｌｏｃａｔｅｄ < b(j) ｏｖｅｒｌａｙｅｄ If the above conditions are met, the data processing unit 42 of the remote support device 40 performs processing to generate and display a composite image in which a 3DCG image with an opacity of 0.25 is superimposed on an image captured by a camera with image priority j.

[0428] Furthermore, the transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ ) is in the following range, i.e., b(j) ｏｖｅｒｌａｙｅｄ ≦b(j) ａｌｌｏｃａｔｅｄIf the above conditions are met, the data processing unit 42 of the remote support device 40 sets the opacity of the 3DCG image to 0, and performs processing to display the image captured by the camera with image priority j as is.

[0429] In this way, the data processing unit 42 of the remote assistance device 40 calculates the transmission bit rate (b(j)) allocated to the camera-captured image received from the autonomously driven vehicle 10. ａｌｌｏｃａｔｅｄ The opacity value of the 3DCG image is determined according to the value of ( ), and a composite image is generated by superimposing the 3DCG image having the determined opacity on the camera image received from the autonomously driven vehicle 10, and the composite image is output to the display unit 43.

[0430] The transmission bit rate (b(j)) allocated to the camera-captured image received from the autonomously driven vehicle 10 ａｌｌｏｃａｔｅｄ The higher the value of b(j), the lower the opacity of the 3DCG image, the higher the output ratio of the camera-captured image to the image displayed on the display unit 43, and the higher the transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ ) is lower, the opacity of the 3DCG image is increased, and the output ratio of the 3DCG image of the displayed image is controlled to be higher.

[0431] The opacity control sequence of the 3DCG image executed by the data processing unit 42 of the remote support device 40 described with reference to FIG. 33 will be described with reference to the flowcharts shown in FIGS.

[0432] The definitions of the parameters shown in the flowcharts of Figures 34 and 35 are as follows, as shown in the table of Figure 36: j: index of the list of images sent by the autonomous vehicle sorted in descending order of priority b(j) ａｌｌｏｃａｔｅｄ : transmission bit rate b(j) allocated to transmission of the image with the jth image priority ｒｅｑｕｉｒｅｄ : Transmission bit rate b(j) required to transmit the image of the jth image priority with a delay of less than a predetermined value ｏｖｅｒｌａｙｅｄ b: The minimum bit rate (predetermined value) required to use the image with the jth image priority as a composite image ｂｕｆｆｅｒB: The bit rate required to prevent image freezing when switching between an image using a camera image and a 3DCG image not using a camera image (= the bit rate required to ensure time for buffering the received image) (predetermined value) ａｌｌｏｃａｔｅｄ : Transmission bit rate of each image (M images) transmitted from the autonomous vehicle (B ａｌｌｏｃａｔｅｄ = {b(1) ａｌｌｏｃａｔｅｄ , b(2) ａｌｌｏｃａｔｅｄ , ..., b(M) ａｌｌｏｃａｔｅｄ ) O ｔａｒｇｅｔ : Target value of 3DCG image opacity to be applied to synthetic image generation O ｃｕｒｒｅｎｔ : Current setting of the 3DCG image opacity applied to the composite image generation

[0433] The following describes the processing of each step shown in the flowchart in Fig. 34. The flowchart in Fig. 34 shows processing that is executed by the remote assistance device 40 each time it receives images captured by a plurality of cameras from the autonomously driven vehicle 10, for example, images corresponding to the six cameras described above with reference to Figs.

[0434] Each received image has an image priority index=j and a transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ The remote support device 40 receives this information together with the image as metadata or attribute information attached to the image, and uses this information to execute processing according to the flowchart shown in FIG.

[0435] (Step S501) First, in step S501, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 determines the transmission bit rate (B ａｌｌｏｃａｔｅｄ )

[0436] Note that M images are, for example, images corresponding to the six cameras described with reference to Figures 3 and 4, and in the example using six cameras described with reference to Figures 3 and 4, M = 6.

[0437] As mentioned above, each received image is assigned a image priority index=j and a transmission bit rate (b(j) ａｌｌｏｃａｔｅｄ ) are assigned to the image, and the remote support device 40 receives this information together with the image as metadata or attribute information attached to the image.

[0438] In step S501, the data processing unit 42 of the remote support device 40 calculates the transmission bit rate (B ａｌｌｏｃａｔｅｄ ), i.e., B ａｌｌｏｃａｔｅｄ = {b(1) ａｌｌｏｃａｔｅｄ , b(2) ａｌｌｏｃａｔｅｄ , ..., b(M) ａｌｌｏｃａｔｅｄ}) The transmission bit rate (B ａｌｌｏｃａｔｅｄ ) information is input via the communication unit 41.

[0439] (Step S502) Step S502 is an initial setting process for the parameter j. In step S502, the data processing unit 42 of the remote support device 40 sets the value of the image priority index j to an initial value j=1. This means that the image with the highest image priority is selected and processed first.

[0440] (Step S503) Next, in step S503, the data processing unit 42 of the remote support device 40 determines whether the value of the image priority index j is greater than the total number of images M, that is, whether the above formula j>M holds. If the above formula holds, processing of the M images to be processed has been completed, and the processing according to this flow ends. On the other hand, if the above formula does not hold, that is, if j≦M, the processing from step S504 onwards is performed for the image with priority index j (≦M) to be processed.

[0441] (Step S504) Next, in step S504, the data processing unit 42 of the remote support device 40 determines whether the following formula is true for the image with priority index j (≦M) to be processed: b(j) ａｌｌｏｃａｔｅｄ ≧b(j) ｏｖｅｒｌａｙｅｄ It is determined whether the above determination formula is satisfied.

[0442] The above determination formula is the transmission bit rate (b(j)) assigned to the image with image priority index j. ａｌｌｏｃａｔｅｄ ) is b(j) ｏｖｅｒｌａｙｅｄ This is an equation for determining whether b(j) is equal to or greater than b(j). ｏｖｅｒｌａｙｅｄ As described above, σj is the minimum bit rate required to use the camera image with image priority j as a composite image to be output to the display unit of the remote support device 40, and is a predetermined value. If the above determination formula is satisfied, the process proceeds to step S505. On the other hand, if the above determination formula is not satisfied, the process proceeds to step S506.

[0443] (Step S505) In step S504, b(j) ａｌｌｏｃａｔｅｄ ≧b(j) ｏｖｅｒｌａｙｅｄ If it is determined that the above judgment formula is satisfied, the data processing unit 42 of the remote support device 40 sets the opacity of the 3DCG image to 0 (=completely transparent) in step S505, and performs processing to display the image captured by the camera with image priority j as is on the display unit 43.

[0444] This process corresponds to an example in which the topmost opacity (Opacity) of the 3DCG images with five different opacities (Opacity) is set to 0 (completely transparent) as described with reference to Fig. 33. In other words, this is a process in which the camera-captured image is output as is to the display unit 43 without using the 3DCG image.

[0445] This process is performed by using the transmission bit rate (b(j)) assigned to the camera-captured image with image priority j received from the autonomously driven vehicle 10. ａｌｌｏｃａｔｅｄ ) is a sufficiently high bit rate and is a process based on the judgment that it is possible to output a clear image captured by a camera.

[0446] After the processing of step S505, the process proceeds to step S510, where the value of the image priority index j is updated (j = j + 1), and the processing from step S503 onwards is performed for the image having the updated value of the image priority index j.

[0447] (Step S506) On the other hand, in step S504, b(j)ａｌｌｏｃａｔｅｄ ≧b(j) ｏｖｅｒｌａｙｅｄ If it is determined that the above determination formula does not hold, the data processing unit 42 of the remote support device 40 determines in step S506 whether the following formula holds: b(j) ａｌｌｏｃａｔｅｄ ≧b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ It is determined whether the above determination formula is satisfied.

[0448] The above determination formula is the transmission bit rate (b(j)) assigned to the image with image priority index j. ａｌｌｏｃａｔｅｄ ) is (b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ ) or greater.

[0449] As previously described with reference to FIG. 33, b(j) ｒｅｑｕｉｒｅｄ is the transmission bit rate required to transmit an image with image priority j with a delay not greater than a predetermined value, ｂｕｆｆｅｒ is the bit rate required to prevent image freezing when switching between an image using a camera-captured image and a 3DCG image not using a camera-captured image, and is the bit rate required to ensure time for buffering the received image.

[0450] In step S506, b(j) ａｌｌｏｃａｔｅｄ ≧b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ If it is determined that the above-mentioned determination formula does not hold, the process proceeds to step S507. On the other hand, if it is determined that the determination formula holds, the process proceeds to step S509.

[0451] (Step S507) In step S506, b(j) ａｌｌｏｃａｔｅｄ ≧b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ If it is determined that the above determination formula does not hold, the process proceeds to step S507.

[0452] Step S507 is to calculate the transmission bit rate (b(j)) assigned to the image with image priority index j. ａｌｌｏｃａｔｅｄ ) is (b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ ) is determined to be less than the predetermined value.

[0453] In this case, the data processing unit 42 of the remote support device 40 sets the opacity of the 3DCG image as follows: Opacity = 1 (completely opaque)

[0454] This corresponds to the use of a 3DCG image with the lowest opacity (Opacity) set to 1 (completely opaque) among the five 3DCG images with different opacities (Opacity) described above with reference to Fig. 33. In other words, the image captured by the camera is not used, and only the 3DCG image is output to the display unit 43.

[0455] This process is performed by using the transmission bit rate (b(j)) assigned to the camera-captured image with image priority j received from the autonomously driven vehicle 10. ａｌｌｏｃａｔｅｄ ) is 0 or has an extremely low bit rate, making it impossible to output a clear camera-captured image.

[0456] (Step S508) After the process of outputting only the 3DCG image to the display unit 43 in step S507, the data processing unit 42 of the remote assistance device 40 executes a process of rendering an object on the 3DCG image displayed on the display unit 43 in step S508.

[0457] As explained above, the 3DCG images generated from the 3D maps stored in the 3D map storage unit 45 do not include real-time surrounding objects such as oncoming vehicles and pedestrians that exist around the autonomous vehicle 10.

[0458] For this reason, the data processing unit 42 of the remote assistance device 40 analyzes metadata including information on surrounding objects such as oncoming vehicles received from the automatic driving control device 20, and performs processing such as displaying images of objects around the automatically driven vehicle 10 on a 3DCG image. By performing such processing, display data that more accurately reproduces the current surrounding situation of the automatically driven vehicle 10 is generated and displayed on the display unit. Details of this processing will be described later.

[0459] After the processing of step S508, the process proceeds to step S510, where the value of the image priority index j is updated (j = j + 1), and the processing from step S503 onwards is performed for the image having the updated value of the image priority index j.

[0460] (Step S509) On the other hand, in step S506, b(j) ａｌｌｏｃａｔｅｄ ≧b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ If it is determined that the above determination formula is satisfied, the process proceeds to step S509.

[0461] Step S509 is a step of determining whether the determination formula in step S504, b(j), ａｌｌｏｃａｔｅｄ ≧b(j) ｏｖｅｒｌａｙｅｄ If it is determined that the above determination formula does not hold, and further, the determination formula in step S506, b(j) ａｌｌｏｃａｔｅｄ ≧b(j) ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ This is a process that is executed when it is determined that the above determination formula is satisfied.

[0462] That is, this step S509 determines the transmission bit rate (b(j)) assigned to the image with image priority index j. ａｌｌｏｃａｔｅｄ ) is in the following range of values: ｒｅｑｕｉｒｅｄ +b ｂｕｆｆｅｒ ) ≦ b(j) ａｌｌｏｃａｔｅｄ < b(j) ｏｖｅｒｌａｙｅｄ

[0463] This corresponds to the processing of using the three images in the middle row of the 3DCG images with five different opacities (Opacity) described above with reference to Figure 33, i.e., 3DCG images with an opacity setting of 0.25 to 0.75.

[0464] In this case, the data processing unit 42 of the remote support device 40 calculates the transmission bit rate (b(j)) assigned to the image with the image priority index j as described above with reference to FIG. ａｌｌｏｃａｔｅｄ33 and calculates the opacity (=0 to 1) of the 3DCG image. Furthermore, the CPU 101 executes a process of superimposing the 3DCG image having the calculated opacity on the image captured by the camera to generate a composite image and output the composite image to the display unit.

[0465] After the processing of step S509, the process proceeds to step S510, where the value of the image priority index j is updated (j = j + 1), and the processing from step S503 onwards is performed for the image having the updated value of the image priority index j.

[0466] Next, referring to the flowchart shown in Figure 35, we will explain the detailed sequence of the process control for changing the opacity of a 3DCG image when updating the image frame of a composite image displayed on the display unit 43 of the remote support device 40, i.e., a composite image of a camera-captured image and a 3DCG image.

[0467] The composite image of the camera-captured image and the 3DCG image displayed on the display unit 43 of the remote support device 40 is a moving image, and the display control unit 122 of the data processing unit 42 of the remote support device 40 calculates the opacity of the 3DCG image for each image frame according to the processing sequence shown in Figure 34.

[0468] However, for example, if the opacity of a 3DCG image changes significantly between consecutive image frames, generating a composite image using the calculated opacity as is and outputting it to the display unit 43 will result in a sudden change in the displayed image, making it difficult to observe.

[0469] The flowchart shown in Figure 35 explains a control sequence for preventing such a situation. Specifically, this flow explains control for gradually changing the opacity value of a 3DCG image to output a composite image that is easy to observe. The processing of each step in the flow shown in Figure 35 will be explained in order below.

[0470] (Step S521) First, in step S521, the data processing unit 42 of the remote support device 40 calculates a target value O of the opacity of the 3DCG image. ｔａｒｇｅｔ Get.

[0471] Target value O of the opacity of the 3DCG image ｔａｒｇｅｔ is calculated according to the flow shown in FIG. 34 described above.

[0472] (Steps S522 to S523) Next, in step S522, the data processing unit 42 of the remote support device 40 checks whether or not a 3DCG image opacity change animation is in progress for the composite image being output to the display unit 43.

[0473] The 3DCG image opacity change animation is an animation that gradually changes the opacity of a 3DCG image, executed according to the flow shown in Fig. 35. If an animation that gradually changes the opacity of a 3DCG image is already running, the animation is stopped in step S523 to prevent double processing, and then the process proceeds to step S524.

[0474] Note that animation stabilization processing may be performed in parallel with the processing of step S522 and subsequent steps. The animation stabilization processing is performed by adjusting the target value O of the opacity of the 3DCG image. ｔａｒｇｅｔ This is a process for preventing display flicker when the value of fluctuates near the threshold value. As a specific example, it is effective to calculate a moving average taking into account the time transition of the calculated opacity and apply the moving average result, or to set the threshold value shown in Fig. 33 to vary by providing hysteresis rather than setting it to a fixed value.

[0475] (Step S524) Next, in step S524, the data processing unit 42 of the remote support device 40 calculates the target value O of the opacity of the 3DCG image. ｔａｒｇｅｔ and the opacity O of the current 3DCG image in the composite image currently being output to the display unit 43. ｃｕｒｒｅｎｔ It is determined whether the difference between |O ｔａｒｇｅｔ -O ｃｕｒｒｅｎｔ|<ε Determine whether the above formula is true.

[0476] If it is determined that the above formula is true, the process proceeds to step S525. On the other hand, if it is determined that the above formula is false, the process proceeds to step S526.

[0477] (Step S525) In step S524, |O ｔａｒｇｅｔ -O ｃｕｒｒｅｎｔ If it is determined that the above formula is true, the data processing unit 42 of the remote support device 40 sets the opacity of the 3DCG image to the target value O ｔａｒｇｅｔ , and generates a composite image, which is output to the display unit 43.

[0478] In this case, the change in opacity of the 3DCG image is small (less than the threshold value ε), the change in the output image is small, and no flickering is perceived.

[0479] (Step S526) In step S524, |O ｔａｒｇｅｔ -O ｃｕｒｒｅｎｔ If it is determined that the above formula is not true, the data processing unit 42 of the remote support device 40 calculates the time parameter T in accordance with the following formula in step S526: T=T ｆｕｌｌ × (O ｔａｒｇｅｔ -O ｃｕｒｒｅｎｔ )

[0480] In addition, T ｆｕｌｌ is a time that is predetermined as the time required to make a natural transition when the opacity of a 3DCG image is transitioned from 0 to 1. The time parameter T calculated according to the above formula is the time required to make a natural transition when the opacity of a 3DCG image is transitioned from 0 to 1. ｃｕｒｒｅｎｔ Target value O ｔａｒｇｅｔ This corresponds to the transition time required to transition smoothly from

[0481] (Step S527) Next, the data processing unit 42 of the remote support apparatus 40 acquires the current time t in step S527.

[0482] (Step S528) Next, in step S528, the data processing unit 42 of the remote support device 40 calculates the 3DCG image opacity change animation start time (t ｓｔａｒｔ ) to the current time t. Δt = t - t ｓｔａｒｔ

[0483] (Step S529) Next, in step S529, the data processing unit 42 of the remote support device 40 calculates a new 3DCG image opacity O to be set at the current time t according to the following calculation formula: ｎｅｗ Calculate O ｎｅｗ =O ｓｔｒａｔ + (O ｔａｒｇｅｔ -O ｓｔａｒｔ ) × (Δt / T)

[0484] The new 3DCG image opacity O calculated according to the above formula ｎｅｗ is set to the 3DCG image opacity at the current time t, the 3DCG image opacity change animation start time (t ｓｔａｒｔ ) is the opacity that allows the opacity change from the original to the original, making it possible to output a natural composite image.

[0485] (Step S530) Next, in step S530, the data processing unit 42 of the remote support device 40 calculates the target value O of the opacity of the 3DCG image. ｔａｒｇｅｔ and the new 3DCG image opacity O calculated in step S529. ｎｅｗ It is determined whether the difference between |O ｔａｒｇｅｔ -O ｎｅｗｔ |<ε Determine whether the above formula is true.

[0486] If it is determined that the above formula is true, the process proceeds to step S531. On the other hand, if it is determined that the above formula is false, the process proceeds to step S532.

[0487] (Step S531) In step S530, |O ｔａｒｇｅｔ -O ｎｅｗ If it is determined that the above formula is true, the data processing unit 42 of the remote support device 40 sets the opacity of the 3DCG image to the target value Oｎｅｗ , and generates a composite image, which is output to the display unit 43.

[0488] In this case, the change in opacity of the 3DCG image is small (less than the threshold value ε), the change in the output image is small, and no flickering is perceived.

[0489] (Step S532) On the other hand, in step S530, |O ｔａｒｇｅｔ -O ｎｅｗ |<ε If it is determined that the above formula does not hold, the data processing unit 42 of the remote assistance device 40 waits in step S532 until the next image frame is received from the autonomously driving vehicle 10, and after waiting, returns to step S527 and executes the processing from step S527 onwards as opacity control processing of the 3DCG image for the newly received image frame.

[0490] By performing the processing according to the flow shown in FIG. 35, the change in the opacity value of the 3DCG image becomes gentler, making it possible to output a composite image that is easy to observe.

[0491] 9. Details of Object Display Control Processing Executed by Remote Support Device Next, details of object display control processing executed by the data processing unit 42 of the remote support device 40 will be described.

[0492] As can be understood from the above explanation, the data processing unit 42 of the remote assistance device 40 displays on the display unit 43 either a camera-captured image received from the autonomously driving vehicle 10, a composite image of a camera-captured image and a 3DCG image, or a 3DCG image.

[0493] When using a 3DCG image as a display image, the data processing unit 42 of the remote assistance device 40 uses the 3D map stored in the 3D map storage unit 45 to generate and display a 3DCG image observed from the position and direction of the camera attached to the autonomously driving vehicle 10.

[0494] However, the 3DCG images generated from the 3D maps stored in the 3D map storage unit 45 do not include real-time surrounding objects such as oncoming vehicles and pedestrians that exist around the autonomous vehicle 10.

[0495] For this reason, the data processing unit 42 of the remote assistance device 40 analyzes metadata including information on surrounding objects such as oncoming vehicles received from the automatic driving control device 20, and performs processing such as superimposing and displaying images of objects around the automatically driven vehicle 10 on a 3DCG image. By performing such processing, display data that more accurately reproduces the current surrounding situation of the automatically driven vehicle 10 can be generated and displayed on the display unit.

[0496] The following describes in detail the object display control process executed by the data processing unit 42 of the remote support device 40. Note that the object display control process described below is a process executed by the display control unit 122 of the data processing unit 42 of the remote support device 40 shown in FIG.

[0497] As previously described with reference to Figures 23 to 29, the automatic driving control device 20 generates metadata consisting of information about surrounding objects such as oncoming vehicles contained in the non-transmitted image and transmits it to the remote assistance device 40.

[0498] The data processing unit 22 of the autonomous driving control device 20 always transmits, for example, object coordinates, bounding box size, and label used to identify the position and size of objects (objects such as vehicles and pedestrians surrounding the autonomous driving vehicle 10). However, for large-volume data such as object images, if the bandwidth of the communication path that can be secured for metadata communication is narrow, the data processing unit 22 may stop transmission, transmit the data as compressed data, or perform filtering to select and transmit data based on the risk level of each object. The risk level of the object is determined by the data processing unit 22 of the autonomous driving control device 20, and this determination result is also transmitted to the remote assistance device 40 as metadata.

[0499] The display control unit 122 of the data processing unit 42 of the remote assistance device 40 extracts object image data, compressed image data of the object, etc. from the metadata transmitted by the data processing unit 22 of the automatic driving control device 20, and generates and outputs an object image to be output to the display unit 43 of the remote assistance device 40.

[0500] The display control unit 122 of the data processing unit 42 of the remote support device 40 executes a process of generating different display data depending on the risk level of the object.

[0501] Referring to FIG. 37, the object display process according to the risk level of the object, which is executed by the display control unit 122 of the data processing unit 42 of the remote assistance device 40, will be described.

[0502] 37 shows an example of object display processing according to the risk level of an object, which is executed by the display control unit 122 of the data processing unit 42 of the remote support device 40. Note that there are four object risk levels, 1 to 4, with object risk level 1 being the highest risk level and object risk level 4 being the lowest risk level.

[0503] 37, the data processing unit 42 of the remote assistance device 40 executes the following object display processing depending on the risk level of the object. For objects with object risk level = 1, 3D object image display processing is executed using uncompressed object image data, i.e., uncompressed object image data. For objects with object risk level = 2, 3D object image display processing is executed using compressed object image data with a low compression rate. For objects with object risk level = 3, 3D object image display processing is executed using compressed object image data with a high compression rate. For objects with object risk level = 4, display processing is executed to display a bounding box surrounding the object area and a label indicating the object type.

[0504] As previously explained with reference to Figures 23 to 29, the automatic driving control device 20 is configured to transmit object image data for surrounding objects such as oncoming vehicles at different compression rates according to the risk level, and the data processing unit 42 of the remote assistance device 40 uses the data received from the automatic driving control device 20 to perform object display processing according to the object risk level as shown in Figure 37.

[0505] 38 shows a specific example of object display processing according to the risk level of an object, which is executed by the display control unit 122 of the data processing unit 42 of the remote support device 40. Fig. 38 shows specific object image display commands according to object risk levels 1 to 4 described with reference to Fig. 37.

[0506] 38(a) is an example of an object display when the object risk level is 1, and is an example of a 3D object image display using uncompressed object image data, i.e., uncompressed object image data. (b) is an example of an object display when the object risk level is 2, and is an example of a 3D object image display using compressed object image data with a low compression rate. (c) is an example of an object display when the object risk level is 3, and is an example of a 3D object image display using compressed object image data with a high compression rate. (d) is an example of an object display when the object risk level is 4, and is an example of a display of a bounding box surrounding the object area and a label indicating the object type.

[0507] In this way, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 executes a process of generating different display data depending on the risk level of the object.

[0508] Next, object image interpolation processing and object image extrapolation processing, which are interpolation processing of object images executed by the display control unit 122 of the data processing unit 42 of the remote support device 40, will be described.

[0509] 23 to 29, the autonomous driving control device 20 generates metadata including information on surrounding objects such as oncoming vehicles and transmits it to the remote assistance device 40. However, for large-volume data such as object images, if the bandwidth of the communication path that can be secured for metadata communication is narrow, transmission may be stopped. Also, images of objects with a low risk level may be filtered out to exclude them from transmission.

[0510] In this way, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs a process of interpolating the object image that could not be received from the automatic driving control device 20 using the object images corresponding to the image frames before and after it.

[0511] The interpolation process executed by the display control unit 122 of the data processing unit 42 of the remote support device 40 has the following two processing modes: (1) An image frame (time t ｎ ) is compared with the object image in the previous image frame (time t ｎ－１ (2) An image frame (time t) in which an object image cannot be received. ｎ ) is compared with the object image in the previous image frame (time t ｎ－１ previous image frame) and the subsequent image frame (time t ｎ＋１ Interior interpolation process that estimates and generates the object image from the object image of the next image frame

[0512] Specific examples of the extrapolation process and the interpolation process will be described with reference to FIGS.

[0513] 39 is a diagram illustrating a specific example of the extrapolation process executed by the display control unit 122 of the data processing unit 42 of the remote support device 40. When an image frame for which an object image cannot be received is reached at time t ｎ The previous image frame (time t ｎ－１ and time t ｎ－２ The object image (image frame) is received.

[0514] The display control unit 122 of the data processing unit 42 of the remote support device 40 determines whether an image frame in which an object image cannot be received is within a time t ｎ The object image of ｎ－１ and time t ｎ－２ The display control unit 122 of the data processing unit 42 of the remote support device 40 estimates and generates the object image from the object image of the image frame at time t ｎThe position and size of the object to be displayed on the image frame of ｎ－１ and time t ｎ－２ (image frame) and estimate the object image at time t ｎ Executes the process of drawing on the image frame.

[0515] Fig. 40 is a diagram illustrating a specific example of the interpolation process executed by the display control unit 122 of the data processing unit 42 of the remote support device 40. The interpolation process is a process aimed at improving the accuracy of the extrapolation process described with reference to Fig. 39. In other words, the interpolation process is a process for improving the accuracy of the extrapolation image by further correcting an interpolated image generated by a previously executed extrapolation process using object images received before and after that time. As shown in the upper part of Fig. 40, at time t ｎ Assume that the image of the image frame of time t is an interpolated image generated by the extrapolation process described with reference to FIG. ｎ－１ previous image frame) and the subsequent image frame (time t ｎ＋１ The object images in the following image frames have been received.

[0516] The display control unit 122 of the data processing unit 42 of the remote support device 40 ｎ For the extrapolated image of the preceding image frame (time t ｎ－１ the object image in the image frame at time t ｎ＋１ Further correction (interpolation processing) is performed using the object image of the image frame (i.e., the image frame of the original image). By performing this correction (interpolation processing), the accuracy of the extrapolated image can be improved.

[0517] The display control unit 122 of the data processing unit 42 of the remote support device 40 may, for example, ｎ The position and size of the extrapolated image are calculated based on the previous image frame (time t ｎ－１ the object image in the image frame at time t ｎ＋１ The accuracy of the extrapolated image is improved by estimating it from the object image of the previous image frame and then correcting it.

[0518] In this way, the display control unit 122 of the data processing unit 42 of the remote assistance device 40 performs a correction process again for the object image estimated and generated by the extrapolation process, using the object images corresponding to the image frames before and after it.

[0519] Through these processes, the operator at the remote support center 30 can reliably identify objects such as oncoming vehicles from the displayed image, and can generate appropriate control information to prevent the autonomous vehicle 10 from colliding with the object and send it to the autonomous driving control device 20.

[0520] [10. Other Examples] Next, other examples will be described.

[0521] In the above-described embodiment, the autonomous driving control device 20 of the autonomous driving vehicle 10 is configured to transmit, for example, transmission video tracks 1 to n for each of the n images corresponding to the multiple (n) cameras previously described with reference to Figures 3 and 4, using a single stream set up between the remote support device 40 of the remote support center 30, as described with reference to Figure 8.

[0522] That is, multiple images, for example, six images corresponding to six cameras, are transmitted using multi-track video streaming, which transmits multiple videos in one stream. The bit rate of the multiple images being transmitted is changed as needed according to the available bandwidth. For example, a larger communication bandwidth (bit rate) is allocated to higher-priority images according to image priority.

[0523] Instead of a multi-track video streaming configuration in which multiple videos are transmitted in a single stream, it is also possible to adopt a configuration in which n images corresponding to multiple (n) cameras are transmitted together in a single video track, as shown in FIG. 41.

[0524] However, in the configuration shown in Fig. 41, it is not possible to control the transmission bit rate for each of n images corresponding to multiple (n) cameras, so processing is performed to uniformly adjust the transmission bit rate for all n images. In other words, the configuration is such that control is performed to change the transmission bit rate of a single video track that transmits n images collectively in accordance with the available bandwidth.

[0525] Furthermore, in the above-described embodiment, the autonomous driving control device 20 of one autonomous vehicle 10 is described as being configured to transmit camera-captured images to one remote assistance device 40, but a configuration such as that shown in Figure 42 is also possible.

[0526] 42 shows a configuration in which an autonomous driving control device 20 of one autonomous driving vehicle 10 transmits camera-captured images to multiple remote support devices 40 a to 40 c. The remote support devices 40 a to 40 c display images controlled according to the above-described embodiment, i.e., composite images of camera-captured images and 3DCG images.

[0527] The images displayed on the display units of the remote assistance devices 40a-c are checked by multiple operators a-c, who then detect obstacles and other obstacles, generate control information, and transmit it to the autonomously driven vehicle 10. This configuration reduces the risk of overlooking obstacles and improves the accuracy of image checking, enabling safer control.

[0528] A configuration such as that shown in Fig. 43 is also possible. In the configuration shown in Fig. 43, multiple autonomously driving vehicles a to c each transmit images captured by a camera mounted on each vehicle to the same remote support device 40. An operator 50 on the remote support device 40 side checks the images transmitted from each vehicle and transmits control information to each vehicle. With such a configuration, it is possible to reduce the number of operators 50.

[0529] 11. Example Hardware Configuration of the Automatic Driving Control Device and the Remote Support Device Next, with reference to FIG. 44 , an example hardware configuration of the automatic driving control device 20 and the remote support device 40 according to the present disclosure will be described.

[0530] The CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to programs stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, it executes processes according to the sequences described in the above-mentioned embodiments. The RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0531] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various switches, a touch panel, a microphone, and a user input unit, and an output unit 307 consisting of a display, a speaker, etc.

[0532] In the case of the autonomous driving control device 20, sensor-acquired information (camera images, object detection information, object distance information, etc.) from various sensors 321 such as cameras and LiDAR is input to the input unit 306. In addition, in the case of the autonomous driving control device 20, the output unit 307 also outputs drive information to a drive unit 322 of the vehicle.

[0533] The CPU 301 receives commands and status data from an input unit 306, executes various processes, and outputs the results of the processes to, for example, an output unit 307. A storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores programs executed by the CPU 301 and various data. A communication unit 309 functions as a transmitter / receiver for data communication via a network such as the Internet or a local area network, and communicates with external devices. In addition to the CPU, the device may also include a GPU (Graphics Processing Unit) as a dedicated processing unit for image information input from a camera, etc.

[0534] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.

[0535] [12. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described above in detail with reference to specific examples. However, it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0536] The technology disclosed in this specification can be configured as follows: (1) An autonomous driving control device having a communication unit that transmits a plurality of camera images taken by a plurality of cameras mounted on a vehicle to a remote support device and receives autonomous driving control information from the remote support device, and a data processing unit that executes transmission control for each of the plurality of camera images, wherein the data processing unit executes a process of calculating an estimated bit rate equivalent to a bandwidth available for image transmission to the remote control device, a process of calculating an image priority for each of the plurality of camera images, and a transmission image selection process that selects the plurality of camera images in descending order of image priority and transmits selected images that are determined to be transmittable to the remote control device at or below the estimated bit rate and with a delay amount not greater than a predetermined value.

[0537] (2) The automatic driving control device described in (1) above, wherein the data processing unit, when the estimated bit rate is equal to or greater than a required bit rate at which it is determined that all of the images captured by the multiple cameras can be transmitted to the remote control device with a delay of a predetermined value or less, executes a transmission process for all of the images captured by the multiple cameras without executing the transmission image selection process; and when the estimated bit rate is less than a required bit rate at which it is determined that all of the images captured by the multiple cameras can be transmitted to the remote control device with a delay of a predetermined value or less, calculates a required bit rate for each combination of images selected from the images captured by the multiple cameras in descending order of image priority, and transmits the largest combination of images for which the calculated required bit rate for each combination of images is equal to or less than the estimated bit rate.

[0538] (3) The automatic driving control device described in (1) or (2), wherein the data processing unit allocates the transmission bit rate of each of the multiple camera-captured images to be transmitted to the remote control device in proportion to the resolution corresponding to the number of pixels of each of the multiple camera-captured images selected as the transmission target.

[0539] (4) The data processing unit calculates the image priority of each of the plurality of camera-captured images by taking into consideration at least one of (a) and (b) above: (a) the number of objects detected from each of the plurality of camera-captured images; and (b) the number of high-risk objects detected from each of the plurality of camera-captured images.

[0540] (5) The automatic driving control device according to any one of (1) to (4), wherein the data processing unit executes transmission bit rate control to set a target bit rate, which is a transmission bit rate when transmitting all of the images captured by the plurality of cameras, between a predetermined minimum bit rate and a predetermined maximum bit rate, and when the estimated bit rate is equal to or greater than the maximum bit rate, sets the target bit rate to the maximum bit rate.

[0541] (6) The automatic driving control device according to any one of (1) to (5), wherein the data processing unit transmits an object image of the object detected from the image captured by the camera to the remote assistance device.

[0542] (7) The automatic driving control device described in (6), wherein the data processing unit executes an object detection process from non-transmitted images that were excluded from the transmission target in the transmission image selection process, and transmits object images of objects detected from the non-transmitted images to the remote assistance device.

[0543] (8) The automatic driving control device described in (6) or (7), wherein the data processing unit executes a process of transmitting all object images of the detected object when an object image transmission bit rate, which is a bandwidth available for transmitting object images, is equal to or greater than a bit rate required for transmitting all object images to the remote control device with a delay amount equal to or less than a predetermined value, and generates and transmits a compressed object image by compressing at least a portion of the object image when the object image transmission bit rate is less than the bit rate required for transmitting all object images to the remote control device with a delay amount equal to or less than a predetermined value.

[0544] (9) The automatic driving control device according to any one of (6) to (8), wherein the data processing unit determines a risk level of a detected object detected from an image, and preferentially selects and transmits object images of objects with a high risk level.

[0545] (10) The data processing unit is configured to perform compression processing using different compression algorithms depending on the risk level of the detected object detected from the image, and generates and transmits a compressed object image using a compression algorithm that sets a lower compression rate for objects with a higher risk level and a higher compression rate for objects with a lower risk level.An automatic driving control device described in any of (6) to (9).

[0546] (11) The automatic driving control device according to any one of (6) to (10), wherein the data processing unit performs an image segmentation process on the image to perform an object detection process.

[0547] (12) The automatic driving control device according to any one of (1) to (11), wherein the communication unit transmits images using a multi-track video streaming method that transmits images captured by multiple cameras in a single stream.

[0548] (13) A remote assistance device having a communication unit that receives multiple camera images taken by multiple cameras mounted on a vehicle from an automatic driving control device, and a data processing unit that executes display control to display the multiple camera images on a display unit, wherein the data processing unit switches and displays an image to be displayed on the display unit in response to the image received from the automatic driving control device between: (a) a camera image received from the automatic driving control device, or (b) a composite image of a camera image received from the automatic driving control device and a 3DCG image, or (c) a 3DCG image, or

[0549] (14) The remote assistance device described in (13) in which the data processing unit sets the image to be displayed on the display unit to the camera image received from the automatic driving control device when the image received from the automatic driving control device has high clarity, and sets the image to be displayed on the display unit to a 3DCG image when no image is received from the automatic driving control device.

[0550] (15) The remote support device according to (13) or (14), wherein the data processing unit controls the opacity of the 3DCG image in a composite image of the camera-captured image and the 3DCG image, generates each of the images (a) to (c), and displays them on a display unit.

[0551] (16) The remote assistance device according to (15), wherein the data processing unit controls the opacity of the 3DCG image according to the value of the transmission bit rate of each of the plurality of camera-captured images determined by the automatic driving control device.

[0552] (17) The remote assistance device according to any one of (13) to (16), wherein the data processing unit displays an object image received from the automatic driving control device on a 3DCG image displayed on a display unit.

[0553] (18) The remote assistance device according to (17), wherein the data processing unit changes and displays a display mode of an object image depending on a risk level of the object.

[0554] (19) The remote assistance device described in any one of (13) to (18), wherein the data processing unit displays an object image generated by interpolation processing using object images of previous and next image frames for an image frame for which an object image could not be received from the automatic driving control device.

[0555] (20) An autonomous driving control system having an autonomous driving control device and a remote assistance device, wherein the autonomous driving control device performs the following processes: a process of calculating an estimated bit rate equivalent to the bandwidth available for transmitting images to the remote control device; a process of calculating an image priority for each of a plurality of camera images taken by a plurality of cameras mounted on a vehicle; and a transmission image selection process of selecting the plurality of camera images in descending order of image priority and transmitting selected images that are determined to be transmittable to the remote control device at or below the estimated bit rate and with an amount of delay not greater than a predetermined value; and the remote assistance device performs the following processes: a process of switching the image to be displayed on a display unit to one of the following images (a) to (c) in accordance with the image received from the autonomous driving control device, (a) a camera image received from the autonomous driving control device, or (b) a composite image of a camera image and a 3DCG image received from the autonomous driving control device, or (c) a 3DCG image; and transmits control information generated from the confirmation result of the displayed image to the autonomous driving control device.

[0556] (21) An image transmission control method executed in an automatic driving control device, wherein the automatic driving control device has a communication unit that transmits multiple camera images taken by multiple cameras mounted on a vehicle to a remote support device and receives automatic driving control information from the remote support device, and a data processing unit that executes transmission control for each of the multiple camera images, wherein the data processing unit executes a process of calculating an estimated bit rate equivalent to a bandwidth available for image transmission to the remote control device, a process of calculating an image priority for each of the multiple camera images, and a transmission image selection process that selects the multiple camera images in descending order of image priority and transmits selected images that are determined to be transmittable to the remote control device at or below the estimated bit rate and with a delay amount below a predetermined value.

[0557] (22) An image display control method executed in a remote assistance device, wherein the remote assistance device has a communication unit that receives multiple camera images taken by multiple cameras mounted on a vehicle from an automatic driving control device, and a data processing unit that executes display control to display the multiple camera images on a display unit, and the data processing unit switches and displays an image to be displayed on the display unit in response to the image received from the automatic driving control device to any of the images (a) to (c), such as: (a) a camera image received from the automatic driving control device, or (b) a composite image of a camera image received from the automatic driving control device and a 3DCG image, or (c) a 3DCG image.

[0558] (23) A program for causing an automatic driving control device to execute an image transmission control process, wherein the automatic driving control device has a communication unit that transmits multiple camera images taken by multiple cameras mounted on a vehicle to a remote support device and receives automatic driving control information from the remote support device, and a data processing unit that executes transmission control for each of the multiple camera images, and the program causes the data processing unit to execute a process of calculating an estimated bit rate equivalent to a bandwidth available for image transmission to the remote control device, a process of calculating an image priority for each of the multiple camera images, and a transmission image selection process that selects the multiple camera images in order of highest image priority and transmits selected images that are determined to be transmittable to the remote control device at or below the estimated bit rate and with a delay amount below a predetermined value.

[0559] (24) A program for causing a remote assistance device to execute an image display control process, the remote assistance device having a communication unit that receives multiple camera images taken by multiple cameras mounted on a vehicle from an automatic driving control device, and a data processing unit that executes display control to display the multiple camera images on a display unit, the program causing the data processing unit to switch and display an image to be displayed on the display unit in response to the image received from the automatic driving control device between: (a) a camera image received from the automatic driving control device, or (b) a composite image of a camera image received from the automatic driving control device and a 3DCG image, or (c) a 3DCG image, any of the above (a) to (c).

[0560] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program from the recording medium to the computer, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0561] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and although each component device may be located in the same housing, it is not limited to this.

[0562] As described above, according to one embodiment of the present disclosure, low-latency image transmission and low-latency remote control are realized by optimally controlling the bit rate of each image transmitted from the autonomous driving control device to the remote control device according to the bandwidth. Specifically, for example, the autonomous driving control device calculates an estimated bit rate corresponding to the bandwidth available for image transmission to the remote control device, calculates image priority for each of multiple camera-captured images, and transmits only high-priority images determined to be capable of low-latency transmission at or below the estimated bit rate. The remote assistance device switches the image displayed on the display unit between a camera-captured image, a composite image of a camera-captured image and a 3DCG image, or a 3DCG image, depending on the image received from the autonomous driving control device. This configuration realizes low-latency image transmission and low-latency remote control by optimally controlling the bit rate of each image transmitted from the autonomous driving control device to the remote control device according to the bandwidth.

[0563] 10 Vehicle 20 Automatic driving control device 21 Sensor 22 Data processing unit 23 Automatic driving control unit (ADS) 24 Communication unit 30 Remote assistance center 40 Remote assistance device 41 Communication unit 42 Data processing unit 43 Display unit 44 Input unit 45 3D map storage unit 50 Operator 111 Risk level classification unit 112 Communication network monitoring unit 113 Data analysis unit 114 Communication control unit 121 Map control unit 122 Display control unit 123 Communication control unit 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / output interface 306 Input unit 307 Output unit 308 Storage unit 309 Communication unit 310 Drive 311 Removable media 321 Sensor 322 Drive unit

Claims

1. An autonomous driving control device having a communication unit that transmits multiple camera images captured by multiple cameras mounted on a vehicle to a remote support device and receives autonomous driving control information from the remote support device, and a data processing unit that executes transmission control of each of the multiple camera images, wherein the data processing unit executes a process of calculating an estimated bit rate equivalent to a bandwidth available for image transmission to the remote control device, a process of calculating an image priority for each of the multiple camera images, and a transmission image selection process that selects the multiple camera images in order of highest image priority and transmits selected images that are determined to be transmittable to the remote control device at or below the estimated bit rate and with a delay amount below a predetermined value.

2. The automatic driving control device according to claim 1, wherein the data processing unit, when the estimated bit rate is equal to or higher than a required bit rate determined to enable all of the images captured by the multiple cameras to be transmitted to the remote control device with a delay amount not greater than a predetermined value, executes a transmission process for all of the images captured by the multiple cameras without executing the transmission image selection process, and when the estimated bit rate is less than the required bit rate determined to enable all of the images captured by the multiple cameras to be transmitted to the remote control device with a delay amount not greater than a predetermined value, calculates a required bit rate for each combination of images selected from the images captured by the multiple cameras in descending order of image priority, and transmits the largest combination of images for which the calculated required bit rate for each image combination is equal to or lower than the estimated bit rate.

3. The automatic driving control device according to claim 1, wherein the data processing unit allocates the transmission bit rate of each of the multiple camera images to be transmitted to the remote control device in proportion to the resolution corresponding to the number of pixels of each of the multiple camera images selected as the transmission target.

4. The automatic driving control device of claim 1, wherein the data processing unit calculates the image priority of each of the plurality of camera-captured images by taking into consideration at least one of (a) and (b): (a) the number of objects detected from each of the plurality of camera-captured images; and (b) the number of high-risk objects detected from each of the plurality of camera-captured images.

5. The automatic driving control device according to claim 1, wherein the data processing unit executes transmission bit rate control to set a target bit rate, which is the transmission bit rate when transmitting all of the images captured by the multiple cameras, between a predetermined minimum bit rate and a predetermined maximum bit rate, and when the estimated bit rate is equal to or greater than the maximum bit rate, sets the target bit rate to the maximum bit rate.

6. The automatic driving control device according to claim 1, wherein the data processing unit transmits an object image of an object detected from the camera-captured image to the remote assistance device.

7. The automatic driving control device according to claim 6, wherein the data processing unit performs an object detection process from non-transmitted images that are excluded from the transmission targets in the transmission image selection process, and transmits object images of objects detected from the non-transmitted images to the remote assistance device.

8. The automatic driving control device according to claim 6, wherein the data processing unit executes a process of transmitting all object images of the detected object when an object image transmission bit rate, which is a bandwidth available for transmitting object images, is equal to or greater than a required bit rate for transmitting all object images to the remote control device with a delay amount not greater than a predetermined value, and generates and transmits a compressed object image by compressing at least a portion of the object image when the object image transmission bit rate is less than the required bit rate for transmitting all object images to the remote control device with a delay amount not greater than a predetermined value.

9. The automatic driving control device according to claim 6, wherein the data processing unit determines a risk level of a detected object detected from an image, and preferentially selects and transmits object images of objects with a high risk level.

10. The automatic driving control device according to claim 6, wherein the data processing unit is configured to execute a compression process applying different compression algorithms depending on the risk level of a detected object detected from an image, and generates and transmits a compressed object image by applying a compression algorithm in which the compression rate is set lower for objects with a higher risk level and higher for objects with a lower risk level.

11. The automatic driving control device according to claim 6, wherein the data processing unit performs an image segmentation process on the image to perform an object detection process.

12. The automatic driving control device according to claim 1, wherein the communication unit performs image transmission using a multi-track video streaming technique that transmits images captured by multiple cameras in a single stream.

13. A remote assistance device having a communication unit that receives multiple camera images taken by multiple cameras mounted on a vehicle from an autonomous driving control device, and a data processing unit that executes display control to display the multiple camera images on a display unit, wherein the data processing unit switches and displays an image to be displayed on the display unit in response to the image received from the autonomous driving control device: (a) a camera image received from the autonomous driving control device, or (b) a composite image of a camera image and a 3DCG image received from the autonomous driving control device, or (c) a 3DCG image, or 14. The remote assistance device described in claim 13, wherein the data processing unit sets the image to be displayed on the display unit to the camera image received from the automatic driving control device when the image received from the automatic driving control device has high clarity, and when no image is received from the automatic driving control device, sets the image to be displayed on the display unit to a 3DCG image.

15. A remote support device as described in claim 13, wherein the data processing unit controls the opacity of a 3DCG image in a composite image of a camera-captured image and a 3DCG image to generate each of the images (a) to (c) and display them on a display unit.

16. The remote assistance device described in claim 15, wherein the data processing unit controls the opacity of the 3DCG image according to the transmission bit rate value of each of the multiple camera images determined by the autonomous driving control device.

17. A remote assistance device as described in claim 13, wherein the data processing unit displays an object image received from the automatic driving control device on a 3DCG image displayed on a display unit.

18. The remote assistance device according to claim 17, wherein the data processing unit changes and displays a display mode of an object image according to a risk level of the object.

19. The remote assistance device according to claim 13, wherein the data processing unit displays an object image generated by interpolation processing using object images from previous and next image frames for image frames in which an object image could not be received from the automatic driving control device.

20. An autonomous driving control system having an autonomous driving control device and a remote assistance device, wherein the autonomous driving control device performs a process of calculating an estimated bit rate equivalent to a bandwidth available for transmitting images to the remote control device, a process of calculating an image priority for each of a plurality of camera images taken by a plurality of cameras mounted on a vehicle, and a transmission image selection process of selecting the plurality of camera images in order of highest image priority and transmitting a selected image that is determined to be transmittable to the remote control device at or below the estimated bit rate and with an amount of delay not greater than a predetermined value, and the remote assistance device performs a process of switching and displaying an image to be displayed on a display unit in response to an image received from the autonomous driving control device, between any of the above (a) to (c), (a) a camera image received from the autonomous driving control device, or (b) a composite image of a camera image and a 3DCG image received from the autonomous driving control device, or (c) a 3DCG image, and transmits control information generated from the confirmation result of the displayed image to the autonomous driving control device.

Images