Remote driving system

Abstract

This technology provides the ability to acquire clear images of the area of ​​interest using an in-vehicle camera, even on roads with a large gradient and where the gradient changes in the direction of travel. [Solution] The remote driving system of the present disclosure comprises one or more processors. When the estimated gradient of the road surface on which the vehicle is traveling is greater than or equal to a threshold and the gradient of the road surface changes in the direction of travel of the vehicle, one or more processors correct the position of the area of ​​interest within the field of view of the onboard camera and the direction of illumination of the vehicle's headlights.

Description

Technical Field

[0001] This disclosure relates to a remote driving system.

Background Art

[0002] Patent Document 1 describes an image processing device for an in-vehicle camera. Specifically, when the vehicle is at a specific location such as an intersection where right or left turns may be made, a curved road, or the entrance / exit of a slope, this image processing device changes the position of the high-quality area within the imaging range so that the attention area, which is an area where the position relative to the vehicle changes according to the behavior of the vehicle, is imaged in high quality.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the remote driving of a vehicle in a remote driving system, generally, the video acquired by an in-vehicle camera is transmitted from the vehicle to a remote driving device, and on the remote driving device side, remote operation of the vehicle by a remote operator is performed based on the received video.

[0005] Here, for example, when the vehicle is in an environment where the gradient of the road surface changes in the traveling direction, the video captured by the in-vehicle camera may not necessarily be the video in the traveling direction of the camera. Therefore, in remote driving, the video transmitted from the vehicle in a situation where the gradient of the road surface changes in the forward direction of the traveling direction may not be the video in the direction required by the remote operator. And in such a case, simply changing the position of the attention area of the in-vehicle camera and imaging as in Patent Document 1 may result in insufficient brightness of the video. In this case, a situation may occur where a clear video in the direction required by the remote operator cannot be provided.

[0006] This disclosure is made in view of the above-mentioned issues and aims to provide a technology that enables the acquisition of clear images of a region of interest by an in-vehicle camera even on roads where the gradient of the road surface changes in the direction of travel. [Means for solving the problem]

[0007] One aspect of this disclosure is a remote driving system that remotely drives a vehicle based on information acquired by on-board sensors, including an on-board camera. This remote driving system comprises one or more processors. The one or more processors correct the position of the area of ​​interest within the field of view of the on-board camera and the direction of illumination of the vehicle's headlights when the estimated gradient of the road surface on which the vehicle is traveling is greater than or equal to a threshold and the gradient of the road surface changes in the direction of travel of the vehicle. [Effects of the Invention]

[0008] According to an example of a remote driving system in this disclosure, when driving on a road where the road surface gradient changes ahead in the direction of travel, the position of the area of ​​interest within the field of view of the onboard camera and the direction of illumination of the vehicle's headlights are corrected. This allows the operator performing the remote driving to be presented with a clear image of the area of ​​interest that has been corrected to the appropriate position according to the change in gradient. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram illustrating the overall configuration of the remote driving system according to an embodiment of the present disclosure. [Figure 2] This is a block diagram showing the functional configuration of a remote driving system according to an embodiment of the present disclosure. [Figure 3] This figure illustrates an overview of the method for calculating the estimated gradient by a vehicle control device according to an embodiment of the present disclosure. [Figure 4] This figure illustrates an overview of the method for calculating the estimated gradient by a vehicle control device according to an embodiment of the present disclosure. [Figure 5]This figure illustrates the position of ROI within the field of view of the in-vehicle camera according to the embodiment of the present disclosure. [Figure 6] This figure illustrates the correction of the illumination direction of the headlights by a vehicle control device according to an embodiment of the present disclosure. [Figure 7] This is a flowchart illustrating the control operations performed by the control device of the remote driving device according to the embodiment of the present disclosure. [Modes for carrying out the invention]

[0010] Embodiments of this disclosure will be described below with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and their descriptions are simplified or omitted.

[0011] Embodiment. Figure 1 is a schematic diagram illustrating the overall configuration of the remote driving system according to this embodiment. As shown in Figure 1, the remote driving system includes a vehicle 100 and a remote driving device 200. The remote driving device 200 and the vehicle 100 can communicate with each other via a communication network.

[0012] Vehicle 100 is a vehicle that is subject to remote driving by remote operator X. Vehicle 100 may be, for example, an autonomous vehicle or a vehicle driven by a driver. In this case, the driver or autonomous driving system of vehicle 100 requests remote driving support from the remote driving device 200 as needed. Upon receiving the request, the remote driving device 200 starts remote driving of the vehicle 100 that is the subject of support. Vehicle 100 is equipped with an on-board camera 101 that takes pictures of the area around the vehicle and acquires video (images) of the area around the vehicle. Vehicle 100 is also equipped with headlights 102.

[0013] The remote driving device 200 presents vehicle information transmitted from the vehicle 100 to the remote operator X via a display device 221 or the like. Based on the vehicle information presented on the display device 221, the remote operator X can recognize the status of the vehicle 100 and the surrounding conditions. The remote operator X inputs control units by operating input devices 230 such as the steering control member 231. Remote driving information generated based on the input control units is transmitted to the vehicle 100, thereby enabling remote driving of the vehicle 100.

[0014] Figure 2 is a block diagram showing the configuration of the remote driving system. The vehicle 100 shown in Figure 2 is equipped with a sensor group 110, a communication device 120, a driving device 130, and a control device 150. In this example, the vehicle 100 is equipped with an autonomous driving system and is capable of autonomous driving.

[0015] The sensor group 110 includes on-board sensors such as recognition sensors, vehicle condition sensors, and position sensors. The recognition sensors recognize (detect) the surrounding conditions of the vehicle 100 and include at least an on-board camera 101. Other examples of recognition sensors include LIDAR (Laser Imaging Detection and Ranging), radar, etc. The vehicle condition sensors detect the state of the vehicle 100. The vehicle condition sensors include speed sensors, acceleration sensors, yaw rate sensors, steering angle sensors, brake hydraulic pressure sensors, etc. The position sensors detect the position and orientation of the vehicle 100. For example, the position sensors include GNSS (Global Navigation Satellite System) sensors.

[0016] The communication device 120 communicates with the outside of the vehicle 100. The communication destinations of the communication device 120 include the remote driving device 200.

[0017] The running device 130 includes a steering actuator, a driving actuator, and a brake actuator. The steering actuator steers the wheels. The driving actuator is a power source that generates driving force. The brake actuator generates braking force.

[0018] The control device 150 acquires vehicle information from the sensor group 110. The vehicle information includes recognition sensor information indicating the recognition result by the recognition sensor, vehicle state information obtained from the vehicle state sensor, and position information obtained from the position sensor. The recognition sensor information includes at least the video around the vehicle 100 acquired by the in-vehicle camera 101. The vehicle state information includes speed information, acceleration information, steering angle information, brake hydraulic pressure information, etc. obtained from a speed sensor, an acceleration sensor, a steering angle sensor, a brake hydraulic pressure sensor, etc. The vehicle information acquired by the control device 150 can be used for the automatic driving or remote driving of the vehicle 100.

[0019] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as the processor 160) and one or more storage devices 170 (hereinafter simply referred to as the storage device 170). The processor 160 executes various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various programs and various information necessary for the processes by the processor 160. By the processor 160 executing the programs stored in the storage device 170, the functions of the control device 150 are realized. Examples of the storage device 170 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The control device 150 may include one or more ECUs (Electronic Control Unit).

[0020] The control device 150 has a function of controlling the running device 130 based on the vehicle information to control the running of the vehicle 10 _.

[0021] The remote driving device 200 includes a communication device 210, an output device 220, an input device 230, and a control device 250. The communication device 210 communicates with the outside world. The communication destination of the communication device 210 includes the vehicle 100.

[0022] The output device 220 outputs various information and presents it to the remote operator X. The output device 220 includes at least a display device 221. The display device 221 displays various information toward the remote operator X. The output device 220 may also include a speaker.

[0023] The input device 230 is a component operated by the remote operator X when remotely driving the vehicle 100. For example, the input device 230 includes the steering control member 231, as well as an accelerator pedal, brake pedal, turn signal, etc. The steering control member 231 is, for example, a steering wheel. The remote control member may also include a touch panel or buttons. The amount of operation input by the remote operator X when remotely driving the vehicle 100 is detected by a sensor installed on the input device 230.

[0024] The control device 250 controls the remote operation device 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as processor 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processors 260 perform various processes. For example, the processor 260 includes a CPU. The storage devices 270 store various programs and information necessary for processing by the processors 260. The functions of the control device 250 are realized when the processors 260 execute the programs stored in the storage devices 270. Examples of storage devices 270 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0025] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information transmitted from the vehicle 100. The control device 250 displays the vehicle information, including images, on the display device 221.

[0026] Furthermore, the control device 250 generates remote operation information based on the manipulated quantities input to the input device 230 by the remote operator X. The remote operation information is information for controlling the vehicle 100 by remote operation. The control device 250 transmits the remote operation information to the vehicle 100 as needed.

[0027] Vehicle 100 receives remote driving information transmitted from the remote driving device 200. Vehicle 100 controls its movement according to the received remote driving information. In this way, vehicle 100 is remotely driven.

[0028] The remote driving system according to this embodiment is configured to perform, in particular, ROI correction control, which changes the position of the ROI (Region of Interest) within the field of view of the onboard camera 101, and illumination direction correction control, which changes the illumination direction of the headlights 102, while the vehicle 100 is being remotely driven.

[0029] By default, the ROI is set to be located in the center of the field of view of the in-vehicle camera 101. The remote driving system corrects the position of the ROI on the screen of the in-vehicle camera 101 when the vehicle 100 meets the ROI correction conditions.

[0030] To perform ROI correction control and illumination direction correction control, the control device 150 of the vehicle 100 has a function to calculate the estimated gradient of the road surface, a function to correct the ROI, and a function to correct the headlight illumination direction.

[0031] Specifically, the control device 150 can calculate the estimated gradient of the road surface on which the vehicle 100 is traveling. Figures 3 and 4 are diagrams illustrating the overview of the method by which the control device 150 calculates the estimated gradient. The estimated gradient calculated by the control device 150 includes the road surface estimated gradient and the image estimated gradient.

[0032] The estimated road surface gradient is an estimate of the absolute gradient of the road on which the vehicle 100 travels, and is calculated using the output of an acceleration sensor mounted on the vehicle 100. That is, for example, as shown in Figure 3, the estimated road surface gradient is calculated using the vertical component a_z of the vehicle 100's acceleration a detected by the acceleration sensor and the gravitational acceleration g.

[0033] The image-estimated gradient is the relative estimated gradient shown by the image in front of the vehicle 100 in the direction of travel, and is a value that indicates the gradient of the road surface under which the vehicle 100 is traveling relative to the road surface gradient in front of the vehicle 100. As shown in Figure 4, the image-estimated gradient is calculated by image analysis using vehicle information such as the ratio of the part above the horizon to the part below the horizon in the image, the height of traffic lights in the image, and the installation position of the on-board camera 101. As the ratio, the ratio of the number of pixels contained in the part above the horizon and the part below the horizon in the image, or the ratio of the area of ​​the upper part and the lower part can be used. The image used to calculate the ratio here may be an image that includes the entire field of view of the on-board camera 101, or an ROI image.

[0034] The calculated estimated gradient information is incorporated as part of the vehicle information and transmitted to the remote driving device 200 via the communication device 120.

[0035] The control device 150 receives an ROI correction amount from the remote driving device 200 via the communication device 120. When the control device 150 receives an ROI correction command, it corrects the position of the ROI within the field of view of the on-board camera 101 according to the ROI correction amount included in the received ROI correction command. For example, as shown in Figure 3, when the gradient of the road surface on which the vehicle 100 is traveling is greater than a threshold and the road surface gradient changes to a horizontal state ahead at an intersection, the position of the ROI within the field of view of the on-board camera 101 is corrected in the direction of the arrow R in the drawing so that the central axis C0 moves to the road surface side (i.e., the lower side) position C1. As a result, as shown in Figure 5, the position of the ROI within the field of view A captured by the on-board camera 101 is corrected from the default position to the corrected ROI position.

[0036] The control device 150 has an upper limit for ROI movement speed and an upper limit for correction amount. When the control device 150 moves the ROI according to the ROI correction amount, it sets the movement speed to a range that does not exceed the upper limit for ROI movement speed. The upper limit for correction amount is set according to the ROI range, the current position of the ROI, and the range of the camera field of view A, so that the entire range of the ROI fits within the field of view A. The movement of the ROI position is limited to the upper limit for correction amount. This prevents the ROI from being set to a position beyond the camera field of view A, even if the ROI correction amount is set to exceed the upper limit for correction amount.

[0037] The control device 150 corrects the illumination direction of the vehicle 100's headlights 102 according to the ROI correction amount. Figure 6 shows the case where the illumination direction of the headlights 102 is corrected according to the ROI correction amount. The correction amount for the illumination direction of the headlights 102 is set according to the ROI correction amount. For example, the relationship between the ROI correction amount and the correction amount for the illumination direction of the headlights 102 is calculated in advance and stored in the storage device 170 so that the headlights 102 illuminate the corrected ROI position. The control device 150 calculates a correction amount according to the ROI correction amount according to the relationship stored in the storage device 170, and corrects the illumination direction of the headlights according to this correction amount as shown in Figure 6.

[0038] If the ROI correction amount is zero, the control device 250 will return the ROI position and the illumination direction of the headlight 102 to their default positions (i.e., positions before correction).

[0039] The remote operation device 200 includes a function to determine whether the ROI correction conditions are met and a function to calculate the ROI correction amount.

[0040] Specifically, the control device 250 of the remote driving device 200 determines whether the vehicle 100 satisfies the pre-set ROI correction conditions based on the vehicle information received from the vehicle 100. In this embodiment, the ROI correction conditions are set to satisfy all of the following conditions 1 to 3. (Condition 1) Vehicle 100 is in a vehicle state that allows ROI correction to be performed. (Condition 2) The estimated road surface gradient is equal to or greater than the first threshold. (Condition 3) The image estimation gradient is greater than or equal to the second threshold.

[0041] (Condition 1) Regarding the vehicle being in a state where ROI correction can be performed, more specifically, the conditions for determining that the vehicle is in a correctable driving state are predetermined, such as the vehicle 100 not being decelerating or accelerating, not turning, and the vehicle speed being below a reference value. Other conditions may also be set. The control device 250 determines whether (Condition 1) is met by determining whether the vehicle 100 satisfies these conditions based on the vehicle information received from the vehicle 100.

[0042] The estimated road surface gradient is an estimated absolute road surface gradient, and the control device 250 determines whether the gradient of the road on which the vehicle 100 travels is greater than or equal to the first threshold by determining whether (condition 2) is met. The first threshold is a value that has been calculated in advance and set in the storage device 270.

[0043] The image-estimated gradient is an estimated value of the relative gradient of the road surface the vehicle is currently traveling on, as shown by the image in front of the vehicle. The control device 250 determines whether the amount of change in the road surface gradient in front of the vehicle is greater than or equal to the second threshold by determining whether (condition 3) is met. Here, the second threshold is a value that has been calculated in advance and set in the storage device 270. The first threshold and the second threshold may be different values. Alternatively, the same threshold may be used for both the first and second thresholds.

[0044] In addition to the above conditions (1) to (3), ROI correction conditions may also be set to other conditions that particularly require ROI correction, such as vehicle 100 being at an intersection.

[0045] The control device 250 calculates an ROI correction amount based on the estimated gradient. The ROI correction amount is calculated based on the acquired estimated gradient. There are no limitations on the method of calculating the ROI correction amount; it is corrected so that the area required by the remote operator X in remote operation is acquired as an ROI. Specifically, for example, the ROI correction amount is set so that the image estimated gradient calculated from the ratio of the part above and the part below the horizontal line within the corrected ROI matches the second threshold. Alternatively, the ROI correction amount is set so that the ratio of the part above and the part below the horizontal line within the corrected ROI is the same as when the road surface gradient does not change even ahead, i.e., when the relative gradient is zero. Furthermore, the ROI correction amount may also recognize objects such as traffic lights that should be displayed as areas of interest in addition to the gradient, and if there are objects that should be displayed, the correction amount may be adjusted so that those objects are reliably displayed.

[0046] The control device 250 further includes a function to update the calculated ROI correction amount in accordance with the ROI operation performed by the remote operator X from the input device 230. The input device 230 has an operating unit that receives ROI operations. This operating unit is installed, for example, at a predetermined position on the steering operating member 231. The ROI operation information input from the input device 230 is input to the control device 250.

[0047] Here, ROI operations include inputs for correction operations that change the ROI correction amount and inputs for cancellation operations that cancel ROI correction. If the ROI operation input from the input device 230 is a cancellation operation, the control device 250 defaults the ROI correction amount, i.e., sets the ROI correction amount to zero. On the other hand, if an ROI position correction operation is input, the control device 250 modifies the ROI correction amount according to the operation amount and updates the ROI correction amount.

[0048] The ROI correction amount calculated by the control device 250 is transmitted to the vehicle 100 via the communication device 210.

[0049] Figure 7 is a flowchart illustrating the control operations performed by the control device 250 of the remote operation device 200. The control operations shown in Figure 7 are repeatedly performed at regular control intervals while remote operation is in progress.

[0050] In step S10 of Figure 7, it is first determined whether the vehicle 100 is in a state where correction can be performed, that is, whether it satisfies the above condition (1). If it is determined in step S10 that the vehicle is not in a state where correction can be performed, the process is terminated.

[0051] If it is determined in step S10 that the vehicle condition is correctable, then in step S11 it is determined whether the estimated road surface gradient is equal to or greater than the first threshold, that is, whether the above condition (2) is met.

[0052] If it is determined in step S11 that the estimated road surface gradient is greater than or equal to the first threshold, then in step S12, it is determined whether or not the image-estimated gradient is greater than or equal to the second threshold, that is, whether or not the above condition (3) is met.

[0053] If it is determined in step S11 that the estimated road surface gradient is not equal to or greater than the first threshold, or if it is determined in step S12 that the estimated image gradient is not equal to or greater than the second threshold, then in step S13 the ROI correction amount is set to the default value, i.e., to zero. When the road surface gradient no longer satisfies the ROI correction conditions, the ROI correction amount is set to the default value, which automatically returns the ROI position to its default position.

[0054] On the other hand, if the image-estimated gradient is determined to be greater than or equal to the second threshold in step S12, the ROI correction amount is then calculated in step S14 and set to the calculated value. The ROI correction amount is set according to the road surface gradient. When the ROI correction conditions (i.e., steps S10 to S12) are met, the ROI correction amount is set according to the estimated gradient, thereby changing the position of the ROI to the position desired by the remote operator X.

[0055] After the ROI correction amount is defaulted in step S13, or after the ROI correction amount is calculated in step S14, the set ROI correction amount is sent to the vehicle 100 in step S15.

[0056] Next, in step S16, it is determined whether or not an ROI correction operation has been input from the input device 230. If it is determined in step S16 that an ROI correction operation has been input, in step S17, the ROI correction amount is modified according to the amount of the correction operation. The ROI correction amount is updated to the modified ROI correction amount. By modifying the ROI correction amount, the position of the ROI can be corrected to the position desired by the remote operator X.

[0057] On the other hand, if it is determined in step S16 that no ROI correction operation has been entered, then in step S18, it is determined whether or not a cancellation operation to cancel the ROI correction has been entered. If it is determined in step S18 that a cancellation operation has been entered, the process ends. If it is determined in step S18 that a cancellation operation has been entered, then in step S19 the ROI correction amount is defaulted to zero.

[0058] After the ROI correction amount is updated in step S17, or after the ROI correction amount is defaulted in step S19, the ROI correction amount is resent to the vehicle 100 in step S20. The process then ends.

[0059] As described above, according to the remote driving system of this embodiment, when the ROI correction conditions are met, the ROI correction amount is automatically calculated and ROI correction is performed. This allows the remote operator X to be appropriately provided with the field of view around the vehicle 100 necessary for remote driving. In addition, the illumination direction of the headlights 102 is corrected according to the ROI correction amount. This allows the image information necessary for driving to be provided in a clearer state. Furthermore, when the ROI correction conditions are no longer met, the ROI and the illumination direction of the headlights 102 are returned to their default positions, so the remote operator X can be provided with the necessary field of view without having to perform any operation to return to the default settings.

[0060] In this embodiment, the functions for determining whether the ROI correction conditions of the remote driving device 200 are met, calculating the ROI correction amount, and receiving vehicle information and transmitting the ROI correction amount may be functions provided by a management server located outside the remote driving device 200. In this case, the remote driving device 200 transmits to the management server information on ROI operations input by the remote operator X, i.e., the amount of the ROI correction operation and the input information for the cancellation operation. Based on the received ROI operation information, the management server updates the ROI correction amount and transmits it to the vehicle 100.

[0061] Alternatively, the functions for determining whether the ROI correction conditions of the remote driving device 200 are met and for calculating the ROI correction amount may be functions provided by the vehicle 100. In this case, the remote driving device 200 receives video information captured with the corrected ROI settings from the vehicle 100. The remote driving device 200 transmits ROI operation information input by the remote operator X, i.e., the amount of the correction operation and the input information for the cancellation operation, to the vehicle 100. The control device 150 of the vehicle 100 updates the ROI correction amount based on the received information and corrects the ROI position based on the updated ROI correction amount. [Explanation of symbols]

[0062] 100 vehicles 101 In-car camera 102 Headlights 110 sensor group 120 Communication equipment 130 Running gear 150 Control device 160 processors 170 Storage device 200 Remote control devices 210 Communication equipment 220 Output device 221 Display device 230 Input devices 231 Steering operating member 250 Control devices 260 processors 270 Storage device

Claims

1. A remote driving system that remotely drives a vehicle based on information acquired by on-board sensors, including an on-board camera, Equipped with one or more processors, The one or more processors correct the position of the area of ​​interest within the field of view of the onboard camera and the direction of illumination of the vehicle's headlights when the estimated gradient of the road surface on which the vehicle is traveling is greater than or equal to a threshold and the gradient of the road surface changes in the direction of travel of the vehicle. Remote driving system.

2. If the vehicle is in a state where correction of the area of ​​interest is not possible, the one or more processors will not perform correction of the position of the area of ​​interest or correction of the direction of illumination of the headlights, even if the gradient is greater than or equal to the threshold. The remote driving system according to claim 1.

3. The vehicle conditions under which correction of the area of ​​interest cannot be performed are when the vehicle is turning, decelerating or accelerating, or when the vehicle speed is above a reference value. The remote driving system according to claim 2.

4. If the gradient becomes smaller than the threshold, one or more of the processors return the position of the region of interest to its position before correction and return the direction of the headlight illumination to its direction before correction. A remote driving system according to any one of claims 1 to 3.

Images