Remote operation support method and device

Abstract

To suppress flickering of the display of a camera video when the camera video in which the effect of a communication delay has been compensated for is provided to an operator.SOLUTION: Video information including a plurality of frames acquired by a camera of a mobile body and respective reference times of those frames is received from the mobile body. Processing is performed using prediction information for a set time ahead of the respective reference times of the plurality of frames to generate a plurality of future frames from the respective frames. Display control of a camera video including the plurality of future frames is performed so that the plurality of future frames are displayed on a display of a terminal operated by an operator at times when the set time has elapsed from the respective reference times of the plurality of frames.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a method and apparatus for assisting an operator in remotely driving a mobile object. [Background technology]

[0002] International Publication No. WO 2018 / 155159 discloses a system that outputs camera images received from a vehicle from a remote device. In this conventional system, when the communication delay time from the vehicle to the remote device is a first time, a first range is cut out from the frame of the camera image and output from the remote output device. In addition, in the conventional system, when the communication delay time from the vehicle to the remote device is a second time, a second range is cut out from the frame of the camera image and output from the remote device.

[0003] The second time is longer than the first time. The second range is narrower than the first range. When the first and second ranges are output, these ranges are enlarged in accordance with the size of the display unit of the remote device. Therefore, when the second range is output, a camera image is displayed on the display unit as if the vehicle had moved forward, compared to when the first range is output. Therefore, it is possible to provide the operator with a camera image in which the effects of communication delays have been compensated for. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 155159 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional system described above, when the communication delay time increases or decreases and crosses the boundary between the first and second times many times, the display mode of the camera image is switched. This causes the display unit to flicker many times, which makes the operator who is looking at the camera image tired. Therefore, improvements to prevent such problems are desired.

[0006] One object of the present disclosure is to provide a technology that can suppress flickering in the display of a camera image when the camera image, in which the effects of communication delays have been compensated for, is provided to an operator. [Means for solving the problem]

[0007] A first aspect of the present disclosure is a method for assisting an operator in remotely driving a moving object, and has the following features. The method includes the steps of receiving video information from the moving body, the video information including a plurality of frames captured by a camera of the moving body and reference times for each of the plurality of frames; performing processing using prediction information that is a set time ahead of each of the reference times of the plurality of frames to generate a plurality of future frames from each of the plurality of frames; and controlling the display of the camera video including the plurality of future frames so that the plurality of future frames are displayed on a display of a terminal operated by the operator at a time when the set time has elapsed from each of the reference times of the plurality of frames.

[0008] A second aspect of the present disclosure is an apparatus for assisting an operator in remotely driving a moving object, and has the following features. The device includes a processor for performing various processes, and the processor is configured to receive from the mobile body video information including a plurality of frames captured by a camera of the mobile body and reference times of the plurality of frames, to perform processing using prediction information for a set time ahead of each of the reference times of the plurality of frames to generate a plurality of future frames from each of the plurality of frames, and to perform display control of the camera video including the plurality of future frames so that the plurality of future frames are displayed on a display of a terminal operated by the operator at a time when the set time has elapsed from each of the reference times of the plurality of frames. Effect of the Invention

[0009] According to the present disclosure, a processing process is performed using prediction information that is a set time ahead of each reference time of a plurality of frames to generate a plurality of future frames. This processing process makes it possible to generate a camera video including a plurality of future frames in which the effects of communication delays have been compensated for. According to the present disclosure, display control of the camera video including these future frames is also performed so that these future frames are displayed on the display at the time when the set time has elapsed from each reference time of the plurality of frames. This display control also makes it possible to display these future frames one after another on the display each time the time when the set time has elapsed from each reference time of the plurality of frames arrives. Therefore, it is possible to suppress flickering of the display of the camera video in which the effects of communication delays have been compensated for. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating remote operation according to an embodiment. [Diagram 2] 11 is a flowchart illustrating a process flow particularly related to the embodiment. [Diagram 3] FIG. 11 is a diagram illustrating an example of estimating a position and an orientation. [Figure 4] 3 is a diagram illustrating a first example of the process in step S13 of FIG. 2. FIG. [Diagram 5] 3 is a diagram illustrating a second example of the process in step S13 of FIG. 2. [Figure 6] FIG. 6 is a diagram illustrating the effects of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and the description thereof will be simplified or omitted.

[0012] 1.Remote operation Fig. 1 is a diagram for explaining remote driving according to an embodiment. Fig. 1 is a diagram showing an example of the overall configuration of a system related to remote driving. In the example shown in Fig. 1, a vehicle VH as an example of a "moving body" of the present disclosure is driven (i.e., remotely driven) by an operator OP who is in a remote cockpit 2 as an example of a "remote driving device" of the present disclosure. The vehicle VH and the remote cockpit 2 communicate with each other via, for example, a wireless network (not shown), and exchange various data.

[0013] 1, a vehicle system 1 mounted on a vehicle VH includes sensors 11, a driving device 12, and a data processing device 13. The vehicle VH is, for example, a vehicle that can be automatically driven by the vehicle system 1. The vehicle VH may also be a vehicle that can be manually driven by a driver of the vehicle VH. In this case, the vehicle system 1 controls the driving of the vehicle VH based on various driver operation amounts input to a driving device (not shown) of the vehicle VH.

[0014] The sensors 11 include a recognition sensor, a position sensor, and a status sensor. The recognition sensor recognizes the surrounding conditions of the vehicle VH. Examples of the recognition sensor include a camera, a millimeter wave radar, and a LiDAR (Light Detection And Ranging). The position sensor acquires position and orientation data of the vehicle VH. Examples of the position sensor include a GNSS (Global Navigation Satellite System) receiver. The status sensor detects the speed, acceleration (e.g., longitudinal acceleration and lateral acceleration), yaw rate, wheel turning angle, steering angle, etc. of the vehicle VH.

[0015] The traveling device 12 accelerates, decelerates, and steers the vehicle VH. The traveling device 12 includes, for example, wheels, a motor, a steering device, and a brake device. The motor drives the wheels. The steering device steers the wheels. The brake device applies a braking force to the vehicle VH. The vehicle VH is accelerated by controlling the motor. The vehicle VH is decelerated by controlling the brake device. The vehicle VH may be braked using regenerative braking controlled by the motor. The vehicle VH is steered by controlling the steering device.

[0016] The data processing device 13 is an example of a "mobile terminal" in the present disclosure. The data processing device 13 includes at least one processor 14 and at least one memory 15. The processor 14 includes a CPU (Central Processing Unit). The memory 15 is a volatile memory such as a DDR memory, and expands various programs used by the processor 14 and temporarily stores various data. The various data include various data acquired from the sensors 11.

[0017] The data processing device 13 communicates with the remote cockpit 2 and exchanges various data with the remote cockpit 2. Examples of various data that the data processing device 13 receives from the remote cockpit 2 include a request to start remote driving of the vehicle VH, a control command for remote driving, and the like. The control commands include drive commands, braking commands, and steering commands required for controlling the traveling device 12. The control commands also include a command to specify the position of the shift lever of the vehicle VH, and a command to operate driving assistance devices such as the headlights and turn signals of the vehicle VH.

[0018] Examples of various data transmitted from the data processing device 13 to the remote cockpit 2 include a request for remote driving by the remote cockpit 2 (operator OP), data related to the driving environment of the vehicle VH, etc. The data related to the driving environment includes internal data such as the speed, acceleration (deceleration), and wheel turning angle (or steering angle) of the vehicle VH, and external data such as an image of the surroundings of the vehicle VH including at least an image in front of the vehicle VH.

[0019] In the embodiment, data associated with the image of the surroundings of the vehicle VH includes data of each reference time of a plurality of frames constituting the image of the surroundings. Examples of each reference time include the time when each of the plurality of frames is acquired by the sensors 11, and the time when each of these frames is transmitted to the outside of the data processing device 13. Each reference time does not need to directly indicate time. For example, it may be a counter that is synchronized in advance between the vehicle VH and the remote cockpit 2 and increases with the passage of time. The data of each reference time, together with the image of the surroundings of the vehicle VH, constitutes the "image information" of the present disclosure.

[0020] The data processing device 13 performs automatic driving control and driving assistance control of the vehicle VH. In the automatic driving control, for example, the control of the traveling device 12 is autonomously performed based on data related to the driving environment of the vehicle VH acquired from the sensors 11. The driving assistance control includes the control of the traveling device 12 based on a control command received from the remote cockpit 2. The control of the traveling device 12 based on an operation input by the driver of the vehicle VH is also included in this driving assistance control.

[0021] The remote cockpit 2 is a device for remote operation by an operator OP. In the example shown in Fig. 1, the remote cockpit 2 includes a driving device 21, a display 22, and a data processing device 23.

[0022] The driving device 21 includes various devices for remote driving. The various devices include, for example, an accelerator pedal, a steering wheel, a brake pedal, and a shift lever. The various devices include a driving assistance device. Examples of the driving assistance device include a headlight operation lever, a turn signal operation lever, and an ignition switch.

[0023] The display 22 outputs an image of the front of the vehicle VH. In addition to the image of the front of the vehicle VH, the display 22 may also output an image of the surroundings of the vehicle VH other than the image of the front (e.g., an image of the sides and rear of the vehicle VH). The display 22 may include two or more displays. In this case, for example, an image of the front of the vehicle VH is output from the main display, and an image other than the image of the front is output from the sub-display.

[0024] The data processing device 23 includes at least one processor 24 and at least one memory 25. The processor 24 includes a CPU. The memory 25 is a volatile memory such as a DDR memory, and expands various programs used by the processor 24 and temporarily stores various data. The various data includes images of the surroundings of the vehicle VH and data associated with the images (i.e., data at each reference time). The various data also includes data related to the operating state of the driving device 21. Examples of the operating state include the amount of depression of the accelerator pedal, the steering angle of the steering wheel, the amount of depression of the brake pedal, and the position of the shift lever. Examples of the processing performed by the processor 24 will be described later.

[0025] 2. Remote driving assistance 2 is a flowchart illustrating a process flow related to the embodiment. The routine shown in FIG. 2 is repeatedly executed by the data processing device 23 (processor 24) at a predetermined cycle.

[0026] In the routine shown in Fig. 2, various information is first acquired (step S11). Examples of the various information acquired in step S11 include various data stored in memory 25. This various data includes an image of the surroundings of vehicle VH and data associated with this image of the surroundings. The various data also includes data related to the operating state of driving device 21. Note that the data related to the operating state is acquired from a time that is a certain time back from the current time to the current time.

[0027] Following the processing of step S11, the position and attitude are estimated (step S12). The position and attitude estimated in step S12 are for at least one of the body and camera of the vehicle VH in the future, a set time ST ahead of each reference time of the multiple frames constituting the image of the surroundings of the vehicle VH. The set time ST is a fixed value, and any time can be applied to this fixed value (for example, several milliseconds to several seconds). When the history of the delay time of communication between the vehicle system 1 (data processing device 13) of the vehicle VH and the remote cockpit 2 is available, the set time ST may be determined by referring to the delay time calculated from this history.

[0028] In the following, for the sake of simplicity, the image of the surroundings of the vehicle VH is taken to be in front of the vehicle VH. In this case, the camera of the vehicle VH is referred to as the "front camera FC". FIG. 3 is a diagram for explaining an example of estimating the position and attitude. In the example shown in FIG. 3, the position and attitude are estimated under the assumption that the vehicle VH makes a steady circular turn from the reference time to a future time that is a set time ST ahead. In addition, the position of the center of gravity GC of the vehicle body VB and the distance from this center of gravity GC to the front camera FC are assumed to be known.

[0029] Assuming that the vehicle VH makes a steady circular turn, it is possible to define a locus T of the center of gravity GC described by the movement of the vehicle VH. The radius R of this locus T is expressed by the following formula (1). R = (1 + KV 2 ) × L / δ (1) In equation (1), K is the stability factor, V is the vehicle speed, L is the wheelbase, and δ is the tire angle.

[0030] 3, the slip angle β of the vehicle body VB is calculated after the calculation of the locus T. The slip angle β is expressed by the following equation (2) with the tire angle δ and the steady-state gain Gβ of the vehicle body slip angle as variables. β = Gβ × δ (2) Gβ = {df(1-V 2 / gLdfCr)} / (1+KV 2 ) · · · (3) In equation (3), df is the front weight distribution, g is the gravitational acceleration, and Cr is the rear wheel normalized cornering power CP.

[0031] 3, after the slip angle β is calculated, the trajectory T is rotated by the slip angle β, with the center of gravity GC as the rotation center, so that the front direction of the front camera FC coincides with the fore-and-aft direction (X direction) of the vehicle body VB.

[0032] Between the reference time and a future time that is the set time ST, the center of gravity GC moves on the trajectory T* after the rotation. Therefore, the position on the trajectory T* that is a moving distance calculated by time-integrating the estimated vehicle speed Ve (where the vehicle speed Ve is constant) expressed by the following formula (4) from the current position of the center of gravity GC is set to be the position of the center of gravity GC after the set time ST has elapsed. Ve = Vm + Acc × Δt (4) In equation (4), Vm is the observed value of the vehicle speed V by the state sensor (vehicle speed sensor) of the vehicle VH, and Acc is the current depression amount of the accelerator pedal in the remote cockpit 2. Acc may be the average of the current depression amount and that before the set time ST has elapsed.

[0033] Since the distance from the center of gravity GC to the front camera FC is known, the position of at least one of the vehicle body VB and the front camera FC after the set time ST has elapsed is estimated from the position of the center of gravity GC after the set time ST has elapsed. Also, the attitude of at least one of the vehicle body VB and the front camera FC after the set time ST has elapsed is estimated from the angle between a tangent line passing through the position of the center of gravity GC after the set time ST has elapsed and a tangent line passing through the current position of the center of gravity GC.

[0034] A second example of estimating the position and attitude will now be described. In this second example, the following equation (5) is used, which is a differential equation relating to the slip angle β and the yaw rate r arranged in the form of a state equation.

number

[0035] In the second example, the slip angle β and yaw rate r based on the latest observation value by the state sensor (for example, an acceleration sensor and a yaw rate sensor) of the vehicle VH are substituted into the state equation shown in formula (5), and the slip angle β and yaw rate r after the set time ST are calculated. The integral value of the yaw rate r from the reference time to a future time beyond the set time ST is the change in the yaw angle of the vehicle body VB. From this change in the yaw angle, the attitude of at least one of the vehicle body VB and the front camera FC after the set time ST has elapsed is estimated. In addition, the time integral of the change in the yaw angle and the integrated value of the vehicle speed V is the lateral change in the vehicle body VB, and the longitudinal change in the vehicle body VB is calculated by the time integral of the estimated vehicle speed Ve (where the vehicle speed Ve is constant) expressed by the above formula (4). Therefore, the position of at least one of the vehicle body VB and the front camera FC after the set time ST has elapsed is also estimated.

[0036] The information on the position and attitude of at least one of the vehicle body VB and the front camera FC estimated by the first example described with reference to Figure 3 or the second example described above corresponds to the "prediction information" of the present disclosure.

[0037] Returning to FIG. 2, following the processing of step S12, a plurality of frames constituting the image in front of the vehicle VH are processed (step S13). The target of the processing of step S13 is a frame stored in the memory 25 at the time of the processing of step S13, and is an unprocessed frame. In other words, among the frames received by the remote cockpit 2 (data processing device 23) at the time of the processing of step S13, a frame that has never been processed by the processing of step S13 of the previous or previous routine is processed in the processing of step S13 of the current routine. The processing of step S13 is performed based on the information of the position and attitude estimated in the processing of step S12. When the processing of step S13 is performed, "a plurality of future frames" are generated from each of the plurality of frames.

[0038] Fig. 4 is a diagram for explaining a first example of the process of step S13. Fig. 4 illustrates one frame FR among a plurality of frames constituting an image in front of the vehicle VH. In the first example, an augmented reality marker AR is generated based on information on the position and attitude of the vehicle body VB after a set time ST has elapsed from the reference time of the frame FR, and the marker AR is superimposed on the frame FR. This generates a future frame FF.

[0039] FIG. 5 is a diagram for explaining a second example of the process of step S13. The second example utilizes projective transformation. In projective transformation, first, coordinates c1 to c4 of four points in a virtual space are set. Then, by perspective projection transformation, these coordinates c1 to c4 are transformed into coordinates on the camera images captured by the cameras CA1 and CA2. The coordinates c1 to c4 on the camera image IM_CA1 and the coordinates c1 to c4 on the camera image IM_CA2 shown in FIG. 5 are examples of coordinates c1 to c4 after perspective projection transformation.

[0040] Once the coordinates c1 to c4 after the perspective projection transformation have been identified, the projection transformation matrix H of the following equation (12) is calculated, which holds between each coordinate x of the coordinates c1 to c4 on the camera image IM_CA1 and each coordinate x' of the coordinates c1 to c4 on the camera image IM_CA2. x' = Hx (12)

[0041] In the second example, the position and orientation information of the front camera FC at a reference time of a certain frame FR is applied to the camera CA1, and the position and orientation information of the front camera FC at a set time ST from this reference time is applied to the camera CA2. Then, once the projective transformation matrix H of the above formula (12) is calculated, this projective transformation matrix H is applied to the entire frame FR. This generates a future frame FF.

[0042] Returning to FIG. 2, following the processing of step S13, display control is performed (step S14). In the processing of step S14, a camera image including a plurality of future frames generated by the processing of step S13 of the previous routine is output from the display 22. When outputting these future frames, the plurality of future frames generated by the processing of step S13 of the previous routine are rectified. In addition, the output timing of the plurality of future frames generated by the processing of step S13 of the previous routine is controlled so that a future frame FF is displayed on the display 22 at a time when a set time ST has elapsed from the reference time of this future frame FF.

[0043] 3.Effects Fig. 6 is a diagram for explaining the effect of the embodiment. The first row of Fig. 6 illustrates frames FR1-FR11 as an example of a plurality of frames constituting an image ahead of the vehicle VH. The image ahead including frames FR1-FR11 is output from an operator OP (display 22). "Comparative Example 1" illustrated in the second row of Fig. 6 illustrates the flow of these frames FR1-FR11 when they are output as is. As can be seen from "Comparative Example 1," the output of frames FR1-FR11 is delayed by the communication delay CD.

[0044] The "Comparative Example 2" shown in the third row of FIG. 6 illustrates the flow of future frames FF1 to FF11 generated from frames FR1 to FR11 by performing the position and attitude estimation process described in step S12 of FIG. 2 and the processing process described in step S13 of FIG. 2 and outputting the frames from the display 22. However, in the "Comparative Example 2", the time when the remote cockpit 2 receives the multiple frames is used instead of the reference time. That is, in the "Comparative Example 2", the future frames FF1 to FF11 are generated based on the position and attitude at the time set time ST after the time when the remote cockpit 2 receives the frames FR1 to FR11. Therefore, in the "Comparative Example 2", the intervals of the output of the future frames FF1 to FF11 are not uniform. The reason for this is that the time when the remote cockpit 2 receives the multiple frames is used, and therefore the variation DV of the communication delay between the multiple frames is not uniform.

[0045] In this regard, according to the embodiment, the processes described in steps S12 and S13 in Fig. 2 are performed using the reference time. Also, the display control described in step S14 is performed. Therefore, as shown in "Example" in the fourth row of Fig. 6, each time the time arrives when the set time ST has elapsed from each reference time of the future frames FF1 to FF11, these future frames can be displayed one after another on the display 22. Therefore, it is possible to suppress flickering of the display of the camera video in which the influence of communication delays has been compensated for. [Explanation of symbols]

[0046] 1...vehicle system, 2...remote cockpit, 13, 23...data processing device, 14, 24...processor, 15, 25...memory, 22...display, AR...augmented reality marker, FC...forward camera, FF, FF1 to FF11...future frames, FR, FR1 to FR11...frames, OP...operator, ST...setting time, VB...vehicle body, VH...vehicle

Claims

1. A method for assisting an operator in remotely driving a moving object, comprising: receiving, from the moving body, image information including a plurality of frames captured by a camera of the moving body and reference times of the plurality of frames; performing processing using prediction information ahead of each reference time of the plurality of frames by a set time, thereby generating a plurality of future frames from each of the plurality of frames; performing a display control of a camera image including the plurality of future frames so that the plurality of future frames are displayed on a display of a terminal operated by the operator at a time when the set time has elapsed from each reference time of the plurality of frames; A remote driving assistance method comprising:

2. 2. The method of claim 1 , The reference time of each of the plurality of frames includes a time when each of the plurality of frames is transmitted from the mobile terminal to the outside. A remote driving assistance method comprising the steps of:

3. 3. The method according to claim 1 or 2, the prediction information includes information on a position and an orientation of the camera at a future time that is the set time ahead of each reference time of the plurality of frames, The processing includes a process of performing a projective transformation based on information on the position and orientation of the camera on each of the plurality of frames. A remote driving assistance method comprising the steps of:

4. 3. The method according to claim 1 or 2, the prediction information includes information on a position and an attitude of the moving object at a future time that is the set time ahead of each reference time of the plurality of frames, The processing includes a process of superimposing an auxiliary image based on information on the position and orientation of the moving object on each of the plurality of frames. A remote driving assistance method comprising the steps of:

5. An apparatus for assisting an operator in remotely driving a moving object, comprising: Equipped with a processor that performs various processes, The processor, receiving, from the moving body, image information including a plurality of frames captured by a camera of the moving body and reference times of the plurality of frames; performing processing using prediction information for a set time ahead of each reference time of the plurality of frames to generate a plurality of future frames from each of the plurality of frames; Display control of camera images including the plurality of future frames is performed so that the plurality of future frames are displayed on a display of a terminal operated by the operator at a time when the set time has elapsed from each reference time of the plurality of frames. A remote driving support device characterized by being configured as follows.

Images