Remote assistance system

Abstract

To provide a technology for enabling an operator to continuously confirm a peripheral situation of a vehicle from the past in remote assistance of the vehicle.SOLUTION: A remote assistance system comprises one or more processors constituted to display a video acquired from a camera loaded on a vehicle, or an infrastructure camera in a display device of a remote operator terminal. The remote assistance system shortens a first past video recorded from first timing in the past to current time to a second period shorter than a first period from the first timing in the past to the current time to be displayed in the display device when it is necessary to display the past video on the display device. The remote assistance system switches display of the display device to a real time video after display of the first past image corresponding to the second period is terminated.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a technology for remotely assisting a vehicle. [Background technology]

[0002] Patent Document 1 discloses a device that captures images of the vehicle's surroundings using a camera mounted on the vehicle and displays the captured images of the vehicle's surroundings on a monitor display. Other documents that demonstrate the technical level of this technical field include Patent Document 2 and Patent Document 3 below. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-218197 [Patent Document 2] Japanese Patent Application Publication No. 02-164198 [Patent Document 3] Patent Publication No. 2021-057724 Summary of the Invention [Problem to be solved by the invention]

[0004] Consider remote vehicle assistance. Until an operator is assigned to a vehicle receiving remote assistance, the operator cannot check the vehicle's surroundings. Furthermore, the vehicle's surroundings change from moment to moment. If remote assistance for a vehicle is provided based on the vehicle's surroundings after an operator is assigned, there is a risk that the vehicle will not be provided with appropriate remote assistance.

[0005] One object of the present disclosure is to provide a technology for remotely supporting a vehicle that allows an operator to continuously check the surrounding conditions of the vehicle from the past. [Means for solving the problem]

[0006] A first aspect of the present disclosure relates to a remote assistance system for remotely assisting a vehicle. The remote assistance system includes one or more processors configured to display images acquired from a camera mounted on a vehicle or an infrastructure camera on a display device of a remote operator terminal. The one or more processors When it is necessary to display the past video on the display device, the first past video recorded from the first timing in the past to the current time is shortened to a second period shorter than the first period from the first timing in the past to the current time, and the shortened period is displayed on the display device; After the display of the first past video corresponding to the second period has ended, the display on the display device is switched to the real-time video. [Effects of the Invention]

[0007] According to the present disclosure, when it is necessary to display past video on a display device, first past video recorded from a first timing in the past to the current time is displayed on the display device after being shortened to a second period that is shorter than the first period from the first timing in the past to the current time. After the display of the first past video corresponding to the second period is completed, the display on the display device switches to real-time video. This allows the operator to continuously check the vehicle's surrounding conditions from the past, enabling appropriate remote support for the vehicle. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram for explaining an overview of a remote support system according to an embodiment; [Figure 2] FIG. 10 is an explanatory diagram showing an example of a scene in which past video footage needs to be displayed. [Figure 3] FIG. 1 is a diagram for explaining an overview of fast-forward playback of past video. [Figure 4] FIG. 10 is an explanatory diagram showing a specific example of fast-forward playback of past video. [Figure 5] FIG. 10 is an explanatory diagram showing a modified example of a scene that requires display of past video. [Figure 6] FIG. 10 is an explanatory diagram showing a modified example of fast-forward playback of past video. DETAILED DESCRIPTION OF THE INVENTION

[0009] A remote support system according to an embodiment of the present disclosure will be described with reference to the accompanying drawings. Elements common to the drawings will be designated by the same reference numerals, and redundant description will be omitted.

[0010] 1. Overview of the remote support system 1 is a diagram illustrating an overview of a remote support system 1 according to an embodiment. The remote support system 1 is a system for remotely supporting a vehicle 2. When a request for remote support is received from the vehicle 2, the remote support system 1 performs remote support for the vehicle 2.

[0011] The remote assistance system 1 includes, for example, a vehicle 2, a remote operator terminal 20, and a management device 100. The vehicle 2, the remote operator terminal 20, and the management device 100 can communicate with each other via a communication network. For example, the vehicle 2 can wirelessly communicate with the remote operator terminal 20 and the management device 100 via a wireless communication network. The remote operator terminal 20 and the management device 100 can communicate with each other via a wired / wireless communication network.

[0012] Vehicle 2 is a target for remote assistance. Vehicle 2 that is a target for remote assistance may be an autonomous vehicle or a manually driven vehicle. Autonomous driving means that the steering, acceleration, and deceleration of vehicle 2 are performed automatically, independent of the driver's operation. If vehicle 2 is an autonomous vehicle, vehicle 2 may be equipped with an autonomous driving function for autonomous driving. When the autonomous driving function determines that autonomous driving of vehicle 2 is difficult, it requests remote assistance from management device 100. For example, the autonomous driving function sends a remote assistance request to management device 100.

[0013] The remote operator terminal 20 is a terminal device used by the remote operator O when remotely supporting the vehicle 2. In other words, the remote operator terminal 20 is configured to be used by the remote operator O to remotely support the vehicle 2. Examples of the remote operator terminal 20 include a cockpit-type terminal, a PC, a tablet, a smartphone, and the like.

[0014] The management device 100 is a computer that manages the remote assistance system 1. The management device 100 may be configured with multiple servers that perform distributed processing. The management device 100 manages the vehicle 2. The management device 100 may also manage multiple remote operators O and multiple remote operator terminals 20. For example, in response to a remote assistance request, the management device 100 assigns the remote operators O and the remote operator terminals 20 to remote assistance for the vehicle 2.

[0015] The vehicle 2 is equipped with a camera 4. The camera 4 captures images of the situation around the vehicle 2 and acquires images (video) showing the situation around the vehicle 2. The vehicle 2 communicates with a remote operator terminal 20 and transmits the video to the remote operator terminal 20. The vehicle 2 may transmit the video to the remote operator terminal 20 via the management device 100.

[0016] If an infrastructure camera 5 is installed at a location where remote assistance for the vehicle 2 is to be provided or in the vicinity thereof, the remote operator O may provide remote assistance for the vehicle 2 using images (video) acquired from the infrastructure camera 5. In this case, the infrastructure camera 5 acquires video showing the situation at the location where remote assistance for the vehicle 2 is to be provided or in the vicinity thereof. The infrastructure camera 5 communicates with the remote operator terminal 20 and transmits the video to the remote operator terminal 20. The infrastructure camera 5 may transmit the video to the remote operator terminal 20 via the management device 100. The infrastructure camera 5 is configured to be able to communicate with the remote operator terminal 20 and the management device 100.

[0017] The remote operator terminal 20 receives the video transmitted from the vehicle 2 or the video transmitted from the infrastructure camera 5. The remote operator terminal 20 presents the video acquired from the camera 4 or the infrastructure camera 5 to the remote operator O. Specifically, the remote operator terminal 20 is equipped with a display device 21, and displays the video acquired from the camera 4 or the infrastructure camera 5 on the display device 21.

[0018] The remote operator O views the image displayed on the display device 21, recognizes the surrounding situation of the vehicle 2, and provides remote support for the vehicle 2. In this way, remote support for the vehicle 2 is realized.

[0019] Here, let us consider the image displayed on the display device 21. The image displayed on the display device 21 may be real-time image corresponding to the current time, or past image showing image from an earlier time than the current time. Past image is image recorded from a past timing to the current time. Past image is obtained by recording image acquired by the camera 4 or the infrastructure camera 5. Past image may be acquired by the management device 100, by the remote operator terminal 20, or by both the management device 100 and the remote operator terminal 20. In other words, the past image displayed on the display device 21 includes at least one of past image recorded by the management device 100 and past image recorded by the remote operator terminal 20. For example, among the past images displayed on the display device 21, past image before the remote operator O is assigned to remotely assist the vehicle 2 may be acquired by the management device 100, and past image after the remote operator O is assigned to remotely assist the vehicle 2 may be acquired by the remote operator terminal 20.

[0020] The past video displayed on the display device 21 is played back, for example, in a time shorter than the total recording time. Playback in a time shorter than the total recording time is called fast-forward playback. The remote operator terminal 20 is configured to fast-forward play back the past video displayed on the display device 21. Note that in fast-forward playback of the past video, the past video may be partially played back in normal mode, or partially played back in slow motion. Details of fast-forward playback of the past video will be described later.

[0021] Consider the criteria for determining whether it is necessary to display the past video on the display device 21. For example, the remote assistance system 1 may determine that it is necessary to display the past video on the display device 21 when it receives a remote assistance request from the vehicle 2. As another example, the remote assistance system 1 may determine that it is necessary to display the past video on the display device 21 when it receives a request from a remote operator O who provides remote assistance to the vehicle 2. As another example, the remote assistance system 1 may determine that it is necessary to display the past video on the display device 21 depending on the scene when the remote assistance request from the vehicle 2 is received. The scene that requires the past video to be displayed on the display device 21 may be determined by, for example, AI. The determination of whether it is necessary to display the past video on the display device 21 may be made by the management device 100, the vehicle 2, or the remote operator terminal 20. Specific examples of scenes that require the past video to be displayed on the display device 21 will be described later.

[0022] The remote assistance system 1 includes one or more processors 110 (hereinafter simply referred to as processors 110) and one or more storage devices 120 (hereinafter simply referred to as storage devices 120). The processor 110 may include a processor 110A in the management device 100 and a processor 110B in the remote operator terminal 20. The storage device 120 may include a storage device 120A in the management device 100 and a storage device 120B in the remote operator terminal 20.

[0023] The processor 110 executes various processes. An example of the processor 110 is a CPU (Central Processing Unit). The storage device 120 stores various information required for processing by the processor 110 (e.g., images acquired by the camera 4, images acquired by the infrastructure camera 5, and past images). Examples of the storage device 120 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), and SSD (Solid State Drive). The processor 110, which executes a remote assistance program, and the storage device 120 may cooperate to realize the functions of the management device 100 and the remote operator terminal 20. The remote assistance program is stored in the storage device 120. Alternatively, the remote assistance program may be recorded on a computer-readable recording medium. The remote assistance program may be provided via a network.

[0024] The processor 110 (remote support system 1) executes a process for displaying past video for remote support. An overview of the process for displaying past video is as follows. When it is necessary to display past video on the display device 21, the processor 110 shortens the first past video recorded from a first timing in the past to the current time to a second period that is shorter than the first period from the first timing in the past to the current time, and displays the shortened first past video on the display device 21. After the processor 110 finishes displaying the first past video corresponding to the second period, it switches the display on the display device 21 to real-time video. This allows the remote operator O to continuously check the surrounding conditions of the vehicle 2 from the past, enabling appropriate remote support for the vehicle.

[0025] 2. Examples of scenes that require past footage to be displayed FIG. 2 is an explanatory diagram showing an example of a scene requiring the display of past video. For example, consider a scene in which vehicle 2, stopped at a stop line SL at an intersection without traffic lights (e.g., a 4-way stop), requests remote assistance such as information on the timing of departure, as shown at time A in FIG. 2. A "4-way stop" refers to an intersection where all vehicles in the four directions are required to stop, stop at the stop line SL of the intersection, and then depart in the order in which they arrive at the intersection. At the time when remote operator O is assigned, another vehicle 3 is stopped at the stop line SL of the intersecting lane that intersects with the vehicle's own lane. If only video after remote operator O is assigned were displayed on display device 21, remote operator O would not be able to recognize whether vehicle 2 or another vehicle 3 stopped at the stop line SL of the intersection first.

[0026] Therefore, after the remote operator O is assigned to the vehicle 2 in the scene, the remote assistance system 1 (remote operator terminal 20) displays the past video on the display device 21. This allows the remote operator O to recognize which of the vehicle 2 or the other vehicle 3 first stopped temporarily at the stop line SL at the intersection. Furthermore, the remote operator terminal 20 fast-forwards the past video displayed on the display device 21, and switches to real-time video when the display of the past video ends. This allows for smooth remote assistance of the vehicle 2. Note that a playback position (e.g., a seek bar) may be displayed in the past video so that the remote operator O can check the remaining playback time of the past video, etc.

[0027] 3. Example of fast forward playback of past footage Overview FIG. 3 is a diagram for explaining an overview of fast-forward playback of past video. Fast-forward playback of past video means shortening the display period of the past video. The display period of the past video indicates the period of past video recorded from a past timing (also referred to as a first past timing) to the current time. The period from the first past timing to the current time is also referred to as a first period, and the video recorded from the first past timing to the current time is also referred to as a first past video. For example, the first past timing may be the time when a remote assistance request is received from the vehicle 2, or may be a time determined depending on a scene that requires the past video to be displayed on the display device 21, or may be a time specified by the remote operator O who provides remote assistance to the vehicle 2.

[0028] Now, consider the current time used to represent the display period of the first past video. The current time when the first past video is displayed on the display device 21 changes from moment to moment. In other words, the display period of the first past video becomes longer as time passes. If the display period of the first past video becomes longer, the start of remote assistance for the vehicle 2 will be delayed. In order to shorten the display period of the first past video, the remote assistance system 1 shortens the first past video corresponding to the period (first period) from a first timing in the past to the current time to a second period that is shorter than the first period from the first timing in the past to the current time, and displays it on the display device 21.

[0029] For example, as shown in Fig. 3, consider a case where the current time when it is determined that the past video needs to be displayed on the display device 21 is time T1, the time of the first timing in the past is time Tp0, and the time when the first past video catches up with the current time, i.e., the time when the first past video switches to the real-time video, is time Tn. In this case, the first period is the period from time Tp0 to time Tn. Meanwhile, the second period, which is shorter than the first period, is the period from time Tp0_A to time Tn, as shown in Fig. 3.

[0030] Here, consider the case where the sampling times of the first past video corresponding to the first period are Tp0, Tp1, Tp2, Tp3, Tp4, T1, T2, ..., Tn, and the sampling times of the second period corresponding to the first past video at the sampling times of the first period are Tp0_A, Tp1_A, Tp2_A, Tp3_A, Tp4_A, T1_A, T2_A, ..., Tn. In this case, each of the sampling intervals in the second period (the period from Tp0_A to Tp1_A, the period from Tp1_A to Tp2_A, the period from Tp2_A to Tp3_A, the period from Tp3_A to Tp4_A, the period from Tp4_A to T1_A, the period from T1_A to T2_A, ...) is shorter than each of the sampling intervals in the first period (the period from Tp0 to Tp1, the period from Tp1 to Tp2, the period from Tp2 to Tp3, the period from Tp3 to Tp4, the period from Tp4 to T1, the period from T1 to T2, ...).

[0031] In this way, by shortening the first past video corresponding to the first period to a second period that is shorter than the first period from the first timing in the past to the current time and displaying it on the display device 21, fast-forward playback of the first past video is achieved.

[0032] 3-2.Specific examples Fig. 4 is an explanatory diagram showing a specific example of fast-forward playback of past video. As shown in Fig. 4(A), a first past video corresponding to a first period can be shortened to a second period that is shorter than the first period from a first timing in the past to the current time, for example, as shown in Fig. 4(B) or Fig. 4(C).

[0033] In the example shown in (B) of Fig. 4, a portion of frames from all frames of the first past video corresponding to the first period are extracted and connected for playback. Examples of the portion of frames include odd-numbered frames, even-numbered frames, etc.

[0034] 4C shows an example of a method for playing back the first past video corresponding to the first period at a frame rate higher than the frame rate used for shooting or recording. In this case, all frames of the first past video can be played back in fast forward.

[0035] 4.Effects According to this embodiment, when it is necessary to display past video on the display device 21, the first past video recorded from a first timing in the past to the current time is shortened to a second period that is shorter than the first period from the first timing in the past to the current time, and is displayed on the display device 21. After the display of the first past video corresponding to the second period is completed, the display on the display device 21 switches to real-time video. This allows the remote operator O to continuously check the surrounding conditions of the vehicle 2 from the past, enabling appropriate remote assistance to the vehicle.

[0036] 5. Variations FIG. 5 is an explanatory diagram showing a modified example of a scene requiring the display of past video. Consider a scene in which vehicle 2 is a passenger transport vehicle (e.g., a bus), vehicle 2 stopped at a bus stop requests remote assistance for the timing of departure, and it is unclear whether there is a moving object (e.g., a child or an animal) underneath vehicle 2 while it is stopped at the bus stop. For example, if there is a moving object underneath vehicle 2 before remote operator O is assigned to vehicle 2, remote operator O cannot confirm whether there is a moving object underneath vehicle 2 while it is stopped, even by looking at the information displayed on display device 21. Therefore, in this scene, after remote operator O is assigned to vehicle 2, remote assistance system 1 displays past video on display device 21.

[0037] Consider the first past timing, which is the start timing of the display period of the past video (first past video) displayed on the display device 21. The timing when the moving object goes under the vehicle 2 is after the vehicle 2 stops at a bus stop. Furthermore, when the moving object goes under the vehicle 2, the moving object approaches the vehicle 2 and makes a movement such as crouching. The first past timing is, for example, the timing when a moving object is first detected within a certain range from the vehicle 2 during a predetermined period before the timing when the remote assistance request from the vehicle 2 is accepted. The predetermined period is, for example, the period from when the vehicle 2 stops at a bus stop to when the remote assistance request is accepted. However, the predetermined period is not limited to this. For example, the predetermined period may be the period from when the vehicle 2 stops at a bus stop to when the remote operator O is assigned to the vehicle 2. This makes it possible to display the first past video on the display device 21 even if the moving object goes under the vehicle 2 during the period from when the remote assistance request is accepted to when the remote operator O is assigned to the vehicle 2.

[0038] According to a modification of the present embodiment, the remote assistance system 1 detects moving objects around the vehicle 2 based on video acquired from the camera 4 or the infrastructure camera 5. Optical flow, which detects object movement, is used as a method for detecting moving objects. When a moving object is detected for the first time within a certain range from the vehicle 2 during a predetermined period prior to receiving a remote assistance request from the vehicle 2, the remote assistance system 1 determines that past video needs to be displayed on the display device 21. The remote assistance system 1 (remote operator terminal 20) displays past video on the display device 21 after the remote operator O is assigned to the vehicle 2 and before the vehicle 2 starts moving. This allows the remote operator O to check whether there is a moving object underneath the parked vehicle 2 by viewing the past video displayed on the display device 21. Furthermore, the remote operator terminal 20 fast-forward plays back the past video displayed on the display device 21 and switches to real-time video when the display of the past video ends. This allows for smooth remote assistance of the vehicle 2.

[0039] FIG. 6 is an explanatory diagram showing a modified example of fast-forward playback of past footage (first past footage). The remote assistance system 1 may perform normal playback of part of the first past footage depending on the detection state of the moving object. For example, the state of the moving object when it is detected may be a state in which the moving object slips under the vehicle 2, a state in which the moving object emerges from under the vehicle 2, etc. Since the period during which the moving object is detected is expected to be short, the first past footage may be played back normally without fast-forward playback while the moving object is being detected.

[0040] Therefore, the remote assistance system 1 may display a portion of the first past video corresponding to the first period on the display device 21 for the same period as the first period, and may shorten the remaining first past video corresponding to the first period to a period shorter than the first period and display it on the display device 21. In this case, the first past video corresponding to a portion of the second period is played back in normal playback, and the first past video corresponding to the remaining period of the second period is played back in fast forward playback. This allows the remote operator O to check without missing whether a moving object has slipped under the stopped vehicle 2. [Explanation of symbols]

[0041] 1... remote support system, 2... vehicle, 3... other vehicle, 4... camera, 5... infrastructure camera, 20... remote operator terminal, 21... display device, 100... management device, 110, 110A, 110B... processor, 120, 120A, 120B... storage device, O... remote operator, SL... stop line

Claims

1. A remote assistance system for remotely assisting a vehicle, comprising: one or more processors configured to display images acquired from the vehicle-mounted camera or infrastructure camera on a display device of a remote operator terminal; the one or more processors: When it is necessary to display a past video on the display device, a first past video recorded from a first timing in the past to a current time is shortened to a second period shorter than a first period from the first timing in the past to the current time, and the shortened period is displayed on the display device; After the display of the first past video corresponding to the second period has ended, the display of the display device is switched to a real-time video. It was configured as Remote support system.

2. The remote assistance system according to claim 1, The one or more processors are configured to fast-forward play the first past video. Remote support system.

3. The remote assistance system according to claim 1, The first timing is a timing at which a remote assistance request is received from the vehicle. Remote support system.

4. The remote assistance system according to claim 1, the one or more processors detect moving objects around the vehicle based on the video; The first timing is a timing at which the moving object is first detected within a certain range from the vehicle during a predetermined period before a timing at which a remote assistance request from the vehicle is received. Remote support system.

5. The remote assistance system according to claim 1, The first timing is designated by an operator who performs the remote assistance for the vehicle. Remote support system.

Images