Remote support system, remote support method, and remote support program

Abstract

To provide a remote support system or the like capable of restricting an error from occurring in a remote support.SOLUTION: A processor of a remote support system receives through a wireless communication line, an image data set to be transmitted from an autonomous travel device for each transmission period in response to photographing of an environment and also a recognition data set to be transmitted from the autonomous travel device for each transmission period by a data capacity smaller than that of the image data set while including at least position information related to a target recognized in the environment. When a reception delay level of the image data set is out of a permission range with respect to this reception period, the processor allows an image data set received in this reception period to be replaced with a processed data set which is generated from the image data set received in the reception period before the preceding period by processing to allow a target image expressing the target to appear at a position predicted in correlation with the recognition data set which is received in this reception period, and then, displays the processed data set in a remote operator.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a remote assistance technology that remotely assists the driving of an autonomous driving device. [Background technology]

[0002] Patent Document 1 discloses an autonomous vehicle and a remote monitoring center that communicates with the autonomous vehicle via a network. The autonomous vehicle transmits camera images of the surroundings of the vehicle captured by a camera to the remote monitoring center. When the autonomous vehicle automatically stops, the remote monitoring center determines whether or not the autonomous vehicle may resume traveling based on the received camera images. When the remote monitoring center determines that the autonomous vehicle may resume traveling, it transmits a start signal to the autonomous vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-87015 A Summary of the Invention [Problem to be solved by the invention]

[0004] As in Patent Document 1, when remotely supporting an autonomous driving device, it is necessary to receive an image data set related to the outside world transmitted from the autonomous driving device. However, since the image data set has a relatively large data capacity, a delay in receiving the image data set may occur due to a communication delay. When remote support is provided based on the image data set, the resulting reception delay may prevent the remote operator from accurately grasping the situation of the outside world, which may result in an error in the remote support.

[0005] An object of the present disclosure is to provide a remote support system capable of suppressing the occurrence of errors in remote support. Another object of the present disclosure is to provide a remote support method capable of suppressing the occurrence of errors in remote support. Yet another object of the present disclosure is to provide a remote support program capable of suppressing the occurrence of errors in remote support. [Means for solving the problem]

[0006] The technical means of the present disclosure for solving the problems will be described below. Note that the claims and the reference characters in parentheses in this section indicate the corresponding relationship with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0007] A first aspect of the present disclosure is a remote assistance system that remotely assists an autonomous vehicle (10) that has a processor (6) and is equipped with an external camera (13) that captures an external environment, the remote assistance system comprising: The processor receiving, via a wireless communication line, an image data set (Di) transmitted from a moving object in each transmission period in response to an image of the outside world, and a recognition data set (Dr) transmitted from the moving object in each transmission period, the recognition data set (Dr) including at least position information on a target (T) recognized from the outside world and having a data capacity smaller than that of the image data set; When a reception delay level of the image data set for the current reception cycle expected according to the transmission cycle falls outside an allowable range, the image data set received in the current reception cycle is replaced with a processed data set (Dp) generated from the image data set received in the previous or previous reception cycle by processing the image data set so that a target image (Ie) representing the target is reflected at a position predicted in correlation with the recognition data set received in the current reception cycle, and the processed data set is displayed to a remote operator (8); The apparatus is configured to execute the following steps:

[0008] A second aspect of the present disclosure is a remote assistance method executed by a processor (6) to remotely assist a moving body (10) equipped with an external camera (13) that captures an external environment, the method comprising: receiving, via a wireless communication line, an image data set (Di) transmitted from a moving object in each transmission period in response to an image of the outside world, and a recognition data set (Dr) transmitted from the moving object in each transmission period, the recognition data set (Dr) including at least position information on a target (T) recognized from the outside world and having a data capacity smaller than that of the image data set; When a reception delay level of the image data set for the current reception cycle expected according to the transmission cycle falls outside an allowable range, the image data set received in the current reception cycle is replaced with a processed data set (Dp) generated from the image data set received in the previous or previous reception cycle by processing the image data set so that a target image (Ie) representing the target is reflected at a position predicted in correlation with the recognition data set received in the current reception cycle, and the processed data set is displayed to a remote operator (8); Includes.

[0009] A third aspect of the present disclosure is a remote assistance program that is stored in a storage medium (7) for remotely assisting automatic driving of a moving body (10) equipped with an external camera (13) that captures the external world, and includes instructions to be executed by a processor (6), The command is, receiving, via a wireless communication line, an image data set (Di) transmitted from the moving body in each transmission period in response to an image of the outside world, and a recognition data set (Dr) transmitted from the moving body in each transmission period, the recognition data set (Dr) including at least position information on a target (T) recognized from the outside world and having a data capacity smaller than that of the image data set; when a reception delay level of the image data set for the current reception cycle expected according to the transmission cycle falls outside an allowable range, the image data set received in the current reception cycle is replaced with a processed data set (Dp) generated from the image data set received in the previous or previous reception cycle by processing the image data set so that a target image (Ie) representing the target is reflected at a position predicted in correlation with the recognition data set received in the current reception cycle, and the processed data set is displayed to a remote operator (8); Includes.

[0010] According to the first to third aspects, when the reception delay level is outside the allowable range, the remote operator can provide remote assistance in accordance with the processed data set generated from the previous or previous image data set instead of the current image data set. Here, the processed data set is a data set generated from the image data set received in the previous or previous reception cycle by processing such that a target image representing a target is reflected at a position predicted in correlation with the recognition data set received in the current reception cycle. Since the recognition data set has a relatively small data capacity and the reception delay can be suppressed compared to the image data set, the remote operator can more accurately grasp the situation of the outside world by the processed data set generated under the influence of a relatively small reception delay. Therefore, errors in remote assistance caused by the reception delay can be suppressed. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an overall configuration of a first embodiment. [Diagram 2] 1 is a block diagram showing a functional configuration of a vehicle control system according to a first embodiment. [Diagram 3] FIG. 2 is a block diagram showing a functional configuration of a control flow in a vehicle according to the first embodiment. [Figure 4] 1 is a block diagram showing a functional configuration of a remote support system according to a first embodiment. [Diagram 5] 1 is a flowchart showing a remote support flow according to the first embodiment. [Figure 6] 3A to 3C are diagrams illustrating an example of a transmission sequence and a reception sequence according to the first embodiment. [Figure 7] FIG. 11 is a diagram for explaining the relationship between the reception delay level of an image data set and an adopted data set according to the first embodiment. [Figure 8] FIG. 11 is a schematic diagram for explaining a process for generating a processed data set. [Figure 9] FIG. 13 is a diagram illustrating an example of a generated processed data set. [Figure 10] FIG. 13 is a diagram showing another example of a generated processed data set. [Figure 11] 13 is a flowchart showing a remote support flow according to a second embodiment. [Figure 12] FIG. 11 is a diagram for explaining the relationship between the reception delay level of an image data set and an adopted data set according to the second embodiment. [Figure 13] 13 is a flowchart showing a remote support flow according to a third embodiment. [Figure 14] FIG. 13 is a diagram for explaining the relationship between the reception delay level of an image data set and an adopted data set according to the third embodiment. [Figure 15] FIG. 13 is a schematic diagram for explaining a process for generating a processed data set in the fourth embodiment. [Figure 16] 13 is a flowchart showing a remote support flow according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. In addition, by assigning the same reference numerals to corresponding components in each embodiment, duplicated descriptions may be omitted. In addition, when only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other parts of the configuration. Furthermore, in addition to the combination of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments can be partially combined together even if not explicitly stated, as long as there is no particular problem with the combination.

[0013] First embodiment 1 is constructed in a remote center 1, which is an external facility of a vehicle 10, which is an autonomously driven mobile body. The remote assistance system 2 is configured to be operable by a plurality of operators, and remotely assists the autonomous driving of the vehicle 10.

[0014] The vehicle 10 remotely assisted by the remote assistance system 2 is a moving body, such as an automobile, capable of traveling on a road with an occupant on board. The vehicle 10 is provided with an autonomous driving mode, the level of which is determined according to the degree of manual intervention by the occupant in the dynamic driving task (DDT). The autonomous driving mode may be realized by autonomous driving control, such as conditional driving automation, advanced driving automation, or full driving automation, in which a system performs all dynamic driving tasks when activated. The autonomous driving mode may be realized by advanced driving assistance control, such as driving assistance or partial driving automation, in which an occupant performs some dynamic driving tasks. The autonomous driving mode may be realized by either one of the autonomous driving control and the advanced driving assistance control, a combination of the two, or a switch between them.

[0015] The vehicle 10 is equipped with a communication system 11, an external sensor 12, an internal sensor 15, and a map database 16 shown in FIG. 1. The communication system 11 acquires communication information usable by the vehicle 10 through wireless communication. The communication system 11 may be of a V2X type that transmits and receives communication signals between the vehicle 10 and a V2X system that exists in the external world of the vehicle 10. The V2X type communication system 11 is at least one of, for example, a Dedicated Short Range Communications (DSRC) communication device and a Cellular V2X (C-V2X) communication device. The V2X type communication system 11 enables the vehicle 10 to communicate wirelessly with the remote assistance system 2.

[0016] The communication system 11 may be of a positioning type that receives a positioning signal from an artificial satellite of a Global Navigation Satellite System (GNSS) that exists in the outside world of the vehicle 10. The external sensor 12 of the positioning type is, for example, a GNSS receiver.

[0017] The external sensor 12 acquires external information as sensor information from the external environment surrounding the vehicle 10. The external sensor 12 is a target detection type that detects targets present in the external world of the vehicle 10. The target detection type external sensor 12 includes, for example, an external camera 13 that captures an image of the external world. The external camera 13 outputs a captured image to the control system 17 at each capture cycle. The external sensor 12 may include a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) 14 that can detect reflected light of irradiated laser light as point cloud data. The external sensor 12 may include at least one of radar, sonar, and the like.

[0018] The internal sensor 15 acquires internal information as sensor information from the internal world, which is the internal environment of the vehicle 10. The internal sensor 15 may be a physical quantity detection type that detects a specific physical quantity of motion in the internal world of the vehicle 10. The physical quantity detection type internal sensor 15 is at least one of a driving speed sensor, an acceleration sensor, a gyro sensor, etc. The internal sensor 15 may be an occupant detection type that detects a specific state of an occupant in the internal world of the vehicle 10. The occupant detection type internal sensor 15 is at least one of a driver status monitor (registered trademark), a biological sensor, a seating sensor, an actuator sensor, an in-vehicle equipment sensor, etc.

[0019] The map database 16 stores map information that can be used by the control system 17. The map database 16 includes at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium. The map database 16 may be a database of a locator that estimates the vehicle's own state quantity including the vehicle's own position. The map database 16 may be a database of a navigation unit that navigates the travel route of the vehicle 10. The map database 16 may be a combination of multiple types of these databases.

[0020] The map database 16 acquires and stores the latest map information by communication with an external center via, for example, a V2X type communication system 11. Here, the map information is digitized in two dimensions or three dimensions as information representing the driving environment of the vehicle 10. In particular, as the three-dimensional map data, digital data of a high-precision map is preferably adopted. The map information may include road information representing at least one of the position, shape, and road surface condition of the road itself. The map information may include marking information representing at least one of the positions and shapes of signs and dividing lines attached to the road. The map information may include structure information representing at least one of the positions and shapes of buildings and traffic lights facing the road.

[0021] The control system 17 is connected to the communication system 11, the external sensor 12, and the map database 16 via at least one of, for example, a LAN (Local Area Network) line, a wire harness, an internal bus, and a wireless communication line. The control system 17 is configured to include at least one dedicated computer.

[0022] The dedicated computer constituting the control system 17 may be an integrated ECU (Electronic Control Unit) that integrates the driving control of the vehicle 10. The dedicated computer constituting the control system 17 may be a judgment ECU that judges a driving task in the driving control of the vehicle 10. The dedicated computer constituting the control system 17 may be a monitoring ECU that monitors the driving control of the vehicle 10. The dedicated computer constituting the control system 17 may be an evaluation ECU that evaluates the driving control of the vehicle 10.

[0023] The dedicated computer constituting the control system 17 may be a navigation ECU that navigates the driving route of the vehicle 10. The dedicated computer constituting the control system 17 may be a locator ECU that estimates the self-state quantity of the vehicle 10. The dedicated computer constituting the control system 17 may be an actuator ECU that controls the driving actuator of the vehicle 10. The dedicated computer constituting the control system 17 may be an HCU (Human Machine Interface (HMI) Control Unit) that controls the presentation of information in the vehicle 10. The dedicated computer constituting the control system 17 may be a computer other than the vehicle 10 that constitutes, for example, an external center or a mobile terminal capable of communicating with the vehicle 10.

[0024] The dedicated computer constituting the control system 17 has at least one memory 18 and one processor 19. The memory 18 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs and data. Here, the term "storage" may refer to accumulation in which data is retained even when the vehicle 10 is turned off, or temporary storage in which data is erased when the vehicle 10 is turned off. The processor 19 includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), a reduced instruction set computer (RISC)-CPU, a complex instruction set computer (CISC)-CPU, a data flow processor (DFP), or a graph streaming processor (GSP).

[0025] In the control system 17, the processor 19 plans a dynamic driving task for the future route, future trajectory, and future action of the vehicle 10 based on sensor information, etc. The control system 17 may execute judgment and planning using a vehicle driving model, such as a simulation model or a machine learning model. At this time, in an unreasonable situation where it is determined that an unreasonable risk exists, the control system 17 generates an assistance request for requesting assistance from the remote operator 8 of the remote center 1, and transmits the assistance request to the remote center 1 via the communication system 11.

[0026] Furthermore, in the control system 17, the processor 19 executes a plurality of instructions included in a control program stored in the memory 18 in order to provide information usable for remote assistance to the remote assistance system 2. In this way, the control system 17 constructs a plurality of functional blocks for providing information usable for remote assistance of the vehicle 10. The plurality of functional blocks constructed in the control system 17 include a monitoring block 20, a recognition block 21, and a communication control block 22 as shown in FIG.

[0027] A control method for providing information usable for remote assistance of the vehicle 10 by cooperation of these blocks 20, 21, and 22 is executed according to the control flow shown in Fig. 3. This control flow is executed repeatedly, for example, while the vehicle 10 is running. Alternatively, this control flow may be executed repeatedly while assistance is requested. Note that each "S" in this control flow represents multiple steps executed by multiple commands included in a control program.

[0028] First, in S10, the monitoring block 20 acquires a captured image from the external camera 13. Next, in S20, the monitoring block 20 generates additional information related to the captured image. The additional information is added to each captured image and is information for identifying each captured image. The additional information includes, for example, at least an order ID indicating the chronological order of the captured images and a timestamp indicating the capture time.

[0029] Then, in S30, the communication control block 22 transmits the captured image with the additional information added thereto as an image data set Di including at least the captured image to the remote support system 2 via the communication system 11. At this time, the communication control block 22 transmits the image data set Di through a first band, which is a wireless communication band having a higher throughput and bandwidth than a second band described below.

[0030] Next, in S40, the recognition block 21 executes image recognition processing on the image dataset Di. As a result, the recognition block 21 generates a recognition dataset Dr including at least a recognition result related to a target T in the outside world reflected in the image dataset Di. The target T includes at least an obstacle that may be an obstacle to the traveling of the vehicle 10. The target T includes at least one type of dynamic object such as another vehicle, a pedestrian, an animal, etc., and a stationary object such as an object fallen on the road or an object installed near the road.

[0031] The recognition data set Dr acquired by the recognition block 21 includes at least position information of the target T within the image frame, i.e., position information in the image coordinate system. The recognition data set Dr may include an identification ID for identifying the target T and type information indicating the type of the target T. The recognition data set Dr may include size information and orientation information of the target T within the image frame. The recognition data set Dr may include position information of the target T in an external coordinate system such as a geographic coordinate system. The recognition data set Dr may include reliability information on the recognition result.

[0032] The recognition block 21 is configured to be capable of generating a recognition dataset Dr as a dataset having a smaller data capacity than the image dataset Di. For example, the recognition block 21 generates the recognition dataset Dr by forming the various information described above into a data type such as an associative array type. In particular, the recognition block 21 is capable of generating a recognition dataset Dr having a smaller data capacity and capable of being transmitted at high speed by serializing the various information in a relatively lightweight format such as JSON, Protocol Buffers, or MessagePack.

[0033] Then, in S50, the communication control block 22 transmits the recognition dataset Dr to the remote support system 2 via the communication system 11. The communication control block 22 may transmit the recognition dataset Dr according to a communication protocol different from that used for transmitting the image dataset Di. For example, the communication control block 22 transmits the recognition dataset Dr according to a communication protocol applied to communication with a relatively small data capacity, such as MQTT. At this time, the communication control block 22 transmits the image dataset Di through a second band having a higher throughput and bandwidth than the first band for transmitting the image dataset Di.

[0034] According to such a control flow in the vehicle 10, the image dataset Di and the recognition dataset Dr are transmitted from the vehicle 10 at each transmission period as shown in Fig. 6. The image dataset Di has a larger data capacity than the recognition dataset Dr. Therefore, as shown in Fig. 6, the transmission of the recognition dataset Dr, which started after the start of the transmission of the image dataset Di, is likely to be completed by the time the transmission of the image dataset Di is completed.

[0035] The remote assistance system 2, which receives each data set Di, Dr from the vehicle 10, includes a communication system 3, a user interface 4, and a remote assistance device 5. The communication system 3 acquires communication information transmitted from the vehicle 10 via wireless communication.

[0036] The user interface 4 is a terminal device used by a remote operator 8 as a user. For example, the user interface 4 is a computer to which a display device such as a display and input devices such as a mouse and a keyboard are connected. The user interface 4 presents information on the vehicle 10 to be assisted to the remote operator 8. For example, the user interface 4 presents an image data set Di or a processed data set Dp periodically delivered from a remote assistance device 5 described later as video data showing the outside world of the vehicle 10. The user interface 4 also accepts instruction inputs for the vehicle 10 to be assisted from the remote operator 8. For example, the user interface 4 accepts instruction inputs such as whether or not a dynamic driving task needs to be performed in the vehicle 10, and the timing of performing the dynamic driving task.

[0037] The remote support device 5 is connected to the communication system 3 and the user interface 4 via at least one of a LAN (Local Area Network) line, a wire harness, an internal bus, and a wireless communication line, for example. The remote support device 5 is configured to include at least one dedicated computer.

[0038] The dedicated computer constituting the remote support device 5 has at least one memory 7 and one processor 6. The memory 7 is at least one type of non-transient substantial storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs and data. Here, storage may be accumulation in which data is retained even when the remote support device 5 is turned off, or temporary storage in which data is erased when the remote support device 5 is turned off. The processor 6 includes at least one type of core, such as a CPU, a GPU, a RISC-CPU, a CISC-CPU, a DFP, or a GSP.

[0039] In the remote assistance device 5, the processor 6 executes a plurality of instructions included in a remote assistance program stored in the memory 7 in order to remotely assist the driving of the vehicle 10. In this way, the remote assistance device 5 constructs a plurality of functional blocks for remotely assisting the driving of the vehicle 10. The plurality of functional blocks constructed in the remote assistance system 2 include a receiving block 110, an image processing block 120, and a display block 130, as shown in FIG.

[0040] A remote assistance method in which the remote assistance system 2 remotely assists the autonomous driving of the vehicle 10 by cooperation of these blocks 110, 120, and 130 is executed according to the remote assistance flow shown in Fig. 5. This remote assistance flow is executed repeatedly while the remote assistance device 5 is running. Note that each "S" in this remote assistance flow means a plurality of steps executed by a plurality of commands included in the remote assistance program.

[0041] First, in S100, the reception block 110 waits to receive transmission data of the image data set Di and the recognition data set Dr from the vehicle 10 in the current reception cycle. The reception cycle is assumed according to the transmission cycle of the image data set Di and the recognition data set Dr. The reception cycle is assumed to be substantially the same period as the transmission cycle, for example, as shown in Fig. 6. When reception of each data set Di, Dr starts, reception processing of each data set Di, Dr is executed in S110 and S120.

[0042] Specifically, in S110, when the reception block 110 starts receiving the transmission data of the recognition data set Dr, it starts executing the reception process of the recognition data set Dr. In S120, when the reception block 110 starts receiving the transmission data of the recognition data set Dr, it starts executing the reception process of the recognition data set Dr. In the reception process, the reception block 110 acquires the transmission data such as packets, and restores the data sets Dr and Di from the transmission data.

[0043] After the reception process starts, the flow proceeds to S130. In S130, the reception block 110 determines whether or not the reception delay level of the current image data set Di falls outside the allowable range Lt.

[0044] Here, the reception delay level is a parameter defined in correlation with the elapsed time from the start point Ps of the current reception cycle to the completion of reception of the image data set Di, i.e., the reception delay time. The reception delay level may be the elapsed time itself or a value related to the elapsed time. In this embodiment, the longer the elapsed time, the higher the reception delay level.

[0045] The allowable range Lt is, for example, a range in which the reception delay level is equal to or less than a threshold level. In other words, the allowable range Lt is a range in which the elapsed time is equal to or less than a threshold time. As an example, the allowable range Lt is defined as a range correlating with the average value μ and standard deviation σ of the reception delay time. Specifically, as shown in FIG. 7, the allowable range Lt is a range in which the reception delay time is equal to or less than μ+1σ. Here, the average value μ of the reception delay time is calculated in advance, for example, for the entire operating range in which the vehicle 10 operates or for each support area obtained by dividing the operating range. The reception block 110 may acquire the elapsed time every period and update the average value μ as appropriate.

[0046] If it is determined that the reception delay level is outside the allowable range Lt, the flow proceeds to S140. In S140, the image processing block 120 processes the image data set Di received in the previous reception cycle to generate a processed data set Dp. Specifically, the image processing block 120 processes the previous image data set Di so that a target image Ie representing the target T appears at a position predicted in correlation with the recognition data set Dr received in the current reception cycle. In the following, the previous image data set Di may be described with the symbol of image data set Di_p in order to distinguish it from the current image data set Di in particular. Also, the previous recognition data set Dr may be distinguished by the symbol of recognition data set Dr_p, and the current recognition data set Dr may be distinguished by the symbol of recognition data set Dr_c.

[0047] To explain the processing in detail, the image processing block 120 first performs image recognition processing on the image data set Di_p to identify the area in which the target T appears. The image processing block 120 recognizes the target T from the image data set Di_p by a method such as instance segmentation, and identifies the area. The image processing block 120 then generates a target image Ie in which only the target T is extracted from the image data set Di_p. In addition, the image processing block 120 generates a target-removed image Ir in which the target T has been removed from the image data set Di_p. The image processing block 120 generates the target-removed image Ir by performing image processing to remove the target T and then interpolate the removed portion by a method such as image inpainting, to generate the target-removed image Ir.

[0048] Then, for the target T extracted as the target image Ie, the image processing block 120 specifies the processing mode of the target image Ie in the target-removed image Ir predicted in correlation with the recognition data set Dr_c. Specifically, the image processing block 120 specifies the image position, image size, and orientation of the target image Ie in the target-removed image Ir according to the position information, image size information, and orientation information within the image frame in the recognition data set Dr_c. The image processing block 120 generates a processed data set Dp by combining the target image Ie, which has been processed in the specified processing mode, with the target-removed image Ir.

[0049] For example, suppose that another vehicle as a target T shown in the image dataset Di_p shown in Fig. 8 travels straight in the oncoming lane of the stopped vehicle 10 and approaches the vehicle 10. The image processing block 120 composites the target image Ie of the other vehicle with the target-removed image Ir at a position corresponding to the position in the image frame in the recognition dataset Dr_c, enlarged to a size according to the recognition dataset Dr_c (see Fig. 9).

[0050] Furthermore, when a target T not included in the recognition data set Dr_c is recognized in the image recognition process executed on the image data set Di_p, the image processing block 120 performs highlighting processing on the corresponding target image Ie. A target T not included in the recognition data set Dr can also be called a target T not recognized by the vehicle 10. For example, when information on a motor-driven bicycle among the targets T shown in FIG. 9 is not included in the recognition data set Dr_c, the image processing block 120 performs a shading process on the corresponding target image Ie and synthesizes it. Alternatively, the image processing block 120 may perform processing such as surrounding the corresponding target image Ie with a frame.

[0051] The image position, image size, and orientation of the target image Ie unrecognized on the vehicle 10 side may be substantially the same as those in the image data set Di_p, for example. Alternatively, when a target T unrecognized on the vehicle 10 side has been continuously recognized by the image processing block 120 since the time before last, the image processing block 120 may estimate the image position, image size, and orientation of the target image Ie by tracking processing.

[0052] In addition, when a target T included in the recognition data set Dr_c is not recognized in the image recognition process executed on the image data set Di_p, the image processing block 120 performs processing to add a provisional target image It. A target T included in the recognition data set Dr_c but not recognized by the image processing block 120 can also be called a target T erroneously recognized on the vehicle 10 side. For example, as shown in Fig. 9, the image processing block 120 synthesizes an arbitrary figure-shaped image as a provisional target image It at a position in the target-removed image Ir correlated with the recognition data set Dr_c.

[0053] Furthermore, the image processing block 120 may process the target-removed image Ir in addition to the target image Ie. Specifically, when the vehicle 10 is moving, the image processing block 120 processes the target-removed image Ir in a processing mode that correlates with the amount of movement and the direction of movement of the vehicle 10. For example, when the vehicle 10 is moving straight on a straight road, the image processing block 120 enlarges the target-removed image Ir to an image size that correlates with the amount of movement (see FIG. 10). In FIG. 10, the portion shown by dot hatching is a portion that is outside the range of the image frame that is actually displayed due to the enlargement. The image processing block 120 may determine the processing mode of the target image Ie in the processed target-removed image Ir in correlation with, for example, the position of the spatial coordinates (for example, latitude and longitude coordinates) in the outside world in the recognition data set Dr. In addition, the amount of movement and the direction of movement of the vehicle 10 may be sequentially acquired from the vehicle 10 via the communication system 3, for example.

[0054] In the following S150, the display block 130 displays the generated processed data set Dp on the user interface 4. That is, the display block 130 replaces the image data set Di that would have been displayed in S160 described below if the reception delay level was within the allowable range Lt with the processed data set Dp and displays it. Specifically, the display block 130 outputs the processed data set Dp to the user interface 4 of the remote operator 8 in charge of the vehicle 10. As a result, the display block 130 displays the processed data set Dp to the remote operator 8 via the user interface 4.

[0055] On the other hand, if it is determined in S130 that the reception delay level of the image data set Di is within the allowable range Lt, the flow proceeds to S160. In S160, the display block 130 outputs the current image data set Di that has been completely received as is to the user interface 4 of the remote operator 8 in charge of the vehicle 10. As a result, the display block 130 displays the image data set Di to the remote operator 8 via the user interface 4.

[0056] According to the first embodiment described above, when the reception delay level falls outside the allowable range Lt, the remote operator 8 can provide remote assistance according to the processed data set Dp generated from the image data set Di_p instead of the current image data set Di. Here, the processed data set Dp is a data set generated from the image data set Di_p received in the previous or previous reception cycle. Furthermore, the processed data set Dp is a data set generated by processing so that the target image Ie representing the target T appears at a position predicted in correlation with the recognition data set Dr_c received in the current reception cycle. Since the recognition data set Dr has a relatively small data capacity and the reception delay can be suppressed compared to the image data set Di, the remote operator 8 can more accurately grasp the situation of the outside world by the processed data set Dp generated under the influence of a relatively small reception delay. Therefore, errors in remote assistance caused by the reception delay can be suppressed.

[0057] Furthermore, according to the first embodiment, when the reception delay level falls on the long side outside the allowable range Lt, the image data set Di received in the current reception cycle is replaced with the processed data set Dp and displayed. The reception delay level is defined in correlation with the elapsed time from the start point Ps of the current reception cycle to the completion of reception of the image data set Di. Therefore, it is possible to avoid support errors caused by an image data set Di whose reception delay is large enough to fall outside the allowable range Lt. In particular, in the first embodiment, an image data set Di whose reception delay is significantly delayed compared to the average reception delay can be replaced with the processed data set Dp. Therefore, since the reception delay can fall within the average range, the variation in reception delay can be suppressed.

[0058] Furthermore, according to the first embodiment, when the reception delay level falls within the permissible range Lt, the replacement with the processed data set Dp is stopped, and the image data set Di received in the current reception cycle is displayed to the remote operator 8. Therefore, when the reception delay is relatively small, it may be easier for the remote operator 8 to grasp the situation of the outside world based on the image data set Di, which has a relatively good image quality compared to the processed data set Dp.

[0059] In addition, according to the first embodiment, the recognition data set Dr includes posture information regarding the recognized target T. Then, the processed data set Dp is generated and displayed so as to show a target image Ie representing the target T in a posture predicted in correlation with the recognition data set Dr received in the current reception cycle. Therefore, the remote operator 8 may be able to grasp the target T in the processed data set Dp in an appearance that is relatively close to the posture of the actual target T. This makes it easier to avoid errors in remote assistance.

[0060] Also, according to the first embodiment, the image data set Di is received through a first band of the wireless communication band, and the recognition data set Dr is received through a second band of the wireless communication band, in which at least one of the throughput and the bandwidth is smaller than the first band. Therefore, for the image data set Di having a relatively large data capacity, the reception delay itself can be suppressed by receiving it at a relatively large throughput or bandwidth. Furthermore, for the recognition data set Dr, even if it is received at a relatively small throughput or bandwidth, the reception delay is unlikely to occur because it is a data set with a relatively small data capacity, and the band is different. This makes it possible to prepare for the reception delay on the image data set Di side while further suppressing the overall delay.

[0061] Second Embodiment As shown in FIGS. 11 and 12, the second embodiment is a modification of the first embodiment.

[0062] 11, in the remote assistance flow in the second embodiment, when it is determined in S130 that the reception delay level is within the allowable range Lt, the flow proceeds to S161. In S161, the reception block 110 determines whether or not the reception delay level of the image data set Di is within the set range Ls.

[0063] The set range Ls is a range whose end point is defined on the shorter side than the end point of the allowable range Lt, as shown in Fig. 12. In other words, the set range Ls is a range in which the reception delay level is equal to or less than a threshold level that is smaller than the threshold level of the allowable range Lt. In other words, the set range Ls is a range in which the elapsed time from the start point Ps of the reception cycle is equal to or less than a threshold time that is shorter than the threshold time of the allowable range Lt.

[0064] As an example, the set range Ls is defined as a range that correlates with the average value μ and standard deviation σ of the reception delay time, similar to the allowable range Lt. Specifically, the allowable range Lt is a range in which the reception delay time is equal to or less than μ-1σ, as shown in FIG.

[0065] If it is determined that the reception delay level is within the set range Ls, the flow proceeds to S162. In S162, the display block 130 displays the image data set Di to the remote operator 8 at a display timing between the end points of the set range Ls and the allowable range Lt. That is, if the display block 130 completes reception of the image data set Di by the end point of the set range Ls, the display block 130 delays the display timing of the image data set Di to the end point. For example, the display block 130 adjusts the display timing so that the time from the start point Ps of the reception cycle to the display of the image data set Di becomes the average value μ of the reception delay time. That is, in this embodiment, the display timing is adjusted so as to fall within the range of μ±1σ from the start point Ps of the reception cycle.

[0066] According to the second embodiment described above, when the reception delay level falls within the set range Ls whose end point is defined on the shorter side than the end point of the allowable range Lt, the replacement with the processed data set Dp is stopped. Then, the image data set Di received in the current reception cycle is displayed to the remote operator 8 at a display timing between the end points of the set range Ls and the allowable range Lt. Therefore, the display intervals of the image data sets Di between reception cycles can be adjusted to be relatively uniform. Therefore, errors in remote assistance caused by variations in reception delay can be further suppressed.

[0067] Third embodiment As shown in FIGS. 13 and 14, the third embodiment is a modification of the first embodiment.

[0068] 13, in the remote assistance flow in the third embodiment, when a reception process of the recognition data set Dr is executed in S110, the flow proceeds to S111. In S111, the image processing block 120 starts generating a processed data set Dp in response to completion of reception of the recognition data set Dr. That is, in S111, as soon as reception of the recognition data set Dr is completed, generation of the processed data set Dp is started in parallel with the reception process of the image data set Di. When the generation of the processed data set Dp and the reception process of the image data set Di are executed, the flow proceeds to S130.

[0069] The allowable range Lt in S130 in this embodiment is a range defined in correlation with the elapsed time from the start point Ps of the receiving cycle until the completion of generation of the processed data set Dp based on the recognition data set Dr received in the current receiving cycle. The allowable range Lt is the elapsed time itself until the completion of generation of the processed data set Dp, for example, as shown in FIG.

[0070] That is, in this embodiment, when the reception delay level of the image dataset Di falls outside the permissible range Lt, it means that the completion of reception of the image dataset Di is later than the completion of generation of the processed dataset Dp. Therefore, in this case, in S150, the processed dataset Dp, which has been generated earlier than the completion of reception of the image dataset Di, is displayed in place of the image dataset Di. Then, when the reception delay level of the image dataset Di is within the permissible range Lt, the image dataset Di, which has been received earlier than the completion of generation of the processed dataset Dp, is displayed in S160.

[0071] According to the third embodiment described above, when the reception delay level falls on the long-term side outside the allowable range Lt, the image data set Di received in the current reception cycle is replaced with the processed data set Dp and displayed on the remote operator 8. Therefore, when the completion of generation of the processed data set Dp is earlier than the completion of reception of the image data set Di, the image data set Di can be immediately replaced with the processed data set Dp and displayed. Therefore, the display delay for the remote operator 8 can be reduced.

[0072] (Fourth embodiment) As shown in FIG. 15, the fourth embodiment is a modification of the first embodiment.

[0073] In the fourth embodiment, the recognition data set Dr transmitted from and received from the vehicle 10 includes a target image Ie obtained by extracting a portion of the image data set Di that includes the target T. The recognition block 21 of the control system 17 recognizes the target T from the image data set Di in the same manner as the image processing block 120, for example, and generates a target image Ie in which only the target T is extracted.

[0074] In particular, the recognition block 21 limits the target T for which the target image Ie is generated to a newly appeared target T that was not recognized before the previous time. The newly appeared target T may be, for example, a target T that has newly entered the angle of view of the image data set Di. The newly appeared target T may be a target T that has been present in the angle of view of the image data set Di since the previous time and has been occluded by another target T. The newly appeared target T may be a target T that has been present in the angle of view of the image data set Di since the previous time and has been given a new identification ID due to a change in appearance caused by a change in posture.

[0075] Corresponding to the above-described processing on the vehicle 10 side, in S110 of the remote assistance flow, the reception block 110 receives and processes a target image Ie relating to a newly appeared target T as one type of the recognition data set Dr.

[0076] Then, in S140, the image processing block 120 generates a processed data set Dp for the newly appearing target T by also using the target image Ie from the vehicle 10 side (see FIG. 15). For example, the image processing block 120 combines the target image Ie with the target-removed image Ir in a processing mode defined in correlation with other information of the recognition data set Dr.

[0077] According to the fourth embodiment described above, the recognition data set Dr received in the current receiving cycle includes a target image Ie obtained by extracting a portion showing the target T from the image data set Di transmitted in the current transmitting cycle. Therefore, even if the target T shown in the current image data set Di is not shown in the image data set Di from the previous image data set Di that is the source of the processing data set Dp, a processing data set Dp showing the target T can be generated. Therefore, the remote operator 8 can more accurately grasp the situation of the outside world.

[0078] Fifth embodiment As shown in FIG. 16, the fifth embodiment is a modification of the first embodiment.

[0079] As shown in Fig. 16, the remote assistance flow of the fifth embodiment proceeds to S1010 after the process of S100. In S101, the image processing block 120 executes a processing preparation process for generating a processing data set Dp before starting a receiving process of a recognition data set Dr_c. For example, the processing preparation process includes a generation process of a target image Ie and a generation process of a target-removed image Ir from the previously received image data set Di. After the process of S101, the flow proceeds to S110 and S120.

[0080] In this case, in the generation process of the processed data set Dp shown in S140a, the image processing block 120 reads out the target image Ie and the target-removed image Ir already prepared in S101. Then, the image processing block 120 executes the processing and synthesis process of each of the images Ie and Ir in a processing mode correlated with the recognition data set Dr.

[0081] The processing preparation process may be executed before or after the timing shown in FIG. 16, so long as it is executed before the determination process in S130.

[0082] (Other embodiments) Although several embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope not departing from the gist of the present disclosure.

[0083] In a modified example, the image processing block 120 may generate the processed data set Dp not from the previous image data set Di, ie, the image data set immediately before, but from the image data set Di before the previous one.

[0084] In a modified example, the allowable range Lt may be set to a larger range, such as μ+2σ from the start point Ps of the reception cycle. The allowable range Lt may be set so that switching between the image data set Di and the processed data set Dp is relatively unlikely to occur for each display cycle.

[0085] In a modified example, the tolerance range Lt may be set to a desired range by the remote operator 8. Alternatively, the tolerance range Lt may be a range in which the occurrence rate of assistance errors is statistically low.

[0086] In a modified example, at least a part of the processing executed by the remote assistance device 5 may be executed by the user interface 4.

[0087] In a modified example of the fourth embodiment, the recognition block 21 may set the target T for generating the target image Ie to a target T whose recognition reliability falls within a low reliability range that does not reach the allowable reliability range.

[0088] In a modified example, the recognition data set Dr may include recognition results of detection data by other external sensors 12, such as the LiDAR 14, instead of or in addition to the recognition results of the image data set Di.

[0089] In a modified example, the dedicated computer constituting the remote support device 5 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), CPLD (Complex Programmable Logic Device), etc. Furthermore, such a digital circuit may have a memory that stores a program.

[0090] In a modified example, the moving object supported by the remote assistance system 2 may be, for example, an autonomous robot capable of transporting luggage or collecting information. In addition to the forms described above, the above-mentioned embodiment and modified examples may be implemented as a control device having at least one processor 6 and one memory 7. For example, the remote assistance system 2 may be implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.). [Explanation of symbols]

[0091] 2: Remote assistance system, 6: Processor, 7: Memory (storage medium), 8: Remote operator, 10: Vehicle (moving object), 13: External camera, Di: Image dataset, Dr: Recognition dataset, Dp: Processing dataset, Ie: Target image, T: Target

Claims

1. A remote support system for remotely supporting the autonomous driving of an autonomously operable mobile body (10) having a processor (6) and equipped with an external camera (13) for photographing the external world, The processor, receiving, via a wireless communication line, an image data set (Di) transmitted from the moving body for each transmission period in response to an image of the outside world, and a recognition data set (Dr) transmitted from the moving body for each transmission period, the recognition data set (Dr) including at least position information on a target (T) recognized from the outside world and having a data capacity smaller than that of the image data set; when a reception delay level of the image data set for a current reception cycle assumed according to the transmission cycle falls outside an allowable range, the image data set received in the current reception cycle is replaced with a processed data set (Dp) generated from the image data set received in the previous or previous reception cycle by processing the image data set so that a target image (Ie) representing the target is reflected at a position predicted in correlation with the recognition data set received in the current reception cycle, and the processed data set is displayed to a remote operator (8); A remote assistance system configured to:

2. Replacing the image data set with the processed data set and displaying it includes:

2. The remote assistance system according to claim 1, further comprising: when the reception delay level, defined in correlation with the elapsed time from the start of the current reception cycle to the completion of reception of the image data set, falls outside the allowable range and is on the long side, replacing the image data set received in the current reception cycle with the processed data set and displaying the processed data set to the remote operator.

3. Replacing the image data set with the processed data set and displaying it includes:

3. The remote assistance system according to claim 2, further comprising: when the reception delay level falls within the allowable range, stopping the replacement with the processed data set and displaying to the remote operator the image data set received in the current reception cycle.

4. Replacing the image data set with the processed data set and displaying it includes:

4. The remote assistance system according to claim 3, further comprising: when the reception delay level falls within a set range in which an end point of the acceptable range is defined on the shorter side, the replacement with the processed data set is stopped, and the image data set received in the current reception cycle is displayed to the remote operator at a display timing between the end points of the set range and the acceptable range.

5. Replacing the image data set with the processed data set and displaying it includes:

2. The remote assistance system according to claim 1, further comprising: when the reception delay level defined in correlation with the elapsed time from the start of the current reception cycle to the completion of reception of the image data set falls on the long side outside the allowable range defined in correlation with the elapsed time from the start of the reception cycle to the completion of generation of the processed data set based on the recognition data set received in the current reception cycle, replacing the image data set received in the current reception cycle with the processed data set and displaying it to the remote operator.

6. the recognition data set includes pose information about the recognized target; Replacing the image data set with the processed data set and displaying it includes: The remote assistance system of claim 1, further comprising generating and displaying the processed data set so as to display the target image representing the target in a predicted pose correlated with the recognition data set received in the current receiving cycle.

7. The remote assistance system of claim 1 , wherein the recognition data set received in the current receiving cycle includes the target image in which a portion of the target is extracted from the image data set transmitted in the current transmitting cycle.

8. Receiving the image dataset and the recognition dataset includes:

2. The remote assistance system of claim 1, further comprising receiving the image data set through a first band among wireless communication bands, and receiving the recognition data set through a second band among the wireless communication bands, the second band having at least one of a throughput and a bandwidth smaller than the first band.

9. A remote support method executed by a processor (6) for remotely supporting the running of a moving body (10) equipped with an external camera (13) that captures the external world, comprising: receiving, via a wireless communication line, an image data set (Di) transmitted from the moving body for each transmission period in response to an image of the outside world, and a recognition data set (Dr) transmitted from the moving body for each transmission period, the recognition data set (Dr) including at least position information on a target (T) recognized from the outside world and having a data capacity smaller than that of the image data set; when a reception delay level of the image data set for a current reception cycle assumed according to the transmission cycle falls outside an allowable range, the image data set received in the current reception cycle is replaced with a processed data set (Dp) generated from the image data set received in the previous or previous reception cycle by processing the image data set so that a target image (Ie) representing the target is reflected at a position predicted in correlation with the recognition data set received in the current reception cycle, and the processed data set is displayed to a remote operator (8); A remote support method comprising:

10. A remote assistance program that is stored in a storage medium (7) for remotely assisting automatic driving of a moving body (10) equipped with an external camera (13) that captures the external world, and includes instructions to be executed by a processor (6), The instruction: receiving, via a wireless communication line, an image data set (Di) transmitted from the moving body for each transmission period in response to an image of the outside world, and a recognition data set (Dr) transmitted from the moving body for each transmission period, the recognition data set (Dr) including at least position information on a target (T) recognized from the outside world and having a data capacity smaller than that of the image data set; when a reception delay level of the image data set for a current reception cycle estimated according to the transmission cycle falls outside an allowable range, the image data set received in the current reception cycle is replaced with a processed data set (Dp) generated from the image data set received in the previous or previous reception cycle by processing the image data set so that a target image (Ie) representing the target is reflected at a position predicted in correlation with the recognition data set received in the current reception cycle, and the processed data set is displayed to a remote operator (8). Remote support programs, including:

Images