Remote support device, remote support method, and remote support program

Abstract

To prevent the position and size of a dynamic target in a composite image from largely deviating from the actual position and size, in receiving a camera image and a target recognition result separately from a movable body to perform remote support of the movable body.SOLUTION: When a target to be noted in an image IMG1 is a dynamic target, a remote support device performs projective transformation of a partial image OBJ1 of the dynamic target included in an image IMG1 used for recognition of the dynamic target. Through the projective transformation of the partial image OBJ1, the remote support device creates a partial image OBJ2 that is obtained in a camera view later than an acquisition timing T1 of the image IMG1 including the partial image OBJ1 by a target movement adjustment time α. The remote support device creates a composite image SIMG1 on the basis of the partial image OBJ2, the image IMG1 in which the partial image OBJ1 was included, and recognition information OR of the dynamic target in the image IMG1, and outputs the composite image from a display.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a technology for remotely supporting a mobile object using images acquired by a camera mounted on the mobile object. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2022-159912 discloses a system for remotely monitoring a vehicle. In this system, first and second data are separately transmitted from the vehicle to a remote server. The first data includes an image (camera image) acquired by a camera mounted on the vehicle and the timing of acquisition of the camera image. The second data includes a target recognition result based on the camera image and the timing of acquisition of the camera image. Based on the acquisition timing of the camera image included in the first and second data, the remote server extracts, in chronological order, camera images acquired at the same time and target recognition results based on the camera images. Then, a composite image in which the target recognition result is superimposed on the extracted camera image is output from a remote monitoring screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-159912 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above system, the difference between the timing of capturing the camera image that generated the target recognition result and the timing at which the target recognition result becomes available for processing by the remote server can be considered the target delay time. This problem arises when the target is a dynamic object such as a pedestrian, bicycle, or other vehicle. In other words, if the dynamic object moves significantly within the target delay time, the position and size of the dynamic object in the composite image will deviate significantly from the actual position and size of the dynamic object. This will result in a decrease in the accuracy of the information output from the remote monitoring screen.

[0005] The present disclosure has been made in view of the above-mentioned problems, and one objective of the present disclosure is to provide a technology for preventing the position and size of a dynamic target in a composite image from deviating significantly from the actual position and size when remotely supporting a moving object by separately receiving a camera image and a target recognition result from the moving object. [Means for solving the problem]

[0006] A first aspect of the present disclosure is an apparatus for remotely supporting a moving object, which has the following features. The device remotely supports the mobile body by outputting a composite image from a display device in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that are noteworthy in the image. The device includes a communication circuit and a processing circuit, the communication circuit being connected to the mobile unit via a communication network, the processing circuit being coupled to the communication circuit. The communication circuit is configured to receive image data including the image from the moving body, and to receive target data from the moving body separately from the image data, the target data including recognition information of the target in the image, and including the acquisition timing of the image used to recognize the target. The processing circuit is configured to: when the target is a dynamic target, predict a movement amount of the dynamic target in a target delay time indicating a timing difference between the acquisition timing of the image used to recognize the dynamic target and the reception timing of the target data including recognition information of the dynamic target by the processing circuit; set a target movement adjustment time that does not monotonically decrease with an increase in the predicted movement amount of the dynamic target; based on information about the relative motion of the dynamic target with respect to the moving body, projectively transform the image included in the image data and a partial image of the dynamic target included in the image used to recognize the dynamic target into a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image; and generate the composite image based on the future partial image, an original image that included an original partial image of the future partial image, and recognition information of the dynamic target in the original image.

[0007] A second aspect of the present disclosure is a method for remotely supporting a moving object, which has the following features. The method remotely supports the mobile body by outputting from a display device a composite image in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that are noteworthy in the image. The method includes receiving image data including the image from the moving body; receiving target data from the moving body, separately from the image data, the target data including recognition information of the target in the image, the target data including acquisition timing of the image used to recognize the target; if the target is a dynamic target, predicting a movement amount of the dynamic target in a target delay time indicating a timing difference between acquisition timing of the image used to recognize the dynamic target and reception timing of the target data including recognition information of the dynamic target; and setting a target movement adjustment time that is monotonically non-decreasing with time; projecting a partial image of the dynamic target included in the image included in the image data and the image used to recognize the dynamic target based on information about the relative movement of the dynamic target with respect to the moving body, into a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image; and generating the composite image based on the future partial image, an original image that included an original partial image of the future partial image, and recognition information of the dynamic target in the original image.

[0008] A third aspect of the present disclosure is a program for remotely supporting a moving object, which has the following features. The program remotely supports the mobile body by causing a computer to function to output from a display device a composite image in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that should be noted in the image. The program includes receiving image data including the image from the moving body; receiving, separately from the image data, target data including recognition information of the target in the image, the target data including acquisition timing of the image used to recognize the target; if the target is a dynamic target, predicting a movement amount of the dynamic target in a target delay time indicating a timing difference between the acquisition timing of the image used to recognize the dynamic target and the reception timing of the target data including the recognition information of the dynamic target; and predicting a movement amount of the dynamic target that does not monotonically decrease with an increase in the predicted movement amount of the dynamic target. The computer is configured to execute the following steps: set a target movement adjustment time; projectively transform, based on information regarding the relative movement of the dynamic target with respect to the moving body, the image included in the image data and a partial image of the dynamic target included in the image used to recognize the dynamic target, into a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image; and generate the composite image based on the future partial image, an original image that included the original partial image of the future partial image, and recognition information of the dynamic target in the original image. [Effects of the Invention]

[0009] According to the present disclosure, when a target is a dynamic target, a projective transformation is performed on a partial image of the dynamic target included in an image included in the image data and an image used to recognize the dynamic target. Projective transformation of the partial image results in a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image including the partial image. According to the present disclosure, a composite image is also generated based on the future partial image, an original image that included the original partial image of the future partial image, and recognition information of the dynamic target in the original image. Therefore, even if the amount of movement of the dynamic target during the target delay time is large, it is possible to prevent the position and size of the dynamic target in the composite image from deviating significantly from the actual position and size. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a remote support system. [Figure 2] FIG. 10 is a diagram illustrating an example of data transmission from a vehicle to a remote operator terminal. [Figure 3] FIG. 10 is a diagram showing an example of a composite image output from a display device. [Figure 4] FIG. 2 is a diagram illustrating a delay time of a target. [Figure 5] 10 is a conceptual diagram for explaining an outline of a movement adjustment process for an image of a dynamic target. FIG. [Figure 6] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 7] FIG. 10 is a diagram illustrating a second example of the movement adjustment time. [Figure 8] FIG. 10 is a diagram illustrating a third example of the movement adjustment time. [Figure 9] FIG. 10 is a diagram illustrating a fourth example of the movement adjustment time. [Figure 10] FIG. 10 is a diagram showing an example of a composite image output from a display device when a projective transformation is performed on an image of a dynamic target. [Figure 11] 10A to 10C are diagrams illustrating a preferred example of a synthetic image generation process. [Figure 12] 10A to 10C are diagrams illustrating a preferred example of a synthetic image generation process. [Figure 13] FIG. 1 is a block diagram showing an example of the configuration of a vehicle. [Figure 14] FIG. 2 is a block diagram showing an example of the configuration of a remote operator terminal. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0012] 1. Remote support system Consider remote support (remote driving) of a moving object. Examples of moving objects that can be remotely supported include vehicles, robots, flying objects, etc. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones, etc. As an example, in the following explanation, consider a case where the moving object is a vehicle. When generalizing, "vehicle" in the following explanation should be read as "moving object."

[0013] FIG. 1 is a schematic diagram showing an example of the configuration of a remote support system 1 according to this embodiment. The remote support system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote support. The remote operator terminal 200 is a terminal device used by a remote operator O when remotely supporting the vehicle 100. The remote operator terminal 200 can also be called a remote support HMI (Human Machine Interface). The management device 300 manages the remote support system 1. Typically, the management device 300 is a management server on the cloud. The management server may be composed of multiple servers that perform distributed processing.

[0014] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 may communicate directly without going through the management device 300.

[0015] The vehicle 100 is equipped with various sensors including a camera CAM. The camera CAM captures images of the surroundings of the vehicle 100 and acquires images IMG showing the situation around the vehicle 100. The sensor detection information SEN includes information obtained by the various sensors. The sensor detection information SEN includes at least the image IMG captured by the camera CAM and recognition information OR of a target (object) that should be noted in this image IMG. The sensor detection information SEN may also include the position and status of the vehicle 100 (e.g., speed, steering angle, etc.). The vehicle 100 transmits the sensor detection information SEN to the remote operator terminal 200.

[0016] The remote operator terminal 200 receives the sensor detection information SEN transmitted from the vehicle 100. The remote operator terminal 200 presents the sensor detection information SEN to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device 220, and displays information such as an image IMG on the display device 220. The remote operator O looks at the displayed information, recognizes the situation around the vehicle 100, and provides remote support for the vehicle 100. In other words, the remote operator O provides remote support for the vehicle 100 by displaying information for the remote operator O on the display device 220.

[0017] The remote support information OPE is information related to remote support by a remote operator O. For example, the remote support information OPE includes the amount of operation by the remote operator O. The remote operator terminal 200 transmits the remote support information OPE to the vehicle 100. The vehicle 100 receives the remote support information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle driving control in accordance with the received remote support information OPE. In this way, remote support for the vehicle 100 is realized.

[0018] 2. Composite image generation process In this embodiment, a composite image SIMG is generated in the remote operator terminal 200. The composite image SIMG is generated based on an image IMG included in the sensor detection information SEN and recognition information OR of a target object to be noted in this image IMG. The image IMG and the recognition information OR are transmitted separately from the vehicle 100 to the remote operator terminal 200.

[0019] FIG. 2 is a diagram illustrating an example of transmission of images IMG and recognition information OR from vehicle 100 to remote operator terminal 200. Image data including a set of images IMG (i.e., video) acquired within a certain period of time is transmitted from vehicle 100 to remote operator terminal 200. This certain period of time corresponds to the transmission interval of the image data. In the example shown in FIG. 2, the set of images IMG1 acquired at time T1 includes images IMG1(T1a), IMG1(T1b), and IMG1(T1c). Images IMG1(T1a), IMG1(T1b), and IMG1(T1c) were acquired at times T1a, T1b, and T1c, respectively.

[0020] Target recognition processing is performed on each of images IMG1(T1a), IMG1(T1b), and IMG1(T1c). Image analysis techniques such as pattern matching and deep learning are used for the recognition processing. When the recognition processing is performed, recognition information OR(T1a), OR(T1b), and OR(T1c) are generated. The recognition information OR includes the timing (timestamp) at which the image IMG1 on which the recognition processing was performed was acquired by the camera CAM. If a notable target is recognized in image IMG1, the coordinates, size, and type of this target in image IMG1 are added to the recognition information OR. In other words, if no notable target is recognized in image IMG1, the recognition information OR includes only timing information.

[0021] Target data including recognition information OR is transmitted from vehicle 100 to remote operator terminal 200. The transmission of target data is performed every time recognition information OR is generated. The transmission of target data may be performed at regular intervals. This regular interval corresponds to the transmission interval of the target data.

[0022] The remote operator terminal 200, which has received the image data and target data, arranges the images IMG1 (i.e., images IMG1(T1a), IMG1(T1b), and IMG1(T1c)) in chronological order based on the timing (timestamp) at which the image IMG1 included in the image data was acquired by the camera CAM. The remote operator terminal 200 also identifies the image IMG1 to be combined with the recognition information OR based on the timing (timestamp) included in the recognition information OR of the target data.

[0023] The remote operator terminal 200 further refers to the coordinates, size, and type of the target included in the recognition information OR of the target data, and superimposes annotation information OA on the image IMG1 to be combined with the target. The annotation information OA is information indicating the position, size, and type of the target in the image IMG1. The information indicating the position and size of the target is a bounding box surrounding the target. The information indicating the type of the target is a character indicating the type.

[0024] A composite image SIMG1 is generated by superimposing annotation information OA on image IMG1. Since the original images IMG1 of composite image SIMG1 (i.e., images IMG1(T1a), IMG1(T1b), and IMG1(T1c)) are arranged in chronological order, a set of composite images SIMG1 (i.e., a video) is output from display device 220 by outputting composite images SIMG1 in chronological order.

[0025] Fig. 3 is a diagram showing an example of a composite image SIMG output from the display device 220. In the example shown in Fig. 3, annotation information OA1 and OA2 are output. Annotation information OA1 is a bounding box surrounding a green light and the words "green light." Annotation information OA2 is a bounding box surrounding a pedestrian and the words "pedestrian."

[0026] 3. Processing using projective transformation Remote support of the vehicle 100 involves a delay between the vehicle 100 and the remote operator terminal 200. The difference between the timing T1 when the image IMG1 in which a target of interest is recognized is captured by the camera CAM and the timing when the recognition information OR becomes available for processing by the remote operator terminal 200 can be said to be the target delay time.

[0027] FIG. 4 is a diagram illustrating the delay time of a target. In the example shown in FIG. 4, image IMG1 is acquired at timing T1, then encoded and transmitted to the remote operator terminal 200. The reason for encoding the image is to reduce communication costs. After the encoded image IMG1 is received by the remote operator terminal 200, it is decoded. This makes image IMG1 processable by the remote operator terminal 200.

[0028] In contrast, the recognition information OR is generated and then transmitted to the remote operator terminal 200. Therefore, the recognition information OR can be processed by the remote operator terminal 200 at timing T3 when the recognition information OR (target data) is received by the remote operator terminal 200. Therefore, the delay time of the target is expressed as the timing difference D1 (=T3-T1) between timing T3 and timing T1.

[0029] The composite image SIMG1 is generated after the timing T3 and when the image IMG1 becomes processable by the remote operator terminal 200. Here, consider the case where the target is a dynamic target such as a pedestrian, bicycle, or other vehicle. When the target is a dynamic target, if the amount of movement of the dynamic target within the target delay time is large, the position and size of the dynamic target in the composite image SIMG1 will deviate significantly from the actual position and size of the dynamic target.

[0030] Therefore, in this embodiment, the movement amount Lo of the dynamic target within the target delay time (timing difference D1) is predicted. Then, based on this movement amount Lo, visual adjustment is made to the image of the dynamic target included in image IMG1. In particular, the remote support system 1 according to this embodiment adjusts the image of the dynamic target included in image IMG1 using "projective transformation." The subject of the movement adjustment process of the dynamic target is, for example, the remote operator terminal 200. However, the subject of the movement adjustment process is not limited to the remote operator terminal 200. At least a part of the movement adjustment process may be executed by the vehicle 100 or the management device 300.

[0031] 5 is a conceptual diagram for explaining an outline of the movement adjustment process of an image of a dynamic target by the remote support system 1. Image IMG1 is an image IMG that is actually captured at time T1 by a camera CAM mounted on the vehicle 100. Image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 acquires image IMG1 after time T1. If it is possible to estimate (predict) images IMG that will be captured in the future from image IMG1, it becomes possible to perform movement adjustment.

[0032] Timing T2 is a target timing for look-ahead and is later than timing T1. The difference between timing T2 and timing T1 corresponds to the "movement adjustment time." The remote support system 1 sets a time α equal to or less than the timing difference D1 shown in FIG. 4 as the movement adjustment time of the dynamic target (target movement adjustment time). In the first example, time α is set to a time equal to the timing difference D1. According to the first example, the position and size of the dynamic target can be adjusted by the movement amount Lo within the movement adjustment time.

[0033] For convenience, the camera CAM at timing T1 will be referred to as the first camera CAM1, and the camera CAM at timing T2 will be referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the combination of the position and orientation of the first camera CAM1 at timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the combination of the predicted position and orientation of the second camera CAM2 at timing T2.

[0034] The remote support system 1 acquires camera information CINF related to the camera CAM mounted on the vehicle 100. The camera information CINF includes installation information and performance information of the camera CAM. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. Because the camera CAM is fixed to the vehicle 100, by using the installation information of the camera CAM, the direction and amount of movement of the vehicle 100 can be converted into the direction and amount of movement of the camera CAM in the camera coordinate system. In other words, the change in the viewpoint of the camera CAM can be estimated based on the installation information of the camera CAM and the direction and amount of movement of the vehicle 100.

[0035] More specifically, the remote support system 1 acquires information (relative movement direction and relative movement amount) regarding the relative movement of the dynamic target to the vehicle 100 during the period from timing T1 to timing T2 (i.e., the target movement adjustment time). For example, the remote support system 1 estimates the relative movement direction and relative movement amount of the dynamic target from timing T1 to timing T2 using optical flow. Information on the speed and steering angle of the vehicle 100 is obtained from sensor detection information SEN provided by the vehicle 100. Alternatively, the steering angle in the steering operation by the remote operator O may be considered as the steering angle of the vehicle 100. The vehicle 100 may be assumed to make a steady circular turn. Then, the remote support system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information CINF (installation information) and the movement amount and movement direction of the vehicle 100 during the movement adjustment time.

[0036] Image IMG1 is an image IMG that is actually captured at time T1 by a camera CAM mounted on vehicle 100. Image IMG1 is transmitted from vehicle 100 to remote operator terminal 200. Based on the recognition information OR, remote operator terminal 200 extracts a partial image OBJ1 of a dynamic target from image IMG1 acquired after time T1. Then, using projective transformation, it estimates (looks ahead) a partial image OBJ2 of a dynamic target that will be captured in the future (future partial image) from partial image OBJ1. Projective transformation is used for this look ahead.

[0037] FIG. 6 is a conceptual diagram for explaining projective transformation. Projective transformation is performed based on perspective projection transformation. Perspective projection transformation is a rendering technique for rendering an object in three-dimensional space on a two-dimensional plane as seen from the camera CAM. To achieve this, perspective projection transformation projects points in three-dimensional space onto a projection plane P, taking into account the viewpoint of the camera CAM. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane perpendicular to the optical axis of the camera CAM. Note that points in three-dimensional space are defined in a three-dimensional world coordinate system (absolute coordinate system). On the other hand, points projected onto the projection plane P are defined in a two-dimensional image coordinate system.

[0038] For example, N virtual points are virtually set in a three-dimensional world coordinate system. N is an integer equal to or greater than 4. The N virtual points as viewed from a first camera CAM1 (first viewpoint) are projected onto a first projection plane P1 associated with the first camera CAM1 by perspective projection transformation. The N virtual points as viewed from a second camera CAM2 (second viewpoint) are projected onto a second projection plane P2 associated with the second camera CAM2 by perspective projection transformation. The second viewpoint is obtained from the difference between the first viewpoint and the second viewpoint. The image coordinates of the virtual points on the first projection plane P1 as viewed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the virtual points on the second projection plane P2 as viewed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on a comparison of the two, a projection transformation matrix H for converting from the first viewpoint to the second viewpoint is calculated. Then, the projective transformation matrix H is applied to the entire image IMG1 actually captured by the first camera CAM1, thereby generating a second image that is expected to be seen from the second viewpoint.

[0039] As another example, a method described in a non-patent document (Koda Matsubara, Manabu Ohmae, "Research on Delay Compensation of Camera Images for Remotely Controlling Vehicles Using Projection Transformation," 19th ITS Symposium 2021, 4-A-12, December 2021) may be used. Specifically, by inverse transformation of the perspective projection transformation, each image coordinate point on image IMG1 (projection plane P) is transformed into a world coordinate point in the world coordinate system. Based on the difference between the first and second viewpoints, the world coordinate points seen from the first viewpoint are transformed into world coordinate points seen from the second viewpoint. Then, by perspective projection transformation, the world coordinate points seen from the second viewpoint are returned to the projection plane P. This generates an image expected to be seen from the second viewpoint. Note that according to the non-patent document, it is assumed that the ground surface S is reflected across the entire image IMG, as shown in FIG. 6.

[0040] Figs. 7 to 9 are diagrams for explaining the second to fourth examples of the movement adjustment time. In the second example shown in Fig. 7, the time α is set to be equal to the timing difference D1 only when the movement amount Lo is greater than the threshold value THLo. According to the second example, the movement adjustment process can be performed only when the position and size of the moving object target in the composite image SIMG1 are likely to deviate. In the third example shown in Fig. 8 and the fourth example shown in Fig. 9, the time α is gradually changed (monotonically non-decreasing) from the preliminary threshold value THpre (<THLo) to the threshold value THLo.

[0041] In the second example, when the movement amount Lo changes so as to cross the threshold value THLo, the execution of the movement adjustment process is switched. Therefore, there is a possibility that the position and size of the moving object target change abruptly before and after the execution of the movement adjustment process, and the image of the moving object target becomes unstable. In this regard, in the third and fourth examples, since the time α is gradually changed, the images of the moving object target before and after the execution of the movement adjustment process can be stabilized.

[0042] Fig. 10 is a diagram showing an example of the composite image SIMG1 output from the display device 220 when a projective transformation is performed on the image of the moving object target. The composite image SIMG1 shown in Fig. 10 is generated based on the image IMG1 and the partial image OBJ2. The annotation information OA1 and OA2 shown in Fig. 10 are the same as those described in Fig. 3. What is important in the description of Fig. 10 is that the position and size (solid line) of the annotation information OA2 (pedestrian) are different from the position and size (dashed line) of the annotation information OA2 in Fig. 3. This is because a projective transformation is performed on the partial image of the moving object target.

[0043] 11 and 12 are diagrams illustrating a preferred example of the process for generating the composite image SIMG1. The composite image SIMG1 shown in FIGS. 11 and 12 is generated based on the image IMG1, partial image OBJ1, and partial image OBJ2. Therefore, the composite image SIMG1 includes a partial image before the projective transformation (i.e., partial image OBJ1) and a partial image after the projective transformation (i.e., partial image OBJ2). By including two types of partial images in the composite image SIMG1, it becomes easier for the remote operator O to predict the trajectory of a dynamic target.

[0044] However, the simultaneous output of two types of partial images may confuse the remote operator O. Therefore, in the example shown in Fig. 11, the display color of the annotation information OA2 attached to the partial image after projective transformation (for example, the color of the bounding box surrounding the pedestrian) is made different from that of the annotation information OA2* attached to the partial image before projective transformation. In the example shown in Fig. 12, a figure AR linking the annotation information OA2* attached to the partial image before projective transformation and the annotation information OA2 attached to the partial image after projective transformation is superimposed on the composite image SIMG1.

[0045] 4. Example of vehicle configuration 4-1.Configuration example 13 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0046] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300.

[0047] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera (CAM), a LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0048] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates braking force.

[0049] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processors 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processors 160 perform various processes. Examples of the processors 160 include a general-purpose processor, a specific-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, and / or a combination thereof. The storage device 170 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), a solid-state drive (SSD), etc. The control device 150 may include one or more electronic control units (ECUs). In general terms, the control device 150 can also be called a processing circuitry.

[0050] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 may be realized by cooperation between the processor 160, which executes the vehicle control program PROG1, and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0051] 4-2. Sensor detection information The control device 150 acquires sensor detection information SEN using the sensor group 120. The sensor detection information SEN is stored in the storage device 170. The sensor detection information SEN includes an image IMG, vehicle state information, position information, target information, etc. The image IMG is captured by a camera CAM. The vehicle state information indicates the state of the vehicle 100 (e.g., speed, steering angle, etc.) detected by the vehicle state sensor. The position information indicates the position and orientation of the vehicle 100 detected by the position sensor.

[0052] The target information is information relating to targets around the vehicle 100. Examples of targets around the vehicle 100 include pedestrians, bicycles, motorcycles, other vehicles (leading vehicles, vehicles running alongside, following vehicles, etc.), white lines, road structures (e.g., curbs, guardrails), poles, traffic lights, signs, etc. The control device 150 can recognize targets around the vehicle 100 by using a recognition sensor. For example, by analyzing an image (IMG), it is possible to identify targets and calculate the relative position of the targets. It is also possible to identify targets and obtain the relative position and relative speed of the targets based on point cloud information obtained by LIDAR. The target information includes the relative position of the targets with respect to the vehicle 100. The target information may further include the relative speed of the targets.

[0053] 4-3.Vehicle driving control The control device 150 executes vehicle driving control to control the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes vehicle driving control by controlling the driving device 130 (steering device, drive device, and brake device).

[0054] The control device 150 may perform automatic driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the sensor detection information SEN. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0055] 4-4.Remote support related processes When remote support of the vehicle 100 is performed, the control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0056] The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. Typically, the control device 150 transmits an image IMG to the remote operator terminal 200. The control device 150 may transmit vehicle state information to the remote operator terminal 200. The control device 150 may transmit target object information to the remote operator terminal 200.

[0057] Furthermore, the control device 150 receives remote support information OPE from the remote operator terminal 200. The remote support information OPE is information related to remote support by the remote operator O. For example, the remote support information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle driving control in accordance with the received remote support information OPE.

[0058] 4-5.Camera information The camera information CINF includes installation information and performance information for each of one or more camera CAMs mounted on the vehicle 100. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. The camera information CINF is stored in the storage device 170. The control device 150 may transmit the camera information CINF to the remote operator terminal 200.

[0059] 5. Example of remote operator terminal configuration 14 is a block diagram showing an example of the configuration of the remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, and an information processing device 250.

[0060] The communication device (communication circuit) 210 communicates with the vehicle 100 and the management device 300.

[0061] The display device 220 displays various information for the remote operator O who provides remote support. In other words, the display device 220 presents various information to the remote operator O by displaying the various information.

[0062] The input device 230 is a member that the remote operator O operates when remotely supporting the vehicle 100. For example, the input device 230 includes remote support members, such as a steering wheel, an accelerator pedal, a brake pedal, and a turn signal.

[0063] The information processing device 250 controls the remote operator terminal 200. The information processing device 250 includes one or more processors 260 (hereinafter simply referred to as processors 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processor 260 executes various processes. Examples of the processor 260 include a general-purpose processor, a special-purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, and / or a combination thereof. The storage device 270 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a HDD, an SSD, etc. In general terms, the information processing device 250 can also be called a processing circuitry.

[0064] The remote support control program PROG2 is a computer program executed by the processor 260. The functions of the information processing device 250 may be realized by cooperation between the processor 260 executing the remote support control program PROG2 and the storage device 270. The remote support control program PROG2 is stored in the storage device 270. Alternatively, the remote support control program PROG2 may be recorded on a computer-readable recording medium. The remote support control program PROG2 may be provided via a network.

[0065] The information processing device 250 communicates with the vehicle 100 via the communication device 210. The information processing device 250 receives sensor detection information SEN transmitted from the vehicle 100. The information processing device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the information processing device 250 presents an image IMG to the remote operator O by displaying the image IMG on the display device 220. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the presented information.

[0066] The remote operator O operates the input device 230. The amount of operation of the input device 230 is detected by a sensor installed on the input device 230. The information processing device 250 generates remote support information OPE that reflects the amount of operation of the input device 230 by the remote operator O. Then, the information processing device 250 transmits the remote support information OPE to the vehicle 100 via the communication device 210.

[0067] The information processing device 250 may receive the camera information CINF transmitted from the vehicle 100. The camera information CINF is stored in the storage device 270.

[0068] The information processing device 250 executes the projective transformation process (movement adjustment process) described in Section 3 above and the composite image generation process described in Section 2 above. The speed and steering angle of the vehicle 100 are obtained from the sensor detection information SEN. The steering angle in the steering operation by the remote operator O may be considered as the steering angle of the vehicle 100. Installation information and performance information of each camera CAM mounted on the vehicle 100 are obtained from the camera information CINF. Based on this information, the information processing device 250 executes the projective transformation process (movement adjustment process) described in Section 3 above. [Explanation of symbols]

[0069] 1. Remote control system 100 vehicles 200 Remote Operator Terminal 210 Communication equipment 220 Display device 250 Information Processing Equipment 300 Management device OA Annotation Information OR recognition information CAM camera CINF Camera Information IMG,IMG1 Images OBJ1,OBJ2 partial images SIMG1 composite image SEN Sensor detection information

Claims

1. 1. A device for remotely supporting a moving body by outputting a composite image from a display device, in which annotation information relating to a target object to be noted in an image obtained by a camera mounted on the moving body is superimposed on the image, the device comprising: a communication circuit connected to the mobile unit via a communication network; a processing circuit coupled to the communication circuit; The communication circuit receiving image data including the image from the mobile object; receiving, from the moving body, target data including recognition information of the target in the image, separately from the image data, the target data including acquisition timing of the image used to recognize the target; The processing circuitry If the target is a dynamic target, predict the amount of movement of the dynamic target in a target delay time that indicates a timing difference between the acquisition timing of the image used to recognize the dynamic target and the reception timing of the target data including recognition information of the dynamic target by the processing circuit; setting a target movement adjustment time that does not monotonically decrease with an increase in the predicted movement amount of the dynamic target; based on information on the relative movement of the dynamic object with respect to the moving body, projectively transforming the image included in the image data and a partial image of the dynamic object included in the image used to recognize the dynamic object into a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image, and generating the composite image based on the future partial image, an original image including the original partial image of the future partial image, and recognition information of the dynamic target in the original image. A remote support device characterized by:

2. 2. The remote support device according to claim 1, The processing circuitry When the predicted movement amount is greater than a threshold movement amount, the target delay time is set as the target movement adjustment time; When the predicted movement amount is equal to or less than the threshold movement amount, the target movement adjustment time is gradually increased from a predetermined time to the target delay time in accordance with an increase in the predicted movement amount. A remote support device characterized by:

3. 2. The remote support device according to claim 1, The processing circuit is configured to generate the composite image based on the future partial image, an original image including an original partial image of the future partial image, recognition information of the dynamic target in the original image, and the original partial image. A remote support device characterized by:

4. 4. The remote support device according to claim 3, The processing circuitry is configured to output a color indicating annotation information regarding the future partial image in the composite image that is different from a color indicating annotation information regarding the original partial image. A remote support device characterized by:

5. 4. The remote support device according to claim 3, The processing circuit is configured to superimpose, on the composite image, a graphic linking annotation information on the future partial image in the composite image with annotation information on the original partial image. A remote support device characterized by:

6. 1. A method for remotely supporting a mobile body by outputting, from a display device, a composite image in which annotation information relating to a target object to be noted in an image obtained by a camera mounted on the mobile body is superimposed on the image, the method comprising: receiving image data including the image from the mobile object; receiving, from the moving body, target data including recognition information of the target in the image, separately from the image data, the target data including acquisition timing of the image used to recognize the target; If the target is a dynamic target, predicting a movement amount of the dynamic target in a target delay time indicating a timing difference between an acquisition timing of the image used to recognize the dynamic target and a reception timing of the target data including recognition information of the dynamic target; setting a target movement adjustment time that does not monotonically decrease with an increase in the predicted movement amount of the dynamic target; based on information on the relative motion of the dynamic object with respect to the moving body, projectively transforming the image included in the image data and the partial image of the dynamic object included in the image used to recognize the dynamic object into a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image; generating the composite image based on the future partial image, an original image containing the original partial image of the future partial image, and recognition information of the dynamic target in the original image. A remote support method comprising:

7. A program for remotely supporting a mobile body by causing a computer to function to output from a display device a composite image in which annotation information relating to a target object that should be noted in an image obtained by a camera mounted on the mobile body is superimposed on the image, the program comprising: receiving image data including the image from the mobile object; receiving, from the moving body, target data including recognition information of the target in the image, separately from the image data, the target data including acquisition timing of the image used to recognize the target; If the target is a dynamic target, predicting a movement amount of the dynamic target in a target delay time indicating a timing difference between an acquisition timing of the image used to recognize the dynamic target and a reception timing of the target data including recognition information of the dynamic target; setting a target movement adjustment time that does not monotonically decrease with an increase in the predicted movement amount of the dynamic target; based on information on the relative motion of the dynamic object with respect to the moving body, projectively transforming the image included in the image data and the partial image of the dynamic object included in the image used to recognize the dynamic object into a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image; generating the composite image based on the future partial image, an original image that included an original partial image of the future partial image, and recognition information of the dynamic target in the original image. A remote support program characterized by:

Images