Remote support device, remote support method, and remote support program

Abstract

To prevent the ambient environment of a movable body shown in a composite image from deviating from the actual ambient environment, in receiving a camera image and a target recognition result separately from the movable body to perform remote support of the movable body.SOLUTION: A remote support device predicts the amount of travel of a movable body in an image delay time indicating a timing difference D1 between a timing T1 at which an image IMG1 is acquired by a camera CAM, and a timing T3 at which a remote operator terminal 200 decodes the image IMG1. When the predicted amount of travel Lv of the movable body exceeds a travel threshold THLv, the remote support device performs, on the basis of information on the movement of the movable body, projective transformation of the image IMG1 into a future image IMG2 obtained in a camera view at a timing T2 later than the timing T1 by a movable body adjustment time α, to create a composite image SIMG1 output from a display 220.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present disclosure relates to a technology for remotely supporting a mobile object using images acquired by a camera mounted on the mobile object. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2022-159912 discloses a system for remotely monitoring a vehicle. In this system, first and second data are separately transmitted from the vehicle to a remote server. The first data includes an image (camera image) acquired by a camera mounted on the vehicle and the timing of acquisition of the camera image. The second data includes a target recognition result based on the camera image and the timing of acquisition of the camera image. Based on the acquisition timing of the camera image included in the first and second data, the remote server extracts, in chronological order, camera images acquired at the same time and target recognition results based on the camera images. Then, a composite image in which the target recognition result is superimposed on the extracted camera image is output from a remote monitoring screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-159912 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above system, the difference between the time when a camera image is acquired and the time when the camera image becomes available for processing on the remote server can be considered the camera image delay time. The camera image delay time includes the time required for communication between the vehicle and the remote server and the time required for processing on the vehicle. The problem here is that if the vehicle moves significantly during the camera image delay time, the vehicle's surrounding environment shown in the composite image may deviate from the actual surrounding environment. This reduces the accuracy of the information output from the remote monitoring screen.

[0005] The present disclosure has been made in view of the above-mentioned problems, and one objective of the present disclosure is to provide a technology for preventing the surrounding environment of a moving object shown in a composite image from deviating from the actual surrounding environment when remotely supporting the moving object by separately receiving a camera image and a target recognition result from the moving object. [Means for solving the problem]

[0006] A first aspect of the present disclosure is an apparatus for remotely supporting a moving object, which has the following features. The device remotely supports the mobile body by outputting a composite image from a display device in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that are noteworthy in the image. The device includes a communication circuit and a processing circuit, the communication circuit being connected to the mobile unit via a communication network, the processing circuit being coupled to the communication circuit. The communication circuit is configured to receive image data from the moving body in which the image and information on the timing of acquisition of the image by the camera are encoded, and to receive target data from the moving body separately from the image data, the target data including recognition information of the target in the image. The processing circuit is configured to predict the amount of movement of the moving body during an image delay time that indicates the timing difference between the acquisition timing of the image included in the image data and the decoding timing of the image data by the processing circuit, and if the predicted amount of movement of the moving body exceeds a movement threshold, set a moving body adjustment time that is less than the image delay time, and if the predicted amount of movement of the moving body exceeds the movement threshold, project the image included in the image data onto a future image obtained from a camera viewpoint that is the moving body adjustment time ahead of the acquisition timing of the image based on information about the movement of the moving body, thereby generating the composite image.

[0007] A second aspect of the present disclosure is a method for remotely supporting a moving object, which has the following features. The method remotely supports the mobile body by outputting from a display device a composite image in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that are noteworthy in the image. The method includes receiving image data from the moving body in which the image and information on the timing of acquisition of the image by the camera are encoded; receiving target data from the moving body separately from the image data, which includes recognition information of the target in the image; predicting the amount of movement of the moving body in an image delay time indicating the timing difference between the acquisition timing of the image included in the image data and the decoding timing of the image data; if the predicted amount of movement of the moving body exceeds a movement threshold, setting a moving body adjustment time that is less than or equal to the image delay time; and if the predicted amount of movement of the moving body exceeds the movement threshold, projecting the image included in the image data onto a future image obtained from a camera viewpoint that is the moving body adjustment time ahead of the acquisition timing of the image, based on information about the movement of the moving body, to generate the composite image.

[0008] A third aspect of the present disclosure is a program for remotely supporting a moving object, which has the following features. The program remotely supports the mobile body by causing a computer to function to output from a display device a composite image in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that should be noted in the image. The program is configured to cause the computer to perform the following operations: receive image data from the moving body in which the image and information on the timing of acquisition of the image by the camera are encoded; receive target data from the moving body separately from the image data, which includes recognition information of the target in the image; predict the amount of movement of the moving body during an image delay time indicating the timing difference between the acquisition timing of the image included in the image data and the decoding timing of the image data; if the predicted amount of movement of the moving body exceeds a movement threshold, set a moving body adjustment time that is less than or equal to the image delay time; and if the predicted amount of movement of the moving body exceeds the movement threshold, project the image included in the image data onto a future image obtained from a camera viewpoint that is the moving body adjustment time ahead of the acquisition timing of the image, based on information about the movement of the moving body, to generate the composite image. [Effects of the Invention]

[0009] According to the present disclosure, when the predicted movement amount of the moving object during the image delay time exceeds a movement threshold, a projective transformation of the image is performed. The projective transformation of the image generates a future image obtained from a camera viewpoint that is the moving object adjustment time ahead of the image capture timing. Therefore, for example, by setting a movement threshold corresponding to the movement amount at which a discrepancy is expected to occur between the surrounding environment of the moving object shown in the composite image and the actual surrounding environment, it is possible to prevent the discrepancy from occurring due to the projective transformation performed when the predicted movement amount of the moving object is large. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a remote support system. [Figure 2] FIG. 10 is a diagram illustrating an example of data transmission from a vehicle to a remote operator terminal. [Figure 3] FIG. 10 is a diagram showing an example of a composite image output from a display device. [Figure 4] FIG. 10 is a diagram illustrating the delay time of an image and the delay time of a target. [Figure 5] FIG. 10 is a conceptual diagram for explaining an overview of a vehicle movement adjustment process. [Figure 6] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 7] 10A and 10B are diagrams illustrating an example of a composite image output from a display device when a vehicle movement adjustment process is performed. [Figure 8] FIG. 10 is a diagram illustrating a first example of a moving object adjustment time. [Figure 9] FIG. 10 is a diagram illustrating a second example of a moving object adjustment time. [Figure 10] FIG. 10 is a diagram illustrating a third example of a moving object adjustment time. [Figure 11] FIG. 10 is a conceptual diagram for explaining an outline of a movement adjustment process for a dynamic target. [Figure 12] 10A and 10B are diagrams illustrating an example of a composite image output from a display device when a movement adjustment process for a dynamic target is performed. [Figure 13] 10A and 10B are diagrams illustrating a case where the vehicle movement adjustment process and the dynamic target movement adjustment process are combined. [Figure 14] FIG. 1 is a block diagram showing an example of the configuration of a vehicle. [Figure 15] FIG. 2 is a block diagram showing an example of the configuration of a remote operator terminal. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0012] 1. Remote support system Consider remote support (remote driving) of a moving object. Examples of moving objects that can be remotely supported include vehicles, robots, flying objects, etc. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones, etc. As an example, in the following explanation, consider a case where the moving object is a vehicle. When generalizing, "vehicle" in the following explanation should be read as "moving object."

[0013] FIG. 1 is a schematic diagram showing an example of the configuration of a remote support system 1 according to this embodiment. The remote support system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote support. The remote operator terminal 200 is a terminal device used by a remote operator O when remotely supporting the vehicle 100. The remote operator terminal 200 can also be called a remote support HMI (Human Machine Interface). The management device 300 manages the remote support system 1. Typically, the management device 300 is a management server on the cloud. The management server may be composed of multiple servers that perform distributed processing.

[0014] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 may communicate directly without going through the management device 300.

[0015] The vehicle 100 is equipped with various sensors including a camera CAM. The camera CAM captures images of the surroundings of the vehicle 100 and acquires images IMG showing the situation around the vehicle 100. The sensor detection information SEN includes information obtained by the various sensors. The sensor detection information SEN includes at least the image IMG captured by the camera CAM and recognition information OR of a target (object) that should be noted in this image IMG. The sensor detection information SEN may also include the position and status of the vehicle 100 (e.g., speed, steering angle, etc.). The vehicle 100 transmits the sensor detection information SEN to the remote operator terminal 200.

[0016] The remote operator terminal 200 receives the sensor detection information SEN transmitted from the vehicle 100. The remote operator terminal 200 presents the sensor detection information SEN to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device 220, and displays information such as an image IMG on the display device 220. The remote operator O looks at the displayed information, recognizes the situation around the vehicle 100, and provides remote support for the vehicle 100. In other words, the remote operator O provides remote support for the vehicle 100 by displaying information for the remote operator O on the display device 220.

[0017] The remote support information OPE is information related to remote support by a remote operator O. For example, the remote support information OPE includes the amount of operation by the remote operator O. The remote operator terminal 200 transmits the remote support information OPE to the vehicle 100. The vehicle 100 receives the remote support information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle driving control in accordance with the received remote support information OPE. In this way, remote support for the vehicle 100 is realized.

[0018] 2. Composite image generation process In this embodiment, a composite image SIMG is generated in the remote operator terminal 200. The composite image SIMG is generated based on an image IMG included in the sensor detection information SEN and recognition information OR of a target object to be noted in this image IMG. The image IMG and the recognition information OR are transmitted separately from the vehicle 100 to the remote operator terminal 200.

[0019] FIG. 2 is a diagram illustrating an example of transmission of images IMG and recognition information OR from vehicle 100 to remote operator terminal 200. Image data including a set of images IMG (i.e., video) acquired within a certain period of time is transmitted from vehicle 100 to remote operator terminal 200. This certain period of time corresponds to the transmission interval of the image data. In the example shown in FIG. 2, the set of images IMG1 acquired at time T1 includes images IMG1(T1a), IMG1(T1b), and IMG1(T1c). Images IMG1(T1a), IMG1(T1b), and IMG1(T1c) were acquired at times T1a, T1b, and T1c, respectively.

[0020] Target recognition processing is performed on each of images IMG1(T1a), IMG1(T1b), and IMG1(T1c). Image analysis techniques such as pattern matching and deep learning are used for the recognition processing. When the recognition processing is performed, recognition information OR(T1a), OR(T1b), and OR(T1c) are generated. The recognition information OR includes the timing (timestamp) at which the image IMG1 on which the recognition processing was performed was acquired by the camera CAM. If a notable target is recognized in image IMG1, the coordinates, size, and type of this target in image IMG1 are added to the recognition information OR. In other words, if no notable target is recognized in image IMG1, the recognition information OR includes only timing information.

[0021] Target data including recognition information OR is transmitted from vehicle 100 to remote operator terminal 200. The transmission of target data is performed every time recognition information OR is generated. The transmission of target data may be performed at regular intervals. This regular interval corresponds to the transmission interval of the target data.

[0022] The remote operator terminal 200, which has received the image data and target data, arranges the images IMG1 (i.e., images IMG1(T1a), IMG1(T1b), and IMG1(T1c)) in chronological order based on the timing (timestamp) at which the image IMG1 included in the image data was acquired by the camera CAM. The remote operator terminal 200 also identifies the image IMG1 to be combined with the recognition information OR based on the timing (timestamp) included in the recognition information OR of the target data.

[0023] The remote operator terminal 200 further refers to the coordinates, size, and type of the target included in the recognition information OR of the target data, and superimposes annotation information OA on the image IMG1 to be combined with the target. The annotation information OA is information indicating the position, size, and type of the target in the image IMG1. The information indicating the position and size of the target is a bounding box surrounding the target. The information indicating the type of the target is a character indicating the type.

[0024] A composite image SIMG1 is generated by superimposing annotation information OA on image IMG1. Since the original images IMG1 of composite image SIMG1 (i.e., images IMG1(T1a), IMG1(T1b), and IMG1(T1c)) are arranged in chronological order, a set of composite images SIMG1 (i.e., a video) is output from display device 220 by outputting composite images SIMG1 in chronological order.

[0025] Fig. 3 is a diagram showing an example of a composite image SIMG output from the display device 220. In the example shown in Fig. 3, annotation information OA1 and OA2 are output. Annotation information OA1 is a bounding box surrounding a green light and the words "green light." Annotation information OA2 is a bounding box surrounding a pedestrian and the words "pedestrian."

[0026] 3. Processing using projective transformation 3-1. Vehicle 100 Movement Adjustment Processing Remote support of the vehicle 100 involves a delay between the vehicle 100 and the remote operator terminal 200. The difference between the timing T1 when the image IMG1 is acquired by the camera CAM and the timing when the image IMG1 becomes processable by the remote operator terminal 200 can be said to be the image delay time. Also, the difference between the timing T1 when the image IMG1 in which a noteworthy target is recognized is acquired by the camera CAM and the timing when the recognition information OR becomes processable by the remote operator terminal 200 can be said to be the target delay time.

[0027] FIG. 4 is a diagram illustrating the image delay time and the target delay time. In the example shown in FIG. 4, image IMG1 is acquired at timing T1, then encoded and transmitted to the remote operator terminal 200. The reason for encoding the image is to reduce communication costs. In order for the encoded image IMG1 to be processable by the remote operator terminal 200, this image IMG1 must be decoded. Therefore, the image delay time is expressed as the timing difference D1 (=T3-T1) between timing T1 and timing T3, when the remote operator terminal 200 decodes image IMG1.

[0028] In contrast, the recognition information OR is generated and then transmitted to the remote operator terminal 200. Therefore, the recognition information OR can be processed by the remote operator terminal 200 at timing T4 when the recognition information OR (target data) is received by the remote operator terminal 200. Therefore, the delay time of the target is expressed as the timing difference D2 (=T4-T1) between this timing T4 and timing T1.

[0029] Focusing on the delay time of the images (i.e., the timing difference D1), the following problem may arise: If the amount of movement of the vehicle 100 during the timing difference D1 (which refers to the amount of movement in at least one of the forward / backward direction and the left / right direction; the same applies hereinafter) is large, the surrounding environment of the vehicle 100 shown in the composite image SIMG1 may deviate from the actual surrounding environment.

[0030] Therefore, in this embodiment, visual adjustment is made to image IMG1 taking into consideration the amount of movement of vehicle 100 during the delay time (timing difference D1) of image IMG1. In particular, the remote support system 1 according to this embodiment makes visual adjustment to image IMG1 by using "projective transformation." The subject of the movement adjustment process is, for example, the remote operator terminal 200. However, the subject of the movement adjustment process is not limited to the remote operator terminal 200. At least a part of the movement adjustment process may be executed by vehicle 100 or the management device 300.

[0031] FIG. 5 is a conceptual diagram for explaining an overview of the movement adjustment process of the vehicle 100 by the remote support system 1. Image IMG1 is an image IMG that is actually captured at time T1 by a camera CAM mounted on the vehicle 100. Image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 acquires image IMG1 after time T1. If it is possible to estimate (predict) images IMG that will be captured in the future from image IMG1, it will be possible to adjust the surrounding environment of the vehicle 100.

[0032] Timing T2 is the target timing for look-ahead and is later than timing T1. The difference between timing T2 and timing T1 corresponds to the movement adjustment time (moving object adjustment time) of vehicle 100. The remote support system 1 predicts the movement amount of vehicle 100 at timing difference D1 shown in FIG. 4. The predicted movement amount Lv of vehicle 100 is calculated, for example, using the following formula: (Forward / backward) Delay time D1 × Vehicle 100 speed v(T1) (Left and right direction) Delay time D1^2 × Lateral acceleration of vehicle 100 Gy(T1) × 0.5 The lateral acceleration Gy used in the predicted left-right movement amount Lv may be estimated using the steering angle MA(T1) and the speed (T1) (Gy(T1) = MA(T1) × v(T1)).

[0033] If the predicted movement amount Lv exceeds the movement threshold THLv, the remote support system 1 sets a time α equal to or less than the timing difference D1 shown in Fig. 4 as the "moving object adjustment time" and performs movement adjustment. Note that the movement threshold THLv can be set in advance as the movement amount at which a discrepancy is expected to occur between the surrounding environment of the vehicle 100 shown in the composite image CIMG1 and the actual surrounding environment.

[0034] For convenience, the camera CAM at timing T1 will be referred to as the first camera CAM1, and the camera CAM at timing T2 will be referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the combination of the position and orientation of the first camera CAM1 at timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the combination of the predicted position and orientation of the second camera CAM2 at timing T2.

[0035] The remote support system 1 acquires camera information CINF related to the camera CAM mounted on the vehicle 100. The camera information CINF includes installation information and performance information of the camera CAM. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. Because the camera CAM is fixed to the vehicle 100, by using the installation information of the camera CAM, the direction and amount of movement of the vehicle 100 can be converted into the direction and amount of movement of the camera CAM in the camera coordinate system. In other words, the change in the viewpoint of the camera CAM can be estimated based on the installation information of the camera CAM and the direction and amount of movement of the vehicle 100.

[0036] The remote support system 1 also calculates the difference between the first viewpoint and the second viewpoint based on the above camera information CINF (installation information) and the predicted movement amount Lv.

[0037] Image IMG1 can be said to be an image IMG captured from a first viewpoint, i.e., an image IMG viewed from the first viewpoint. An image IMG expected to be captured from a second viewpoint, i.e., an image IMG expected to be seen from the second viewpoint, will be referred to as "image IMG2" hereinafter. The remote support system 1 converts image IMG1 viewed from the first viewpoint into image IMG2 viewed from the second viewpoint based on the difference between the first and second viewpoints. In other words, the remote support system 1 predicts (reads ahead) image IMG2 viewed from the second viewpoint based on image IMG1 viewed from the first viewpoint. Projective transformation is used for this look-ahead.

[0038] FIG. 6 is a conceptual diagram for explaining projective transformation. Projective transformation is performed based on perspective projection transformation. Perspective projection transformation is a rendering technique for rendering an object in three-dimensional space on a two-dimensional plane as seen from the camera CAM. To achieve this, perspective projection transformation projects points in three-dimensional space onto a projection plane P, taking into account the viewpoint of the camera CAM. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane perpendicular to the optical axis of the camera CAM. Note that points in three-dimensional space are defined in a three-dimensional world coordinate system (absolute coordinate system). On the other hand, points projected onto the projection plane P are defined in a two-dimensional image coordinate system.

[0039] For example, N virtual points are virtually set in a three-dimensional world coordinate system. N is an integer equal to or greater than 4. The N virtual points as viewed from a first camera CAM1 (first viewpoint) are projected onto a first projection plane P1 associated with the first camera CAM1 by perspective projection transformation. The N virtual points as viewed from a second camera CAM2 (second viewpoint) are projected onto a second projection plane P2 associated with the second camera CAM2 by perspective projection transformation. The second viewpoint is obtained from the difference between the first viewpoint and the second viewpoint. The image coordinates of the virtual points on the first projection plane P1 as viewed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the virtual points on the second projection plane P2 as viewed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on a comparison of the two, a projection transformation matrix H for converting from the first viewpoint to the second viewpoint is calculated. Then, the projective transformation matrix H is applied to the entire image IMG1 actually captured by the first camera CAM1, thereby generating an image IMG2 that is expected to be seen from the second viewpoint.

[0040] As another example, a method described in a non-patent document (Koda Matsubara, Manabu Ohmae, "Research on Delay Compensation of Camera Images for Remotely Controlling Vehicles Using Projection Transformation," 19th ITS Symposium 2021, 4-A-12, December 2021) may be used. Specifically, by inverse transformation of the perspective projection transformation, each image coordinate point on image IMG1 (projection plane P) is transformed into a world coordinate point in the world coordinate system. Based on the difference between the first and second viewpoints, the world coordinate points seen from the first viewpoint are transformed into world coordinate points seen from the second viewpoint. Then, by perspective projection transformation, the world coordinate points seen from the second viewpoint are returned to the projection plane P. This generates image IMG2 as expected to be seen from the second viewpoint. Note that according to the non-patent document, it is assumed that the ground surface S is reflected across the entire image IMG, as shown in FIG. 6.

[0041] FIG. 7 is a diagram showing an example of a composite image SIMG output from the display device 220 when the movement adjustment process of the vehicle 100 is performed. The annotation information OA1 and OA2 shown in FIG. 7 is the same as that described in FIG. 3. What is important in the description of FIG. 7 is that the composite image SIMG2 is generated based on the image IMG2. The generation of the composite image SIMG2 is performed when the predicted movement amount Lv exceeds the movement threshold value THLv. Therefore, by outputting the composite image SIMG2 from the display device 220, even when the movement amount of the vehicle 100 in the image delay time is large, it is possible to approximate the surrounding environment of the vehicle 100 shown in the composite image SIMG2 to the actual surrounding environment.

[0042] FIGS. 8 to 10 are diagrams for explaining the first to third examples of the moving body adjustment time. In the first example shown in FIG. 8, the time α (moving body adjustment time) is set to a time equal to the timing difference D1. However, in the first example, the execution of the movement adjustment process is switched when the predicted movement amount Lv changes so as to straddle the movement threshold value THLv. Therefore, the composite image SIMG may change significantly before and after the execution of the movement adjustment process, and the image of the surrounding environment of the vehicle 100 becomes unstable.

[0043] On the other hand, in the second example shown in FIG. 9 and the third example shown in FIG. 10, when the predicted movement amount Lv exceeds a preliminary threshold value (projection conversion threshold value) THpre (<THLv), the time α is set and the movement adjustment process is performed. Also, in the second and third examples, the time α is gradually changed (monotonically non-decreasing) from zero to the timing difference D1 from the preliminary threshold value THpre to the movement threshold value THLv. Therefore, the image of the surrounding environment of the vehicle 100 can be stabilized before and after the execution of the movement adjustment process.

[0044] 3-2. Moving Object Movement Adjustment Process In this section, we focus on the target delay time (timing difference D2 in FIG. 4). The problem related to the image delay time (i.e., timing difference D1 in FIG. 4) described in Section 3-1 above can also be considered when the target included in the composite image SIMG is a dynamic target. That is, when the target is a dynamic target such as a pedestrian, bicycle, or other vehicle, if the relative movement amount (referring to the relative movement amount in at least one of the forward / backward direction and the left / right direction; the same applies hereinafter) of the dynamic target with respect to the vehicle 100 at the timing difference D2 is large, the position and size of the dynamic target in the composite image SIMG1 may significantly deviate from the actual position and size of the dynamic target. Therefore, in this embodiment, a "projective transformation" is also performed on the image of the dynamic target included in image IMG1, taking into account the relative movement amount of the dynamic target at the timing difference D2.

[0045] 11 is a conceptual diagram for explaining an outline of the movement adjustment process of a dynamic target by the remote support system 1. Image IMG1 is an image IMG that is actually captured at timing T1 by a camera CAM mounted on the vehicle 100. Image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 predicts the relative movement amount of the dynamic target at the timing difference D2 shown in FIG. 4 using optical flow.

[0046] Timing T2 shown in FIG. 11 is the target timing for look-ahead and is later than timing T1. The difference between timing T2 and timing T1 corresponds to the movement adjustment time (target adjustment time) of the dynamic target. If the predicted movement amount Lo of the dynamic target exceeds the movement threshold THLo, the remote support system 1 sets the time β equal to or less than the timing difference D2 shown in FIG. 4 as the "target adjustment time" and performs movement adjustment. Note that, like the movement threshold THLv, the movement threshold THLo can be set in advance as the movement amount at which a discrepancy is expected to occur between the position and size of the dynamic target shown in the composite image CIMG1 and the actual position and size of this dynamic target.

[0047] When adjusting the movement of the dynamic target, the remote operator terminal 200 extracts a partial image OBJ1 of the dynamic target from an image IMG1 acquired after timing T1 based on the recognition information OR. Then, using projective transformation, it estimates (predicts) a partial image OBJ2 of the dynamic target that will be captured in the future (future partial image) from the partial image OBJ1.

[0048] The remote support system 1 acquires camera information CINF related to the camera CAM mounted on the vehicle 100. Then, based on the installation information of the camera CAM included in the camera information CINF, it estimates a change in the viewpoint of the camera CAM. Up to this point, this is the same as the movement adjustment process of the vehicle 100 described in FIG.

[0049] In the target movement adjustment process, information regarding the relative movement of the dynamic target with respect to the vehicle 100 during the period from timing T1 to timing T2 (i.e., target adjustment time) is acquired. For example, the remote support system 1 estimates the relative movement amount of the dynamic target from timing T1 to timing T2 using optical flow. Then, the remote support system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information CINF (installation information) and the relative movement amount of the dynamic target during the target adjustment time. Based on this difference, the remote support system 1 converts the image IMG1 viewed from the first viewpoint into an image IMG2 viewed from the second viewpoint.

[0050] Fig. 12 is a diagram showing an example of a composite image SIMG output from the display device 220 when projective transformation is performed on a partial image of a dynamic target. The annotation information OA1 and OA2 shown in Fig. 12 are the same as those explained in Figs. 3 and 7. What is important in explaining Fig. 12 is that the position and size (solid line) of the annotation information OA2 (pedestrian) are different from the position and size (dashed line) of the annotation information OA2 in Fig. 7. The reason for this is that projective transformation is performed on the partial image of the dynamic target.

[0051] The object adjustment time (time β) may be set to a time equal to the timing difference D2, or may be set to a time shorter than the timing difference D2. When setting the object adjustment time to a time shorter than the timing difference D2, based on the same concept as the preliminary threshold THpre described in FIGS. 9 and 10, a preliminary threshold (projection conversion threshold) THpre (<THLo) smaller than the movement threshold THLo may be set. And in this case, the time β may be set so as to gradually change (monotonically non-decrease) from zero to the timing difference D2 from the preliminary threshold THpre to the movement threshold THLo. Thereby, the position and size of the dynamic object can be stabilized before and after the execution of the movement adjustment process.

[0052] 3-3. Combination of Movement Adjustment Processes The movement adjustment process of the vehicle 100 described in Section 3-1 and the movement adjustment process of the dynamic object described in Section 3-2 may be performed in combination. FIG. 13 is a diagram for explaining the case of combining the movement adjustment process of the vehicle 100 and the movement adjustment process of the dynamic object. As shown in FIG. 13, when the predicted movement amount Lv of the vehicle 100 is less than or equal to the movement threshold THLv and the predicted movement amount Lo of the dynamic object is less than or equal to the movement threshold THLo, the movement adjustment process is not performed. On the other hand, when the predicted movement amount Lv exceeds the movement threshold THLv and the predicted movement amount Lo exceeds the movement threshold THLo, the movement adjustment processes of the vehicle 100 and the dynamic object are performed.

[0053] When the predicted movement amount Lv is less than or equal to the movement threshold THLv and the predicted movement amount Lo exceeds the movement threshold THLo, only the movement adjustment process of the dynamic object is performed. When the predicted movement amount Lo is less than or equal to the movement threshold THLo and the predicted movement amount Lv exceeds the movement threshold THLv, only the movement adjustment process of the vehicle 100 is performed. Thus, the two types of movement adjustment processes described above can be appropriately executed based on the comparison between the predicted movement amount and the movement threshold.

[0054] 4. Configuration Example of Vehicle 4-1. Configuration Example 14 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0055] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300.

[0056] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera (CAM), a LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0057] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates braking force.

[0058] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processors 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processors 160 perform various processes. Examples of the processors 160 include a general-purpose processor, a specific-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, and / or a combination thereof. The storage device 170 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), a solid-state drive (SSD), etc. The control device 150 may include one or more electronic control units (ECUs). In general terms, the control device 150 can also be called a processing circuitry.

[0059] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 may be realized by cooperation between the processor 160, which executes the vehicle control program PROG1, and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0060] 4-2. Sensor detection information The control device 150 acquires sensor detection information SEN using the sensor group 120. The sensor detection information SEN is stored in the storage device 170. The sensor detection information SEN includes an image IMG, vehicle state information, position information, target information, etc. The image IMG is captured by a camera CAM. The vehicle state information indicates the state of the vehicle 100 (e.g., speed, steering angle, etc.) detected by the vehicle state sensor. The position information indicates the position and orientation of the vehicle 100 detected by the position sensor.

[0061] The target information is information relating to targets around the vehicle 100. Examples of targets around the vehicle 100 include pedestrians, bicycles, motorcycles, other vehicles (leading vehicles, vehicles running alongside, following vehicles, etc.), white lines, road structures (e.g., curbs, guardrails), poles, traffic lights, signs, etc. The control device 150 can recognize targets around the vehicle 100 by using a recognition sensor. For example, by analyzing an image (IMG), it is possible to identify targets and calculate the relative position of the targets. It is also possible to identify targets and obtain the relative position and relative speed of the targets based on point cloud information obtained by LIDAR. The target information includes the relative position of the targets with respect to the vehicle 100. The target information may further include the relative speed of the targets.

[0062] 4-3.Vehicle driving control The control device 150 executes vehicle driving control to control the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes vehicle driving control by controlling the driving device 130 (steering device, drive device, and brake device).

[0063] The control device 150 may perform automatic driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the sensor detection information SEN. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0064] 4-4.Remote support related processes When remote support of the vehicle 100 is performed, the control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0065] The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. Typically, the control device 150 transmits an image IMG to the remote operator terminal 200. The control device 150 may transmit vehicle state information to the remote operator terminal 200. The control device 150 may transmit target object information to the remote operator terminal 200.

[0066] Furthermore, the control device 150 receives remote support information OPE from the remote operator terminal 200. The remote support information OPE is information related to remote support by the remote operator O. For example, the remote support information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle driving control in accordance with the received remote support information OPE.

[0067] 4-5.Camera information The camera information CINF includes installation information and performance information for each of one or more camera CAMs mounted on the vehicle 100. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. The camera information CINF is stored in the storage device 170. The control device 150 may transmit the camera information CINF to the remote operator terminal 200.

[0068] 5. Example of remote operator terminal configuration 15 is a block diagram showing an example of the configuration of the remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, and an information processing device 250.

[0069] The communication device (communication circuit) 210 communicates with the vehicle 100 and the management device 300 .

[0070] The display device 220 displays various information for the remote operator O who provides remote support. In other words, the display device 220 presents various information to the remote operator O by displaying the various information.

[0071] The input device 230 is a member that the remote operator O operates when remotely supporting the vehicle 100. For example, the input device 230 includes remote support members, such as a steering wheel, an accelerator pedal, a brake pedal, and a turn signal.

[0072] The information processing device 250 controls the remote operator terminal 200. The information processing device 250 includes one or more processors 260 (hereinafter simply referred to as processors 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processor 260 executes various processes. Examples of the processor 260 include a general-purpose processor, a special-purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, and / or a combination thereof. The storage device 270 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a HDD, an SSD, etc. In general terms, the information processing device 250 can also be called a processing circuitry.

[0073] The remote support control program PROG2 is a computer program executed by the processor 260. The functions of the information processing device 250 may be realized by cooperation between the processor 260 executing the remote support control program PROG2 and the storage device 270. The remote support control program PROG2 is stored in the storage device 270. Alternatively, the remote support control program PROG2 may be recorded on a computer-readable recording medium. The remote support control program PROG2 may be provided via a network.

[0074] The information processing device 250 communicates with the vehicle 100 via the communication device 210. The information processing device 250 receives sensor detection information SEN transmitted from the vehicle 100. The information processing device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the information processing device 250 presents an image IMG to the remote operator O by displaying the image IMG on the display device 220. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the presented information.

[0075] The remote operator O operates the input device 230. The amount of operation of the input device 230 is detected by a sensor installed on the input device 230. The information processing device 250 generates remote support information OPE that reflects the amount of operation of the input device 230 by the remote operator O. Then, the information processing device 250 transmits the remote support information OPE to the vehicle 100 via the communication device 210.

[0076] The information processing device 250 may receive the camera information CINF transmitted from the vehicle 100. The camera information CINF is stored in the storage device 270.

[0077] The information processing device 250 executes the projective transformation process (movement adjustment process) described in Section 3 above and the composite image generation process described in Section 2 above. The speed and steering angle of the vehicle 100 are obtained from the sensor detection information SEN. The steering angle in the steering operation by the remote operator O may be considered as the steering angle of the vehicle 100. Installation information and performance information of each camera CAM mounted on the vehicle 100 are obtained from the camera information CINF. Based on this information, the information processing device 250 executes the projective transformation process (movement adjustment process) described in Section 3 above. [Explanation of symbols]

[0078] 1. Remote support system 100 vehicles 200 Remote Operator Terminal 210 Communication equipment 220 Display device 250 Information Processing Equipment 300 Management device OA Annotation Information OR recognition information CAM camera CINF Camera Information IMG,IMG1,IMG2 images OBJ1,OBJ2 partial images SIMG1,SIMG2 composite image SEN Sensor detection information

Claims

1. 1. A device for remotely supporting a moving body by outputting a composite image from a display device, in which annotation information relating to a target object to be noted in an image obtained by a camera mounted on the moving body is superimposed on the image, the device comprising: a communication circuit connected to the mobile unit via a communication network; a processing circuit coupled to the communication circuit; The communication circuit receiving image data from the moving object in which the image and information on the timing of acquisition of the image by the camera are encoded; receiving target data from the moving body, the target data including recognition information of the target in the image, separately from the image data; The processing circuitry predicting a movement amount of the moving object in an image delay time indicating a timing difference between an acquisition timing of the image included in the image data and a decoding timing of the image data by the processing circuit; If the predicted movement amount of the moving object exceeds a movement threshold, a moving object adjustment time is set to be equal to or less than the image delay time; When the predicted movement amount of the moving object exceeds the movement threshold, the image included in the image data is projectively transformed onto a future image obtained from a camera viewpoint that is the moving object adjustment time ahead of the acquisition timing of the image, based on information about the movement of the moving object, to generate the composite image. A remote support device characterized by:

2. 2. The remote support device according to claim 1, The processing circuitry When the predicted movement amount of the moving object exceeds a projective transformation threshold that is lower than the movement threshold, projective transformation of the image onto the future image based on information about the movement of the moving object is started; and gradually increasing the moving object adjustment time from zero to the image delay time as the predicted movement amount of the moving object increases from the projective transformation threshold to the movement threshold. A remote support device characterized by:

3. 3. The remote support device according to claim 1, The processing circuitry calculating a predicted movement amount of the moving body in each of a forward / backward direction and a left / right direction of the moving body; When at least one of the predicted movement amount of the moving body in the forward / backward direction and the predicted movement amount of the moving body in the left / right direction exceeds the movement threshold, a projective transformation of the image onto the future image is performed based on information about the movement of the moving body. A remote support device characterized by:

4. 2. The remote support device according to claim 1, the target recognition information includes information on the acquisition timing of the image used to recognize the target, The processing circuitry further comprises: If the target is a dynamic target, predict a relative movement amount of the dynamic target with respect to the moving body during a target delay time indicating a timing difference between an acquisition timing of the image used to recognize the dynamic target and a reception timing of the target data by the processing circuit; If the predicted relative movement amount of the dynamic target exceeds a relative movement threshold, set a target adjustment time that is equal to or less than the target delay time; When the predicted relative movement amount of the dynamic target exceeds the relative movement threshold, the image included in the image data and the partial image of the dynamic target included in the image used to recognize the dynamic target are projectively transformed onto a future partial image obtained from a camera viewpoint that is the target adjustment time ahead of the acquisition timing of the image, based on information regarding the relative movement of the moving body of the dynamic target, to generate the composite image. A remote support device characterized by:

5. 6. The remote support device according to claim 5, The processing circuitry When the predicted relative movement amount of the dynamic target exceeds a projective transformation threshold that is lower than the relative movement threshold, starting a projective transformation of the partial image onto the future partial image based on information about the relative movement of the dynamic target; gradually increasing the target adjustment time from zero to the target delay time as the predicted relative movement of the dynamic target increases from the projective transformation threshold to the relative movement threshold. A remote support device characterized by:

6. 6. The remote support device according to claim 4, The processing circuitry calculating a predicted relative movement amount of the dynamic target in each of the forward / backward direction and the left / right direction of the moving body; When at least one of the predicted relative movement amount of the dynamic target in the forward / backward direction and the predicted movement amount of the dynamic target in the left / right direction exceeds the relative movement threshold, projective transformation of the partial image onto the future partial image is performed based on information about the relative movement of the dynamic target. A remote support device characterized by:

7. 1. A method for remotely supporting a mobile body by outputting, from a display device, a composite image in which annotation information relating to a target object to be noted in an image obtained by a camera mounted on the mobile body is superimposed on the image, the method comprising: receiving image data from the moving object in which the image and information on the timing of acquisition of the image by the camera are encoded; receiving target data from the moving body, separately from the image data, the target data including recognition information of the target in the image; predicting a movement amount of the moving object during an image delay time indicating a timing difference between an acquisition timing of the image included in the image data and a decoding timing of the image data; If the predicted movement amount of the moving object exceeds a movement threshold, setting a moving object adjustment time equal to or less than the image delay time; When the predicted movement amount of the moving object exceeds the movement threshold, the image included in the image data is projectively transformed onto a future image obtained from a camera viewpoint that is the moving object adjustment time ahead of the acquisition timing of the image, based on information about the movement of the moving object, to generate the composite image. A remote support method comprising:

8. A program for remotely supporting a mobile body by causing a computer to function to output from a display device a composite image in which annotation information relating to a target object that should be noted in an image obtained by a camera mounted on the mobile body is superimposed on the image, the program comprising: receiving image data from the moving object in which the image and information on the timing of acquisition of the image by the camera are encoded; receiving target data from the moving body, separately from the image data, the target data including recognition information of the target in the image; predicting a movement amount of the moving object during an image delay time indicating a timing difference between an acquisition timing of the image included in the image data and a decoding timing of the image data; If the predicted movement amount of the moving object exceeds a movement threshold, setting a moving object adjustment time equal to or less than the image delay time; When the predicted movement amount of the moving object exceeds the movement threshold, based on information about the movement of the moving object, projectively transform the image included in the image data onto a future image obtained from a camera viewpoint that is the moving object adjustment time ahead of the acquisition timing of the image, thereby generating the composite image. A remote support program characterized by:

Images