Remote support device, remote support method, and remote support program

Abstract

To prevent a delay of output of a composite image in which a target recognition result is superimposed on a camera image from a remote support screen, in receiving the camera image and the target recognition result separately from a movable body to perform remote support of the movable body.SOLUTION: A remote support device sets an image delay compensation time α equal to or less than an image delay time indicating a timing difference D1 between a timing T1 at which an image IMG1 is acquired by a camera CAM, and a timing T3 at which a remote operator terminal 200 decodes the image IMG1. The remote support device performs, on the basis of information on the movement of a movable body, projective transformation of the image IMG1 into a future image IMG2 obtained in a camera view at a timing T2 later than the timing T1 by the image delay compensation time α. The remote support device creates a composite image SIMG1 on the basis of the future image IMG2 and recognition information OR of a target in the image IMG1, and outputs the composite image from a display.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a technology for remotely supporting a mobile object using images acquired by a camera mounted on the mobile object. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2022-159912 discloses a system for remotely monitoring a vehicle. In this system, first and second data are separately transmitted from the vehicle to a remote server. The first data includes an image (camera image) acquired by a camera mounted on the vehicle and the timing of acquisition of the camera image. The second data includes a target recognition result based on the camera image and the timing of acquisition of the camera image. Based on the acquisition timing of the camera image included in the first and second data, the remote server extracts, in chronological order, camera images acquired at the same time and target recognition results based on the camera images. Then, a composite image in which the target recognition result is superimposed on the extracted camera image is output from a remote monitoring screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-159912 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above system, the difference between the timing of acquiring a camera image and the timing when this camera image becomes available for processing on the remote server can be referred to as the delay time of the camera image. The delay time of the camera image includes the time required for communication between the vehicle and the remote server and the time required for processing on the vehicle. A delay time equivalent to the delay time of the camera image also exists in the target recognition result. The delay time of the target recognition result can be referred to as the difference between the timing of acquiring the camera image that is the basis of the target recognition result and the timing when this target recognition result becomes available for processing on the remote server.

[0005] The problem here is that the timing when the camera image becomes available for processing on the remote server and the timing when the target recognition results based on the camera image become available for processing on the remote server do not match. This timing mismatch means that after one piece of information becomes available for processing, you have to wait until the other piece of information becomes available for processing. This leads to a delay in the output of the composite image from the remote monitoring screen.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and one objective of the present disclosure is to provide a technology for suppressing delays in output from a remote support screen of a composite image in which a target recognition result is superimposed on a camera image, when remote support for a moving object is provided by separately receiving a camera image and a target recognition result from the moving object. [Means for solving the problem]

[0007] A first aspect of the present disclosure is an apparatus for remotely supporting a moving object, which has the following features. The device remotely supports the mobile body by outputting a composite image from a display device in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that are noteworthy in the image. The device includes a communication circuit and a processing circuit, the communication circuit being connected to the mobile unit via a communication network, the processing circuit being coupled to the communication circuit. The communication circuit is configured to receive image data from the moving body in which the image and information on the timing of acquisition of the image by the camera are encoded, and to receive target data from the moving body separately from the image data, the target data including recognition information of the target in the image. The processing circuit is configured to set an image delay compensation time that is equal to or less than an image delay time that indicates the timing difference between the acquisition timing of the image included in the image data and the decoding timing of the image data by the processing circuit, and based on information regarding the movement of the moving body, projectively transform the image included in the image data into a future image obtained from a camera viewpoint that is the image delay compensation time ahead of the acquisition timing of the image, and generate the composite image based on the future image and recognition information of the target in the original image of the future image.

[0008] A second aspect of the present disclosure is a method for remotely supporting a moving object, which has the following features. The method remotely supports the mobile body by outputting from a display device a composite image in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that are noteworthy in the image. The method includes receiving image data from the moving body in which the image and information on the acquisition timing of the image by the camera are encoded; receiving target data from the moving body separately from the image data, the target data including recognition information of the target in the image; setting an image delay compensation time that is equal to or less than an image delay time that indicates the timing difference between the decoding timing of the image data and the acquisition timing of the image included in the image data; projecting the image included in the image data into a future image obtained from a camera viewpoint that is the image delay compensation time ahead of the acquisition timing of the image, based on information on the movement of the moving body; and generating the synthetic image based on the future image and the recognition information of the target in the original image of the future image.

[0009] A third aspect of the present disclosure is a program for remotely supporting a moving object, which has the following features. The program remotely supports the mobile body by causing a computer to function to output from a display device a composite image in which an image obtained by a camera mounted on the mobile body is superimposed with annotation information regarding objects that should be noted in the image. The program is configured to cause the computer to execute the following steps: receive image data from the moving body in which the image and information on the acquisition timing of the image by the camera are encoded; receive target data from the moving body separately from the image data, which includes recognition information of the target in the image; set an image delay compensation time that is equal to or less than an image delay time that indicates the timing difference between the decoding timing of the image data and the acquisition timing of the image included in the image data; projectively transform the image included in the image data into a future image obtained from a camera viewpoint that is the image delay compensation time ahead of the acquisition timing of the image, based on information on the movement of the moving body; and generate the composite image based on the future image and recognition information of the target in the original image of the future image. [Effects of the Invention]

[0010] According to the present disclosure, a processing circuit performs a projective transformation of an image. Projective transformation of an image generates a future image obtained from a camera viewpoint that is an image delay compensation time ahead of the image capture timing. Therefore, for example, before an image captured by a camera at a time TA can be processed by the processing circuit, a future image can be obtained based on an image captured by a camera at a time TB prior to the time TA, and a composite image can be generated based on this future image. Therefore, it is possible to suppress delays in output of the composite image from the display device. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a remote support system. [Figure 2] FIG. 10 is a diagram illustrating an example of data transmission from a vehicle to a remote operator terminal. [Figure 3] FIG. 10 is a diagram showing an example of a composite image output from a display device. [Figure 4] FIG. 10 is a diagram illustrating the delay time of an image and the delay time of a target. [Figure 5] FIG. 10 is a conceptual diagram for explaining an outline of image delay compensation processing. [Figure 6] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 7] 10A and 10B are diagrams illustrating an example of a composite image output from a display device when a projective transformation is performed on an image. [Figure 8] FIG. 10 is a conceptual diagram for explaining an outline of a target movement adjustment process. [Figure 9] FIG. 10 is a diagram showing an example of a composite image output from a display device when a projective transformation is performed on a partial image of a target. [Figure 10] FIG. 1 is a block diagram showing an example of the configuration of a vehicle. [Figure 11] FIG. 2 is a block diagram showing an example of the configuration of a remote operator terminal. DETAILED DESCRIPTION OF THE INVENTION

[0012] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0013] 1. Remote support system Consider remote support (remote driving) of a moving object. Examples of moving objects that can be remotely supported include vehicles, robots, flying objects, etc. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones, etc. As an example, in the following explanation, consider a case where the moving object is a vehicle. When generalizing, "vehicle" in the following explanation should be read as "moving object."

[0014] FIG. 1 is a schematic diagram showing an example of the configuration of a remote support system 1 according to this embodiment. The remote support system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote support. The remote operator terminal 200 is a terminal device used by a remote operator O when remotely supporting the vehicle 100. The remote operator terminal 200 can also be called a remote support HMI (Human Machine Interface). The management device 300 manages the remote support system 1. Typically, the management device 300 is a management server on the cloud. The management server may be composed of multiple servers that perform distributed processing.

[0015] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 may communicate directly without going through the management device 300.

[0016] The vehicle 100 is equipped with various sensors including a camera CAM. The camera CAM captures images of the surroundings of the vehicle 100 and acquires images IMG showing the situation around the vehicle 100. The sensor detection information SEN includes information obtained by the various sensors. The sensor detection information SEN includes at least the image IMG captured by the camera CAM and recognition information OR of a target (object) that should be noted in this image IMG. The sensor detection information SEN may also include the position and status of the vehicle 100 (e.g., speed, steering angle, etc.). The vehicle 100 transmits the sensor detection information SEN to the remote operator terminal 200.

[0017] The remote operator terminal 200 receives the sensor detection information SEN transmitted from the vehicle 100. The remote operator terminal 200 presents the sensor detection information SEN to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device 220, and displays information such as an image IMG on the display device 220. The remote operator O looks at the displayed information, recognizes the situation around the vehicle 100, and provides remote support for the vehicle 100. In other words, the remote operator O provides remote support for the vehicle 100 by displaying information for the remote operator O on the display device 220.

[0018] The remote support information OPE is information related to remote support by a remote operator O. For example, the remote support information OPE includes the amount of operation by the remote operator O. The remote operator terminal 200 transmits the remote support information OPE to the vehicle 100. The vehicle 100 receives the remote support information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle driving control in accordance with the received remote support information OPE. In this way, remote support for the vehicle 100 is realized.

[0019] 2. Composite image generation process In this embodiment, a composite image SIMG is generated in the remote operator terminal 200. The composite image SIMG is generated based on an image IMG included in the sensor detection information SEN and recognition information OR of a target object to be noted in this image IMG. The image IMG and the recognition information OR are transmitted separately from the vehicle 100 to the remote operator terminal 200.

[0020] FIG. 2 is a diagram illustrating an example of transmission of images IMG and recognition information OR from vehicle 100 to remote operator terminal 200. Image data including a set of images IMG (i.e., video) acquired within a certain period of time is transmitted from vehicle 100 to remote operator terminal 200. This certain period of time corresponds to the transmission interval of the image data. In the example shown in FIG. 2, the set of images IMG1 acquired at time T1 includes images IMG1(T1a), IMG1(T1b), and IMG1(T1c). Images IMG1(T1a), IMG1(T1b), and IMG1(T1c) were acquired at times T1a, T1b, and T1c, respectively.

[0021] Target recognition processing is performed on each of images IMG1(T1a), IMG1(T1b), and IMG1(T1c). Image analysis techniques such as pattern matching and deep learning are used for the recognition processing. When the recognition processing is performed, recognition information OR(T1a), OR(T1b), and OR(T1c) are generated. The recognition information OR includes the timing (timestamp) at which the image IMG1 on which the recognition processing was performed was acquired by the camera CAM. If a notable target is recognized in image IMG1, the coordinates, size, and type of this target in image IMG1 are added to the recognition information OR. In other words, if no notable target is recognized in image IMG1, the recognition information OR includes only timing information.

[0022] Target data including recognition information OR is transmitted from vehicle 100 to remote operator terminal 200. The transmission of target data is performed every time recognition information OR is generated. The transmission of target data may be performed at regular intervals. This regular interval corresponds to the transmission interval of the target data.

[0023] The remote operator terminal 200, which has received the image data and target data, arranges the images IMG1 (i.e., images IMG1(T1a), IMG1(T1b), and IMG1(T1c)) in chronological order based on the timing (timestamp) at which the image IMG1 included in the image data was acquired by the camera CAM. The remote operator terminal 200 also identifies the image IMG1 to be combined with the recognition information OR based on the timing (timestamp) included in the recognition information OR of the target data.

[0024] The remote operator terminal 200 further refers to the coordinates, size, and type of the target included in the recognition information OR of the target data, and superimposes annotation information OA on the image IMG1 to be combined with the target. The annotation information OA is information indicating the position, size, and type of the target in the image IMG1. The information indicating the position and size of the target is a bounding box surrounding the target. The information indicating the type of the target is a character indicating the type.

[0025] A composite image SIMG1 is generated by superimposing annotation information OA on image IMG1. Since the original images IMG1 of composite image SIMG1 (i.e., images IMG1(T1a), IMG1(T1b), and IMG1(T1c)) are arranged in chronological order, a set of composite images SIMG1 (i.e., a video) is output from display device 220 by outputting composite images SIMG1 in chronological order.

[0026] Fig. 3 is a diagram showing an example of a composite image SIMG output from the display device 220. In the example shown in Fig. 3, annotation information OA1 and OA2 are output. Annotation information OA1 is a bounding box surrounding a green light and the words "green light." Annotation information OA2 is a bounding box surrounding a pedestrian and the words "pedestrian."

[0027] 3. Processing using projective transformation 3-1. Image delay compensation processing Remote support of the vehicle 100 involves a delay between the vehicle 100 and the remote operator terminal 200. The difference between the timing T1 when the image IMG1 is acquired by the camera CAM and the timing when the image IMG1 becomes processable by the remote operator terminal 200 can be said to be the image delay time. Also, the difference between the timing T1 when the image IMG1 in which a noteworthy target is recognized is acquired by the camera CAM and the timing when the recognition information OR becomes processable by the remote operator terminal 200 can be said to be the target delay time.

[0028] FIG. 4 is a diagram illustrating the image delay time and the target delay time. In the example shown in FIG. 4, image IMG1 is acquired at timing T1, then encoded and transmitted to the remote operator terminal 200. The reason for encoding the image is to reduce communication costs. In order for the encoded image IMG1 to be processable by the remote operator terminal 200, this image IMG1 must be decoded. Therefore, the image delay time is expressed as the timing difference D1 (=T3-T1) between timing T1 and timing T3, when the remote operator terminal 200 decodes image IMG1.

[0029] In contrast, the recognition information OR is generated and then transmitted to the remote operator terminal 200. Therefore, the recognition information OR can be processed by the remote operator terminal 200 at timing T4 when the recognition information OR (target data) is received by the remote operator terminal 200. Therefore, the delay time of the target is expressed as the timing difference D2 (=T4-T1) between this timing T4 and timing T1.

[0030] To generate the composite image SIMG1, image IMG1 and the recognition information OR of this image IMG1 are required. However, the timing T3 at which image IMG1 is decoded is later than the timing T4 at which the recognition information OR is received. Therefore, the recognition information OR must wait for the decoding of image IMG1 for a period of time equivalent to the timing difference D3 (=T4-T3) between timing T3 and timing T4.

[0031] Therefore, in this embodiment, visual delay compensation is performed on image IMG1, taking into account the delay time (timing difference D1) of image IMG1. In particular, the remote support system 1 according to this embodiment performs visual delay compensation on image IMG1 by using "projective transformation." The subject of the delay compensation process is, for example, the remote operator terminal 200. However, the subject of the delay compensation process is not limited to the remote operator terminal 200. At least a part of the delay compensation process may be performed by the vehicle 100 or the management device 300.

[0032] 5 is a conceptual diagram for explaining an overview of image delay compensation processing by the remote support system 1. Image IMG1 is an image IMG that is actually captured at time T1 by a camera CAM mounted on the vehicle 100. Image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 acquires image IMG1 after time T1. If it is possible to estimate (predict) images IMG that will be captured in the future from image IMG1, it becomes possible to perform delay compensation.

[0033] Timing T2 is the target timing for look-ahead and is later than timing T1. The difference between timing T2 and timing T1 corresponds to the "delay compensation time." The remote support system 1 sets a time α equal to or less than the timing difference D1 shown in FIG. 4 as the image delay compensation time (image delay compensation time). In the first example, time α is set to a time equal to the timing difference D1. According to the first example, the composite image CIMG1 can be generated after the reception of the recognition information OR is completed (for example, at timing T4). In other words, it is possible to output the composite image CIMG1 at an earlier timing by eliminating the timing difference D3 (= T4 - T3) described in FIG. 4.

[0034] In the second example, the time α is set to a time that is equal to or greater than the timing difference D2 and less than the timing difference D1. According to the second example, it is possible to shorten the timing difference D3 (= T4 - T3) described in FIG. 4. In the third example, the time α is set to a time that is equal to or greater than the timing difference D3 and less than the timing difference D2. According to the third example, it is possible to generate the composite image CIMG1 after the timing at which reception of the recognition information OR is completed (i.e., at timing T4) and before timing T3. In either case, the remote support system 1 sets timing T2 to be after timing T1 by the image delay compensation time.

[0035] For convenience, the camera CAM at timing T1 will be referred to as the first camera CAM1, and the camera CAM at timing T2 will be referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the combination of the position and orientation of the first camera CAM1 at timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the combination of the predicted position and orientation of the second camera CAM2 at timing T2.

[0036] The remote support system 1 acquires camera information CINF related to the camera CAM mounted on the vehicle 100. The camera information CINF includes installation information and performance information of the camera CAM. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. Because the camera CAM is fixed to the vehicle 100, by using the installation information of the camera CAM, the direction and amount of movement of the vehicle 100 can be converted into the direction and amount of movement of the camera CAM in the camera coordinate system. In other words, the change in the viewpoint of the camera CAM can be estimated based on the installation information of the camera CAM and the direction and amount of movement of the vehicle 100.

[0037] More specifically, the remote support system 1 acquires information (movement direction and amount) regarding the movement of the vehicle 100 during the period from timing T1 to timing T2 (i.e., the image delay compensation time). For example, the remote support system 1 estimates the movement direction and amount of movement of the vehicle 100 during the period from timing T1 to timing T2 based on the speed and steering angle of the vehicle 100 at timing T1 and the image delay compensation time. Information on the speed and steering angle of the vehicle 100 is obtained from sensor detection information SEN provided by the vehicle 100. Alternatively, the steering angle in the steering operation by the remote operator O may be considered as the steering angle of the vehicle 100. The vehicle 100 may be assumed to make a steady circular turn. Then, the remote support system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information CINF (installation information) and the amount and direction of movement of the vehicle 100 during the image delay compensation time.

[0038] Image IMG1 can be said to be an image IMG captured from a first viewpoint, i.e., an image IMG viewed from the first viewpoint. An image IMG expected to be captured from a second viewpoint, i.e., an image IMG expected to be seen from the second viewpoint, will be referred to as "image IMG2" hereinafter. The remote support system 1 converts image IMG1 viewed from the first viewpoint into image IMG2 viewed from the second viewpoint based on the difference between the first and second viewpoints. In other words, the remote support system 1 predicts (reads ahead) image IMG2 viewed from the second viewpoint based on image IMG1 viewed from the first viewpoint. Projective transformation is used for this look-ahead.

[0039] FIG. 6 is a conceptual diagram for explaining projective transformation. Projective transformation is performed based on perspective projection transformation. Perspective projection transformation is a rendering technique for rendering an object in three-dimensional space on a two-dimensional plane as seen from the camera CAM. To achieve this, perspective projection transformation projects points in three-dimensional space onto a projection plane P, taking into account the viewpoint of the camera CAM. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane perpendicular to the optical axis of the camera CAM. Note that points in three-dimensional space are defined in a three-dimensional world coordinate system (absolute coordinate system). On the other hand, points projected onto the projection plane P are defined in a two-dimensional image coordinate system.

[0040] For example, N virtual points are virtually set in a three-dimensional world coordinate system. N is an integer equal to or greater than 4. The N virtual points as viewed from a first camera CAM1 (first viewpoint) are projected onto a first projection plane P1 associated with the first camera CAM1 by perspective projection transformation. The N virtual points as viewed from a second camera CAM2 (second viewpoint) are projected onto a second projection plane P2 associated with the second camera CAM2 by perspective projection transformation. The second viewpoint is obtained from the difference between the first viewpoint and the second viewpoint. The image coordinates of the virtual points on the first projection plane P1 as viewed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the virtual points on the second projection plane P2 as viewed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on a comparison of the two, a projection transformation matrix H for converting from the first viewpoint to the second viewpoint is calculated. Then, the projective transformation matrix H is applied to the entire image IMG1 actually captured by the first camera CAM1, thereby generating an image IMG2 that is expected to be seen from the second viewpoint.

[0041] As another example, a method described in a non-patent document (Koda Matsubara, Manabu Ohmae, "Research on Delay Compensation of Camera Images for Remotely Controlling Vehicles Using Projection Transformation," 19th ITS Symposium 2021, 4-A-12, December 2021) may be used. Specifically, by inverse transformation of the perspective projection transformation, each image coordinate point on image IMG1 (projection plane P) is transformed into a world coordinate point in the world coordinate system. Based on the difference between the first and second viewpoints, the world coordinate points seen from the first viewpoint are transformed into world coordinate points seen from the second viewpoint. Then, by perspective projection transformation, the world coordinate points seen from the second viewpoint are returned to the projection plane P. This generates image IMG2 as expected to be seen from the second viewpoint. Note that according to the non-patent document, it is assumed that the ground surface S is reflected across the entire image IMG, as shown in FIG. 6.

[0042] FIG. 7 is a diagram showing an example of a composite image SIMG output from display device 220 when a projective transformation is performed on an image. Annotation information OA1 and OA2 shown in FIG. 7 are the same as those described in FIG. 3. What is important in the description of FIG. 7 is that composite image SIMG2 is generated based on image IMG2. The generation of composite image SIMG2 is possible before the timing at which image IMG1 is decoded (i.e., timing T3 in FIG. 4). Therefore, by outputting composite image SIMG2 from display device 220, the delay in the output of composite image SIMG caused by the image delay time is compensated for.

[0043] 3-2. Dynamic target movement adjustment processing Section 3-1 above focuses on the image delay time. In this section, we focus on the target delay time (timing difference D2 in Figure 4). By performing image delay compensation processing, a future image IMG2 can be generated. However, if the target is a dynamic target such as a pedestrian, bicycle, or other vehicle, and the amount of movement of this dynamic target within the target delay time is large, the position and size of the dynamic target in image IMG2 may significantly deviate from the actual position and size of this dynamic target. Therefore, in this embodiment, a "projective transformation" is also performed on the image of the dynamic target included in image IMG1.

[0044] 8 is a conceptual diagram for explaining an outline of the movement adjustment process of a dynamic target by the remote support system 1. Image IMG1 is an image IMG that is actually captured at time T1 by a camera CAM mounted on the vehicle 100. Image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. Based on the recognition information OR, the remote operator terminal 200 extracts a partial image OBJ1 of the dynamic target from image IMG1 acquired after time T1. Then, using projective transformation, it estimates (predicts) a partial image OBJ2 of the dynamic target that will be captured in the future (future partial image) from the partial image OBJ1.

[0045] Timing T2 shown in FIG. 8 is a target timing for look-ahead, and is later than timing T1. The difference between timing T2 and timing T1 corresponds to the "movement adjustment time." The remote support system 1 sets a time β equal to or less than the timing difference D2 shown in FIG. 4 as the movement adjustment time of the dynamic target (target movement adjustment time). In the first example, time β is set to a time equal to the timing difference D2. According to the first example, it is possible to bring the position and size of the dynamic target at the time equivalent to the timing difference D2 closer to the actual position and size.

[0046] In the second example, the time β is set to a time less than the timing difference D2. Even in the second example, it is possible to prevent the position and size of the dynamic target at the time corresponding to the timing difference D2 from deviating significantly from the actual position and size. In either case, the remote support system 1 sets the timing T2 after the timing T1 by the target movement adjustment time.

[0047] The remote support system 1 acquires camera information CINF about the camera CAM mounted on the vehicle 100. Then, based on the installation information of the camera CAM included in the camera information CINF, it estimates the change in the viewpoint of the camera CAM. Up to this point, this is the same as the video delay compensation process described in FIG.

[0048] In the target movement adjustment process, information (relative movement direction and relative movement amount) regarding the relative movement of the dynamic target with respect to the vehicle 100 during the period from timing T1 to timing T2 (i.e., target movement adjustment time) is acquired. For example, the remote support system 1 estimates the relative movement direction and relative movement amount of the dynamic target from timing T1 to timing T2 using optical flow. Then, the remote support system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information CINF (installation information) and the relative movement amount and relative movement direction of the dynamic target during the target movement adjustment time. Based on this difference, the remote support system 1 converts the image IMG1 viewed from the first viewpoint into an image IMG2 viewed from the second viewpoint.

[0049] Fig. 9 is a diagram showing an example of a composite image SIMG output from the display device 220 when projective transformation is performed on a partial image of a dynamic target. The annotation information OA1 and OA2 shown in Fig. 9 are the same as those explained in Figs. 3 and 7. What is important in explaining Fig. 9 is that the position and size (solid line) of the annotation information OA2 (pedestrian) are different from the position and size (dashed line) of the annotation information OA2 in Fig. 7. The reason for this is that projective transformation is performed on the partial image of the dynamic target.

[0050] 4. Example of vehicle configuration 4-1.Configuration example 10 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0051] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300.

[0052] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera (CAM), a LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0053] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates braking force.

[0054] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processors 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processors 160 perform various processes. Examples of the processors 160 include a general-purpose processor, a specific-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, and / or a combination thereof. The storage device 170 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), a solid-state drive (SSD), etc. The control device 150 may include one or more electronic control units (ECUs). In general terms, the control device 150 can also be called a processing circuitry.

[0055] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 may be realized by cooperation between the processor 160, which executes the vehicle control program PROG1, and the storage device 170. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0056] 4-2. Sensor detection information The control device 150 acquires sensor detection information SEN using the sensor group 120. The sensor detection information SEN is stored in the storage device 170. The sensor detection information SEN includes an image IMG, vehicle state information, position information, target information, etc. The image IMG is captured by a camera CAM. The vehicle state information indicates the state of the vehicle 100 (e.g., speed, steering angle, etc.) detected by the vehicle state sensor. The position information indicates the position and orientation of the vehicle 100 detected by the position sensor.

[0057] The target information is information relating to targets around the vehicle 100. Examples of targets around the vehicle 100 include pedestrians, bicycles, motorcycles, other vehicles (leading vehicles, vehicles running alongside, following vehicles, etc.), white lines, road structures (e.g., curbs, guardrails), poles, traffic lights, signs, etc. The control device 150 can recognize targets around the vehicle 100 by using a recognition sensor. For example, by analyzing an image (IMG), it is possible to identify targets and calculate the relative position of the targets. It is also possible to identify targets and obtain the relative position and relative speed of the targets based on point cloud information obtained by LIDAR. The target information includes the relative position of the targets with respect to the vehicle 100. The target information may further include the relative speed of the targets.

[0058] 4-3.Vehicle driving control The control device 150 executes vehicle driving control to control the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes vehicle driving control by controlling the driving device 130 (steering device, drive device, and brake device).

[0059] The control device 150 may perform automatic driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the sensor detection information SEN. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0060] 4-4.Remote support related processes When remote support of the vehicle 100 is performed, the control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0061] The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. Typically, the control device 150 transmits an image IMG to the remote operator terminal 200. The control device 150 may transmit vehicle state information to the remote operator terminal 200. The control device 150 may transmit target object information to the remote operator terminal 200.

[0062] Furthermore, the control device 150 receives remote support information OPE from the remote operator terminal 200. The remote support information OPE is information related to remote support by the remote operator O. For example, the remote support information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle driving control in accordance with the received remote support information OPE.

[0063] 4-5.Camera information The camera information CINF includes installation information and performance information for each of one or more camera CAMs mounted on the vehicle 100. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. The camera information CINF is stored in the storage device 170. The control device 150 may transmit the camera information CINF to the remote operator terminal 200.

[0064] 5. Example of remote operator terminal configuration 14 is a block diagram showing an example of the configuration of the remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, and an information processing device 250.

[0065] The communication device (communication circuit) 210 communicates with the vehicle 100 and the management device 300.

[0066] The display device 220 displays various information for the remote operator O who provides remote support. In other words, the display device 220 presents various information to the remote operator O by displaying the various information.

[0067] The input device 230 is a member that the remote operator O operates when remotely supporting the vehicle 100. For example, the input device 230 includes remote support members, such as a steering wheel, an accelerator pedal, a brake pedal, and a turn signal.

[0068] The information processing device 250 controls the remote operator terminal 200. The information processing device 250 includes one or more processors 260 (hereinafter simply referred to as processors 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processor 260 executes various processes. Examples of the processor 260 include a general-purpose processor, a special-purpose processor, a CPU, a GPU, an ASIC, an FPGA, an integrated circuit, and / or a combination thereof. The storage device 270 stores various information. Examples of the storage device 170 include a volatile memory, a non-volatile memory, a HDD, an SSD, etc. In general terms, the information processing device 250 can also be called a processing circuitry.

[0069] The remote support control program PROG2 is a computer program executed by the processor 260. The functions of the information processing device 250 may be realized by cooperation between the processor 260 executing the remote support control program PROG2 and the storage device 270. The remote support control program PROG2 is stored in the storage device 270. Alternatively, the remote support control program PROG2 may be recorded on a computer-readable recording medium. The remote support control program PROG2 may be provided via a network.

[0070] The information processing device 250 communicates with the vehicle 100 via the communication device 210. The information processing device 250 receives sensor detection information SEN transmitted from the vehicle 100. The information processing device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the information processing device 250 presents an image IMG to the remote operator O by displaying the image IMG on the display device 220. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the presented information.

[0071] The remote operator O operates the input device 230. The amount of operation of the input device 230 is detected by a sensor installed on the input device 230. The information processing device 250 generates remote support information OPE that reflects the amount of operation of the input device 230 by the remote operator O. Then, the information processing device 250 transmits the remote support information OPE to the vehicle 100 via the communication device 210.

[0072] The information processing device 250 may receive the camera information CINF transmitted from the vehicle 100. The camera information CINF is stored in the storage device 270.

[0073] The information processing device 250 performs the projective transformation processing (delay compensation processing and movement adjustment processing) described in Section 3 above, and the synthetic image generation processing described in Section 2 above. The speed and steering angle of the vehicle 100 are obtained from the sensor detection information SEN. The steering angle in the steering operation by the remote operator O may be considered as the steering angle of the vehicle 100. Installation information and performance information of each camera CAM mounted on the vehicle 100 are obtained from the camera information CINF. Based on this information, the information processing device 250 performs the projective transformation processing (delay compensation processing and movement adjustment processing) described in Section 3 above. [Explanation of symbols]

[0074] 1. Remote support system 100 vehicles 200 Remote Operator Terminal 210 Communication equipment 220 Display device 250 Information Processing Equipment 300 Management device OA Annotation Information OR recognition information CAM camera CINF Camera Information IMG,IMG1,IMG2 images OBJ1,OBJ2 partial images SIMG1,SIMG2 composite image SEN Sensor detection information

Claims

1. 1. A device for remotely supporting a moving body by outputting a composite image from a display device, in which annotation information relating to a target object to be noted in an image obtained by a camera mounted on the moving body is superimposed on the image, the device comprising: a communication circuit connected to the mobile unit via a communication network; a processing circuit coupled to the communication circuit; The communication circuit receiving image data from the moving object in which the image and information on the timing of acquisition of the image by the camera are encoded; receiving target data from the moving body, the target data including recognition information of the target in the image, separately from the image data; The processing circuitry setting an image delay compensation time equal to or less than an image delay time indicating a timing difference between an acquisition timing of the image included in the image data and a decoding timing of the image data by the processing circuit; Based on information about the motion of the moving object, projectively transform the image included in the image data into a future image obtained from a camera viewpoint that is the image delay compensation time ahead of the acquisition timing of the image; and generating the composite image based on the future image and recognition information of the target in the original image of the future image. A remote support device characterized by:

2. 2. The remote support device according to claim 1, the target recognition information includes information on the acquisition timing of the image used to recognize the target, The processing circuit is configured to set the image delay compensation time to a time equal to or greater than a target delay time indicating a timing difference between the acquisition timing of the image used to recognize the target and the reception timing of the target data by the processing circuit. A remote support device characterized by:

3. 2. The remote support device according to claim 1, the target recognition information includes information on the acquisition timing of the image used to recognize the target, The processing circuit is configured to set the image delay compensation time to a time equal to or greater than the difference between a target delay time indicating a timing difference between a reception timing of the target data by the processing circuit and a timing of acquisition of the image used to recognize the target, and the image delay time. A remote support device characterized by:

4. 2. The remote support device according to claim 1, the target recognition information includes information on the acquisition timing of the image used to recognize the target, The processing circuitry further comprises: setting a target movement adjustment time equal to or less than a target delay time indicating a timing difference between an acquisition timing of the image used for recognizing the target and a reception timing of the target data by the processing circuit; If the target is a dynamic target, based on information on the relative movement of the dynamic target with respect to the moving body, a partial image of the dynamic target included in the image included in the image data and the image used to recognize the dynamic target is projectively transformed into a future partial image obtained from a camera viewpoint that is the target movement adjustment time ahead of the acquisition timing of the image, and generating the composite image based on the future partial image, the future image, and recognition information of the dynamic target in the original image of the future image. A remote support device characterized by:

5. 5. The remote support device according to claim 4, The processing circuitry is configured to set the image delay compensation time to a time equal to the target delay time. A remote support device characterized by:

6. 1. A method for remotely supporting a mobile body by outputting, from a display device, a composite image in which annotation information relating to a target object to be noted in an image obtained by a camera mounted on the mobile body is superimposed on the image, the method comprising: receiving image data from the moving object in which the image and information on the timing of acquisition of the image by the camera are encoded; receiving target data from the moving body, separately from the image data, the target data including recognition information of the target in the image; setting an image delay compensation time equal to or less than an image delay time indicating a timing difference between a decoding timing of the image data and an acquisition timing of the image included in the image data; projecting the image included in the image data into a future image obtained from a camera viewpoint that is the image delay compensation time ahead of the acquisition timing of the image based on information about the motion of the moving object; generating the composite image based on the future image and recognition information of the target in the original image of the future image. A remote support method comprising:

7. A program for remotely supporting a mobile body by causing a computer to function to output from a display device a composite image in which annotation information relating to a target object that should be noted in an image obtained by a camera mounted on the mobile body is superimposed on the image, the program comprising: receiving image data from the moving object in which the image and information on the timing of acquisition of the image by the camera are encoded; receiving target data from the moving body, separately from the image data, the target data including recognition information of the target in the image; setting an image delay compensation time equal to or less than an image delay time indicating a timing difference between a decoding timing of the image data and an acquisition timing of the image included in the image data; projecting the image included in the image data into a future image obtained from a camera viewpoint that is the image delay compensation time ahead of the acquisition timing of the image based on information about the motion of the moving object; generating the composite image based on the future image and recognition information of the target in the original image of the future image. A remote support program characterized by:

Images