Remote image display method and remote image display system

Abstract

Provided is a remote image display method and a remote image display system that reduce the occurrence of a left-right adverse condition when an operator who performs remote operation based on a sensor image misjudges the front of a mobile body. A sensor image generated based on data obtained from a remote sensor is displayed on a display of a terminal that communicates with the remote sensor. The remote sensor includes a sensor mounted on the mobile body. The sensor image includes a mobile body image that represents an image generated based on data obtained from the sensor mounted on the mobile body. A right region and a left region with the front of the mobile body as a reference are defined in the mobile body image. Identification aids of the right and the left are respectively superimposed on the right region and the left region. The color of the identification aid of the right is different from the color of the identification aid of the left.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method and a system of displaying a sensor image generated based on data acquired from a remote sensor at a terminal in communication with the remote sensor. BACKGROUND

[0002] U.S. Patent No. 11034299 discloses a system provided with two electronic mirrors arranged on both sides of a driver's seat of a vehicle. The existing system outputs an image of a right side and a right rear of the vehicle from the right electronic mirror and an image of a left side and a left rear of the vehicle from the left electronic mirror. The existing system overlaps an enlarged image of another vehicle in a partial area containing an image of the other vehicle in a case where the other vehicle is recognized to approach from the rear of the vehicle.

[0003] PRIOR ART DOCUMENTS

[0004] Patent Document 1: U.S. Patent No. 11034299

[0005] Patent Document 2: Japanese Patent Application Publication No. 2020-72401

[0006] Patent Document 3: Japanese Patent Application Publication No. H9-271016

[0007] Patent Document 4: Japanese Patent Application Publication No. 2020-161039

[0008] Patent Document 5: Japanese Patent Application Publication No. 2011-35729 SUMMARY

[0009] PROBLEMS TO BE SOLVED BY THE INVENTION

[0010] According to the existing system, a driver seated at the driver's seat can intuitively grasp which of the right and left rear of the vehicle the other vehicle is approaching from. The reason is because the driver understands that the electronic mirrors have the function of side mirrors and can instantly understand the position of the other vehicle depending on the arrangement position of the electronic mirror outputting an image containing the other vehicle.

[0011] Consider a scenario of remote operation of a mobile body represented by a vehicle. In the remote operation, a sensor image is generated based on data acquired from a sensor of the mobile body, and the image is provided to an operator. As the sensor image, an image of a front center and an image of a rear center of the mobile body can be exemplified. Typically, the images of the front center and the rear center are generated based on data acquired from a camera mounted on the mobile body.

[0012] It is a problem here that the operator can erroneously judge that the right region of the image of the rear center belongs to the "left" of the mobile body and the left region of the image belongs to the "right" of the mobile body when taking the front of the mobile body as a reference. In addition, the operator can erroneously judge that the right region of the image of the front center belongs to the "right" of the mobile body and the left region of the image belongs to the "left" of the mobile body when confusing the image of the front center and the image of the rear center.

[0013] An object of the present disclosure is to provide a technology capable of mitigating the generation of the adverse condition of the operator of remote operation based on a sensor image erroneously judging right and left when taking the front of a mobile body as a reference.

[0014] Technical solution for solving the problem

[0015] The first viewpoint is a method of displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal in communication with the remote sensor, having the following features.

[0016] The remote sensor includes a sensor mounted on a mobile body.

[0017] The sensor image includes a mobile body image representing an image generated based on data acquired from a sensor mounted on the mobile body.

[0018] The method includes:

[0019] a step of setting a reference line extending in the front-rear direction of the mobile body on an imaginary horizontal plane passing through a reference point of the mobile body;

[0020] a step of defining, using the reference line, a right space and a left space with the front of the mobile body as a reference for an imaginary space including the imaginary horizontal plane;

[0021] a step of setting, using the imaginary space in which the right space and the left space are defined, a right region and a left region with the front of the mobile body as a reference for the mobile body image;

[0022] a step of superimposing, in the right region and the left region of the mobile body image, a right recognition aid and a left recognition aid, respectively, the color of the right recognition aid being different from the color of the left recognition aid; and

[0023] a step of outputting, from the display, the mobile body image superimposed with the right and left recognition aids.

[0024] The second viewpoint has the following features in addition to the first viewpoint.

[0025] The right and left recognition assistance includes coloring performed on a circular arc drawn in a concentric circle shape on the virtual horizontal plane with a reference point of the moving body as a center, coloring performed on at least a part of each of the right region and the left region, coloring performed on a static target recognized in the moving body image, coloring performed on a bounding box that encloses a dynamic target recognized in the moving body image, coloring performed on a trajectory of the dynamic target, and coloring performed on a trajectory of the moving body.

[0026] The third aspect further has the following feature in the second aspect.

[0027] The right and left recognition assistance is a combination of at least one selected from coloring performed on the circular arc, coloring performed on at least a part of each of the right region and the left region, coloring performed on the static target, and coloring performed on the trajectory of the moving body, and at least one selected from coloring performed on the bounding box and coloring performed on the trajectory of the dynamic target.

[0028] The fourth aspect further has the following feature in the second aspect or the third aspect.

[0029] The method further includes:

[0030] a step of determining whether a dynamic target that crosses a boundary that separates the right region from the left region is recognized in the moving body image; and

[0031] in a case where it is determined that a dynamic target that crosses the boundary is recognized in the moving body image, a step of making the bounding boxes that enclose the dynamic target that crosses the boundary into two layers, and applying the colors of the right and left recognition assistance to the bounding boxes, respectively.

[0032] The fifth aspect further has the following feature in the second aspect or the third aspect.

[0033] The method further includes:

[0034] a step of determining whether a dynamic target that crosses a boundary that separates the right region from the left region is recognized in the moving body image; and

[0035] in a case where it is determined that a dynamic target that crosses the boundary is recognized in the moving body image, a step of applying a color different from any color of the right and left recognition assistance to the bounding boxes that enclose the dynamic target that crosses the boundary.

[0036] The sixth aspect further has the following feature in any one of the second aspect to the fifth aspect.

[0037] The method further includes:

[0038] a step of determining whether the total number of the dynamic objects is above an upper limit; and

[0039] in a case where it is determined that the total number is above the upper limit, a step of selecting the dynamic object to which the bounding box is to be applied in accordance with a predetermined criterion.

[0040] The seventh aspect further includes the following feature in any of the first through sixth aspects.

[0041] The moving body image includes a front image and a rear image of the moving body.

[0042] The step of superimposing the right and left recognition aids on the moving body image includes:

[0043] a step of setting a color of the right recognition aid superimposed on the front image and a color of the right recognition aid superimposed on the rear image to be the same color of a color system; and

[0044] a step of setting a color of the left recognition aid superimposed on the front image and a color of the left recognition aid superimposed on the rear image to be the same color of a color system.

[0045] The eighth aspect further includes the following feature in the seventh aspect.

[0046] The method further includes:

[0047] a step of determining whether the number of the dynamic objects recognized in the right and left regions of the front image is one or more;

[0048] a step of determining whether a dynamic object is recognized in at least one of the right and left regions of the rear image; and

[0049] in a case where it is determined that the number of the dynamic objects recognized in the right and left regions of the front image is one or more and a dynamic object is recognized in at least one of the right and left regions of the rear image, a step of using coloring performed on a bounding box that encloses the dynamic objects recognized in the front and rear images as the right and left recognition aids.

[0050] The ninth aspect further includes the following feature in any of the first through eighth aspects.

[0051] the boundary that separates the right region from the left region includes a boundary having a predetermined shape,

[0052] The method further includes a step of overlapping, on the boundary identification aid having the predetermined shape, an identification aid of the right side and an identification aid of the left side, the color of the identification aid of the right side and the color of the identification aid of the left side being different from the color of the identification aid of the right side and the color of the identification aid of the left side overlapped on the infrastructure image.

[0053] The tenth aspect has the following feature in addition to any one of the first through ninth aspects.

[0054] The sensor image includes an infrastructure image representing an image generated based on data obtained from an infrastructure camera at a distance of a predetermined value or less from the mobile body.

[0055] The terminal further includes a display that displays the infrastructure image.

[0056] The method further includes:

[0057] a step of setting, using the imaginary space defining the right side space and the left side space, a right side region and a left side region of the infrastructure image with reference to the front of the mobile body;

[0058] the step of overlapping the right side and the left side identification aids on the right side region and the left side region of the infrastructure image, the color of the identification aid of the right side overlapped on the right side of the infrastructure image being a color of the same color system as the color of the identification aid of the right side overlapped on the mobile body image, and the color of the identification aid of the left side overlapped on the left side of the infrastructure image being a color of the same color system as the color of the identification aid of the left side overlapped on the mobile body image; and

[0059] a step of outputting, from a display that displays the infrastructure image, the infrastructure image on which the right side and the left side identification aids are overlapped.

[0060] The eleventh aspect has the following feature in addition to the tenth aspect.

[0061] The method further includes:

[0062] a step of setting an additional reference line orthogonal to the reference line on the imaginary horizontal plane; and

[0063] a step of defining, using the additional reference line, a right front side space, a left front side space, a right rear side space, and a left rear side space by respectively dividing the right side space and the left side space in half in the front-rear direction of the mobile body.

[0064] In a case where the right front space, the left front space, the right rear space, and the left rear space are defined, in a step of setting, to the infrastructure image, right and left regions with the front of the mobile body as a reference, right front, left front, right rear, and left rear regions with the front of the mobile body as a reference are set to the infrastructure image.

[0065] In a case where the right front space, the left front space, the right rear space, and the left rear space are defined, in a step of overlapping, with the right and left regions of the infrastructure image, identification aids of right and left, identification aids of right front, left front, right rear, and left rear are overlapped in the right front, left front, right rear, and left rear regions, respectively. The color of the identification aid overlapped with the right front of the infrastructure image is different from the color of the identification aid of the right rear, and the color of the identification aid overlapped with the left front of the infrastructure image is different from the color of the identification aid of the left rear.

[0066] The 12th aspect has the following feature in addition to any one of the 1st to 11th aspects.

[0067] The terminal further includes a display that displays a map image of the surroundings of the mobile body.

[0068] The method further includes:

[0069] In a step of overlapping, with the map image, an icon of the mobile body and pictographs of right and left with the front of the mobile body as a reference, the color of the pictograph of the right is the same color system as the color of the identification aid overlapped with the right of the mobile body image, and the color of the pictograph of the left is the same color system as the color of the identification aid overlapped with the left of the mobile body image; and

[0070] A step of outputting, from a display that displays the map image, the map image in which the icon of the mobile body and the pictographs of right and left are overlapped.

[0071] The 13th aspect is a method of displaying, on a display of a terminal that communicates with a remote sensor, a sensor image generated based on data acquired from the remote sensor, and has the following feature.

[0072] The sensor image includes an infrastructure image that represents an image generated based on data acquired from an infrastructure camera that is within a predetermined distance from a mobile body.

[0073] The method includes:

[0074] setting a reference line extending in the front-rear direction of the moving body on an imaginary horizontal plane passing through a reference point of the moving body;

[0075] defining, using the reference line, a rightward space and a leftward space with reference to the front of the moving body in an imaginary space including the imaginary horizontal plane;

[0076] setting, using the imaginary space in which the rightward space and the leftward space are defined, a rightward region and a leftward region with reference to the front of the moving body in the infrastructure image;

[0077] overlapping the rightward and leftward recognition aids in the rightward region and the leftward region of the infrastructure image, respectively, the color of the rightward recognition aid being different from the color of the leftward recognition aid; and

[0078] outputting, from the display, the infrastructure image in which the rightward and leftward recognition aids are overlapped.

[0079] The 14th aspect is a system that displays a sensor image generated based on data acquired from a remote sensor on a display of a terminal that communicates with the remote sensor, having the following features.

[0080] The remote sensor includes a sensor mounted on a moving body.

[0081] The sensor image includes a moving body image that represents an image generated based on data acquired from a sensor mounted on the moving body.

[0082] The moving body,

[0083] setting a reference line extending in the front-rear direction of the moving body on an imaginary horizontal plane passing through a reference point of the moving body,

[0084] defining, using the reference line, a rightward space and a leftward space with reference to the front of the moving body in an imaginary space including the imaginary horizontal plane,

[0085] setting, using the imaginary space in which the rightward space and the leftward space are defined, a rightward region and a leftward region with reference to the front of the moving body in the moving body image,

[0086] overlapping the rightward and leftward recognition aids in the rightward region and the leftward region of the moving body image, respectively,

[0087] transmitting, to the terminal, the moving body image in which the rightward and leftward recognition aids are overlapped.

[0088] The color of the recognition aid on the right is different from the color of the recognition aid on the left.

[0089] The terminal outputs an image of the moving body from the display, which overlaps the right and left sides for recognition assistance.

[0090] The 15th point is a system that displays a sensor image generated based on data obtained from a remote sensor on a display of a terminal communicating with the remote sensor, and has the following characteristics.

[0091] The sensor images include infrastructure images, which represent images generated based on data obtained from infrastructure cameras located at a predetermined distance from the moving body.

[0092] The infrastructure camera sends images of the infrastructure to the terminal.

[0093] The terminal,

[0094] A reference line extending in the front-rear direction of the moving body is set on an imaginary horizontal plane passing through the reference point of the moving body.

[0095] Using the baseline, define a right space and a left space with the front of the moving body as the reference for the imaginary space containing the imaginary horizontal plane.

[0096] Using the hypothetical space that defines the right space and the left space, the infrastructure image is set with a right region and a left region based on the front of the moving body.

[0097] The right and left regions of the infrastructure image overlap with the right and left recognition aids, respectively.

[0098] The display outputs an image of the infrastructure with the right and left sides overlapping for recognition assistance.

[0099] The color of the recognition aid on the right is different from the color of the recognition aid on the left.

[0100] The effects of the invention

[0101] According to viewpoint 1 or viewpoint 14, the moving object image is defined with a right region and a left region based on the front of the moving object, and the moving object image is output from the display with colors that differ according to the right and left regions. Therefore, it can reduce the undesirable situation where the operator misidentifies the left and right sides of the moving object image based on the front of the moving object.

[0102] According to the second viewpoint, as the recognition assistance of the right and left, coloring performed on the circular arc, coloring performed on at least a part of each of the right region and the left region, coloring performed on the static target, coloring performed on the bounding box that surrounds the dynamic target, coloring performed on the trajectory of the dynamic target, and coloring performed on the trajectory of the moving body can be used.

[0103] In a case where no dynamic target is recognized in the moving body image, the bounding box that surrounds the dynamic target is not overlapped. Therefore, the operator can misrecognize the right and left when the front of the moving body is taken as a reference. The same problem also occurs in a case where the number of the recognized dynamic targets is small and in a case where the recognized dynamic targets are biased to the right region or the left region. In this regard, according to the third viewpoint, the recognition assistance of at least one of the bounding box whose recognition of the dynamic target is a prerequisite and the trajectory of the dynamic target is combined with at least one of the recognition assistance other than these. Therefore, the above problem can be prevented from occurring.

[0104] In a case where the bounding box that surrounds the dynamic target is used as the recognition assistance, what kind of bounding box is given to the dynamic target that crosses the boundary that separates the right region and the left region becomes a problem. In this regard, according to the fourth viewpoint, the color of the recognition assistance of the right and left can be respectively applied to the bounding box that is provided as two layers. In addition, according to the fifth viewpoint, a color different from any of the colors of the recognition assistance of the right and left can be applied to the bounding box that surrounds the dynamic target that crosses the boundary. Therefore, according to the fourth viewpoint or the fifth viewpoint, the appropriate bounding box can also be given to the dynamic target that crosses the boundary.

[0105] In a case where many dynamic targets are recognized in the moving body image, there is a problem that the bounding box that surrounds the dynamic targets can reduce the visual recognition. In this regard, according to the sixth viewpoint, in a case where it is determined that the total number of the dynamic targets is equal to or more than an upper limit, the dynamic target to which the bounding box is given can be selected in accordance with a predetermined criterion. Therefore, the number of the bounding boxes overlapped on the moving body image can be reduced to an appropriate number, and the reduction of the visual recognition caused by the bounding box can be suppressed.

[0106] According to the seventh viewpoint, in the front image and the rear image, the recognition assistance of the right is set to the color of the same color scheme, and the recognition assistance of the left is also set to the color of the same color scheme. Therefore, the occurrence of the adverse condition that the operator misrecognizes the right and left when the front of the moving body is taken as a reference can be reduced in the front image and the rear image.

[0107] The problem explained in the fourth or fifth aspect also applies to the dynamic target objects recognized in the front image and the rear image. For this, according to the eighth aspect, in the case where it is determined that the number of the dynamic target objects recognized in the right and left regions of the front image is one or more, respectively, and a dynamic target object is recognized in at least one of the right and left regions of the rear image, the bounding box is used as the recognition aid for the right and left. Thus, the bounding box superimposed on the front image and the rear image can be used to reduce the occurrence of the adverse condition of the operator misrecognizing the right and left when the front of the moving body is used as the reference.

[0108] According to the ninth aspect, even in the case where the boundary separating the right region and the left region has a predetermined shape, the recognition aid different from either of the colors of the recognition aids for the right and left can be applied to the boundary.

[0109] According to the tenth aspect, the recognition aids superimposed on the right and left of the moving body image are also superimposed on the infrastructure image. According to the tenth aspect, in addition, the color of the recognition aid superimposed on the right region of the infrastructure image is the same color system as the color of the recognition aid superimposed on the right region of the moving body image, and the color of the recognition aid superimposed on the left region of the infrastructure image is the same color system as the color of the recognition aid superimposed on the left region of the moving body image. Thus, it is expected to contribute to the accurate judgment of the operator who confirms both the moving body image and the infrastructure image, and to be useful for the safe movement of the moving body achieved by remote operation.

[0110] According to the eleventh aspect, the right front region, the left front region, the right rear region, and the left rear region with the front of the moving body as the reference are set to the infrastructure image, and the recognition aids for the right front, the left front, the right rear, and the left rear are superimposed on these regions, respectively. In addition, since the color of the recognition aid for the right front is different from the color of the recognition aid for the right rear, and the color of the recognition aid for the left front is different from the color of the recognition aid for the left rear, the infrastructure image can be given the information of the front and rear directions of the moving body. This is expected to contribute to the accurate judgment of the operator, and to be useful for the safe movement of the moving body achieved by remote operation.

[0111] According to the twelfth aspect, the icon of the moving body is superimposed on the map image, the pictogram of the right and left with the front of the moving body as the reference is surrounded by the icon, and the output is performed from the display. The color of the pictogram of the right is the same color system as the color of the recognition aid superimposed on the right of the moving body image. On the other hand, the color of the pictogram of the left is the same color system as the color of the recognition aid superimposed on the left of the moving body image. Thus, the immediate recognition of the front and rear of the moving body by the operator can be achieved.

[0112] According to the 13th aspect or the 15th aspect, the right region and the left region with reference to the front of the mobile body are set to the infrastructure image generated on the basis of the data acquired from the infrastructure camera at a distance of a predetermined value or less from the mobile body. According to the 13th aspect or the 15th aspect, in addition, the infrastructure image colored with colors different according to the right and the left is output from the display. Therefore, it is possible to reduce the occurrence of the undesirable situation in which the operator misrecognizes the right and the left with reference to the front of the mobile body in the infrastructure image. BRIEF DESCRIPTION OF DRAWINGS

[0113] Figure 1 is a diagram for explaining remote operation taken as a premise of the first embodiment.

[0114] Figure 2 is a diagram for explaining an example of a sensor image (mobile body image).

[0115] Figure 3 is a diagram for explaining a problem in remote operation.

[0116] Figure 4 is a diagram showing a first example of recognition assistance.

[0117] Figure 5 is a diagram showing a second example of recognition assistance.

[0118] Figure 6 is a diagram showing a third example of recognition assistance.

[0119] Figure 7 is a diagram showing a fourth example of recognition assistance.

[0120] Figure 8 is a diagram showing a fifth example of recognition assistance.

[0121] Figure 9 is a diagram showing a sixth example of recognition assistance.

[0122] Figure 10 is a block diagram showing a configuration example of a display system related to the first embodiment.

[0123] Figure 11 is a block diagram showing a configuration example of a function particularly associated with the first embodiment.

[0124] Figure 12 is a diagram for explaining a processing example of the right and left defining section shown in Figure 11 .

[0125] Figure 13 is a diagram showing a first modification of the fourth example of recognition assistance explained in Figure 7 .

[0126] Figure 14 is a view showing a second modification of the fourth example of the recognition assistance explained in Figure 7

[0127] Figure 15 is a view showing a third modification of the fourth example of the recognition assistance explained in Figure 7

[0128] Figure 16 is a view showing a fourth modification of the fourth example of the recognition assistance explained in Figure 7

[0129] Figure 17 is a view showing a remote operation taken as a premise of the second embodiment.

[0130] Figure 18 is a view showing examples of sensor images (moving body images and infrastructure images).

[0131] Figure 19 is a view showing a case where the first example of the recognition assistance explained in Figure 4

[0132] Figure 20 is a block diagram showing a configuration example of a display system to which the second embodiment is applied.

[0133] Figure 21 is a block diagram showing a configuration example of functions particularly associated with the second embodiment.

[0134] Figure 22 is a view showing a processing example of the right and left defining sections shown in Figure 21

[0135] Figure 23 is a view showing a processing example of the right and left regions in the infrastructure image.

[0136] Figure 24 is a view showing a processing example of superimposing various icons on the map image.

[0137] Explanation of Reference Signs

[0138] 1 moving body

[0139] 2 remote operation device (terminal)

[0140] 3, 3a, 3b infrastructure camera

[0141] 11, 11a, 11b sensor

[0142] 12, 22, 32 data processing device​​​​​

[0143] 12a, 22a, 32a processor

[0144] 12b, 22b, 32b memory

[0145] 21, 21a, 21b, 26, 26a, 26b, 29 display

[0146] 31, 31a, 31b camera body

[0147] AS_L1 to AS_L23, AS_L11a to AS_L11d, AS_R1 to AS_R23, AS_R11a, AS_R11c, AS_R11d, AS_B1 to AS_B6 recognition aid

[0148] BD1, BD2 boundary

[0149] BS boundary surface

[0150] ICN1, ICN3 icon

[0151] LA left region

[0152] LS left space

[0153] LFA left front region

[0154] LRA left rear region

[0155] MIM map image

[0156] OB1, OB2, OB3 dynamic object

[0157] OP operator

[0158] PG_L, PG_R pictogram

[0159] RA right region

[0160] RL reference

[0161] RS right space

[0162] RFA right front region

[0163] RRA right rear region

[0164] VHS virtual horizontal plane

[0165] VSP virtual space

[0166] SIM sensor image

[0167] SIM_1 moving body image

[0168] SIM 11a front image

[0169] SIM 11b rear image

[0170] SIM 3, SIM 3a, SIM 3b infrastructure image DETAILED DESCRIPTION

[0171] Hereinafter, a display method and a display system according to an embodiment of the present disclosure will be described with reference to the drawings. Further, the display method according to the embodiment is realized by a computer processing performed in the display system according to the embodiment. In each drawing, the same or equivalent portions are denoted by the same reference numerals, and the description thereof is simplified or omitted.

[0172] First Embodiment

[0173] First, reference will be made to Figures 1 to 16 The first embodiment of the present disclosure will be described.

[0174] 1. SUMMARY

[0175] 1-1. Remote Operation

[0176] Figure 1 is a diagram for explaining the remote operation which is a premise of the first embodiment. In Figure 1 A mobile body 1 and a remote operation device (hereinafter also referred to as "terminal") 2 are described in

[0177] As the remote operation, remote driving in which an operation directly related to the behavior (for example, running, stopping, turning, and the like) of the mobile body 1 is taken as a target can be exemplified. The remote operation also includes remote command and remote assistance. The remote command is performed taking an operation (turning on a light, and the like) not directly related to the behavior of the mobile body 1 as a target. The remote assistance is performed in order to assist the recognition and judgment of the mobile body 1.

[0178] The mobile body 1 and the terminal 2 communicate via a base station (not illustrated) of a network. For example, the data processing device 12 mounted on the mobile body 1 determines whether remote operation is required. Also, in a case where it is determined that remote operation is required, the data processing device 12 transmits a remote operation request to the terminal 2. The data processing device 12 transmits a sensor image SIM to the terminal 2 at the time of transmitting the remote operation request or before transmitting the remote operation request. The sensor image SIM is generated on the basis of data acquired from the sensor 11 mounted on the mobile body 1. Hereinafter, the sensor image SIM transmitted from the mobile body 1 is also referred to as "mobile body image SIM_1".

[0179] The data processing device 22 of the terminal 2 transmits various instructions INS to the mobile body 1 in response to the remote operation request. As the instructions INS transmitted from the data processing device 22 to the mobile body 1, an instruction based on the judgment of the operator OP can be exemplified. For example, the operator OP watches the sensor image SIM (mobile body image SIM_1) output from the display 21 of the terminal 2, and understands the content of the remote operation request. Also, the operator OP judges an action that the mobile body 1 should take, and inputs an instruction corresponding to the action to the terminal 2. As the action that the mobile body 1 should take, traveling, stopping, evasive avoidance with respect to an obstacle ahead, emergency retreat, and the like can be exemplified. The data processing device 22 generates an instruction INS corresponding to the input data from the operator OP, and transmits to the mobile body 1.

[0180] Figure 2 is a diagram illustrating an example of the mobile body image SIM_1. In Figure 2 a vehicle that is the mobile body 1 and sensors 11a and 11b mounted on the vehicle are depicted. Figure 2 The direction DR of the arrow illustrated is a traveling direction of the mobile body 1. The sensor 11a is a sensor 11 that detects a situation ahead of the mobile body 1, and the sensor 11b is a sensor 11 that detects a situation behind the mobile body 1. The sensors 11a and 11b are typically cameras, but can also generate the mobile body image SIM_1 on the basis of data acquired from a LIDAR (Laser Imaging Detection and Ranging). Therefore, the sensors 11a and 11b can also be LIDARs.

[0181] Figure 2The front image SIM_11a described in FIG. 1 is an example of the moving body image SIM_1 generated on the basis of data acquired from the sensor 11a. The front image SIM_11a includes an image of a dynamic target object OB1 (pedestrian) present on the right front shoulder (sidewalk) of the moving body 1. On the other hand, the rear image SIM_11b is an example of the moving body image SIM_1 generated on the basis of data acquired from the sensor 11b. The rear image SIM_11b includes an image of a dynamic target object OB2 (pedestrian) present on the left rear shoulder (sidewalk) of the moving body 1.

[0182] 1-2. Problems in remote operation

[0183] Figure 3 is a diagram for explaining the problems in remote operation. Figure 3 The two moving body images SIM_1 described above are combined. The front image SIM_11a is output to the display 21a, and the rear image SIM_11b is output to the display 21b. Figure 2 The operator OP who has watched the front image SIM_11a output to the display 21a can easily determine in which of the right front and left front of the moving body 1 the dynamic target object OB1 is present. On the other hand, the operator OP who has watched the rear image SIM_11b output to the display 21b can possibly misjudge in which of the right rear and left rear of the moving body 1 the dynamic target object OB2 is present.

[0184] The reason is that, as to whether the posture reflected in a mirror is recognized as a posture in which the left and right are reversed or as a posture in which neither is reversed, it is clarified from a psychological experiment that there are individual differences (Y. Takano et al., "Mirror reversal: Empirical tests of competing accounts", Quarterly Journal of Experimental Psychology Society, 2007, 60(11), 1555-1584). According to this psychological experiment, it is reported that the subjects who answered that the posture reflected in a mirror is reversed in the left and right are 53.9% of the whole, and the subjects who answered that neither is reversed are 45.1% of the whole.

[0185] The reason is that, as to whether the posture reflected in a mirror is recognized as a posture in which the left and right are reversed or as a posture in which neither is reversed, it is clarified from a psychological experiment that there are individual differences (Y. Takano et al., "Mirror reversal: Empirical tests of competing accounts", Quarterly Journal of Experimental Psychology Society, 2007, 60(11), 1555-1584). According to this psychological experiment, it is reported that the subjects who answered that the posture reflected in a mirror is reversed in the left and right are 53.9% of the whole, and the subjects who answered that neither is reversed are 45.1% of the whole.

[0186] That is, the result indicates that 53.9% of the operators OP who viewed the rear image SIM_11b were likely to recognize that the left and right of the rear image SIM_11b were reversed from the left and right of the front image SIM_11a, and 45.1% of the operators OP were likely to recognize that no such reversal occurred. Such individual differences are an obstacle to performing safe remote operation. In addition, in the case where the front image SIM_11a and the rear image SIM_11b are misread, the likelihood that the operators OP misrecognize the left and right of these moving body images SIM_1, when the front of the moving body 1 is taken as a reference, is high.

[0187] 1-3. Recognition assistance

[0188] In view of such a problem, in the first embodiment, the moving body image SIM_1 on which "recognition assistance" is overlaid is output from the display 21. The recognition assistance coloration overlaid on the moving body image SIM_1 has a color that differs depending on the left and right when the front of the moving body 1 is taken as a reference. Specifically, the recognition assistance coloration overlaid on the left region of the front image SIM_11a has the same color tone as the recognition assistance overlaid on the left region of the rear image SIM_11b, and the recognition assistance coloration overlaid on the right region of the front image SIM_11a has the same color tone as the recognition assistance overlaid on the right region of the rear image SIM_11b.

[0189] Here, the right region and the left region are determined by a boundary that is provided so as to extend in the front-rear direction of the moving body 1. The setting method of the right region and the left region will be described later. The same color tone refers to a combination of colors that have the same hue but different brightness or saturation, a combination of colors that have the same hue but different brightness and saturation (different in hue), or a combination of colors that have the same hue but are adjacent in hue (adjacent colors) and have the same brightness and saturation. Hereinafter, an example of the recognition assistance will be described.

[0190] 1-3-1. First example

[0191] Figure 4 is a view that shows the first example of the recognition assistance. In Figure 4 , two kinds of moving body images SIM_1 are described in the upper and lower portions, respectively. The front image SIM_11a is output from the display 21a, and the rear image SIM_11b is output from the display 21b. Up to this point, the description is the same as that of Figure 3

[0192] The front image SIM_11a and the rear image SIM_11b are each divided into a left region LA and a right region RA by a boundary BD1. In addition, the boundary BD1 is shown in Figure 4 ​In the first example, the right region RA of the front image SIM_11a overlaps the circular arc-shaped recognition aids AS_R1 to AS_R3, and the right region RA of the rear image SIM_11b also overlaps the circular arc-shaped recognition aids AS_R4 to AS_R6. These circular arcs are parts of the circumference of concentric circles described on an imaginary horizontal plane centered on a reference point of the moving body 1 (for example, the center position of the rear wheel axle of the vehicle, the position at which the sensor 11a or 11b is provided, or the position of the driver's seat). For example, the recognition aid AS_R1 and the recognition aid AS_R4 constitute the same circle, the recognition aid AS_R2 and the recognition aid AS_R5 constitute the same circle, and the recognition aid AS_R3 and the recognition aid AS_R6 constitute the same circle.

[0193] In the first example, the right region RA of the front image SIM_11a overlaps the circular arc-shaped recognition aids AS_R1 to AS_R3, and the right region RA of the rear image SIM_11b also overlaps the circular arc-shaped recognition aids AS_R4 to AS_R6. These circular arcs are parts of the circumference of concentric circles described on an imaginary horizontal plane centered on a reference point of the moving body 1 (for example, the center position of the rear wheel axle of the vehicle, the position at which the sensor 11a or 11b is provided, or the position of the driver's seat). For example, the recognition aid AS_R1 and the recognition aid AS_R4 constitute the same circle, the recognition aid AS_R2 and the recognition aid AS_R5 constitute the same circle, and the recognition aid AS_R3 and the recognition aid AS_R6 constitute the same circle.

[0194] The colors of the recognition aids AS_R1 to AS_R6 use the same color scheme, but the colors can be made uniform for the recognition aids that constitute the same circle. For example, the same color can be used for the color of the recognition aid AS_R1 and the color of the recognition aid AS_R4, the same color can be used for the color of the recognition aid AS_R2 and the color of the recognition aid AS_R5, and the same color can be used for the color of the recognition aid AS_R3 and the color of the recognition aid AS_R6. In this case, a color matching technique (gradation) in which at least one of the brightness and the saturation of the recognition aids is regularly changed according to the distance from the reference point of the moving body 1 can also be used.

[0195] In the first example, the right region RA of the front image SIM_11a overlaps the circular arc-shaped recognition aids AS_R1 to AS_R3, and the right region RA of the rear image SIM_11b also overlaps the circular arc-shaped recognition aids AS_R4 to AS_R6. These circular arcs are parts of the circumference of concentric circles described on an imaginary horizontal plane centered on a reference point of the moving body 1 (for example, the center position of the rear wheel axle of the vehicle, the position at which the sensor 11a or 11b is provided, or the position of the driver's seat). For example, the recognition aid AS_R1 and the recognition aid AS_R4 constitute the same circle, the recognition aid AS_R2 and the recognition aid AS_R5 constitute the same circle, and the recognition aid AS_R3 and the recognition aid AS_R6 constitute the same circle.

[0196] 1-3-2. Second Example

[0197] Figure 5 is a diagram showing a second example of a recognition aid. In Figure 5 , two moving body images SIM_1 are described in the upper and lower portions, respectively. Furthermore, the Figure 5The output manner of the front image SIM_11a and the rear image SIM_11b in the and the division of the left area LA and the right area RA based on the boundary BD1 are common to the 1st example.

[0198] In the 2nd example, the recognition aid AS_R7 that covers the entire right area RA of the front image SIM_11a is overlaid, and the recognition aid AS_R8 that covers the entire right area RA of the rear image SIM_11b is overlaid. The area covered by the recognition aid AS_R7 can also be a part of the right area RA of the front image SIM_11a, and the area covered by the recognition aid AS_R8 can also be a part of the right area RA of the rear image SIM_11b. In this case, the shape of the recognition aid AS_R7 and the recognition aid AS_R8 is not particularly limited, and can be a circle, an ellipse, a polygon, a star polygon, or a straight line (a solid line, a dotted line, a single-dot chain line, or the like). The colors of the recognition aid AS_R7 and the recognition aid AS_R8 use the same color system.

[0199] In the 2nd example, the recognition aid AS_R7 that covers the entire right area RA of the front image SIM_11a is overlaid, and the recognition aid AS_R8 that covers the entire right area RA of the rear image SIM_11b is overlaid. The area covered by the recognition aid AS_R7 can also be a part of the right area RA of the front image SIM_11a, and the area covered by the recognition aid AS_R8 can also be a part of the right area RA of the rear image SIM_11b. In this case, the shape of the recognition aid AS_R7 and the recognition aid AS_R8 is not particularly limited, and can be a circle, an ellipse, a polygon, a star polygon, or a straight line (a solid line, a dotted line, a single-dot chain line, or the like). The colors of the recognition aid AS_R7 and the recognition aid AS_R8 use the same color system.

[0200] 1-3-3. 3rd example

[0201] Figure 6 is a diagram that shows the 3rd example of the recognition aid. In Figure 6 In the and the in the , two kinds of moving body images SIM_1 are described in the upper part and the lower part, respectively. In addition, the output manner of the front image SIM_11a and the rear image SIM_11b in the and the division of the left area LA and the right area RA based on the boundary BD1 are common to the 1st example. Figure 6 In the and the in the , two kinds of moving body images SIM_1 are described in the upper part and the lower part, respectively. In addition, the output manner of the front image SIM_11a and the rear image SIM_11b in the and the division of the left area LA and the right area RA based on the boundary BD1 are common to the 1st example.

[0202] In the third example, the recognition aid AS_R9 that covers the lane recognized in the right area RA of the front image SIM_11a is overlaid, and the recognition aid AS_R10 that covers the lane recognized in the right area RA of the rear image SIM_11b is overlaid. These lanes are one example of the static target object included in the front image SIM_11a. The recognition of the static target object is performed by a publicly known method such as machine learning, for example. As other examples of the static target object, a guardrail, a road structure (a signal machine, a traffic sign), and the like can be given. The same color system color is used for the colors of the recognition aid AS_R9 and the recognition aid AS_R10.

[0203] In the third example, the recognition aid AS_R9 that covers the lane recognized in the right area RA of the front image SIM_11a is overlaid, and the recognition aid AS_R10 that covers the lane recognized in the right area RA of the rear image SIM_11b is overlaid. These lanes are one example of the static target object included in the front image SIM_11a. The recognition of the static target object is performed by a publicly known method such as machine learning, for example. As other examples of the static target object, a guardrail, a road structure (a signal machine, a traffic sign), and the like can be given. The same color system color is used for the colors of the recognition aid AS_R9 and the recognition aid AS_R10.

[0204] 1-3-4. Fourth Example

[0205] Figure 7 is a drawing that shows the fourth example of the recognition aid. In Figure 7 In the upper and lower portions, two mobile body images SIM_1 are described. In addition, the output method of the front image SIM_11a and the rear image SIM_11b in Figure 7 is common to the first example.

[0206] In the fourth example, the recognition aid AS_R11 (a bounding box) that surrounds the dynamic target object OB1 recognized in the right area RA of the front image SIM_11a is overlaid. In the fourth example, the recognition aid AS_L11 (a bounding box) that surrounds the dynamic target object OB2 recognized in the left area LA of the rear image SIM_11b is further overlaid. The recognition of the dynamic target object is performed by a publicly known method such as machine learning, for example. As examples of the dynamic target object, in addition to the pedestrian shown in Figure 7 , a bicycle, a motorcycle, and an automobile can be given.

[0207] The recognition assistance AS_L11 is colored with a color that is not the same color scheme as the recognition assistance AS_R11. The recognition assistance AS_L11 is colored with a color that is the same color scheme as the recognition assistance AS_R11 in a case where a dynamic object OB2 is recognized in the right area RA of the rear image SIM_11b.

[0208] 1-3-5. Fifth Example

[0209] Figure 8 is a diagram that shows the fifth example of the recognition assistance. In Figure 8 In the upper and lower portions, two mobile body images SIM_1 are described, respectively. Further, regarding the output manner of the front image SIM_11a and the rear image SIM_11b in Figure 8

[0210] In the fifth example, the trajectory of the mobile body 1 is overlaid on the front image SIM_11a and the rear image SIM_11b. The trajectory of the mobile body 1 overlaid on the front image SIM_11a is specifically the future trajectory of the mobile body 1 with reference to the current time point, which is generated based on, for example, vehicle control data including at least one of driving, braking, and steering of the mobile body 1. The trajectory of the mobile body 1 overlaid on the rear image SIM_11b is the past trajectory of the mobile body 1 with reference to the current time point, which is generated based on, for example, historical record data of the lateral position of the mobile body 1 with respect to a reference position set on the lane.

[0211] In the fifth example, the recognition assistance AS_R12 is overlaid on the trajectory of the mobile body 1 included in the right area RA of the front image SIM_11a, and the recognition assistance AS_L12 is overlaid on the trajectory of the mobile body 1 included in the left area LA of the image. The recognition assistance AS_L12 is colored with a color that is not the same color scheme as the recognition assistance AS_R12.

[0212] In the fifth example, the recognition assistance AS_R13 is additionally overlaid on the trajectory of the mobile body 1 included in the right area RA of the rear image SIM_11b, and the recognition assistance AS_L13 is overlaid on the trajectory of the mobile body 1 included in the left area LA of the image. The recognition assistance AS_L13 is colored with a color that is not the same color scheme as the recognition assistance AS_R13.

[0213] The color of the recognition assistance AS_L13 can use a color of the same color scheme as the color of the recognition assistance AS_L12, or can use the same color. The color of the recognition assistance AS_R13 can use a color of the same color scheme as the color of the recognition assistance AS_R12, or can use the same color. ​

[0214] The idea of the 5th example can also be applied to the trajectory of the dynamic target recognized in the moving body image SIM_1. In this case, the dynamic target included in the moving body image SIM_1 is recognized by a known method such as machine learning, and the past trajectory of the dynamic target is determined. Also, the past trajectory can be depicted in the moving body image SIM_1, and the recognition aid is overlaid along the trajectory. Further, the future trajectory of the dynamic target can also be predicted from the determined past trajectory. In this case, the future trajectory of the dynamic target is depicted in the moving body image SIM_1, and the recognition aid is overlaid along the trajectory.

[0215] 1-3-6. 6th Example

[0216] Figure 9 is a diagram showing the 6th example of the recognition aid. In Figure 9 In the upper and lower portions, two moving body images SIM_1 are depicted. As for Figure 9 The output method of the front image SIM_11a and the rear image SIM_11b in

[0217] Unlike the 1st to 5th examples, in the 6th example, a region shape is presented in which the boundary BD1 has a certain width in the left-right direction. In the 6th example, the recognition aids AS_B1 to AS_B3 in the shape of a circular arc are overlaid on the boundary BD1 of the front image SIM_11a, and the recognition aids AS_B4 to AS_B6 in the shape of a circular arc are overlaid on the boundary BD1 of the rear image SIM_11b. The recognition aid AS_B1 is located between the recognition aid AS_R1 and the recognition aid AS_L1. The recognition aids AS_B2 to AS_B6 are respectively located between the recognition aid AS_Rk and the recognition aid AS_Lk (k = 2 to 6).

[0218] The explanation of the recognition aids AS_B1 to AS_B6 is basically the same as that of the recognition aids AS_R1 to AS_R6. However, the recognition aids AS_B1 to AS_B6 are colored in a color that is not the same color system as the recognition aids AS_R1 to AS_R6, and also not the same color system as the recognition aids AS_L1 to AS_L6 (that is, a third color).

[0219] In the 6th example, the position of the boundary BD1 can also be moved in the left-right direction. For example, if the position of the driver's seat of the moving body 1 is set as the reference point of the moving body 1, the position of the boundary BD1 will be moved. This variation can also be applied to the 1st to 5th examples.

[0220] In the sixth example, the shape of the boundary BD1 can also be arbitrarily deformed. As other shapes of the boundary BD1, shapes such as a triangle and a trapezoid can be cited. For example, if the apex is set as the reference point of the mobile body 1 and the remaining two apexes of the triangle are set on the periphery of the front image SIM 11a and the rear image SIM 11b, respectively, it is possible to set the boundary BD1 having a triangle on these images. Further, by making the width in the left-right direction of the boundary BD1 gradually change from the front to the rear of the mobile body 1, it is possible to set the boundary BD1 having a trapezoid on these images.

[0221] Thus, according to the first embodiment, the recognition aid colored with colors different according to the left-right with the front of the mobile body 1 as the reference is overlaid on the mobile body image SIM 1, which is output from the display 21. Therefore, it is possible to reduce the occurrence of the undesirable situation that the operator OP misrecognizes the left-right in the mobile body image SIM 1 when the front of the mobile body 1 is taken as the reference.

[0222] 2. System configuration example

[0223] Next, a configuration example of the display system related to the first embodiment will be described.

[0224] 2-1. Overall configuration example

[0225] Figure 10 is a block diagram showing a configuration example of the display system related to the first embodiment. In Figure 10 In the example shown in FIG. 1, as the configuration of the display system related to the first embodiment, the mobile body 1 provided with the sensor 11 and the data processing device 12 and the terminal 2 provided with the display 21, the data processing device 22, and the input device 23 are described.

[0226] The sensor 11 is mounted on the mobile body 1. The sensor 11 detects the situation of at least the front and the rear of the mobile body 1. As the sensor 11, a camera and a LIDAR can be cited. As a specific sensor 11, the sensors 11a and 11b shown in FIG. 2 can be cited. However, the sensor 11a can also include a three-base sensor that detects the situation of the front center, the right front, and the left front of the mobile body 1. Further, the sensor 11b can also include a three-base sensor that detects the situation of the rear center, the right rear, and the left rear of the mobile body 1. Figure 2 Further, the sensor 11 can also include a sensor that detects the situation of the right side and the left side of the mobile body 1. In addition, the sensor that detects the situation of the right side (or the left side) of the mobile body 1 can also detect the situation of the right rear (or the left rear) of the mobile body 1.

[0227]

[0228] ​The data processing device 12 is a microcomputer mounted on the mobile body 1. The data processing device 12 is provided with at least one processor 12a and at least one memory 12b. The processor 12a includes a CPU (Central Processing Unit). The memory 12b is a volatile memory such as a DDR memory, and performs expansion of various programs used by the processor 12a and temporary storage of various data. Various data acquired from the sensor 11, the mobile body image SIM_1 transmitted from the mobile body 1 to the terminal 2, and the instruction INS received by the mobile body 1 from the terminal 2 are stored in the memory 12b.

[0229] The processor 12a performs various processes associated with generation of the mobile body image SIM_1 by executing a program for image processing stored in the memory 12b. In addition, the processor 12a performs various processes for transmitting the generated mobile body image SIM_1 to the terminal 2 by executing a program for image transmission stored in the memory 12b. Further, the processor 12a further performs various processes associated with control of the mobile body 1 based on the instruction INS received from the terminal 2 by executing a program for remote operation stored in the memory 12b.

[0230] The display 21 is a display device that outputs the mobile body image SIM_1. The display 21 is provided, for example, at a facility of a business that provides a remote operation service for the mobile body 1. The display 21 is connected to the data processing device 22 in a wired or wireless manner. In the latter case, the display 21 can be connected to the data processing device 22 via a base station of a network. The total number of the displays 21 is not particularly limited, but it is preferable that at least the images of the front and the rear of the mobile body 1 (for example, the front image SIM_11a and the rear image SIM_11b) are respectively output. Therefore, the total number of the displays 21 is preferably at least two.

[0231] The data processing device 22 is a computer for performing remote operation of the mobile body 1. The data processing device 22 is provided with at least one processor 22a and at least one memory 22b. The processor 22a includes a CPU (Central Processing Unit). The memory 22b is a volatile memory such as a DDR memory, and performs expansion of various programs used by the processor 22a and temporary storage of various data. The mobile body image SIM_1 received by the terminal 2 from the mobile body 1, various data from the input device 23, and the instruction INS transmitted from the terminal 2 to the mobile body 1 are stored in the memory 22b.

[0232] The processor 22a performs various processes associated with output control of the mobile body image SIM_1 by executing a program for output processing stored in the memory 22b. In addition, the processor 22a performs various processes for generating the instruction INS transmitted to the mobile body 1 by executing a program for remote operation stored in the memory 22b. The processor 22a further performs various processes for transmitting the generated instruction INS to the mobile body 1 by executing a program for instruction transmission.

[0233] The input device 23 is operated by the operator OP. The input device 23 has, for example, an input section operated by the operator OP, and a control circuit that generates and outputs a signal based on input data. As the input section, a mouse, a keyboard, a key, and a switch can be exemplified. As other examples of the input section, a steering wheel, a shift lever, an accelerator pedal, and a brake pedal can be exemplified. As a signal generated based on an operation input, a signal for changing (enlarging, reducing, and the like) a display mode of the mobile body image SIM_1 on the display 21, a signal corresponding to an action that the mobile body 1 should take, and the like can be exemplified.

[0234] 2-2. Functional Configuration Example

[0235] 2-2-1. Functional Configuration Example of Data Processing Device 12

[0236] Figure 11 is a block diagram showing a configuration example of functions particularly associated with the first embodiment. In the example shown in Figure 11 The data processing device 12 has a data reception section 13, an image generation section 14, a left-right definition section 15, an identification assistance superimposition section 16, and a data transmission section 17. The functions of each block shown in the data processing device 12 are realized by the processor 12a executing various programs. In addition, a part of the functions of the data processing device 12 (for example, the image generation section 14, the left-right definition section 15, and the identification assistance superimposition section 16) can also be realized by performing program processing by a server outside the mobile body 1.

[0237] The data reception section 13 receives various data detected by the sensor 11. The various data include data indicating a situation in front of the mobile body 1 and data indicating a situation behind the mobile body 1. The data indicating the situation in front can also include data indicating situations in the front center, the right front, and the left front of the mobile body 1. The data indicating the situation behind can also include data indicating situations in the back center, the right back, and the left back of the mobile body 1. The various data acquired by the sensor 11 can also include data indicating situations in the right side and the left side of the mobile body 1. The data reception section 13 transmits the various data to the image generation section 14.

[0238] The image generation section 14 generates a sensor image SIM (moving body image SIM_1) based on various data received from the data reception section 13. In the case where the sensor 11 is a camera, the image generation section 14 generates the moving body image SIM_1 based on a color image detected by the camera. In the case where the sensor 11 is a LiDAR, the image generation section 14 generates the moving body image SIM_1 based on point cloud data detected by the LIDAR. The image generation section 14 transmits the generated moving body image SIM_1 to the left-right definition section 15 and the recognition assistance superimposition section 16.

[0239] The left-right definition section 15 defines left and right in the moving body image SIM_1. The left-right definition section 15 first sets a boundary. Figure 12 is a diagram that explains a processing example of the left-right definition section 15. Furthermore, in Figure 12 , a processing example is explained on the premise that the moving body 1 is a vehicle. In Figure 12 the example shown in FIG. 10, a virtual horizontal surface VHS that passes through a reference point (for example, the center position of the rear wheel axle of the vehicle) of the moving body 1 is first set (step S1). The virtual horizontal surface VHS is, for example, a surface that is parallel to the lane on which the moving body 1 travels.

[0240] After the virtual horizontal surface VHS is set, a reference line RL that extends in the front-rear direction of the moving body 1 is set on the virtual horizontal surface VHS (step S1). The front-rear direction is, for example, a direction that connects the center of the front of the vehicle body of the moving body 1 with the center of the rear of the vehicle body. In the example shown in FIG. 10, the reference line RL is set by projecting a straight line that passes through these centers onto the virtual horizontal surface VHS. Figure 12

[0241] After the reference line RL is set, a plane that contains the reference line RL and is orthogonal to the virtual horizontal surface VHS (hereinafter also referred to as a "boundary surface BS") is set (step S2). After the boundary surface BS is set, it is possible to define a right space RS and a left space LS that are based on the front of the moving body 1 in a virtual space VSP that contains the boundary surface BS and the virtual horizontal surface VHS.

[0242] The left-right definition section 15 applies the virtual space VSP that contains the boundary surface BS and the virtual horizontal surface VHS to the moving body image SIM_1 (step S2). Then, it is possible to set a boundary BD1 that is drawn by the boundary surface BS on the moving body image SIM_1. When the boundary BD1 is set, it is possible to set a right region RA and a left region LA that are based on the front of the moving body 1 on the moving body image SIM_1.

[0243] ​The left-right defining section 15 transmits data of the right area RA and the left area LA in the moving body image SIM_1 to the recognition assistance superimposing section 16. The left-right defining section 15, for example, transmits data of pixel coordinates constituting the boundary BD1 in the moving body image SIM_1 to the recognition assistance superimposing section 16. In the case where the circular arc-shaped recognition assistance is superimposed on the moving body image SIM_1 (i.e., in the case of the first example explained above), the left-right defining section 15 transmits data of pixel coordinates constituting the virtual horizontal surface VHS in the moving body image SIM_1 to the recognition assistance superimposing section 16. Figure 4 In the case of the first example explained above, the left-right defining section 15 transmits data of pixel coordinates constituting the virtual horizontal surface VHS in the moving body image SIM_1 to the recognition assistance superimposing section 16.

[0244] The recognition assistance superimposing section 16 superimposes recognition assistance on the moving body image SIM_1 on the basis of the moving body image SIM_1 received from the image generating section 14 and various data associated with the moving body image SIM_1 received from the left-right defining section 15.

[0245] In the case of referring to the first example explained above, the recognition assistance superimposing section 16 uses the data of pixel coordinates constituting the virtual horizontal surface VHS in the moving body image SIM_1, and draws a virtual concentric circle centered on the reference point of the moving body 1 on the virtual horizontal surface VHS. Next, the recognition assistance superimposing section 16 applies the virtual concentric circle to the moving body image SIM_1, and superimposes the recognition assistance AS_R1 to AS_R6 on the right area RA and the recognition assistance AS_L1 to AS_L6 on the left area LA. Figure 4 In the case of referring to the second example explained above, the recognition assistance superimposing section 16 superimposes the recognition assistance AS_R7 and AS_R8 covering the entire right area RA and the recognition assistance AS_L7 and AS_L8 on the left area LA. Further, in order to suppress a decrease in visual recognition of the moving body image SIM_1 due to the superimposed recognition assistance, it is preferable to color the right area RA and the left area LA with a transmissive color in the case of the second example. In other examples, in the case where the color is not a transmissive color, it is preferable to change the transmittance of the color according to time.

[0246] Figure 5 In the case of referring to the second example explained above, the recognition assistance superimposing section 16 superimposes the recognition assistance AS_R7 and AS_R8 covering the entire right area RA and the recognition assistance AS_L7 and AS_L8 on the left area LA. Further, in order to suppress a decrease in visual recognition of the moving body image SIM_1 due to the superimposed recognition assistance, it is preferable to color the right area RA and the left area LA with a transmissive color in the case of the second example. In other examples, in the case where the color is not a transmissive color, it is preferable to change the transmittance of the color according to time.

[0247] In the case of referring to the second example explained above, the recognition assistance superimposing section 16 superimposes the recognition assistance AS_R7 and AS_R8 covering the entire right area RA and the recognition assistance AS_L7 and AS_L8 on the left area LA. Further, in order to suppress a decrease in visual recognition of the moving body image SIM_1 due to the superimposed recognition assistance, it is preferable to color the right area RA and the left area LA with a transmissive color in the case of the second example. In other examples, in the case where the color is not a transmissive color, it is preferable to change the transmittance of the color according to time. Figure 6 ​In the third example described above, the recognition assistance overlap unit 16 first identifies the target object contained in the moving object image SIM_1. If a static target object is identified within the moving object image SIM_1, the recognition assistance overlap unit 16 overlaps recognition assistance AS_R9, AS_R10, AS_L9, or AS_L10 that would cover the pixel coordinates of the static target object, depending on the region where the static target object is identified. Furthermore, to suppress the decrease in visual recognizability of the static target object due to the overlapped recognition assistance, it is preferable in the third example to use a transmissive color, or to change the transmissive value of a non-transmissive color over time.

[0248] In reference Figure 7 In the fourth example described above, similar to the third example, the recognition assistance overlap unit 16 first identifies the target object contained in the moving object image SIM_1. If a dynamic target object contained in the moving object image SIM_1 is identified, the recognition assistance overlap unit 16 overlaps a recognition assistance AS_R11 or AS_L11 as a bounding box surrounding the dynamic target object, based on the region where the dynamic target object is identified.

[0249] Consider the following scenarios: In the cases of Example 4 and the variations in Example 5 (coloring the trajectory of the moving object), no moving objects contained in the moving object image SIM_1 are detected at all. Furthermore, consider the scenario where the total number of moving objects contained in the moving object image SIM_1 is small. Further, consider the scenario where the identified moving objects in the moving object image SIM_1 are biased towards either the right region RA or the left region LA. Additionally, consider the scenario where the total number of moving objects is excessive.

[0250] When no moving objects are detected, the moving object image SIM_1 is not colored. Therefore, in this case, there is a high possibility of left-right misidentification when the front of the moving object 1 is used as a reference. In addition, when the total number of moving objects is small, or when the detected moving objects are biased towards either the right region RA or the left region LA, there is still a possibility of left-right misidentification when the front of the moving object 1 is used as a reference. On the other hand, when the total number of moving objects is too large, the visual recognizability of the moving object image SIM_1 may actually decrease.

[0251] Therefore, in the fourth example, it is preferable to count the number of moving objects contained in the moving object image SIM_1, and preferably to count the number of moving objects contained in the right region RA and the left region LA respectively. Furthermore, it is preferable to determine whether the number of moving objects contained in the right region RA and the left region LA of the front image SIM_11a is one or more, and whether at least one of the moving objects contained in the right region RA and the left region LA of the rear image SIM_11b contains a moving object. And, preferably, if the determination result is affirmative, the recognition aid AS_R11 or AS_L11 as a bounding box is used.

[0252] Figure 13 It indicates a reference. Figure 7 The diagram shows the first variation of the fourth example of the previously explained recognition aid. Figure 13 In the example shown, a positive judgment was obtained in the determination based on dynamic targets. Specifically, in Figure 13 In the example shown, a moving object is identified in both the right region RA and the left region LA of the front image SIM_11a, and a moving object is identified in the left region LA of the rear image SIM_11b. Therefore, recognition aid AS_R11a overlaps in the right region RA of the front image SIM_11a, and recognition aid AS_L11a overlaps in the left region LA. Additionally, recognition aid AS_L11b overlaps in the left region LA of the rear image SIM_11b.

[0253] On the other hand, if a negative determination result is obtained based on the determination of dynamic targets, it is preferable to appropriately combine the 1st to 3rd or 5th cases (however, limited to the coloring of the trajectory of static targets) into the 4th case. Figure 14 It indicates a reference. Figure 7 The diagram shows the second variation of the fourth example of the previously explained recognition aid. Figure 14 In the example shown, a negative judgment result was obtained in the determination based on dynamic targets. Therefore, in Figure 14 In the example shown, the identification aids AS_R7 and AS_R8 and the identification aids AS_L7 and AS_L8 described in Example 2 are added to the moving body image SIM_1.

[0254] When the total number of dynamic targets is too large, it is preferable to select dynamic targets that overlap with recognition aids AS_R11 or AS_L11 (i.e., dynamic targets with bounding boxes) according to a predetermined criterion. In this case, for example, it is determined whether the total number of dynamic targets in the image region being viewed (e.g., the right region RA of the foreground image SIM_11a) exceeds the upper limit.

[0255] Further, in a case where the total number of dynamic objects is determined to be equal to or more than the upper limit, it is determined that the total number of dynamic objects is too large, and a dynamic object satisfying any one of the criteria (i) to (iii) is extracted.

[0256] (i) a dynamic object whose distance from the mobile body 1 is within a predetermined distance

[0257] (ii) a dynamic object whose time to collision (TTC) with the mobile body 1 is within a predetermined time

[0258] (iii) a dynamic object for which monitoring by the operator OP is particularly required

[0259] It is preferable to repeatedly perform extraction of a dynamic object until the total number of dynamic objects is lower than the upper limit. In this case, for example, it is preferable to perform a change of extraction criteria such that a dynamic object satisfying two of the criteria (i) to (iii) is extracted. The recognition aid AS_R11 or AS_L11 as a bounding box is overlaid so as to surround the dynamic object thus extracted.

[0260] Further, the first and second modifications of the fourth example described above can be applied to the modification of the fifth example. For the description in this case, the description in the above-described description is replaced with "trajectory of a dynamic object" as appropriate.

[0261] In the case of the fourth example, other problems are also envisaged. That is, in a case where a dynamic object is located at a position across the boundary BD1, it is difficult to determine which color of the recognition aids AS_R11 or AS_L11 to adopt. Thus, in this case, for example, the proportion of the area of the dynamic object in the right region RA and the left region LA is calculated. Further, it is preferable to adopt the same color as the color of the bounding box surrounding the dynamic object included in the region in which the area proportion is high as the color of the dynamic object across the boundary BD1.

[0262] However, which side of the colors of the recognition aids AS_R11 and AS_L11 to adopt as the color of the dynamic object across the boundary BD1 can hinder recognition of the left and right by the operator OP. Thus, as the second modification of the fourth example, it is preferable to color the bounding box surrounding the dynamic object across the boundary BD1 with a color that is not the same color system as the color of the recognition aid AS_R11 and is not the same color system as the color of the recognition aid AS_L11 (that is, a third color).

[0263] As the third example of coloring of the bounding box surrounding the dynamic object across the boundary BD1, an example in which the color is changed midway through the bounding box is also considered. Figure 15 is a reference to Figure 7The diagram shows the third variation of the fourth example of the previously explained recognition aid. Figure 15 In the example shown, the bounding box of the right region RA partially overlaps with the identification aid AS_R11c, and the bounding box of the left region LA partially overlaps with the identification aid AS_L11c. The color of the identification aid AS_R11c is the same as the color of the bounding box surrounding the dynamic object contained in the right region RA, and the color of the identification aid AS_L11c is the same as the color of the bounding box surrounding the dynamic object contained in the left region LA.

[0264] As a third example of coloring the bounding box that surrounds a dynamic target object across boundary BD1, we also consider an example of changing the color along the way of the bounding box. Figure 15 It indicates a reference. Figure 7 The diagram for the third variation of the fourth example, as explained earlier. In Figure 15 In the example shown, the bounding box of the right region RA partially overlaps with the identification aid AS_R11c, and the bounding box of the left region LA partially overlaps with the identification aid AS_L11c. The color of the identification aid AS_R11c is the same as the color of the bounding box surrounding the dynamic object contained in the right region RA, and the color of the identification aid AS_L11c is the same as the color of the bounding box surrounding the dynamic object contained in the left region LA.

[0265] As a fourth example of coloring the bounding box that surrounds a dynamic target object across boundary BD1, we also consider an example of making the bounding box two layers. Figure 16 It means in Figure 7 The diagram shows the fourth variation of the fourth example of the recognition aid described earlier. Figure 16 In the example shown, the two bounding boxes surrounding the dynamic target object across boundary BD1 overlap with recognition aids AS_R11d and AS_L11d. The color of recognition aid AS_R11d is the same as the color of the bounding box surrounding the dynamic target object contained in the right region RA, and the color of recognition aid AS_L11d is the same as the color of the bounding box surrounding the dynamic target object contained in the left region LA.

[0266] In addition, Figure 15 and Figure 16 In the example shown, the recognition aids AS_R7 and AS_R8 and recognition aids AS_L7 and AS_L8 described in Example 2 are appended to the front image SIM_11a. However, the addition of these recognition aids is arbitrary.

[0267] In the case where the coloring of the bounding box that encloses the dynamic target object across the border BD1 is colored in a combination of warning colors such as "red" and "blue", the following problem is envisaged. That is, since the coloring of the bounding box changes from "red" to "blue" in association with the movement of the dynamic target object, the operator OP can misinterpret that the state of the dynamic target object has become a safe state. In addition, since the coloring of the bounding box changes from "blue" to "red", the operator OP can misinterpret that the state of the dynamic target object has become a dangerous state. Therefore, it is preferable to use a warning color with reduced saturation and brightness, or not to use a warning color, for the color of the recognition aid.

[0268] Returning to Figure 11 , the function of the data processing device 12 will be described. The recognition aid superimposition section 16 transmits the mobile body image SIM_1 on which the recognition aid is superimposed to the data transmission section 17. The data transmission section 17 encodes the mobile body image SIM_1 on which the recognition aid is superimposed, and transmits it to the terminal 2. At the time of the encoding process, the mobile body image SIM_1 can also be compressed.

[0269] The data transmission section 17 can also transmit the recognition data (ID data) of the mobile body 1 and / or data associated with the mobile body image SIM_1 together with the mobile body image SIM_1. As the data associated with the mobile body image SIM_1, data of the time at which the mobile body image SIM_1 is acquired, internal state data (speed, acceleration, rudder angle, etc.) of the mobile body 1 at that time can be exemplified.

[0270] 2-2-2. Example of functional configuration of data processing device 22

[0271] In Figure 11 the example shown, the data processing device 22 is provided with a data reception section 24 and a display control section 25. The functions of the respective blocks of the data processing device 22 are realized by the processor 22a executing various programs.

[0272] The data reception section 24 receives various data from the mobile body 1. The various data include the mobile body image SIM_1. The various data can also include the recognition data (ID data) of the mobile body 1, data associated with the mobile body image SIM_1. Examples of the data associated with the mobile body image SIM_1 are as already described. The data reception section 24 decodes the various data received from the mobile body 1, and transmits them to the display control section 25. In the case where the various data are compressed, the data reception section 24 decodes them at the time of the decoding process.

[0273] The display control section 25 performs various controls for outputting various data received from the data reception section 24 to the display 21. The various controls include output control of the mobile body image SIM_1. In the output control of the mobile body image SIM_1, for example, processing for displaying the front image SIM_11a on the display 21a and the rear image SIM_11b on the display 21b is performed. In another example, processing for displaying the front image SIM_11a on the main area of the display 21 and displaying data associated with the mobile body image SIM_1 on a sub area of the display 21 is performed. Further, in another example, processing for displaying the rear image SIM_11b on the sub area instead of the data associated with the mobile body image SIM_1 is performed.

[0274] 3. Effects

[0275] According to the first embodiment described above, the recognition aid colored in colors different according to the left and right based on the front of the mobile body 1 is overlaid on the mobile body image SIM_1, and is output from the display 21. Therefore, it is possible to reduce the occurrence of the bad situation that the operator OP misrecognizes the left and right when the front of the mobile body 1 is taken as a reference in the mobile body image SIM_1. This is expected to contribute to the accurate judgment of the operator OP at the time of remote operation of the mobile body 1, and is useful for the safe movement of the mobile body 1 achieved by remote operation.

[0276] Further, according to various modifications of the fourth example of the recognition aid, it is possible to prevent the occurrence of various bad situations assumed when the boundary box as the recognition aid is overlaid on the mobile body image SIM_1. This is also expected to contribute to the accurate judgment of the operator OP at the time of remote operation of the mobile body 1, and is useful for the safe movement of the mobile body 1 achieved by remote operation.

[0277] Second Embodiment

[0278] Next, with reference to Figures 17 to 24 The second embodiment of the present disclosure will be described. Further, the description that is repetitive of the description in the first embodiment is appropriately omitted.

[0279] 1. Summary

[0280] 1-1. Remote Operation

[0281] Figure 17 is a view that describes the remote operation that is a premise of the second embodiment. In Figure 17 The mobile body 1 and the terminal 2 are further described. The description up to this point is the same as that of Figure 1 The infrastructure camera 3 and the display 26 are further described in Figure 17

[0282] ​The infrastructure camera 3 is a camera provided to a structure (for example, a ceiling, a pillar, a wall surface, and the like of a facility structure of a road structure, a parking lot, a factory, and the like). The total number of the infrastructure cameras 3 is one or more. The infrastructure camera 3 and the terminal 2 respectively communicate via a base station of a network. The infrastructure camera 3 transmits a sensor image SIM to the terminal 2. Hereinafter, the sensor image SIM transmitted from the infrastructure camera 3 is also referred to as "infrastructure image SIM 3". The infrastructure image SIM 3 is generated on the basis of data acquired from the infrastructure camera 3.

[0283] The display 26 is connected to the data processing device 22 as with the display 21. In the example shown, the display 26 outputs the infrastructure image SIM 3. As with the total number of the infrastructure cameras 3, the total number of the displays 26 for outputting the infrastructure image SIM 3 is one or more. Further, the infrastructure image SIM 3 can also be output from the display 21 together with the mobile body image SIM 1. In this case, the display 26 can be omitted. For example, according to a picture-in-picture technique, it is possible to display the mobile body image SIM 1 in a part of the area of the display 21 and to display the infrastructure image SIM 3 in the other area of the display 21. Figure 17

[0284] The position where the infrastructure camera 3 is provided is known. In the second embodiment, the mobile body 1 always transmits position data of the mobile body 1 to the terminal 2. The data processing device 22 determines the infrastructure camera 3 whose distance from the current position of the mobile body 1 is a predetermined value or less, for example, on the basis of the position data of the mobile body 1 and the position data of the infrastructure camera 3. As the predetermined value, for example, a distance at which at least a part of the mobile body 1 enters a visual angle of the infrastructure camera 3 is set for each infrastructure camera 3. In other examples, the predetermined value is a distance at which at least a part of the mobile body 1 is expected to enter a visual angle of the infrastructure camera 3 in the future from the current time by a predetermined time (for example, several seconds) elapsing. In the case where the infrastructure camera 3 is determined, the data processing device 22 outputs the infrastructure image SIM 3 generated on the basis of data acquired from the determined infrastructure camera 3 from the display 26.

[0285] ​In this example, a case where a remote operation request is received from the mobile body 1, or a case where the mobile body image SIM_1 is received from the mobile body 1 prior to the transmission of the remote operation request is considered. The operator OP views the mobile body image SIM_1 output from the display 21 and the infrastructure image SIM_3 output from the display 26, and understands the content of the remote operation request. Alternatively, the operator OP views the infrastructure image SIM_3 output from the display 26, and understands the situation in which the mobile body 1 is placed. Also, the operator OP determines the action that the mobile body 1 should take, and inputs an instruction corresponding to the action to the terminal 2.

[0286] The data processing device 22 can also determine the infrastructure camera 3 that is located within a predetermined value or less from the current position of the mobile body 1, in a case where a remote operation request is received from the mobile body 1, or in a case where the mobile body image SIM_1 is received from the mobile body 1 prior to the transmission of the remote operation request. In a case where the infrastructure camera 3 is determined, the data processing device 22 outputs the infrastructure image SIM_3 generated on the basis of data acquired from the determined infrastructure camera 3, from the display 26.

[0287] Figure 18 is a diagram that illustrates an example of the mobile body image SIM_1 and the infrastructure image SIM_3. In Figure 18 a vehicle that is the mobile body 1 and a sensor 11a mounted on the vehicle are depicted. Figure 18 The direction DR of the arrow shown is the traveling direction of the mobile body 1. Also, in Figure 18 the camera main body 31a of the infrastructure camera 3a and the camera main body 31b of the infrastructure camera 3b are described. In Figure 18 In the example shown in

[0288] The front image SIM_11a described in Figure 18 is one example of the mobile body image SIM_1 generated on the basis of data acquired from the sensor 11a. The front image SIM_11a contains the image of a dynamic target object OB3 (pedestrian) that exists in the right front of the mobile body 1. The infrastructure image SIM_3a is generated on the basis of data acquired from the camera main body 31a, and here contains the images of the dynamic target object OB3 and the mobile body 1. The infrastructure image SIM_3b is generated on the basis of data acquired from the camera main body 31b, and here contains the image of the mobile body 1 (the image of the rear of the mobile body 1).

[0289] 1-2. Recognition Assistance

[0290] Similar to the first embodiment, in the second embodiment, a recognition aid is also overlaid on the mobile body image SIM_1 and output from the display 21. In the second embodiment, a recognition aid is further overlaid on the infrastructure image SIM_3 and output from the display 26. As an example of a recognition aid overlaid on the infrastructure image SIM_3, the referenced image can be cited. Figures 4 to 9 The examples described in examples 1 through 6 are identical. As a representative example, for reference... Figure 4 Let's illustrate the first example of the recognition assistance we've already discussed.

[0291] Figure 19 It is a reference Figure 4 The diagram illustrates the first example of recognition assistance applied to the infrastructure image SIM_3. Figure 19 The document describes the front view SIM_11a and the infrastructure images SIM_3a and SIM_3b. The front view SIM_11a is output on display 21a, the infrastructure image SIM_3a is output on display 26a, and the infrastructure image SIM_3b is output on display 26b.

[0292] exist Figure 19 In the example shown, the foreground image SIM_11a is divided into a left region LA and a right region RA by boundary BD1. The right region RA is overlapped with arc-shaped recognition aids AS_R1 to AS_R3, and the left region LA is overlapped with arc-shaped recognition aids AS_L1 to AS_L3. The recognition aids AS_L1 to AS_L3 are colored with a color that is not from the same color family as the recognition aids AS_R1 to AS_R3. Up to this point, compared with... Figure 4 The description of the front image SIM_11a is the same.

[0293] exist Figure 19 In the example shown, infrastructure images SIM_3a and SIM_3b are further divided into a left region LA and a right region RA by boundary BD2. In infrastructure image SIM_3a, arc-shaped recognition aids AS_R14 and AS_R15 overlap in the right region RA, and arc-shaped recognition aids AS_L14 and AS_L15 overlap in the left region LA. In infrastructure image SIM_3b, arc-shaped recognition aids AS_R16 and AS_R17 overlap in the right region RA, and arc-shaped recognition aids AS_L16 and AS_L17 overlap in the left region LA.

[0294] The circular arcs depicted in the infrastructure images SIM_3a and SIM_3b are, like the circular arcs described in the front image SIM_11a and the rear image SIM_11b, parts of the circumference of concentric circles described on a notional horizontal plane with the reference point of the mobile body 1 (e.g., the current position of the mobile body 1) as the center. For example, the recognition aid AS_R1 constitutes the same circle as the recognition aids AS_R14 and AS_R16, and the recognition aid AS_R2 constitutes the same circle as the recognition aids AS_R15 and AS_R17. The recognition aid AS_L1 constitutes the same circle as the recognition aids AS_L14 and AS_L16, and the recognition aid AS_L2 constitutes the same circle as the recognition aids AS_L15 and AS_L17.

[0295] Thus, according to the second embodiment, the infrastructure image SIM_3 is generated on the basis of data obtained from the infrastructure camera 3 at a distance of a predetermined value (a distance at which it is expected that at least a part of the mobile body 1 will enter the angle of view of the infrastructure camera 3 a predetermined time from the current time, etc.) or less from the current position of the mobile body 1, and this infrastructure image SIM_3, on which the recognition aids colored in colors that differ according to the left and right with the front of the mobile body 1 as the reference are superimposed, is output from the display 26. Therefore, it is expected that this will contribute to accurate judgment by the operator OP at the time of remote operation of the mobile body 1, and will be useful for safe movement of the mobile body 1 achieved by remote operation.

[0296] 2. System configuration example

[0297] 2-1. Overall configuration example

[0298] Figure 20 is a block diagram showing an example of the configuration of the display system to which the second embodiment is directed. In the example shown in Figure 20 In the example shown in

[0299] As for the configuration of the mobile body 1, it is as explained in the first embodiment. The terminal 2 is provided with the displays 21 and 26, the data processing device 22, and the input device 23. The infrastructure image SIM_3 generated on the basis of data obtained from the infrastructure camera 3 is output in the display 26.

[0300] The infrastructure camera 3 has a camera main body 31 and a data processing device 32. The camera main body 31 is basically the same as the camera itself as the sensor 11. The data processing device 32 has at least one processor 32a and at least one memory 32b. The processor 32a performs various processes associated with generation of the infrastructure image SIM_3 by executing a program for image processing saved in the memory 32b. In addition, the processor 32a performs various processes for transmitting the generated infrastructure image SIM_3 to the terminal 2 by executing a program for image transmission saved in the memory 32b.

[0301] 2-2. Functional configuration example

[0302] 2-2-1. Functional configuration example of data processing device 32

[0303] Figure 21 is a block diagram showing a configuration example of functions particularly associated with the second embodiment. In Figure 21 the example shown in FIG. 2-2, the data processing device 32 has a data reception section 33, an image generation section 34, and a data transmission section 35. The functions of each block shown in the data processing device 32 are realized by the processor 32a executing various programs.

[0304] The data reception section 33 receives the color image detected by the camera main body 31. The data reception section 33 transmits the color image to the image generation section 34.

[0305] The image generation section 34 generates the sensor image SIM (the infrastructure image SIM_3) based on the color image received from the data reception section 33. The image generation section 34 transmits the generated infrastructure image SIM_3 to the data transmission section 35.

[0306] The data transmission section 35 encodes the infrastructure image SIM_3 received from the image generation section 34 and transmits it to the terminal 2. At the time of encoding processing, the infrastructure image SIM_3 can also be compressed. In the data transmitted by the data transmission section 35, identification data of the infrastructure camera 3, position data (for example, latitude and longitude data) of the infrastructure camera 3, and data associated with the infrastructure image SIM_3 can also be included together with the infrastructure image SIM_3. As the data associated with the infrastructure image SIM_3, data of the time at which the infrastructure image SIM_3 was acquired can be exemplified.

[0307] 2-2-2. Functional configuration example of data processing device 22

[0308] In Figure 21 the example shown in FIG. 2-2, the data processing device 22 has a data reception section 24. The description up to this point is the same as that of the data processing device 32. Figure 11 the data processing device 22.Figure 21 In the example shown, the data processing apparatus 22 is also provided with display control sections 25a and 25b, a left-right defining section 27, and an identification assistance superimposition section 28. The functions of the respective blocks of the data processing apparatus 22 are realized by the processor 22a executing various programs.

[0309] The data reception section 24 receives various data from the mobile body 1. The various data includes the mobile body image SIM_1. The various data can also include identification data of the mobile body 1, position data (GPS data) of the mobile body 1, and data associated with the mobile body image SIM_1. The position data is, for example, latitude and longitude data acquired by a GPS (Global Positioning System) receiver mounted on the mobile body 1. Examples of the data associated with the mobile body image SIM_1 are as already explained. The data reception section 24 decodes the various data received from the mobile body 1 and sends it to the display control section 25a. The function of the display control section 25a is the same as that of the display control section 25 explained in Figure 11

[0310] In addition, the data reception section 24 receives various data from the infrastructure camera 3. The various data includes the infrastructure image SIM_3. The various data can also include identification data of the infrastructure camera 3, position data of the infrastructure camera 3, and data associated with the infrastructure image SIM_3. Examples of the data associated with the infrastructure image SIM_3 are as already explained. The data reception section 24 decodes the various data received from the infrastructure camera 3 and sends it to the left-right defining section 27 and the identification assistance superimposition section 28.

[0311] The left-right defining section 27 defines left and right in the infrastructure image SIM_3. The left-right defining section 27 first sets a boundary. Figure 22 is a diagram explaining a processing example of the left-right defining section 27. Furthermore, in Figure 22 , explanation is made of a processing example of the infrastructure image SIM_3a explained with reference to Figure 18 , explanation is made of a processing example of the infrastructure image SIM_3a explained with reference to Figure 22 In the example shown, first, based on the position data of the mobile body 1, an imaginary horizontal plane VHS passing through a reference point of the mobile body 1 (for example, the current position of the mobile body 1) is set (step S1).

[0312] After the imaginary horizontal plane VHS is set, a reference line RL extending in the front-rear direction of the mobile body 1 is set on the imaginary horizontal plane VHS (step S1). For the front-rear direction, for example, estimation is made based on a history record of the position data of the mobile body 1 in a period of a predetermined time back from the current time. In Figure 22 ​In the example shown, the reference line RL is set by projecting a straight line extending in the presumed direction onto the virtual horizontal plane VHS.

[0313] After the reference line RL is set, a plane (i.e., the boundary surface BS) that contains the reference line RL and is orthogonal to the virtual horizontal plane VHS is set (step S2). After the boundary surface BS is set, the right space RS and the left space LS with the front of the mobile body 1 as a reference can be defined in a virtual space VSP that contains the boundary surface BS and the virtual horizontal plane VHS.

[0314] The right-left defining section 27 applies the virtual space VSP that contains the boundary surface BS and the virtual horizontal plane VHS to the infrastructure image SIM_3 (step S2). Thus, the boundary BD2 drawn by the boundary surface BS can be set on the infrastructure image SIM_3. When the boundary BD2 is set, the right region RA and the left region LA with the front of the mobile body 1 as a reference can be set on the infrastructure image SIM_3.

[0315] The right-left defining section 27 transmits data of the right region RA and the left region LA in the infrastructure image SIM_3 to the recognition assistance superimposing section 28. The right-left defining section 27, for example, transmits data of pixel coordinates that constitute the boundary BD2 in the infrastructure image SIM_3 to the recognition assistance superimposing section 28. In the case where the recognition assistance is superimposed in a circular arc shape on the infrastructure image SIM_3 (i.e., with reference to the first example described above), the right-left defining section 27 transmits data of pixel coordinates that constitute the virtual horizontal plane VHS in the infrastructure image SIM_3 to the recognition assistance superimposing section 28. Figure 19 In the case of the first example described above, the right-left defining section 27 transmits data of pixel coordinates that constitute the virtual horizontal plane VHS in the infrastructure image SIM_3 to the recognition assistance superimposing section 28.

[0316] The recognition assistance superimposing section 28 superimposes recognition assistance on the infrastructure image SIM_3 on the basis of the infrastructure image SIM_3 received from the data receiving section 24 and various data associated with the infrastructure image SIM_3 received from the right-left defining section 27.

[0317] In the case where the recognition assistance is superimposed in a circular arc shape on the infrastructure image SIM_3 (i.e., with reference to the first example described above), the recognition assistance superimposing section 28 superimposes the recognition assistance on the infrastructure image SIM_3 on the basis of the data of pixel coordinates that constitute the virtual horizontal plane VHS in the infrastructure image SIM_3. Figure 19 In the case of the first example described above, the recognition assistance superimposing section 28 draws a virtual concentric circle with the reference point of the mobile body 1 as a center on the virtual horizontal plane VHS using the data of pixel coordinates that constitute the virtual horizontal plane VHS in the infrastructure image SIM_3. Next, the recognition assistance superimposing section 28 applies the virtual concentric circle to the infrastructure image SIM_3, superimposes the recognition assistance AS_R14 to AS_R17 on the right region RA, and superimposes the recognition assistance AS_L14 to AS_L17 on the left region LA.

[0318] In the first embodiment, the recognition assistance is superimposed on the infrastructure image SIM_3 on the basis of the data of pixel coordinates that constitute the boundary BD2 in the infrastructure image SIM_3. Figures 5 to 9Examples 2 through 6 of the described recognition assistance can also be applied to the second implementation. The explanation of these examples will be provided by... Figure 11 In the description of “identification auxiliary overlap 16”, “moving body image SIM_1” is appropriately replaced with “infrastructure image SIM_3” for explanation.

[0319] return Figure 21 Next, the functions of the data processing device 22 will be explained. The identification assist overlay unit 28 sends the infrastructure image SIM_3, overlaid with identification assistance, to the display control unit 25b. The display control unit 25b performs various controls for outputting various data received from the data receiving unit 24 to the display 26. These various controls include output control of the infrastructure image SIM_3. For example, the output control of the infrastructure image SIM_3 includes processing for displaying the infrastructure image SIM_3a on the display 26a and the infrastructure image SIM_3b on the display 26b.

[0320] When the infrastructure image SIM_3 is output from the display 26, there is a problem that it is difficult for the operator (OP) to immediately identify the front and rear of the moving body 1 included in the infrastructure image SIM_3. Therefore, in this case, it is preferable to use a reference... Figure 19 The first example described above is modified by subdividing the right region RA and the left region LA in the moving body image SIM_1 and the infrastructure image SIM_3 in the front-back direction.

[0321] Figure 23 This diagram illustrates an example of subdivision processing of the right region RA and the left region LA in an infrastructure image SIM_3. Furthermore, this subdivision processing can be performed using a reference... Figure 21 The left and right definitions, as explained in section 27, are used for this purpose. (And...) Figure 19 Similarly, in Figure 23 The image depicts the foreground image SIM_11a and infrastructure images SIM_3a and SIM_3b. The foreground image SIM_11a and... Figure 19 The images shown are the same.

[0322] exist Figure 23 In the example shown, a boundary BD3 is added to the infrastructure images SIM_3a and SIM_3b. Boundary BD3 can be set using the same method as boundary BD2. Specifically, first, it is set on the imaginary horizontal plane VHS and referenced... Figure 22The line orthogonal to the previously described baseline RL (hereinafter also referred to as the "additional baseline"). Next, a plane containing this additional baseline and a plane orthogonal to the imaginary horizontal plane VHS (hereinafter also referred to as the "additional boundary plane"). Then, an imaginary space containing this additional boundary plane and the imaginary horizontal plane VHS is applied to the infrastructure image SIM_3. Thus, a boundary (additional boundary) BD3 drawn by the additional boundary plane can be set on the infrastructure image SIM_3.

[0323] When boundary BD3 is set, the right region RA can be segmented into the right front region RFA and the right rear region RRA, and the left region LA can be segmented into the left front region LFA and the left rear region LRA. By using data from these regions, forward and backward orientation information can be assigned to the infrastructure images SIM_3a and SIM_3b. Furthermore, this processing can be performed using a reference... Figure 21 The identification assistance overlap 28, as explained, is used.

[0324] exist Figure 23 The diagram shows infrastructure images SIM_3a and SIM_3b with recognition assistance superimposed in four regions. Specifically, in infrastructure image SIM_3a, recognition assistance AS_R18 and AS_R19 are superimposed in the right front region RFA, AS_L18 and AS_L19 are superimposed in the left front region LFA, AS_R20 is superimposed in the right rear region RRA, and AS_L20 is superimposed in the left rear region LRA. In infrastructure image SIM_3b, recognition assistance AS_R21 and AS_R22 are superimposed in the right rear region RRA, AS_L21 and AS_L22 are superimposed in the left rear region LRA, AS_R23 is superimposed in the right front region RFA, and AS_L23 is superimposed in the left front region LFA.

[0325] Figure 23 The identification aids shown are colored based on the regions where these aids overlap. That is, in Figure 23 In the example shown, the recognition aids AS_R18, AS_R19, and AS_R23 in the right front region RFA, which overlaps with the infrastructure image SIM_3, are colored with a different color than the recognition aids AS_R20, AS_R21, and AS_R22 in the right rear region RRA, which overlaps with the infrastructure image SIM_3. However, the recognition aids AS_R18 to AS_R23 are colored with a color of the same color family as the recognition aids AS_R1 to AS_R3 in the foreground image SIM_11a. That is, the recognition aids AS_R18, AS_R19, and AS_R23 are colored with a color of the same color family as, but different from, the recognition aids AS_R20, AS_R21, and AS_R22.

[0326] In addition, the recognition aids AS_L18, AS_L19, and AS_L23 that overlap the front-left area LFA of the infrastructure image SIM_3 are colored in the same color tone as the recognition aids AS_L20, AS_L21, and AS_L22 that overlap the rear-left area LRA of the infrastructure image SIM_3, and in different colors. Further, the colors of the recognition aids AS_L18 to AS_L23 are not the same color tone as the colors of the recognition aids AS_R18 to AS_R23, which is a matter that is easily understood in accordance with the gist of the present disclosure.

[0327] As another example that contributes to the immediate recognition of the front and rear of the mobile body 1 included in the infrastructure image SIM_3, a display that outputs a map image of the vicinity of the current position of the mobile body 1 is considered to be additionally provided. The current position of the mobile body 1 can be grasped from the position data, and thus a map image of the vicinity of the current position is acquired from a map database or the like, an icon that imitates the infrastructure camera 3, an icon that imitates the mobile body 1, and a pictogram that is colored in the same color tone as the colors of the recognition aids output from the display 26 are overlaid on the map image, and are output from the display.

[0328] Figure 24 is a diagram that explains a processing example of superimposing various icons on a map image. In Figure 24 In the example illustrated in FIG. 29, a map image MIM in which an icon ICN3 that imitates the infrastructure camera 3 and an icon ICN1 that imitates the mobile body 1 are superimposed is output from the display 29. In addition, a pictogram PG_R that is colored in the same color tone as the colors of the recognition aids AS_R14 to AS_R17 output from the display 26 and a pictogram PG_L that is colored in the same color tone as the colors of the recognition aids AS_L14 to AS_L17 are superimposed on the map image MIM.

[0329] 3. Effects

[0330] According to the second embodiment described above, the infrastructure image SIM_3 is generated on the basis of data obtained from the infrastructure cameras 3 that are within a predetermined distance from the current position of the mobile body 1. Further, the infrastructure image SIM_3 in which the recognition aids that are colored in different colors according to the left and right with the front of the mobile body 1 as a reference are superimposed is output from the display 26. Thus, it is expected that this contributes to the accurate judgment of the operator OP at the time of remote operation of the mobile body 1, and is useful for the safe movement of the mobile body 1 achieved by remote operation.

[0331] In addition, according to the first embodiment described above, the infrastructure image SIM_3 is generated on the basis of data obtained from the infrastructure cameras 3 that are within a predetermined distance from the current position of the mobile body 1. Further, the infrastructure image SIM_3 in which the recognition aids that are colored in different colors according to the left and right with the front of the mobile body 1 as a reference are superimposed is output from the display 26. Thus, it is expected that this contributes to the accurate judgment of the operator OP at the time of remote operation of the mobile body 1, and is useful for the safe movement of the mobile body 1 achieved by remote operation. Figure 23The first example of the recognition aid deformed as explained above, in which the infrastructure image SIM_3 is overlaid with the recognition aid colored in colors different according to the right front, left front, right rear, and left rear from the front of the mobile body 1, is output from the display 26. Thus, immediate recognition of the front and rear of the mobile body 1 by the operator OP is realized. This is also expected to contribute to accurate judgment of the operator OP at the time of remote operation of the mobile body 1, and is useful for safe movement of the mobile body 1 realized by remote operation.

[0332] In addition, according to the deformation of the map image utilizing the deformation of the map image explained above in the first embodiment, Figure 24 immediate recognition of the front and rear of the mobile body 1 by the operator OP is realized. Also, the relative position of the infrastructure camera 3 to the mobile body 1, and the traveling direction of the mobile body 1 can be grasped immediately. This is also expected to contribute to accurate judgment of the operator OP at the time of remote operation of the mobile body 1, and is useful for safe movement of the mobile body 1 realized by remote operation.

[0333] In addition, the deformation of the map image MIM utilizing the deformation of the map image explained above in the first embodiment can also be applied to the second embodiment. In this case, the image obtained by removing the icon ICN3 from the map image MIM shown in FIG. 21 is output from the display 29. In addition, the pictograms PG_R and PG_L are colored in the same color system as the colors of the recognition aid output from the display 21. Figure 24 Figure 24 In addition, the deformation of the map image MIM utilizing the deformation of the map image explained above in the first embodiment can also be applied to the second embodiment. In this case, the image obtained by removing the icon ICN3 from the map image MIM shown in FIG. 21 is output from the display 29. In addition, the pictograms PG_R and PG_L are colored in the same color system as the colors of the recognition aid output from the display 21.

[0334] Third Embodiment

[0335] Finally, the third embodiment of the present disclosure is explained. In addition, the explanation repeated in the first and second embodiments is appropriately omitted.

[0336] In the second embodiment, the mobile body image SIM_1 and the infrastructure image SIM_3 are combined and output from the displays 21 and 26. In the third embodiment, only the infrastructure image SIM_3 is output from the display 26. That is, in the third embodiment, the generation of the mobile body image SIM_1 is not performed, or the output of the mobile body image SIM_1 from the display 21 is not performed. In the case where the infrastructure image SIM_3 is output from the display 26, the display 21 can be omitted. The infrastructure image SIM_3 can be output from the display 21, in which case the display 26 can be omitted.

[0337] ​The following case is envisaged: the mobile body image SIM_1 is not output from the display 21 prior to receiving the remote operation request from the mobile body 1. For example, in a case where the operator OP is on standby for remote operation requests from a plurality of mobile bodies 1, a plurality of infrastructure images SIM_3 generated on the basis of data acquired from the infrastructure cameras 3 are preferably output from a plurality of displays 26, respectively.

[0338] In this case, if a remote operation request is received from one of the plurality of mobile bodies 1, the output of the infrastructure images SIM_3 other than the infrastructure image SIM_3 corresponding to the infrastructure camera 3 that is a predetermined value or less from the mobile body 1 is ended. Also, the mobile body image SIM_1 of the mobile body 1 that is being remotely operated is output from the display 21.

Claims

1. A remote image display method, comprising displaying a sensor image generated based on data obtained from a remote sensor on a display of a terminal communicating with the remote sensor, characterized in that, The remote sensor includes a sensor mounted on the moving body. The sensor images include moving body images, which represent images generated based on data obtained from sensors mounted on the moving body, and include front and rear images of the moving body. The remote image display method includes: The step of setting a reference line extending in the front-rear direction of the moving body on an imaginary horizontal plane passing through the reference point of the moving body; Using the baseline, the steps are as follows: defining the right space and left space of the imaginary space containing the imaginary horizontal plane with the front of the moving body as the reference. The steps are as follows: using the imaginary space that defines the right space and the left space, the steps are as follows: setting the right region and the left region of the front image and the rear image respectively, with the front of the moving body as the reference. In the step of overlapping the right and left recognition aids in the right and left regions of the front and rear images respectively, the color of the right recognition aid is different from the color of the left recognition aid; as well as The step of outputting the front and rear images, which are respectively superimposed on the right and left recognition aids, from the display.

2. The remote image display method according to claim 1, characterized in that, The right and left identification aids include coloring an arc drawn concentrically on an imaginary horizontal plane centered on the reference point of the moving object; coloring at least a portion of each region of the right and left regions; coloring a static target identified in the moving object image; coloring a bounding box surrounding a dynamic target identified in the moving object image; coloring the trajectory of the dynamic target; and coloring the trajectory of the moving object.

3. The remote image display method according to claim 2, characterized in that, The right and left identification aids are a combination of at least one selected from coloring the arc, coloring at least a portion of each region of the right and left regions, coloring the static target, and coloring the trajectory of the moving object, and at least one selected from coloring the bounding box and coloring the trajectory of the dynamic target.

4. The remote image display method according to claim 2, characterized in that, Also includes: The step of determining whether a dynamic target object is identified in the moving object image that crosses the boundary separating the right region and the left region; and If it is determined that a dynamic target object across the boundary is identified in the moving object image, the step of making the bounding box surrounding the dynamic target object across the boundary into two layers and applying the right and left recognition aid colors to these bounding boxes respectively.

5. The remote image display method according to claim 2, characterized in that, Also includes: The step of determining whether a dynamic target object is identified in the moving object image that crosses the boundary separating the right region and the left region; and In the case that a dynamic target object is identified across the boundary in the moving object image, the step of applying a color different from either of the right and left identification aids to the bounding box surrounding the dynamic target object across the boundary.

6. The remote image display method according to any one of claims 2 to 5, characterized in that, Also includes: The step of determining whether the total number of the dynamic targets is above the upper limit; and The step of selecting dynamic target objects assigned to the bounding box according to a predetermined criterion when the total number is determined to be above the upper limit.

7. The remote image display method according to any one of claims 1 to 5, characterized in that, The step of overlaying the right and left recognition aids onto the front and rear images respectively further includes: The step of setting the color of the recognition aid on the right side of the image overlapping the front image and the color of the recognition aid on the right side of the image overlapping the rear image to colors of the same color family; and The step of setting the color of the recognition aid on the left side of the image overlapping the front image and the color of the recognition aid on the left side of the image overlapping the rear image to colors of the same color family.

8. The remote image display method according to claim 7, characterized in that, Also includes: The step of determining whether the number of dynamic targets identified in the right region and the left region of the image in front is 1 or more; The step of determining whether a dynamic target object is identified in at least one of the right region and the left region of the rear image; as well as If it is determined that the number of dynamic targets identified in the right region and the left region of the front image is 1 or more, and a dynamic target is identified in at least one of the right region and the left region of the rear image, a step is taken to assist in the identification of the right and left regions by coloring the bounding box surrounding the dynamic targets identified in the front image and the rear image.

9. The remote image display method according to any one of claims 1 to 5 and 8, characterized in that, The boundary separating the right region from the left region includes a boundary with a predetermined shape. The remote image display method further includes the following step: in the boundary overlap recognition aid with a predetermined shape, the color of the recognition aid overlapping the boundary with the predetermined shape is different from the color of either the right or left recognition aid.

10. The remote image display method according to any one of claims 1 to 5 and 8, characterized in that, The sensor images include infrastructure images, which are images generated based on data obtained from infrastructure cameras located at a predetermined distance from the moving body. The terminal also includes a display showing an image of the infrastructure. The remote image display method further includes: The step of defining a right region and a left region for the infrastructure image based on the front of the moving body using the imaginary space that defines the right space and the left space; In the step of overlapping right and left recognition aids respectively in the right and left regions of the infrastructure image, the color of the recognition aid overlapping the right side of the infrastructure image is a color of the same color family as the recognition aid overlapping the right side of the moving object image, and the color of the recognition aid overlapping the left side of the infrastructure image is a color of the same color family as the recognition aid overlapping the left side of the moving object image; and The step of outputting an image of the infrastructure that overlaps the right and left sides of the display to aid in identification from the display of the infrastructure image.

11. The remote image display method according to claim 10, characterized in that, Also includes: The step of setting an additional reference line orthogonal to the baseline on the imaginary horizontal plane; and The steps of using the additional baseline to divide the right space and the left space in the front-rear direction of the moving body into two parts to define the right front space, left front space, right rear space, and left rear space are as follows: Given the defined right front space, left front space, right rear space, and left rear space, In the step of defining a right and left region based on the front of the moving body in the infrastructure image, a right front region, a left front region, a right rear region, and a left rear region based on the front of the moving body are defined in the infrastructure image. In the step of overlapping right and left recognition aids in the right and left regions of the infrastructure image, respectively, the recognition aids for the right front region, the left front region, the right rear region, and the left rear region are respectively overlapped with the right front, left front, right rear, and left rear recognition aids. The color of the recognition aid superimposed on the right front of the infrastructure image is different from the color of the recognition aid superimposed on the right rear, and the color of the recognition aid superimposed on the left front of the infrastructure image is different from the color of the recognition aid superimposed on the left rear.

12. The remote image display method according to any one of claims 1 to 5, 8, and 11, characterized in that, The terminal also includes a display showing a map image of the area surrounding the moving body. The remote image display method further includes: In the step of overlaying the icon of the moving object onto the map image and the pictograms to the right and left of the icon of the moving object, with the front of the moving object as a reference, the color of the right pictogram is a color of the same color family as the color of the recognition aid overlaid on the right side of the moving object image, and the color of the left pictogram is a color of the same color family as the color of the recognition aid overlaid on the left side of the moving object image; and The step of outputting a map image from a display showing the map image, which overlaps the icons of the moving objects and the pictographs on the right and left.

13. A remote image display method, comprising displaying a sensor image generated based on data obtained from a remote sensor on a display of a terminal communicating with the remote sensor, characterized in that, The sensor images include infrastructure images and moving body images. The infrastructure images are generated based on data obtained from infrastructure cameras located at a predetermined distance from the moving body. The moving body images are generated based on data obtained from sensors mounted on the moving body, and include an image of the front of the moving body. The remote image display method includes: The step of setting a reference line extending in the front-rear direction of the moving body on an imaginary horizontal plane passing through the reference point of the moving body; Using the baseline, the steps are as follows: defining the right space and left space of the imaginary space containing the imaginary horizontal plane with the front of the moving body as the reference. The steps involve using the imaginary space that defines the right space and the left space to set the right region and the left region of the infrastructure image and the moving body image respectively, with the front of the moving body as the reference. In the step of overlapping the right and left recognition aids respectively in the right and left regions of the infrastructure image and the moving body image, the color of the right recognition aid is different from the color of the left recognition aid; as well as The step of outputting the infrastructure image and the moving body image, which are respectively superimposed on the right and left sides for recognition assistance, from the display.

14. A remote image display system for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal communicating with the remote sensor, characterized in that, The remote sensor includes a sensor mounted on the moving body. The sensor images include moving body images, which represent images generated based on data obtained from sensors mounted on the moving body, and include front and rear images of the moving body. The moving body, A reference line extending in the front-rear direction of the moving body is set on an imaginary horizontal plane passing through the reference point of the moving body. Using the baseline, define a right space and a left space with the front of the moving body as the reference for the imaginary space containing the imaginary horizontal plane. Using the imaginary space that defines the right space and the left space, right and left regions are respectively set for the front and rear images, with the front of the moving object as the reference. In the front and rear images, right and left recognition aids are respectively superimposed on the right and left regions, respectively, and the color of the right recognition aid is different from the color of the left recognition aid. The terminal receives the front and rear images, which are respectively superimposed with the right and left recognition aids. The terminal outputs a front image and a rear image from the display, which are superimposed with the right and left recognition aids.

15. A remote image display system for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal communicating with the remote sensor, characterized in that, The sensor images include infrastructure images and moving body images. The infrastructure images are generated based on data obtained from infrastructure cameras located at a predetermined distance from the moving body. The moving body images are generated based on data obtained from sensors mounted on the moving body, and include an image of the front of the moving body. The infrastructure camera sends images of the infrastructure to the terminal. The terminal, A reference line extending in the front-rear direction of the moving body is set on an imaginary horizontal plane passing through the reference point of the moving body. Using the baseline, define a right space and a left space with the front of the moving body as the reference for the imaginary space containing the imaginary horizontal plane. Using the hypothetical space that defines the right space and the left space, right and left regions are respectively set for the infrastructure image and the moving body image, with the front of the moving body as the reference. In the infrastructure image and the moving body image, right and left recognition aids are respectively superimposed on the right and left regions, respectively, and the color of the right recognition aid is different from the color of the left recognition aid. The display outputs an image of the infrastructure and the moving body that are superimposed on the right and left sides respectively to aid in identification.

Images