Remote image display method and remote image display system

Abstract

To reduce the occurrence of a problem in which an operator who performs remote work on the basis of a sensor image misjudges left and right when the front of a moving object is taken as a reference.SOLUTION: A sensor image generated on the basis of data acquired from a remote sensor is displayed on the display of a terminal in communication with the remote sensor. The remote sensor include a sensor mounted on a moving object. The sensor image includes a moving object image that is generated on the basis of data acquired from the sensor mounted on the mobile object. In the moving object image, right and left regions are set with respect to the front of the moving object. Right and left recognition aids are superimposed on the right and left regions, respectively. The color of the right recognition aid is different from the color of the left recognition aid.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a method and system for displaying a sensor image generated based on data acquired from a remote sensor on a terminal in communication with the remote sensor. [Background technology]

[0002] US Patent No. 11034299 discloses a system equipped with two electronic mirrors provided on both sides of the driver's seat of a vehicle. In this conventional system, an image of the right side and right rear of the vehicle is output from the right electronic mirror, and an image of the left side and left rear of the vehicle is output from the left electronic mirror. When this conventional system recognizes another vehicle approaching from behind the vehicle, it superimposes an enlarged image of the other vehicle on a part of the image including the other vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 11034299 [Patent Document 2] JP 2020-72401 A [Patent Document 3] Japanese Patent Application Publication No. 9-271016 [Patent Document 4] JP 2020-161039 A [Patent Document 5] JP 2011-35729 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the conventional system, the driver sitting in the driver's seat can intuitively know whether another vehicle is approaching from the right rear or the left rear of the vehicle, because the driver knows that these electronic mirrors have the function of side mirrors, and can instantly understand the location of the other vehicle from the installation position of the electronic mirror that outputs an image including the other vehicle.

[0005] Consider a situation where a remote operation of a moving object, such as a vehicle, is performed. In this remote operation, a sensor image is generated based on data acquired from a sensor of the moving object, and this is provided to an operator. Examples of this sensor image include an image of the front center and an image of the rear center of the moving object. Typically, the images of the front center and the rear center are generated based on data acquired from a camera mounted on the moving object.

[0006] The problem here is that, when the front of the moving body is used as a reference, the operator may erroneously judge that the right area of ​​the rear center image corresponds to the "left" of the moving body, and the left area of ​​this image corresponds to the "right" of the moving body. Also, if the front center image and the rear center image are mixed up, the operator may erroneously judge that the right area of ​​the front center image corresponds to the "right" of the moving body, and the left area of ​​this image corresponds to the "left" of the moving body.

[0007] One objective of the present disclosure is to provide technology that can reduce the occurrence of problems such as an operator performing remote work based on sensor images misjudging left and right when using the area in front of a moving object as a reference point. [Means for solving the problem]

[0008] A first aspect is a method for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal communicating with the remote sensor, and has the following features. The remote sensor includes a sensor mounted on a vehicle. The sensor images include moving body images that show images generated based on data acquired from a sensor mounted on the moving body. The method comprises: setting a reference line extending in a front-rear direction of the moving body on a virtual horizontal plane passing through a reference point of the moving body; using the reference line to define right and left spaces in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; setting right and left areas in the moving body image based on a front of the moving body using the virtual space in which the right and left spaces are defined; a step of superimposing right and left recognition aids on the right and left regions of the moving object image, respectively, the right recognition aid being a different color from the left recognition aid; outputting the moving object image on which the right and left recognition aids are superimposed from the display; Includes.

[0009] The second aspect has the same features as the first aspect, as follows. The right and left recognition assistance includes coloring applied to arcs drawn concentrically on the virtual horizontal plane centered on a reference point of the moving body, coloring applied to at least a portion of each of the right and left regions, coloring applied to static targets recognized in the moving body image, coloring applied to bounding boxes surrounding dynamic targets recognized in the moving body image, coloring applied to the trajectory of the dynamic targets, and coloring applied to the trajectory of the moving body.

[0010] The third aspect has the same features as the second aspect as follows. The right and left recognition assistance is a combination of at least one selected from a color applied to the arc, a color applied to at least a portion of each of the right and left regions, a color applied to the static target, and a color applied to the trajectory of the moving body, and at least one selected from a color applied to the bounding box and a color applied to the trajectory of the dynamic target.

[0011] The fourth aspect of the present invention is the second or third aspect further characterized by the following. The method further comprises: determining whether a dynamic target straddling a boundary separating the right region and the left region is recognized in the moving object image; when it is determined that the boundary-spanning dynamic object has been recognized in the moving object image, doubling the bounding box surrounding the boundary-spanning dynamic object and applying the right and left recognition assistance colors to these bounding boxes, respectively; Includes.

[0012] The fifth aspect has the second or third aspect further characterized by the following: The method further comprises: determining whether a dynamic target straddling a boundary separating the right region and the left region is recognized in the moving object image; When it is determined that the boundary-spanning dynamic object has been recognized in the moving object image, applying a color different from both the right and left recognition assistance colors to a bounding box surrounding the boundary-spanning dynamic object; Includes.

[0013] A sixth aspect is any one of the second to fifth aspects, further comprising the following features. The method further comprises: determining whether the total number of the dynamic targets is equal to or greater than an upper limit; selecting a dynamic object to which the bounding box is assigned according to a predetermined criterion when it is determined that the total number is equal to or greater than the upper limit; Includes.

[0014] A seventh aspect is any one of the first to sixth aspects, further comprising the following features. The moving object images include front and rear images of the moving object. The step of superimposing the right and left recognition assistance on the moving object image includes: setting a color of the right recognition assistance superimposed on the front image and a color of the right recognition assistance superimposed on the rear image to the same color; setting a color of the left recognition aid superimposed on the front image and a color of the left recognition aid superimposed on the rear image to the same color; Includes.

[0015] The eighth aspect has the same features as the seventh aspect as follows. The method further comprises: determining whether the number of dynamic targets recognized in the right and left regions of the forward image is equal to or greater than one; determining whether a dynamic target is recognized in at least one of the right and left regions of the rear image; adopting a color applied to a bounding box surrounding the dynamic target recognized in the front and rear images as a recognition aid for the right and left when the number of dynamic targets recognized in each of the right and left regions of the front image is equal to or greater than one and it is determined that a dynamic target has been recognized in at least one of the right and left regions of the rear image; Includes.

[0016] A ninth aspect is any one of the first to eighth aspects, further comprising the following features. the boundary separating the right region and the left region includes a boundary having a predetermined shape; The method further comprises: The method includes the step of overlaying a recognition aid on the shaped boundary, the recognition aid being a color different from the colors of both the right and left recognition aids.

[0017] A tenth aspect is any one of the first to ninth aspects, further comprising the following feature. The sensor image includes an infrastructure image showing an image generated based on data acquired from an infrastructure camera whose distance from the moving object is equal to or less than a predetermined value. The terminal further includes a display for displaying the infrastructure image. The method further comprises: setting right and left areas based on a front of the moving object in the infrastructure image using the virtual space in which the right and left spaces are defined; a step of superimposing right and left recognition aids on the right and left regions of the infrastructure image, respectively, wherein the color of the right recognition aid superimposed on the infrastructure image is the same color as that of the right recognition aid superimposed on the moving body image, and the color of the left recognition aid superimposed on the infrastructure image is the same color as that of the left recognition aid superimposed on the moving body image; outputting the infrastructure image on which the right and left recognition aids are superimposed from a display that displays the infrastructure image; Includes.

[0018] The eleventh aspect has the same features as the tenth aspect as follows. The method further comprises: setting an additional reference line on the virtual horizontal plane that is perpendicular to the reference line; dividing the right and left spaces into two in a front-rear direction of the moving body using the additional reference line to define right front, left front, right rear, and left rear spaces; Includes. When the right front, left front, right rear and left rear spaces are defined, in a step of setting right and left areas based on the front of the moving body in the infrastructure image, the right front, left front, right rear and left rear areas based on the front of the moving body are set in the infrastructure image. When the right front, left front, right rear, and left rear spaces are defined, in the step of superimposing right and left recognition aids on the right and left regions of the infrastructure image, respectively, the right front, left front, right rear, and left rear recognition aids are superimposed on the right front, left front, right rear, and left rear regions, respectively. The right front recognition aid and the right rear recognition aid superimposed on the infrastructure image have different colors, and the left front recognition aid and the left rear recognition aid superimposed on the infrastructure image have different colors.

[0019] A twelfth aspect is any one of the first to eleventh aspects, further comprising the following feature. The terminal further includes a display for displaying a map image of the surroundings of the mobile object. The method further comprises: a step of superimposing on the map image an icon of the moving body and pictograms surrounding the icon of the moving body and right and left pictograms based on a front of the moving body, the right pictogram being a color of the same color family as the right recognition aid superimposed on the moving body image, and the left pictogram being a color of the same color family as the left recognition aid superimposed on the moving body image; outputting the map image on which the icon of the moving object and the right and left pictograms are superimposed from a display that displays the map image; Includes.

[0020] A thirteenth aspect is a method for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal communicating with the remote sensor, the method having the following features. The sensor image includes an infrastructure image showing an image generated based on data acquired from an infrastructure camera whose distance from a moving object is equal to or less than a predetermined value. The method comprises: setting a reference line extending in a front-rear direction of the moving body on a virtual horizontal plane passing through a reference point of the moving body; using the reference line to define right and left spaces in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; setting right and left areas based on a front of the moving object on the infrastructure image by using the virtual space in which the right and left spaces are defined; a step of superimposing right and left recognition aids on the right and left regions of the infrastructure image, respectively, the right recognition aid being a different color from the left recognition aid; outputting the infrastructure image with the right and left recognition aids superimposed thereon from the display; Includes.

[0021] A fourteenth aspect is a system for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal that communicates with the remote sensor, the system having the following features. The remote sensor includes a sensor mounted on a vehicle. The sensor images include moving body images that show images generated based on data acquired from a sensor mounted on the moving body. The moving body is A reference line is set on a virtual horizontal plane passing through a reference point of the moving body, the reference line extending in a front-rear direction of the moving body; Using the reference line, right and left spaces are defined in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; Using the virtual space in which the right and left spaces are defined, right and left areas based on a front of the moving body are set in the moving body image; Superimposing right and left recognition aids on the right and left regions of the moving object image, respectively; The moving object image with the right and left recognition aids superimposed thereon is transmitted to the terminal. The right recognition aid is of a different color than the left recognition aid. The terminal outputs the moving object image with the right and left recognition aids superimposed thereon from the display.

[0022] A fifteenth aspect is a system for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal that communicates with the remote sensor, the system having the following features. The sensor image includes an infrastructure image showing an image generated based on data acquired from an infrastructure camera whose distance from a moving object is equal to or less than a predetermined value. The infrastructure camera transmits the infrastructure image to the terminal. The terminal includes: A reference line is set on a virtual horizontal plane passing through a reference point of the moving body, the reference line extending in a front-rear direction of the moving body; Using the reference line, right and left spaces are defined in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; setting right and left areas based on a front of the moving object in the infrastructure image using the virtual space in which the right and left spaces are defined; Superimposing right and left recognition aids on the right and left regions of the infrastructure image, respectively; The infrastructure image with the right and left recognition aids superimposed thereon is output from the display. The right recognition aid is of a different color than the left recognition aid. Effect of the Invention

[0023] According to the first or fourteenth aspect, right and left regions based on the front of the moving body are set in the moving body image, and the moving body image with different colors on the right and left is output from the display. This makes it possible to reduce the occurrence of a problem in which an operator erroneously recognizes the left and right when the front of the moving body is used as a reference in the moving body image.

[0024] According to the second aspect, as recognition aids for the right and left, coloring applied to arcs, coloring applied to at least a portion of each of the right and left regions, coloring applied to static targets, coloring applied to bounding boxes surrounding dynamic targets, coloring applied to the trajectory of dynamic targets, and coloring applied to the trajectory of moving bodies can be used.

[0025] If a dynamic target is not recognized in a moving body image, a bounding box surrounding the dynamic target is not superimposed on the moving body image. Therefore, the operator may mistakenly recognize the left and right when the front of the moving body is used as a reference. In addition, similar problems occur when the number of recognized dynamic targets is small and when the recognized dynamic targets are biased toward the right or left region. In this regard, according to the third viewpoint, recognition assistance for at least one of the bounding box and the trajectory of the dynamic target, which are the premise for recognition of the dynamic target, is combined with at least one other recognition assistance. Therefore, it is possible to prevent the occurrence of the above-mentioned problems.

[0026] When using a bounding box surrounding a dynamic target as a recognition aid, the question arises as to what kind of bounding box should be assigned to a dynamic target that crosses the boundary separating the right and left regions. In this regard, according to the fourth aspect, the double bounding boxes can each be assigned a color for the right and left recognition aids. Also, according to the fifth aspect, a color different from the colors for the right and left recognition aids can be assigned to a bounding box surrounding a dynamic target that crosses a boundary. Therefore, according to the fourth or fifth aspect, it becomes possible to assign an appropriate bounding box to a dynamic target that crosses a boundary.

[0027] When many dynamic targets are recognized in a moving object image, there is a problem that the bounding boxes surrounding the targets reduce visibility. In this regard, according to the sixth aspect, when it is determined that the total number of dynamic targets is equal to or greater than the upper limit, dynamic targets to which bounding boxes are assigned can be selected according to a predetermined criterion. Therefore, it is possible to reduce the number of bounding boxes superimposed on a moving object image to an appropriate number and suppress the reduction in visibility caused by the bounding boxes.

[0028] According to the seventh aspect, the same color is set for the recognition assistance on the right side in the front and rear images, and the same color is set for the recognition assistance on the left side in the front and rear images. Therefore, it is possible to reduce the occurrence of a problem in which the operator erroneously recognizes left and right when the front of the moving object is used as a reference in the front and rear images.

[0029] The problems described in the fourth and fifth aspects also apply to dynamic targets recognized in the front and rear images. In this regard, according to the eighth aspect, when the number of dynamic targets recognized in the right and left regions of the front image is one or more, and it is determined that a dynamic target has been recognized in at least one of the right and left regions of the rear image, a bounding box is adopted as a recognition aid for the right and left. Therefore, by using the bounding box superimposed on the front and rear images, it is possible to reduce the occurrence of a problem in which an operator erroneously recognizes the left and right when the front of the moving object is used as a reference.

[0030] According to the ninth aspect, even if the boundary separating the right and left regions has a predetermined shape, it is possible to provide recognition aids to this boundary in a color different from the colors of both the right and left recognition aids.

[0031] According to the tenth aspect, the right and left recognition assistants superimposed on the moving body image are also superimposed on the infrastructure image. According to the tenth aspect, the color of the recognition assistant superimposed on the right region of the infrastructure image is the same color as the color of the recognition assistant superimposed on the right region of the moving body image, and the color of the recognition assistant superimposed on the left region of the infrastructure image is the same color as the color of the recognition assistant superimposed on the left region of the moving body image. Therefore, it is expected to help an operator who checks both the moving body image and the infrastructure image to make an accurate judgment, and to contribute to the safe movement of moving bodies by remote work.

[0032] According to the eleventh aspect, right front, left front, right rear, and left rear areas based on the front of the moving object are set on the infrastructure image, and recognition assistance for the right front, left front, right rear, and left rear is superimposed on these areas, respectively. In addition, the color of the recognition assistance for the right front is different from the color of the recognition assistance for the right rear, and the color of the recognition assistance for the left front is different from the color of the recognition assistance for the left rear superimposed on the infrastructure image, so that it is possible to add information on the forward and backward directions of the moving object to the infrastructure image. This is expected to help the operator make accurate judgments and contribute to safe movement of the moving object by remote operation.

[0033] According to the twelfth aspect, an icon of a moving object and pictograms to the right and left of the icon, which surround the icon and are based on the front of the moving object, are superimposed on a map image and output from a display. The color of the right pictogram is the same color as the color of the right recognition aid superimposed on the moving object image. On the other hand, the color of the left pictogram is the same color as the color of the left recognition aid superimposed on the moving object image. This enables the operator to instantly recognize the front and rear of the moving object.

[0034] According to the thirteenth or fifteenth aspect, right and left regions based on the front of the moving body are set in an infrastructure image generated based on data acquired from an infrastructure camera whose distance from the moving body is equal to or less than a predetermined value. According to the thirteenth or fifteenth aspect, an infrastructure image in which the right and left are colored differently is output from a display. This makes it possible to reduce the occurrence of a problem in which an operator erroneously recognizes left and right when the front of the moving body in the infrastructure image is used as a reference. [Brief description of the drawings]

[0035] [Figure 1] FIG. 2 is a diagram for explaining a remote operation on which the first embodiment is based. [Diagram 2] FIG. 2 is a diagram for explaining an example of a sensor image (moving object image). [Diagram 3] FIG. 1 is a diagram illustrating a problem in remote work. [Figure 4]FIG. 13 is a diagram showing a first example of recognition assistance. [Diagram 5] FIG. 13 is a diagram showing a second example of recognition assistance. [Figure 6] FIG. 13 is a diagram showing a third example of recognition assistance. [Figure 7] FIG. 13 is a diagram showing a fourth example of recognition assistance. [Figure 8] FIG. 13 is a diagram showing a fifth example of recognition assistance. [Figure 9] FIG. 13 is a diagram showing a sixth example of recognition assistance. [Figure 10] 1 is a block diagram showing an example of the configuration of a display system according to a first embodiment. [Figure 11] FIG. 2 is a block diagram showing an example of a functional configuration particularly related to the first embodiment. [Figure 12] 12 is a diagram for explaining an example of processing by the left-right definition unit shown in FIG. 11; [Figure 13] FIG. 8 shows a first variant of the fourth example of recognition assistance described in FIG. 7. [Figure 14] FIG. 8 shows a second variant of the fourth example of recognition assistance described in FIG. 7. [Figure 15] FIG. 8 shows a third variant of the fourth example of recognition assistance described in FIG. 7. [Figure 16] FIG. 8 shows a fourth variant of the fourth example of recognition assistance described in FIG. 7. [Figure 17] FIG. 11 is a diagram for explaining a remote operation on which the second embodiment is based. [Figure 18] 1A to 1C are diagrams illustrating examples of sensor images (moving body images and infrastructure images). [Figure 19] FIG. 5 is a diagram illustrating a case where the first example of recognition assistance described in FIG. 4 is applied to an infrastructure image. [Figure 20] FIG. 11 is a block diagram showing an example of the configuration of a display system according to a second embodiment. [Figure 21] FIG. 13 is a block diagram showing an example of a functional configuration particularly related to the second embodiment. [Figure 22] 22 is a diagram for explaining an example of processing by the left-right definition unit shown in FIG. 21. [Diagram 23]11A and 11B are diagrams illustrating an example of a process for dividing right and left areas in an infrastructure image. [Figure 24] 11A and 11B are diagrams illustrating an example of a process for superimposing various icons on a map image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Hereinafter, a display method and a display system according to an embodiment of the present disclosure will be described with reference to the drawings. The display method according to the embodiment is realized by computer processing performed in the display system according to the embodiment. In each drawing, the same or corresponding parts are given the same reference numerals, and the description thereof will be simplified or omitted.

[0037] First embodiment First, a first embodiment of the present disclosure will be described with reference to FIGS.

[0038] 1. Overview 1-1.Remote work Fig. 1 is a diagram for explaining remote operation on which the first embodiment is based. Fig. 1 illustrates a moving body 1 and a remote operation device (hereinafter also referred to as a "terminal") 2. The moving body 1 is an example of a target that receives remote operation by the terminal 2. Examples of the moving body 1 include vehicles and robots such as logistics robots and work robots.

[0039] An example of a remote operation is remote driving, which is intended to perform operations directly related to the behavior of the moving body 1 (e.g., running, stopping, turning, etc.). Remote operations also include remote commands and remote assistance. Remote commands are performed to perform operations not directly related to the behavior of the moving body 1 (e.g., turning on the lights). Remote assistance is performed to assist the moving body 1 in its recognition and judgment.

[0040] The mobile object 1 and the terminal 2 communicate with each other via a base station (not shown) of a network. For example, the data processing device 12 mounted on the mobile object 1 determines whether or not remote work is required. If it is determined that remote work is required, the data processing device 12 transmits a remote work request to the terminal 2. When transmitting the remote work request, or prior to transmitting the remote work request, the data processing device 12 transmits a sensor image SIM to the terminal 2. The sensor image SIM is generated based on data acquired from a sensor 11 mounted on the mobile object 1. Hereinafter, the sensor image SIM transmitted from the mobile object 1 is also referred to as a "mobile object image SIM_1."

[0041] The data processing device 22 of the terminal 2 responds to the remote operation request and transmits various commands INS to the moving body 1. Examples of the commands INS transmitted from the data processing device 22 to the moving body 1 include commands based on the judgment of the operator OP. For example, the operator OP understands the contents of the remote operation request by looking at the sensor image SIM (moving body image SIM_1) output from the display 21 of the terminal 2. Then, the operator OP determines the action that the moving body 1 should take, and inputs instructions corresponding to this action to the terminal 2. Examples of the actions that the moving body 1 should take include proceeding, stopping, offset avoidance of an obstacle ahead, emergency evacuation, etc. The data processing device 22 generates a command INS according to the input data from the operator OP and transmits it to the moving body 1.

[0042] FIG. 2 is a diagram for explaining an example of a moving body image SIM_1. In FIG. 2, a vehicle is depicted as a moving body 1, and sensors 11a and 11b mounted on the vehicle are depicted. The direction DR of the arrow shown in FIG. 2 is the traveling direction of the moving body 1. The sensor 11a is a sensor 11 that detects the situation ahead of the moving body 1, and the sensor 11b is a sensor 11 that detects the situation behind the moving body 1. The sensors 11a and 11b are typically cameras, but the moving body image SIM_1 can also be generated based on data acquired from a LIDAR (Laser Imaging Detection and Ranging). Therefore, the sensors 11a and 11b may be LIDAR.

[0043] 2 is an example of a moving object image SIM_1 generated based on data acquired from the sensor 11a. The forward image SIM_11a includes an image of a dynamic object OB1 (pedestrian) existing on the road shoulder (sidewalk) to the right front of the moving object 1. On the other hand, the rearward image SIM_11b is an example of a moving object image SIM_1 generated based on data acquired from the sensor 11b. The rearward image SIM_11b includes an image of a dynamic object OB2 (pedestrian) existing on the road shoulder (sidewalk) to the left rear of the moving object 1.

[0044] 1-2.Challenges in remote work Fig. 3 is a diagram for explaining the problem in remote work. In Fig. 3, a combination of two types of moving object images SIM_1 described with reference to Fig. 2 is depicted in the upper and lower parts, respectively. The front image SIM_11a is output to the display 21a, and the rear image SIM_11b is output to the display 21b.

[0045] The operator OP, viewing the forward image SIM_11a outputted on the display 21a, can easily determine whether the dynamic object OB1 is present on the right front side or the left rear side of the moving body 1. On the other hand, the operator OP, viewing the rear image SIM_11b outputted on the display 21b, may erroneously determine whether the dynamic object OB2 is present on the right rear side or the left rear side of the moving body 1.

[0046] The reason for this is that psychological experiments have revealed that there are individual differences in whether people perceive their image reflected in a mirror as being reversed left to right or as nothing being reversed (Y. Takano et al., “Mirror reversal: Empirical tests of competing accounts”, Quarterly Journal of Experimental Psychology Society, 2007, 60(11), 1555-1584). According to the psychological experiment, 53.9% of the subjects answered that the image reflected in the mirror was reversed left to right, while 45.1% answered that nothing was reversed.

[0047] In other words, this result suggests that 53.9% of operators OP who viewed the rear image SIM_11b may recognize that the left and right of the rear image SIM_11b are reversed from that of the front image SIM_11a, while 45.1% of operators OP may recognize that this reversal does not occur. The existence of such individual differences is an obstacle to the safe performance of remote work. In addition, if an operator OP confuses the front image SIM_11a and the rear image SIM_11b and views them, there is a high possibility that the operator OP will misrecognize the left and right in these moving body images SIM_1 when the front of the moving body 1 is used as a reference.

[0048] 1-3. Recognition aid In view of such problems, in the first embodiment, the moving object image SIM_1 on which the "recognition aid" is superimposed is output from the display 21. The recognition aid superimposed on the moving object image SIM_1 is colored in different colors on the left and right sides with respect to the front of the moving object 1. Specifically, the recognition aid superimposed on the left region of the forward image SIM_11a is colored in the same color system as the recognition aid superimposed on the left region of the rearward image SIM_11b, and the recognition aid superimposed on the right region of the forward image SIM_11a is colored in the same color system as the recognition aid superimposed on the right region of the rearward image SIM_11b.

[0049] Here, the right and left regions are specified by setting boundaries that extend in the forward and backward directions of the moving body 1. The method of setting the right and left regions will be described later. Colors of the same family refer to a combination of colors with different lightness or saturation but the same hue, a combination of colors with different lightness and saturation (different tones) but the same hue, or a combination of colors with the same lightness and saturation (same tone) but adjacent in terms of hue (adjacent colors). An example of recognition assistance will be described below.

[0050] 1-3-1. First example Fig. 4 is a diagram showing a first example of recognition assistance. In Fig. 4, two types of moving object images SIM_1 are depicted in the upper and lower parts, respectively. A forward image SIM_11a is output to the display 21a, and a rearward image SIM_11b is output to the display 21b. Up to this point, the explanation is the same as that of Fig. 3.

[0051] The front image SIM_11a and the rear image SIM_11b are divided into a left area LA and a right area RA by a boundary BD1. Note that the boundary BD1 is shown in FIG. 4 for convenience of explanation, and the boundary BD1 is not superimposed on the front image SIM_11a and the rear image SIM_11b. The left area LA and the right area RA are specified by using the boundary BD1, and a method for setting these areas will be described later.

[0052] In the first example, arc-shaped recognition assistants AS_R1 to AS_R3 are superimposed on the right area RA of the forward image SIM_11a, and arc-shaped recognition assistants AS_R4 to AS_R6 are also superimposed on the right area RA of the rear image SIM_11b. These arcs are part of the circumference of concentric circles drawn on a virtual horizontal plane centered on a reference point of the moving object 1 (for example, the center position of the rear wheel axle of the vehicle, the installation position of the sensor 11a or 11b, the position of the driver's seat). For example, the recognition assistants AS_R1 and AS_R4 form the same circle, the recognition assistants AS_R2 and AS_R5 form the same circle, and the recognition assistants AS_R3 and AS_R6 form the same circle.

[0053] Although the recognition assistants AS_R1 to AS_R6 are of the same color family, the colors can be the same for each recognition assistant constituting the same circle. For example, the same color can be used for the recognition assistants AS_R1 and AS_R4, the same color can be used for the recognition assistants AS_R2 and AS_R5, and the same color can be used for the recognition assistants AS_R3 and AS_R6. In this case, a color scheme technique (gradation) may be used that regularly changes at least one of the brightness and saturation of the recognition assistants according to the distance from the reference point of the moving object 1.

[0054] In the first example, the recognition assistants AS_L1 to AS_L3 in the arc shape are superimposed on the left area LA of the forward image SIM_11a, and the recognition assistants AS_L4 to AS_L6 in the arc shape are superimposed on the left area LA of the rearward image SIM_11b. The explanation for the recognition assistants AS_L1 to AS_L6 is basically the same as that for the recognition assistants AS_R1 to AS_R6. However, the recognition assistants AS_L1 to AS_L6 are colored in a different color (different color) from the recognition assistants AS_R1 to AS_R6.

[0055] 1-3-2. Second example Fig. 5 is a diagram showing a second example of recognition assistance. In Fig. 5, two types of moving object images SIM_1 are depicted in the upper and lower parts, respectively. Note that the output manner of the front image SIM_11a and the rear image SIM_11b in Fig. 5 and the division into the left area LA and the right area RA by the boundary BD1 are the same as in the first example.

[0056] In the second example, the recognition assistant AS_R7 covering the entire right area RA of the forward image SIM_11a is superimposed, and the recognition assistant AS_R8 covering the entire right area RA of the rearward image SIM_11b is superimposed. The area covered by the recognition assistant AS_R7 may be a part of the right area RA of the forward image SIM_11a, and the area covered by the recognition assistant AS_R8 may be a part of the right area RA of the rearward image SIM_11b. In this case, the shapes of the recognition assistant AS_R7 and the recognition assistant AS_R8 are not particularly limited, and may be a circle, an ellipse, a polygon, a star-shaped polygon, or a straight line (such as a solid line, a dashed line, or a dashed dotted line). The recognition assistant AS_R7 and the recognition assistant AS_R8 are of the same color system.

[0057] In the second example, a recognition assistant AS_L7 covering the entire left area LA of the forward image SIM_11a is superimposed, and a recognition assistant AS_L8 covering the entire left area LA of the rearward image SIM_11b is superimposed. The explanation for the recognition assistants AS_L7 and AS_L8 is basically the same as that for the recognition assistants AS_R7 and AS_R8. However, the recognition assistants AS_L7 and AS_L8 are colored in a color different from that of the recognition assistants AS_R7 and AS_R8.

[0058] 1-3-3.Third example Fig. 6 is a diagram showing a third example of recognition assistance. In Fig. 6, two types of moving object images SIM_1 are depicted in the upper and lower parts, respectively. Note that the output manner of the front image SIM_11a and the rear image SIM_11b in Fig. 6 and the division into the left area LA and the right area RA by the boundary BD1 are the same as in the first example.

[0059] In the third example, a recognition assistant AS_R9 covering a recognized lane in the right area RA of the forward image SIM_11a is superimposed, and a recognition assistant AS_R10 covering a recognized lane in the right area RA of the rearward image SIM_11b is superimposed. These lanes are examples of static targets included in the forward image SIM_11a. Recognition of static targets is performed by a known method such as machine learning. Other examples of static targets include guardrails, road structures (traffic lights, traffic signs), etc. The recognition assistant AS_R9 and the recognition assistant AS_R10 are of the same color system.

[0060] In the third example, a recognition assistant AS_L9 covering the recognized lane in the left area LA of the forward image SIM_11a is superimposed, and a recognition assistant AS_L10 covering the recognized lane in the left area LA of the rearward image SIM_11b is superimposed. The explanation for the recognition assistants AS_L9 and AS_L10 is basically the same as that for the recognition assistants AS_R9 and AS_R10. However, the recognition assistants AS_L9 and AS_L10 are colored in a color different from that of the recognition assistants AS_R9 and AS_R10.

[0061] 1-3-4. Fourth Example Fig. 7 is a diagram showing a fourth example of recognition assistance. In Fig. 7, two types of moving object images SIM_1 are depicted in the upper and lower parts, respectively. Note that the output manner of the front image SIM_11a and the rear image SIM_11b in Fig. 7 and the division into the left area LA and the right area RA by the boundary BD1 are the same as in the first example.

[0062] In the fourth example, a recognition assistant AS_R11 (bounding box) surrounding a dynamic object OB1 recognized in a right area RA of the forward image SIM_11a is superimposed. In the fourth example, a recognition assistant AS_L11 (bounding box) surrounding a dynamic object OB2 recognized in a left area LA of the forward image SIM_11a is also superimposed. The recognition of the dynamic object is performed by a known method such as machine learning. Examples of the dynamic object include a pedestrian shown in FIG. 7, a bicycle, a motorcycle, and an automobile.

[0063] The recognition assistance AS_L11 is colored in a color different from that of the recognition assistance AS_R11. If the moving target OB2 is recognized in the right area RA of the rear image SIM_11b, the recognition assistance AS_L11 is colored in a color similar to that of the recognition assistance AS_R11.

[0064] 1-3-5. Fifth Example Fig. 8 is a diagram showing a fifth example of recognition assistance. In Fig. 8, two types of moving object images SIM_1 are depicted in the upper and lower parts, respectively. Note that the output manner of the front image SIM_11a and the rear image SIM_11b in Fig. 8, and the division into the left area LA and the right area RA by the boundary BD1 are the same as in the first example.

[0065] In the fifth example, the trajectory of the moving object 1 is superimposed on the front image SIM_11a and the rear image SIM_11b. The trajectory of the moving object 1 superimposed on the front image SIM_11a is specifically the future trajectory of the moving object 1 based on the current time point, and is generated, for example, based on vehicle control data including at least one of driving, braking, and steering of the moving object 1. The trajectory of the moving object 1 superimposed on the rear image SIM_11b is the past trajectory of the moving object 1 based on the current time point, and is generated, for example, based on history data of the lateral position of the moving object 1 relative to a reference position set on a lane.

[0066] In the fifth example, a recognition assistance AS_R12 is superimposed on the trajectory of the moving object 1 included in the right area RA of the forward image SIM_11a, and a recognition assistance AS_L12 is superimposed on the trajectory of the moving object 1 included in the left area LA of this image. The recognition assistance AS_L12 is colored in a color different from that of the recognition assistance AS_R12.

[0067] In the fifth example, a recognition assistance AS_R13 is superimposed on the trajectory of the moving object 1 included in the right area RA of the rear image SIM_11b, and a recognition assistance AS_L13 is superimposed on the trajectory of the moving object 1 included in the left area LA of this image. The recognition assistance AS_L13 is colored in a color different from that of the recognition assistance AS_R13.

[0068] The color of the recognition assistance AS_L13 can be the same as that of the recognition assistance AS_L12, or the same color can be used. The color of the recognition assistance AS_R13 can be the same as that of the recognition assistance AS_R12, or the same color can be used.

[0069] The idea of ​​the fifth example can also be applied to the trajectory of a dynamic target recognized in the moving body image SIM_1. In this case, for example, a dynamic target included in the moving body image SIM_1 is recognized by a known method such as machine learning, and the past trajectory of this dynamic target is identified. Then, this past trajectory may be drawn on the moving body image SIM_1, and a recognition assistance may be superimposed along this. Furthermore, the future trajectory of the dynamic target may be predicted from the identified past trajectory. In this case, the future trajectory of this dynamic target may be drawn on the moving body image SIM_1, and a recognition assistance may be superimposed along this.

[0070] 1-3-6. Sixth Example Fig. 9 is a diagram showing a sixth example of recognition assistance. In Fig. 9, two types of moving object images SIM_1 are depicted in the upper and lower parts, respectively. The output manner of the front image SIM_11a and the rear image SIM_11b in Fig. 9 is the same as in the first example.

[0071] Unlike the first to fifth examples, in the sixth example, the boundary BD1 has an area shape with a certain width in the left-right direction. In the sixth example, the arc-shaped recognition assistants AS_B1 to AS_B3 are superimposed on the boundary BD1 of the front image SIM_11a, and the arc-shaped recognition assistants AS_B4 to AS_B6 are superimposed on the boundary BD1 of the rear image SIM_11b. The recognition assistant AS_B1 is located between the recognition assistants AS_R1 and AS_L1. The recognition assistants AS_B2 to AS_B6 are located between the recognition assistants AS_Rk and AS_Lk (k=2 to 6), respectively.

[0072] The explanation for the recognition assistants AS_B1 to AS_B6 is basically the same as that for the recognition assistants AS_R1 to AS_R6. However, the recognition assistants AS_B1 to AS_B6 are colored in a color different from the color of the recognition assistants AS_R1 to AS_R6 and a color different from the color of the recognition assistants AS_L1 to AS_L6 (i.e., a third color).

[0073] In the sixth example, the position of the boundary BD1 may be moved in the left-right direction. For example, if the position of the driver's seat of the moving object 1 is set as the reference point of the moving object 1, the position of the boundary BD1 moves. This modification is similarly applied to the first to fifth examples.

[0074] In the sixth example, the shape of the boundary BD1 may be arbitrarily changed. Other shapes of the boundary BD1 include a triangle and a trapezoid. For example, by setting a vertex at the reference point of the moving object 1 and setting the remaining two vertices of the triangle on the periphery of the front image SIM_11a and the rear image SIM_11b, the boundary BD1 having a triangle can be set on these images. Also, by gradually changing the width of the boundary BD1 in the left-right direction from the front to the rear of the moving object 1, the boundary BD1 having a trapezoid can be set on these images.

[0075] In this way, according to the first embodiment, a recognition aid in which the left and right sides are colored in different colors with respect to the front of the moving object 1 as a reference is superimposed on the moving object image SIM_1, and this is output from the display 21. Therefore, it is possible to reduce the occurrence of a problem in which the operator OP erroneously recognizes the left and right sides with respect to the front of the moving object 1 in the moving object image SIM_1 as a reference.

[0076] 2. System configuration example Next, a configuration example of the display system according to the first embodiment will be described.

[0077] 2-1. Overall configuration example Fig. 10 is a block diagram showing a configuration example of the display system according to the first embodiment. In the example shown in Fig. 10, a moving object 1 including a sensor 11 and a data processing device 12, and a terminal 2 including a display 21, a data processing device 22, and an input device 23 are depicted as the configuration of the display system according to the first embodiment.

[0078] The sensor 11 is mounted on the moving body 1. The sensor 11 detects the situation at least in front and behind the moving body 1. Examples of the sensor 11 include a camera and LIDAR. Specific examples of the sensor 11 include sensors 11a and 11b shown in FIG. 2. However, the sensor 11a may include three sensors that detect the situation in the front center, right front, and left front of the moving body 1. Furthermore, the sensor 11b may include three sensors that detect the situation in the rear center, right rear, and left rear of the moving body 1.

[0079] Furthermore, the sensor 11 may include a sensor that detects the situation on the right side and the left side of the moving body 1. The sensor that detects the situation on the right side (or the left side) of the moving body 1 may also detect the situation on the right rear (or the left rear) of the moving body 1.

[0080] The data processing device 12 is a microcomputer mounted on the moving object 1. The data processing device 12 includes at least one processor 12a and at least one memory 12b. The processor 12a includes a CPU (Central Processing Unit). The memory 12b is a volatile memory such as a DDR memory, and expands various programs used by the processor 12a and temporarily stores various data. Various data acquired from the sensor 11, a moving object image SIM_1 transmitted by the moving object 1 to the terminal 2, and a command INS received by the moving object 1 from the terminal 2 are stored in the memory 12b.

[0081] The processor 12a executes a program for image processing stored in the memory 12b to perform various processes related to the generation of the moving body image SIM_1. The processor 12a also executes a program for image transmission stored in the memory 12b to perform various processes for transmitting the generated moving body image SIM_1 to the terminal 2. The processor 12a further executes a program for remote operation stored in the memory 12b to perform various processes related to the control of the moving body 1 based on the command INS received from the terminal 2.

[0082] The display 21 is a display device on which the moving object image SIM_1 is output. The display 21 is installed, for example, in the facility of a business providing remote operation services for the moving object 1. The display 21 and the data processing device 22 are connected by wire or wirelessly. In the latter case, the display 21 may be connected to the data processing device 22 via a base station of a network. The total number of displays 21 is not particularly limited, but it is desirable that at least the front and rear images of the moving object 1 (for example, the front image SIM_11a and the rear image SIM_11b) are output separately. Therefore, it is desirable that the total number of displays 21 is at least two.

[0083] The data processing device 22 is a computer for performing remote operations on the moving object 1. The data processing device 22 includes at least one processor 22a and at least one memory 22b. The processor 22a includes a CPU (Central Processing Unit). The memory 22b is a volatile memory such as a DDR memory, and expands various programs used by the processor 22a and temporarily stores various data. The moving object image SIM_1 that the terminal 2 receives from the moving object 1, various data from the input device 23, and the command INS that the terminal 2 transmits to the moving object 1 are stored in the memory 22b.

[0084] The processor 22a executes a program for output processing stored in the memory 12b to perform various processes related to output control of the moving body image SIM_1. The processor 22a also executes a program for remote operation stored in the memory 22b to perform various processes for generating a command INS to be transmitted to the moving body 1. The processor 22a further executes a program for command transmission to perform various processes for transmitting the generated command INS to the moving body 1.

[0085] The input device 23 is operated by an operator OP. The input device 23 includes, for example, an input unit operated by the operator OP, and a control circuit that generates and outputs a signal based on input data. Examples of the input unit include a mouse, a keyboard, a button, and a switch. Other examples of the input unit include a steering wheel, a shift lever, an accelerator pedal, and a brake pedal. Examples of signals generated based on an operational input include a signal for changing (enlarging, reducing, etc.) the display mode of the moving body image SIM_1 on the display 21, a signal corresponding to an action to be taken by the moving body 1, and the like.

[0086] 2-2. Example of functional configuration 2-2-1. Example of functional configuration of data processing device 12 Fig. 11 is a block diagram showing an example of the configuration of functions particularly related to the first embodiment. In the example shown in Fig. 11, a data processing device 12 includes a data receiving unit 13, an image generating unit 14, a left / right defining unit 15, a recognition assistance superimposing unit 16, and a data transmitting unit 17. The functions of each block shown in the data processing device 12 are realized by the processor 12a executing various programs. Note that some of the functions of the data processing device 12 (for example, the image generating unit 14, the left / right defining unit 15, and the recognition assistance superimposing unit 16) may be realized by program processing by a server external to the moving body 1.

[0087] The data receiving unit 13 receives various data detected by the sensor 11. The various data include data indicating the situation in front of the moving body 1 and data indicating the situation in the rear of the moving body 1. The data indicating the situation in the front may include data indicating the situation in the center front, right front, and left front of the moving body 1. The data indicating the situation in the rear may include data indicating the situation in the center rear, right rear, and left rear of the moving body 1. The various data acquired by the sensor 11 may include data indicating the situation on the right side and left side of the moving body 1. The data receiving unit 13 transmits the various data to the image generating unit 14.

[0088] The image generating unit 14 generates a sensor image SIM (moving object image SIM_1) based on various data received from the data receiving unit 13. When the sensor 11 is a camera, the image generating unit 14 generates the moving object image SIM_1 based on a color image detected by the camera. When the sensor 11 is a LiDAR, the image generating unit 14 generates the moving object image SIM_1 based on point cloud data detected by the LIDAR. The image generating unit 14 transmits the generated moving object image SIM_1 to the left / right defining unit 15 and the recognition assistance superimposing unit 16.

[0089] The left / right definition unit 15 defines the left and right in the moving body image SIM_1. First, the left / right definition unit 15 sets a boundary. FIG. 12 is a diagram for explaining a processing example by the left / right definition unit 15. Note that in FIG. 12, the processing example is explained on the assumption that the moving body 1 is a vehicle. In the example shown in FIG. 12, first, a virtual horizontal plane VHS passing through a reference point of the moving body 1 (for example, the center position of the rear wheel axle of the vehicle) is set (step S1). The virtual horizontal plane VHS is, for example, a plane parallel to the lane on which the moving body 1 is traveling.

[0090] Once the virtual horizontal plane VHS is set, a reference line RL extending in the front-rear direction of the moving body 1 is set on the virtual horizontal plane VHS (step S1). The front-rear direction is, for example, a direction connecting the center of the front of the body of the moving body 1 and the center of the rear of the body. In the example shown in Fig. 12, the reference line RL is set by projecting a straight line passing through these centers onto the virtual horizontal plane VHS.

[0091] Once the reference line RL is set, a plane including the reference line RL and perpendicular to the virtual horizontal plane VHS (hereinafter also referred to as a "boundary plane BS") is set (step S2). Once the boundary plane BS is set, a right space RS and a left space LS can be defined in the virtual space VSP including the boundary plane BS and the virtual horizontal plane VHS, with the front of the moving object 1 as the reference.

[0092] The left-right definition unit 15 applies the virtual space VSP including the boundary surface BS and the virtual horizontal plane VHS to the moving body image SIM_1 (step S2). Then, the boundary BD1 depicted by the boundary surface BS can be set on the moving body image SIM_1. When the boundary BD1 is set, the right area RA and the left area LA based on the front of the moving body 1 can be set on the moving body image SIM_1.

[0093] The left / right definition unit 15 transmits data of the right area RA and the left area LA in the moving object image SIM_1 to the recognition assistance superimposition unit 16. The left / right definition unit 15 transmits, for example, data of pixel coordinates constituting the boundary BD1 in the moving object image SIM_1 to the recognition assistance superimposition unit 16. When an arc-shaped recognition assistance is superimposed on the moving object image SIM_1 (that is, in the case of the first example described with reference to FIG. 4), the left / right definition unit 15 transmits data of pixel coordinates constituting the virtual horizontal plane VHS in the moving object image SIM_1 to the recognition assistance superimposition unit 16.

[0094] The recognition assistance superimposing unit 16 superimposes recognition assistance on the moving object image SIM_1 based on the moving object image SIM_1 received from the image generating unit 14 and various data related to the moving object image SIM_1 received from the left / right defining unit 15.

[0095] In the case of the first example described with reference to Fig. 4, the recognition assistance superimposition unit 16 uses pixel coordinate data constituting a virtual horizontal plane VHS in the moving object image SIM_1 to draw virtual concentric circles centered on the reference point of the moving object 1 on this virtual horizontal plane VHS. Next, the recognition assistance superimposition unit 16 applies these virtual concentric circles to the moving object image SIM_1, and superimposes recognition assistance elements AS_R1 to AS_R6 on the right area RA and superimposes recognition assistance elements AS_L1 to AS_L6 on the left area LA.

[0096] In the second example described with reference to Fig. 5, the recognition assistance superimposition unit 16 superimposes the recognition assistances AS_R7 and AS_R8 covering the entire right area RA, and superimposes the recognition assistances AS_L7 and AS_L8 on the left area LA. In order to prevent the visibility of the moving object image SIM_1 from being reduced by the superimposed recognition assistance, it is preferable to color the right area RA and the left area LA using a transparent color in the second example. In another example, when a non-transparent color is used for coloring, it is preferable to change the transparency value of this color according to time.

[0097] In the case of the third example described with reference to Fig. 6, the recognition assistance superimposing unit 16 first recognizes a target included in the moving body image SIM_1. When a static target included in the moving body image SIM_1 is recognized, the recognition assistance superimposing unit 16 superimposes the recognition assistance AS_R9, AS_R10, AS_L9 or AS_L10 covering the pixel coordinates constituting the static target according to the area in which the static target is recognized. Note that in order to prevent the visibility of the static target from being reduced by the superimposed recognition assistance, in the case of the third example, it is desirable to use a transparent color or to change the transmission value of a non-transparent color according to time.

[0098] 7, the recognition assistance superimposing unit 16 first recognizes a target included in the moving body image SIM_1, as in the third example. When a dynamic target included in the moving body image SIM_1 is recognized, the recognition assistance superimposing unit 16 superimposes the recognition assistance AS_R11 or AS_L11 as a bounding box surrounding the dynamic target, depending on the area in which the dynamic target is recognized.

[0099] In the fourth example and the modified fifth example (coloring the trajectory of the dynamic target), it is assumed that the dynamic target included in the moving object image SIM_1 is not recognized at all. It is also assumed that the total number of dynamic targets included in the moving object image SIM_1 is small. It is also assumed that the dynamic targets recognized in the moving object image SIM_1 are biased toward one of the right area RA and the left area LA. It is also assumed that the total number of dynamic targets is too large.

[0100] If no dynamic targets are recognized at all, the moving body image SIM_1 is not colored. Therefore, in this case, there is a high possibility of misidentifying left and right when the front of the moving body 1 is used as a reference. Also, if the total number of dynamic targets is small, and if the recognized dynamic targets are biased toward one of the right area RA and the left area LA, there remains a possibility of misidentifying left and right when the front of the moving body 1 is used as a reference. On the other hand, if the total number of dynamic targets is too large, there is a possibility that the visibility of the moving body image SIM_1 will decrease instead.

[0101] Therefore, in the fourth example, it is desirable to count the number of dynamic targets included in the moving object image SIM_1, and it is desirable to count the number of dynamic targets included in the right area RA and the left area LA, respectively. In addition, it is desirable to determine whether or not the number of dynamic targets included in the right area RA and the left area LA of the forward image SIM_11a is 1 or more, and whether or not a dynamic target is included in at least one of the dynamic targets included in the right area RA and the left area LA of the rearward image SIM_11b. Then, if the result of this determination is positive, it is desirable to employ the recognition assistance AS_R11 or AS_L11 as a bounding box.

[0102] FIG. 13 is a diagram showing a first modification of the fourth example of the recognition assistance described with reference to FIG. 7. In the example shown in FIG. 13, a positive determination result is obtained in the determination based on the dynamic target. Specifically, in the example shown in FIG. 13, one dynamic target is recognized in each of the right area RA and the left area LA of the forward image SIM_11a, and one dynamic target is recognized in the left area LA of the rearward image SIM_11b. Therefore, the recognition assistance AS_R11a is superimposed on the right area RA of the forward image SIM_11a, and the recognition assistance AS_L11a is superimposed on the left area LA. In addition, the recognition assistance AS_L11b is superimposed on the left area LA of the rearward image SIM_11b.

[0103] On the other hand, if a negative determination result is obtained in the determination based on the dynamic target, it is desirable to appropriately combine the first to third examples or the fifth example (limited to coloring the trajectory of the static target) with the fourth example. FIG. 14 is a diagram showing a second modification of the fourth example of the recognition assistance described with reference to FIG. 7. In the example shown in FIG. 14, a negative determination result is obtained in the determination based on the dynamic target. Therefore, in the example shown in FIG. 14, the recognition assistances AS_R7 and AS_R8 and the recognition assistances AS_L7 and AS_L8 described in the second example are added to the moving object image SIM_1.

[0104] When the total number of dynamic targets is too large, it is desirable to select dynamic targets on which the recognition assistance AS_R11 or AS_L11 is superimposed (i.e., dynamic targets on which bounding boxes are assigned) according to a predetermined criterion. In this case, for example, it is determined whether the total number of dynamic targets in the image area of ​​interest (for example, the right area RA of the forward image SIM_11a) is equal to or greater than an upper limit.

[0105] Then, when it is determined that the total number of dynamic targets is equal to or greater than the upper limit, it is determined that the total number of dynamic targets is too large, and dynamic targets that satisfy any one of the criteria (i) to (iii) are extracted. (i) A dynamic target within a predetermined distance from the moving object 1 (ii) A dynamic target whose predicted time to collision (TTC) with the moving object 1 is within a predetermined time (iii) Dynamic targets that require special monitoring by the operator OP

[0106] It is desirable to repeatedly extract dynamic targets until the total number of dynamic targets falls below the upper limit. In this case, it is desirable to change the extraction criteria, for example, to extract dynamic targets that simultaneously satisfy two of the criteria (i) to (iii). The recognition assistance AS_R11 or AS_L11 as a bounding box is superimposed so as to surround the dynamic targets extracted in this manner.

[0107] The first and second modifications of the fourth example described above can be applied to the modification of the fifth example. In this case, the "bounding box" in the above description can be appropriately replaced with the "trajectory of a dynamic target".

[0108] In the case of the fourth example, another problem is also expected. That is, when the dynamic target is located so as to straddle the boundary BD1, it is difficult to determine which color of the recognition assistance AS_R11 or AS_L11 should be adopted. Therefore, in this case, for example, the ratio of the area of ​​the dynamic target in the right area RA and the left area LA is calculated. Then, it is desirable to adopt the same color as the color of the bounding box surrounding the dynamic target included in the area with the high area ratio as the color of the dynamic target straddling the boundary BD1.

[0109] However, adopting either one of the colors of the recognition assistance AS_R11 and AS_L11 as the color of the dynamic target straddling the boundary BD1 may hinder the operator OP from recognizing left and right. Therefore, as a second modification of the fourth example, it is desirable to paint the bounding box surrounding the dynamic target straddling the boundary BD1 with a color that is not similar to the recognition assistance AS_R11 and is not similar to the recognition assistance AS_L11 (i.e., a third color).

[0110] As a third example of coloring the bounding box surrounding the dynamic target straddling the boundary BD1, an example of changing the color in the middle of the bounding box is also considered. FIG. 15 is a diagram showing a third modification of the fourth example of the recognition assistance described with reference to FIG. 7. In the example shown in FIG. 15, the recognition assistance AS_R11c is superimposed on the part of the bounding box located in the right region RA, and the recognition assistance AS_L11c is superimposed on the part of the bounding box located in the left region LA. The color of the recognition assistance AS_R11c is the same as the color of the bounding box surrounding the dynamic target included in the right region RA, and the color of the recognition assistance AS_L11c is the same as the color of the bounding box surrounding the dynamic target included in the left region LA.

[0111] As a third example of coloring the bounding box surrounding the dynamic target straddling the boundary BD1, an example of changing the color in the middle of the bounding box is also considered. FIG. 15 is a diagram showing a third modification of the fourth example described with reference to FIG. 7. In the example shown in FIG. 15, a recognition assistant AS_R11c is superimposed on a portion of the bounding box located in the right region RA, and a recognition assistant AS_L11c is superimposed on a portion of the bounding box located in the left region LA. The color of the recognition assistant AS_R11c is the same as the color of the bounding box surrounding the dynamic target included in the right region RA, and the color of the recognition assistant AS_L11c is the same as the color of the bounding box surrounding the dynamic target included in the left region LA.

[0112] As a fourth example of coloring the bounding box surrounding the dynamic target straddling the boundary BD1, an example of doubling the bounding box is also considered. FIG. 16 is a diagram showing a fourth modification of the fourth example of the recognition assistance described in FIG. 7. In the example shown in FIG. 16, the recognition assistance AS_R11d and AS_L11d are superimposed as a double bounding box surrounding the dynamic target straddling the boundary BD1. The color of the recognition assistance AS_R11d is the same as the color of the bounding box surrounding the dynamic target included in the right area RA, and the color of the recognition assistance AS_L11d is the same as the color of the bounding box surrounding the dynamic target included in the left area LA.

[0113] 15 and 16, the recognition assistants AS_R7 and AS_R8 and the recognition assistants AS_L7 and AS_L8 described in the second example are added to the front image SIM_11a. However, the addition of these recognition assistants is optional.

[0114] Incidentally, when the coloring of the bounding box surrounding the dynamic target straddling the boundary BD1 is a combination including warning colors such as "red" and "blue," the following problem is expected. That is, when the coloring of the bounding box changes from "red" to "blue" as the dynamic target moves, the operator OP may mistakenly believe that the state of the dynamic target has changed to a safe state. Also, when the coloring of the bounding box changes from "blue" to "red," the operator OP may mistakenly believe that the state of the dynamic target has become dangerous. Therefore, it is desirable to use a warning color with reduced saturation and brightness as the color for the recognition assistance, or to not use a warning color at all.

[0115] Returning to Fig. 11, the description of the functions of the data processing device 12 will be continued. The recognition assistance superimposing unit 16 transmits the moving object image SIM_1 on which the recognition assistance is superimposed to the data transmitting unit 17. The data transmitting unit 17 encodes the moving object image SIM_1 on which the recognition assistance is superimposed and transmits it to the terminal 2. During the encoding process, the moving object image SIM_1 may be compressed.

[0116] The data transmission unit 17 may transmit, together with the moving body image SIM_1, identification data (ID data) of the moving body 1 and data related to the moving body image SIM_1. Examples of data related to the moving body image SIM_1 include data on the time when the moving body image SIM_1 was acquired and internal situation data (speed, acceleration, steering angle, etc.) of the moving body 1 at this time.

[0117] 2-2-2. Example of functional configuration of data processing device 22 11, the data processing device 22 includes a data receiving unit 24 and a display control unit 25. The functions of each block shown in the data processing device 22 are realized by the processor 22a executing various programs.

[0118] The data receiving unit 24 receives various data from the moving object 1. The various data includes a moving object image SIM_1. The various data may include identification data (ID data) of the moving object 1 and data related to the moving object image SIM_1. Examples of data related to the moving object image SIM_1 have already been described. The data receiving unit 24 decodes the various data received from the moving object 1 and transmits it to the display control unit 25. If the various data are compressed, the data receiving unit 24 expands the data in the decoding process.

[0119] The display control unit 25 performs various controls for outputting various data received from the data receiving unit 24 to the display 21. The various controls include output control of the moving object image SIM_1. In the output control of the moving object image SIM_1, for example, a process is performed for displaying a forward image SIM_11a on the display 21a and a rearward image SIM_11b on the display 21b. In another example, a process is performed for displaying the forward image SIM_11a in the main area of ​​the display 21 and displaying data related to the moving object image SIM_1 in a sub-area of ​​the display 21. In yet another example, a process is performed for displaying a rearward image SIM_11b in the sub-area instead of data related to the moving object image SIM_1.

[0120] 3.Effects According to the first embodiment described above, a recognition aid in which the left and right sides are colored in different colors with respect to the front of the moving body 1 as a reference is superimposed on the moving body image SIM_1, and this is output from the display 21. Therefore, it is possible to reduce the occurrence of a problem in which the operator OP erroneously recognizes the left and right sides with respect to the front of the moving body 1 as a reference in the moving body image SIM_1. This is expected to help the operator OP make accurate judgments when remotely operating the moving body 1, and to contribute to the safe movement of the moving body 1 by remote operation.

[0121] Moreover, according to various modifications of the fourth example of the recognition assistance, it is possible to prevent various problems that are expected to occur when a bounding box as a recognition assistance is superimposed on the moving body image SIM_1. This is also expected to help the operator OP make accurate decisions when remotely operating the moving body 1, and contribute to the safe movement of the moving body 1 by remote operation.

[0122] Second embodiment Next, a second embodiment of the present disclosure will be described with reference to Figures 17 to 24. Note that descriptions that overlap with the description of the first embodiment will be omitted as appropriate.

[0123] 1. Overview 1-1.Remote work Fig. 17 is a diagram for explaining a remote operation on which the second embodiment is based. Fig. 17 illustrates a moving object 1 and a terminal 2. The explanation up to this point is the same as that of Fig. 1. Fig. 17 further illustrates an infrastructure camera 3 and a display 26.

[0124] The infrastructure camera 3 is a camera installed on a structure (for example, a facility structure such as a road structure, or a ceiling, pillar, or wall of a parking lot or factory). The total number of infrastructure cameras 3 is one or more. The infrastructure camera 3 and the terminal 2 communicate with each other via a base station of the network. The infrastructure camera 3 transmits a sensor image SIM to the terminal 2. Hereinafter, the sensor image SIM transmitted from the infrastructure camera 3 is also referred to as an "infrastructure image SIM_3". The infrastructure image SIM_3 is generated based on data acquired from the infrastructure camera 3.

[0125] Like the display 21, the display 26 is connected to the data processing device 22. In the example shown in FIG. 17, an infrastructure image SIM_3 is output to the display 26. Like the total number of infrastructure cameras 3, the total number of displays 26 for outputting the infrastructure image SIM_3 is one or more. Note that the infrastructure image SIM_3 may be output from the display 21 together with the moving object image SIM_1. In this case, the display 26 may be omitted. For example, according to the picture-in-picture technology, it is possible to display the moving object image SIM_1 in a part of the display 21 and display the infrastructure image SIM_3 in another part of the display 21.

[0126] The position where the infrastructure camera 3 is installed is known. In the second embodiment, the moving object 1 constantly transmits the position data of the moving object 1 to the terminal 2. The data processing device 22, for example, based on the position data of the moving object 1 and the position data of the infrastructure camera 3, identifies the infrastructure camera 3 whose distance from the current position of the moving object 1 is equal to or less than a predetermined value. For example, the predetermined value is set for each infrastructure camera 3 as a distance at which at least a part of the moving object 1 falls within the angle of view of the infrastructure camera 3. In another example, the predetermined value is a distance at which at least a part of the moving object 1 is expected to fall within the angle of view of the infrastructure camera 3 in the future after a predetermined time (for example, several seconds) has elapsed from the current time. When the infrastructure camera 3 is identified, the data processing device 22 outputs the infrastructure image SIM_3 generated based on the data acquired from the identified infrastructure camera 3 from the display 26.

[0127] In this example, consider the case where a remote operation request is received from a moving object 1, or where a moving object image SIM_1 is received from the moving object 1 prior to transmission of this remote operation request. The operator OP looks at the moving object image SIM_1 output from the display 21 and the infrastructure image SIM_3 output from the display 26 to understand the content of the remote operation request. Or, looks at the infrastructure image SIM_3 output from the display 26 to understand the situation in which the moving object 1 is placed. Then, the operator OP determines the action that the moving object 1 should take, and inputs an instruction corresponding to this action to the terminal 2.

[0128] When the data processing device 22 receives a remote operation request from the moving object 1, or when the data processing device 22 receives a moving object image SIM_1 from the moving object 1 prior to transmitting the remote operation request, the data processing device 22 may identify an infrastructure camera 3 whose distance from the current position of the moving object 1 is equal to or less than a predetermined value. When an infrastructure camera 3 is identified, the data processing device 22 outputs an infrastructure image SIM_3 generated based on data acquired from the identified infrastructure camera 3 from the display 26.

[0129] FIG. 18 is a diagram for explaining an example of a moving object image SIM_1 and an infrastructure image SIM_3. FIG. 18 depicts a vehicle as a moving object 1 and a sensor 11a mounted on the vehicle. The direction DR of the arrow shown in FIG. 18 is the traveling direction of the moving object 1. FIG. 18 also depicts a camera body 31a of the infrastructure camera 3a and a camera body 31b of the infrastructure camera 3b. In the example shown in FIG. 18, the infrastructure cameras 3a and 3b are infrastructure cameras 3 whose distance from the current position of the moving object 1 is equal to or less than a predetermined value. The camera body 31a is located to the front right of the moving object 1, and the camera body 31b is located to the rear left of the moving object.

[0130] The forward image SIM_11a depicted in Fig. 18 is an example of a moving body image SIM_1 generated based on data acquired from the sensor 11a. The forward image SIM_11a includes an image of a dynamic object OB3 (pedestrian) present to the right front of the moving body 1. The infrastructure image SIM_3a is generated based on data acquired from the camera body 31a, and includes images of the dynamic object OB3 and the moving body 1. The infrastructure image SIM_3b is generated based on data acquired from the camera body 31b, and includes an image of the moving body 1 (an image of the rear part of the moving body 1).

[0131] 1-2. Recognition aid As in the first embodiment, in the second embodiment, a recognition aid is superimposed on a moving object image SIM_1, and this is output from the display 21. In the second embodiment, a recognition aid is also superimposed on an infrastructure image SIM_3, and this is output from the display 26. Examples of the recognition aid superimposed on the infrastructure image SIM_3 include the same examples as the first to sixth examples described with reference to Figs. 4 to 9. As a representative example, the first example of the recognition aid described with reference to Fig. 4 will be described.

[0132] Fig. 19 is a diagram illustrating a case where the first example of the recognition assistance described with reference to Fig. 4 is applied to an infrastructure image SIM_3. Fig. 19 illustrates a forward image SIM_11a and infrastructure images SIM_3a and SIM_3b. The forward image SIM_11a is output to the display 21a, the infrastructure image SIM_3a is output to the display 26a, and the infrastructure image SIM_3b is output to the display 26b.

[0133] In the example shown in Fig. 19, the forward image SIM_11a is divided into a left area LA and a right area RA by a boundary BD1. Arc-shaped recognition assistants AS_R1 to AS_R3 are superimposed on the right area RA, and arc-shaped recognition assistants AS_L1 to AS_L3 are superimposed on the left area LA. The recognition assistants AS_L1 to AS_L3 are colored in a color different from the color of the recognition assistants AS_R1 to AS_R3. Up to this point, the explanation is the same as that for the forward image SIM_11a in Fig. 4.

[0134] In the example shown in Fig. 19, the infrastructure images SIM_3a and SIM_3b are divided into a left area LA and a right area RA by a boundary BD2. In the infrastructure image SIM_3a, the right area RA is superimposed with arc-shaped recognition assistants AS_R14 and AS_R15, and the left area LA is superimposed with arc-shaped recognition assistants AS_L14 and AS_L15. In the infrastructure image SIM_3b, the right area RA is superimposed with arc-shaped recognition assistants AS_R16 and AS_R17, and the left area LA is superimposed with arc-shaped recognition assistants AS_L16 and AS_L17.

[0135] The arcs drawn on the infrastructure images SIM_3a and SIM_3b, like the arcs drawn on the forward image SIM_11a and the rearward image SIM_11b, are parts of the circumference of concentric circles drawn on a virtual horizontal plane centered on a reference point of the moving object 1 (e.g., the current position of the moving object 1). For example, the recognition assistance AS_R1 forms the same circle as the recognition assistance AS_R14 and AS_R16, and the recognition assistance AS_R2 forms the same circle as the recognition assistance AS_R15 and AS_R17. The recognition assistance AS_L1 forms the same circle as the recognition assistance AS_L14 and AS_L16, and the recognition assistance AS_L2 forms the same circle as the recognition assistance AS_L15 and AS_L17.

[0136] In this way, according to the second embodiment, an infrastructure image SIM_3 is generated based on data obtained from an infrastructure camera 3 whose distance from the current position of the moving object 1 is equal to or less than a predetermined value (such as a distance at which at least a part of the moving object 1 is expected to be included in the angle of view of the infrastructure camera 3 in the future after a predetermined time has passed from the current time). Then, a recognition aid colored in different colors on the left and right with respect to the front of the moving object 1 as a reference is superimposed on this infrastructure image SIM_3, and this is output from the display 26. Therefore, it is expected to help the operator OP make accurate decisions when remotely working on the moving object 1, and contribute to safe movement of the moving object 1 during remote work.

[0137] 2. System configuration example 2-1. Overall configuration example Fig. 20 is a block diagram showing a configuration example of a display system according to the second embodiment. In the example shown in Fig. 20, a moving object 1, a terminal 2, and an infrastructure camera 3 are depicted as the configuration of the display system according to the second embodiment.

[0138] The configuration of the moving object 1 is as described in the first embodiment. The terminal 2 includes displays 21 and 26, a data processing device 22, and an input device 23. An infrastructure image SIM_3 generated based on data obtained from the infrastructure camera 3 is output to the display 26.

[0139] The infrastructure camera 3 includes a camera body 31 and a data processing device 32. The configuration of the camera body 31 is basically the same as that of the camera serving as the sensor 11. The data processing device 32 includes at least one processor 32a and at least one memory 32b. The processor 32a executes an image processing program stored in the memory 32b to perform various processes related to the generation of the infrastructure image SIM_3. The processor 32a also executes an image transmission program stored in the memory 32b to perform various processes for transmitting the generated moving object image SIM_1 to the terminal 2.

[0140] 2-2. Example of functional configuration 2-2-1. Example of functional configuration of data processing device 32 Fig. 21 is a block diagram showing an example of a functional configuration particularly related to the second embodiment. In the example shown in Fig. 21, a data processing device 32 includes a data receiving unit 33, an image generating unit 34, and a data transmitting unit 35. The functions of each block shown in the data processing device 32 are realized by the processor 32a executing various programs.

[0141] The data receiving unit 33 receives the color image detected by the camera body 31. The data receiving unit 33 transmits this color image to the image generating unit .

[0142] The image generating unit 34 generates a sensor image SIM (infrastructure image SIM_3) based on the color image received from the data receiving unit 33. The image generating unit 34 transmits the generated infrastructure image SIM_3 to the data transmitting unit 35.

[0143] The data transmission unit 35 encodes the infrastructure image SIM_3 received from the image generation unit 34 and transmits it to the terminal 2. During the encoding process, the infrastructure image SIM_3 may be compressed. The data transmitted by the data transmission unit 35 may include, together with the infrastructure image SIM_3, identification data of the infrastructure camera 3, position data of the infrastructure camera 3 (e.g., latitude and longitude data), and data related to the infrastructure image SIM_3. An example of the data related to the infrastructure image SIM_3 is data on the time when the infrastructure image SIM_3 was acquired.

[0144] 2-2-2. Example of functional configuration of data processing device 22 In the example shown in Fig. 21, the data processing device 22 includes a data receiving unit 24. The explanation so far is the same as that of Fig. 11. In the example shown in Fig. 21, the data processing device 22 further includes display control units 25a and 25b, a left / right definition unit 27, and a recognition assistance superimposition unit 28. The functions of each block shown in the data processing device 22 are realized by the processor 22a executing various programs.

[0145] The data receiving unit 24 receives various data from the moving object 1. The various data includes a moving object image SIM_1. The various data may include identification data of the moving object 1, position data (GPS data) of the moving object 1, and data related to the moving object image SIM_1. The position data is, for example, latitude and longitude data acquired by a GPS (Global Positioning System) receiver mounted on the moving object 1. Examples of data related to the moving object image SIM_1 have already been described. The data receiving unit 24 decodes the various data received from the moving object 1 and transmits it to the display control unit 25a. The function of the display control unit 25a is the same as the function of the display control unit 25 described in FIG. 11.

[0146] The data receiving unit 24 also receives various data from the infrastructure camera 3. The various data includes the infrastructure image SIM_3. The various data may include identification data of the infrastructure camera 3, position data of the infrastructure camera 3, and data related to the infrastructure image SIM_3. Examples of data related to the infrastructure image SIM_3 have already been described. The data receiving unit 24 decodes the various data received from the infrastructure camera 3 and transmits it to the left / right definition unit 27 and the recognition assistance superimposition unit 28.

[0147] The left / right definition unit 27 defines the left and right in the infrastructure image SIM_3. First, the left / right definition unit 27 sets a boundary. FIG. 22 is a diagram for explaining an example of processing by the left / right definition unit 27. Note that FIG. 22 explains an example of processing for the infrastructure image SIM_3a explained with reference to FIG. 18. In the example shown in FIG. 22, first, a virtual horizontal plane VHS passing through a reference point of the moving object 1 (for example, the current position of the moving object 1) is set based on the position data of the moving object 1 (step S1).

[0148] Once the virtual horizontal plane VHS is set, a reference line RL extending in the forward / backward direction of the moving object 1 is set on the virtual horizontal plane VHS (step S1). The forward / backward direction is estimated, for example, based on a history of position data of the moving object 1 during a period going back a predetermined time from the current time. In the example shown in Fig. 22, the reference line RL is set by projecting a straight line extending in this estimated direction onto the virtual horizontal plane VHS.

[0149] Once the reference line RL is set, a plane including the reference line RL and perpendicular to the virtual horizontal plane VHS (i.e., a boundary plane BS) is set (step S2). Once the boundary plane BS is set, a right space RS and a left space LS can be defined in the virtual space VSP including the boundary plane BS and the virtual horizontal plane VHS, with the front of the moving object 1 as the reference.

[0150] The left / right definition unit 27 applies the virtual space VSP including the boundary surface BS and the virtual horizontal plane VHS to the infrastructure image SIM_3 (step S2). Then, the boundary BD2 depicted by the boundary surface BS can be set on the infrastructure image SIM_3. When the boundary BD2 is set, the right area RA and the left area LA based on the front of the moving object 1 can be set on the infrastructure image SIM_3.

[0151] The left / right definition unit 27 transmits data of the right area RA and the left area LA in the infrastructure image SIM_3 to the recognition assistance superimposition unit 28. The left / right definition unit 27 transmits, for example, data of pixel coordinates constituting the boundary BD2 in the infrastructure image SIM_3 to the recognition assistance superimposition unit 28. When an arc-shaped recognition assistance is superimposed on the infrastructure image SIM_3 (that is, in the case of the first example described with reference to FIG. 19), the left / right definition unit 27 transmits data of pixel coordinates constituting the virtual horizontal plane VHS in the infrastructure image SIM_3 to the recognition assistance superimposition unit 28.

[0152] The recognition assistance superimposing unit 28 superimposes recognition assistance on the infrastructure image SIM_3 based on the infrastructure image SIM_3 received from the data receiving unit 24 and various data related to the infrastructure image SIM_3 received from the left / right defining unit 27.

[0153] In the case of the first example described with reference to Fig. 19, the recognition assistance superimposing unit 28 uses pixel coordinate data constituting a virtual horizontal plane VHS in the infrastructure image SIM_3, and draws virtual concentric circles centered on the reference point of the moving object 1 on this virtual horizontal plane VHS. Next, the recognition assistance superimposing unit 28 applies these virtual concentric circles to the infrastructure image SIM_3, and superimposes recognition assistance elements AS_R14 to AS_R17 on the right region RA, and superimposes recognition assistance elements AS_L14 to AS_L17 on the left region LA.

[0154] The second to sixth examples of the recognition assistance described in the first embodiment with reference to Figures 5 to 9 can also be applied to the second embodiment. These examples can be explained by appropriately replacing "moving object image SIM_1" in the description of "recognition assistance superimposition unit 16" in Figure 11 with "infrastructure image SIM_3".

[0155] Returning to Fig. 21, the description of the functions of the data processing device 22 will be continued. The recognition assistance superimposing unit 28 transmits the infrastructure image SIM_3 on which the recognition assistance is superimposed to the display control unit 25b. The display control unit 25b performs various controls for outputting the various data received from the data receiving unit 24 to the display 26. The various controls include output control of the infrastructure image SIM_3. In the output control of the infrastructure image SIM_3, for example, processing is performed for displaying the infrastructure image SIM_3a on the display 26a and displaying the infrastructure image SIM_3b on the display 26b.

[0156] However, when the infrastructure image SIM_3 is output from the display 26, there is a problem that it is difficult for the operator OP to instantly recognize the front and rear of the moving object 1 included in the infrastructure image SIM_3. Therefore, in this case, it is preferable to modify the first example described with reference to Fig. 19 and subdivide the right area RA and the left area LA in the moving object image SIM_1 and the infrastructure image SIM_3 in the front-rear direction.

[0157] Fig. 23 is a diagram for explaining an example of subdivision processing of the right area RA and the left area LA in the infrastructure image SIM_3. This subdivision processing can be performed by the left / right definition unit 27 described with reference to Fig. 21. Fig. 23 depicts a front image SIM_11a and infrastructure images SIM_3a and SIM_3b, as in Fig. 19. The front image SIM_11a is the same as that shown in Fig. 19.

[0158] In the example shown in FIG. 23, a boundary BD3 is added to the infrastructure images SIM_3a and SIM_3b. The boundary BD3 can be set by a method similar to that of setting the boundary BD2. Specifically, first, a line (hereinafter also referred to as an "additional reference line") perpendicular to the reference line RL described with reference to FIG. 22 is set on the virtual horizontal plane VHS. Next, a plane (hereinafter also referred to as an "additional boundary surface") that includes this additional reference line and perpendicular to the virtual horizontal plane VHS is set. Then, a virtual space including this additional boundary surface and the virtual horizontal plane VHS is applied to the infrastructure image SIM_3. Then, a boundary (additional boundary) BD3 drawn by the additional boundary surface can be set on the infrastructure image SIM_3.

[0159] When the boundary BD3 is set, the right area RA can be divided into a right front area RFA and a right rear area RRA, and the left area LA can be divided into a left front area LFA and a left rear area LRA. By using the data of these areas, information on the front and rear directions can be added to the infrastructure images SIM_3a and SIM_3b. This process can be performed by the recognition assistance superimposition unit 28 described with reference to FIG. 21.

[0160] 23 shows an infrastructure image SIM_3a and an infrastructure image SIM_3b in which recognition aids are superimposed on four types of regions. Specifically, recognition aids AS_R18 and AS_R19 are superimposed on the right front region RFA of the infrastructure image SIM_3a, recognition aids AS_L18 and AS_L19 are superimposed on the left front region LFA, recognition aids AS_R20 are superimposed on the right rear region RRA, and recognition aids AS_L20 are superimposed on the left rear region LRA of the infrastructure image SIM_3b. Recognition aids AS_R21 and AS_R22 are superimposed on the right rear region RRA, recognition aids AS_L21 and AS_L22 are superimposed on the left rear region LRA, recognition aids AS_R23 are superimposed on the right front region RFA, and recognition aids AS_L23 are superimposed on the left front region LFA.

[0161] The recognition assistants shown in Fig. 23 are colored according to the areas where these recognition assistants are superimposed. That is, in the example shown in Fig. 23, the recognition assistants AS_R18, AS_R19, and AS_R23 superimposed on the right front area RFA of the infrastructure image SIM_3 are colored in a color different from the recognition assistants AS_R20, AS_R21, and AS_R22 superimposed on the right rear area RRA of the infrastructure image SIM_3. However, the recognition assistants AS_R18 to AS_R23 are colored in the same color system as the recognition assistants AS_R1 to AS_R3 of the forward image SIM_11. That is, the recognition assistants AS_R18, AS_R19, and AS_R23 are colored in the same color system as the recognition assistants AS_R20, AS_R21, and AS_R22, but different colors.

[0162] Further, the recognition assistants AS_L18, AS_L19, and AS_L23 superimposed on the left front area LFA of the infrastructure image SIM_3 are colored in a color similar to, but different from, the color of the recognition assistants AS_L20, AS_L21, and AS_L22 superimposed on the left rear area LRA of the infrastructure image SIM_3. Note that it is easily understood from the gist of this disclosure that the colors of the recognition assistants AS_L18 to AS_L23 are not similar to the colors of the recognition assistants AS_R18 to AS_R23.

[0163] Another example of assisting in immediate recognition of the area ahead and behind the moving body 1 included in the infrastructure image SIM_3 is to provide a separate display that outputs a map image of the area around the current location of the moving body 1. Since the current location of the moving body 1 can be ascertained from the location data, a map image of the area around the current location is acquired from a map database or the like, and an icon imitating the infrastructure camera 3, an icon imitating the moving body 1, and a pictogram colored in the same color as the recognition aid output from the display 26 are superimposed on this map image and output from the display.

[0164] Fig. 24 is a diagram for explaining a processing example of superimposing various icons on a map image. In the example shown in Fig. 24, a map image MIM on which an icon ICN3 imitating an infrastructure camera 3 and an icon ICN1 imitating a moving object 1 are superimposed is output from the display 29. In addition, a pictogram PG_R colored in the same color as the recognition assistance AS_R14 to AS_R17 output from the display 26 and a pictogram PG_L colored in the same color as the recognition assistance AS_L14 to AS_L17 are superimposed on the map image MIM.

[0165] 3.Effects According to the second embodiment described above, an infrastructure image SIM_3 is generated based on data obtained from an infrastructure camera 3 whose distance from the current position of the moving object 1 is equal to or less than a predetermined value. Then, a recognition aid colored in different colors on the left and right sides based on the front of the moving object 1 is superimposed on this infrastructure image SIM_3, and this is output from the display 26. Therefore, it is expected to help the operator OP make accurate decisions when remotely operating the moving object 1, and to contribute to safe movement of the moving object 1 during remote operation.

[0166] 23, a recognition aid in which the right front, left front, right rear, and left rear are colored in different colors with respect to the front of the moving object 1 as a reference is superimposed on the infrastructure image SIM_3, and this is output from the display 26. This allows the operator OP to instantly recognize the front and rear of the moving object 1. This is also expected to help the operator OP make accurate decisions when remotely operating the moving object 1, and to contribute to the safe movement of the moving object 1 by remote operation.

[0167] 24, the transformation using the map image allows the operator OP to instantly recognize the front and rear of the moving object 1. In addition, it is possible to instantly grasp the relative positions of the infrastructure camera 3 and the moving object 1 and the traveling direction of the moving object 1. This is also expected to help the operator OP make accurate decisions when remotely operating the moving object 1, and to contribute to the safe movement of the moving object 1 by remote operation.

[0168] The modification using the map image MIM described in Fig. 24 may be applied to the first embodiment. In this case, the map image MIM shown in Fig. 24 with the icon ICN3 removed is output from the display 29. In addition, the pictograms PG_R and PG_L are colored in the same color as the recognition assistance color output from the display 21.

[0169] Third embodiment Finally, a third embodiment of the present disclosure will be described below, in which explanations that overlap with those of the first and second embodiments will be omitted as appropriate.

[0170] In the second embodiment, the moving object image SIM_1 and the infrastructure image SIM_3 are combined and output from the displays 21 and 26. In the third embodiment, only the infrastructure image SIM_3 is output from the display 26. That is, in the third embodiment, the moving object image SIM_1 is not generated, or the moving object image SIM_1 is not output from the display 21. When the infrastructure image SIM_3 is output from the display 26, the display 21 may be omitted. The infrastructure image SIM_3 may be output from the display 21, in which case the display 26 may be omitted.

[0171] It is assumed that it is desirable not to output the moving body image SIM_1 from the display 21 before receiving a remote operation request from the moving body 1. For example, when the operator OP waits for remote operation requests from multiple moving bodies 1, it is desirable to output multiple infrastructure images SIM_3 generated based on data acquired from the infrastructure camera 3 from multiple displays 26, respectively.

[0172] In this case, when a remote operation request is received from one of the multiple moving bodies 1, the output of infrastructure images SIM_3 other than the infrastructure image SIM_3 corresponding to the infrastructure camera 3 whose distance from the moving body 1 is equal to or less than a predetermined value is terminated. Then, the moving body image SIM_1 of the moving body 1 performing the remote operation is output from the display 21. [Explanation of symbols]

[0173] 1. Mobile 2 Remote operation device (terminal) 3, 3a, 3b Infrastructure Camera 11, 11a, 11b Sensor 12, 22, 32 Data processing device 12a, 22a, 32a processors 12b, 22b, 32b memory 21, 21a, 21b, 26, 26a, 26b, 29 Display 31, 31a, 31b Camera body Recognition assistance for AS_L1~AS_L23, AS_L11a~AS_L11d, AS_R1~AS_R23, AS_R11a, AS_R11c, AS_R11d, AS_B1~AS_B6 BD1, BD2 boundaries BS boundary surface ICN1, ICN3 icons LA left region LS left space LFA left front region LRA left rear region MIM map image OB1, OB2, OB3 dynamic object markers OP operator PG_L, PG_R pictograms RA right region RL reference RS right space RFA right front region RRA right rear region VHS virtual horizontal plane VSP virtual space SIM sensor image SIM_1 moving object image SIM_11a front image SIM_11b rear image SIM_3, SIM_3a, SIM_3b infrastructure images

Claims

1. A remote image display method for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal that communicates with the remote sensor, comprising: The remote sensor includes a sensor mounted on a moving object; the sensor image is a moving body image showing an image generated based on data acquired from a sensor mounted on the moving body, and includes a moving body image including a front image and a rear image of the moving body; setting a reference line extending in a front-rear direction of the moving body on a virtual horizontal plane passing through a reference point of the moving body; using the reference line to define right and left spaces in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; setting right and left areas based on a front of the moving object as a reference in the front and rear images, respectively, using the virtual space in which the right and left spaces are defined; superimposing right and left recognition aids on the right and left regions of the front and rear images, respectively, the right recognition aid being a different color than the left recognition aid; outputting the front and rear images on the display with the right and left recognition aids respectively superimposed thereon; 13. A method for remote image display comprising:

2. The right and left recognition assistance includes coloring applied to arcs drawn concentrically on the virtual horizontal plane with the reference point of the moving body as a center, coloring applied to at least a portion of each of the right and left regions, coloring applied to static targets recognized in the moving body image, coloring applied to bounding boxes surrounding dynamic targets recognized in the moving body image, coloring applied to the trajectory of the dynamic targets, and coloring applied to the trajectory of the moving body.

2. The method of claim 1, wherein the image is displayed remotely.

3. The right and left recognition assistance is a combination of at least one selected from a color applied to the arc, a color applied to at least a part of each of the right and left regions, a color applied to the static target, and a color applied to the trajectory of the moving object, and at least one selected from a color applied to the bounding box and a color applied to the trajectory of the dynamic target.

3. The method of claim 2, wherein the image is displayed remotely.

4. determining whether a dynamic target straddling a boundary separating the right region and the left region is recognized in the moving object image; when it is determined that the boundary-spanning dynamic object has been recognized in the moving object image, doubling the bounding box surrounding the boundary-spanning dynamic object and applying the right and left recognition assistance colors to these bounding boxes, respectively; 4. The method of claim 2, further comprising:

5. determining whether a dynamic target straddling a boundary separating the right region and the left region is recognized in the moving object image; When it is determined that the boundary-spanning dynamic object has been recognized in the moving object image, applying a color different from both the right and left recognition assistance colors to a bounding box surrounding the boundary-spanning dynamic object; 4. The method of claim 2, further comprising:

6. determining whether the total number of the dynamic targets is equal to or greater than an upper limit; selecting a dynamic object to which the bounding box is assigned according to a predetermined criterion when it is determined that the total number is equal to or greater than the upper limit; 6. The method for remote image display according to claim 2, further comprising:

7. superimposing the right and left recognition aids onto the front and rear images, respectively; setting a color of the right recognition assistance superimposed on the front image and a color of the right recognition assistance superimposed on the rear image to the same color; setting a color of the left recognition aid superimposed on the front image and a color of the left recognition aid superimposed on the rear image to the same color; 7. The method for remote image display according to claim 1, further comprising:

8. determining whether the number of dynamic targets recognized in each of the right and left regions of the forward image is equal to or greater than one; determining whether a dynamic target is recognized in at least one of the right and left regions of the rear image; adopting a color applied to a bounding box surrounding the dynamic object recognized in the front and rear images as a recognition aid for the right and left when the number of dynamic objects recognized in each of the right and left regions of the front image is equal to or greater than one and it is determined that a dynamic object has been recognized in at least one of the right and left regions of the rear image; 8. The method of claim 7, further comprising:

9. the boundary separating the right region and the left region includes a boundary having a predetermined shape; A method for remote image display according to any one of claims 1 to 8, further comprising a step of superimposing a recognition aid on the boundary having the predetermined shape, the color of the recognition aid superimposed on the boundary having the predetermined shape being different from the color of either the right or left recognition aid.

10. the sensor image includes an infrastructure image showing an image generated based on data acquired from an infrastructure camera whose distance from the moving object is equal to or less than a predetermined value; The terminal further includes a display that displays the infrastructure image; setting right and left areas based on a front of the moving object in the infrastructure image using the virtual space in which the right and left spaces are defined; a step of superimposing right and left recognition aids on the right and left regions of the infrastructure image, respectively, wherein the color of the right recognition aid superimposed on the infrastructure image is the same color as that of the right recognition aid superimposed on the moving body image, and the color of the left recognition aid superimposed on the infrastructure image is the same color as that of the left recognition aid superimposed on the moving body image; outputting the infrastructure image on which the right and left recognition aids are superimposed from a display that displays the infrastructure image; 10. The method for remote image display according to claim 1, further comprising:

11. setting an additional reference line on the virtual horizontal plane that is perpendicular to the reference line; dividing the right and left spaces into two in a front-rear direction of the moving body using the additional reference line to define right front, left front, right rear, and left rear spaces; Further comprising: When the right front, left front, right rear and left rear spaces are defined, In the step of setting right and left regions based on a front of the moving body in the infrastructure image, right front, left front, right rear, and left rear regions based on a front of the moving body are set in the infrastructure image, In the step of superimposing right and left recognition aids on the right and left regions of the infrastructure image, right front, left front, right rear, and left rear recognition aids are superimposed on the right front, left front, right rear, and left rear regions, respectively; The remote image display method described in claim 10, characterized in that the color of the right front recognition aid superimposed on the infrastructure image is different from the color of the right rear recognition aid, and the color of the left front recognition aid superimposed on the infrastructure image is different from the color of the left rear recognition aid.

12. The terminal further includes a display for displaying a map image of the surroundings of the mobile object; a step of superimposing on the map image an icon of the moving body and pictograms surrounding the icon of the moving body and right and left pictograms based on a front of the moving body, the right pictogram being a color of the same color family as the right recognition aid superimposed on the moving body image, and the left pictogram being a color of the same color family as the left recognition aid superimposed on the moving body image; outputting the map image on which the icon of the moving object and the right and left pictograms are superimposed from a display that displays the map image; The method for remote image display according to any one of claims 1 to 11, further comprising:

13. A remote image display method for displaying a sensor image generated based on data acquired from a remote sensor on a display of a terminal that communicates with the remote sensor, comprising: The sensor images include an infrastructure image showing an image generated based on data acquired from an infrastructure camera that is at a distance from the moving body that is equal to or less than a predetermined value, and a moving body image showing an image generated based on data acquired from a sensor mounted on the moving body, the moving body image including a forward image of the moving body, setting a reference line extending in a front-rear direction of the moving body on a virtual horizontal plane passing through a reference point of the moving body; using the reference line to define right and left spaces in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; setting right and left areas based on a front of the moving object as a reference for each of the infrastructure image and the moving object image, using the virtual space in which the right and left spaces are defined; a step of superimposing right and left recognition aids on the right and left regions of the infrastructure image and the moving object image, respectively, the right recognition aid being a different color from the left recognition aid; outputting, from the display, the infrastructure image and the moving object image on which the right and left recognition assistance are respectively superimposed; 13. A method for remote image display comprising:

14. A remote image display system that displays a sensor image generated based on data acquired from a remote sensor on a display of a terminal that communicates with the remote sensor, The remote sensor includes a sensor mounted on a moving object; the sensor image is a moving body image showing an image generated based on data acquired from a sensor mounted on the moving body, and includes a moving body image including a front image and a rear image of the moving body; The moving body, A reference line is set on a virtual horizontal plane passing through a reference point of the moving body, the reference line extending in a front-rear direction of the moving body; Using the reference line, right and left spaces are defined in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; Using the virtual space in which the right and left spaces are defined, right and left areas based on a front of the moving object are set in the front and rear images, respectively; superimposing right and left recognition aids on the right and left regions of the front and rear images, respectively; The color of the right recognition aid is different from that of the left recognition aid, Transmitting the moving object image onto which the right and left recognition assistance are respectively superimposed to the terminal; The terminal outputs the front and rear images on which the right and left recognition aids are superimposed from the display. A remote image display system comprising:

15. A remote image display system that displays a sensor image generated based on data acquired from a remote sensor on a display of a terminal that communicates with the remote sensor, The sensor images include an infrastructure image showing an image generated based on data acquired from an infrastructure camera that is at a distance from the moving body that is equal to or less than a predetermined value, and a moving body image showing an image generated based on data acquired from a sensor mounted on the moving body, the moving body image including a forward image of the moving body, The infrastructure camera transmits the infrastructure image to the terminal; The terminal, A reference line is set on a virtual horizontal plane passing through a reference point of the moving body, the reference line extending in a front-rear direction of the moving body; Using the reference line, right and left spaces are defined in a virtual space including the virtual horizontal plane, with a front of the moving object as a reference; using the virtual space in which the right and left spaces are defined, right and left areas based on a front of the moving body are set for each of the infrastructure image and the moving body image; Superimposing right and left recognition aids on the right and left regions of the infrastructure image and the moving object image, respectively; The color of the right recognition aid is different from that of the left recognition aid, The infrastructure image and the moving object image on which the right and left recognition assistance are respectively superimposed are output from the display. A remote image display system comprising:

Images