Image display device and image display method

Abstract

To provide a technique capable of making an objective viewpoint position appropriate.SOLUTION: An image display device calculates a region which can photograph a photographing image containing an image of an operation position as a photographing available region, extracts a position contained in the photographing available region as a viewpoint position from positions in a movable path where a mobile body moved in the past on the basis of operation information and the photographing available region, and generates an overlapping image obtained by overlapping a corresponded position image indicating the operation position to a photographing image photographed at the viewpoint position on the basis of the photographing image, the operation position, and the viewpoint position.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image display device and an image display method.

Background Art

[0002] An operator who is not used to remotely operating a moving object to be operated has difficulty remotely operating the moving object only with a captured image that is captured in real time from a camera mounted on the moving object because the operator cannot grasp the vehicle feeling of the moving object. Therefore, a technique has been proposed to improve the operability of the moving object by superimposing an image of the moving object on a captured image that can objectively view the current moving object among past captured images captured by the camera (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when an operator remotely operates a moving object indoors where there are many obstacles, an image viewed from a viewpoint position that is separated from the rear of the moving object by a certain distance may not be preferable for the operator.

[0005] For example, consider a case where an operator remotely operates a moving object to move back after it turns right and moves to the end of an indoor passage while viewing an image seen from a viewpoint position at a certain distance behind the moving object. In this case, immediately after the moving object turns right, there is a problem that the moving object cannot be displayed due to obstacles such as walls or shelves constituting the passage. Also, when the moving object returns, if an attempt is made to secure a certain distance, the viewpoint position is greatly changed, and thus there is a problem that the operator cannot easily grasp the situation of the moving object. As described above, in order to perform remote operation indoors with many obstacles, it was necessary to optimize the objective viewpoint position serving as a reference for the display of the moving object and its surroundings.

[0006] In particular, not only for the work by the operator's remote operation, but also for a moving object that autonomously performs work, an operation is assumed in which the operator remotely operates the moving object only when opening a situation where the moving object cannot autonomously perform work. In such an operation, since the operator does not monitor the moving object until starting the remote operation, in order to appropriately grasp the situation of the moving object at the start of the remote operation, an image viewed from an optimized objective viewpoint position was necessary.

[0007] Therefore, the present disclosure has been made in view of the above problems, and an object thereof is to provide a technology capable of optimizing an objective viewpoint position.

Means for Solving the Problem

[0008] The image display device according to the present disclosure includes an acquisition unit that acquires a request for remote operation of a mobile body, work information indicating work including movement of the mobile body, and a captured image captured from the mobile body, a work position calculation unit that calculates a work position of the mobile body when the request is acquired based on the request and the work information, work environment information including map information indicating a work space of the mobile body and obstacle information regarding obstacles in the work space, a photographable area calculation unit that calculates, based on the work position and the work environment information, an area in which the captured image including an image of the work position can be captured as a photographable area, a first viewpoint position extraction unit that extracts, based on the work information and the photographable area, a position included in the photographable area among positions in a movement path along which the mobile body has moved in the past as a viewpoint position, a superimposed image generation unit that generates a superimposed image in which a corresponding position image indicating the work position is superimposed on the captured image captured at the viewpoint position based on the captured image, the work position, and the viewpoint position, and a display unit that displays the superimposed image.

Effect of the Invention

[0009] According to the present disclosure, an area in which a captured image including an image of a work position can be captured is calculated as a photographable area, and based on the work information and the photographable area, a position included in the photographable area among positions in a movement path along which the mobile body has moved in the past is extracted as a viewpoint position, and based on the captured image, the work position, and the viewpoint position, a superimposed image in which a corresponding position image indicating the work position is superimposed on the captured image captured at the viewpoint position is generated. With such a configuration, an objective viewpoint position can be optimized.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0011] <Embodiment 1> Conventionally, in environments such as factories, an environment that is easy for mobile robots to work in has been prepared, and through the actions of employees following the rules, an operation in which the mobile robots are fully automated has been realized. Examples of an environment that is easy for mobile robots to work in include, for example, the floor color and illuminance being constant, the passageways being wide with few blind spots, and no obstacles being installed in the passageways. Examples of the rules that employees follow include, for example, avoiding mobile robots and not occupying the movement paths of mobile robots.

[0012] On the one hand, due to the performance improvement of devices such as batteries and sensors that make up mobile robots, the popularization of safety sensors, and the improvement of situation judgment ability due to the progress of AI (artificial intelligence) technology, the autonomous work execution ability of mobile robots has been increasing. Along with this, the demand for autonomous mobile robots that work autonomously not only in factories but also in human workspaces has been increasing, and as a result, the operation of autonomous mobile robots that share the workspace with humans has been increasing.

[0013] However, it is difficult for autonomous mobile robots to achieve fully automated operation in a workspace designed for humans. For example, in commercial facilities visited by customers or general offices, decorations that give people a good impression are provided, and people who share the environment with autonomous mobile robots often do not give special consideration to the autonomous mobile robots. Therefore, it is difficult to anticipate all the situations that autonomous mobile robots may encounter and corresponding countermeasures until they can operate completely autonomously in a human workspace.

[0014] By the way, in order to prioritize the safety of people sharing the workspace, when an autonomous mobile robot cannot confirm safety due to insufficient information from sensors or determines that there are people in a place where there is a possibility of contact, the driving of the autonomous mobile robot is often stopped. In addition, people can flexibly grasp the situation from image information and the like more than autonomous mobile robots even in unexpected situations, so they can appropriately grasp the risk of contact accidents. In addition, people can also draw attention to the surroundings and predict the movements of people nearby from the video.

[0015] In consideration of these, when an autonomous mobile robot gets into a situation where it cannot perform tasks autonomously, an operation has been proposed in which an operator returns the autonomous mobile robot to work through remote operation. There are many situations in which an autonomous mobile robot can be returned to work through remote operation by an operator, and the situations and countermeasures that should be anticipated in advance can be reduced. Therefore, with such an operation, the applicable range of autonomous mobile robots can be expanded even within the working space of people.

[0016] In such an operation, since the operator has few opportunities to remotely operate an autonomous mobile robot, it is assumed that the operator remotely operates multiple autonomous mobile robots as multiple operation targets by one person. Since the working environment is diverse, it is important for the operator to be able to easily grasp the situations of multiple autonomous mobile robots in order to appropriately perform remote operations such as recovery work on the autonomous mobile robots.

[0017] In the case of recovery work on an autonomous driving vehicle traveling on a public road rather than indoor recovery work, since the objects that need to be paid attention to can be limited to the road, the operator can easily grasp the situation, shape, and movement characteristics of the autonomous driving vehicle. For this reason, even with information from a first-person perspective obtained from in-vehicle sensors such as LiDER (Light Detection and Ranging) or an omnidirectional camera, the operator can easily grasp the situation of the autonomous driving vehicle.

[0018] However, in an indoor environment, since the objects that need to be paid attention to, as well as the shape and movement characteristics of the operation target, are diverse, it is difficult for the operator to grasp the situation, shape, and movement characteristics of the operation target from the first-person perspective video captured by the camera mounted on the operation target. Therefore, in order for the operator to easily grasp the situation of the operation target, etc., a video including the operation target and its surroundings viewed from an objective viewpoint position is more preferable than a video viewed from the viewpoint of the operation target itself such as a first-person perspective.

[0019] Images from such an objective viewpoint position can be obtained using a fixed camera installed on the ceiling of the work space or the like. However, in a place with many desks and shelves, such as an office environment, the desks and shelves become blind spots, and there are areas where the floor conditions cannot be grasped with such a fixed camera. Also, for example, in commercial facilities, many surveillance cameras are installed, but in office environments, surveillance cameras are installed only at entrances and exits, or there are few indoor surveillance cameras.

[0020] Therefore, by processing the images from the camera mounted on the autonomous mobile robot and generating an image viewed from an objective viewpoint position from the images of the camera, it is possible to reduce the blind spots in the work space. Also, by using the images from the camera mounted on the autonomous mobile robot that requires remote operation as the images to be processed, it is possible to increase the possibility of capturing images at locations where the autonomous mobile robot can no longer perform its work autonomously.

[0021] In the following-described Embodiment 1, it is configured to be able to generate and display an image showing the position of a remotely operable mobile body viewed from an objective viewpoint position from the images taken from the mobile body. Hereinafter, the mobile body capable of image capturing and remote operation will be described as an autonomous mobile robot, but it is not limited to this, and for example, other mobile bodies that operate by remote control, or ride-on type mobile bodies, etc. may also be used. Even with these mobile bodies, similar to the autonomous mobile robot described below, when there is an obstacle to the movement of the mobile body, the operator can perform remote operation while viewing an image from an objective viewpoint position.

[0022] FIG. 1 is a block diagram showing the configuration of an image display device according to Embodiment 1. The work information of the autonomous mobile robot and the captured images taken by the camera are input to the image display device of FIG. 1.

[0023] The work information is information indicating work including the movement of the autonomous mobile robot, and indicates the work that the autonomous mobile robot has performed and the work that the autonomous mobile robot plans to perform in the future. The work information may include, for example, movement route information indicating the positions where the autonomous mobile robot has moved in chronological order, information detected in chronological order by sensors mounted on the autonomous mobile robot, log information of drive information of a drive unit that drives the autonomous mobile robot, and the like. Further, the work information may be information indicating not only the work of the autonomous mobile robot but also the work of peripheral devices such as surveillance cameras and automatic doors in the work space of the autonomous mobile robot.

[0024] The captured image is an image captured by the autonomous mobile robot and is an image captured by a camera mounted on the autonomous mobile robot. Hereinafter, a camera mounted on a moving body such as an autonomous mobile robot may also be referred to as a "mounted camera". In the first embodiment, it is assumed that the mounted camera is an omnidirectional camera capable of photographing all directions of the autonomous mobile robot. The captured image is, for example, an image captured in chronological order, and may be an image in video format or an image in still image format. Further, the captured image may include distance information indicating the distance between the captured object and the mounted camera. Hereinafter, for the sake of simplicity of explanation, it will be described that the captured image is a still image format image captured by the mounted camera and does not include distance information. Note that the captured image may be a still image format image extractable from a video format image.

[0025] In the example of FIG. 1, the work environment information is information that changes little during the execution of work by the autonomous mobile robot, and may be input to the image display device in the same manner as the work information and the captured image, or may be stored in the image display device in advance. The work environment information includes map information indicating the work space of the autonomous mobile robot and obstacle information regarding obstacles in the work space. The work environment information may include mobile body information such as the shape and type of the autonomous mobile robot, and the installation positions and shapes of desks, shelves, surveillance cameras, and automatic doors in the work space. The map information may include, for example, information on a three-dimensional space measured in advance and information on a three-dimensional space calculated from the time-series information of a distance sensor mounted on the autonomous mobile robot.

[0026] Although not shown in FIG. 1, information for remotely operating the autonomous mobile robot is input to the autonomous mobile robot. The autonomous mobile robot is remotely operated based on the information.

[0027] The image display device in FIG. 1 includes an acquisition unit 1, a work position calculation unit 2, a photographable area calculation unit 3, a self-viewpoint position extraction unit 4 which is a first viewpoint position extraction unit, a superimposed image generation unit 5, and a display unit 6. Hereinafter, the components of the image display device will be described in detail.

[0028] <Acquisition unit 1> The acquisition unit 1 acquires a remote operation request which is a request for remotely operating the autonomous mobile robot, work information, and a captured image. The remote operation request is a request for remote operation to the operator to resume autonomous work when it is determined that it is difficult for the autonomous mobile robot to perform work autonomously. When it is determined that it is difficult for a management system that manages a plurality of autonomous mobile robots as a whole or the autonomous mobile robot to perform autonomous work, etc., the remote operation request requests the operator to remotely operate the autonomous mobile robot by the remote operation request, and outputs the remote operation request to the acquisition unit 1.

[0029] The remote operation request includes information specifying the autonomous mobile robot for which remote operation is required, and may also include other additional information. The additional information includes, for example, the request time, work priority, reason for request, remote operation plan, and the like. The work priority is set to "high" when the risk of the autonomous mobile robot causing harm to people is high, such as in the case of a collision with a person or an accident in a situation where a person is nearby. It is set to "medium" when the risk is low and the work is urgent, and set to "low" when the risk is low and the work is not urgent. The reason for request indicates, for example, an error code and the reason for requesting remote operation, and the remote operation plan shows a countermeasure for the reason for request.

[0030] The acquisition unit 1 not only acquires the remote operation request but also acquires information about the autonomous mobile robot that requested the remote operation. The information about the autonomous mobile robot includes the work information of the autonomous mobile robot and the captured images related thereto. When the work environment information is not stored in advance, the acquisition unit 1 acquires the work environment information. After acquiring the remote operation request and the like, the process proceeds to the work position calculation unit 2.

[0031] <Work position calculation unit 2> The work position calculation unit 2 calculates, based on the remote operation request and the work information, the position where the autonomous mobile robot was working when the remote operation request was acquired as the work position.

[0032] In the first embodiment, first, the work position calculation unit 2 determines from the work information acquired by the acquisition unit 1 whether the actual work being performed by the autonomous mobile robot when the remote operation request was acquired is a specific work performed around a specific position or a movement between specific positions. The specific work is a general work that is not for the purpose of movement, such as loading / unloading or transfer work at an article delivery point.

[0033] When it is determined that the implementation work is a specific work, the work position calculation unit 2 determines the point where a series of operations in the specific work are being performed as the work position. When it is determined that the implementation work is a movement between specific positions, the work position calculation unit 2 determines the movement range within a predetermined period up to the time point of the remote operation request, or the movement range within a predetermined distance up to the stop position after the time point of the remote operation request as the work position. The predetermined period is, for example, a specified time such as 10 seconds up to the request time point, or a specified time such as 5 seconds up to the request time point that does not include the time after stopping. The fixed distance is, for example, a specified distance such as 5 m up to the stop position after the request time point, or a distance proportional to the size of the autonomous mobile robot. After calculating the work position, the process proceeds to the photographable area calculation unit 3.

[0034] <Photographable area calculation unit 3> The photographable area calculation unit 3 calculates, based on the work environment information and the work position calculated by the work position calculation unit 2, the area where a photographed image including an image of the work position can be photographed as the photographable area.

[0035] FIG. 2 is a bird's-eye view for explaining an example of a method for calculating the photographable area. FIG. 2 shows a scene where a remote operation is requested when the autonomous mobile robot is performing a transfer operation on a shelf. FIG. 2 shows the autonomous mobile robot 21 that requested the remote operation, indicated by a black circle, the work position 22 indicated by hatching, an obstacle 23 such as a shelf, and the photographable area 24 surrounded by a dashed line.

[0036] In the first embodiment, first, the shootable area calculation unit 3 combines the work position 22 calculated by the work position calculation unit 2 and obstacles 23 such as shelves indicated by the obstacle information included in the work environment information on the map of the map information included in the work environment information. Then, the shootable area calculation unit 3 calculates, as the shootable area 24, an area where a shootable image including an image of the work position 22 can be taken without being blocked by the obstacles 23. For example, the shootable area calculation unit 3 calculates, as the shootable area 24, an area obtained by collecting points where straight lines can be drawn to each area of the work position 22 without hitting the obstacles 23.

[0037] The method for calculating the shootable area 24 may be a geometric calculation method in which the work position 22 and the obstacles 23 are projected onto the map of the map information as polyhedrons and the vertices thereof are connected, or a search calculation method for determining whether appropriately extracted representative points are points in the shootable area 24. Note that the method for calculating the shootable area 24 is not limited to these. Also, in the example of FIG. 2, the shootable area calculation unit 3 calculates the shootable area 24 in a plane such as a bird's-eye view, but may also calculate the shootable area 24 in a three-dimensional space. After the shootable area 24 is calculated, the process proceeds to the own viewpoint position extraction unit 4.

[0038] <Own viewpoint position extraction unit 4> The own viewpoint position extraction unit 4 extracts, as the viewpoint position, a position included in the shootable area 24 among the positions in the movement path where the autonomous mobile robot 21 has moved in the past, based on the work information and the shootable area 24 calculated by the shootable area calculation unit 3. The own viewpoint position extraction unit 4 uses, for example, the above-described movement path information included in the work information when extracting the viewpoint position. The viewpoint position is a position that becomes the viewpoint of the image displayed on the display unit 6. Note that the own viewpoint position extraction unit 4 may extract the viewpoint position in consideration of the optical system information of the camera mounted on the autonomous mobile robot 21.

[0039] FIG. 3 is a bird's-eye view for explaining an example of a method for extracting a viewpoint position. In the first embodiment, first, the own-vehicle viewpoint position extraction unit 4 superimposes the movement path 25 along which the autonomous mobile robot has moved so far on the photographable area 24 based on the movement path information, and extracts, as an extraction path, a portion of the movement path 25 that is included in the photographable area 24. In the example of FIG. 3, continuous extraction paths 25a and 25b are extracted.

[0040] Then, the own-vehicle viewpoint position extraction unit 4 calculates an evaluation value for each position in the extraction path based on the distance from the work position 22, the distance from the boundary of the photographable area 24, and the time elapsed since passing through the extraction path. For example, the own-vehicle viewpoint position extraction unit 4 superimposes the autonomous mobile robot 21 on a captured image at a position in the extraction path based on the movement path information, the optical system information of the camera mounted on the autonomous mobile robot 21, and the information indicating the size of the autonomous mobile robot 21. Then, the own-vehicle viewpoint position extraction unit 4 may increase the evaluation value of the position as the ratio of the autonomous mobile robot 21 occupying the captured image approaches a certain range. Also, for example, the own-vehicle viewpoint position extraction unit 4 may decrease the evaluation value of the position as the distance from the boundary of the photographable area 24 of the position in the extraction path becomes smaller than a certain value. Also, for example, the own-vehicle viewpoint position extraction unit 4 may decrease the evaluation value of the position as the period from the time when the autonomous mobile robot 21 passed through the position in the extraction path to the current time becomes longer.

[0041] Note that the position at which the evaluation value is calculated may be, for example, a position obtained by appropriately dividing the extraction path and each position within each division, or a position at which some index for calculating an evaluation value such as the shooting distance takes a maximum value. Also, the position at which the evaluation value is calculated is not limited to these.

[0042] Next, the own-vehicle viewpoint position extraction unit 4 determines, for each extraction path, the position with the highest evaluation value as a candidate for the viewpoint position. Then, the own-vehicle viewpoint position extraction unit 4 compares the evaluation values of the viewpoint position candidates and determines the candidate with the highest evaluation value as the viewpoint position.

[0043] In the example of FIG. 3, the own-vehicle viewpoint position extraction unit 4 calculates an evaluation value at each position within the extraction path 25a, and determines the position with the highest evaluation value as the candidate 26a for the viewpoint position. Similarly, the own-vehicle viewpoint position extraction unit 4 calculates an evaluation value at each position within the extraction path 25b, and determines the position with the highest evaluation value as the candidate 26b for the viewpoint position. The own-vehicle viewpoint position extraction unit 4 compares the evaluation values of the candidates 26a and 26b for the viewpoint position, and determines the candidate with the highest evaluation value as the viewpoint position. For example, although the candidate 26b is a past capture of the captured image compared to the candidate 26a, if a certain distance can be ensured from the work position 22 compared to the candidate 26a, the evaluation value of the candidate 26b becomes higher than the evaluation value of the candidate 26a, and the candidate 26b is determined as the viewpoint position. After the extraction of the viewpoint position, the process proceeds to the superimposed image generation unit 5.

[0044] <superimposed image generation unit 5> The superimposed image generation unit 5 generates a superimposed image in which a corresponding position image indicating the work position 22 is superimposed on the captured image captured at the viewpoint position, based on the captured image, the work position 22 calculated by the work position calculation unit 2, and the viewpoint position extracted by the own-vehicle viewpoint position extraction unit 4. In the first embodiment, a work image indicating the work being performed by the autonomous mobile robot 21 when the remote operation request acquisition unit 1 acquired the remote operation request is used for the corresponding position image.

[0045] In the first embodiment, first, the superimposed image generation unit 5 extracts, from the captured images acquired by the acquisition unit 1, the captured image captured when the autonomous mobile robot 21 passed through the viewpoint position extracted by the own-vehicle viewpoint position extraction unit 4, as the objective viewpoint environment image. For example, when there are a plurality of captured images for the viewpoint position because the autonomous mobile robot 21 passes through or stops at the viewpoint position a plurality of times, the superimposed image generation unit 5 extracts the latest captured image among them as the objective viewpoint environment image. Also, for example, when there is no captured image captured when passing through the viewpoint position, the superimposed image generation unit 5 extracts the captured image captured before and after the time when passing through the viewpoint position and at the time closest to that time as the objective viewpoint environment image.

[0046] Next, the superimposed image generation unit 5 generates a work image that reproduces the work being performed by the autonomous mobile robot 21 when the remote operation request is acquired by the acquisition unit 1 based on the work information. The work to be reproduced is, for example, the work being performed that is the calculation target of the work position by the work position calculation unit 2. The work image may include only the operation of the autonomous mobile robot 21 without including an image of the environment such as the background. The work image may be, for example, an image that operates a 3D model of the autonomous mobile robot 21.

[0047] Next, the superimposed image generation unit 5 generates a work situation reproduction image, which is a superimposed image, by superimposing the work image on the objective viewpoint environment image. Since methods for calibrating the coordinate systems of each image among a plurality of images are widely known, detailed descriptions thereof are omitted. After generating the work situation reproduction image, the process proceeds to the information display operation unit.

[0048] In the above description, the superimposed image generation unit 5 directly uses the captured image taken at the viewpoint position as the objective viewpoint environment image, but this is not limited thereto as described in other embodiments. Further, the superimposed image generation unit 5 may superimpose the name of the article shown in the captured image and additional information obtained by other functions on the objective viewpoint environment image.

[0049] <Display unit 6> The display unit 6 displays the work situation reproduction image, which is a superimposed image. Note that the display unit 6 may also display information useful for remote operation, such as the request status of remote operation, facility map information, information on the operation target, and captured images from the past over a long period of time. Further, the display unit 6 may switch the display information according to the operation of the operator. Further, the display unit 6 is configured to be communicable with the superimposed image generation unit 5 by a wireless technology such as wireless LAN (Local Area Network), and is a display device that can be viewed by an operator located remotely from the autonomous mobile robot 21.

[0050] <Summary of Embodiment 1> According to the image display device according to the first embodiment as described above, among the movement paths 25 that the autonomous mobile robot 21 has moved in the past, the positions included in the photographable area 24 are extracted as viewpoint positions, and based on the photographed image, the working position 22, and the viewpoint position, it is possible to generate and display a work situation reproduction image in which a corresponding position image indicating the working position 22 is superimposed on the photographed image taken at the viewpoint position. According to such a configuration, the operator can easily grasp the situation of the autonomous mobile robot 21 by checking the work situation reproduction image viewed from an objective viewpoint position in a situation where the autonomous mobile robot 21 can no longer continue working.

[0051] In particular, in indoor work with many obstacles, the display of the working position of the autonomous mobile robot 21 viewed from an objective viewpoint position may be obstructed by obstacles or the like. In contrast, in the first embodiment, since the objective viewpoint position can be optimized by the photographable area 24, it is possible to suppress the display of the working position of the autonomous mobile robot 21 viewed from an objective viewpoint position from being obstructed by obstacles or the like. In addition, the operator can eliminate the trouble of searching for a photographed image for generating a work situation reproduction image from the photographed images acquired by the acquisition unit 1.

[0052] Also, in the first embodiment, the corresponding position image indicating the working position 22 includes a work image indicating the work that the autonomous mobile robot 21 was performing when a remote operation request was acquired. According to such a configuration, the operator can easily grasp the work of the autonomous mobile robot 21 that has fallen into a state where it cannot autonomously perform the work.

[0053] <Modification Example> In the first embodiment, the own-vehicle viewpoint position extraction unit 4 determines not only the extraction of the viewpoint position such as the determination of the candidate for the viewpoint position, but also the determination of the viewpoint position itself, but it is not limited to this. For example, the extraction of the viewpoint position in the own-vehicle viewpoint position extraction unit 4 includes the determination of the candidate for the viewpoint position and the calculation of the evaluation value of the candidate, and the operator may determine the viewpoint position from among the candidates for the viewpoint position based on the evaluation value. In this case, the own-vehicle viewpoint position extraction unit 4 may display the candidate for the viewpoint position and the evaluation value of the candidate to the operator.

[0054] FIG. 4 is a diagram showing an example of the display of the own-vehicle viewpoint position extraction unit 4 to the operator. In the example of FIG. 4, the autonomous mobile robot 21, the work position 22, the obstacle 23, the photographable area 24, and the movement path 25 of the autonomous mobile robot 21 are displayed on the map. Further, among the extraction paths 25a and 25b within the photographable area 24, the positions with high evaluation values are displayed to the operator as candidates 26a and 26b for the viewpoint position. In such a case, the own-vehicle viewpoint position extraction unit 4 may determine the candidate selected by the operator from among the candidates 26a and 26b as the viewpoint position.

[0055] Note that the operator may not determine the viewpoint position from among the candidates 26a and 26b, but may determine an arbitrary point on the extraction paths 25a and 25b as the viewpoint position. In this case, the own-vehicle viewpoint position extraction unit 4 may highlight the extraction paths 25a and 25b within the photographable area 24. Further, the extraction paths 25a and 25b may be displayed so that the photographing time can be identified by the color or thickness of the line. For example, the longer the period from the time when the autonomous mobile robot 21 passes through the position within the extraction path to the current time, the lighter the color of the line at that position, or the color tone such as red and blue of the line at that position may be changed, or the line at that position may be made thinner.

[0056] Further, when the camera mounted on the autonomous mobile robot 21 is not an omnidirectional camera, as shown in FIG. 5, the photographing ranges 27a and 27b of the mounted camera may be displayed for the candidates 26a and 26b. According to such a configuration, even if the mounted camera is not an omnidirectional camera, an appropriate photographed image can be used for the work situation reproduction image.

[0057] For example, the extraction of the viewpoint position in the own-vehicle viewpoint position extraction unit 4 may include determination of candidate viewpoint positions and calculation of evaluation values thereof, and the superimposed image generation unit 5 may determine the viewpoint position from among the candidate viewpoint positions based on the evaluation values.

[0058] <Embodiment 2> In the second embodiment, the texture image of the three-dimensional space of the surrounding environment is updated using the time-series captured images, so that a three-dimensional virtual environment image that can be used as the objective viewpoint environment image of the superimposed image generation unit 5 and can be changed in time series is generated.

[0059] FIG. 6 is a block diagram showing the configuration of the image display device according to the second embodiment. The configuration of the image display device in FIG. 6 is the same as the configuration of the image display device in FIG. 1 with an environment image generation unit 7 added.

[0060] In the second embodiment, the work environment information includes the position and orientation information of the three-dimensional space of the fixed objects in the work space, and the position and orientation information of the three-dimensional space indicates the three-dimensional objects of the fixed objects.

[0061] The environment image generation unit 7 generates a three-dimensional virtual environment image that can be changed in time series by attaching the time-series captured images as textures to the three-dimensional objects based on the time-series captured images and the position and orientation information included in the work environment information. Hereinafter, an example of generating the virtual environment image will be described.

[0062] First, in the second embodiment, the environment image generation unit 7 extracts, as the in-view object information, the position and orientation information indicating the three-dimensional objects that can be photographed from the viewpoint position extracted by the own-vehicle viewpoint position extraction unit 4 among the position and orientation information. The in-view object information includes, for example, information about which surface of which three-dimensional object can be photographed.

[0063] Then, until a remote operation is requested, the environment image generation unit 7 determines in chronological order whether the captured images captured and stored include a captured image corresponding to the three-dimensional object of the in-vision object information.

[0064] When it is determined that the captured images include a captured image corresponding to the three-dimensional object, the environment image generation unit 7 generates a virtual environment image by attaching the captured image as a texture to the three-dimensional object in chronological order. As a result, a three-dimensional virtual environment image that can be changed in chronological order as viewed from the viewpoint position extracted by the own-vehicle viewpoint position extraction unit 4 is generated.

[0065] When it is not determined that the captured images include a captured image corresponding to the three-dimensional object, the environment image generation unit 7 extracts a captured image captured at the viewpoint position extracted by the own-vehicle viewpoint position extraction unit 4 from among the captured images acquired by the acquisition unit 1.

[0066] The superimposed image generation unit 5 generates a work situation reproduction image by using the virtual environment image generated by the environment image generation unit 7 or the extracted captured image as the objective viewpoint environment image described in the first embodiment. In the first embodiment, a static captured image captured at the viewpoint position is used as the objective viewpoint environment image, but in the second embodiment, it is possible to use a three-dimensional virtual environment image that can be changed in chronological order as the objective viewpoint environment image. When a work image showing the work of the autonomous mobile robot 21 is superimposed on the objective viewpoint environment image composed of the virtual environment image, by matching the time of the work image and the time of the objective viewpoint environment image, the degree of reproduction of the situation of the autonomous mobile robot 21 can be increased.

[0067] Note that the time of the work image and the time of the objective viewpoint environment image may be different. FIG. 7 is an example of a screen for generating a work situation reproduction image by making the time of the work image and the time of the objective viewpoint environment image different. At the lower part of the screen in FIG. 7, there is a seek bar for specifying the time of the work image of the autonomous mobile robot, the time of the viewpoint position defining the three-dimensional object, and the time of the captured image pasted as a texture on the three-dimensional object. In this example, the time of the viewpoint position and the time of the captured image are included in the time of the objective viewpoint environment image.

[0068] By the operator performing an operation of specifying the time on the seek bar, the objective viewpoint environment image is reproduced with the combination of the times specified for the work image, the viewpoint position, and the captured image. Note that various methods for combining the three-dimensional object and the captured image have already been proposed, and it is not limited to the above.

[0069] <Summary of Embodiment 2> According to the second embodiment, a three-dimensional virtual environment image that can be changed in time series is generated based on the time-series captured images and the position and orientation information of the three-dimensional space of the fixed objects in the work space. With such a configuration, the operator can more easily grasp the situation of the autonomous mobile robot 21 that has fallen into a state where the work cannot be performed autonomously.

[0070] <Embodiment 3> In the third embodiment, it is configured to calculate the position and orientation information of the moving obstacle.

[0071] FIG. 8 is a block diagram showing the configuration of the image display device according to the third embodiment. The configuration of the image display device in FIG. 8 is the same as the configuration of the image display device in FIG. 1 with the addition of a moving obstacle information calculation unit 8.

[0072] Note that the object according to the third embodiment includes at least one of a fixed obstacle and a moving obstacle. The fixed obstacle is, for example, an obstacle with little change in position and orientation, such as a wall, a door, a shelf, and a desk. The moving obstacle is, for example, a moving body such as a person and a mobile robot, and an obstacle with a lot of change in position and orientation such as a chair.

[0073] In the third embodiment, the obstacle information in the work environment information includes the position and orientation information of the fixed obstacles in the work space.

[0074] The moving obstacle information calculation unit 8 calculates the position and orientation information of the moving obstacles that are displayed on the display unit 6 separately from the fixed obstacles based on the position and orientation information of the objects in the work space and the position and orientation information of the fixed obstacles included in the work environment information. The position and orientation information of the objects in the work space is, for example, the position and orientation information of an object including at least one of a fixed obstacle and a moving obstacle recognized by a sensor mounted on an autonomous mobile robot 21 or the like. The moving obstacle information calculation unit 8 calculates the position and orientation information of the moving obstacles by subtracting the position and orientation information of the fixed obstacles from the position and orientation information of the objects in the work space. For example, the distance information recognized by a LiDER or a stereo camera may be used as the position and orientation information of the object. In this case, the moving obstacle information calculation unit 8 may calculate the information of the moving obstacles by removing the position and orientation information of the fixed obstacles from the distance information.

[0075] The moving obstacles are, for example, highlighted by color, characters, and other image effects in the objective viewpoint environment image of the superimposed image generation unit 5. The highlighting in the objective viewpoint environment image of the superimposed image generation unit 5 is reflected in the work situation reproduction image displayed on the display unit 6. Note that the moving obstacles indicated by the calculated position and orientation information may be highlighted in the work situation reproduction image of the display unit 6 without passing through the objective viewpoint environment image of the superimposed image generation unit 5.

[0076] In the above configuration, when a model of a moving obstacle is obtained in advance, the moving obstacle information calculation unit 8 may recognize the moving obstacle by model matching from the objects in the captured image that does not include distance information, and calculate the position and orientation information of the moving obstacle. For example, models of moving obstacles such as chairs, people, and mobile robots are included in the work environment information, and the moving obstacle information calculation unit 8 performs model matching with reference to the models. For people, instead of precise model matching, general model matching for people may be performed. Further, the moving obstacle information calculation unit 8 may generate tag information for distinguishing objects such as people, mobile robots, and chairs by tagging the objects. This tag information may be used to change the highlighting expression according to the type of the moving obstacle, or add the name of the moving obstacle in the objective viewpoint environment image of the superimposed image generation unit 5 or the work situation reproduction image of the display unit 6.

[0077] In the position and orientation information of the fixed obstacle included in the work environment information, the fixed obstacle and the moving obstacle may be switched during operation. For example, when the recognition result based on the distance information is different from the previous recognition result, the fixed obstacle at the previous recognition time may be switched to a moving obstacle. Conversely, when the recognition result based on the distance information does not change for a certain period, the moving obstacle at the previous recognition time may be switched to a fixed obstacle.

[0078] <Summary of Embodiment 3> Since the influence of the fixed obstacle is assumed in advance when planning the work, the possibility of inhibiting the work execution of the autonomous mobile robot 21 is low. On the other hand, since the influence of the moving obstacle cannot be fully assumed in advance when planning the work, the possibility of inhibiting the work execution of the autonomous mobile robot 21 is relatively high.

[0079] Therefore, according to the third embodiment, the position and orientation information of the moving obstacle displayed on the display unit 6 is calculated separately from the fixed obstacle. With such a configuration, since the moving obstacle can be displayed separately from the fixed obstacle, the operator can more easily grasp the situation of the autonomous mobile robot 21 that has fallen into a state where it cannot perform work autonomously.

[0080] <Embodiment 4> In the fourth embodiment, the image of another camera other than the camera mounted on the autonomous mobile robot 21 that has requested remote operation is configured to be available.

[0081] FIG. 9 is a block diagram showing the configuration of the image display device according to the fourth embodiment. The configuration of the image display device in FIG. 9 is the same as the configuration of the image display device in FIG. 1, with an additional other-machine viewpoint position extraction unit 9, which is a second viewpoint position extraction unit.

[0082] The other-machine viewpoint position extraction unit 9 calculates a different viewpoint position for a camera other than the camera mounted on the autonomous mobile robot 21 that has requested remote operation. Hereinafter, the case where the other camera is a camera mounted on another autonomous mobile robot (i.e., another moving body) other than the autonomous mobile robot 21 that has requested remote operation will be mainly described.

[0083] Except for the points where the movement paths are different, the processing of the other-machine viewpoint position extraction unit 9 is the same as the processing of the own-machine viewpoint position extraction unit 4. That is, the other-machine viewpoint position extraction unit 9 extracts, as different viewpoint positions, the positions included in the photographable area 24 among the positions on different movement paths that another autonomous mobile robot has moved along in the past.

[0084] Specifically, after the photographable area 24 is calculated, the other-machine viewpoint position extraction unit 9 overlays a different movement path along which another autonomous mobile robot has moved so far on the photographable area 24 based on the movement path information of the other autonomous mobile robot, and extracts the portion of the different movement path included in the photographable area 24 as a different extraction path. The other-machine viewpoint position extraction unit 9 calculates the evaluation value of each position in the different extraction path and determines a different viewpoint position based on the evaluation value.

[0085] The superimposed image generation unit 5 superimposes a corresponding position image on another captured image taken from another viewpoint position based on another captured image taken from another autonomous mobile robot, the working position 22, and another viewpoint position determined by the other-robot viewpoint position extraction unit 9. By performing such superimposition, the superimposed image generation unit 5 generates a reproduced working condition image to be displayed on the display unit 6. Note that the superimposed image generation unit 5 may compare the evaluation value of the viewpoint position calculated by the own-robot viewpoint position extraction unit 4 with the evaluation value of another viewpoint position calculated by the other-robot viewpoint position extraction unit 9, and generate a reproduced working condition image only for the one with the higher evaluation value among them.

[0086] The superimposed image generation unit 5 may use an image of, for example, a surveillance camera fixed to the ceiling instead of the image of the on-board camera of the autonomous mobile robot. In the image of the surveillance camera or the like, the other viewpoint position is not changed.

[0087] Also, when an appropriate other viewpoint position cannot be obtained, the operator may move another autonomous mobile robot by remote operation.

[0088] FIG. 10 is a diagram showing an example of display to the operator. In FIG. 10, the viewpoint position 28 and the shooting range 29 of the fixed surveillance camera and the moving destination 30 of another autonomous mobile robot are added to the display in FIG. 4. When a map like that in FIG. 10 is displayed to the operator and the operator selects the moving destination 30 on the map, another autonomous mobile robot may be configured to move to the moving destination 30. Note that another autonomous mobile robot can still take a picture even after the autonomous mobile robot 21 requests remote operation. Therefore, when the display unit 6 displays an image taken by another autonomous mobile robot, whether the image is an image after the remote operation request or an image before the remote operation request may be discriminatively displayed by the color of a display icon or the like. Also, the display unit 6 may display an image of the shooting time specified by a seek bar or the like.

[0089] <Summary of Embodiment 4> According to the fourth embodiment, an image captured by another moving body such as another autonomous mobile robot can be used. As a result, when there is no appropriate captured image within the shootable area 24 in the captured image of the autonomous mobile robot 21, or when the operator desires an image with a shooting direction different from the captured image, the image captured by another moving body can be used. As a result, the operator can more easily grasp the situation of the autonomous mobile robot 21 that has fallen into a state where it cannot autonomously perform work.

[0090] <Embodiment 5> In the fifth embodiment, the configuration is such that a simulation image can be generated when work is virtually performed on the autonomous mobile robot 21 based on a remote operation from an operator.

[0091] FIG. 11 is a block diagram showing the configuration of the image display device according to the fifth embodiment. The configuration of the image display device in FIG. 11 is the same as the configuration of the image display device in FIG. 1, with the addition of a virtual image generation unit 10.

[0092] The virtual image generation unit 10 generates a simulation image when work is virtually performed on the autonomous mobile robot 21 based on a remote operation from an operator. Before operating the autonomous mobile robot 21 by remote operation, the virtual image generation unit 10 generates a simulation image when operating a 3D model or the like of the autonomous mobile robot 21 according to the remote operation. The simulation image is superimposed and displayed on the work situation reproduction image on the display unit 6. Note that the simulation image may be superimposed on the objective viewpoint environment image of the superimposed image generation unit 5 so as to be superimposed and displayed on the work situation reproduction image on the display unit 6.

[0093] <Summary of Embodiment 5> It is assumed that the operator remotely operates a plurality of autonomous mobile robots. In such an operation, a remote operation mistake after a remote operation request may lead to an accident. In particular, since there are a wide variety of types of autonomous mobile robots and the operation methods and behaviors differ for each autonomous mobile robot, such a possibility is considered to be relatively high.

[0094] In contrast, according to the fifth embodiment, a simulation image is generated when the autonomous mobile robot 21 virtually performs an operation based on a remote operation from an operator. According to such a configuration, before actually remotely operating the autonomous mobile robot 21, the operator can check whether there is any problem with the remote operation by viewing the simulation image superimposed on the work situation reproduction image. As a result, the operator can change the remote operation until it is confirmed that there is no problem with the remote operation, so that remote operation errors can be reduced.

[0095] <Embodiment 6> In the sixth embodiment, when the work being performed by the autonomous mobile robot 21 when a remote operation request is acquired is an operation from the autonomous mobile robot 21 to the object to be worked on, the configuration is such that an image of the object to be worked on that has been photographed can be acquired.

[0096] FIG. 12 is a block diagram showing the configuration of the image display device according to the sixth embodiment. The configuration of the image display device in FIG. 12 is the same as the configuration of the image display device in FIG. 1, with the addition of the object to be worked on image acquisition unit 11.

[0097] The object to be worked on image acquisition unit 11 acquires an image of the object to be worked on that has been photographed when the work being performed by the autonomous mobile robot 21 when a remote operation request is acquired is an operation from the autonomous mobile robot 21 to the object to be worked on. The operation from the autonomous mobile robot 21 to the object to be worked on includes operations using the end effector of the autonomous mobile robot 21 or the like. For photographing the image of the object to be worked on, for example, among the mounted cameras, a mobile camera that can photograph a three-dimensional space image in real time and can be moved from the main body of the autonomous mobile robot 21 is used. The object to be worked on image acquisition unit 11 may change the viewpoint position of the mobile camera by moving the mobile camera so that an image that is easy to recognize the object to be worked on is photographed.

[0098] First, in the sixth embodiment, the workpiece object image acquisition unit 11 determines whether a three-dimensional space image of the workpiece object or the end effector is required. For example, when the operation content being performed by the autonomous mobile robot 21 at the time a remote operation request is acquired is an operation on the workpiece object using the end effector, it is determined that a three-dimensional space image is required. Also, when the end effector is being operated, or when there is some object near the end effector, etc., it may also be determined that a three-dimensional space image is required. When it is determined that a three-dimensional space image is required, the workpiece object image acquisition unit 11 determines the viewpoint position of the mobile camera that is photographing the workpiece object as the workpiece object photographing position.

[0099] The display unit 6 displays the workpiece object image from the workpiece object photographing position to the operator. When only the end effector operates before and after the operation at the time of a remote operation request, the display unit 6 may display only the workpiece object image without displaying the objective viewpoint environment image.

[0100] Incidentally, the image suitable for the operator varies depending on the operation content. In an operation to move the autonomous mobile robot 21 body, an image at an objective viewpoint position where the entire autonomous mobile robot 21 can be confirmed is suitable for the operator to judge the situation. On the other hand, when the autonomous mobile robot 21 performs an operation such as transferring a workpiece object using the end effector, a three-dimensional space image centered on the end effector and the workpiece object and with a freely changeable viewpoint position is suitable for the operator to judge the situation.

[0101] In particular, when operating the object to be worked on, it is difficult to grasp the sense of distance from an image with only a fixed viewpoint, so it is desirable that the operator can confirm the work while changing the viewpoint position. On the other hand, when performing an operation to move the object to be worked on significantly, it is desirable that the operator can simultaneously confirm both the state of the main body of the autonomous mobile robot 21 and the state of the object to be worked on by the end effector. Therefore, the display unit 6 may selectively perform a display in which the objective viewpoint environment image and the object image to be worked on are arranged side by side, and a display of either the objective viewpoint environment image or the object image to be worked on, based on the work content at the time of a remote operation request. Note that the display in which the objective viewpoint environment image and the object image to be worked on are arranged side by side is useful not only when the operator confirms the work at the time of a remote operation request but also when the operator performs a remote operation.

[0102] The object image acquisition unit 11 may reduce blind spots or the like by combining a three-dimensional space image captured at a time point before the autonomous mobile robot 21 starts working (for example, at the time of system startup or another work time point) with a real-time object image to be worked on. Further, when the three-dimensional space image can be expanded, the expanded three-dimensional space image may be displayed on the display unit 6. When such a display is performed, if the operator changes the viewpoint position within the three-dimensional space image, the viewpoint position of the mobile camera may be changed. According to such a configuration, the operator can confirm the object to be worked on and the end effector by changing the viewpoint position within the three-dimensional space image.

[0103] <Summary of Embodiment 6> According to the sixth embodiment, when the work being performed by the autonomous mobile robot 21 when a remote operation request is acquired is work from the autonomous mobile robot 21 to the object to be worked on, an image of the object to be worked on being photographed is acquired. According to such a configuration, the operator can easily grasp the work performed by the autonomous mobile robot 21 that has fallen into a state where it cannot autonomously perform the work on the object to be worked on.

[0104] <Embodiment 7> In the seventh embodiment, the autonomous mobile robot 21 is configured to be movable so that a captured image can be acquired at a point in the work space where a predetermined time has elapsed since the captured image was captured.

[0105] FIG. 13 is a block diagram showing the configuration of the image display device according to the seventh embodiment. The configuration of the image display device in FIG. 13 is the same as the configuration of the image display device in FIG. 1, with a shooting position acquisition unit 12 and a plan creation unit 13 added.

[0106] The shooting position acquisition unit 12 acquires, as the shooting position, a point in the work space where a predetermined time has elapsed since the captured image was captured. The shooting position acquisition unit 12 manages the time when the captured image was captured at each point in the work space based on the work information and the work environment information. If there is a point where the captured image has not been captured for a certain period of time, such as 15 minutes, the shooting position acquisition unit 12 acquires that point as the shooting position.

[0107] The plan creation unit 13 creates a plan for the path along which the autonomous mobile robot 21 should move, that is, a plan for the movement path of the autonomous mobile robot 21, based on the shooting position. The plan creation unit 13 plans the movement path and the work instruction based on the work plan of the autonomous mobile robot 21. When the shooting position acquisition unit 12 has acquired the shooting position, the plan creation unit 13 plans the movement path and the work instruction in consideration of the shooting position. The shooting position may be used as the destination of the movement path of the autonomous mobile robot 21 or as a waypoint on the movement path. Since the method of planning the movement path of the autonomous mobile robot 21 is widely known, a detailed description thereof is omitted.

[0108] In addition, similar to the display of candidate viewpoint positions and the selection of candidates by the operator described in the modification of Embodiment 1, the display of candidate shooting positions and the selection of candidates by the selector may be performed. For example, the shooting position acquisition unit 12 may calculate a degree of requirement proportional to the time elapsed since the shooting image was taken for each candidate shooting position, and display each candidate shooting position and its degree of requirement. Then, the shooting position acquisition unit 12 may increase the degree of requirement of the candidate shooting position selected by the operator by a certain value, and acquire the candidate shooting position whose degree of requirement is equal to or higher than the threshold value as the shooting position.

[0109] <Summary of Embodiment 7> According to the seventh embodiment, in the work space, the autonomous mobile robot 21 is moved to a point where a predetermined time has elapsed since the shooting image was taken. With such a configuration, the shooting image at that point can be updated, and the possibility of obtaining an appropriate shooting image for the objective viewpoint environment image can be increased.

[0110] <Embodiment 8> In the eighth embodiment, the autonomous mobile robot 21 is configured to be movable so that a shooting image can be acquired at a point in the work space where a person has completed the work.

[0111] FIG. 14 is a block diagram showing the configuration of the image display device according to the eighth embodiment. The configuration of the image display device in FIG. 14 is the same as the configuration of the image display device in FIG. 1, with the addition of a plan creation unit 13 and a work completion position acquisition unit 14.

[0112] The work completion position acquisition unit 14 acquires, as the shooting position, a point in the work space where a person has completed the work. For example, the work completion position acquisition unit 14 acquires, as the position where the work has been completed, that is, the shooting position, a point where a person has moved after staying for a certain period of time. The determination of the movement of the person may be made by tracking using a camera image or the like, or may be made from the work information of the worker when the person is an employee.

[0113] The planning unit 13 is the same as the planning unit 13 according to Embodiment 7.

[0114] <Summary of Embodiment 8> After a person has completed their work, there is a high possibility that the installation status of the installed equipment has changed. For example, in an office environment, changes to the installation status of installed equipment include moving chairs. In a sales facility for goods, changes to the installation status of installed equipment include changing the installation position of goods. In a restaurant, after the staff and customers have completed their work, there is a high possibility that the installation status of the installed equipment has changed.

[0115] Therefore, according to the eighth embodiment, in the working space, the point where a person has completed their work is moved to the autonomous mobile robot 21. According to such a configuration, since it is possible to take an image from the environment after a person has completed their work, where there is a high possibility that the working environment has changed, it is possible to secure an image of a point where remote operation by an operator is likely to be requested.

[0116] <Other Modification Examples> The acquisition unit 1, the working position calculation unit 2, the shootable area calculation unit 3, the own vehicle viewpoint position extraction unit 4, and the superimposed image generation unit 5 shown in FIG. 1 above are hereinafter referred to as "acquisition unit 1 etc.". The acquisition unit 1 etc. is realized by the processing circuit 81 shown in FIG. 15. That is, the processing circuit 81 is an acquisition unit 1 that acquires a request for remote operation of a moving body, work information indicating work including the movement of the moving body, and a captured image captured from the moving body, and based on the request and the work information, calculates the working position of the moving body when the request is acquired. A working position calculation unit 2, work environment information including map information indicating the working space of the moving body and obstacle information regarding obstacles in the working space, and based on the working position, a shootable area calculation unit 3 that calculates the area where a captured image including an image of the working position can be captured as the shootable area, and based on the work information and the shootable area, an own vehicle viewpoint position extraction unit 4 that extracts, as the viewpoint position, a position included in the shootable area among the positions in the movement path where the moving body has moved in the past, and based on the captured image, the working position, and the viewpoint position, a superimposed image generation unit 5 that generates a superimposed image in which a corresponding position image indicating the working position is superimposed on the captured image captured at the viewpoint position. For the processing circuit 81, dedicated hardware may be applied, or a processor that executes a program stored in a memory may be applied. Examples of the processor include a central processing unit, a processing device, an arithmetic device, a microprocessor, a microcomputer, and a DSP (Digital Signal Processor).

[0117] When the processing circuit 81 is dedicated hardware, the processing circuit 81 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each part of the acquisition unit 1 etc. may be realized by a circuit in which the processing circuit is distributed, or the functions of each part may be realized by a single processing circuit.

[0118] When the processing circuit 81 is a processor, functions such as the acquisition unit 1 are realized in combination with software or the like. Note that software or the like includes, for example, software, firmware, or software and firmware. Software or the like is described as a program and stored in a memory. As shown in FIG. 16, the processor 82 applied to the processing circuit 81 reads and executes a program stored in the memory 83 to realize the functions of each part. That is, when executed by the processing circuit 81, the image display device performs a step of acquiring a request for remote operation of a moving body, work information indicating work including the movement of the moving body, and a captured image captured from the moving body; a step of calculating a work position of the moving body at the time when the request is acquired based on the request and the work information; a step of calculating, as a photographable area, an area in the captured image including an image of the work position based on work environment information including map information indicating the work space of the moving body and obstacle information regarding obstacles in the work space and the work position; a step of extracting, as a viewpoint position, a position included in the photographable area among positions in the movement path along which the moving body has moved in the past based on the work information and the photographable area; a step of generating a superimposed image in which a corresponding position image indicating the work position is superimposed on the captured image captured at the viewpoint position based on the captured image, the work position, and the viewpoint position; and a step of displaying the superimposed image. In other words, this program can be said to cause a computer to execute procedures and methods such as the acquisition unit 1.Here, the memory 83 may be a non-volatile or volatile semiconductor memory such as, for example, RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), their drive devices, or any storage medium to be used in the future.

[0119] Above, the configuration in which each function such as the acquisition unit 1 is realized by either hardware or software has been described. However, it is not limited to this, and a configuration in which a part of the acquisition unit 1 etc. is realized by dedicated hardware and another part is realized by software etc. may be used. For example, for the acquisition unit 1, its function can be realized by the processing circuit 81 as dedicated hardware, and for the rest, the processing circuit 81 as the processor 82 reads and executes the program stored in the memory 83 to realize its function.

[0120] As described above, the processing circuit 81 can realize each of the above functions by hardware, software, etc., or a combination thereof.

[0121] Also, the image display device described above can also be applied to an image display system constructed as a system by appropriately combining an information processing device, a communication terminal including a mobile terminal such as a mobile phone, a smartphone, and a tablet, the function of an application installed in the information processing device, and a server. In this case, each function or each component of the image display device described above may be distributed and arranged in each device constructing the system, or may be concentrated and arranged in any one device.

[0122] Note that it is possible to freely combine each embodiment and each modification example, or to appropriately modify or omit each embodiment and each modification example.

[0123] Hereinafter, various aspects of the present disclosure will be collectively described as appendices.

[0124] (Appendix 1) An acquisition unit that acquires a request for remote operation of a moving body, work information indicating work including the movement of the moving body, and a captured image captured from the moving body; A work position calculation unit that calculates the work position of the moving body when the request is acquired based on the request and the work information; A work environment information including map information indicating the work space of the moving body and obstacle information regarding obstacles in the work space, and a photographable area calculation unit that calculates, based on the work environment information and the work position, a photographable area including an image of the work position in the captured image as a photographable area; A first viewpoint position extraction unit that extracts, based on the work information and the photographable area, a position included in the photographable area among positions in the movement path where the moving body has moved in the past as a viewpoint position; A superimposed image generation unit that generates a superimposed image in which a corresponding position image indicating the work position is superimposed on the captured image captured at the viewpoint position based on the captured image, the work position, and the viewpoint position; A display unit that displays the superimposed image An image display device comprising:

[0125] (Appendix 2) The image display device according to Appendix 1, wherein the corresponding position image includes a work image indicating the work being performed by the moving body when the request is acquired.

[0126] (Appendix 3) The work environment information includes position and orientation information of a three-dimensional space regarding fixed objects in the work space, An image display device according to appended note 1 or appended note 2, further comprising an environmental image generation unit that generates a three-dimensional virtual environment image that can be changed in time series based on the time-series captured images and the position and orientation information, and uses the captured image thus generated as the captured image used by the superimposed image generation unit.

[0127] (Appended note 4) The obstacle information in the work environment information includes the position and orientation information of fixed obstacles in the work space. An image display device according to any one of appended notes 1 to 3, further comprising a moving obstacle information calculation unit that calculates the position and orientation information of a moving obstacle that is displayed on the display unit separately from the fixed obstacle based on the position and orientation information of an object in the work space and the position and orientation information of the fixed obstacle in the work space.

[0128] (Appended note 5) Further comprising a second viewpoint position extraction unit that extracts, as another viewpoint position, a position included in the photographable area among positions on another movement path along which another moving body has moved in the past. The superimposed image generation unit generates the superimposed image to be displayed on the display unit by superimposing the corresponding position image on the another captured image taken from the another viewpoint position based on the another captured image taken from the another moving body, the work position, and the another viewpoint position. An image display device according to any one of appended notes 1 to 4.

[0129] (Appended note 6) An image display device according to any one of appended notes 1 to 5, further comprising a virtual image generation unit that generates a simulation image when the superimposed image is superimposed and displayed on the display unit and a work is virtually performed on the moving body based on a remote operation from an operator.

[0130] (Appended note 7) When the operation being performed by the mobile body when the request is acquired is an operation from the mobile body to the object to be worked on, the image display device according to any one of Appendices 1 to 6 further includes an object to be worked on image acquisition unit that acquires an image of the object to be worked on displayed on the display unit.

[0131] (Appendix 8) A photographing position acquisition unit that acquires, as a photographing position, a point in the working space where a predetermined time has elapsed since the photographed image was taken; A path planning unit that creates a plan of a path along which the mobile body should move based on the photographing position; The image display device according to any one of Appendices 1 to 7, further comprising:

[0132] (Appendix 9) A work completion position acquisition unit that acquires, as a photographing position, a point in the working space where a person has completed work; A path planning unit that creates a plan of a path along which the mobile body should move based on the photographing position; The image display device according to any one of Appendices 1 to 7, further comprising:

[0133] (Appendix 10) Acquire a request for remote operation of a mobile body, work information indicating work including the movement of the mobile body, and a photographed image taken from the mobile body. Based on the request and the work information, calculate the working position of the mobile body when the request is acquired. Based on the work environment information including the map information indicating the working space of the mobile body and the obstacle information regarding obstacles in the working space, and the working position, calculate the area where the photographed image including the image of the working position can be taken as the photographable area. Based on the work information and the photographable area, extract, as the viewpoint position, positions within the movement path along which the mobile body has moved in the past and included in the photographable area. Based on the captured image, the working position, and the viewpoint position, a superimposed image is generated by superimposing a corresponding position image indicating the working position on the captured image captured at the viewpoint position. An image display method for displaying the superimposed image.

Explanation of Signs

[0134] 1 Acquisition unit, 2 Working position calculation unit, 3 Shootable area calculation unit, 4 Own-vehicle viewpoint position extraction unit, 5 Superimposed image generation unit, 6 Display unit, 7 Environment image generation unit, 8 Moving obstacle information calculation unit, 9 Other-vehicle viewpoint position extraction unit, 10 Virtual image generation unit, 11 Object image acquisition unit for work, 12 Shooting position acquisition unit, 13 Plan creation unit, 14 Working completion position acquisition unit, 21 Autonomous mobile robot, 22 Working position, 23 Obstacle, 24 Shootable area, 25 Movement path.

Claims

1. An acquisition unit that acquires a request for remote operation of a moving body, work information indicating work including the movement of the moving body, and a captured image captured from the moving body; A work position calculation unit that calculates a work position of the moving body at the time when the request is acquired based on the request and the work information; A photographable area calculation unit that calculates, as a photographable area, an area in which the captured image including the image of the work position can be captured based on work environment information including map information indicating the work space of the moving body and obstacle information regarding obstacles in the work space, and the work position; A first viewpoint position extraction unit that extracts, as a viewpoint position, a position included in the photographable area among positions in a movement path along which the moving body has moved in the past based on the work information and the photographable area; A superimposed image generation unit that generates a superimposed image in which a corresponding position image indicating the work position is superimposed on the captured image captured at the viewpoint position based on the captured image, the work position, and the viewpoint position; A display unit that displays the superimposed image An image display device comprising:

2. The image display device according to claim 1, wherein the corresponding position image includes a work image indicating the work being performed by the moving body at the time when the request is acquired.

3. The image display device according to claim 1 or claim 2, wherein the work environment information includes position and orientation information of a three-dimensional space regarding fixed objects in the work space, and further comprises an environment image generation unit that generates a three-dimensional virtual environment image that can be changed in time series as the captured image used by the superimposed image generation unit based on the time-series captured images and the position and orientation information.

4. The image display device according to claim 1 or claim 2, wherein the obstacle information in the work environment information includes position and orientation information of fixed obstacles in the work space, and further comprises a moving obstacle information calculation unit that calculates position and orientation information of a moving obstacle that is displayed on the display unit separately from the fixed obstacles based on position and orientation information of an object in the work space and the position and orientation information of the fixed obstacles in the work space.

5. The image display device according to claim 1 or claim 2, further comprising a second viewpoint position extraction unit that extracts, as another viewpoint position, a position included in the photographable area among positions on another movement path along which another moving body has moved in the past, wherein the superimposed image generation unit An image display device that generates the superimposed image to be displayed on the display unit by superimposing the corresponding position image on the other captured image captured from the other moving body based on the other captured image, the working position, and the other viewpoint position.

6. The image display device according to claim 1 or claim 2, further comprising a virtual image generation unit that generates a simulation image when the work is virtually performed on the moving body based on the remote operation from the operator, and the simulation image is superimposed and displayed with the superimposed image on the display unit.

7. The image display device according to claim 1 or claim 2, further comprising a workpiece image acquisition unit that acquires an image of the workpiece captured by the display unit when the work being performed by the moving body when the request is acquired is work from the moving body to the workpiece.

8. The image display device according to claim 1 or claim 2, a shooting position acquisition unit that acquires, as a shooting position, a point in the work space where a predetermined time has elapsed since the captured image was captured; and a path planning unit that creates a plan for the path along which the moving body should move based on the shooting position. The image display device further comprising.

9. The image display device according to claim 1 or claim 2, a work completion position acquisition unit that acquires, as a shooting position, a point in the work space where a person has completed the work; and a path planning unit that creates a plan for the path along which the moving body should move based on the shooting position. The image display device further comprising.

10. Acquire a request for remote operation of the moving body, work information indicating work including the movement of the moving body, and a captured image captured from the moving body, calculate the working position of the moving body when the request is acquired based on the request and the work information, calculate, as a shootable area, an area in the captured image including the image of the working position based on the work environment information including the map information indicating the working space of the moving body and the obstacle information regarding obstacles in the working space, and the working position, extract, as a viewpoint position, a position included in the shootable area among the positions in the movement path along which the moving body has moved in the past based on the work information and the shootable area. Based on the captured image, the working position, and the viewpoint position, a superimposed image is generated by superimposing a corresponding position image indicating the working position on the captured image captured at the viewpoint position, An image display method for displaying the superimposed image.

Images