Remote control support device and remote control support method

Abstract

To improve the work efficiency of remote operators. [Solution] The remote control support device (60) comprises a virtual image generation unit (63) that generates a virtual image, a processed image generation unit (61) that generates a processed image by performing DR (Diminished Reality) processing on the actual image from the remote operator's viewpoint and AR (Augmented Reality) processing to synthesize the generated virtual image, and an output unit (62) that outputs the processed image to a display (30).

Description

Technical Field

[0001] The present disclosure relates to a remote operation support device and a remote operation support method.

Background Art

[0002] Conventionally, techniques for performing various processes on real images captured by a camera or the like are known. For example, Patent Document 1 describes a technique for providing information regarding the state or operation guide of a robot to a user of the robot using an augmented reality-compatible display in a real image or a real environment. In the robot system of Patent Document 1, an image of the robot that has been augmented realityized is displayed on an AR-compatible display.

[0003] In addition, Patent Document 2 describes a crane operation assist device capable of visualizing a portion that is blocked by a spreader and becomes a blind spot. In Patent Document 2, a technique for generating a composite image by synthesizing an enlarged image and a reference image captured by a camera, and synthesizing a virtual image of the spreader through the composite image is described.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, it is known that the work efficiency of remotely operating various work devices while visually observing the work video displayed on a display is lower than the work performed by actually boarding a work machine or the like at the site. One of the reasons for the decrease in work efficiency is that the amount of information obtained from the real image displayed on the display is small.

[0006] One aspect of this disclosure aims to improve the work efficiency of remote operators. [Means for solving the problem]

[0007] To solve the above problems, a remote operation support device according to one aspect of the present disclosure is a remote operation support device that outputs work video of work performed by remote operation to a display, comprising: a virtual video generation unit that generates a virtual video representing information about the work; a processed video generation unit that generates a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of the remote operator, which erases or makes transparent occluding objects in the area to be removed, and synthesizes an image that models the background hidden by overlapping with the occluding objects in the area to be removed, and AR (Augmented Reality) processing which synthesizes the virtual video generated on the real video or the video on which the DR processing has been performed; and an output unit that outputs the processed video to the display.

[0008] To solve the above problems, a remote operation support method according to one aspect of the present disclosure is a remote operation support method performed by a remote operation support device that outputs work video of work performed by remote operation to a display, and includes: a virtual video generation step of generating a virtual video representing information about the work; a processed video generation step of generating a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of a remote operator, which erases or makes transparent occluding objects in an area to be removed, and synthesizes an image that models the background hidden by overlapping with the occluding objects in the area to be removed, and AR (Augmented Reality) processing which synthesizes the virtual video generated on the real video or the video on which the DR processing has been performed; and an output step of outputting the processed video to the display. [Effects of the Invention]

[0009] According to one aspect of this disclosure, the work efficiency of the remote operator can be improved. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example of a remote control system. [Figure 2] This is a flowchart showing the flow of the remote control support method performed by the remote control support device. [Figure 3] This figure shows an example of an image displayed on a screen. [Figure 4] This is a block diagram showing an example of a remote control system. [Figure 5] This is a schematic diagram showing the overview of a rolling mill. [Figure 6] This figure shows an example of an image displayed on a screen. [Figure 7] This figure shows an example of an image displayed on a screen. [Figure 8] This figure shows an example of an image displayed on a screen. [Modes for carrying out the invention]

[0011] [Embodiment 1] One embodiment of this disclosure will be described in detail below.

[0012] [Overview of the remote control system] Referring to Figure 1, an overview of the remote control system 100 will be described. Figure 1 is a block diagram showing an example of the remote control system 100. The remote control system 100 is a system for a remote operator to remotely control a work device using a remote control device 10. As shown in Figure 1, the remote control system 100 comprises a remote control device 10, a work robot 20, a display 30, a camera 40, and a remote control support device 60. The remote control system 100 may further include a camera 50.

[0013] The remote operation device 10 is a device that is arranged at a position away from the work robot 20 and is operated by a remote operator. The remote operator operates the remote operation device 10 while checking the work video displayed on the display 30. The remote operation device 10 is communicably connected to the work robot 20 by wire or wirelessly.

[0014] The work robot 20 is a robot that is remotely operated by the remote operation device 10. The work robot 20 is an example of a work device that performs a predetermined process on a work target, such as physical processing or chemical processing, by moving, transforming, joint driving, or exerting various functions through remote operation. The work robot 20 is, for example, a robot that performs work in fields such as industry or medicine. Industries include agriculture, forestry, fishery, manufacturing, construction, and mining, etc.

[0015] The display 30 is arranged at a position where the remote operator can visually observe it. The display 30 is connected to the remote operation support device 60. The display 30 displays an image based on the video signal output from the remote operation support device 60. An image from the perspective of the remote operator is displayed on the display 30. That is, an image from a perspective suitable for the remote operator to perform work is displayed on the display 30. An image captured by the camera 40 (described later) is displayed on the display 30. The number of displays 30 may be one or more.

[0016] The camera 40 captures the work site where the work robot 20 is performing work. The camera 40 captures an image from the perspective of the remote operator as the work video to be displayed on the display 30. The camera 40 includes an image pickup device using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like. The camera 40 is connected to the remote operation support device 60 by wire or wirelessly. The actual image captured by the camera 40 is output to the remote operation support device 60. The number of cameras 40 may be one or more.

[0017] The camera 50 photographs the work site where the work robot 20 operates from a perspective different from that of the camera 40. The camera 50 is a camera that photographs a background that is hidden by overlapping with an obstruction in the camera 40 from the perspective of the remote operator. The obstruction includes, for example, the work subject that is the main body of the work performed by the work robot 20, and the wall of the work building or other equipment that blocks the field of view of the camera 40. The work subject includes the work robot 20 and a moving body that moves by the operation of the work robot 20. In the following description, it will be described assuming that the work subject is an obstruction. Also, in the following description, in the video from the perspective of the remote operator, the background that is hidden by overlapping with the work subject will be referred to as the "hidden background".

[0018] The camera 50 includes an image pickup device using a CCD, a CMOS, or the like. The camera 50 is connected to the remote operation support device 60 by wire or wirelessly. The actual video captured by the camera 50 is output to the remote operation support device 60. The number of cameras 50 may be one or a plurality. Also, the remote operation system 100 may not include the camera 50. Also, when color information is not required and only the distance information to other objects needs to be obtained, the remote operation system 100 may include a sensor for obtaining the site configuration of the work site, such as LiDAR (Light Detection And Ranging), instead of the camera 50 equipped with an image pickup device.

[0019] The remote operation support device 60 outputs the work video of the work performed by the remote operation of the work robot 20 to the display 30. The remote operation support device 60 includes a processed video generation unit 61, an output unit 62, and a virtual video generation unit 63.

[0020] The processed video generation unit 61 generates a processed video by applying DR (Diminished Reality) processing and AR (Augmented Reality) processing to the actual video from the remote operator's viewpoint. AR processing will be described later. DR processing is a process that erases or makes transparent the work subject within the area to be removed, and composites a hidden background image into the area to be removed. DR processing allows the remote operator to see the work target area that would otherwise be hidden by the overlapping work subject in the video from the remote operator's viewpoint. The work target area is the part that the work subject is working on, and is the part that the remote operator can see. In DR processing, "erasure" is a process that makes the work subject completely transparent, while "transparency" is a process that makes the work subject semi-transparent (see-through).

[0021] The details of the DR processing performed by the processed image generation unit 61 will now be explained. First, the processed image generation unit 61 observes the hidden background and models it by performing geometric processing. The processed image generation unit 61 models the hidden background from (i) images of the hidden background acquired by real-time shooting, (ii) images of the hidden background acquired by prior shooting, or (iii) images of the hidden background acquired by estimation from the actual images, or images of the hidden background generated by combining these.

[0022] Next, the processed video generation unit 61 determines the area to be removed. The area to be removed is at least a portion of the video from the remote operator's viewpoint and includes the work area where the worker performs work on the work target. The processed video generation unit 61 determines the area to be removed based on the division of the actual video using segmentation technology, the outline of the worker or the work target, site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof.

[0023] The segmentation of real video using segmentation technology is performed, for example, by a remote operator manually segmenting an arbitrary area from the real video. That is, the processing image generation unit 61 may determine the area to be removed according to the instructions of the remote operator. The contour of the work subject or work target may be a contour recognized by image processing such as CV (Computer Vision) technology. Also, if the geometric shape of the work subject is known, the contour of the work subject may be recognized based on the geometric shape of the work subject that has been registered in advance, such as a three-dimensional model. The positional relationship between the contour of the work subject and / or work target and the area to be removed that includes the work subject and / or work target may be predetermined. That is, the processing image generation unit 61 may determine the area to be removed based on the contour of the work subject and / or work target and the predetermined positional relationship.

[0024] Site configuration information may be information represented by detection results from sensors that detect the site configuration of the work site. Site configuration refers to the configuration of the work site, which is composed of various parts present at the work site, such as the main work unit, the work target area, the work building, and other equipment. Site configuration information includes, for example, information representing the positional relationship of each part present at the work site and the shape of each part. Site configuration information is preferably 3D data. LiDAR can be given as an example of a sensor that detects the site configuration. In this case, the processing image generation unit 61 determines the area to be removed based on the 3D data representing the site configuration information. Note that site configuration information may also be 1D or 2D data.

[0025] Furthermore, the processed image generation unit 61 may determine the area to be removed by combining, for example, on-site configuration information with information obtainable through image processing such as segmentation technology or CV technology, or pre-registered information. By combining various types of information in this way, the processed image generation unit 61 can determine the area to be removed at an appropriate location even in cases of poor recognition of the geometric shape of the work subject and / or work target. Poor recognition of geometric shape occurs, for example, when (i) the geometric shape of the work subject and / or work target cannot be recognized or is misrecognized by image processing, or when the pre-registered geometric shape of the work subject and / or work target differs from the actual geometric shape of the work subject and / or work target.

[0026] The processed image generation unit 61 changes the size of the removal area or moves the removal area in response to the movement of the camera 40 that captures the actual image and the movement or deformation of the work subject or work target. The movement of the camera 40 that captures the actual image and the movement of the work subject or work target may be recognized by image processing such as CV technology or by physical sensors. The deformation of the work subject or work target may be recognized by image processing such as CV technology. Preferably, the removal area includes an area larger than the work subject, that is, an area outside the area of ​​the work subject. Next, the processed image generation unit 61 erases or makes transparent the work subject within the removal area and synthesizes an image that models the hidden background within the removal area.

[0027] The processed image generation unit 61 may change the transparency of the work subject within the removal target area according to the ratio of the hidden area of ​​the work target area when making the work subject transparent. The ratio of the hidden area of ​​the work target area is the ratio of the area of ​​the work target area that is hidden by the work subject when the work subject overlaps with the work target area, relative to the area of ​​the work target area in the video from the remote operator's viewpoint. In addition, the processed image generation unit 61 may make the outline of the work subject transparent, that is, make a part of the work subject within the removal target area transparent, according to the ratio of the hidden area of ​​the work target area.

[0028] Table 1 shows an example of the transparency of the work area, set according to the percentage of the hidden area of ​​the work area. [Table 1]

[0029] Table 1's "Threshold" represents the percentage of the hidden area of ​​the work target. Table 1's "Alpha Blend" is a value that represents the transparency of the work subject within the removal target area. An Alpha Blend value of "0" indicates 0% transparency, and "1" indicates 100% transparency. That is, if the Alpha Blend value is "0", the work subject is not transparent, and if the Alpha Blend value is "1", the work subject is completely transparent. Table 1's "Contour" indicates whether or not to allow transparency to the contour portion of the work subject within the removal target area. "ON" means that the contour portion of the work subject within the removal target area is not transparent, and "OFF" means that the contour portion of the work subject within the removal target area is transparent.

[0030] For example, as shown in Table 1, the transparency of the work object within the removal area may be increased as the proportion of the hidden area of ​​the work object increases. For example, if the proportion of the hidden area of ​​the work object is 20% or more but less than 50%, the transparency of the work object should be set to 0.1 (10%), if the proportion of the hidden area of ​​the work object is 50% or more but less than 70%, the transparency of the work object should be set to 0.3 (30%), and if the proportion of the hidden area of ​​the work object is 70% or more, the transparency of the work object should be set to 0.7 (70%). Also, if the proportion of the hidden area of ​​the work object is 20% or more but less than 50%, only the outline portion of the work object within the removal area may be made opaque, while the portion near the centroid of the work object within the removal area may be made transparent. That is, if the proportion of the hidden area of ​​the work object is low, a part of the work object may be made transparent. If the proportion of the hidden area of ​​the work object is 50% or more, the outline portion of the work object within the removal area should be made transparent.

[0031] Furthermore, the processed image generation unit 61 may erase the work subject within the removal target area if the proportion of the hidden area of ​​the work target area exceeds a predetermined proportion. Also, when erasing the work subject within the removal target area, the processed image generation unit 61 may change the size of the area to which the work subject is erased according to the proportion of the hidden area of ​​the work target area. In addition, the processed image generation unit 61 may erase the work subject within the removal target area that is transparent if the proportion of the hidden area of ​​the work target area exceeds a predetermined proportion. For example, if the proportion of the hidden area of ​​the work target area is 20% or more but less than 70%, the work subject may be made transparent, and if the proportion of the hidden area of ​​the work target area is 70% or more, the work subject may be erased. Furthermore, the processed image generation unit 61 may erase a part of the work subject within the removal target area and make the unerased part of the work subject transparent. For example, the processed image generation unit 61 may erase the part near the centroid of the work subject within the target area and make the contour part of the work subject within the removal target area transparent.

[0032] The processed video generation unit 61 further performs AR processing to generate a processed video that combines a virtual image generated on the actual video from the remote operator's viewpoint or the video that has undergone DR processing. The processed video generation unit 61 combines the virtual image generated by the virtual video generation unit 63, described later, with the actual video from the remote operator's viewpoint or the video that has undergone DR processing. The processed video generation unit 61 may also combine the virtual image in the area corresponding to the area to be removed in the video that has undergone DR processing. The processed video generation unit 61 may also superimpose and combine the virtual image in the area corresponding to the area to be removed, on at least a part of the work subject, or on at least a part of the work target that is hidden by the work subject by overlapping with the work subject.

[0033] The output unit 62 outputs a first video signal representing the actual image or a second video signal representing the processed image to the display 30. Specifically, the output unit 62 outputs a first video signal representing the actual image captured by the camera 40, or a second video signal representing the processed image obtained by performing DR processing and AR processing on the actual image captured by the camera 40 to the display 30.

[0034] The virtual image generation unit 63 generates a virtual image representing information related to the work performed by the work robot 20. The virtual image generation unit 63 outputs the generated virtual image to the processing image generation unit 61. The virtual image generation unit 63 may generate a virtual image based on various information related to the work performed by the work robot 20 stored in the memory unit of the remote operation support device 60. Alternatively, the virtual image generation unit 63 may generate a virtual image based on various information related to the work performed by the work robot 20 that has been input. For example, the virtual image generation unit 63 may generate a virtual image based on various information input by a remote operator operating the input device of the remote operation support device 60. Alternatively, the virtual image generation unit 63 may generate a virtual image based on various information input from ancillary equipment connected to the work robot 20. The virtual image generation unit 63 may generate a virtual image in which the various information is represented by numbers, figures, symbols, graphs, or indicators.

[0035] The virtual image generation unit 63 may generate virtual images that graph various types of information. The virtual image generation unit 63 may also generate virtual images in a shape corresponding to the geometric shape of at least a part of the work subject or at least a part of the work target. That is, virtual images representing various types of information may be generated in a shape corresponding to the geometric shape of the work subject or work target so that the various types of information can be superimposed and displayed on the work subject or work target. For example, the virtual image generation unit 63 may determine the shape of the virtual image to be superimposed and displayed on the work subject or work target based on the contour of the work subject or work target.

[0036] The virtual image generation unit 63 may adjust the color of the displayed virtual image to make the virtual image representing various information easier to see. For example, the virtual image generation unit 63 may generate a virtual image with warm colors when the values ​​of various information are abnormal, and a virtual image with cool colors when the values ​​of various information are normal. Alternatively, the virtual image generation unit 63 may detect the color of the work subject or the work target area using image processing such as CV technology, and generate a virtual image using a color different from the color of the work subject or the work target area. The color of the virtual image may be a color input or selected by the remote operator using an input device.

[0037] [Flowchart of Remote Operation Support Method] Referring to Figure 2, an example of a remote operation support method performed by the remote operation support device 60 will be described. Figure 2 is a flowchart showing the flow of the remote operation support method performed by the remote operation support device 60.

[0038] As shown in Figure 2, in step S1, the processed video generation unit 61 acquires images captured by cameras 40 and 50. In step S1, the processed video generation unit 61 acquires real-time images captured by camera 40. In step S1, the processed video generation unit 61 may also acquire in real time images of the hidden background of the work subject captured by camera 50. If the remote control system 100 does not have camera 50, in step S1, the processed video generation unit 61 may acquire images of the hidden background of the work subject that have been previously captured by camera 40.

[0039] In step S2, the virtual image generation unit 63 generates a virtual image based on various information related to the work performed by the work robot 20. In step S2, the virtual image generation unit 63 may generate a virtual image in which the various information is represented by a graph or the like. In step S2, the virtual image generation unit 63 may also make the virtual image a shape corresponding to the geometric shape of the work subject or the work target. In step S2, the virtual image generation unit 63 may also adjust the color of the displayed virtual image to make the virtual image representing the various information easier to see. In step S2, the virtual image generation unit 63 outputs the generated virtual image to the processing image generation unit 61.

[0040] In step S3, the processed video generation unit 61 generates a processed video by performing DR processing on the actual video captured by the camera 40. In step S3, the processed video generation unit 61 observes the hidden background using the camera 40 and / or the camera 50 and models the hidden background. In step S3, the processed video generation unit 61 performs geometric processing on (i) the video of the hidden background captured in real time by the camera 50, (ii) the video of the hidden background captured in advance by the camera 40, or (iii) the video of the hidden background estimated from the background around the work subject based on the actual video captured by the camera 40, or a video generated by combining these, to model the hidden background.

[0041] In step S3, the processed video generation unit 61 determines the area to be removed based on the region division of the actual video captured by the camera 40 using segmentation technology, the outline of the work subject or work target, site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof. In step S3, the processed video generation unit 61 changes the size of the area to be removed or moves the area to be removed in accordance with the movement of the camera 40 that captures the actual video and the movement or deformation of the work subject or work target. In step S3, the processed video generation unit 61 erases or makes transparent the work subject within the area to be removed and synthesizes an image that models the hidden background within the area to be removed.

[0042] In step S3, the processed image generation unit 61 performs AR processing to combine the generated virtual image with the actual image from camera 40 or the image after DR processing. In step S3, the processed image generation unit 61 combines the virtual image generated by the virtual image generation unit 63 with the actual image from camera 40. In step S3, the processed image generation unit 61 may also combine the virtual image with the area to be removed in the image after DR processing. In step S3, the processed image generation unit 61 may perform AR processing after DR processing on the actual image from camera 40, or perform DR processing after AR processing. In step S3, the processed image generation unit 61 may superimpose the virtual image, which has been generated in a shape corresponding to the geometric shape of the work subject or work target, onto the geometric shape of at least a part of the corresponding work subject or at least a part of the work target in the area corresponding to the area to be removed.

[0043] In step S4, the output unit 62 outputs the generated processed video to the display 30.

[0044] [Example of images displayed on the screen] Referring to Figure 3, an example of the video output to the display 30 by the remote control support device 60 will be described. Figure 3 is a diagram showing an example of the video displayed on the display 30. For the sake of explanation, components having the same function as those described in the above embodiment will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0045] First, let's describe the remotely controlled work device 20A. The work device 20A shown in Figure 3 is a device for pressing a switch SW. The work device 20A is an example of a work robot 20 provided by the remote control system 100. The switch SW is an example of a work target. The work device 20A comprises a robot arm 25 and a work unit 26 (see Figure 4). The work unit 26 is provided at the tip of the robot arm 25. The work unit 26 is the part for pressing the switch SW. The remote operator remotely controls the work device 20A while checking the image displayed on the display 30, and presses the switch SW with the work unit 26. The robot arm 25 and the work unit 26 are the main components of the work.

[0046] A camera 40 is installed at a distance from the work device 20A to capture video footage of the work device 20A in operation. Camera 40 is a fixed-point camera whose shooting position does not move. In this example, a camera 50 for capturing the hidden background is not installed. In the processed video displayed on the display 30, the hidden background is observed using video footage of the hidden background previously captured by camera 40.

[0047] Next, the video output to the display 30 by the remote control support device 60 will be described. Reference numeral 300 in Figure 3 indicates a processed video in which only DR processing has been applied to the actual video from the camera 40. Reference numeral 301 in Figure 3 indicates a processed video in which both DR processing and AR processing have been applied to the actual video from the camera 40.

[0048] As shown in Figure 3, the removal target area R1 is determined based on the contour of the work device 20A and the contour of the switch SW. Since the geometric shape of the work device 20A is known, the contour of the work device 20A is recognized based on the geometric shape of the work device 20A that has been registered in advance. The contour of the switch SW may be recognized by image processing or by the geometric shape of the switch SW that has been registered in advance. The removal target area R1 moves in accordance with the movement of the work unit 26. The size of the removal target area R1 changes according to the size of the work unit 26, which changes with the movement in the perspective direction relative to the camera 40.

[0049] As shown by reference numeral 301 in Figure 3, a virtual image 260 is displayed on the work device 20A in the area R1 to be removed. In this embodiment, the virtual image 260 is displayed on the contour portion of the tip of the work unit 26 within the area of ​​the work device 20A included in the area R1 to be removed, and represents the shape of the tip of the work unit 26. As shown by reference numeral 300 in Figure 3, in the case of a processed image where only DR processing has been performed, the work unit 26 is transparent, making it difficult for the remote operator to see the work unit 26. As shown by reference numeral 301 in Figure 3, in the case of a processed image where both DR processing and AR processing have been performed, the virtual image 260 is displayed, making it easy for the remote operator to see the work unit 26.

[0050] Furthermore, since the processed video undergoes DR and AR processing, the area where the virtual image is displayed will differ significantly from the actual image for the remote operator. Therefore, considering the increased cognitive load on the remote operator, the virtual image may be displayed only on the part considered most important in the task. In the example shown in Figure 3, the virtual image 260 is displayed only on the tip of the work section 26, which is considered most important in the task. This reduces the cognitive load on the remote operator.

[0051] In Figure 3, the processed image shown by reference numeral 301 combines the virtual image 260 with the processed image that has undergone DR processing. Therefore, the virtual image 260 is displayed without transparency in the removal target area R1. Alternatively, DR processing may be performed on the image obtained by combining the virtual image 260 with the actual image from camera 40. In this case, the virtual image 260 is displayed transparently in the removal target area R1.

[0052] Generally, in remote operation, if there are areas that are difficult for the remote operator to see on the display screen from their perspective, this can be addressed by installing additional cameras to capture those areas. However, as the number of cameras increases, the remote operator needs to select the appropriate screen as needed, and if they cannot select the appropriate screen, work efficiency decreases. Also, in some work sites, it is difficult to install multiple cameras, so it is desirable to be able to acquire various information from a single camera.

[0053] Furthermore, it is generally known that in remote operation, the image displayed on the screen is two-dimensional, making it difficult to grasp three-dimensional space using only the display image, thus reducing work efficiency. Therefore, in order for inexperienced remote operators to perform remote work efficiently, it is necessary to display images from different viewpoints on at least two displays. If three-dimensional images are displayed on the screen, it becomes possible to grasp three-dimensional space with just one image. However, there are problems such as motion sickness, making it difficult for remote operators to work for long periods of time.

[0054] According to the remote operation support device 60 described above, DR processing makes the work target area, which is not visible to the operator, visible, and AR processing allows for the additional display of work information. As a result, the remote operator can obtain various information from a single video. This improves the work efficiency of the remote operator.

[0055] Furthermore, with a configuration that superimposes a virtual image onto the area corresponding to the area to be removed, the virtual image is superimposed onto the area to be removed, which the remote operator focuses on most during the operation. This reduces the amount of eye movement required by the remote operator to check the virtual image. As a result, the remote operator can more easily check the work information.

[0056] Furthermore, in a configuration that displays a virtual image corresponding to the shape of the work subject or work target, the virtual image corresponding to the geometric shape of the work subject or work target that has been erased or made transparent by DR processing is displayed in the processed image. This makes it easier for the remote operator to recognize which part of the information the virtual image displayed in the processed image represents. [Embodiment 2] Other embodiments of this disclosure will be described below with reference to Figures 4 to 8. For the sake of clarity, components having the same function as those described in the above embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0057] Referring to Figure 4, the remote control system 100A according to this embodiment will be described. Figure 4 is a block diagram showing an example of the remote control system 100A. The remote control system 100A according to Embodiment 2 further includes a state sensor 70 and an environmental sensor 80 in addition to the configuration of the remote control system 100 described above.

[0058] The state sensor 70 detects the state of the work subject or the work target. The state sensor 70 detects parameters that represent the state of the work subject or work target, which are necessary for the work performed by the work robot 20. The state sensor 70 is a sensor that detects, for example, temperature, acceleration, angular velocity, current, voltage, time, or pressure. The state sensor 70 is connected to the remote control support device 60 by wire or wireless connection and outputs the detection results to the remote control support device 60.

[0059] The environmental sensor 80 detects the state of the surrounding environment of the work subject or the work target. The environmental sensor 80 detects parameters representing the surrounding environment of the work subject or the work target that are necessary for the work performed by the work robot 20. The environmental sensor 80 is a sensor that detects, for example, temperature, humidity, or atmospheric pressure. The environmental sensor 80 is connected to the remote control support device 60 by wire or wireless connection and outputs the detection results to the remote control support device 60.

[0060] In this embodiment, the virtual image generation unit 63 may generate a virtual image representing the detection results of the state sensor 70 and / or the environmental sensor 80. The virtual image generation unit 63 acquires the detection results output from the state sensor 70 and / or the environmental sensor 80. The virtual image generation unit 63 may generate a virtual image that graphs the detection results of the state sensor 70 and / or the environmental sensor 80. The virtual image generation unit 63 may also make the virtual image representing the detection results of the state sensor 70 and / or the environmental sensor 80 correspond to the geometric shape of the work subject or the work target. The virtual image generation unit 63 may also generate a virtual image representing the simulation results obtained by simulating the work using a computer. The computer simulation is performed based on the operating status of the work robot 20, the detection results of the state sensor 70 and / or the environmental sensor 80, or information regarding the operation of ancillary equipment, etc.

[0061] [Example of images displayed on the screen] Referring to Figures 5 to 8, an example of the video output to the display 30 by the remote control support device 60 will be explained.

[0062] First, the remotely operated rolling mill 20B will be described with reference to Figure 5. Figure 5 is a schematic diagram showing an overview of the rolling mill 20B. As shown in Figure 5, the rolling mill 20B hot-rolls rolled material 250 such as steel. The rolling mill 20B is equipped with a plurality of rolling mills 200. Each rolling mill 200 has work rollers 202 positioned to sandwich the rolled material 250 from above and below, and backup rollers 201 supporting each work roller 202. The rolled material 250 sandwiched by the work rollers 202 is pressed and rolled by each work roller 202.

[0063] In the conveying direction D of the rolled material 250, a looper 210 is positioned between the rolling mills 200 to maintain the rolled material 250 at an appropriate angle and tension. The looper 210 is controlled by a controller (not shown) and adjusts the angle and tension of the rolled material 250 by rotating around its axis.

[0064] The rolling mill 20B is equipped with a cooling device 220 for cooling the rolled material 250. The cooling device 220 cools the rolled material 250 by spraying cooling water or the like onto the rolled material 250. The rolling mill 20B is equipped with a temperature sensor 230. The temperature sensor 230 is a sensor that detects the temperature distribution of the rolled material 250 in the width direction. The rolling mill 20B is equipped with a meandering sensor 240. The meandering sensor 240 is a sensor that detects the rolled material 250's tendency to meander in the width direction. The temperature sensor 230 and the meandering sensor 240 are examples of state sensors.

[0065] For example, in the finish rolling process of a hot rolling mill, the rolled material 250 is continuously rolled using multiple rolling mills 200 so that the dimensions of the rolled material 250 after the finish rolling process, such as the thickness or width of the plate, reach the desired values. In addition, the rolled material 250 is cooled using a cooling device 220 so that it reaches the desired temperature. Furthermore, if there is a large variation in the temperature of the rolled material 250 in the width direction, the rolled material 250 will meander. If the amount of meandering of the rolled material 250 becomes large, it will interfere with other parts and hinder the operation of the rolling mill 20B. The remote operator checks the detection results of the temperature sensor 230 and the meandering sensor 240 and remotely operates the rolling mill 20B to ensure smooth rolling operations.

[0066] Next, with reference to Figures 6 to 8, the video output to the display 30 by the remote control support device 60 will be described. Figures 6 to 8 are examples of video displayed on the display 30.

[0067] In Figure 6, reference numeral 600 indicates the actual image captured by camera 40. Camera 40 is capturing area X (see Figure 5) where the cooling device 220 is cooling the rolled material 250. The cooling water Y sprayed from the cooling device 220 is the work object, and the rolled material 250 to which the cooling water Y is sprayed is the work target. As shown by reference numeral 600, the remote operator has difficulty seeing the rolled material 250 from the actual image captured by camera 40 due to the cooling water Y sprayed from the cooling device 220. Reference numeral 601 in Figure 6 indicates a processed image in which only DR processing has been performed to remove the cooling water Y from the actual image captured by camera 40. As shown by reference numeral 601, the remote operator can see the rolled material 250 from the processed image after DR processing because the cooling water Y has been removed. The area to be removed is determined relative to the area to which the cooling water Y is sprayed, and in this embodiment, it is determined to be the same area as area X.

[0068] As shown in Figure 7, the remote control support device 60 performs AR processing on the processed image after DR processing to erase the cooling water Y. As shown by reference numeral 700 in Figure 7, the remote control support device 60 may superimpose a virtual image 250A corresponding to the shape of the rolled material 250 onto the rolled material 250 and display it on the display 30. The virtual image 250A is an image representing the tension of the rolled material 250 detected by the looper 210. The virtual image 250A changes color according to the tension value. A bar plot 251 is displayed on the display 30, and the color of the virtual image 250A changes to the color associated with the bar plot 251 according to the tension value.

[0069] As shown by reference numeral 701 in Figure 7, the remote control support device 60 may display a virtual image 260, which corresponds to the shape of a part of the rolled material 250, superimposed on a part of the rolled material 250 on the display 30. The virtual image 260 is displayed superimposed on the virtual image 250A. The virtual image 260 is an image representing the temperature distribution of the rolled material 250 detected by the temperature sensor 230. The virtual image 260 changes color according to the temperature. A bar plot 261 is displayed on the display 30, and the color of the virtual image 260 changes to the color associated with the bar plot 261 according to the temperature.

[0070] As shown by reference numeral 702 in Figure 7, the remote control support device 60 may display a virtual image 270, which corresponds to the shape of a part of the rolled material 250, superimposed on a part of the rolled material 250 on the display 30. The virtual image 270 is displayed superimposed on the virtual image 250A. The virtual image 270 is an image that represents the degree to which the rolled material 250 is flat in the width direction, as detected by the meandering sensor 240. The virtual image 270 changes color according to the degree of flatness. A bar plot 271 is displayed on the display 30, and the color of the virtual image 270 changes to the color associated with the bar plot 271 according to the degree of flatness.

[0071] The remote control support device 60 may also display a virtual image on the display 30, which is a graph of the values ​​detected by each part. As shown in Figure 8, the remote control support device 60 may display on the display 30 a graph 800 showing the tension value of the rolled material 250 detected by the looper 210 and the change in the angle of the rolled material 250 over time. The remote control support device 60 may also display on the display 30 a graph 801 showing the temperature distribution of the rolled material 250 detected by the temperature sensor 230. The remote control support device 60 may also display on the display 30 a graph 802 showing the degree of plate-like deformation (meandering) of the rolled material 250 detected by the meandering sensor 240. The remote control support device 60 may display each graph 800, 801, and 802 superimposed on the rolled material 250, or it may be displayed in a position within the removal target area that does not overlap with the main work unit or the work target part.

[0072] The remote control support device 60 may display a processed image on the display 30 that is a composite of virtual images representing the generated simulation results. The remote control support device 60 may also display actual numerical values ​​as virtual images on the display 30. Furthermore, the remote control support device 60 may display information such as the sum load, which is the total load of the rolled material 250 in the transport direction D, the differential load, which is the difference in load of the rolled material 250 in the transport direction D, the elongation difference ratio of the rolled material 250, or the reduction ratio as virtual images on the display 30. The remote control support device 60 may display these virtual images superimposed on the rolled material 250, or they may be displayed in a position within the removal target area that does not overlap with the main work unit or the work target part. Furthermore, the remote control support device 60 may display a color selected by the remote operator using an input device, along with the virtual image, on the display 30.

[0073] With the above configuration, the remote operator does not need to check a separate display that shows the detection results of the looper 210, temperature sensor 230, and meandering sensor 240, or a separate display that shows the simulation results, in addition to the display 30 that shows the work video. In other words, the work and work monitoring can be done solely from the processed video displayed on the display 30. This makes it possible to reduce work-related problems.

[0074] [Examples of implementation using software] The function of the remote control support device 60 (hereinafter referred to as "the device") is a program that causes the device to function as a computer, and can be realized by a program that causes the computer to function as each control block of the device.

[0075] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0076] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0077] 〔summary〕 A remote operation support device according to Embodiment 1 of the present disclosure is a remote operation support device that outputs work video of work performed by remote operation to a display, comprising: a virtual video generation unit that generates a virtual video representing information related to the work; a processed video generation unit that generates a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of a remote operator, which erases or makes transparent occluding objects in an area to be removed, and synthesizes an image that models the background hidden by overlapping with the occluding objects in the area to be removed, and AR (Augmented Reality) processing which synthesizes the virtual video generated on the real video or the video on which the DR processing has been performed; and an output unit that outputs the processed video to the display.

[0078] According to the above embodiment 1, DR processing makes objects that are not visible due to obstructions visible, and AR processing allows for the additional display of work information. As a result, the remote operator can obtain various information from a single video. This improves the work efficiency of the remote operator.

[0079] In the remote control support device according to Embodiment 2 of the present disclosure, in Embodiment 1 described above, the processed video generation unit may synthesize the generated virtual video into the region corresponding to the region to be removed in the actual video or the video on which the DR processing has been performed.

[0080] According to the above embodiment 2, since a virtual image is superimposed on the area to be removed, which the remote operator focuses on most intently during the operation, the amount of eye movement required by the remote operator to check the virtual image can be reduced. This makes it easier for the remote operator to check the work information.

[0081] In the remote control support device according to embodiment 3 of the present disclosure, in embodiment 2 described above, the virtual image generation unit generates the virtual image in a shape corresponding to the geometric shape of at least a part of the occluding object, or at least a part of the target object that overlaps with the occluding object and is hidden by the occluding object, and the processing image generation unit generates the processed image by superimposing the virtual image, which was generated in a shape corresponding to the geometric shape, onto the geometric shape of at least a part of the occluding object or at least a part of the target object in the region corresponding to the removal target area.

[0082] According to embodiment 3 described above, a virtual image corresponding to the geometric shape of the occluding object or target part that has been erased or made transparent by DR processing is displayed in the processed image. This makes it easier for the remote operator to recognize which part of the image the virtual image displayed in the processed image represents.

[0083] In the remote operation support device according to Embodiment 4 of the present disclosure, in any of Embodiments 1 to 3 above, the virtual image generation unit may generate information representing the detection result of a state sensor that detects the state of the occluding object or a target part that is hidden by the occluding object by overlapping with the occluding object, or an environmental sensor that detects the state of the surrounding environment of the occluding object or the target part, or a virtual image representing the simulation result obtained by simulating the operation using a computer.

[0084] According to embodiment 4 described above, the remote operator does not need to check a separate display that shows the sensor detection results or the simulation results, which are provided separately from the display that shows the work video. In other words, the work and work monitoring can be done solely from the processed video displayed on the display. This makes it possible to reduce work-related problems.

[0085] In any of the embodiments 1 to 4 described above, the remote operation support device according to embodiment 5 of the present disclosure may, in the processing image generation unit, model the background image obtained by real-time shooting, the background image obtained by prior shooting, the background image obtained by estimation from the actual image, or the background image generated by combining these, and synthesize it with the area to be removed. The area to be removed may be determined based on the area division of the actual image using segmentation technology, the outline of the obstruction or the target part that is the target of the remote operator's view, or site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof.

[0086] A remote operation support method according to aspect 6 of the present disclosure is a remote operation support method performed by a remote operation support device that outputs work video of work performed by remote operation to a display, and includes: a virtual video generation step of generating a virtual video representing information about the work; a processed video generation step of generating a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of a remote operator, which erases or makes transparent occluding objects in an area to be removed, and synthesizes an image that models the background hidden by overlapping with the occluding objects in the area to be removed, and AR (Augmented Reality) processing which synthesizes the virtual video generated on the real video or the video on which the DR processing has been performed; and an output step of outputting the processed video to the display.

[0087] This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure. [Explanation of Symbols]

[0088] 10 Remote control device 20 Industrial Robots 30 displays 40 Cameras 50 Cameras 60 Remote control support device 61 Processing and Video Generation Unit 62 Output section 63 Virtual Image Generation Unit 100 Remote Control Systems

Claims

1. A remote operation support device that outputs video of work performed by remote operation to a display, A virtual image generation unit that generates a virtual image representing information related to the aforementioned work, A processing video generation unit generates a processed video by performing DR (Diminished Reality) processing on a real video from the remote operator's point of view, which erases or makes transparent obstacles within the area to be removed, and composites it with a video that models the background that is hidden by overlapping with the obstacles within the area to be removed, and AR (Augmented Reality) processing which composites the virtual video generated on the real video or the video after the DR processing, The system includes an output unit that outputs the processed video to the display, Remote control support device.

2. The remote control support device according to claim 1, wherein the processed image generation unit synthesizes the generated virtual image with the region corresponding to the region to be removed in the actual image or the image on which the DR processing has been performed.

3. The virtual image generation unit generates the virtual image in a shape corresponding to the geometric shape of at least a part of the occluding object, or at least a part of the target object that overlaps with the occluding object and is hidden by the occluding object. The processing image generation unit generates the processed image by superimposing the virtual image, which is generated in a shape corresponding to the geometric shape in the region corresponding to the area to be removed, onto the geometric shape of at least a part of the corresponding occluding object or at least a part of the target area. The remote control support device according to claim 2.

4. The remote operation support device according to any one of claims 1 to 3, wherein the virtual image generation unit generates a virtual image representing information representing the detection result of a state sensor that detects the state of the occluding object or a target part that is hidden by the occluding object by overlapping with the occluding object, or an environmental sensor that detects the state of the surrounding environment of the occluding object or the target part, or a simulation result obtained by simulating the operation using a computer.

5. The aforementioned processed video generation unit, The background image obtained by real-time shooting, the background image obtained by prior shooting, the background image obtained by estimation from the actual image, or the background image generated by combining these, is modeled and composited onto the area to be removed. A remote operation support device according to any one of claims 1 to 3, which determines the area to be removed based on the division of the actual image using segmentation technology, the outline of the obstruction or the target part that is to be seen by the remote operator, or site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof.

6. A remote operation support method performed by a remote operation support device that outputs video of work performed by remote operation to a display, A virtual image generation step that generates a virtual image representing information related to the aforementioned work, A process for generating a processed image involves performing a DR (Diminished Reality) process on a real video from the remote operator's perspective, which removes or makes transparent obstacles within the area to be removed, and then compositing it with an image that models the background that is hidden by overlapping with the obstacles within the area to be removed; and an AR (Augmented Reality) process which combines the virtual image generated with the real video or the image after the DR process to generate a processed image. The process includes an output step of outputting the processed video to the display, Remote operation support method.

Images