Remote control support device and remote control support method

Abstract

To improve the work efficiency of remote operators. [Solution] The remote operation support device (60) includes a processed video generation unit (61) that generates a processed video by performing DR (Diminished Reality) processing on the actual video from the remote operator's viewpoint, an output unit that outputs a first video signal representing the actual video or a second video signal representing the processed video to a display (30), and a video switching unit (63) that switches the video signal output to the display (30) to the first video signal or the second video signal.

Description

Technical Field

[0001] The present disclosure relates to a remote operation support device and a remote operation support method.

Background Art

[0002] Conventionally, techniques for performing various processes on real-time images captured by a camera or the like are known. As one of such techniques, DR (Diminished Reality) technology is known. The DR technology makes a target part located in the hidden background of a shielding object visible by erasing or transmitting the shielding object that shields the target part.

[0003] For example, Patent Document 1 describes a crane operation assist device capable of visualizing a portion that is blocked by a spreader and becomes a blind spot. Patent Document 1 describes a technique of generating a composite image by synthesizing an enlarged image and a reference image captured by a camera, and synthesizing by transmitting a virtual image of the spreader through the composite image.

[0004] Further, Patent Document 2 describes a technique of generating an image along a line of sight in the direction of the vehicle from the operator, including a blind spot position located on the opposite side of the operator with respect to the vehicle, as an image for remote parking. In the image for remote parking, the automobile to be parked is displayed as a see-through image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Incidentally, it is known that remotely operating various work devices while visually observing work images displayed on a screen results in lower work efficiency compared to actually operating the equipment on-site. One reason for this decrease in work efficiency is the error (shift) in the hidden background of the processed image when processed using DR technology is displayed on a screen. It is known that even if the error in the hidden background is slight, remote operators may still have doubts about the hidden background in processed images using DR technology, which prevents improvement in work efficiency.

[0007] One aspect of this disclosure aims to improve the work efficiency of remote operators. [Means for solving the problem]

[0008] To solve the above problems, a remote operation support device according to one aspect of the present disclosure outputs a work video of a task performed by remote operation to a display, and comprises: a processed video generation unit that generates a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of the remote operator, which erases or makes transparent an obstruction in the area to be removed, and synthesizes a video that models the background that is hidden by overlapping with the obstruction in the area to be removed; an output unit that outputs a first video signal representing the real video or a second video signal representing the processed video to the display; and a video switching unit that switches the video signal output to the display to the first video signal or the second video signal.

[0009] To solve the above problems, a remote operation support method according to one aspect of the present disclosure is a remote operation support method performed by a remote operation support device that outputs a work video of a work performed by remote operation to a display, and includes: a processed video generation step that generates a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of the remote operator, which erases or makes transparent an obstruction in the area to be removed, and synthesizes a video that models the background that is hidden by overlapping with the obstruction in the area to be removed; an output step that outputs a first video signal representing the real video or a second video signal representing the processed video to the display; and a video switching step that switches the video signal output to the display to the first video signal or the second video signal. [Effects of the Invention]

[0010] According to one aspect of this disclosure, the work efficiency of the remote operator can be improved. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example of a remote control system. [Figure 2] This is a flowchart showing the flow of the remote control support method performed by the remote control support device. [Figure 3] This is a schematic diagram showing the overview of an overhead crane system. [Figure 4] This figure shows an example of an image displayed on a screen. [Figure 5] This figure shows an example of an image displayed on a screen. [Figure 6] This figure shows an example of an image displayed on a screen. [Figure 7] This figure shows an example of an image displayed on a screen. [Figure 8] This figure shows an example of a processed image displayed on a screen. [Modes for carrying out the invention]

[0012] [Embodiment 1] One embodiment of this disclosure will be described in detail below.

[0013] [Overview of the remote control system] Referring to Figure 1, an overview of the remote control system 100 will be described. Figure 1 is a block diagram showing an example of the remote control system 100. The remote control system 100 is a system for a remote operator to remotely control a work device using a remote control device 10. As shown in Figure 1, the remote control system 100 comprises a remote control device 10, a work robot 20, a display 30, a camera 40, and a remote control support device 60. The remote control system 100 may further include a camera 50.

[0014] The remote control device 10 is positioned away from the work robot 20 and is operated by a remote operator. The remote operator operates the remote control device 10 while viewing the work video displayed on the display 30. The remote control device 10 is connected to the work robot 20 via wired or wireless communication.

[0015] The work robot 20 is a robot that is remotely controlled by a remote control device 10. The work robot 20 is an example of a work device that performs predetermined processing on a work object, such as physical or chemical processing, by moving, changing shape, driving joints, or performing various functions through remote control. The work robot 20 is a robot that performs tasks in fields such as industry or medicine. Industries include agriculture, forestry and fisheries, manufacturing, construction, and mining.

[0016] The display 30 is arranged at a position visible to the remote operator. The display 30 is connected to the remote operation support device 60. The display 30 displays an image based on the video signal output from the remote operation support device 60. An image from the perspective of the remote operator is displayed on the display 30. That is, an image from a perspective suitable for the remote operator to perform work is displayed on the display 30. An image captured by the camera 40 (described later) is displayed on the display 30. The number of displays 30 may be one or more than one.

[0017] The camera 40 captures the work site where the work robot 20 is performing work. The camera 40 captures an image from the perspective of the remote operator as the work image to be displayed on the display 30. The camera 40 includes an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 40 is connected to the remote operation support device 60 by wire or wirelessly. The actual image captured by the camera 40 is output to the remote operation support device 60. The number of cameras 40 may be one or more than one.

[0018] The camera 50 captures the work site where the work robot 20 is performing work from a perspective different from that of the camera 40. The camera 50 is a camera that captures the background that will be hidden by overlapping with an obstacle in the camera 40 from the perspective of the remote operator. The obstacle includes, for example, the work object that is the main body of the work performed by the work robot 20, and the wall of the work building or other equipment that blocks the field of view of the camera 40. The work object includes the work robot 20 and the moving body that moves due to the operation of the work robot 20. In the following description, the work object is described as an obstacle. Also, in the following description, in the image from the perspective of the remote operator, the background that is hidden by overlapping with the work object is referred to as the "hidden background".

[0019] Camera 50 is equipped with an image sensor such as a CCD or CMOS. Camera 50 is connected to the remote control support device 60 by wire or wireless connection. The actual video captured by camera 50 is output to the remote control support device 60. There may be one camera 50 or multiple cameras 50. The remote control system 100 does not need to be equipped with cameras 50. Also, if color information is not required and it is sufficient to simply acquire distance information to other objects, the remote control system 100 may be equipped with a sensor for acquiring the site configuration of the work site, such as LiDAR (Light Detection And Ranging), instead of a camera 50 equipped with an image sensor.

[0020] The remote control support device 60 outputs video footage of the work performed by the remote control of the work robot 20 to the display 30. The remote control support device 60 comprises a processing video generation unit 61, an output unit 62, and a video switching unit 63. The remote control support device 60 may further include a determination unit 64.

[0021] The processed video generation unit 61 generates a processed video by performing DR (Diminished Reality) processing on the actual video from the remote operator's perspective. DR processing is a process that erases or makes transparent the work subject within the area to be removed, and composites a hidden background image into the area to be removed. Through DR processing, the remote operator can see the work target area that would otherwise be hidden by the work subject in the video from the remote operator's perspective. The work target area is the part that is the target of work performed by the work subject, and is the part that the remote operator can see. In DR processing, "erasure" is a process that makes the work subject completely transparent, and "transparency" is a process that makes the work subject semi-transparent (see-through).

[0022] The details of the DR processing performed by the processed image generation unit 61 will now be explained. First, the processed image generation unit 61 observes the hidden background and models it by performing geometric processing. The processed image generation unit 61 models the hidden background from (i) images of the hidden background acquired by real-time shooting, (ii) images of the hidden background acquired by prior shooting, or (iii) images of the hidden background acquired by estimation from the actual images, or images of the hidden background generated by combining these.

[0023] Next, the processed video generation unit 61 determines the area to be removed. The area to be removed is at least a portion of the video from the remote operator's viewpoint and includes the work area where the worker performs work on the work target. The processed video generation unit 61 determines the area to be removed based on the division of the actual video using segmentation technology, the outline of the worker or the work target, site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof.

[0024] The segmentation of real video using segmentation technology is performed, for example, by a remote operator manually segmenting an arbitrary area from the real video. That is, the processing image generation unit 61 may determine the area to be removed according to the instructions of the remote operator. The contour of the work subject or work target may be a contour recognized by image processing such as CV (Computer Vision) technology. Also, if the geometric shape of the work subject is known, the contour of the work subject may be recognized based on the geometric shape of the work subject that has been registered in advance, such as a three-dimensional model. The positional relationship between the contour of the work subject and / or work target and the area to be removed that includes the work subject and / or work target may be predetermined. That is, the processing image generation unit 61 may determine the area to be removed based on the contour of the work subject and / or work target and the predetermined positional relationship.

[0025] Site configuration information may be information represented by detection results from sensors that detect the site configuration of the work site. Site configuration refers to the configuration of the work site, which is composed of various parts present at the work site, such as the main work unit, the work target area, the work building, and other equipment. Site configuration information includes, for example, information representing the positional relationship of each part present at the work site and the shape of each part. Site configuration information is preferably 3D data. LiDAR can be given as an example of a sensor that detects the site configuration. In this case, the processing image generation unit 61 determines the area to be removed based on the 3D data representing the site configuration information. Note that site configuration information may also be 1D or 2D data.

[0026] Furthermore, the processed image generation unit 61 may determine the area to be removed by combining, for example, on-site configuration information with information obtainable through image processing such as segmentation technology or CV technology, or pre-registered information. By combining various types of information in this way, the processed image generation unit 61 can determine the area to be removed at an appropriate location even in cases of poor recognition of the geometric shape of the work subject and / or work target. Poor recognition of geometric shape occurs, for example, when (i) the geometric shape of the work subject and / or work target cannot be recognized or is misrecognized by image processing, or when the pre-registered geometric shape of the work subject and / or work target differs from the actual geometric shape of the work subject and / or work target.

[0027] The processed image generation unit 61 changes the size of the removal area or moves the removal area in response to the movement of the camera 40 that captures the actual image and the movement or deformation of the work subject or work target. The movement of the camera 40 that captures the actual image and the movement of the work subject or work target may be recognized by image processing such as CV technology or by physical sensors. The deformation of the work subject or work target may be recognized by image processing such as CV technology. Preferably, the removal area includes an area larger than the work subject, that is, an area outside the area of ​​the work subject.

[0028] Next, the processed image generation unit 61 erases or makes transparent the work subject within the removal target area and synthesizes an image that models the hidden background within the removal target area. When the processed image generation unit 61 makes transparent the work subject within the removal target area, it may change the transparency of the work subject within the removal target area according to the ratio of the hidden area of ​​the work target area. The ratio of the hidden area of ​​the work target area is the ratio of the area of ​​the work target area that is hidden by the work subject when the work subject overlaps with the work target area, relative to the area of ​​the work target area in the image from the remote operator's viewpoint. In addition, the processed image generation unit 61 may make the outline portion of the work subject transparent, that is, make a part of the work subject within the removal target area transparent, according to the ratio of the hidden area of ​​the work target area.

[0029] Table 1 shows an example of the transparency of the work area, set according to the percentage of the hidden area of ​​the work area. [Table 1]

[0030] Table 1's "Alpha Blend" value represents the transparency of the work object within the removal area. An Alpha Blend value of "0" indicates 0% transparency, and "1" indicates 100% transparency. That is, an Alpha Blend value of "0" means the work object is not transparent, and an Alpha Blend value of "1" means the work object is completely transparent. Table 1's "Contour" value indicates whether or not the contour portion of the work object within the removal area is transparent. "ON" means the contour portion of the work object within the removal area is not transparent, and "OFF" means the contour portion of the work object within the removal area is transparent.

[0031] As shown in Table 1, if the percentage of the hidden area of ​​the work area is less than 20%, DR processing is not performed (not performed), and if the percentage of the hidden area of ​​the work area is 20% or more, DR processing may be performed. For example, as shown in Table 1, the transparency of the work object within the removal target area may be increased as the percentage of the hidden area of ​​the work area increases. For example, if the percentage of the hidden area of ​​the work area is 20% or more but less than 50%, the transparency of the work object should be set to 0.1 (10%), if the percentage of the hidden area of ​​the work area is 50% or more but less than 70%, the transparency of the work object should be set to 0.3 (30%), and if the percentage of the hidden area of ​​the work area is 70% or more, the transparency of the work object should be set to 0.7 (70%). In addition, if the percentage of the hidden area of ​​the work area is 20% or more but less than 50%, only the outline portion of the work object within the removal target area may not be transparent, while the portion near the centroid of the work object within the removal target area may be transparent. That is, if the percentage of the hidden area of ​​the work area is low, a portion of the work object may be made transparent. If the hidden area of ​​the work area is 50% or more, the outline of the work object within the removal area will be made transparent.

[0032] Furthermore, the processed image generation unit 61 may erase the work subject within the removal target area if the proportion of the hidden area of ​​the work target area exceeds a predetermined proportion. Also, when erasing the work subject within the removal target area, the processed image generation unit 61 may change the size of the area to which the work subject is erased according to the proportion of the hidden area of ​​the work target area. In addition, the processed image generation unit 61 may erase the work subject within the removal target area that is transparent if the proportion of the hidden area of ​​the work target area exceeds a predetermined proportion. For example, if the proportion of the hidden area of ​​the work target area is 20% or more but less than 70%, the work subject may be made transparent, and if the proportion of the hidden area of ​​the work target area is 70% or more, the work subject may be erased. Furthermore, the processed image generation unit 61 may erase a part of the work subject within the removal target area and make the unerased part of the work subject transparent. For example, the processed image generation unit 61 may erase the part near the centroid of the work subject within the target area and make the contour part of the work subject within the removal target area transparent.

[0033] The output unit 62 outputs a first video signal representing the actual image or a second video signal representing the processed image to the display 30. Specifically, the output unit 62 outputs a first video signal representing the actual image captured by the camera 40, or a second video signal representing the processed image obtained by applying DR processing to the actual image captured by the camera 40, to the display 30.

[0034] The video switching unit 63 switches the video signal output to the display 30 to either the first video signal or the second video signal. The video switching unit 63 switches the video signal output by the output unit 62 to the display 30 to either the first video signal or the second video signal. The video switching unit 63 may also switch the video signal output to the display 30 to either the first video signal or the second video signal depending on the determination result of the determination unit 64.

[0035] The determination unit 64 determines whether or not to output a second video signal to the display 30. The determination unit 64 may also determine to output a second video signal to the display 30 if the percentage of the hidden area of ​​the work target in the video from the remote operator's viewpoint is equal to or greater than a predetermined threshold. The percentage of the hidden area of ​​the work target may be calculated, for example, based on the outline of the work subject detected based on the video captured by the camera 40, and the outline of the work target.

[0036] Furthermore, the determination unit 64 may determine whether or not to output a second video signal to the display 30 based on input from the remote operator. For example, if the remote operator operates an input button to display a processed video on the display 30, the determination unit 64 may determine to output a second video signal. Alternatively, if the remote operator inputs to display a real video on the display 30, the determination unit 64 may determine not to output a second video signal. Note that if the remote operation support device 60 switches the video displayed on the display 30 only by manual input from the remote operator, it does not need to be equipped with a determination unit 64.

[0037] [Flowchart of Remote Operation Support Method] Referring to Figure 2, an example of a remote operation support method performed by the remote operation support device 60 will be described. Figure 2 is a flowchart showing the flow of the remote operation support method performed by the remote operation support device 60.

[0038] As shown in Figure 2, in step S1, the processed video generation unit 61 acquires images captured by cameras 40 and 50. In step S1, the processed video generation unit 61 acquires real-time images captured by camera 40. In step S1, the processed video generation unit 61 may also acquire in real time images of the hidden background of the work subject captured by camera 50. If the remote control system 100 does not have camera 50, in step S1, the processed video generation unit 61 may acquire images of the hidden background of the work subject that have been previously captured by camera 40.

[0039] In step S2, the processed video generation unit 61 generates a processed video by performing DR processing on the actual video captured by the camera 40. In step S2, the processed video generation unit 61 observes the hidden background using the camera 40 and / or the camera 50 and models the hidden background. In step S2, the processed video generation unit 61 performs geometric processing on (i) the video of the hidden background captured in real time by the camera 50, (ii) the video of the hidden background captured in advance by the camera 40, or (iii) the video of the hidden background estimated from the background around the work subject based on the actual video captured by the camera 40, or a video generated by combining these, to model the hidden background.

[0040] In step S2, the processed video generation unit 61 determines the area to be removed based on the region division of the actual video captured by the camera 40 using segmentation technology, the outline of the work subject or work target, site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof. In step S2, the processed video generation unit 61 changes the size of the area to be removed or moves the area to be removed in accordance with the movement of the camera 40 that captures the actual video and the movement or deformation of the work subject or work target.

[0041] In step S2, the processed image generation unit 61 erases or makes transparent the work subject within the area to be removed, and synthesizes an image that models the hidden background within the area to be removed. In step S2, the processed image generation unit 61 may change the transparency of the work subject within the area to be removed according to the proportion of the hidden area of ​​the work target. In addition, in step S2, the processed image generation unit 61 may make the outline portion of the work subject transparent, that is, make a part of the work subject within the area to be removed transparent, according to the proportion of the hidden area of ​​the work target. In addition, in step S2, the processed image generation unit 61 may erase the work subject within the area to be removed according to the proportion of the hidden area of ​​the work target.

[0042] In step S3, the determination unit 64 determines whether or not to output a second video signal to the display 30. In step S3, the determination unit 64 may determine whether or not to output a second video signal to the display 30 based on the ratio of the hidden area of ​​the work target. Alternatively, in step S3, the determination unit 64 may determine whether or not to output a second video signal based on whether or not an input has been received from a remote operator to display the processed video on the display 30.

[0043] In step S4, if the determination unit 64 determines that it will not output the second video signal (NO in S3), the video switching unit 63 switches the video signal to be output to the display 30 to the first video signal. In step S4, the output unit 62 outputs the first video signal to the display 30.

[0044] In step S5, if the determination unit 64 determines to output the second video signal (YES in S3), the video switching unit 63 switches the video signal to be output to the display 30 to the second video signal. In step S5, the output unit 62 outputs the second video signal to the display 30.

[0045] In step S6, the determination unit 64 determines whether the work has been completed. If the determination unit 64 determines that the work has not been completed (NO in S6), step S3 is executed again. If the determination unit 64 determines that the work has been completed (NO in S6), the remote control support device 60 completes the work shown in Figure 4.

[0046] [Example of image displayed on the screen 1] An example of the image displayed on the display 30 will be described. First, the remotely operated overhead crane device 20A will be described with reference to Figure 3. Figure 3 is a schematic diagram showing an overview of the overhead crane device 20A. For the sake of explanation, components having the same function as those described in the above embodiment will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0047] The overhead crane device 20A shown in Figure 3 is a device for transporting steel plates M. The overhead crane device 20A is an example of a work robot 20 provided by the remote control system 100. As shown in Figure 3, the overhead crane device 20A is equipped with a transport rack 21, a magnet 22, and a chain 23.

[0048] The transport rack 21 is attached to a main beam (not shown) that travels along a rail installed on the ceiling. The transport rack 21 can be raised and lowered relative to the main beam by a lifting device (not shown). A magnet 22 attracts a steel plate M by magnetic force. The magnet 22 is attached to the transport rack 21 by a chain 23. The transport rack 21, magnet 22, chain 23, and steel plate M are the main components of the transport operation performed by the overhead crane device 20A. The steel plate M is an example of a moving object that moves due to the operation of the overhead crane device 20A.

[0049] A camera 40 is mounted on the side of the transport rack 21 to which the magnet 22 is attached, i.e., on the underside of the transport rack 21. Note that the camera 40 is not shown in Figure 3. The remote operator remotely operates the overhead crane device 20A while checking the display 30 which shows the image from the camera 40, and sequentially transports the steel plates M loaded in the first loading area P1 to the second loading area P2. A camera 50 is installed near the second loading area P2 and photographs the area including the second loading area P2. Note that a camera 50 that photographs the second loading area P2 is not required.

[0050] The overhead crane device 20A lowers the transport rack 21 in the direction indicated by arrow D1 relative to the main beam, which is stopped at a position corresponding to the first loading area P1. After lowering the transport rack 21 to a predetermined position, the overhead crane device 20A supplies current to the magnet 22 to attract the steel plate M placed on the first loading area P1. After attracting the steel plate M with the magnet 22, the overhead crane device 20A raises the transport rack 21 relative to the main beam in the direction indicated by arrow D1. After raising the transport rack 21 to a predetermined position, the overhead crane device 20A moves the main beam in the direction indicated by arrow D2.

[0051] The overhead crane device 20A lowers the transport rack 21 in the direction indicated by arrow D1 relative to the main beam which has stopped at a position corresponding to the second loading area P2. After lowering the transport rack 21 to the predetermined position, the overhead crane device 20A stops supplying current to the magnet 22, thereby placing the steel plate M, which was attracted to the magnet 22, onto the second loading area P2. The second loading area P2 is the work area for the transport operation of the steel plate M performed by the overhead crane device 20A.

[0052] After the overhead crane device 20A places the steel plates M on the second loading area P2, it raises the transport rack 21 relative to the main beam in the direction indicated by arrow D1. Subsequently, the overhead crane device 20A moves the main beam in the direction indicated by arrow D2 and stops the main beam at a position corresponding to the first loading area P1. By repeating the above operations, the overhead crane device 20A sequentially transports the steel plates M placed on the first loading area P1 to the second loading area P2 and loads the steel plates M onto the second loading area P2.

[0053] Next, with reference to Figure 4, an example of the video output to the display 30 by the remote control support device 60 will be described. Figure 4 is a diagram showing an example of the video displayed on the display 30. Reference numeral 400 in Figure 4 indicates the actual video from the camera 40 displayed on the display 30 when the transport rack 21 is positioned at its highest position relative to the main beam. Reference numeral 401 in Figure 4 indicates the processed video displayed on the display 30 when the transport rack 21 is positioned at its highest position relative to the main beam. Reference numeral 402 in Figure 4 indicates the processed video displayed on the display 30 when the transport rack 21 is lowered relative to the main beam.

[0054] As shown by reference numeral 400 in Figure 4, in the actual image from camera 40, the remote operator can see the magnet 22 and the steel plate M attached to the magnet 22. However, in the actual image from camera 40, the remote operator cannot see the second mounting area P2 located behind the steel plate M. By switching the image displayed on the display 30 from the actual image to a processed image, the remote operator can see the second mounting area P2, which is the work target.

[0055] As shown by reference numeral 401 in Figure 4, in the processed image, the magnet 22 and the steel plate M attached to the magnet 22, which are included in the removal target area R1 indicated by the dashed line, are transparent. Therefore, the remote operator can see the second mounting area P2 located behind the steel plate M. The removal target area R1 is determined based on the contour of the steel plate M. In Figure 4, the contours of the transparent magnet 22 and steel plate M are shown by dashed lines.

[0056] The remote operator remotely controls the overhead crane device 20A while checking the processed image on the display 30 to stop the main beam at a position corresponding to the second loading area P2. The remote operator also lowers the transport rack 21 in the direction indicated by arrow D1. The main beam and transport rack 21 are operated so that the steel plate M is within the range of the second loading area P2, as shown in the processed image indicated by reference numeral 402 in Figure 4. The remote operation support device 60 may also accept input from the remote operator to switch the image displayed on the display 30 from the processed image to the actual image during the process of placing the steel plate M in the second loading area P2. This allows the remote operator to check for errors in the hidden background and, according to the error, switch the image displayed on the display 30 between the processed image and the actual image, thereby enabling the steel plate M to be placed accurately and efficiently within the range of the second loading area P2.

[0057] In Figure 4, the remote control support device 60 determines the area including the entire steel plate M as the removal target area R1, but the configuration is not limited to this. As shown in Figure 5, the remote control support device 60 may also determine the area near the contour of the steel plate M as the removal target area R2.

[0058] Figure 5 shows an example of a processing image displayed on the display 30. Reference numeral 500 in Figure 5 indicates the processing image displayed on the display 30 when the transport rack 21 is positioned at its highest position relative to the main beam. Reference numeral 501 in Figure 5 indicates the processing image displayed on the display 30 when the transport rack 21 is lowered relative to the main beam. In Figure 5, the transparent parts of the outlines of the magnet 22 and the steel plate M are shown with dashed lines.

[0059] As shown by reference numeral 500 in Figure 5, when the transport rack 21 is positioned at its highest point relative to the main beam, only the vicinity of the outlines of the magnet 22 and steel plate M included in the removal area R2 is transparent. The second placement area P2, which is not included in the removal area R2, is hidden behind the non-transparent steel plate M, so the remote operator cannot see the second placement area P2 in the image on the display 30. When the transport rack 21 is lowered by remote control, as shown by reference numeral 501 in Figure 5, the outline of the second placement area P2 is included in the removal area R2. In the removal area R2, the magnet 22 and steel plate M are transparent, so the remote operator can see the outline of the second placement area P2 in the image on the display 30. The remote operator uses the outline of the second placement area P2 as a guide to place the steel plate M on the second placement area P2. Thus, the removal area R2 may be determined to match the main part of the remote operation.

[0060] [Example of image displayed on the screen 2] Referring to Figures 6 and 7, an example of the video output to the display 30 by the remote control support device 60 will be described. Figures 6 and 7 are diagrams showing an example of the video displayed on the display 30. For the sake of explanation, components having the same function as those described in the above embodiment will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0061] First, the remotely operated work device 20B will be described. The work device 20B shown in Figures 6 and 7 is a device for pressing a switch SW. The work device 20B is an example of a work robot 20 provided by the remote operation system 100. The switch SW is an example of a work target. The work device 20B comprises a robot arm 25 and a work unit 26 (see Figure 7). The work unit 26 is provided at the tip of the robot arm 25. The work unit 26 is the part for pressing the switch SW. The remote operator remotely operates the work device 20B while checking the image displayed on the display 30, and presses the switch SW with the work unit 26. The robot arm 25 and the work unit 26 are the main components of the work.

[0062] A camera 40 is installed at a distance from the work device 20B to capture video footage of the work device 20B in operation. Camera 40 is a fixed-point camera whose shooting position does not move. In this example, a camera 50 for capturing the hidden background is not installed. In the processed video displayed on the display 30, the hidden background is observed using video footage of the hidden background previously captured by camera 40.

[0063] Next, the video output to the display 30 by the remote control support device 60 will be described. As shown in Figures 6 and 7, the removal target area R3 is determined based on the contour of the work device 20B and the contour of the switch SW. Since the geometric shape of the work device 20B is known, the contour of the work device 20B is recognized based on the geometric shape of the work device 20B that has been registered in advance. The contour of the switch SW may be recognized by image processing or by the geometric shape of the switch SW that has been registered in advance. The removal target area R3 moves in accordance with the movement of the work unit 26. The size of the removal target area R3 changes according to the size of the work unit 26, which changes with the movement in the near and far directions relative to the camera 40.

[0064] In the video shown in Figure 6, the work device 20B and the switch SW are not in an overlapping position, so the actual video captured by the camera 40 is displayed on the display 30. Note that in Figure 6, for the sake of explanation, the removal target area R3 is shown, but the removal target area R3 is not shown in the actual video. The remote operator remotely operates the work device 20B while checking the actual video displayed on the display 30, bringing the work unit 26 closer to the switch SW.

[0065] As shown in Figure 7, when the work unit 26 is brought closer to the switch SW, causing the robot arm 25 and the work unit 26 to overlap with the switch SW, the image displayed on the display 30 switches from the real image to the processed image. As a result, a portion of the robot arm 25 and the work unit 26 included in the removal target area R3 become transparent, allowing the remote operator to see the switch SW. The remote operation support device 60 may also accept input from the remote operator to switch the image displayed on the display 30 from the processed image to the real image, or from the real image to the processed image, during the operation of pressing the switch SW by the work device 20B.

[0066] According to the remote control support device 60 described above, the image displayed on the display 30 can be switched between the actual image and the processed image. By switching the image displayed on the display 30, the remote operator can compare the actual image and the processed image, and thus recognize the degree of error in the hidden background of the processed image. This improves the work efficiency of the remote operator.

[0067] Furthermore, in the configuration where the determination unit 64 performs the determination, the actual video or processed video is displayed on the display 30 according to the determination result of the determination unit 64. Therefore, the remote operator can switch the video displayed on the display 30 without operating the remote operation support device 60. This reduces the workload of the remote operator.

[0068] With the configuration in which the determination unit 64 makes a determination according to the percentage of the hidden area of ​​the work target, if the work target is hidden by the work subject by overlapping with the work subject in the video from the remote operator's perspective by a predetermined percentage or more, the system can switch from the actual video to the processed video. In other words, if the work subject is larger than the work target and the work subject is hidden by the work subject, the processed video is displayed on the display 30. This can further improve the work efficiency of the remote operator.

[0069] [Embodiment 2] Other embodiments of this disclosure will be described below with reference to Figure 8. Figure 8 is a diagram showing an example of a processed image displayed on the display 30. For the sake of explanation, components having the same function as those described in the above embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated. In this embodiment, the remote operation support device 60 outputs a processed image to the display 30 that has undergone Augmented Reality (AR) processing, which synthesizes a virtual image with the actual image captured by the camera 40 or the image that has undergone DR processing. The virtual image is an image that represents information related to the work performed by remote operation.

[0070] Figure 8 shows a video of the work performed by the work device 20B. In Figure 8, the processed video displayed on the display 30 is created by applying both DR processing and AR processing to the actual video captured by the camera 40. In the removal target area R3, a virtual image 260 is displayed on the work device 20B. In this embodiment, the virtual image 260 is displayed on the contour portion of the tip of the work unit 26 within the area of ​​the work device 20B included in the removal target area R3. That is, the virtual image 260 is a video representing the tip shape of the work unit 26 in order to clarify the tip position of the work unit 26 for pressing the switch SW.

[0071] More specifically, the processing image generation unit 61 of the remote control support device 60 further performs AR processing on the actual image from the camera 40, compositing a virtual image 260 into the area corresponding to the removal target area R3. In this embodiment, the processing image generation unit 61 generates a virtual image 260 that matches the tip shape of the work unit 26 based on the geometric shape of the work device 20B which has been registered in advance. The processing image generation unit 61 generates a processed image by compositing the generated virtual image 260 into the tip position of the work unit 26 in the processed image after DR processing. The processing image generation unit 61 performs AR processing on the actual image from the camera 40 after DR processing. Therefore, the virtual image 260 is displayed without transparency in the removal target area R3. Alternatively, the processing image generation unit 61 may perform DR processing on the actual image from the camera 40 after AR processing. In this case, the virtual image 260 will be displayed transparently in the removal target area R3.

[0072] Furthermore, the virtual image is not limited to being superimposed and displayed on the work subject, but may also be superimposed and displayed on the work target. For example, the virtual image may be an image representing the shape of a switch SW. Also, the virtual image is not limited to an image representing the geometric shape of the work subject or work target. For example, the virtual image may be an image of information representing the detection result of a state sensor that detects the state of the work subject or work target, or an environmental sensor that detects the state of the surrounding environment of the work subject or work target. In this case, the virtual image representing the detection result of the state sensor or environmental sensor may be superimposed and displayed on the work subject or work target, or it may be displayed within the removal target area so as not to overlap with the work subject and work target. In addition, in the processed image after AR processing, the virtual image may be displayed outside the area corresponding to the removal target area determined in DR processing.

[0073] The processed video generation unit 61 may generate a processed video in which only AR processing has been applied to the actual video from the camera 40. In this case, the display 30 may display the processed video in which only AR processing has been applied to the actual video from the camera 40. The remote operation support device 60 may display the processed video in which DR processing and / or AR processing has been applied on the display 30 based on input from the remote operator using the input device.

[0074] Generally, in remote operation, if there are areas that are difficult for the remote operator to see on the display screen from their perspective, this can be addressed by installing additional cameras to capture those areas. However, as the number of cameras increases, the remote operator needs to select the appropriate screen as needed, and if they cannot select the appropriate screen, work efficiency decreases. Also, in some work sites, it is difficult to install multiple cameras, so it is desirable to be able to acquire various information from a single camera.

[0075] Furthermore, it is generally known that in remote operation, the image displayed on the screen is two-dimensional, making it difficult to grasp three-dimensional space using only the display image, thus reducing work efficiency. Therefore, in order for inexperienced remote operators to perform remote work efficiently, it is necessary to display images from different viewpoints on at least two displays. If three-dimensional images are displayed on the screen, it becomes possible to grasp three-dimensional space with just one image. However, there are problems such as motion sickness, making it difficult for remote operators to work for long periods of time.

[0076] With the above configuration, DR processing makes objects invisible due to obstructions visible, and AR processing allows for the additional display of information about the work site. As a result, remote workers can obtain various information from a single video feed. This can further improve the work efficiency of remote operators.

[0077] [Examples of implementation using software] The function of the remote control support device 60 (hereinafter referred to as "the device") is a program that causes the device to function as a computer, and can be realized by a program that causes each control block of the device to function as a computer.

[0078] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0079] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0080] Furthermore, some or all of the functions of each of the above control blocks can also be implemented by logic circuits. For example, an integrated circuit in which logic circuits functioning as each of the above control blocks are formed is also included in the scope of this disclosure. In addition, it is also possible to implement the functions of each of the above control blocks by, for example, a quantum computer.

[0081] 〔summary〕 A remote operation support device according to Embodiment 1 of the present disclosure is a remote operation support device that outputs work video of work performed by remote operation to a display, and comprises: a processed video generation unit that generates a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of the remote operator, which erases or makes transparent occluding objects in the area to be removed, and synthesizes a video that models the background that is hidden by overlapping with the occluding objects in the area to be removed; an output unit that outputs a first video signal representing the real video or a second video signal representing the processed video to the display; and a video switching unit that switches the video signal output to the display to the first video signal or the second video signal.

[0082] According to the above embodiment 1, the image displayed on the screen can be switched between the actual image and the processed image. By switching the image displayed on the screen, the remote operator can compare the actual image and the processed image, and thus recognize the degree of error in the hidden background of the processed image. This improves the work efficiency of the remote operator.

[0083] The remote control support device according to Embodiment 2 of the present disclosure further comprises a determination unit that determines whether or not to output the second video signal to the display, and the video switching unit may switch the video signal to be output to the display to the first video signal if the determination unit determines that the second video signal is not to be output, and switch the video signal to be output to the display to the second video signal if the determination unit determines that the second video signal is to be output.

[0084] According to the above embodiment 2, the actual video or processed video is displayed on the display according to the determination result by the determination unit. Therefore, the remote operator can switch the video displayed on the display without operating the remote operation support device. This reduces the workload of the remote operator.

[0085] In the remote operation support device according to Embodiment 3 of the present disclosure, in Embodiment 2 described above, the determination unit may determine that it will output the second video signal to the display if, in the video from the remote operator's viewpoint, the ratio of the area of ​​the target part that is hidden by the obstruction as the obstruction overlaps with the target part to the area of ​​the target part that is to be seen by the remote operator is equal to or greater than a predetermined threshold.

[0086] According to embodiment 3 described above, if the target part is obscured by an obstruction in the video from the remote operator's perspective by more than a predetermined percentage, the system can switch from the actual video to a processed video. In other words, if the obstruction is larger than the target part and the target part is obscured by the obstruction, the processed video is displayed on the screen. This can further improve the work efficiency of the remote operator.

[0087] In the remote operation support device according to Embodiment 4 of the present disclosure, in any of Embodiments 1 to 3 above, the processing video generation unit may model the background video obtained by real-time shooting, the background video obtained by prior shooting, the background video obtained by estimation from the actual video, or the background video generated by combining these, and synthesize it with the area to be removed. The removal area may be determined based on the area division of the actual video using segmentation technology, the outline of the obstruction or the target part that is the target of the remote operator's view, or site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof.

[0088] In the remote control support device according to aspect 5 of the present disclosure, in any of aspects 1 to 4 above, the processed video generation unit may generate a processed video in which an Augmented Reality (AR) process is performed to synthesize a virtual video with the actual video or the video on which the DR process has been performed.

[0089] According to embodiment 5 described above, DR processing makes objects that are not visible due to obstructions visible, and AR processing additionally displays information about the work site. As a result, remote workers can obtain various information from a single video. This can further improve the work efficiency of the remote operator.

[0090] A remote operation support method according to aspect 6 of the present disclosure is a remote operation support method performed by a remote operation support device that outputs a work video of a task performed by remote operation to a display, and includes: a processed video generation step that generates a processed video by performing DR (Diminished Reality) processing on a real video from the viewpoint of a remote operator, which erases or makes transparent an obstruction in an area to be removed, and synthesizes a video that models the background that is hidden by overlapping with the obstruction in the area to be removed; an output step that outputs a first video signal representing the real video or a second video signal representing the processed video to the display; and a video switching step that switches the video signal output to the display to the first video signal or the second video signal.

[0091] This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure. [Explanation of Symbols]

[0092] 10 Remote control device 20 Industrial Robots 30 displays 40 Cameras 50 Cameras 60 Remote control support device 61 Processing and Video Generation Unit 62 Output section 63 Video switching section 64 Judgment section 100 Remote Control Systems

Claims

1. A remote operation support device that outputs video of work performed by remote operation to a display, A processing video generation unit generates a processed video by performing DR (Diminished Reality) processing on the actual video from the remote operator's point of view, which removes or makes transparent the obstructions within the area to be removed, and composites it with a video that models the background that is hidden by overlapping with the said obstructions within the area to be removed. An output unit that outputs a first video signal representing the actual image, or a second video signal representing the processed image, to the display, The system includes a video switching unit that switches the video signal output to the display to either the first video signal or the second video signal. Remote control support device.

2. The system further includes a determination unit that determines whether or not to output the second video signal to the display, The aforementioned video switching unit is If the determination unit determines that it will not output the second video signal, it switches the video signal to be output to the display to the first video signal. If the determination unit determines that the second video signal should be output, the video signal output to the display is switched to the second video signal. The remote control support device according to claim 1.

3. The determination unit determines that, in the video from the remote operator's perspective, if the ratio of the area of ​​the target part that is hidden by the obstruction when the obstruction overlaps with the target part to the area of ​​the target part that is visible to the remote operator is equal to or greater than a predetermined threshold, it will output the second video signal to the display. The remote control support device according to claim 2.

4. The aforementioned processed video generation unit, The background image obtained by real-time shooting, the background image obtained by prior shooting, the background image obtained by estimation from the actual image, or the background image generated by combining these, is modeled and composited onto the area to be removed. The removal target area is determined based on the following: segmentation of the actual video using segmentation technology, the outline of the obstruction or the target part that is visible to the remote operator, site configuration information representing the site configuration of the work site where the work is performed, or a combination thereof. The remote control support device according to claim 1.

5. The remote control support device according to any one of claims 1 to 4, wherein the processed image generation unit generates a processed image by performing AR (Augmented Reality) processing to synthesize a virtual image with the actual image or the image that has undergone DR processing.

6. A remote operation support method performed by a remote operation support device that outputs video of work performed by remote operation to a display, A process for generating a processed image is performed by applying DR (Diminished Reality) processing to the actual video footage from the remote operator's perspective, which involves erasing or making transparent the occluding objects within the area to be removed, and then compositing it with an image that models the background that would otherwise be hidden by overlapping with the occluding objects within the area to be removed. Output step of outputting a first video signal representing the actual image or a second video signal representing the processed image to the display, The process includes a video switching step of switching the video signal output to the display to the first video signal or the second video signal, Remote operation support method.

Images