Remote work support system, remote work support method, and program

Abstract

To suppress reduction in quality of an important region in a captured image while reducing a communication load to be applied on a network by the captured image obtained by imaging a work space.SOLUTION: A remote work support system includes: a calculation unit which calculates two-dimensional coordinates in a worker-side screen, corresponding to three-dimensional coordinates passing through a section related to work; an estimation unit which estimates, as an important region, a region in the worker-side screen corresponding to the two-dimensional coordinates to obtain information indicating the important region; an encoding processing unit which encodes a captured image obtained by imaging a work space on the basis of the information indicating the important region, to obtain an encoded captured image; a transmission control unit which controls transmitting the encoded captured image; an acquisition unit which acquires the encoded captured image; a decoding unit which decodes the encoded captured image to obtain a decoded captured image; and an instructor-side display control unit which displays the decoded captured image on an instructor-side screen which is visually recognized by an instructor who issues an instruction to a worker.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a remote work support system, a remote work support method, and a program.

Background Art

[0002] In recent years, due to a shortage of skilled workers, skilled workers are not always deployed at the work site. Instead, a skilled worker in a location different from the work site acts as an instructor and gives instructions to workers who perform work at the work site remotely. The importance of remote work support has been increasing. As an example of technology related to remote work support, various remote work support systems are known (for example, see Patent Document 1).

[0003] For example, there is known a remote work support system in which a worker terminal transmits video of a space where work is performed by a worker (hereinafter also referred to as a "work space") and audio collected from the worker to an instructor terminal, the instructor terminal presents the video and audio to the instructor, and the instructor gives instructions to the worker while viewing the video and audio (for example, see Non-Patent Document 1).

[0004] More specifically, in such a remote work support system, the worker-side system superimposes a finger video showing the finger shape of the instructor on a captured image obtained by capturing the work space, and displays the captured image on which the finger video is superimposed. Further, the worker-side system transmits the captured image to the instructor-side system, and the instructor-side system superimposes the finger video on the captured image received from the worker-side system and displays the captured image on which the finger video is superimposed.

[0005] This allows the instructor to grasp how the finger shape of the instructor is conveyed to the worker. Note that various techniques are known as techniques for detecting the finger shape of the instructor (for example, see Non-Patent Document 2).

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Patent No. 6568130 [Non-patent literature]

[0007] [Non-Patent Document 1] Shunsuke Ichihara, Yusuke Suzuki, "Field Evaluation Experiment of a Remote Work Support System Using Hand Gestures and Drawing Functions," [online], March 2, 2017, Information Processing Society of Japan Interaction 2017, [Retrieved June 21, 2022], Internet<URL:http: / / www.interaction-ipsj.org / proceedings / 2017 / data / pdf / 1-502-13.pdf> [Non-Patent Document 2] Ultraleap Corporation website, [online], [searched June 21, 2022], Internet<URL:https: / / www.ultraleap.com> [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Generally, the higher the quality of the captured image obtained from imaging the workspace, the greater the communication load on the network caused by transmitting the image. Conversely, the lower the quality of the captured image, the lower the overall visibility of the image.

[0009] Therefore, it is desirable to provide a technology that can reduce the communication load on the network from images obtained by imaging the workspace, while suppressing the degradation of quality in important areas of the captured images. [Means for solving the problem]

[0010] To solve the above problems, according to one aspect of the present invention, a remote work support system is provided, comprising: a detection unit that detects a plurality of three-dimensional coordinates through which an object undergoing work passes, based on sensor data obtained from the object through a sensor; an identification unit that identifies a part of the object undergoing work that is subject to the work, based on the plurality of three-dimensional coordinates; a calculation unit that calculates two-dimensional coordinates on a worker-side screen, which is visible to the worker, corresponding to the three-dimensional coordinates through which the part undergoing work passes; an estimation unit that estimates an important area on the worker-side screen corresponding to the two-dimensional coordinates and obtains information indicating the important area; an encoding processing unit that encodes an image of the workspace where the work is performed based on the information indicating the important area and obtains an encoded image; a transmission control unit that controls the transmission of the encoded image; an acquisition unit that acquires the encoded image; a decoding unit that decodes the encoded image to obtain a decoded image; and an instructor-side display control unit that displays the decoded image on an instructor-side screen, which is visible to the instructor giving instructions to the worker.

[0011] The encoding processing unit may set the compression ratio for the non-important region, which is a region other than the important region in the captured image of the workspace, to be higher than the compression ratio for the important region.

[0012] The estimation unit may estimate the important area as the region on the worker's screen, with the central position being the position indicated by the two-dimensional coordinates on the worker's screen that correspond to the three-dimensional coordinates through which the part involved in the work passes.

[0013] The calculation unit calculates a two-dimensional direction on the operator's screen corresponding to the three-dimensional direction of the part being worked on, based on a plurality of three-dimensional coordinates through which the part being worked on passes, and the estimation unit may estimate the area on the operator's screen corresponding to the two-dimensional coordinates and the two-dimensional direction as the important area.

[0014] The identifying unit may estimate the degree of curvature of each of the multiple parts constituting the object involved in the work based on the multiple three-dimensional coordinates, and identify the part involved in the work based on the degree of curvature of each of the multiple parts.

[0015] The specified part may identify the part among the plurality of parts in which the degree of curvature is below a threshold as the part related to the work.

[0016] The specified unit may identify the part of the body involved in the operation based on the plurality of three-dimensional coordinates and a trained model obtained by learning using a predetermined machine learning algorithm.

[0017] The remote work support system comprises a worker-side system used by the worker and an instructor-side system used by the instructor, wherein the worker-side system comprises the encoding processing unit and the transmission control unit, and the instructor-side system may comprise the acquisition unit, the decoding unit and the instructor-side display control unit.

[0018] The instructioner-side display control unit may display on the instructioner's screen an image on which the object related to the operation, superimposed on the decoded captured image, corresponds to the object.

[0019] The remote work support system may include a worker-side display control unit that displays on the worker's screen an image corresponding to an object related to the work, superimposed on an image of the workspace captured.

[0020] The object involved in the work is the fingers of the worker or the person giving the instruction, and the part involved in the work may be one or more fingers that make up the fingers.

[0021] The object involved in the aforementioned work may include the first object used by the worker.

[0022] The object involved in the aforementioned work may include a second object used by the person giving the instructions.

[0023] Also, according to another aspect of the present invention for solving the above problems, based on sensor data obtained by a sensor from an object related to work, detecting a plurality of three-dimensional coordinates through which the object related to the work passes; identifying a part related to the work in the object related to the work based on the plurality of three-dimensional coordinates; calculating two-dimensional coordinates on a screen on the operator side visually recognized by an operator corresponding to the three-dimensional coordinates passing through the part related to the work; estimating an area on the screen on the operator side corresponding to the two-dimensional coordinates as an important area and obtaining information indicating the important area; encoding an imaging image of a work space in which the work is performed based on the information indicating the important area to obtain an encoded imaging image; controlling transmission of the encoded imaging image; obtaining the encoded imaging image; decoding the encoded imaging image to obtain a decoded imaging image; and displaying the decoded imaging image on a screen on the instructor side visually recognized by an instructor who gives an instruction to the operator. A remote work support method is provided.

[0024] Also, according to another aspect of the present invention for solving the above problems, a computer is provided with a detection unit that detects a plurality of three-dimensional coordinates through which an object related to work passes based on sensor data obtained by a sensor from the object related to work, a specification unit that specifies a part related to the work in the object related to the work based on the plurality of three-dimensional coordinates, a calculation unit that calculates two-dimensional coordinates on the operator side screen visually recognized by an operator corresponding to the three-dimensional coordinates passing through the part related to the work, an estimation unit that estimates an area on the operator side screen corresponding to the two-dimensional coordinates as an important area and obtains information indicating the important area, an encoding processing unit that performs encoding on a captured image of a work space in which the work is performed based on the information indicating the important area to obtain an encoded captured image, a transmission control unit that controls transmission of the encoded captured image, an acquisition unit that acquires the encoded captured image, a decoding unit that decodes the encoded captured image to obtain a decoded captured image, and an instructor side display control unit that displays the decoded captured image on an instructor side screen visually recognized by an instructor who gives an instruction to the operator, and a program is provided to function as a remote work support system including these components.

Advantages of the Invention

[0025] As described above, according to the present invention, a technique is provided that can reduce the communication load imposed on a network by a captured image obtained by imaging a work space and suppress a decrease in the quality of an important area in the captured image.

Brief Description of the Drawings

[0026] [Figure 1] It is a diagram showing a functional configuration example of a remote work support system according to an embodiment of the present invention. [Figure 2] It is a block diagram showing a functional configuration example of an operator side device 14. [Figure 3] It is a block diagram showing a functional configuration example of an instructor side device 24. [Figure 4] It is a diagram showing a display example of a decoded work captured video by an instructor side display unit 26 before an instruction is given. [Figure 5] This figure shows an example of three-dimensional skeletal information of fingers detected by the finger detection unit 241. [Figure 6] This diagram illustrates an example of a method for calculating the degree of curvature of the first joint of the index finger. [Figure 7] This figure shows an example of how a hand image superimposed on an image of the work being performed is displayed on the worker's side display unit 16. [Figure 8] This is a diagram to explain the estimation of the pre-correction region. [Figure 9] This is a diagram illustrating the estimation of the corrected region. [Figure 10] This figure illustrates the estimation of important regions, including the pre-correction region R1 and the post-correction region R2. [Figure 11] This figure shows an example of how the hand image superimposed on the decoded work image M12 is displayed by the user-side display unit 26. [Figure 12] This flowchart shows an example of the operation of the worker-side system 10. [Figure 13] This flowchart shows an example of the operation of the instruction-side system 20. [Figure 14] This figure shows the hardware configuration of an information processing device 900 as an example of the instruction-side system 10 according to an embodiment of the present invention. [Modes for carrying out the invention]

[0027] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0028] (1. Details of the Embodiment) Details of embodiments of the present invention will be described below.

[0029] (1-1. Configuration of the Remote Work Support System) First, an example of the configuration of a remote work support system according to an embodiment of the present invention will be described. Figure 1 is a diagram showing an example of the functional configuration of a remote work support system according to an embodiment of the present invention. As shown in Figure 1, the remote work support system 1 includes a worker-side system 10, an instructor-side system 20, and a network 30.

[0030] The worker-side system 10 is the system used by the worker within the remote work support system 1. The worker performs their work at a location (i.e., a remote location) away from the supervisor who gives them instructions. On the other hand, the supervisor-side system 20 is the system used by the supervisor within the remote work support system 1. The worker-side system 10 and the supervisor-side system 20 are connected to a network 30 and are configured to communicate with each other via the network 30.

[0031] In remote work support, voice communication is generally used between the supervisor and the worker. However, since the configuration necessary for voice communication is not relevant to the description of the configuration of the remote work support system 1 according to the embodiment of the present invention, the description of the configuration necessary for voice communication will be omitted.

[0032] As shown in Figure 1, the worker-side system 10 comprises a work imaging unit 12, a worker-side device 14, and a worker-side display unit 16. On the other hand, as shown in Figure 1, the instructor-side system 20 comprises a finger imaging unit 22, an instructor-side device 24, and an instructor-side display unit 26.

[0033] (Working Image Unit 12) The work imaging unit 12 obtains an image (hereinafter also referred to as "work imaging image") by imaging the work space, which is the space in which the worker performs the work. In the following description, it is mainly assumed that the work imaging image is a moving image consisting of multiple frames that are captured continuously in chronological order. A moving image can also be referred to as a "video." Therefore, in the following description, the term "work imaging video" will be used as an example of a work imaging image. However, the work imaging image may also be a still image.

[0034] The work imaging unit 12 is preferably positioned to face the same direction as the worker's line of sight. Therefore, it is preferable that the work imaging unit 12 is integrated with the AR display. However, the work imaging unit 12 may be configured as separate hardware from the AR display.

[0035] (Operator-side device 14) The operator-side device 14 is implemented by a computer. An example of the functional configuration of the operator-side device 14 will be described with reference to Figure 2.

[0036] Figure 2 is a block diagram showing an example of the functional configuration of the operator-side device 14. As shown in Figure 2, the operator-side device 14 comprises a control unit 140, a communication unit 148, and a storage unit 149. The functions of the control unit 140, the communication unit 148, and the storage unit 149 will be described below.

[0037] (Control unit 140) The control unit 140 performs various arithmetic operations. For example, the control unit 140 may include an arithmetic unit such as a CPU (Central Processing Unit), and its functions may be realized when a program stored in ROM (Read Only Memory) is loaded into RAM by the arithmetic unit and executed. In this case, a computer-readable recording medium on which the program is stored may also be provided. Alternatively, these blocks may be composed of dedicated hardware or a combination of multiple hardware components.

[0038] The data necessary for calculations performed by the arithmetic unit is appropriately stored in the storage unit 149. As shown in Figure 2, the control unit 140 comprises a determination result acquisition unit 141, an encoding unit 142, a transmission control unit 143, and a worker-side display control unit 144. The details of the functions of the determination result acquisition unit 141, the encoding unit 142, the transmission control unit 143, and the worker-side display control unit 144 will be described later.

[0039] (Communications Section 148) The communication unit 148 is configured with a communication interface and communicates with the instructioner-side device 24 via the network 30.

[0040] (Storage unit 149) The memory unit 149 is a memory capable of storing programs and data for operating the control unit 140. The memory unit 149 can also temporarily store various data required during the operation of the control unit 140. For example, the memory unit 149 may be a non-volatile memory.

[0041] (Worker side display section 16) The operator-side display unit 16 has the function of displaying various information to the operator in accordance with the control unit 140. More specifically, the operator-side display unit 16 has a screen that is visible to the operator (hereinafter also referred to as the "operator-side screen"), and displays various information on the operator-side screen in accordance with the control unit 140.

[0042] For example, the worker-side display unit 16 may be a liquid crystal display (LCD) device or an OLED (Organic Light Emitting Diode) device. Furthermore, the worker-side display unit 16 may be a head-mounted display worn on the worker's head or a stationary display.

[0043] (Finger imaging section 22) The finger imaging unit 22 obtains an image (hereinafter also referred to as "finger image") by imaging the fingers of the person making the instruction. In the following description, we will mainly assume that the finger image is a moving image (video). Therefore, in the following description, we will use the term "finger image video" as an example of a finger image. However, the finger image may also be a still image.

[0044] Furthermore, the hands of the person giving the instruction may be an example of an object involved in the work, as will be explained later. Also, the hands may include the area of ​​the body from the wrist to the tips of each of the multiple fingers that make up the hand. In addition, the hand imaging unit 22 may be replaced with other sensors. That is, the hand imaging video may be an example of sensor data obtained by a sensor.

[0045] (Instructor side device 24) Let's return to Figure 1 and continue the explanation. The instruction-side device 24 is implemented by a computer. Referring to Figure 3, we will explain an example of the functional configuration of the instruction-side device 24.

[0046] Figure 3 is a block diagram showing an example of the functional configuration of the instructioner-side device 24. As shown in Figure 3, the instructioner-side device 24 comprises a control unit 240, a communication unit 248, and a storage unit 249. The functions of the control unit 240, the communication unit 248, and the storage unit 249 will be described below.

[0047] (Control unit 240) The control unit 240 performs various arithmetic operations. For example, the control unit 240 may include an arithmetic unit such as a CPU (Central Processing Unit), and its functions may be realized when a program stored in ROM (Read Only Memory) is loaded into RAM by the arithmetic unit and executed. In this case, a computer-readable recording medium on which the program is stored may also be provided. Alternatively, these blocks may be composed of dedicated hardware or a combination of multiple hardware components.

[0048] The data necessary for calculations performed by the arithmetic unit is appropriately stored in the storage unit 249. As shown in Figure 3, the control unit 240 includes a finger detection unit 241, a finger determination unit 242, a determination result output unit 243, an encoded data acquisition unit 244, a decoding unit 245, and an instruction-side display control unit 246. The details of the functions of each of the finger detection unit 241, finger determination unit 242, determination result output unit 243, encoded data acquisition unit 244, decoding unit 245, and instruction-side display control unit 246 will be described later.

[0049] (Communications Section 248) The communication unit 248 is configured with a communication interface and communicates with the operator-side device 14 via the network 30.

[0050] (Storage unit 249) The memory unit 249 is a memory capable of storing programs and data for operating the control unit 240. The memory unit 249 can also temporarily store various data required during the operation of the control unit 240. For example, the memory unit 249 may be a non-volatile memory.

[0051] (Instructor side display section 26) The instructioner-side display unit 26 has the function of displaying various information to the instructioner in accordance with the control unit 240. More specifically, the instructioner-side display unit 26 has a screen that is visible to the instructioner (hereinafter also referred to as the "instructor-side screen"), and displays various information on the instructioner-side screen in accordance with the control unit 240.

[0052] For example, the instructor-side display unit 26 may be a liquid crystal display (LCD) device or an OLED (Organic Light Emitting Diode) device. Furthermore, the instructor-side display unit 26 may be a head-mounted display worn on the instructor's head or a stationary display.

[0053] (Network 30) Network 30 connects the worker-side system 10 and the supervisor-side system 20. Network 30 can also function as a communication path between the worker-side system 10 and the supervisor-side system 20.

[0054] (Before the person giving instructions begins) This section describes the process before the instruction is given by the instructor. In the worker-side system 10, the encoding unit 142 acquires the work image captured by the work imaging unit 12. The encoding unit 142 performs encoding on the acquired work image captured based on information indicating important areas (in other words, information indicating which areas are important areas). Here, before the instruction is given by the instructor, the entire work image captured is initialized as a non-important area. Therefore, the encoding unit 142 may perform encoding on the entire work image captured using the same compression ratio.

[0055] The transmission control unit 143 controls the communication unit 148 so that the encoded work image is transmitted to the instruction-side device 24 via the network 30. In the instruction-side system 20, the encoded data acquisition unit 244 acquires the encoded work image received by the communication unit 248, and the decoding unit 245 decodes the acquired encoded work image to obtain the decoded work image.

[0056] The instruction-side display control unit 246 controls the instruction-side display unit 26 so that the decoded work image is displayed. An example of the display of the decoded work image will be explained with reference to Figure 4.

[0057] Figure 4 shows an example of the display of the decoded work image on the instructioner-side display unit 26 before an instruction is given. Referring to Figure 4, the instructioner-side display unit 26 has an instructioner-side screen 261. The instructioner-side display control unit 246 controls the instructioner-side display unit 26 so that the decoded work image, M11, is displayed on the instructioner-side screen 261. The work image M11 shows the work space, and the work object 61 that is the target of the work is present in the work space.

[0058] Here, we assume that the work object 61 is a printer, and the worker is attempting to make the work object 61 print on paper by operating the work object 61 (for example, by pressing a button on the printer), and that the supervisor is giving instructions on how to operate the work object 61. However, the work object 61 is not limited to a printer. Furthermore, the work performed by the worker is not limited to operations on the work object 61.

[0059] The supervisor can understand the situation in the work space by viewing the work image M11, and can consider giving instructions that are appropriate to the situation in the work space.

[0060] (After the person giving the instructions begins giving instructions) Next, we will explain what happens after the instructor begins giving instructions. Referring to Figure 4, it can be seen that in the space where the instructor gives instructions (hereinafter also referred to as the "instruction space"), the instructor begins giving instructions by changing the shape of their fingers U1 to the shape of a gesture for operating on the work object 61.

[0061] When the fingers U1 of the person giving the instruction enter the imaging range of the finger imaging unit 22, the finger imaging unit 22 images the fingers U1 of the person giving the instruction, thereby obtaining an image of the fingers.

[0062] (Finger detection unit 241) The finger detection unit 241 acquires finger image footage obtained by the finger imaging unit 22. Based on the finger image footage, the finger detection unit 241 functions as an example of a detection unit that detects multiple three-dimensional coordinates through which the finger U1 passes. These multiple three-dimensional coordinates through which the finger U1 passes may be three-dimensional coordinates indicating each position of the finger U1's skeleton. Hereinafter, these multiple three-dimensional coordinates through which the finger U1 passes will also be referred to as "three-dimensional skeletal information of the finger." The finger detection unit 241 stores the three-dimensional skeletal information of the finger and a timestamp indicating the current time in the storage unit 249.

[0063] The three-dimensional skeletal information of the fingers may be detected in any way. For example, the three-dimensional skeletal information of the fingers may be detected using Leap Motion®, developed by Ultraleap Inc. Leap Motion® is a technology that tracks hand movements based on the detection results of an infrared stereo camera using infrared light emitted from multiple LEDs (Light Emitting Diodes) onto the hand.

[0064] Now, with reference to Figure 5, we will explain the three-dimensional skeletal information of the fingers.

[0065] Figure 5 shows an example of three-dimensional skeletal information of fingers detected by the finger detection unit 241. Referring to Figure 5, the right hand Rh of the person giving the instruction is present in the finger image obtained by the finger imaging unit 22. As shown in Figure 5, the finger detection unit 241 detects the three-dimensional coordinates of each vertex of the right hand Rh in the instruction space as an example of three-dimensional skeletal information of the fingers, based on the finger image.

[0066] Specifically, the vertices whose three-dimensional coordinates are detected by the finger detection unit 241 may include fingertips, finger joints, and vertices of polygons that make up the palm. Figure 5 shows the tip P1 of the thumb of the right hand Rh as an example of a vertex.

[0067] (Finger determination unit 242) The finger determination unit 242 functions as an example of a identification unit that identifies the finger used to give instructions for a task on the instructor's finger U1, based on the three-dimensional skeletal information of the finger detected by the finger detection unit 241. The finger used to give instructions for a task may be an example of a part of the body involved in the task.

[0068] More specifically, the finger determination unit 242 obtains three-dimensional skeletal information of the fingers stored in the memory unit 249, which is associated with a timestamp indicating a time when the elapsed time up to the current time is less than a predetermined time. Then, based on the obtained three-dimensional skeletal information of the fingers, the finger determination unit 242 identifies the finger U1 of the person giving the instruction that is used for instruction. In the example shown in Figure 4, the finger used for instruction is the index finger of the right hand of the person giving the instruction.

[0069] Various methods can be applied to identify which finger is used for instruction. As an example, the finger determination unit 242 estimates the degree of curvature of each of the multiple fingers that make up the instructioner's finger U1 based on the three-dimensional skeletal information of the finger, and identifies the finger used for instruction based on the degree of curvature of each of the multiple fingers. Here, we assume that the multiple fingers that make up the instructioner's finger U1 are the thumb, index finger, middle finger, ring finger, and little finger.

[0070] For example, since an instructor tends to extend the fingers used to give instructions for a task, fingers with a higher degree of bend are likely to be less related to the task. Therefore, the finger determination unit 242 may identify the fingers that make up the instructor's fingers U1 and whose degree of bend is below a threshold as the fingers used for giving instructions.

[0071] The degree of bending of each finger can be calculated using the degree of bending of each finger joint. Referring to Figure 6, an example of a method for calculating the degree of bending of the first joint of the index finger will be explained.

[0072] In the following explanation, the Euclidean norm of vector A is denoted as ||A||, and the normalized vector of vector A is denoted as normalize(A). In this case, ||normalize(A)||=1 and normalize(A)×||A||=A. Also, in the following explanation, the dot product of vector A and vector B is denoted as dot(A,B).

[0073] Figure 6 is a diagram illustrating an example of a method for calculating the degree of curvature of the first joint of the index finger. Referring to Figure 6, the three-dimensional coordinates E of the tip of the index finger, J of the first joint of the index finger, and R of the second joint of the index finger are shown. The finger determination unit 242 can calculate the dot product ip shown in the following equation (1) as the degree of curvature of the first joint of the index finger.

[0074] ip:=dot(normalize(EJ),normalize(RJ)) ...(1)

[0075] The finger determination unit 242 determines whether the dot product ip calculated in this way and the predetermined threshold thrbend satisfy the following equation (2).

[0076] ip>thrbend ···(2)

[0077] For example, the value of the threshold thrbend may be cos(135°), but it may also be any other value. The finger determination unit 242 calculates the degree of bending of the second joint of the index finger, the degree of bending of the first and second joints of the middle finger, ring finger, and little finger, and the degree of bending of the first joint of the thumb, similar to the degree of bending of the first joint of the index finger.

[0078] The finger determination unit 242 determines whether at least one of the bending degrees of the first and second joints of the index finger exceeds the threshold thrbend. The finger determination unit 242 identifies the index finger as the finger to be used for pointing if both the bending degrees of the first and second joints of the index finger are below the threshold thrbend. On the other hand, the finger determination unit 242 does not identify the index finger as the finger to be used for pointing if at least one of the bending degrees of the first and second joints of the index finger exceeds the threshold thrbend.

[0079] Similar to the index finger, the middle, ring, and little fingers are identified as fingers used for pointing if the degree of curvature of both the first and second joints is below the threshold thrbend, and are not identified as fingers used for pointing if at least one of the degrees of curvature of the first and second joints exceeds the threshold thrbend.

[0080] On the other hand, the finger determination unit 242 determines whether the degree of bending of the first joint of the thumb exceeds the threshold thrbend. If the degree of bending of the first joint of the thumb is less than or equal to the threshold thrbend, the finger determination unit 242 identifies the thumb as the finger to be used for pointing. On the other hand, if the degree of bending of the first joint of the thumb exceeds the threshold thrbend, the finger determination unit 242 does not identify the thumb as the finger to be used for pointing.

[0081] (Decision result output unit 243) The judgment result output unit 243 acquires the three-dimensional skeletal information of the fingers detected by the finger detection unit 241 and the information indicating the finger used for instruction, which has been identified by the finger determination unit 242. For example, if the finger used for instruction is the index finger, the information indicating the finger used for instruction is the information indicating the index finger. The judgment result output unit 243 then controls the communication unit 248 so that the three-dimensional skeletal information of the fingers and the information indicating the finger used for instruction are transmitted to the operator-side device 14 by the communication unit 248.

[0082] (Judgment result acquisition unit 141) In the worker-side system 10, the judgment result acquisition unit 141 acquires three-dimensional skeletal information of the fingers and information indicating the finger to be used for instruction, which are received by the communication unit 148.

[0083] (Operator-side display control unit 144) The operator-side display control unit 144 generates a three-dimensional finger model based on the three-dimensional skeletal information of the fingers, and generates a two-dimensional finger image by projecting the generated three-dimensional finger model using a predetermined projection formula. Such a finger image may be an example of an image corresponding to an object involved in the work. For example, such a projection formula may be an formula that represents orthogonal projection (parallel projection). The operator-side display control unit 144 then superimposes the generated finger image onto the work image obtained by the work image capture unit 12, and displays the finger image superimposed on the work image on the operator-side display unit 16.

[0084] Figure 7 shows an example of display by the worker-side display unit 16 of a hand image superimposed on a work image. As shown in Figure 7, the worker-side display control unit 144 superimposes the generated hand image U2 onto the work image M12 obtained by the work image unit 12, and displays the hand image U2 superimposed on the work image M12 on the worker-side display unit 16.

[0085] (encoding unit 142) The encoding unit 142 functions as an example of a calculation unit that calculates two-dimensional coordinates on the operator-side screen 161 (Figure 7) that correspond to the three-dimensional coordinates passing through the finger used for instruction, based on the three-dimensional skeletal information of the fingers and the information indicating the finger used for instruction.

[0086] More specifically, the encoding unit 142 calculates the two-dimensional coordinates on the operator-side screen 161 that correspond to the three-dimensional coordinates of the fingertip used for instruction. For example, if the information indicating the finger used for instruction is information indicating the index finger, the encoding unit 142 calculates the two-dimensional coordinates on the operator-side screen 161 that correspond to the three-dimensional coordinates of the fingertip constituting the index finger.

[0087] Furthermore, the conversion formula for converting the three-dimensional coordinates passing through the finger used for instruction into two-dimensional coordinates corresponding to the worker-side screen 161 may be the same as the projection formula for projecting a three-dimensional finger model onto a two-dimensional finger image. Also, in the following explanation, the two-dimensional coordinates on the worker-side screen 161 that correspond to the three-dimensional coordinates passing through the finger used for instruction will also simply be referred to as the "two-dimensional coordinates of the finger used for instruction."

[0088] The encoding unit 142 functions as an example of an estimation unit that estimates an area on the operator-side screen 161 corresponding to the two-dimensional coordinates of the finger used for instruction as an important area and obtains information indicating the important area. For example, the encoding unit 142 may estimate the important area by taking into account the position of the finger used for instruction. More specifically, the encoding unit 142 may estimate the important area as an area on the operator-side screen 161 with the position indicated by the two-dimensional coordinates of the finger used for instruction as the central position (hereinafter also referred to as the "pre-correction area").

[0089] Alternatively, the encoding unit 142 may estimate the important region by taking into account not only the position of the finger used for instruction but also the direction of the finger used for instruction. More specifically, the encoding unit 142 may calculate the two-dimensional direction on the operator-side screen 161 corresponding to the three-dimensional direction of the finger used for instruction (hereinafter also referred to as the "two-dimensional direction of the finger used for instruction") based on multiple three-dimensional coordinates passing through the finger used for instruction, and estimate the region on the operator-side screen 161 corresponding to the two-dimensional coordinates and the two-dimensional direction of the finger used for instruction (hereinafter also referred to as the "corrected region") as the important region.

[0090] The following section describes an example of a method for estimating the important region, which includes both the pre-correction and post-correction regions, with reference to Figures 8 to 10. While this section primarily describes the estimation of the important region when the finger used for pointing is the index finger, the important region can be estimated in the same way when a finger other than the index finger is used for pointing.

[0091] Figure 8 is a diagram illustrating the estimation of the pre-correction region. As shown in Figure 8, it is assumed that the working image M12 is divided into multiple regions. Figure 8 shows an example where each region is a 16x16 pixel rectangular area. As a result of this division, the working image M12 is divided vertically into 6 sections and horizontally into 8 sections, resulting in a total of 48 rectangular regions. However, the size of the rectangular regions and the number of divisions of the working image M12 are not limited to this example.

[0092] Here, let E be the three-dimensional coordinate of the tip of the index finger, which is used as an example of a finger used for instruction, and let proj() be the projection formula. At this time, the encoding unit 142 can calculate the two-dimensional coordinate F of the tip of the index finger in the work image M12, which corresponds to the three-dimensional coordinate E of the tip of the index finger, as shown in equation (3) below.

[0093] F:=proj(E) ···(3)

[0094] The encoding unit 142 estimates the region in the work image M12 as the pre-correction region R1, with the position indicated by the two-dimensional coordinate F of the tip of the index finger in the work image M12 as the central position.

[0095] In the example shown in Figure 8, a rectangular area consisting of a 3x3 grid centered on a single region containing the tip of the index finger is estimated as the pre-correction region R1. However, the vertical and horizontal sizes of the pre-correction region R1 are not limited to this example. Furthermore, the encoding unit 142 does not include the area outside the working image M12 in the pre-correction region R1.

[0096] Figure 9 is a diagram illustrating the estimation of the corrected region. Here, J is the three-dimensional coordinate of the first joint of the index finger, which is used as an example of a finger used for instruction. At this time, the encoding unit 142 can calculate the two-dimensional coordinate of the first joint of the index finger in the work image M12, which corresponds to the three-dimensional coordinate J of the first joint of the index finger, as proj(J).

[0097] Let D be the two-dimensional direction on the operator-side screen 161 that corresponds to the three-dimensional direction of the index finger. At this time, the encoding unit 142 can calculate the two-dimensional direction D of the index finger, as an example of a finger used for instruction, based on the two-dimensional coordinates proj(E) of the tip of the index finger and the two-dimensional coordinates proj(J) of the first joint of the index finger, as shown in equation (4) below.

[0098] D:=normalize(proj(E)-proj(J)) ···(4)

[0099] Furthermore, the encoding unit 142 can calculate the position T (hereinafter also referred to as the "shifted two-dimensional coordinate of the tip of the index finger") obtained by shifting the position indicated by the two-dimensional coordinate proj(E) of the tip of the index finger by a predetermined shift amount H in the two-dimensional direction D of the index finger, as shown in the following equation (5).

[0100] T:=proj(E)+D×H ···(5)

[0101] The shift amount H can be determined in any way. For example, the shift amount H may be set to the size of each region into which the work image M12 is divided (e.g., 16 pixels). Alternatively, the shift amount H may be set to be larger the longer the index finger is in the work image M12. As an example, the length of the index finger in the work image M12 may be the distance between the two-dimensional coordinate proj(E) of the tip of the index finger and the two-dimensional coordinate proj(J) of the first joint of the index finger.

[0102] Here, it is assumed that not only the pre-correction region R1 (Figure 8), but also the region corresponding to the two-dimensional coordinate T after the shift of the tip of the index finger, is an important region that the person giving the instruction should pay attention to. Therefore, the encoding unit 142 estimates the region in the work image M12 with the two-dimensional coordinate T after the shift of the tip of the index finger as the central position as the corrected region R2.

[0103] In the example shown in Figure 9, the corrected region R2 is estimated to be a rectangular area consisting of a 3x3 grid centered on a single region that includes the tip of the index finger after shifting in the pointing direction. However, the vertical and horizontal sizes of the corrected region R2 are not limited to this example. Furthermore, the encoding unit 142 does not include the area outside the working image M12 in the corrected region R2.

[0104] Figure 10 is a diagram illustrating the estimation of the critical region, which includes the pre-correction region R1 and the post-correction region R2. Referring to Figure 10, the pre-correction region R1 and the post-correction region R2 are shown. The encoding unit 142 estimates the critical region R3 by adding the pre-correction region R1 and the post-correction region R2. This allows for a more comprehensive estimation of the critical region R3 that the person giving the instruction should pay attention to in the work image M12. If multiple fingers are used for instruction, the encoding unit 142 can estimate the critical region for each of the fingers used for instruction and then obtain the final critical region by adding up the estimated critical regions.

[0105] The encoding unit 142 functions as an example of an encoding processing unit that performs encoding on the work image M12 based on information indicating important regions to obtain the encoded work image M12. More specifically, the encoding unit 142 makes the compression ratio of the non-important regions of the work image M12, which are regions other than the important region R3, higher than the compression ratio of the encoding on the important region R3. The specific encoding methods for the important region and the non-important region are not particularly limited.

[0106] (Transmission control unit 143) The transmission control unit 143 controls the communication unit 148 so that the encoded work image M12 obtained by the encoding unit 142 and information indicating the important region R3 are transmitted to the instructioner-side device 24 via the network 30. By transmitting the encoded work image M12 in this way, it is possible to reduce the communication load that the work image M12 places on the network 30 while suppressing the deterioration of the quality of the important region R3 in the work image M12.

[0107] (Encoded data acquisition unit 244) The encoded data acquisition unit 244 functions as an example of an acquisition unit that acquires the encoded working image M12 and information indicating the important region R3 received by the communication unit 248.

[0108] (Decoding unit 245) The decoding unit 245 decodes the encoded work image M12 acquired by the encoded data acquisition unit 244 to obtain the decoded work image M12. More specifically, based on the encoded work image M12 and information indicating the important region R3, the decoding unit 245 performs decoding on non-important regions, which are regions other than the important region R3, corresponding to the encoding for the non-important regions, and performs decoding on the important region R3 corresponding to the encoding for the important region R3.

[0109] (Instructor-side display control unit 246) The instructor-side display control unit 246 displays the decoded work image M12 obtained by the decoding unit 245 on the instructor-side display unit 26. The instructor-side display control unit 246 also generates a three-dimensional finger model based on the three-dimensional skeletal information of the fingers obtained by the finger detection unit 241, and generates a two-dimensional finger image by projecting the generated three-dimensional finger model using a predetermined projection formula. The projection by the instructor-side display control unit 246 may be performed in the same manner as the projection by the worker-side display control unit 144.

[0110] Then, the user-side display control unit 246 superimposes the generated finger video onto the decoded work image M12 obtained by the decoding unit 245, and displays the finger video superimposed on the decoded work image M12 on the user-side display unit 26.

[0111] Figure 11 shows an example of the display by the instructor's display unit 26 of a finger image superimposed on the decoded work image M12. As shown in Figure 11, the instructor's display control unit 246 superimposes the generated finger image U2 onto the decoded work image M12 obtained by the decoding unit 245, and displays the finger image U2 superimposed on the work image M12 on the instructor's display unit 26. Referring to Figure 11, the instructor's finger U1 located in the instruction space is shown.

[0112] As described above, the compression ratio applied to the non-critical region of the working image M12, which is the region other than the critical region R3, is higher than the compression ratio applied to the critical region R3. Therefore, as shown in Figure 11, the non-critical region of the decoded working image M12, which is the region other than the critical region R3, is of "low quality," while the critical region R3 is of "high quality."

[0113] The above describes an example of the configuration of the remote work support system 1 according to an embodiment of the present invention.

[0114] (1-2. Operation of the Remote Work Support System) Next, with reference to Figures 12 and 13 (and Figures 1 to 11 as appropriate), an example of the operation of the remote work support system 1 according to an embodiment of the present invention will be described. First, with reference to Figure 12, an example of the operation of the worker-side system 10 will be described, and then, with reference to Figure 13, an example of the operation of the instructor-side system 20 will be described.

[0115] Figure 12 is a flowchart showing an example of the operation of the worker-side system 10. In the worker-side system 10, the work imaging unit 12 obtains a work image by imaging the work space, which is the space in which the worker performs work. The worker-side display control unit 144 acquires the work image obtained by the work imaging unit 12 (S101) and displays the work image on the worker-side display unit 16 (S102).

[0116] The encoding unit 142 encodes the work image based on the information indicating the important region to obtain the encoded work image (S103). More specifically, the encoding unit 142 makes the compression ratio of the non-important region, which is the region of the work image M12 other than the important region, higher than the compression ratio of the encoding of the important region. The transmission control unit 143 controls the communication unit 148 so that the encoded work image obtained by the encoding unit 142 and the information indicating the important region are transmitted to the instructioner-side device 24 via the network 30 (S104).

[0117] If the communication unit 148 does not receive the three-dimensional skeletal information of the fingers and the information indicating the finger to be used for instruction (NO in S110), the operation proceeds to S101. On the other hand, if the communication unit 148 does receive the three-dimensional skeletal information of the fingers and the information indicating the finger to be used for instruction (YES in S110), the judgment result acquisition unit 141 acquires the three-dimensional skeletal information of the fingers and the information indicating the finger to be used for instruction that was received by the communication unit 148.

[0118] The operator-side display control unit 144 generates a three-dimensional finger model based on the three-dimensional skeletal information of the fingers, and generates a two-dimensional finger image by projecting the generated three-dimensional finger model using a predetermined projection formula. The operator-side display control unit 144 then superimposes the generated finger image onto the work image captured on the operator-side display unit 16 (S111).

[0119] The encoding unit 142 calculates two-dimensional coordinates on the operator-side screen 161 that correspond to the three-dimensional coordinates passing through the finger used for instruction, based on the three-dimensional skeletal information of the fingers and the information indicating the finger used for instruction. The encoding unit 142 estimates the area on the operator-side screen 161 corresponding to the two-dimensional coordinates of the finger used for instruction as an important area and obtains information indicating the important area (S112). Then, the operation proceeds to S101.

[0120] Figure 13 is a flowchart showing an example of the operation of the instruction-side system 20. In the instruction-side system 20, the communication unit 248 receives the encoded work image and information indicating important areas (S201), and the encoded data acquisition unit 244 acquires the encoded work image and information indicating important areas received by the communication unit 248.

[0121] The decoding unit 245 decodes the encoded work image acquired by the encoded data acquisition unit 244 to obtain the decoded work image (S202). More specifically, based on the encoded work image and information indicating important regions, the decoding unit 245 performs decoding corresponding to the encoding for non-important regions, which are regions other than the important regions, and performs decoding corresponding to the encoding for important regions, for the important regions.

[0122] The instructioner-side display control unit 246 displays the decoded work image obtained by the decoding unit 245 on the instructioner-side display unit 26 (S203). Next, the finger imaging unit 22 obtains a finger image by imaging the instructioner's fingers (S204). The finger detection unit 241 acquires the finger image obtained by the finger imaging unit 22. Based on the finger image, the finger detection unit 241 attempts to detect the three-dimensional skeletal information of the fingers, which is the multiple three-dimensional coordinates that the instructioner's fingers pass through.

[0123] If the finger detection unit 241 does not detect three-dimensional skeletal information of the fingers (NO in S210), the operation proceeds to S201.

[0124] On the other hand, if the finger detection unit 241 detects three-dimensional skeletal information of the fingers (YES in S210), the user-side display control unit 246 generates a three-dimensional finger model based on the three-dimensional skeletal information of the fingers, and generates a two-dimensional finger image by projecting the generated three-dimensional finger model using a predetermined projection formula. The user-side display control unit 246 then superimposes the generated finger image onto the decoded work image displayed on the user-side display unit 26 (S211).

[0125] The finger determination unit 242 identifies the finger used to give instructions for a task on the user's hand based on the three-dimensional skeletal information of the fingers detected by the finger detection unit 241, and obtains information indicating the finger used for the instruction (S212).

[0126] The judgment result output unit 243 acquires the three-dimensional skeletal information of the fingers detected by the finger detection unit 241 and the information indicating the finger to be used for instruction, which has been identified by the finger determination unit 242. The judgment result output unit 243 then controls the communication unit 248 so that the three-dimensional skeletal information of the fingers and the information indicating the finger to be used for instruction are transmitted to the operator-side device 14 by the communication unit 248 (S213). Then the operation proceeds to S201.

[0127] The above describes an example of the operation of the remote work support system 1 according to an embodiment of the present invention.

[0128] (1-3. Effects) As described above, according to an embodiment of the present invention, a remote work support system 1 is provided, comprising a finger detection unit 241, a finger determination unit 242, an encoding unit 142, a transmission control unit 143, an encoded data acquisition unit 244, a decoding unit 245, and an instructor-side display control unit 246.

[0129] The finger detection unit 241 detects multiple three-dimensional coordinates through which the instructor's fingers pass, based on the finger image captured by the finger imaging unit 22. The finger determination unit 242 identifies the finger used to give instructions for the task on the instructor's finger, based on the multiple three-dimensional coordinates.

[0130] The encoding unit 142 calculates two-dimensional coordinates on the worker-side screen 161, which is visible to the worker, that correspond to the three-dimensional coordinates through which the finger used for instruction passes. It then estimates the area on the worker-side screen 161 corresponding to the two-dimensional coordinates as an important area and obtains information indicating the important area. Based on the information indicating the important area, the encoding unit 142 encodes the work image captured in the work space where the work is performed to obtain the encoded image.

[0131] The transmission control unit 143 controls the transmission of the encoded work image. The encoded data acquisition unit 244 acquires the encoded work image. The decoding unit 245 decodes the encoded work image to obtain the decoded work image. The instructor-side display control unit 246 displays the decoded work image on the instructor-side screen 261, which is visible to the instructor.

[0132] With this configuration, it is possible to reduce the communication load on the network from the work image obtained by imaging the work space, while suppressing the degradation of quality in important areas of the work image.

[0133] The effects of the remote work support system 1 according to the embodiment of the present invention have been described above.

[0134] (2. Hardware Configuration Example) Next, an example of the hardware configuration of the operator-side system 10 according to an embodiment of the present invention will be described.

[0135] In the following, an example of the hardware configuration of the information processing device 900 will be described as an example of the hardware configuration of the worker-side system 10 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the worker-side system 10. Therefore, the hardware configuration of the worker-side system 10 may be modified by removing unnecessary components from the hardware configuration of the information processing device 900 described below, or by adding new components. Note that the hardware configuration of the instructor-side system 20 can also be implemented in the same way as the hardware configuration of the worker-side system 10.

[0136] Figure 14 shows the hardware configuration of an information processing device 900 as an example of a worker-side system 10 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0137] The CPU 901 functions as both an arithmetic processing unit and a control unit, controlling the overall operation of the information processing unit 900 according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. These are interconnected by a host bus 904, which consists of a CPU bus and other components.

[0138] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not always necessary to configure the host bus 904, bridge 905, and external bus 906 separately; these functions may be implemented on a single bus.

[0139] The input device 908 consists of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers, and an input control circuit that generates input signals based on the user's input and outputs them to the CPU 901. The user operating the information processing device 900 can input various types of data to the information processing device 900 or instruct it to perform processing operations by operating this input device 908.

[0140] The output device 909 includes, for example, display devices such as CRT (Cathode Ray Tube) display devices, liquid crystal display (LCD) devices, OLED (Organic Light Emitting Diode) devices, lamps, and audio output devices such as speakers.

[0141] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). This storage device 910 drives the hard disk and stores programs executed by the CPU 901 and various data.

[0142] The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network. The communication device 911 may support either wireless or wired communication.

[0143] The above describes an example of the hardware configuration of the operator-side system 10 according to an embodiment of the present invention.

[0144] (3. Summary) Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these examples. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to fall within the technical scope of the present invention.

[0145] For example, in the above, the example of an object involved in the work was mainly assumed to be the fingers of the person giving instructions for the work. However, instead of the fingers of the person giving instructions, the fingers of the worker may be used as the object involved in the work. In this case, the finger imaging unit 22 may be provided in the worker-side system 10 instead of the person giving instructions system 20. The finger imaging unit 22 will then image the fingers of the worker, and the finger detection unit 241 will detect the three-dimensional skeletal information of the worker's fingers using the same method as the method used to detect the three-dimensional skeletal information of the fingers of the person giving instructions.

[0146] Furthermore, the objects involved in the work are not limited to the hands. That is, the objects involved in the work may include objects used by the worker (first object), or objects used by the supervisor giving instructions for the work (second object). The objects involved in the work may also include various tools used by the worker or supervisor (for example, tools used for the work (screwdriver, etc.), medical instruments, etc.). Alternatively, the objects involved in the work may include objects used by a robot.

[0147] Furthermore, the above mainly described the case in which the finger determination unit 242 identifies a finger that satisfies a predetermined condition (specifically, the condition that the degree of curvature is below a threshold) as the finger to be used for instruction. However, the finger determination unit 242 may also identify the finger to be used for instruction based on three-dimensional skeletal information of the finger and a trained model obtained by learning using a predetermined machine learning algorithm.

[0148] For example, the learning using a machine learning algorithm may be machine learning using a neural network (e.g., deep learning). The pre-trained model does not need to be updated after the remote work support system 1 is put into use, or it may be updated as appropriate during the use of the remote work support system 1.

[0149] For example, the instructor or worker may input correct data for the body part being worked on into the remote work support system 1 while referring to the captured video of their fingers. In this case, the remote work support system 1 can perform additional training on a pre-trained model based on the correct data for the body part being worked on and the captured video of their fingers. This can improve the accuracy of the model even while the remote work support system 1 is in use.

[0150] Furthermore, the above primarily assumed a case where the work image is divided into multiple rectangular regions. However, the shape of each of the multiple regions into which the work image is divided is not limited to rectangular regions. Also, as mentioned above, the size of each of the multiple rectangular regions into which the work image is divided is not limited to 16x16 pixels.

[0151] Furthermore, the above mainly assumes that the finger detection unit 241 and the finger determination unit 242 are provided on the instructor-side device 24. However, at least one of the finger detection unit 241 and the finger determination unit 242 may be provided on the worker-side device 14, or on a server (not shown).

[0152] Furthermore, at least one of the calculation unit and the estimation unit, which calculate the two-dimensional coordinates of the fingers used for instruction, of the encoding unit 142 may also be provided on the operator-side device 14 or on a server (not shown).

[0153] In the above, the hand video displayed on the instructor-side display unit 26 is primarily assumed to be superimposed on the work image by the instructor-side display control unit 246. With this configuration, the hand video does not need to be transmitted from the worker-side system 10 to the instructor-side system 20, which reduces the communication load on the network 30. In addition, the instructor-side system 20 can independently decide whether or not to superimpose the hand video.

[0154] However, the finger video displayed on the instructor-side display unit 26 may be superimposed on the work video by the worker-side display control unit 144. With this configuration, the same finger video frame is superimposed on the same work video frame in both the worker-side system 10 and the instructor-side system 20. In other words, this configuration has the effect of eliminating the time difference between the finger video frames superimposed on the same work video frame in both the worker-side system 10 and the instructor-side system 20. [Explanation of symbols]

[0155] 1. Remote work support system 10. Worker-side system 12 Working Imaging Unit 14. Operator-side equipment 140 Control Unit 141 Judgment result acquisition part 142 Encoding section 143 Transmission Control Unit 144 Operator-side display control unit 148 Communications Department 149 Memory section 16 Operator side display section 161 Operator's screen 20. Instructor-side system 22 Finger imaging section 24 Instructor side device 240 Control Unit 241 Finger detection unit 242 Hand and finger determination section 243 Judgment Result Output Unit 244 Encoded Data Acquisition Unit 245 Decoding section 246 Instructor-side display control unit 248 Communications Department 249 Storage section 26 Instructor side display section 261 Instructor side screen 30 Networks

Claims

1. A detection unit that detects multiple three-dimensional coordinates through which an object undergoing work passes, based on sensor data obtained from the object through a sensor, A identifying unit that identifies the part of the object to which the work is performed based on the plurality of three-dimensional coordinates, A calculation unit that calculates two-dimensional coordinates on the worker's screen, as seen by the worker, corresponding to the three-dimensional coordinates passing through the part involved in the aforementioned work, An estimation unit that estimates the area on the operator's screen corresponding to the two-dimensional coordinates as an important area and obtains information indicating the important area, An encoding processing unit performs encoding on an image of the workspace where the work is performed, based on the information indicating the important area, to obtain an encoded image. A transmission control unit that controls the transmission of the encoded image after capturing, An acquisition unit that acquires the encoded image, A decoding unit that decodes the encoded image to obtain a decoded image, An instruction-side display control unit that displays the decoded captured image on a screen on the instructioner's side, which is visible to the instructioner who gives instructions to the worker, A remote work support system equipped with the following features.

2. The encoding processing unit makes the compression ratio applied to the non-critical region, which is a region other than the critical region, of the captured image in which the workspace is captured, higher than the compression ratio applied to the critical region. The remote work support system according to claim 1.

3. The estimation unit estimates the important region as the area on the worker's screen, with the central position being the position indicated by the two-dimensional coordinates on the worker's screen that correspond to the three-dimensional coordinates through which the part involved in the work passes. The remote work support system according to claim 1.

4. The calculation unit calculates the two-dimensional direction on the operator's screen corresponding to the three-dimensional direction of the part being worked on, based on a plurality of three-dimensional coordinates through which the part being worked on passes. The estimation unit estimates the area on the operator's screen corresponding to the two-dimensional coordinates and the two-dimensional direction as the important area. The remote work support system according to claim 1.

5. The specified unit estimates the degree of curvature of each of the multiple parts constituting the object involved in the work based on the multiple three-dimensional coordinates, and identifies the part involved in the work based on the degree of curvature of each of the multiple parts. The remote work support system according to claim 1.

6. The specified part identifies the part among the plurality of parts in which the degree of curvature is below a threshold as the part to be worked on. The remote work support system according to claim 5.

7. The specified unit identifies the part of the body involved in the work based on the plurality of three-dimensional coordinates and a trained model obtained by learning using a predetermined machine learning algorithm. The remote work support system according to claim 1.

8. The remote work support system is The system comprises a worker-side system used by the worker and a supervisor-side system used by the supervisor, The aforementioned worker-side system is: The system comprises the encoding processing unit and the transmission control unit, The aforementioned instruction-side system is: The system comprises the acquisition unit, the decoding unit, and the instructioner-side display control unit. The remote work support system according to claim 1.

9. The instructioner-side display control unit displays the image corresponding to the object involved in the operation, which is superimposed on the decoded captured image, on the instructioner-side screen. The remote work support system according to claim 1.

10. The remote work support system is The system includes a worker-side display control unit that displays an image on the worker's screen that corresponds to an object related to the work, superimposed on an image of the workspace captured. The remote work support system according to claim 1.

11. The object involved in the aforementioned work is the fingers of the worker or the person giving the instruction. The part involved in the aforementioned work is one or more fingers that make up the fingers. A remote work support system according to any one of claims 1 to 10.

12. The object involved in the aforementioned work includes the first object used by the worker, A remote work support system according to any one of claims 1 to 10.

13. The object involved in the aforementioned work includes a second object used by the person giving the instructions. A remote work support system according to any one of claims 1 to 10.

14. Based on sensor data obtained from the object involved in the work, the system detects multiple three-dimensional coordinates through which the object passes. Based on the aforementioned plurality of three-dimensional coordinates, the part of the object to which the work is performed is identified, To calculate the two-dimensional coordinates on the worker's screen, as seen by the worker, that correspond to the three-dimensional coordinates passing through the part involved in the aforementioned work, The area on the operator's screen corresponding to the two-dimensional coordinates is estimated as an important area, and information indicating the important area is obtained. Based on the information indicating the important region, encoding is performed on the captured image of the workspace where the work is performed to obtain the encoded captured image. Controlling the transmission of the encoded image after capturing, To acquire the encoded image, The above-mentioned encoded image is decoded to obtain the decoded image. The decoded image is displayed on a screen on the supervisor's side, which is visible to the supervisor giving instructions to the worker. A remote work support method that includes the following features.

15. Computers, A detection unit that detects multiple three-dimensional coordinates through which an object undergoing work passes, based on sensor data obtained from the object through a sensor, A identifying unit that identifies the part of the object to which the work is performed based on the plurality of three-dimensional coordinates, A calculation unit that calculates two-dimensional coordinates on the worker's screen, as seen by the worker, corresponding to the three-dimensional coordinates passing through the part involved in the aforementioned work, An estimation unit that estimates the area on the operator's screen corresponding to the two-dimensional coordinates as an important area and obtains information indicating the important area, An encoding processing unit performs encoding on an image of the workspace where the work is performed, based on the information indicating the important area, to obtain an encoded image. A transmission control unit that controls the transmission of the encoded image after capturing, An acquisition unit that acquires the encoded image, A decoding unit that decodes the encoded image to obtain a decoded image, An instruction-side display control unit that displays the decoded captured image on a screen on the instructioner's side, which is visible to the instructioner who gives instructions to the worker, A program that functions as a remote work support system equipped with the necessary features.

Images