Work support device, work support method, and program

Abstract

To solve a problem such that when a user attempts to manipulate a virtual object, there may be a case where the user touches a structure existing in a real space.SOLUTION: A work support device is provided that includes: a structure detection unit for detecting a position of a structure in a real world; an initial position setting unit for setting an initial value of a display position in a three-dimensional space of a virtual object; an interference determination unit for determining whether or not an operator touches the structure in the real world based on a position of the structure and a display position of the virtual object when the operator tries to manipulate the virtual object; and a position adjustment unit for adjusting the display position of the virtual object based on determination results in the interference determination unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0007] ,

[0001] The present invention relates to a work support device, a work support method, and a program.

[0002] Conventionally, augmented reality (AR) technology or mixed reality (MR) technology that overlays and displays virtual objects on an image showing the real space has been known (see, for example, Patent Document 1). Patent Document 1 Japanese Patent Application Laid-Open No. 2018-63567

Summary of the Invention

Problems to be Solved by the Invention

[0003] When a user tries to operate a virtual object, the user may come into contact with a structure existing in the real space.

Means for Solving the Problems

[0004] In order to solve the above problems, in a first aspect of the present invention, a work support device is provided. The work support device may include a display unit that displays an image of a three-dimensional space including a structure in the real world and a virtual object operated by an operator. The work support device may include a structure detection unit that detects the position of the structure in the real world. The work support device may include an initial position setting unit that sets an initial value of the display position of the virtual object in the three-dimensional space. The work support device may include an interference determination unit that determines whether the operator contacts the structure in the real world when the operator tries to operate the virtual object, based on the position of the structure and the display position of the virtual object. The work support device may include a position adjustment unit that adjusts the display position of the virtual object based on the determination result in the interference determination unit.

[0005] The interference determination unit may determine whether the position of the structure and the display position of the virtual object overlap.

[0006] The interference determination unit may determine whether the passage area through which the operator passes to operate the virtual object and the position of the structure overlap.

[0007] The work support device may include a gesture detection unit that detects the operator's gestures. The initial position setting unit may set the initial value of the display position of the virtual object based on the operator's gestures.

[0008] The position adjustment unit may adjust the display position based on operator information relating to the operator. Operator information may include static information that does not change over time regarding the operator. Operator information may also include dynamic information that changes over time regarding the operator.

[0009] The position adjustment unit may adjust the display position based on structural information relating to the structure.

[0010] The position adjustment unit may adjust the display position based on virtual object information relating to the virtual object.

[0011] The position adjustment unit may adjust the orientation of the virtual object based on the judgment result from the interference determination unit.

[0012] A second aspect of the present invention provides a work support method using a display unit that displays an image of a three-dimensional space including a structure in the real world and a virtual object operated by an operator. The work support method may detect the position of the structure in the real world. The work support method may set an initial value for the display position of the virtual object in three-dimensional space. When the operator attempts to operate the virtual object, the work support method may determine whether the operator will come into contact with the structure in the real world, based on the position of the structure and the display position of the virtual object. The work support method may adjust the display position of the virtual object based on the determination result.

[0013] In a third aspect of the present invention, a program is provided for causing a computer to execute a work support method according to the second aspect.

[0014] It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. Furthermore, subcombinations of these features may also constitute an invention. [Brief explanation of the drawing]

[0015] [Figure 1] This figure illustrates an example of the use of a work support device 100 according to one embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the functional configuration of the work support device 100. [Figure 3] This diagram illustrates an example of a gesture by operator 30. [Figure 4] This diagram illustrates an example of the operation of the interference detection unit 106. [Figure 5] This diagram illustrates another example of the operation of the interference detection unit 106. [Figure 6] This diagram illustrates another example of the operation of the interference detection unit 106. [Figure 7] This figure shows an example of the operation of the position adjustment unit 108. [Figure 8] This is a flowchart outlining the work support method using the work support device 100. [Figure 9] An example of a computer 2200 in which multiple aspects of work support methods may be embodied, either entirely or partially, is shown. [Modes for carrying out the invention]

[0016] The present invention will be described below through embodiments of the invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0017] FIG. 1 is a diagram for explaining an example of use of a work support device 100 according to an embodiment of the present invention. The work support device 100 superimposes and displays a real image of the real world and a virtual object 20 operated by an operator 30. In the example of FIG. 1, the work support device 100 displays an image of a structure 10 in the real world and the virtual object 20 on a common three-dimensional space. In FIG. 1, the three-dimensional space is shown in an XYZ coordinate system. Note that the operator 30 may or may not be included in the three-dimensional space displayed by the work support device 100. The work support device 100 may display three-dimensional holography or may display a pseudo three-dimensional space using the parallax of the images presented to both eyes. As an example, the work support device 100 includes a wearable terminal of a head-mounted display (HMD) type worn by the operator 30.

[0018] The virtual object 20 in this example is an object for inputting control information for controlling the structure 10, for example. The structure 10 is equipment arranged in a factory or the like, for example. The structure 10 may be manufacturing equipment or may be equipment for other purposes.

[0019] Depending on the position where the virtual object 20 is displayed, when the operator 30 tries to operate the virtual object 20, the operator 30 may contact the structure 10. When the operator 30 operates the virtual object 20 displayed at a predetermined initial position, the work support device 100 in this example determines whether or not the operator 30 contacts the structure 10. For example, the work support device 100 determines that the operator 30 contacts the structure 10 when at least a part of the display position of the virtual object 20 in the three-dimensional space coincides with the position where the structure 10 exists.

[0020] When the work support device 100 determines that the operator 30 contacts the structure 10, the work support device 100 adjusts the display position of the virtual object 20. The adjusted display position of the virtual object 20 may be, for example, a position where the display position of the virtual object 20 and the position where the structure 10 exists do not even partially coincide. By displaying the virtual object 20 at the adjusted position, the possibility that the operator 30 contacts the structure 10 can be reduced.

[0021] FIG. 2 is a block diagram showing a functional configuration example of the work support device 100. The work support device 100 includes a structure detection unit 104, an initial position setting unit 114, an interference determination unit 106, a position adjustment unit 108, and a display unit 122. The work support device 100 may further include at least any one of an image acquisition unit 102, a sensor unit 110, a gesture detection unit 112, a gesture information storage unit 116, a work support information storage unit 118, and a control unit 120. Functions other than the display unit 122, the sensor unit 110, and the image acquisition unit 102 in the work support device 100 may be realized by a computer. Each component of the work support device 100 may be housed in a single housing such as a head-mounted display, or may be distributed and housed in a plurality of housings. Each component may communicate with each other by wire or wirelessly.

[0022] The display unit 122 provides an image of a three-dimensional space including the structure 10 and the virtual object 20 to the operator 30. The display unit 122 may be a liquid crystal display device, an organic EL display device, a hologram display device, or other display device. The control unit 120 controls the display unit 122 to display an image. The control unit 120 may output data of an image to be displayed on the display unit 122.

[0023] The structure detection unit 104 detects the position of the structure 10 in the real world. The structure detection unit 104 may detect the position of the structure 10 in the three-dimensional space displayed by the display unit 122 as the position of the structure 10 in the real world. The structure detection unit 104 may detect the relative position between the structure 10 and the operator 30. The structure detection unit 104 may detect the position of the structure 10 from an image obtained by imaging the structure 10. In this example, the structure detection unit 104 detects the position of the structure 10 from an image of the structure 10 acquired by the image acquisition unit 102. The image acquisition unit 102 is a camera having an image pickup device such as a CCD. The image acquisition unit 102 may acquire distance information from the image acquisition unit 102 to each part of the structure 10. Thereby, the structure detection unit 104 can detect the position of each part of the structure 10 in the three-dimensional space.

[0024] The initial position setting unit 114 sets the initial value of the display position of the virtual object 20 in three-dimensional space. At the stage when the initial value is set, the display unit 122 does not have to display the virtual object 20 in three-dimensional space. The initial position setting unit 114 may set the initial value of the display position of the virtual object 20 when the operator 30 inputs an indication that the virtual object 20 should be displayed. The input from the operator 30 may be a gesture made by the operator 30 or a voice from the operator 30. In another example, the operator 30 may use an input device having a button or the like to input an indication that the virtual object 20 should be displayed. In this example, the work support device 100 detects the hand gestures of the operator 30.

[0025] The sensor unit 110 monitors the movements of at least a part of the operator's body 30. The sensor unit 110 may have a camera that captures images of at least a part of the operator's body, or an acceleration sensor attached to the operator's body 30.

[0026] The gesture detection unit 112 detects an action corresponding to a preset gesture from the actions of the operator 30 detected by the sensor unit 110. The gesture information storage unit 116 stores information for detecting one or more preset gestures. For example, the gesture information storage unit 116 may store information indicating the actions of the operator 30 for each gesture. The gesture information storage unit 116 may also store what kind of input from the operator 30 each gesture represents. The gesture information storage unit 116 stores at least the gestures for displaying the virtual object 20. The work support device 100 may be capable of displaying multiple types of virtual objects 20. Each virtual object 20 provides a different type of control over the structure 10. The gestures for displaying each virtual object 20 may be different. The gesture information storage unit 116 may store the gestures for each virtual object 20. The gesture detection unit 112 may extract gestures corresponding to the actions of the operator 30 from the gesture information storage unit 116.

[0027] The initial position setting unit 114 sets the initial position for displaying the virtual object 20 when the gesture detection unit 112 detects a gesture for displaying the virtual object 20. The initial position of the virtual object 20 may be a fixed relative position with respect to the operator 30 (or the head-mounted display of the work support device 100), regardless of the type of virtual object 20. The initial position of the virtual object 20 may also be set according to information about the operator 30 or the virtual object 20. For example, the initial position setting unit 114 may set the initial position of the virtual object 20 differently depending on whether the operator 30 is right-handed or left-handed. It is preferable that the initial position of the virtual object 20 be set to a position that is easy for the operator 30 to operate.

[0028] The initial position setting unit 114 may also set an initial value for the display position of the virtual object 20 based on the gesture of the operator 30. The initial position setting unit 114 may set the initial position of the virtual object 20 differently depending on whether the operator 30 makes a gesture with their right hand or their left hand. For example, if the operator 30 makes a gesture with their right hand, the initial position setting unit 114 may set the initial position to a position that is easy to operate with the right hand. Furthermore, if the operator 30 selects and operates one of several operable parts of the virtual object 20 by gesture, the initial position of the virtual object 20 may be set according to the position of the selected operable part. In other words, the initial position of the virtual object 20 may be set to a position that makes it easy to operate the operable part.

[0029] Furthermore, the initial position setting unit 114 may set different initial positions for the virtual objects 20 depending on the type of virtual object 20. For example, if the part of the structure 10 to be controlled differs for each type of virtual object 20, the initial position of the virtual object 20 may be set near the respective part of the structure to be controlled. The initial position setting unit 114 may extract the initial position of the virtual object 20 specified by the gesture from the work support information storage unit 118.

[0030] The work support information storage unit 118 may store the shape of each virtual object 20 in three-dimensional space, the display content of the operated part included in the virtual object 20, and the initial position of the virtual object 20. The operated part refers to the part operated by the operator 30. The operated part may include, for example, buttons, levers, etc. The initial position of the virtual object 20 may be the position of the entire area occupied by the virtual object 20 in three-dimensional space. The information indicating the initial position of the virtual object 20 may be the coordinates of each point of the virtual object 20 in three-dimensional space.

[0031] The interference determination unit 106 determines whether the operator 30 will come into contact with the structure 10 when the operator 30 attempts to operate the virtual object 20, based on the position of the structure 10 and the initial display position of the virtual object 20. The interference determination unit 106 may determine that the operator 30 will come into contact with the structure 10 if there is a possibility that the operator 30 will come into contact with the structure 10. The interference determination unit 106 may also determine that the operator 30 will come into contact with the structure 10 if the position of the structure 10 and the position of the virtual object 20 in three-dimensional space overlap at least partially.

[0032] The position adjustment unit 108 adjusts the display position of the virtual object 20 based on the determination result from the interference determination unit 106. If the position adjustment unit 108 determines that the operator 30 will not come into contact with the structure 10, it may use the initial position as the display position of the virtual object 20. If the position adjustment unit 108 determines that the operator 30 will come into contact with the structure 10, it changes the display position of the virtual object 20 from the initial position. The position adjustment unit 108 adjusts the display position of the virtual object 20 so that the possibility of the operator 30 coming into contact with the structure 10 is reduced. The position adjustment unit 108 may adjust the display position of the virtual object 20 so that the structure 10 and the virtual object 20 do not overlap even partially in three-dimensional space. Note that the adjustment of the display position includes at least one of moving the position of the virtual object 20 in three-dimensional space and changing the orientation of the virtual object 20 (i.e., rotating it). The position adjustment unit 108 may move the display position of the virtual object 20 within the region between the initial position of the virtual object 20 and the operator 30.

[0033] The control unit 120 displays the virtual object 20 at the display position set by the position adjustment unit 108. The shape of the virtual object 20 and the display content such as text information may be obtained from the work support information storage unit 118.

[0034] Figure 3 illustrates an example of a gesture by operator 30. In this example, the gesture detection unit 112 detects the movement of operator 30's hand 32 as a gesture. For example, the gesture detection unit 112 detects the state in which each finger of operator 30's hand 32 is extended as a display gesture for displaying the virtual object 20.

[0035] When a display gesture is detected, the control unit 120 displays the virtual object 20 at the display position adjusted by the position adjustment unit 108. The virtual object 20 has an operable part 22 that can be operated by the operator 30. The operable part 22 may simulate a structure that changes state, such as a button or a lever.

[0036] In this example, the gesture detection unit 112 detects a state in which some fingers of the operator's hand 32 are extended as an operation gesture for manipulating the virtual object 20. When an operation gesture is detected, the control unit 120 makes the operated part 22 operable by the operator 30. For example, when an operation gesture is detected, the control unit 120 may determine that the operated part 22 has been pressed if the hand 32 overlaps with the operated part 22. If the control unit 120 determines that the operated part 22 has been pressed, it may change the state of the operated part 22 and display it. For example, it may change the color of the operated part 22 to display it differently. The control unit 120 may also notify an external party that the operated part 22 has been operated. The structure 10 may be controlled in response to this notification. The work support device 100 may control the structure 10 using this notification, or an external device that receives this notification from the work support device 100 may control the structure 10.

[0037] The virtual object 20 may have multiple operable parts 22. Each operable part 22 may be operated by a common operation gesture, or an operation gesture may be set for each operable part 22.

[0038] Figure 4 illustrates an example of the operation of the interference determination unit 106. In Figure 4, a part of the structure 10 and a virtual object 20 are shown. The interference determination unit 106 may determine that the operator 30 is in contact with the structure 10 if the coordinates of at least one point of the virtual object 20 are included in the area of ​​the structure 10. The interference determination unit 106 may also determine that the operator 30 is in contact with the structure 10 if the coordinates of at least one point of the operated part 22, which is the target of the operator's gesture, are included in the area of ​​the structure 10. In other words, even if a part of the virtual object 20 overlaps with the structure 10, if the operated part 22 does not overlap with the structure 10, the interference determination unit 106 may determine that the operator 30 is not in contact with the structure 10.

[0039] Figure 5 illustrates another example of the operation of the interference determination unit 106. In Figure 5, in addition to a part of the structure 10 and the virtual object 20, the passage area 34 through which the operator 30's body passes when the operator 30 attempts to manipulate the virtual object 20 is shown.

[0040] The interference determination unit 106 may determine the passage area 34 based on operator information relating to the operator 30 and the position of the operated part 22. The operator information may include static information relating to the operator 30 that does not change over time. For example, the static information may include the operator 30's height, arm length, shoulder position, etc. This information may be determined from an image of the operator 30. For example, when the operator 30 extends their hand 32 towards the operated part 22, the interference determination unit 106 may use the static information to determine the passage area 34 that the operator 30 can pass through.

[0041] The operator information may also include dynamic information that changes over time regarding the operator 30. For example, the dynamic information may include the relative position of the operator 30 with respect to the operated part 22, information on whether the hand 32 that performed the gesture is the right or left hand, and the current position of the hand 32. This information may be determined from an image of the operator 30. For example, the interference determination unit 106 may use the dynamic information to determine the passable area 34 that the operator 30 can pass through when the operator 30 extends their hand 32 towards the operated part 22. The interference determination unit 106 may also use both static and dynamic information to determine the passable area 34.

[0042] In this example, the passage region 34 is the region connecting the hand 32, which performs a gesture to display or manipulate the virtual object 20, and the operated part 22. The passage region 34 may be a region that connects the hand 32 and the operated part 22 in a straight line, or it may be a region that connects them in a curved line. The interference determination unit 106 may determine that the operator 30 is in contact with the structure 10 if at least a part of the passage region 34 overlaps with the structure 10.

[0043] The position adjustment unit 108 adjusts the position of the virtual object 20 when it determines that the operator 30 is about to come into contact with the structure 10, thereby reducing the possibility of the operator 30 coming into contact with the structure 10. The position adjustment unit 108 may adjust the display position of the virtual object 20 based on the operator information described above. For example, the position adjustment unit 108 adjusts the display position of the virtual object 20 so that the passage area 34, determined based on the operator information described above, does not come into contact with the structure 10. The position adjustment unit 108 may also adjust the display position of the virtual object 20 based on operator information (including at least one of static information and dynamic information) to make it easier for the operator 30 to operate. In this case as well, the position adjustment unit 108 adjusts the display position of the virtual object 20 on the condition that the passage area 34 does not come into contact with the structure 10.

[0044] The position adjustment unit 108 may adjust the display position of the virtual object 20 based on virtual object information relating to the virtual object 20. For example, the virtual object information includes the position and size of the operated part 22 in the virtual object 20. The position adjustment unit 108 may adjust the position of the virtual object 20 so that the operated part 22 in the virtual object 20 does not overlap with the structure 10. The position adjustment unit 108 may also adjust the display position of the virtual object 20 so that the passage area 34 for the operated part 22 does not overlap with the structure 10.

[0045] Figure 6 illustrates another example of the operation of the interference detection unit 106. Figure 6 shows the arrangement of the structure 10 and the virtual object 20 in the XY plane (top view). In this example, the structure 10 has a movable part 16. The movable part 16 is a part whose position in three-dimensional space is variable. The movable part 16 is, for example, an openable and closable door.

[0046] The interference determination unit 106 in this example may determine whether the operator 30 will come into contact with the structure 10 based on structural information relating to the structure 10. The structural information may include, for example, information indicating the range of motion 12 of the movable part 16. The range of motion 12 may be the maximum range in which the movable part 16 can move. The structural information may be pre-registered in the work support device 100.

[0047] The interference determination unit 106 may determine that the operator 30 is in contact with the structure 10 if at least a portion of the virtual object 20 overlaps with the movable range 12. The interference determination unit 106 may determine that the operator 30 is in contact with the structure 10 if at least a portion of the operated portion 22 of the virtual object 20 overlaps with the movable range 12. The interference determination unit 106 may determine that the operator 30 is in contact with the structure 10 if the passage area 34 overlaps with the movable range 12.

[0048] The position adjustment unit 108 adjusts the position of the virtual object 20 when it determines that the operator 30 is about to come into contact with the structure 10, thereby reducing the possibility of the operator 30 coming into contact with the structure 10. The position adjustment unit 108 may adjust the display position of the virtual object 20 based on the structure information described above. For example, the position adjustment unit 108 adjusts the display position of the virtual object 20 so that at least a part of the virtual object 20 does not overlap with the movable range 12. The position adjustment unit 108 may also adjust the display position of the virtual object 20 so that at least a part of the operated part 22 of the virtual object 20 does not overlap with the movable range 12. The position adjustment unit 108 may also adjust the display position of the virtual object 20 so that the passage area 34 for the operated part 22 does not overlap with the movable range 12.

[0049] Furthermore, the structure 10 may include hazardous parts 14. Hazardous parts 14 are, for example, parts that become high voltage, parts that become hot, parts with sharp shapes, etc. The structure information may include information indicating a prohibited area 18 centered on the hazardous parts 14. The position adjustment unit 108 may adjust the display position of the virtual object 20 so that at least a part of the virtual object 20 does not overlap with the prohibited area 18. The position adjustment unit 108 may adjust the display position of the virtual object 20 so that at least a part of the operated part 22 of the virtual object 20 does not overlap with the prohibited area 18. The position adjustment unit 108 may adjust the display position of the virtual object 20 so that the passage area 34 for the operated part 22 does not overlap with the prohibited area 18.

[0050] Figure 7 shows an example of the operation of the position adjustment unit 108. In this example, the position adjustment unit 108 adjusts the orientation of the virtual object 20 when the passage area 34 overlaps with the structure 10, the movable range 12, or the restricted area 18, as shown in Figure 6. In this example, the virtual object 20 does not overlap with the structure 10, the movable range 12, or the restricted area 18.

[0051] By changing the orientation of the virtual object 20, the position of the passage area 34 changes. The passage area 34 moves by revolving around the virtual object 20 as the virtual object 20 rotates. The position adjustment unit 108 rotates the virtual object 20 so that the passage area 34 does not overlap with the structure 10, the movable range 12, or the restricted range 18. This allows the operator 30 to move to a position where the virtual object 20 does not come into contact with the structure 10, etc., even when the virtual object 20 is manipulated.

[0052] Figure 8 is a flowchart outlining the work support method using the work support device 100. The processing in the work support method is the same as the processing of the work support device 100 described in Figures 1 to 7. In addition to the processing described in Figure 8, the work support method may also include the processing described in Figures 1 to 7.

[0053] First, in the imaging stage S802, the image acquisition unit 102 starts imaging the real space and the operator 30, etc. The image acquisition unit 102 may capture a video so that it can detect changes in the real space and the operator 30 over time.

[0054] In the gesture reading stage S804, the sensor unit 110 and the gesture detection unit 112 read the gestures of the operator 30. In the gesture detection stage S806, the gesture detection unit 112 determines whether the gestures of the operator 30 correspond to gestures pre-set in the gesture information storage unit 116, etc. If the gestures of the operator 30 do not correspond to the set gestures, the process in S806 is repeated.

[0055] If the operator 30's gesture is one that indicates the display of the virtual object 20, the initial position setting unit 114 sets the initial position of the virtual object 20 in three-dimensional space during the initial position setting stage S808. Next, during the interference detection stage S810, the interference determination unit 106 determines whether the operator 30 will come into contact with the structure 10. If contact is determined to occur, the position adjustment unit 108 adjusts the display position of the virtual object 20 during the position adjustment stage S812. After it is determined in S810 that there is no contact, or after the display position has been adjusted in S812, the control unit 120 acquires information such as the shape of the virtual object 20 to be displayed in three-dimensional space during the display information reading stage S814. Next, during the composite image display stage S816, the display unit 122 displays a composite image that includes an image of the structure 10 in the real world and an image of the virtual object 20.

[0056] In S806, if the operator 30's gesture indicates an action to operate the virtual object 20, in the operation determination stage S818, the control unit 120 determines whether the virtual object 20 has been operated by the operator 30. If the operator 30 does not operate the operated part 22 of the virtual object 20 within a predetermined period after detecting the gesture, the process from S806 is repeated. If the virtual object 20 has been operated by the operator 30, in the update stage S820, the control unit 120 updates the display content of the virtual object 20 according to the operation content and executes the process in S816. After the process in S816, the process from S806 is repeated. This prevents the operator 30 from coming into contact with the structure 10 when operating the virtual object 20.

[0057] Figure 9 shows an example of a computer 2200 in which multiple embodiments of the work support method may be fully or partially implemented. The computer 2200 is equipped with a program that causes the computer 2200 to execute the work support method described in Figures 1 to 8.

[0058] A program installed on computer 2200 can cause computer 2200 to function as an operation or one or more sections of an apparatus according to an embodiment of the present invention, or to execute such operation or one or more sections, and / or to cause computer 2200 to execute a method or a step of such method according to an embodiment of the present invention. Such a program may be executed by CPU 2212 to cause computer 2200 to perform a particular operation associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0059] The computer 2200 according to this embodiment includes a CPU 2212, RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0060] The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 retrieves image data generated by the CPU 2212 from a frame buffer provided in RAM 2214 or from itself, and displays the image data on the display device 2218.

[0061] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides them to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

[0062] The ROM 2230 stores boot programs and / or programs that depend on the computer 2200's hardware, which are executed by the computer 2200 when activated. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0063] The program is provided on a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium and installed on a hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable medium, and executed by the CPU 2212. The information processing described within these programs is read by the computer 2200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the manipulation or processing of information in accordance with the use of the computer 2200.

[0064] For example, when communication is performed between a computer 2200 and an external device, the CPU 2212 may execute a communication program loaded into RAM 2214 and, based on the processing described in the communication program, instruct the communication interface 2222 to perform communication processing. Under the control of the CPU 2212, the communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as RAM 2214, a hard disk drive 2224, a DVD-ROM 2201, or an IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area provided on the recording medium.

[0065] Furthermore, the CPU 2212 may read all or necessary parts of files or databases stored on external storage media such as the hard disk drive 2224, DVD-ROM drive 2226 (DVD-ROM 2201), or IC card into the RAM 2214, and perform various types of processing on the data in the RAM 2214. The CPU 2212 then writes the processed data back to the external storage media.

[0066] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 2212 may perform various types of processing on the data read from RAM 2214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 2214. The CPU 2212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 2212 may search among the multiple entries for an entry that matches the condition for which the attribute value of the first attribute is specified, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0067] The programs or software modules described above may be stored on or near computer 2200 on a computer-readable medium. Alternatively, recording media such as hard disks or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as computer-readable media, thereby providing programs to computer 2200 via the network.

[0068] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0069] It should be noted that the execution order of operations, procedures, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before," "prior to," etc., and can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," "next," etc. for convenience, it does not mean that it is essential to perform the operations in that order. [Explanation of symbols]

[0070] 10...Structure, 12...Movable range, 14...Dangerous area, 16...Movable part, 18...Prohibited area, 20...Virtual object, 22...Operated part, 30...Operator, 32...Hand, 34...Passage area, 100...Work support device, 102...Image acquisition unit, 104...Structure detection unit, 106...Interference determination unit, 108...Position adjustment unit, 110...Sensor unit, 112...Gesture detection unit, 114...Initial position setting unit, 116...Gesture information storage unit, 118...Work support information storage unit, 120...Control unit, 122...Display unit

Claims

1. A display unit that displays an image of a three-dimensional space including structures in the real world and virtual objects manipulated by the operator, A structure detection unit for detecting the position of the structure in the real world, An initial position setting unit that sets the initial value of the display position of the virtual object in the three-dimensional space, An interference determination unit determines, based on the position of the structure and the display position of the virtual object, whether the operator comes into contact with the structure in the real world when the operator attempts to operate the virtual object. A position adjustment unit adjusts the display position of the virtual object based on the determination result in the interference determination unit, The gesture detection unit detects the operator's gestures. Equipped with, The initial position setting unit sets the initial value of the display position of the virtual object based on the operator's gesture. Work support device.

2. A display unit that displays an image of a three-dimensional space including structures in the real world and virtual objects manipulated by the operator, A structure detection unit for detecting the position of the structure in the real world, An initial position setting unit that sets the initial value of the display position of the virtual object in the three-dimensional space, An interference determination unit determines, based on the position of the structure and the display position of the virtual object, whether the operator comes into contact with the structure in the real world when the operator attempts to operate the virtual object. A position adjustment unit adjusts the display position of the virtual object based on the determination result of the interference determination unit. Equipped with, The position adjustment unit will use the initial value as the display position if it is determined that the operator will not come into contact with the structure. Work support device.

3. A display unit that displays an image of a three-dimensional space including structures in the real world and virtual objects manipulated by the operator, A structure detection unit for detecting the position of the structure in the real world, An initial position setting unit that sets the initial value of the display position of the virtual object in the three-dimensional space, An interference determination unit determines, based on the position of the structure and the display position of the virtual object, whether the operator comes into contact with the structure in the real world when the operator attempts to operate the virtual object. A position adjustment unit adjusts the display position of the virtual object based on the determination result of the interference determination unit. Equipped with, The interference determination unit determines whether the passage region, which is the area between the operator's hand and the virtual object, overlaps with the position of the structure. Work support device.

4. The interference determination unit determines whether the position of the structure and the display position of the virtual object overlap. A work support device according to any one of claims 1 to 3.

5. The position adjustment unit further adjusts the display position based on operator information relating to the operator. A work support device according to any one of claims 1 to 4.

6. The operator information includes static information that does not change over time with respect to the operator. The work support device according to claim 5.

7. The operator information includes dynamic information that changes over time with respect to the operator. The work support device according to claim 5.

8. The position adjustment unit further adjusts the display position based on structural information relating to the structure. A work support device according to any one of claims 1 to 7.

9. The position adjustment unit further adjusts the display position based on virtual object information relating to the virtual object. A work support device according to any one of claims 1 to 8.

10. The position adjustment unit further adjusts the orientation of the virtual object based on the determination result in the interference determination unit. A work support device according to any one of claims 1 to 9.

11. A work support method using a display unit that displays an image of a three-dimensional space including structures in the real world and virtual objects manipulated by an operator, The position of the structure in the real world is detected, The operator's gesture is detected, Based on the operator's gesture, the initial value of the display position of the virtual object in the three-dimensional space is set. When the operator attempts to manipulate the virtual object, it is determined whether the operator will come into contact with the structure in the real world, based on the position of the structure and the display position of the virtual object. The display position of the virtual object is adjusted based on the determination result. Work support method.

12. A work support method using a display unit that displays an image of a three-dimensional space including structures in the real world and virtual objects manipulated by an operator, The position of the structure in the real world is detected, The initial value of the display position of the virtual object in the three-dimensional space is set, When the operator attempts to manipulate the virtual object, it is determined whether the operator will come into contact with the structure in the real world, based on the position of the structure and the display position of the virtual object. Based on the determination result, the display position of the virtual object is adjusted. If it is determined that the operator will not come into contact with the structure, the initial value will be used as the display position. Work support method.

13. A work support method using a display unit that displays an image of a three-dimensional space including structures in the real world and virtual objects manipulated by an operator, The position of the structure in the real world is detected, The initial value of the display position of the virtual object in the three-dimensional space is set, When the operator attempts to manipulate the virtual object, whether the operator comes into contact with the structure in the real world is determined by whether the passage area, which is the area between the operator's hand and the virtual object, overlaps with the position of the structure. The display position of the virtual object is adjusted based on the determination result. Work support method.

14. A program for causing a computer to perform the work support method described in any one of claims 11 to 13.

Images