Work support device, work support method, and program

Abstract

To provide a work support apparatus capable of efficiently manipulating a state of a virtual object.SOLUTION: A work support apparatus 100 comprises: a display unit 122 for displaying an image in a three-dimensional space; a target selection unit 104 for selecting at least one target object to be operated by an operator from one or more virtual objects; an area identifying unit 106 for identifying a designated area in the three-dimensional space based on gesture information indicating a state of an operator's body, and identifying a related object from actual objects and virtual objects included in the designated area; and a parameter changing unit 119 for changing parameters related to the target object based on the related object.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a work support device, a work support method, and a program.

[0002] Conventionally, in augmented reality (AR) technology or mixed reality (MR) technology that overlays and displays virtual objects on an image showing the real space, a technology for operating the state of virtual objects is known (see, for example, Patent Document 1). Patent Document 1: Japanese Patent Application Laid-Open No. 2017-27206

Summary of the Invention

Problems to be Solved by the Invention

[0003] It is preferable that the state of a virtual object can be efficiently operated.

Means for Solving the Problems

[0004] In order to solve the above problems, in a first aspect of the present invention, a work support device is provided. The work support device may include a display unit that displays an image of a three-dimensional space including a real object showing a structure in the real world and a virtual object operated by an operator. The work support device may include an object selection unit that selects at least one target object to be an operation target from one or more virtual objects. The work support device may include a region identification unit that identifies a specified region in the three-dimensional space based on gesture information indicating the state of the operator's body and identifies related objects from the real objects and virtual objects included in the specified region. The work support device may include a parameter change unit that changes parameters related to the target object based on the related objects.

[0005] The parameter change unit may change the position parameter of the target object based on the position parameter of the related object in the three-dimensional space.

[0006] The region identification unit may identify a designated region based on the position through which the operator's line of sight passes in three-dimensional space.

[0007] The region identification unit may identify the real or virtual object that the operator's line of sight first passes over as the associated object.

[0008] The display unit may display a depth position indicator that specifies the position in the depth direction of three-dimensional space. The area identification unit may identify related objects based on the position specified by the depth position indicator.

[0009] The region identification unit may rotate and display the specified region when it detects pre-configured gesture information.

[0010] The region identification unit may rotate and display the specified region if multiple objects contained within the specified region overlap in the depth direction.

[0011] The display unit may display a list of real and virtual objects included in the specified area. The area identification unit may identify objects selected from the list as related objects.

[0012] A virtual object may contain multiple modifiable parameters. The parameter modification section may select the parameter to modify based on the operator's gesture information.

[0013] A second aspect of the present invention provides a work support method. The work support method may display an image of a three-dimensional space including real objects representing structures in the real world and virtual objects manipulated by an operator. The work support method may select at least one target object from one or more virtual objects to be manipulated by the operator. The work support method may identify a designated area in the three-dimensional space based on gesture information indicating the state of the operator's body, and identify related objects from the real and virtual objects included in the designated area. The work support method may change parameters relating to the target object based on the related objects.

[0014] In a third aspect of the present invention, a program is provided for causing a computer to execute a work support method according to the second aspect.

[0015] It should be noted that the above summary of the invention does not list all the necessary features of the present invention. Furthermore, subcombinations of these features may also constitute an invention. [Brief explanation of the drawing]

[0016] [Figure 1] This figure illustrates an image 12 displayed by a work support device according to one embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the functional configuration of the work support device 100. [Figure 3] This diagram illustrates an example of how an operator can manipulate a target object using gestures. [Figure 4] This diagram illustrates an example of selecting a target object from virtual object 20. [Figure 5] This diagram illustrates an example of selecting related objects. [Figure 6] This diagram illustrates an example of changing the parameters of a target object. [Figure 7] This figure shows Image 12 after changing the position parameters of the target object. [Figure 8] This is a diagram for explaining an example of changing group parameters of target objects. [Figure 9] This is a diagram for explaining an example of parameters assigned to the virtual object 20. [Figure 10] This is a diagram for explaining another example of a method for selecting an object. [Figure 11] This is a diagram for explaining another example of an operation when an operator selects an object. [Figure 12] This is a flowchart showing an overview of the work support method. [Figure 13] An example of a computer 2200 is shown in which multiple aspects of the work support method may be embodied in whole or in part.

Mode for Carrying Out the Invention

[0017] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0018] FIG. 1 is a diagram for explaining an image 12 displayed by a work support device according to an embodiment of the present invention. The work support device displays an image 12 in which a real object 10 indicating a structure existing in the real world and a virtual object 20 operated by an operator are superimposed. In this specification, the real object 10 and the virtual object 20 may sometimes be simply referred to as objects. The image 12 in the example of FIG. 1 is an image showing the real object 10 and the virtual object 20 on a common three-dimensional space. In FIG. 1, the three-dimensional space shown in the image 12 is shown in an XYZ coordinate system. Note that the three-dimensional space shown in the image 12 may or may not include an operator. The image 12 may be a three-dimensional holography or a pseudo three-dimensional image using a parallax of an image presented to both eyes. As an example, the work support device includes a wearable terminal of a head-mounted display (HMD) type that is worn by an operator and presents the image 12.

[0019] The virtual object 20 in this example is an object for managing information about the real object 10. The real object 10 is, for example, equipment arranged in a factory or the like. The real object 10 may be manufacturing equipment or may be equipment for other purposes.

[0020] The virtual object 20 is displayed in association with any one of the real objects 10. The virtual object 20 in this example has one or more tags (tags 22 and 24 in FIG. 1) to which information is assigned. The information assigned to each tag can be edited by the operator. In this specification, the information managed by the virtual object 20 may be referred to as a parameter.

[0021] Character information is assigned to the tag 22 in this example. The tag 22 may display the assigned character information. The character information is, for example, information indicating the state of the real object 10. The state of the real object 10 may be, for example, a state such as whether the equipment is in operation or under repair (or maintenance). The operator may edit the character information by operating the tag 22. The tag 22 may be assigned any character string input by the operator, or may be assigned a character string selected by the operator from a plurality of preset types of character strings.

[0022] Information indicating the correspondence between the virtual object 20 and the real object 10 is assigned to the tag 24 in this example. The tag 24 may be displayed at a position overlapping the corresponding real object 10. The operator may edit the correspondence between the virtual object 20 and the real object 10 by moving the tag 24. The virtual object 20 in the image 12 of FIG. 1 shows a state where it is not yet associated with the real object 10.

[0023] Depending on the operation performed on the virtual object 20, the operation may take some time. For example, in a three-dimensional space, when moving the tag 24 of the virtual object 20 in the depth direction of the three-dimensional space shown in image 12, the amount of movement in the depth direction may be determined by the length of the operation time. In other words, the virtual object 20 is moved in the depth direction while a movement operation such as pressing a button is being performed, and the movement of the virtual object 20 is stopped when the movement operation is finished. In such a case, the operation time increases in proportion to the amount of movement in the depth direction. The depth direction refers to the direction away from the operator or the direction towards the operator in three-dimensional space. If image 12 is an image displayed on the XZ plane, the depth direction is the Z axis direction perpendicular to the XZ plane.

[0024] Figure 2 is a block diagram showing an example of the functional configuration of the work support device 100. The work support device 100 assists the operator in performing tasks on the image 12. The work support device 100 assists in changing parameters (i.e., information assigned to tags) assigned to the virtual object 20.

[0025] The work support device 100 includes a target selection unit 104, a region identification unit 106, a parameter change unit 119, and a display unit 122. The work support device 100 may further include at least one of the following: an image acquisition unit 102, a sensor unit 110, a gesture detection unit 112, a gesture information storage unit 116, a parameter information storage unit 118, and a control unit 120. Functions in the work support device 100 other than the display unit 122, the sensor unit 110, and the image acquisition unit 102 may be implemented by a computer. Each component of the work support device 100 may be housed in a single housing such as a head-mounted display, or it may be distributed and housed in multiple housings. Each component may communicate with each other by wire or wireless.

[0026] The display unit 122 provides the operator with an image 12 of a three-dimensional space including real objects 10 and virtual objects 20. The display unit 122 may be a liquid crystal display device, an organic EL display device, a hologram display device, or other display device. The control unit 120 controls the display unit 122 to display the image 12. The control unit 120 may output the image data to be displayed on the display unit 122.

[0027] The image acquisition unit 102 acquires an image of the real world. The image acquisition unit 102 acquires information regarding the position in the depth direction of each structure included in the image. The image acquisition unit 102 may acquire a three-dimensional image of the real world. In this example, the image acquisition unit 102 is a camera having an image sensor such as a CCD. The image acquisition unit 102 may acquire distance information from the image acquisition unit 102 to each part of the structure.

[0028] The control unit 120 generates the image 12 shown in Figure 1 based on the image of the structure and information regarding its depth position acquired by the image acquisition unit 102. The control unit 120 places the real object 10 representing the structure in the three-dimensional space shown in the image 12. In this example, the control unit 120 places the real object 10 in the three-dimensional space of the image 12 based on the information regarding the depth position of the structure.

[0029] The control unit 120 also places the virtual object 20, as shown in Figure 1, in the three-dimensional space shown in Image 12. The control unit 120 may place the virtual object 20 in Image 12 when it receives input from the operator indicating that the virtual object 20 should be displayed. The input from the operator may be a gesture made by the operator's body or a voice command from the operator. In another example, the operator may use an input device having a button or the like to input that the virtual object 20 should be displayed. The work support device 100 in this example detects gestures that include at least one of the operator's hand and gaze.

[0030] The sensor unit 110 monitors the movements of at least a part of the operator's body. The sensor unit 110 may have a camera that captures images of at least a part of the operator's body, or an accelerometer attached to the operator's body.

[0031] The gesture detection unit 112 detects an action that corresponds to a pre-set gesture from the operator's actions detected by the sensor unit 110. The gesture information storage unit 116 stores information for detecting one or more pre-set gestures. For example, the gesture information storage unit 116 may store gesture information indicating the operator's actions for each gesture. The gesture information storage unit 116 may also store what kind of input from the operator each gesture represents. The gesture information storage unit 116 stores at least gestures for changing the parameters of the virtual object 20. The gesture information storage unit 116 may also store gestures for displaying a new virtual object 20. The gesture detection unit 112 may extract the gesture that corresponds to the operator's actions from the gesture information storage unit 116.

[0032] The target selection unit 104 selects at least one target object from one or more virtual objects 20 included in the image 12 to be the target of the operator's operation. The target selection unit 104 may select the target object based on the operator's gestures. For example, after the operator makes a gesture indicating that they intend to select a target object, the target selection unit 104 may select the virtual object 20 specified by the operator as the target object. The operator's specification may be made by gestures such as the operator's gaze.

[0033] The region identification unit 106 identifies a designated region in the three-dimensional space shown in image 12 based on gesture information detected by the operator's gesture. The region identification unit 106 also identifies related objects from the real objects 10 and virtual objects 20 included in the designated region. The region identification unit 106 may identify the designated region and related objects when the operator makes a gesture indicating that they want to identify related objects. For example, the region identification unit 106 may detect the position through which the operator's line of sight passes when the operator makes a predetermined gesture with their hand. The region identification unit 106 may designate the region through which the operator's line of sight passes in the three-dimensional space shown in image 12 as the designated region.

[0034] The region identification unit 106 identifies an object as a related object if the designated region contains only one object. If the designated region contains multiple objects, the region identification unit 106 identifies one or more objects as related objects according to predetermined rules or processing methods. For example, the region identification unit 106 may identify the object closest to the user (i.e., the object closest to the user in 3D space) as a related object.

[0035] The parameter modification unit 119 modifies parameters related to the target object based on the identified related object. For example, the parameter modification unit 119 modifies the position parameter, which indicates the position of the target object in three-dimensional space, according to the position parameter of the related object. The parameter modification unit 119 may match the position parameter of the target object with the position parameter of any part of the related object. This process allows the target object to be moved efficiently in three-dimensional space. In other words, since the target object can be moved by specifying an already existing object, it is not necessary to specify the amount of movement of the target object in the depth direction using operation time, etc. Therefore, the operation time related to moving the target object can be shortened.

[0036] The parameter information storage unit 118 stores the parameters associated with each object. The parameter information storage unit 118 may also store image data showing the shape, etc., of each object when it is displayed. The parameter modification unit 119 may read the parameter values ​​associated with each object from the parameter information storage unit 118. The parameter modification unit 119 may store the modified parameter values ​​of the target object in the parameter information storage unit 118. The parameter modification unit 119 notifies the control unit 120 of the modified parameter values ​​of the target object. The control unit 120 modifies the image data of the target object displayed by the display unit 122 based on the modified parameters. For example, the control unit 120 controls the position of the target object in image 12 based on the position parameters after movement.

[0037] Figure 3 illustrates an example of an operation on a target object using the operator's gestures. In this example, the gesture detection unit 112 detects the movement of the operator's hand 32 as a gesture. For example, the gesture detection unit 112 detects the state in which each finger of the operator's hand 32 is extended as a target selection gesture for selecting a target object.

[0038] The target selection unit 104 identifies a virtual object 20 specified by the operator as the target object after the target selection gesture is detected. The target selection unit 104 may identify a virtual object 20 in image 12 where the operator's gaze remains for a predetermined period of time or longer as the target object after the target selection gesture is detected. The target selection unit 104 may identify a virtual object 20 selected by the operator's hand 32 movement as the target object after the target selection gesture is detected. The target selection unit 104 may identify the target object by other means.

[0039] Furthermore, the gesture detection unit 112 may detect a state in which some fingers of the operator's hand 32 are extended as an operation gesture for manipulating the target object, the virtual object 20. The gesture detection unit 112 then detects operations performed by the operator on the virtual object 20 after the operation gesture has been detected. For example, the gesture detection unit 112 detects operations performed on the virtual object 20 by the hand 32 in the operation gesture state.

[0040] Each virtual object 20 has multiple tags (tags 22, 24, and 26) that can be manipulated by the operator. Each tag is assigned a changeable parameter. The parameter change unit 119 selects the parameter to change according to the operator's gesture information. In this example, the operator may select the tag whose parameter to change by pointing to one of the tags with the hand 32 in the operation gesture state. The display unit 122 may display an image of the hand 32 in three-dimensional space.

[0041] In this example, tag 22 is assigned a state parameter containing state information, tag 24 is assigned a position parameter containing location information, and tag 26 is assigned a group parameter containing group information. The state parameter is, for example, a parameter that indicates the status of the equipment of the real object 10 corresponding to the virtual object 20. The status of the equipment is, for example, whether or not it is undergoing renovation (i.e., maintenance and repair, etc.), whether or not it is operating normally, whether or not it is scheduled for renovation, etc. The position information is information such as the coordinates of the virtual object 20 in three-dimensional space. The group information is information that indicates which group the virtual object 20 or the real object 10 belongs to. The groups are, for example, a group that indicates real objects 10 that are being renovated in conjunction at the same time, a group that indicates real objects 10 that operate in conjunction during normal operation, a group that indicates real objects 10 that were introduced at the same time, etc.

[0042] The tags assigned to the virtual object 20 are not limited to these. Each virtual object 20 may have tags to which various other types of information are assigned. For example, a virtual object 20 may have tags containing text information entered by an operator or the like. A virtual object 20 may also have a tag containing correspondence information indicating which real object the virtual object 20 is associated with.

[0043] Figure 4 illustrates an example of selecting a target object from the virtual objects 20. As described above, when the target selection unit 104 detects a target selection gesture, it selects a target object from one or more virtual objects 20 whose parameters are to be changed.

[0044] If image 12 contains only one virtual object 20, the target selection unit 104 may select that virtual object 20 as the target object. If image 12 contains multiple virtual objects 20, the target selection unit 104 selects the virtual object 20 selected by the operator as the target object.

[0045] As described above, the target selection unit 104 may detect a passage area 51 through which the operator's line of sight passes in the three-dimensional space shown in the image 12. The work support device 100 may have a sensor that detects the direction of the operator's eyes. The passage area 51 may be specified by other means. For example, the passage area 51 may be defined by a straight line extending in a direction specified by the operator's palm or fingers. The target selection unit 104 selects a virtual object 20 that overlaps with the passage area 51 as the target object. If there are multiple virtual objects 20 that overlap with the passage area 51, the target selection unit 104 may select the virtual object 20 closest to the user as the target object, or it may select a virtual object 20 determined by other means as the target object.

[0046] Figure 5 illustrates an example of selecting related objects. The region identification unit 106 identifies related objects to be used to change the parameters of the target object. The related objects may be either the real object 10 or the virtual object 20.

[0047] The region identification unit 106 may start the process of identifying the associated object when an associated selection gesture by the operator is detected. The associated selection gesture is, for example, a gesture made by the hand 32.

[0048] In this example, the region identification unit 106 identifies a designated region 52 based on the position through which the operator's line of sight passes in the three-dimensional space shown in image 12. The designated region 52 may be a linear region through which the operator's line of sight passes in the three-dimensional space.

[0049] The region identification unit 106 detects objects that exist in the designated region 52. The region identification unit 106 may also detect objects that overlap with the designated region 52. In this example, the real object 10-1 exists in the designated region 52.

[0050] The region identification unit 106 identifies an object as the associated object if there is only one object in the designated region 52. If there are multiple objects overlapping the designated region 52, the region identification unit 106 may select the object closest to the viewer (i.e., the object that the operator's line of sight passes through first) as the associated object, or it may select an object determined by another method as the associated object. In the example in Figure 5, the real object 10-1 is selected as the associated object.

[0051] Figure 6 illustrates an example of changing the parameters of a target object. In this example, the position parameter of the target object (virtual object 20) is changed. The parameter change unit 119 may start the process of changing the parameters of the target object when an operation gesture is detected.

[0052] The parameter modification unit 119 modifies the parameters of the target object (virtual object 20) based on the parameters of the related object (real object 10-1). In this example, the position parameters of the target object are modified based on the position parameters of the related object. The parameter modification unit 119 may make the parameters of the target object match those of the related object. In this example, the position parameters of the target object are made to match those of the related object.

[0053] The parameter modification unit 119 may determine which parameter of the target object to modify based on which tag (see Figure 3) the operator selects using an operation gesture. In this example, the operator selects tag 24 of the virtual object 20. The parameter modification unit 119 may then match the parameter corresponding to the selected tag with the parameter of an already specified related object.

[0054] Furthermore, the region identification unit 106 may identify a designated region based on the position where the operator moves any of the tags on the two-dimensional plane (XZ plane) of the image 12. For example, if the operator moves tag 24 on the two-dimensional plane, the region overlapping with the moved tag 24 in three-dimensional space may be designated as the designated region. The region identification unit 106 may determine related objects from the objects included in the designated region. In this case as well, if there is only one object in the designated region 52, the region identification unit 106 identifies that object as the related object. If there are multiple objects that overlap with the designated region 52, the region identification unit 106 may select the object closest to the viewer as the related object, or it may select an object determined by another method as the related object.

[0055] The process of identifying related objects by moving tags and the process of identifying related objects by detecting gaze, etc., may be performed individually or both. When both processes are performed, if the object specified by moving tags matches the object specified by detecting gaze, etc., that object may be identified as a related object.

[0056] Figure 7 shows image 12 after changing the position parameters of the target object. In this example, the position parameter of tag 24 of virtual object 20 has been changed to match the position parameter of real object 10-1. Tag 24 is displayed in a position that overlaps with the image of real object 10. Tags other than tag 24 of virtual object 20 may be displayed in positions that do not overlap with other objects. This process makes it easy to change the position parameters of the target object.

[0057] Figure 8 illustrates an example of changing the group parameters of a target object. In this example, virtual object 20-3 is the target object, and virtual object 20-1 is the related object. In this example, virtual object 20-3 is selected as the target object by selecting tag 26-3 of virtual object 20-3. Also, virtual object 20-1 is designated as the related object by moving tag 26-3 in two dimensions to a position that overlaps with tag 26-1 of virtual object 20-1. Each virtual object 20 is already associated with a real object 10. Virtual object 20-1 may also be identified as a related object by line of sight, etc.

[0058] Before the operation, tags 26-3 and 26-1 indicate that they belong to different groups. The parameter modification unit 119 may match the parameters of tag 26-3 to those of tag 26-1 if tag 26-3 is moved to a position where it overlaps with tag 26-1. In other examples, the parameters of tag 26-1 may be matched to those of tag 26-3. In other words, the parameters of each tag are modified so that the real object 10-1 corresponding to the virtual object 20-1 and the real object 10-3 corresponding to the virtual object 20-3 belong to the same group. In this example, by moving tag 26 in two dimensions, other tags 26 in three-dimensional space can be specified. This makes it easy to edit each object in three-dimensional space.

[0059] Figures 4 to 8 illustrate the editing of position parameters or group parameters, but the editing of state parameters is similar. The operator can select and manipulate tag 22 using an operation gesture to match the state parameters of the target object with those of the related object.

[0060] Figure 9 illustrates an example of parameters assigned to a virtual object 20. Each virtual object 20 may be assigned the position parameter, state parameter, and group parameter described above. Each virtual object 20 may be assigned multiple group parameters. In other words, a virtual object 20 may belong to multiple groups. Furthermore, the groups of each virtual object 20 may be divided hierarchically. Each virtual object 20 may be assigned a group parameter that indicates the group at each level of the hierarchy. This allows the group parameters of the virtual object 20 to be edited on a hierarchical basis.

[0061] Each virtual object 20 may include device parameters indicating the corresponding real object 10, device information parameters indicating information about the real object 10, and maintenance information parameters indicating information about the maintenance of the corresponding real object 10. Device parameters are information for identifying the device, such as the manufacturer and model number of the device. Device information parameters are information such as the purpose, performance, and administrator of the device. Maintenance information parameters are information such as the maintenance procedure for the real object 10, the next maintenance date, and the previous maintenance date.

[0062] Each virtual object 20 may be assigned a character parameter that indicates arbitrary character information entered by the operator. The character parameter may be, for example, a note regarding the operation of the real object 10. The character parameters may be grouped together, for example. For example, if a group of virtual objects 20 is edited, the character parameters assigned to multiple virtual objects 20 belonging to the edited group may be combined into one and managed so that they can be referenced from each virtual object 20. This allows for the simultaneous reference of comments regarding the real object 10 of the same group.

[0063] Figure 10 illustrates another example of how to select an object. This example explains how to select related objects, but the same process can be used to select a target object.

[0064] In this example, a designated region 52 identified by line of sight or other means contains multiple real-world objects 10. The multiple real-world objects 10 are arranged overlapping in the depth direction (Z-axis direction) of the three-dimensional space shown in image 12.

[0065] The region identification unit 106 may rotate and display multiple objects included in the designated region 52 when it detects gesture information indicating a pre-set rotation operation. The region identification unit 106 may rotate and display multiple objects included in the designated region 52 on the condition that the multiple objects included in the designated region 52 overlap in the depth direction. In other words, if a rotation operation gesture is detected and the multiple objects in the designated region 52 overlap, the multiple objects in the designated region 52 may be rotated.

[0066] Rotating multiple objects in the specified region 52 refers to changing the viewpoint position and line of sight relative to the multiple objects in the specified region 52. In the example in Figure 10, an image observed from a viewpoint position far away from the multiple objects in the Z-axis direction (i.e., parallel to the specified region 52) with a line of sight parallel to the Z-axis direction is converted to an image observed from a viewpoint position far away from the multiple objects in the X-axis direction (i.e., perpendicular to the specified region 52) with a line of sight parallel to the X-axis direction. This allows multiple objects that were overlapping in the depth direction to be displayed separately.

[0067] The region identification unit 106 may identify the object selected by the operator in the rotated image as a related object. The operator may select an object by means of gaze detection or tag movement, as in other examples. According to this example, related objects can be easily selected even when multiple objects overlap in the depth direction.

[0068] Furthermore, in both cases—when the designated area 52 is rotated and displayed, and when a pre-set zoom gesture is detected—the area including the designated area 52 may be enlarged and displayed. This makes it easy to select related objects even when multiple objects are densely packed together.

[0069] Furthermore, the designated area 52 may be specified by methods other than gaze. For example, if a pre-set area setting gesture is detected, the area identification unit 106 may designate the area specified by the gesture as the designated area 52. For example, if a predetermined range is specified on the XZ plane of the image 12 by moving a finger or the like, the designated area 52 may designate that range as the designated area 52.

[0070] The area identification unit 106 may display a list of multiple objects if the designated area 52 contains multiple objects. The objects may include both virtual objects 20 and real objects 10, or only one of the two types of objects. The designated area 52 may identify the object selected by the operator from the list as the related object.

[0071] Figure 11 illustrates another example of operation when an operator selects an object. In this example, the display unit 122 displays a depth position indicator 60 that specifies the position in the depth direction of the three-dimensional space shown in image 12. The depth position indicator 60 has position information in three-dimensional space (e.g., three-dimensional coordinates). The depth position indicator 60 is movable along the Z axis in three-dimensional space. The display unit 122 may display a line 61 in a two-dimensional plane (XZ plane) that simulates the Z axis direction. The operator moves the depth position indicator 60 along the line 61 in the two-dimensional plane. Each position on the line 61 is assigned a coordinate in three-dimensional space. The control unit 120 may calculate the depth position of the depth position indicator 60 from its position on the line 61 in the two-dimensional plane.

[0072] When selecting an object, the operator may specify the depth position by manipulating the depth position indicator 60. The area identification unit 106 may identify objects located at the depth position specified by the depth position indicator 60 as related objects among the objects included in the aforementioned designated area 52. The display unit 122 may display the XZ plane 62 at the depth position specified by the depth position indicator 60. The XZ plane 62 may be a transparent or semi-transparent plane so that objects located behind the XZ plane 62 can be seen. In this example as well, the position of the depth position indicator 60 can be specified on a two-dimensional plane, so the position in the depth direction in three-dimensional space can be specified in a short time.

[0073] Figure 12 is a flowchart illustrating the overview of the work support method. Figure 12 shows a portion of the processing in the work support method. The work support method may perform the same processing as the work support device 100 described in Figures 1 to 11.

[0074] First, in the image display stage S1202, an image 12 of a three-dimensional space is displayed, which includes real objects 10 representing structures in the real world and virtual objects 20 manipulated by the operator. The processing in S1202 is the same as the processing in the display unit 122.

[0075] Next, in the target selection stage S1204, at least one target object to be operated on by the operator is selected from one or more virtual objects 20. The processing in S1204 is the same as the processing in the target selection unit 104.

[0076] Next, in the designated area identification stage S1206, the designated area 52 in the three-dimensional space shown by image 12 is identified based on gesture information indicating the state of the operator's body. Furthermore, in the related object identification stage S1208, related objects are identified from the real objects 10 and virtual objects 20 included in the designated area 52. The processing in S1206 and S1208 is the same as the processing in the area identification unit 106.

[0077] Next, in the target object manipulation step S1210, the operator's operation of the target object is accepted. In S1210, operation by operation gesture is accepted. The processing in S1210 is the same as the processing in the gesture detection unit 112.

[0078] Next, in the parameter change stage S1212, the parameters of the target object are changed based on the related objects. The processing in S1212 is the same as the processing in the parameter change unit 119. Through this processing, the depth position in three-dimensional space can be efficiently identified and the parameters of the target object can be efficiently changed.

[0079] Figure 13 shows an example of a computer 2200 in which multiple embodiments of the work support method may be fully or partially implemented. The computer 2200 is equipped with a program that causes the computer 2200 to execute the work support method described in Figures 1 to 12.

[0080] A program installed on computer 2200 can cause computer 2200 to function as an operation or one or more sections of an apparatus according to an embodiment of the present invention, or to execute such operation or one or more sections, and / or to cause computer 2200 to execute a method or a step of such method according to an embodiment of the present invention. Such a program may be executed by CPU 2212 to cause computer 2200 to perform a particular operation associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0081] The computer 2200 according to this embodiment includes a CPU 2212, RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0082] The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 retrieves image data generated by the CPU 2212 from a frame buffer provided in RAM 2214 or from itself, and displays the image data on the display device 2218.

[0083] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides them to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

[0084] The ROM 2230 stores boot programs and / or programs that depend on the computer 2200's hardware, which are executed by the computer 2200 when activated. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0085] The program is provided on a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium and installed on a hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable medium, and executed by the CPU 2212. The information processing described within these programs is read by the computer 2200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the manipulation or processing of information in accordance with the use of the computer 2200.

[0086] For example, when communication is performed between a computer 2200 and an external device, the CPU 2212 may execute a communication program loaded into RAM 2214 and, based on the processing described in the communication program, instruct the communication interface 2222 to perform communication processing. Under the control of the CPU 2212, the communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as RAM 2214, a hard disk drive 2224, a DVD-ROM 2201, or an IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area provided on the recording medium.

[0087] Furthermore, the CPU 2212 may read all or necessary parts of files or databases stored on external storage media such as the hard disk drive 2224, DVD-ROM drive 2226 (DVD-ROM 2201), or IC card into the RAM 2214, and perform various types of processing on the data in the RAM 2214. The CPU 2212 then writes the processed data back to the external storage media.

[0088] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 2212 may perform various types of processing on the data read from RAM 2214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 2214. The CPU 2212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 2212 may search among the multiple entries for an entry that matches the condition for which the attribute value of the first attribute is specified, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0089] The programs or software modules described above may be stored on or near computer 2200 on a computer-readable medium. Alternatively, recording media such as hard disks or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as computer-readable media, thereby providing programs to computer 2200 via the network.

[0090] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0091] It should be noted that the execution order of operations, procedures, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before," "prior to," etc., and can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," "next," etc. for convenience, it does not mean that it is essential to perform the operations in that order. [Explanation of Symbols]

[0092] 10...Real object, 12...Image, 20...Virtual object, 22, 24, 26...Tags, 32...Hand, 51...Passage area, 52...Specified area, 60...Depth position indicator, 62...XZ plane, 100...Work support device, 102...Image acquisition unit, 104...Target selection unit, 106...Area identification unit, 110...Sensor unit, 112...Gesture detection unit, 116...Gesture information storage unit, 118...Parameter information storage unit, 119...Parameter change unit, 120...Control unit, 122...Display unit

Claims

1. A display unit that displays an image of a three-dimensional space including real objects representing structures in the real world and virtual objects manipulated by the operator, A target selection unit that selects at least one target object from one or more virtual objects to be operated on by the operator, A region identification unit identifies a designated region in the three-dimensional space based on gesture information indicating the physical state of the operator, and identifies related objects from the real objects and virtual objects included in the designated region. A parameter modification unit that modifies parameters relating to the target object based on the aforementioned related object. Equipped with, The virtual object includes a first tag to which a state parameter indicating the state of the real object is assigned, and a second tag to which a position parameter including the position information of the virtual object in the depth direction in the three-dimensional space is assigned. The region identification unit, when the second tag is moved on the two-dimensional plane of the image based on the operator's gesture information, identifies the region that overlaps with the second tag after the movement as the designated region. The parameter modification unit changes the display position of the target object in the three-dimensional space by modifying the position parameter of the second tag based on the position parameter of the related object. Work support device.

2. After the target object is moved, the second tag is displayed in a position that overlaps with the associated object, and the first tag is displayed in a position that does not overlap with the associated object. The work support device according to claim 1.

3. When the operator moves the second tag in the image, the second tag is moved separately from the first tag. After changing the position parameter of the target object, the first tag is concatenated to the second tag and displayed. The work support device according to claim 1.

4. The virtual object further includes a third tag to which a group parameter is assigned, indicating which group the corresponding real object belongs. The parameter modification unit modifies the group parameter so that the real object corresponding to the first virtual object and the real object corresponding to the second virtual object belong to the same group when the third tag of the first virtual object is moved to a position where it overlaps with the third tag of the second virtual object. The work support device according to claim 1.

5. When the operator moves the third tag in the image, the third tag is moved separately from the first tag and the second tag. The work support device according to claim 4.

6. The region identification unit identifies the designated region based on the position through which the operator's line of sight passes in the three-dimensional space, identifies the related object included in the designated region, and identifies the object as the related object if the related object identified by the movement of the second tag matches the related object identified by the operator's line of sight. A work support device according to any one of claims 1 to 5.

7. When the region identification unit identifies the related object based on the operator's line of sight, it identifies the real object or virtual object that the operator's line of sight first passes over as the related object. The work support device according to claim 6.

8. The display unit displays a depth position indicator that specifies the position in the depth direction of the three-dimensional space, The region identification unit identifies the associated object based on the position specified by the depth position indicator. A work support device according to any one of claims 1 to 6.

9. When the region identification unit detects the pre-set gesture information, it rotates and displays the designated region. A work support device according to any one of claims 1 to 6.

10. The region identification unit rotates and displays the designated region on the condition that multiple objects included in the designated region overlap in the depth direction of the three-dimensional space. The work support device according to claim 9.

11. The display unit displays a list of the real objects and virtual objects included in the designated area. The region identification unit identifies the object selected from the list as the associated object. A work support device according to any one of claims 1 to 10.

12. The system displays an image of a three-dimensional space that includes real-world objects representing structures and virtual objects manipulated by the user. From one or more virtual objects, select at least one target object that will be the target of the operator's operation, Based on the gesture information indicating the physical state of the operator, a designated area in the three-dimensional space is identified, and related objects are identified from the real objects and virtual objects included in the designated area. Based on the aforementioned related object, modify the parameters relating to the target object. A work support method, The virtual object includes a first tag to which a state parameter indicating the state of the real object is assigned, and a second tag to which a position parameter including the position information of the virtual object in the depth direction in the three-dimensional space is assigned. When identifying the designated area, if the second tag is moved on the two-dimensional plane of the image based on the operator's gesture information, the area overlapping with the second tag after the move is identified as the designated area. By changing the position parameter of the second tag based on the position parameter of the associated object, the display position of the target object in the three-dimensional space is changed. Work support method.

13. The virtual object further includes a third tag to which a group parameter is assigned, indicating which group the corresponding real object belongs. When the third tag of the first virtual object is moved to a position where it overlaps with the third tag of the second virtual object, the group parameter is changed so that the real object corresponding to the first virtual object and the real object corresponding to the second virtual object belong to the same group. The work support method according to claim 12.

14. A program for causing a computer to perform the work support method described in claim 12 or 13.

Images