Cross-reality device, processing device, generation method, processing method, program and storage medium

Abstract

To provide a cross-reality device, a processing device, a generation method, a processing method, a program, and a storage medium with which virtual objects can be prepared more easily.SOLUTION: A cross-reality device comprises an imaging device, a display device, and a processing device. The imaging device is configured to acquire an image. The display device is configured to display a virtual space. The processing device is configured to generate an object in the virtual space in response to a command input by a person, and change a position and a size of the generated object in accordance with movement of the hand.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a cross-reality device, a processing device, a generating method, a processing method, a program, and a storage medium. [Background technology]

[0002] In recent years, cross reality (XR) devices are being used to make work easier. XR devices are a general term for mixed reality (MR) devices, virtual reality (VR) devices, augmented reality (AR) devices, and alternative reality (SR) devices. By displaying virtual objects on XR devices, work can be supported in various ways. There is a demand for technology that can more easily prepare these virtual objects for XR devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a cross-reality device, a processing device, a generation method, a processing method, a program, and a storage medium that can more easily prepare virtual objects. [Means for solving the problem]

[0005] A cross reality device according to an embodiment includes a camera, a display device, and a processing device. The camera captures an image. The display device displays a virtual space. The processing device detects a person's hand from the image. The processing device generates an object in the virtual space in response to a command input by the person, and changes a position and a size of the generated object in response to a movement of the hand. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 3] FIG. 3 is a schematic diagram showing the state of real space. [Figure 4] FIG. 4 is a schematic diagram showing the state during work. [Diagram 5] 5(a) to 5(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 6] 6(a) to 6(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 7] 7(a) to 7(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 8] 8(a) to 8(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 9] 9(a) to 9(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 10] 10(a) and 10(b) are schematic diagrams for explaining an embodiment of the present invention. [Figure 11] 11(a) to 11(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 12] 12(a) and 12(b) are schematic diagrams for explaining an embodiment of the present invention. [Figure 13]13(a) to 13(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 14] 14(a) to 14(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 15] 15(a) and 15(b) are schematic diagrams for explaining an embodiment of the present invention. [Figure 16] 16(a) to 16(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 17] 17(a) to 17(c) are schematic diagrams for explaining an embodiment of the present invention. [Figure 18] FIG. 18 is a schematic diagram for explaining an embodiment of the present invention. [Figure 19] 19(a) and 19(b) are schematic diagrams for explaining an embodiment of the present invention. [Figure 20] FIG. 20 is a flowchart illustrating a processing method according to an embodiment. [Figure 21] FIG. 21 is a flowchart illustrating a processing method according to an embodiment. [Figure 22] 22(a) and 22(b) are schematic diagrams showing the state during work. [Diagram 23] FIG. 23 is a schematic diagram showing the state during work. [Figure 24] 24(a) and 24(b) are schematic diagrams showing the state during work. [Diagram 25] 25(a) and 25(b) are schematic diagrams showing the state during work. [Figure 26] FIG. 26 is a flowchart illustrating a processing method according to an embodiment. [Figure 27] FIG. 27 is a schematic diagram for explaining an embodiment of the present invention. [Figure 28] 28(a) and 28(b) are schematic diagrams for explaining an embodiment of the present invention. [Figure 29] FIG. 29 is a schematic diagram for explaining an embodiment of the present invention. [Diagram 30] FIG. 30 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] The embodiment of the present invention is used when a user generates an object to be displayed in a virtual space. By using the embodiment, the user can generate a virtual object more easily.

[0009] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. The processing system 1 according to the embodiment includes a processing device 10, an imaging device 20, a display device 30, an input device 40, and a storage device 50.

[0010] The processing device 10 executes various processes related to the generation of virtual objects. For example, the processing device 10 executes detection of a person's hand appearing in an image, detection of a command, generation of an object, storage of the object, etc. The processing device 10 includes a central processing unit (CPU), a memory for storing or executing a program, various interfaces, etc.

[0011] The image capturing device 20 captures images of a part of a person (user)'s body and the surroundings. The image capturing device 20 continuously captures images and stores them in the storage device 50. The image capturing device 20 includes, for example, a camera capable of capturing RGB images. It is more preferable that the camera is capable of capturing depth information in addition to the RGB images. The display device 30 displays a virtual space different from the real world to the user. The display device 30 includes a monitor or a projector.

[0012] The input device 40 is used by a user to input data to the processing device 10. The input device 40 includes a microphone. A mouse, a keyboard, a touchpad, or the like may also be used as the input device 40. In addition to the input device 40, data may also be input to the processing device 10 using hand gestures, a virtual input device, or the like.

[0013] The storage device 50 stores data necessary for processing by the processing device 10, data obtained by processing by the processing device 10, etc. The storage device 50 includes, for example, one or more selected from a Hard Disk Drive (HDD) and a Solid State Drive (SSD).

[0014] The processing device 10 is electrically connected to the photographing device 20, the display device 30, the input device 40, and the storage device 50. The processing device 10 may be connected to the photographing device 20, the display device 30, the input device 40, and the storage device 50 by wired communication or wireless communication. A network HDD (NAS) or the like may be used as the storage device 50, and the processing device 10 and the storage device 50 may be connected via a network.

[0015] 1, the processing device 10 specifically includes functions as an acquisition unit 11, a detection unit 12, a control unit 13, a generation unit 14, and an output unit 15. The acquisition unit 11 acquires images acquired by the imaging device 20, data input by the input device 40, and the like in real time.

[0016] The detection unit 12 detects the user's hand appearing in the image. The detection unit 12 measures the three-dimensional position of each point of the detected hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may be used as the position of the hand. Or, the position of the center of the entire hand may be used as the position of the hand.

[0017] The detection unit 12 repeats hand detection for continuously acquired images and performs hand tracking. The detection unit 12 also detects hand gestures from time-series changes in the detected hand positions. For example, the detection unit 12 calculates a similarity between the change in hand position and the hand movement of each predefined hand gesture. When the similarity for any hand gesture exceeds a preset threshold, the detection unit 12 determines that the user's hand movement indicates that hand gesture.

[0018] When the acquisition unit 11 acquires voice data, the detection unit 12 detects a voice command from the voice. For example, the detection unit 12 executes voice recognition and converts the user's utterance into a character string. The detection unit 12 determines whether the utterance includes a character string of any of the predefined voice commands. When the utterance includes a character string of any of the voice commands, the detection unit 12 determines that the user is uttering that voice command.

[0019] Both hand gestures and voice commands are types of commands. A definition file in which multiple commands and the functions of each command are defined is prepared in advance. The control unit 13 transmits an instruction to the generation unit 14 to execute a function corresponding to the detected command (hand gesture or voice command).

[0020] The generation unit 14 generates virtual objects and changes the position, size, shape, etc. of the objects in response to instructions. The output unit 15 outputs a video signal of the virtual space including the generated or changed objects to the display device 30. The display device 30 displays the virtual space based on the input video signal.

[0021] Hereinafter, the details of the embodiment of the present invention will be described with reference to a specific example. Here, an example in which the processing system 1 is implemented as an MR device will be described. In the MR device, a virtual space is displayed superimposed on a real space. A user can interact with an object displayed in the virtual space.

[0022] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. The processing system 1 shown in Fig. 1 is implemented as, for example, an MR device 100 shown in Fig. 2. The MR device 100 includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0023] The processing device 150 is an example of the processing device 10. The projection device 121 and the projection device 122 are an example of the display device 30. The image camera 131 and the depth camera 132 are an example of the imaging device 20. The microphone 141 is an example of the input device 40. The storage device 170 is an example of the storage device 50.

[0024] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0025] The projection device 121 and the projection device 122 display the detection result of the user's body, a virtual object, etc. on the lens 111 and the lens 112. The user is a person who wears the MR device 100. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0026] The lenses 111 and 112 are optically transparent. A user can view the real world through the lenses 111 and 112. The user can also view information projected onto the lenses 111 and 112 by the projection devices 121 and 122. Information is displayed superimposed on the real space through projection by the projection devices 121 and 122.

[0027] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0028] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0029] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0030] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0031] FIG. 3 is a schematic diagram showing the state of real space. An article 200 shown in FIG. 3 exists in a real space. The article 200 is a cylindrical member. The inside of the article 200 is hollow, and fastening points 201 to 206 exist at the bottom of the hollow. In this example, a screw fastening operation is performed on the article 200. In the fastening operation, a wrench and an extension bar are used to fasten the screws into each of the fastening points 201 to 206. During the fastening operation, the MR device is used to support this operation. For example, the appropriate posture of the tool, the fastening position, the position where the hands are placed when fastening, and the like are shown by the virtual object.

[0032] A marker 210 is provided near the article 200 to be worked on. In the illustrated example, the marker 210 is an AR marker. As described below, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 150 sets a three-dimensional coordinate system of the virtual space based on a plurality of points indicated by a hand gesture.

[0033] FIG. 4 is a schematic diagram showing the state during work. When a screw is fastened to the article 200, the worker places the screw at any of the fastening locations. The worker fits one end of the extension bar 251 onto the screw. The worker fits a wrench 252 onto the other end of the extension bar 251. As shown in FIG. 4 , the worker holds both ends of the wrench 252 with both hands. In this state, the worker turns the wrench 252 to rotate the extension bar 251, thereby tightening the screw at the fastening location.

[0034] When performing work using a screw, it is preferable that the tool is used in an appropriate posture. If the tool is used in an inappropriate posture, the object may be damaged. There is also a possibility that the worker may be injured. Here, an example will be described in which a virtual jig is provided so that the worker uses the tool in an appropriate posture.

[0035] First, when preparing an object, the position of the article 200 relative to the marker 210 is adjusted. The positional relationship between the article 200 and the marker 210 when preparing the object is adjusted to be the same as the positional relationship between the article 200 and the marker 210 when the work is performed. This allows the prepared object to be displayed overlapping the article 200 at an appropriate position during the work.

[0036] Figures 5(a) to 5(c), 6(a) to 6(c), 7(a) to 7(c), 8(a) to 8(c), 9(a) to 9(c), 10(a), 10(b), 11(a) to 11(c), 12(a), and 12(b) are schematic diagrams for explaining an embodiment of the present invention. A specific example of the embodiment will be described with reference to the drawings. First, the processing device 10 detects the left hand 301 and the right hand 302 of the user by hand tracking. The user is a person wearing the MR device. As shown in FIG. 5(a), the user points with his / her finger at a position where an object is to be generated. At this time, the position of the fingertip is measured by the processing device 10. Next, as shown in FIG. 5(b), the user speaks the shape of the object to be generated. The shape is registered in advance as a voice command in the storage device 50. The processing device 10 detects the voice command included in the utterance 311. As shown in FIG. 5(c), the processing device 10 generates and displays an object 321 having a shape corresponding to the voice command.

[0037] Next, the user adjusts the position of the generated object 321. First, as shown in FIG. 6(a), the user touches a part of the object 321 with the fingers of one hand. The user points to the destination of the part of the object 321 with the fingers of the other hand. The amount and direction of movement of the object 321 are specified by the fingers of both hands of the user. In this state, as shown in FIG. 6(b), the user utters a voice command for moving the object. Based on the detection result of the fingertip and the detection result of the voice command included in the utterance 312, the processing device 10 moves the object 321 as shown in FIG. 6(c). Note that if it is not necessary to move the object 321 from the generated position, the movement process can be omitted.

[0038] Next, the user adjusts the size of the object 321. First, as shown in FIG. 7(a), the user indicates the width of the object 321 with the fingers of both hands. As shown in FIG. 7(b), the user specifies the dimension to be adjusted by speech 313. The processing device 10 changes the width of the object 321 based on the detection result of the fingertips and the detection result of the voice command included in the speech 313, as shown in FIG. 7(c). In the illustrated example, the cylindrical object 321 is set so that the depth also changes according to the change in width. The width and depth may be adjustable separately.

[0039] After the width is adjusted, the user indicates the height of the object 321 with the fingers of both hands, as shown in Fig. 8(a). The user specifies the dimension to be adjusted by speech 314, as shown in Fig. 8(b). The processing device 10 changes the height of the object 321 based on the detection result of the fingertips and the detection result of the voice command included in the utterance 314, as shown in Fig. 8(c).

[0040] An attribute can also be assigned to the generated object. As shown in FIG. 9(a), the user indicates the range of the object to which the attribute is assigned with the fingers of both hands. In the illustrated example, the range between the fingers of the left hand 301 and the fingers of the right hand 302 in the height direction is shown. The user utters a voice command corresponding to the attribute to be assigned, as shown in FIG. 9(b). The processing device 10 detects the voice command included in the utterance 315. As shown in FIG. 9(c), the processing device 10 assigns the attribute of "jig first stage" to the range 321a of the object 321 indicated by the fingers of both hands.

[0041] The user can further assign attributes to the object 321 in the same manner as the methods shown in Figs. 9(a) to 9(c). For example, as shown in Fig. 10(a), the user points with his / her finger at area 321b located below area 321a. The user assigns the attribute of "jig, second tier" to area 321b by speech 316. As shown in Fig. 10(b), the user points with his / her finger at area 321c located below area 321b. The user assigns the attribute of "jig, third tier" to area 321c by speech 317.

[0042] In this example, the virtual "jig" represents an area where the tool must not enter during work. The jig is positioned so that the tool and the jig will come into contact when the tool is used in an incorrect orientation. During actual work, the processing device 10 calculates the distance between the object to which the attribute of "jig" is assigned and the tool. If the distance falls below a predetermined threshold and it is determined that the tool is in contact with the jig, the processing device 10 issues an alert.

[0043] Furthermore, the deeper the tool is inserted in the cavity of the article 200 in an incorrect position, the stronger the alert issued by the processing device 10. That is, the alert issued when the tool contacts the first stage of the jig, the alert issued when the tool contacts the second stage of the jig, and the alert issued when the tool contacts the third stage of the jig are different from each other. The alert issued when the tool contacts the second stage of the jig is stronger than the alert issued when the tool contacts the first stage of the jig, and weaker than the alert issued when the tool contacts the third stage of the jig. For example, the deeper the tool is inserted in an incorrect position, the stronger the message, the louder the warning sound, or the darker the warning color issued by the processing device 10.

[0044] The shape of the object 321 may be processed. For example, the user forms a hole in the object 321 for passing the extension bar 251 through. First, as shown in FIG. 11(a), the user points with his / her finger to the position where the hole is to be formed. As shown in FIG. 11(b), the user utters a voice command for forming the hole. The processing device 10 detects the voice command included in the utterance 318. As shown in FIG. 11(c), the processing device 10 forms a hole 322 in the object 321 in response to the voice command.

[0045] To adjust the size of the hole 322, the user selects the hole 322 with a finger or a voice command. The user can adjust the position, width, depth, and height of the hole 322 in a manner similar to that shown in Figures 6(a) to 8(c).

[0046] Once the shape, position, size, and attributes of the object have been set, the user saves the generated object. For example, the user points at the object with his / her finger and utters a voice command to save it. The processing device 10 detects the voice command included in the utterance and saves the data of the object pointed at with the finger.

[0047] As a command for generating or changing an object, various hand gestures may be used other than voice as shown in Fig. 12(a). Alternatively, as shown in Fig. 12(b), a panel 325 that is a virtual input device may be displayed. The user selects (touches) a command displayed on the panel 325. The processing device 10 detects the selected command by hand tracking. The processing device 10 executes processing according to the command.

[0048] The order of setting the position, size, and attributes of an object can be changed as appropriate. If adjustment of the position or size is not required, or if setting of attributes is not required, those steps may be omitted.

[0049] As described above, the processing device 10 can generate an object in a virtual space in response to a command input by a user. Also, the processing device 10 can adjust the position and size of the object as appropriate in response to the command and hand movement. Furthermore, the processing device 10 can accept attribute settings for the object. The processing device 10 associates the position, size, and attributes with an ID for identifying each object and stores them.

[0050] Figures 13(a) to 13(c), 14(a) to 14(c), 15(a), 15(b), 16(a) to 16(c), 17(a) to 17(c), 18, 19(a), and 19(b) are schematic diagrams for explaining an embodiment of the present invention. 5(a) to 11(c) have been described as examples in which an object is generated to indicate an area where a tool must not enter. From FIG. 13(a) onwards, an example is described in which an object is generated to indicate a position where a hand is placed during work.

[0051] First, as shown in FIG. 13(a), the user points with a finger of one hand to a position where an object is to be generated. The user utters a voice command for setting the position of a part of the object. The processing device 10 detects the voice command included in the utterance 331. The processing device 10 measures the position of the fingertip and acquires the position. Next, as shown in FIG. 13(b), the user points with a finger of the other hand to a position of another part of the object and utters a voice command for generating the object. The processing device 10 detects the voice command included in the utterance 332 and generates an object 341a as shown in FIG. 13(c).

[0052] In the illustrated example, the one hand is a left hand 301. The other hand is a right hand 302. In this example, it is preset that the position of the fingers of the left hand 301 is recognized as the start point of a vector, and the position of the fingers of the right hand 302 is recognized as the end point of the vector. Objects of preset shapes are generated at the start point and end point of the vector, respectively.

[0053] The user can adjust the position and size of object 341a, similar to object 321. As shown in Fig. 14(a), the user may be able to move object 341a in parallel by speech 333 and finger movement. Or, as shown in Fig. 14(b), the user may be able to set a rotation axis AX for movement by speech 334 and pointing. As shown in Fig. 14(c), the user can move object 341a around rotation axis AX by speech 335 and finger movement.

[0054] Next, the user touches the object 341a with a finger as shown in FIG. 15(a). As shown in FIG. 15(b), an attribute is assigned to the object 341a by speech 336. In this example, the object 341a is provided at a position where the hand should be placed during work. Therefore, if the work is performed appropriately, the hand touches the object 341a. The user assigns to the object 341a a function of detecting that the hand has touched the object 341a by the "guide ball" of speech 336. During actual work, the processing device 10 calculates the distance between the hand and the object 341a. When the distance falls below a predetermined threshold, the processing device 10 determines that the hand has touched the object 341a. Instead of the hand, a function of detecting that a tool has touched the object 341a may be assigned to the object 341a.

[0055] Next, as shown in Fig. 16(a), the user indicates the position where another object is to be generated with the fingers of one hand. The user indicates the orientation of the object with the fingers of the other hand. As shown in Fig. 16(b), the user utters a voice command to generate an object. The processing device 10 detects the voice command included in the utterance 337, and generates an object 342 as shown in Fig. 16(c). The object 342 displays data related to the work during the work. The orientation of the object 342 displaying the data can be specified by the left hand 301 and the right hand 302.

[0056] As shown in Fig. 17(a), the user touches the object 342 with a finger. As shown in Fig. 17(b), the user indicates data to be displayed on the object 342 by speech 338. The processing device 10 detects the voice command and content included in the utterance 338, and causes the object 342 to display the data, as shown in Fig. 17(c).

[0057] Thereafter, by repeatedly generating an object similar to the object 341a, objects 341a to 341f are set corresponding to the fastening points 201 to 206 of the article 200, respectively, as shown in Fig. 18. Alternatively, a command for copying the object 341a may be prepared. For example, the user can use the copy command to duplicate an object identical to an object that has already been generated.

[0058] The start point to the end point of an object or the orientation of an object may be indicated by the fingers of both hands, or by the orientation of the palm of a hand, as shown in Fig. 19(a). In the illustrated example, the orientation of an object is set to the orientation from the back of the hand to the palm.

[0059] As shown in Fig. 19(b), a command for aligning the positions of multiple objects may be prepared. In the illustrated example, the top ends of multiple objects are aligned to the top end of object 341a pointed to by a finger by speech 339. In addition to the illustrated example, it is also possible to align multiple objects at the bottom end, left end, right end, front end, or back end. Center alignment in the vertical direction, center alignment in the horizontal direction, center alignment in the depth direction, etc. may also be performed.

[0060] 20 and 21 are flow charts illustrating a processing method according to an embodiment. FIG. 20 shows a method for generating an object. First, the processing device 10 performs hand tracking to obtain the three-dimensional position of the fingertip (step S11). The processing device 10 refers to a command master 51 stored in the storage device 50. The command master 51 registers the ID, name, and function of each command. The processing device 10 detects a command by comparing the command registered in the command master 51 with an input from the user (hand movement or voice). When the processing device 10 detects a command for generating an object, it generates the object (step S12).

[0061] When the processing device 10 detects a command to change the size of the object, it changes the size of the object (step S13). When the processing device 10 detects a command to give an attribute to the object, it gives the attribute to the object (step S14). When the processing device 10 detects a command to process the object, it processes the object (step S15).

[0062] When the generation of the object, the change in size, and the addition of attributes are completed, the user speaks a command to save the object. When the processing device 10 detects the command, it saves the generated object (step S16). The object is saved in the object master 52 of the storage device 50. In the object master 52, the ID, shape, size, attributes, etc. of each object are registered.

[0063] The processing device 10 judges whether the generation of objects is to be continued (step S17). For example, the processing device 10 continues the generation of objects until a command indicating the end of the generation of objects is received. A termination condition such as the number of objects to be generated may be determined in advance. When the termination condition is satisfied, the processing device 10 terminates the generation of objects. If the generation of objects is to be continued, step S12 is executed again.

[0064] The order of steps S12 to S15 can be changed as appropriate. The position of the object may be adjusted between steps S11 and S16. In this example, the final display position of the object is determined by associating the object with a task step, which will be described later.

[0065] Fig. 21 shows a method for arranging objects and linking them to tasks. When an object is generated, each object is arranged and linked to a task so that the object is displayed in accordance with an item during an actual task.

[0066] First, the processing device 10 performs hand tracking and acquires the three-dimensional position of the fingertip (step S21). Next, the processing device 10 detects a marker in the image and sets the base point of a three-dimensional coordinate system based on the marker (step S22). The processing device 10 accepts the selection of a task step to be associated with the object, and reads the selected task step from the task master 53 (step S23).

[0067] For example, when an article is manufactured, a plurality of processes are performed. One process is made up of one or more work steps. In one work step, a fastening operation may be performed. The work master 53 includes data related to the work steps and is stored in the storage device 50. The work master 53 includes an ID of each work step, a name of each work step, an ID of an object to be worked on in each work step, a name of an object to be worked on, and the like. When a screw is fastened in a certain work step, the ID of each fastening point, the position of each fastening point, data of the tool to be used, a target torque value, and the like are also stored in the work master 53.

[0068] The work step is selected by the user. The work step may be linked to a marker for setting a base point. In this case, the processing device 10 reads the work step linked to the marker.

[0069] The processing device 10 accepts the selection of the object to be linked, and reads the selected object from the object master 52 (step S24). The processing device 10 displays the read object (step S25). The processing device 10 accepts a command input from the user, and moves the object in response to the command (step S26). At this time, the size, orientation, etc. of the object may be adjusted. The processing device 10 saves the position of the object after the movement in the object master 52. The processing device 10 also saves the data of the object in the object master 52, and links it to the data of the read work step (step S27).

[0070] If the object to be placed is associated with a fastening point, the fastening point is further selected in step S23. The processing device 10 associates the data of the placed object with the ID of the selected fastening point, and stores the data in the task master 53.

[0071] The processing device 10 judges whether the object arrangement is to be continued (step S28). For example, the processing device 10 continues the object arrangement until a command indicating the end of the object arrangement is received. A termination condition may be determined in advance. When the termination condition is satisfied, the processing device 10 ends the object arrangement. If the object arrangement is to be continued, step S24 is executed again.

[0072] Through the above process, the creation of objects, the placement of objects, and the association of objects with work steps are completed. The prepared objects are displayed to the worker during the actual work. The worker can perform the work more smoothly by seeing the objects displayed.

[0073] Advantages of the embodiment will be described. Displaying virtual objects on an XR device can assist with work in a variety of ways. Objects in a virtual space are usually generated using dedicated software on a PC. For this reason, it is difficult for people with little specialized knowledge of software to set up the objects. For example, a worker can easily guess what kind of object should be displayed in what position to make it easier to work. However, general workers are not familiar with software related to virtual spaces and find it difficult to set up objects.

[0074] Regarding this problem, according to an embodiment of the present invention, a virtual object can be generated using an XR device. Specifically, the processing device 10 detects a person's hand from an image acquired by the imaging device 20. The processing device 10 generates a virtual object in a virtual space based on a hand gesture or a voice command. In other words, a person wearing an XR device can make the processing device 10 generate an object simply by inputting a hand gesture or a voice command. In addition, the processing device 10 detects the movement of the person's hand by hand tracking, and changes the position and size of the object according to the hand movement. A person can adjust the position and size of an object by moving their hand while inputting a command to change the position and size.

[0075] According to the embodiment, the generation, position adjustment, and size adjustment of an object can be executed by using commands and hand movements prepared in advance, so that even a person lacking in specialized knowledge can easily prepare a virtual object.

[0076] As a specific example, for a task, a task master 53 in which the procedure of the task is defined is generally prepared in advance. If a command master in which hand gestures, voice commands, etc. are defined is prepared, an embodiment of the present invention can be implemented using the task master 53. For example, a worker involved in the task or a manager thereof prepares an object to support the task. By appropriately linking the prepared object with the data of the task master 53, the prepared object can be displayed during the actual task.

[0077] When an object is prepared to support a task in real space, the position, orientation, etc. of the object must be matched to an item in real space. That is, the three-dimensional coordinate system of the virtual space when the object is prepared is set to be the same as the three-dimensional coordinate system of the virtual space when the task is performed. Also, the positional relationship between the origin of the three-dimensional coordinate system when the object is prepared and the task target is set to be the same as the positional relationship between the origin of the three-dimensional coordinate system and the task target when the task is performed.

[0078] In order to facilitate these settings, it is preferable that a marker 210 for setting a base point is prepared in real space as shown in Fig. 3. When an object is prepared, the processing device 10 detects the marker 210 from the image and sets a three-dimensional coordinate system with the marker 210 as the base point. When work is performed, the processing device 10 also detects the marker 210 from the image and sets a three-dimensional coordinate system with the marker 210 as the base point. If the positional relationship between the marker 210 and the item being worked on does not change, the prepared object can be displayed in accordance with the item during work.

[0079] An attribute indicating the function of the object can also be set for the object. In the above example, the attribute is given the function of detecting when a specific object comes into contact with the object. If the attribute of "jig" is given to an object, the processing device 10 outputs an alert when a tool comes into contact with the object. If the attribute of "guide ball" is given to an object, the processing device 10 estimates that a screw is being turned into the corresponding fastening point when a hand or tool comes into contact with the object.

[0080] It is particularly preferable that the objects are displayed corresponding to the fastening points. When a large item is manufactured, there are many fastening points, and it may be difficult for the worker to grasp all of the fastening points. In addition, as shown in FIG. 3, if some fastening points are located in places that are difficult to see, it may happen that a screw is left unfastened.

[0081] When an object is displayed corresponding to a fastening point, it is preferable that data regarding the fastening point is linked to the object. It is also preferable that the object is provided with a function (attribute) of detecting contact with a hand or a tool. According to this method, when a hand or a tool is in contact with an object, it can be estimated that a screw is being turned into the fastening point corresponding to the object. Based on the estimation result, the processing device 10 may automatically create a work record indicating which fastening point the screw has been turned into.

[0082] 22(a), 22(b), 23, 24(a), 24(b), 25(a) and 25(b) are schematic diagrams showing the state during work. When a fastening operation is to be performed, a work step is selected and input to the processing device 10. The processing device 10 reads data of the input work step from the work master 53. The processing device 10 reads an object linked to the work step.

[0083] When a screw is fastened to the article 200 shown in Fig. 3, the object 321, objects 341a to 341f, and object 342 are read as shown in Fig. 22(a). The processing device 10 displays the object 321, objects 341a to 341f, and object 342 in the virtual space so as to be superimposed on the article 200 in the real space as shown in Fig. 22(b).

[0084] The fastening points 201-206, not shown in FIG. 22(b), are located at the bottom of the cavity of the article 200. It is difficult for the worker to visually recognize the fastening points 201-206. The objects 341a-341f are displayed at positions where the hands should be placed when screws are fastened into the fastening points 201-206, respectively. By displaying the objects 341a-341f, the worker can easily understand where to position the hands when working.

[0085] Each of the objects 341a to 341f includes a spherical portion away from the fastening point and a linear portion connecting the spherical portion and the fastening point. The linear portion indicates the appropriate position and posture of the tool when the hand is located on the spherical portion. As shown in the figure, the position where the hand should be located, the position of the tool, and the posture of the tool are indicated, so that the worker can perform the work more smoothly.

[0086] When a hand comes into contact with any of the objects 341a to 341f, the processing device 10 detects the contact. In the example shown in Fig. 23, the hand comes into contact with the object 341f. The processing device 10 can estimate that a screw is fastened at a fastening point corresponding to the object 341f. Here, the fastening point estimated to be where a screw is fastened by contacting an object is referred to as an "estimated point."

[0087] Preferably, a digital tool is used in the work. The processing device 10 receives detection values ​​from the digital tool. The processing device 10 can use the detection values ​​to determine whether screw tightening to the estimated location has been completed. When it is determined that screw tightening has been completed, the processing device 10 inputs the work result into the work record. This method makes it possible to automatically create a more accurate work record.

[0088] For example, the digital tool is a digital torque wrench or a digital torque driver, and the detection value is a torque value detected by the digital torque wrench or the digital torque driver. The digital torque wrench or the digital torque driver detects the torque value and transmits it to the processing device 10. If the torque value exceeds a predetermined threshold, the processing device 10 determines that the screw tightening is completed. The digital torque wrench or the digital torque driver may determine whether or not a torque exceeding a predetermined threshold has been detected. In that case, the digital torque wrench or the digital torque driver may output the determination result as the detection value instead of the torque value. The digital torque wrench or the digital torque driver may output both the determination result and the torque value. The processing device 10 may link the received detection value to data related to the estimated location.

[0089] An attribute may be assigned to the object so that a change in the display mode of the object, output of a message, or output of a sound is executed in response to contact of a predetermined object. For example, the processing device 10 changes the display mode of the object 341f. The display mode of the object 341f that is contacted by a hand is different from the display mode of the object 341f that is not contacted by a hand. In the illustrated example, the color of the object 341f is changed. The size or shape of the object 341f may be changed. The contact of the hand with the object 341f may be notified by a message or sound. This allows the user to easily determine whether or not a predetermined object is contacting the virtual object.

[0090] There is a case where a screw is fastened at one location, and then another screw is fastened at another location, and then the screw at the one location is retightened. In this case, the processing device 10 may change the display mode of the objects 341a to 341f according to the number of times the screw is fastened. In the example shown in FIG. 24(a), a screw is fastened at the fastening location corresponding to the object 341f. No screw is fastened at the fastening locations corresponding to the other objects 341a to 341e. In the example shown in FIG. 24(b), the screw at the fastening location corresponding to the object 341f is retightened. The screws at the fastening locations corresponding to the other objects 341a to 341e have not yet been retightened. The number of times the screw is fastened at the fastening location corresponding to the object 341f is different from the number of times the screw is fastened at the fastening locations corresponding to the objects 341a to 341e. Therefore, the display mode of the object 341f is different from the display mode of the objects 341a to 341e.

[0091] The object 321 is assigned a jig attribute. For example, as shown in FIG. 25(a), a plurality of markers 251a are attached to the extension bar 251. The processing device 10 detects the plurality of markers 251a from the image. The processing device 10 measures the position of each marker 251a. The positional relationship between the plurality of markers 251a and the extension bar 251 is registered in advance. Based on this positional relationship, the processing device 10 calculates the position P1 of the extension bar 251 from the position of each marker 251a. When the position P1 of the extension bar 251 overlaps with a portion of the object 321 other than the hole 322, the processing device 10 determines that the extension bar 251 is in contact with the object 321.

[0092] As with the object 341f, an attribute for indicating contact to the user can be added to the object 321. In the example shown in Fig. 25(b), the processing device 10 displays a message 326 as an alert. The processing device 10 may change the color, size, shape, etc. of the object 321.

[0093] By assigning attributes to objects, it becomes possible to support the fastening work in various ways. The specific functions of the attributes are registered in the command master 51. The above-mentioned various functions may be registered in one attribute. Alternatively, only a single function may be registered in one attribute, and multiple attributes may be assigned to one object.

[0094] FIG. 26 is a flowchart illustrating a processing method according to an embodiment. Fig. 26 shows a process flow during actual work. For example, the first processing device executes the processes shown in Figs. 20 and 21 to prepare an object. After that, the second processing device executes the processes shown in Fig. 26. The second processing device may be the same as the first processing device, or may be different from the first processing device. That is, the MR device used in the work may be the same as the MR device used when preparing the object, or may be different.

[0095] First, the processing device 10 accepts the selection of a work step (step S31). The work step is selected by the worker. The work step to be executed may be instructed by a higher-level system, and the processing device 10 may accept the selection based on the instruction. Alternatively, the processing device 10 may determine the work step to be executed based on data obtained from the imaging device 20 or other sensors, and the processing device 10 may accept the selection based on the determination result.

[0096] The processing device 10 acquires data of the selected task step by referring to the task master 53. The task master 53 includes task step data 53a, base point data 53b, and fastening point data 53c.

[0097] The work step data 53a stores a work step ID, a work step name, the ID of the item to be worked on, the item name, the tool model and tool ID used, etc. The tool model indicates a classification of tools based on structure, external shape, performance, etc. The processing device 10 can accept any of the work step ID, work step name, item ID, or item name as a work selection. When there are multiple candidates for the work step selection, the processing device 10 may output an inquiry as to which candidate to select.

[0098] The processing device 10 refers to the base point data 53b. The base point data 53b stores a method for setting a base point for each task. The processing device 10 acquires the method for setting a base point for the selected task, and sets a base point based on an input by the worker (step S32). As described above, it is preferable to use a marker to set the base point.

[0099] The processing device 10 refers to the fastening point data 53c. In the fastening point data 53c, the fastening point ID is stored, and for each fastening point ID, the fastening position, angle, extension bar type, torque value, screw tightening count, mark color, object shape, and display mode are stored. The fastening position indicates the position of each fastening point. The angle indicates the angle of the tool or extension bar when tightening a screw at each fastening point. The extension bar type indicates the classification of the extension bar according to the structure, external shape, or performance. The torque value indicates the magnitude of the torque required to tighten a screw at each fastening point. The screw tightening count indicates the number of times the screw is tightened at each fastening point. The mark color is the color of the mark indicating the completion of the screw tightening. The object shape indicates the shape of the object displayed corresponding to each fastening point. The display mode indicates the mode of each object displayed. The display mode is set for each screw tightening count. The screw tightening count, object shape, display mode, etc. can be assigned to the object as attributes by a command when the object is generated.

[0100] The processing device 10 displays the prepared object based on the set base point and the data of the object master 52 (step S33). The processing device 10 repeatedly judges whether a predetermined object has come into contact with the virtual object (step S34). This allows detection of a tool coming into contact with a "jig" object, a hand coming into contact with a "guide ball" object, and the like.

[0101] When the contact of the hand with the "guide ball" object is detected, the processing device 10 judges whether or not a detection value is received from the digital tool (step S35). If a detection value is not received, the processing device 10 executes step S34 again.

[0102] When a hand touches the "guide ball" object and a detection value is received, the processing device 10 links data related to the fastening point to the detection value and records the data in the history data 54 (work record) (step S36). For example, a torque value is linked to the fastening point ID and recorded. In the illustrated example, a work step ID, a tool ID, a tool model, and the like are also linked.

[0103] The processing device 10 judges whether the work step selected in step S31 has been completed (step S37). If the work step has not been completed, the display of the object in step S33 continues.

[0104] In the above, an example in which the processing system 1 according to the embodiment is implemented as an MR device has been described. However, the present invention is not limited to this example, and the processing system 1 according to the embodiment may be implemented as a VR device.

[0105] As an example, an object of an item to be worked on is generated in a virtual space. For that object, an object to assist the work is generated as in the above example. A user of a VR device can experience fastening work in a virtual space including these objects.

[0106] 2, lenses 111 and 112 that do not transmit light are used. Only the virtual space is reflected in the lenses 111 and 112. In order to further enhance the sense of immersion, a goggle-type or helmet-type VR device may be used.

[0107] 27, 28(a), 28(b), and 29 are schematic diagrams for explaining an embodiment of the present invention. 27 shows a virtual space V displayed on the display device 30. In the illustrated example, a virtual desk, shelf, and the like are displayed in the virtual space V. The processing device 10 detects the user's hand and displays the detected hand in the virtual space V.

[0108] As shown in Fig. 28(a), the user points with his / her finger at the position where an object is to be generated in the virtual space V, and utters a work object ID and a voice command. In this example, the work object ID is registered in advance as a voice command. The user specifies the work object ID by utterance 401.

[0109] The processing device 10 detects the utterance 401 and refers to the task target ID stored in the task master. As shown in Fig. 28(b), the processing device 10 generates an object 411 of the same size as the article having the task target ID in the virtual space V. After that, the user can adjust the position or size of the object 411 by a method similar to the method shown in Figs. 6(a) to 8(c). Furthermore, attributes may be assigned to the object 411.

[0110] Thereafter, the object 321, the objects 341a to 341f, and the object 342 are generated by the methods shown in Fig. 5(a) to Fig. 11(c) and the methods shown in Fig. 13(a) to Fig. 17(c). As a result, objects for simulating the fastening work are prepared in the virtual space V as shown in Fig. 29.

[0111] When a VR device is used, the object to be worked on and the object to assist the work are generated in the same virtual space. Therefore, it is not necessary to match the position or size of the object to assist the work to an object in the real space. Therefore, the base point can be set arbitrarily. As shown in FIG. 3, there is no need to use a marker 210.

[0112] In the above example, an example was described in which an XR device is used to assist in work. The embodiment of the present invention is also effective in generating virtual objects for purposes other than work. According to the embodiment of the present invention, even a person with little specialized knowledge about virtual space can easily generate an object in a virtual space.

[0113] In the above-described examples, the example of tightening a screw into a fastening point has been mainly described. The embodiment of the present invention is applicable not only to the case where a screw is tightened into a fastening point, but also to the case where a screw in a fastening point is loosened. For example, a screw in a fastening point is loosened when performing maintenance, inspection, or repair of a product. For example, by displaying a virtual object when loosening a screw, a worker can perform the work more smoothly.

[0114] FIG. 30 is a schematic diagram showing a hardware configuration. 30 is used as the processing device 10 or the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0115] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0116] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0117] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) and the like. The storage device 94 may be used as the storage device 50 or the storage device 170.

[0118] An input interface (I / F) 95 can connect the computer 90 to the input device 40. The CPU 91 can read various data from the input device 40 via the input I / F 95.

[0119] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to the display device 30 via the output I / F 96, and cause the display device 30 to display information.

[0120] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0121] The data processing by the processing device 10 or the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0122] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0123] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0124] The processing system 1 may be implemented by a device other than an XR device. For example, the processing system 1 is implemented by using a general-purpose PC. In that case, a monitor is used as the display device 30. A keyboard, a microphone, a touchpad, a microphone, or the like is used as the input device 40. The photographing device 20 photographs a user at a position away from the user. The user inputs commands to the processing device 10 using the input device 40 while checking the display device 30.

[0125] Embodiments of the invention include the following features. (Feature 1) An imaging device for acquiring an image; A display device that displays a virtual space; a processing device for detecting a human hand from the image; The processing device includes: generating an object in the virtual space in response to a command input by the person; Changing the position and size of the generated object in response to the hand movement. Cross reality devices. (Feature 2) The cross reality device described in feature 1, wherein when a marker present in real space is photographed by the photographing device, the processing device sets a three-dimensional coordinate system of the virtual space based on the marker and changes the position and the size of the object in the three-dimensional coordinate system. (Feature 3) the processing device is capable of accepting a setting of an attribute indicating a function of the object; 3. A cross reality device according to feature 1 or 2, wherein the function is detection of contact of a predetermined object with the object. (Feature 4) 4. The cross reality device according to any one of features 1 to 3, wherein the object is stored in association with data relating to a fastening point of an article in real space. (Feature 5) 5. The cross reality device according to any one of features 1 to 4, wherein the command is input by voice or hand gesture. (Feature 6) The processing device causes the display device to display an object on which a plurality of commands are displayed; 5. The cross reality device according to any one of features 1 to 4, which accepts input of the command selected by the person. (Feature 7) 7. The cross reality device according to any one of Features 1 to 6, wherein the display device displays the virtual space by superimposing it on the real space. (Feature 8) Detects human hands from images, generating an object in a virtual space in response to a command input by said person; A processing device that changes the position and size of the generated object in response to the hand movement. (Feature 9) A first processing device, Detects human hands from images, generating an object in a virtual space in response to a command input by said person; Changing the position and size of the object in response to the hand movement. How to create an object. (Feature 10) The first processing device is Detecting markers from the image, A three-dimensional coordinate system is set based on the marker; modifying the position and the size of the object in the three-dimensional coordinate system; 10. The method of producing according to feature 9. (Feature 11) The second processing device Displaying the object generated by the generation method according to feature 9 or 10 on a display device; Detecting contact between a predetermined object and the object from an image showing the predetermined object; Processing methods. (Feature 12) On the computer, Detecting human hands from images, detecting a command entered by said person; generating an object in a virtual space in response to the detected command; changing a position and a size of the generated object in response to the movement of the hand; program. (Feature 13) A storage medium storing the program according to feature 12.

[0126] According to the embodiments described above, it is possible to provide a cross reality device, a processing device, a generation method, a processing method, a program, and a storage medium that can more easily prepare virtual objects.

[0127] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0128] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0129] 1: Processing system, 10: Processing device, 11: Acquisition unit, 12: Detection unit, 13: Control unit, 14: Generation unit, 15: Output unit, 20: Shooting device, 30: Display device, 40: Input device, 50: Storage device, 51: Command master, 52: Object master, 53: Work master, 53a: Work step data, 53b: Base point data, 53c: Fastening point data, 54: History data, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: MR device, 101: Frame, 111, 112: Lens, 121, 122: Projection device, 131: Image camera, 132: Depth camera, 140: sensor, 141: microphone, 150: processing device, 160: battery, 170: storage device, 200: item, 201-206: fastening point, 210: marker, 251: extension bar, 251a: marker, 252: wrench, 301: left hand, 302: right hand, 311-318: speech, 321: object, 321a-321c: range, 322: hole, 325: panel, 326: message, 331-339: speech, 341a-341f, 342: object, 401: speech, 411: object, AX: rotation axis, V: virtual space

Claims

1. An imaging device for acquiring an image; A display device that displays a virtual space; a processing device for detecting a human hand from the image; The processing device includes: generating an object in the virtual space in response to a command input by the person; Changing the position and size of the generated object in response to the hand movement. Cross reality devices.

2. The cross reality device of claim 1, wherein when a marker present in real space is photographed by the photographing device, the processing device sets a three-dimensional coordinate system of the virtual space based on the marker and changes the position and the size of the object in the three-dimensional coordinate system.

3. the processing device is capable of accepting a setting of an attribute indicating a function of the object; The cross reality device of claim 1 , wherein the function is detection of contact of a predetermined object with the object.

4. The cross reality device according to claim 1 , wherein the object is stored in association with data relating to fastening points of an article in real space.

5. The cross reality device of claim 1 , wherein the commands are input by voice or hand gestures.

6. The processing device causes the display device to display an object on which a plurality of commands are displayed; The cross reality device of claim 1 , which accepts input of the command selected by the person.

7. 7. The cross reality device according to claim 1, wherein the display device displays the virtual space by superimposing it on the real space.

8. Detects human hands from images, generating an object in a virtual space in response to a command input by said person; A processing device that changes the position and size of the generated object in response to the hand movement.

9. A first processing device, Detects human hands from images, generating an object in a virtual space in response to a command input by said person; Changing the position and size of the object in response to the hand movement. How to create an object.

10. The first processing device is Detecting markers from the image, A three-dimensional coordinate system is set based on the marker; modifying the position and the size of the object in the three-dimensional coordinate system; The method of claim 9.

11. The second processing device, Displaying the object generated by the generation method according to claim 9 or 10 on a display device; Detecting contact between a predetermined object and the object from an image showing the predetermined object; Processing methods.

12. On the computer, Detecting human hands from images, detecting a command entered by said person; generating an object in a virtual space in response to the detected command; changing a position and a size of the generated object in response to the movement of the hand; program.

13. A storage medium storing the program according to claim 12.

Images