Mixed reality device, processing method, program and storage medium

Abstract

To provide a mixed reality device with improved convenience, and to provide a processing method, a program, and a storage medium for improving the convenience of the mixed reality device.SOLUTION: A mixed reality device is configured to set an origin of a virtual space by using a marker set in a real space. The mixed reality device is configured to display a virtual object at a preset position in a three-dimensional coordinate system based on the origin. The mixed reality device is configured to acquire a moving amount of the marker with respect to a reference position of the marker, the reference position of the marker being acquired when the origin is set. The mixed reality device is configured to move the origin according to the moving amount when the moving amount exceeds a preset threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] SUMMARY OF THE PRESENT EMBODIMENTS An embodiment of the present invention relates to a mixed reality device, a processing method, a program, and a storage medium. [Background technology]

[0002] Conventionally, mixed reality devices have been used for efficient work. Mixed reality devices can display a virtual space overlaid on a real space, and can provide various information to workers. Workers can perform work more efficiently by referring to the information displayed by the mixed reality device. There is a demand for technology that can further improve the convenience of mixed reality devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a mixed reality device having better convenience. Also, the problem to be solved by the present invention is to provide a processing method, a program, and a storage medium capable of further improving the convenience of the mixed reality device. [Means for solving the problem]

[0005] A mixed reality device according to an embodiment sets a base point of a virtual space using a marker provided in a real space. The mixed reality device displays a virtual object at a preset position in a three-dimensional coordinate system based on the base point. The mixed reality device acquires a movement amount of the marker relative to a reference position of the marker when the base point was set. When the movement amount exceeds a preset threshold, the mixed reality device moves the base point according to the movement amount. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. [Diagram 3] FIG. 3 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Diagram 5] FIG. 5 is a schematic diagram illustrating the state of work. [Figure 6] FIG. 6 is a schematic diagram showing an example of a tool. [Figure 7] FIG. 7 is a schematic diagram illustrating the state of work. [Figure 8] 8(a) and 8(b) are schematic diagrams for explaining processing by the mixed reality device according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram for explaining a method of calculating the amount of movement. [Figure 10] FIG. 10 is a schematic diagram for explaining a method of calculating the amount of movement. [Figure 11] FIG. 11 is a flowchart illustrating a processing method according to an embodiment. [Figure 12] FIG. 12 is a flowchart showing a method for correcting the display position of a virtual object. [Figure 13] FIG. 13 is a flowchart showing a method for controlling the operation of the MR device. [Figure 14] FIG. 14 is a schematic diagram showing a display example by a mixed reality device according to a modified example. [Figure 15] FIG. 15 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram illustrating a mixed reality device according to an embodiment. The embodiment of the present invention relates to a mixed reality device (MR device). For example, as shown in Fig. 1, the MR device 100 according to the embodiment includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0009] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0010] The projection device 121 and the projection device 122 display the recognition result of the body of the worker, a virtual object, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0011] The lenses 111 and 112 are optically transparent. A person wearing the MR device 100 can visually recognize the state of real space through the lenses 111 and 112. The person wearing the MR device 100 can also visually recognize information projected onto the lenses 111 and 112 by the projection devices 121 and 122. Information (virtual space) is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0012] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0013] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0014] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0015] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0016] FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. A worker wearing the MR device 100 works on, for example, an article 200 shown in Fig. 2. The article 200 is a cylindrical member and has fastening points 201 to 208. The worker uses a wrench and an extension bar to turn the screws into the fastening points 201 to 208 in order.

[0017] When a task is performed using the MR device 100, a three-dimensional coordinate system in a virtual space is set with a specific object as a reference. In the example shown in FIG. 2, a marker 210 is provided near the task object. The marker 210 is an AR marker. As will be described later, the marker 210 is provided to set the base point of the three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210.

[0018] At the start of the work, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a base point in the virtual space based on the position and orientation of the marker 210. When the base point is set, a three-dimensional coordinate system of the virtual space is determined. By setting the base point based on a predetermined object that exists in the real space, it is possible to display a virtual object in correspondence with the object in the real space.

[0019] The processing device 150 further calculates the position of the MR device 100. As an example, the processing device 150 calculates the position and orientation of the MR device 100 by a function of spatial mapping. In the MR device 100, the depth camera 132 measures the distance between the MR device 100 and the surrounding objects. Surface information of the surrounding objects is obtained from the measurement results by the depth camera 132. The surface information includes the position and orientation of the object's surface. For example, the surface of each object is represented by a plurality of meshes, and the position and orientation are calculated for each mesh. The processing device 150 calculates the relative position and orientation of the MR device 100 with respect to the surfaces of the surrounding objects from the surface information. When the marker 210 is recognized, the position of each surface is also represented in a three-dimensional coordinate system with the marker 210 as the base point. The position and orientation of the MR device 100 in the three-dimensional coordinate system are calculated from the positional relationship between the surface of the object and the MR device 100. The spatial mapping is repeatedly performed at a predetermined interval.

[0020] FIG. 3 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. During work, the image camera 131 and the depth camera 132 capture images of the article 200, the worker's left hand 251, and the worker's right hand 252. The processing device 150 recognizes the left hand 251 and the right hand 252 from the captured images by hand tracking. The processing device 150 may cause the projection device 121 and the projection device 122 to display the recognition results on the lens 111 and the lens 112. Hereinafter, the processing device causing the projection device to display information on the lens is also simply referred to as "the processing device displays information."

[0021] 3, the processing device 150 displays the recognition results of the left hand 251 and the right hand 252 superimposed on the hands in real space. In the illustrated example, a plurality of virtual objects 261 and a plurality of virtual objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The plurality of virtual objects 261 indicate the plurality of joints of the left hand 251, respectively. The plurality of virtual objects 262 indicate the plurality of joints of the right hand 252, respectively. Instead of the joints, virtual objects (meshes) indicating the surface shapes of the left hand 251 and the right hand 252, respectively, may be displayed. The meshes are obtained by spatial mapping.

[0022] When the processing device 150 recognizes the left hand 251 and the right hand 252, it measures the position of each hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may be used as the position of the hand. Or, the position of the center of the entire hand may be used as the position of the hand.

[0023] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. The processing device 150 displays virtual objects 301-308 and virtual objects 311-318 as shown in Fig. 4. The virtual objects 311-318 are displayed at positions away from the fastening points 201-205, respectively. The virtual objects 311-318 are displayed between the fastening points 201-205 and the virtual objects 301-308, respectively. The virtual objects 311-318 indicate to which fastening points the virtual objects 301-308 correspond, respectively.

[0024] In the illustrated example, the virtual objects 301-308 are spherical, and the virtual objects 311-318 are rod-shaped. As long as the worker can visually recognize each virtual object, the shape of each virtual object is not limited to this example. For example, the virtual objects 301-308 may be cubes, and the virtual objects 311-318 may be linear. Also, virtual objects similar to the virtual objects 301-308 and the virtual objects 311-318 are displayed for the fastening points 206-208 not shown in FIG. 4.

[0025] FIG. 5 is a schematic diagram illustrating the state of work. For example, as shown in Fig. 5, a wrench 280 and an extension bar 290 are used to turn a screw into a fastening point. As an example, when fastening a screw into fastening point 203, an operator places a screw into a threaded hole of fastening point 203. The operator fits one end of extension bar 290 onto the screw. The operator fits the head of wrench 280 onto the other end of extension bar 290. The operator holds the head of wrench 280 with one hand and holds the grip of wrench 280 with the other hand. By turning wrench 280, the screw is tightened into fastening point 203 via extension bar 290.

[0026] At this time, the worker positions the extension bar 290 so that the extension bar 290 is close to or in contact with the virtual object 313. The worker also grips the head of the wrench 280 so that his or her hand comes into contact with the virtual object 303. By displaying the virtual object, the worker can easily grasp the positions of the tool and hand when turning the screw into the fastening point 203. This can improve the work efficiency.

[0027] After the virtual object is displayed, the processing device 150 may determine whether a predetermined object has come into contact with the virtual object. For example, the processing device 150 determines whether a hand has come into contact with the virtual object 303. Specifically, the processing device 150 calculates the distance between the position of the hand and the virtual object 303. If either distance is less than a preset threshold, the processing device 150 determines that the hand has come into contact with the virtual object. As an example, in FIG. 5, the diameter of the virtual object 303 (sphere) corresponds to the threshold. The sphere indicates the range within which it is determined that the hand has come into contact with the virtual object.

[0028] FIG. 6 is a schematic diagram showing an example of a tool. The processing device 150 may determine whether the tool has come into contact with the virtual object 303. For example, as shown in FIG. 6, a plurality of markers 281 are attached to the wrench 280. The processing device 150 recognizes the plurality of markers 281 from an image captured by the image camera 131. The processing device 150 measures the position of each marker 281. A positional relationship between the plurality of markers 281 and the head 282 of the wrench 280 is registered in advance. The processing device 150 calculates the position of the head 282 based on the positions of at least three recognized markers 281 and the positional relationship registered in advance. The processing device 150 calculates the distance between the position of the head 282 and the virtual object 303. If any of the distances is less than a preset threshold, the processing device 150 determines that the wrench 280 has come into contact with the virtual object.

[0029] If the distance is less than the threshold, it can be estimated that a screw is being turned at a fastening point corresponding to one of the virtual objects 303. In the example shown in Fig. 5, it is estimated that an operation is being performed at the fastening point 203 corresponding to the virtual object 303 due to contact between the hand and the virtual object 303.

[0030] Fig. 7 is a schematic diagram illustrating an example of a work situation. Fig. 8(a) and Fig. 8(b) are schematic diagrams for explaining a process by the mixed reality device according to the embodiment. 7, for example, the article 200 is placed on a dolly 221. Casters 222 are attached to the bottom surface of the dolly 221. The dolly 221 is movable relative to the floor surface. The marker 210 is attached to the dolly 221, for example. The article 200 is placed so as not to move relative to the dolly 221 during work. Therefore, the positional relationship between the article 200 and the marker 210 is substantially fixed.

[0031] The virtual object is displayed at a preset position in a three-dimensional coordinate system based on the marker 210. In the example shown in Fig. 5, the position of the fastening point 203, the length of the extension bar 290, etc. are registered in advance. The virtual object 313 is displayed between the position of the fastening point 203 and a position that is separated from the fastening point 203 by the length of the extension bar 290. The virtual object 303 is displayed at a position that is separated from the fastening point 203 by the length of the extension bar 290. As a result, the virtual objects 303 and 313 are displayed corresponding to the fastening point 203.

[0032] The worker can move the article 200 so that it is easier to work. For example, the worker rotates the article 200 while working so that the fastening point where the screw is to be tightened is located closer to the worker. At this time, if the article 200 moves without updating the position of the base point, the display position of the virtual object shifts from the position where it should be displayed. As a result, as shown in FIG. 8(a), the virtual objects 303 and 313 are displayed shifted from the fastening point 203.

[0033] The processing device 150 acquires the amount of movement of the article 200 while the virtual object is being displayed so that the virtual object is appropriately displayed. Here, "movement" includes translational movement and rotational movement. "Amount of movement" includes the amount of translational movement and the amount of rotational movement. When the amount of movement exceeds a preset threshold, the processing device 150 moves the base point according to the amount of movement. As a result, the display positions of the virtual objects 303 and 313 change, as shown in FIG. 8(b). The virtual objects 303 and 313 are displayed in more appropriate positions.

[0034] 9 and 10 are schematic diagrams for explaining a method of calculating the amount of movement. The amount of movement is calculated using one of the following first to fourth methods.

[0035] In the first method, as shown in FIG. 9, the photographing device 1 is installed at a location where the article 200 is provided. The photographing device 1 photographs the article 200. A marker 215 is attached to the article 200 at a position visible from the photographing device 1. The marker 215 is different from the marker 210. As long as the position of the marker 215 is fixed with respect to the article 200, the marker 215 may be attached to an object other than the article 200. In order to improve the recognition accuracy of the marker 215, it is preferable that the marker 215 has a color that is easily distinguishable from other objects. Also, in order to improve the calculation accuracy of the movement amount, it is preferable that a plurality of markers 215 are attached.

[0036] The photographing device 1 photographs the marker 215 and acquires the image. The processing device 2 recognizes the marker 215 from the image and calculates the position of the marker 215 in the image. The processing device 2 holds the position of the marker 215 at the time when the three-dimensional coordinate system is set by the processing device 150 as a reference position. When an image is acquired, the processing device 2 calculates the difference between the reference position and the recognized position of the marker 215 as the amount of movement.

[0037] In the second method, the photographing device 1 photographs the article 200 in the same manner as in the first method. The marker 215 is not required. The processing device 2 recognizes the article 200 from the image and calculates the position of the article 200 in the image. A pre-trained model is used to recognize the article 200. In order to improve the accuracy of the recognition, it is preferable that the model includes a convolutional neural network (CNN). The processing device 2 holds the position of the article 200 at the time when the three-dimensional coordinate system is set by the processing device 150 as a reference position. When an image is acquired, the processing device 2 calculates the difference between the reference position and the recognized position of the article 200 as the amount of movement.

[0038] In the third method, a sensor 223 is used to obtain the amount of movement. The sensor 223 is attached to the article 200 or the dolly 221. As long as the position of the sensor 223 is fixed with respect to the article 200, the sensor 223 may be attached to an object other than the article 200 or the dolly 221. For example, the sensor 223 includes one or more selected from an acceleration sensor or an angular velocity sensor. The processing device 2 calculates the amount of movement of the article 200 from the acceleration or angular velocity detected by the sensor 223. Alternatively, the sensor 223 may include a radio wave receiver, and an indoor positioning system may be used to obtain the amount of movement of the article 200. For indoor positioning, a beacon, a Radio Frequency Identification (RFID), an Indoor MEssaging System (IMES), an Ultra-Wide Band (UWB) wireless communication, or the like is used.

[0039] In the fourth method, the spatial mapping function of the MR device 100 is used. In the spatial mapping, surface information of the surrounding objects is calculated from the depth image acquired by the depth camera 132. The change in the position of the surface is calculated by repeating the calculation of the surface information. In addition, the movement amount of the MR device 100 is calculated from the detection result of the sensor 140. The difference between the change in the position of the surface of the article 200 and the movement amount of the MR device 100 corresponds to the actual movement amount of the article 200.

[0040] When the amount of movement is calculated by any one of the first to fourth methods, the processing device 150 acquires the amount of movement. The processing device 150 compares the amount of movement with a threshold value. The threshold value is set in advance depending on the size of the virtual object, the allowable amount of deviation of the display position, and the like. The amount of movement is repeatedly calculated at a predetermined interval. It is preferable that the calculation interval be short in order to shorten the period during which the virtual object is misaligned with respect to the article 200 when the article 200 moves.

[0041] FIG. 11 is a flowchart illustrating a processing method according to an embodiment. 11 is executed, task master data 170a, base point master data 170b, tool master data 170c, and fastening point master data 170d are prepared in advance. Each master data is stored in the storage device 170.

[0042] First, a task to be performed is selected (step S1). A task ID, a task name, an item ID, and an item name are registered in the task master data 170a. A task is specified by a task ID, a task name, an ID of an item to be worked on, or an item name. The processing device 150 accepts the selection of a task. For example, the task to be performed is selected by a worker. The task to be performed may be selected by a higher-level system, and the processing device 150 may accept the selection. The processing device 150 may determine the task to be performed based on data obtained from the image camera 131 or another sensor. The processing device 150 selects a task based on the result of the determination.

[0043] Next, the image camera 131 captures the marker 210. The processing device 150 sets a base point of a three-dimensional coordinate system based on the position and orientation of the marker 210 (step S2). At this time, the processing device 150 refers to the base point master data 170b. The base point master data 170b stores a method for setting a base point for each task. The processing device 150 acquires the method for setting a base point for the selected task, and sets the base point according to the setting method.

[0044] After setting the base point, the processing device 150 displays a virtual object (step S3). When the virtual object is displayed, the tool master data 170c and the fastening point master data 170d are appropriately referred to.

[0045] The tool master data 170c stores the ID, tool type, tool length, socket type, socket length, etc. of the tool used for each task. The tool type indicates the classification of the tool based on the structure, external shape, or performance. The tool length is the length from the center of rotation to the grip when the tool is used for screw tightening. The socket type indicates the classification of the socket based on the structure or external shape. The socket length indicates the length of the socket in the direction connecting the tool and the screw when the screw is tightened. The processing device 150 acquires data of the tool used in the task selected in step S1 from the tool master data 170c. When an extension bar is used, the type, length, etc. of the extension bar are also stored in the tool master data 170c. The processing device 150 further acquires data related to the extension bar from the tool master data 170c.

[0046] The fastening point master data 170d stores, for each fastening point, the ID of the fastening point, the position of the fastening point, the required torque value, and the number of times the screw is tightened. The fastening position indicates the position where the fastening point is located, and is specified by the position of the three-dimensional coordinate system set in step S2. The number of times the screw is tightened indicates the number of times the screw is tightened that is required for each fastening point. If a mark is applied to the screw after tightening, the color of the mark is also registered.

[0047] The processing device 150 recognizes a predetermined object from the image captured by the image camera 131 (step S4). The "predetermined object" is an object that is to be determined to be in contact with a virtual object. The processing device 150 calculates the position of the predetermined object, and determines whether the distance between the object and the virtual object is less than a threshold (step S5). For example, if the distance is equal to the threshold, the predetermined object comes into contact with the virtual object.

[0048] If the distance is equal to or greater than the threshold, step S4 is executed again. If the distance is less than the threshold, the processing device 150 presumes that an operation is being performed on the location corresponding to the virtual object.

[0049] The processing device 150 records the work related to the estimated location and saves it in the history data 170e (step S6). For example, the torque value detected by the tool is linked to the ID of the work and the ID of the estimated location. As shown in the figure, the processing device 150 may further link the type and ID of the tool used, the number of times the screw was tightened, and the mark recognition result to the ID of the fastening location. The mark is recognized by the processing device 150 from an image taken by the image camera 131. The processing device 150 extracts a block of pixels of the mark color from the image and counts the number of pixels in that block. If the number of pixels exceeds a preset threshold, it is determined that a mark has been attached.

[0050] The processor 150 determines whether the work is complete (step S7). If the work is not complete, step S3 is executed again.

[0051] FIG. 12 is a flowchart showing a method for correcting the display position of a virtual object. The correction method M1 shown in Fig. 12 is executed after step S3 shown in Fig. 11. The correction method M1 is repeated until step S6 is executed.

[0052] First, the processing device 150 acquires the amount of movement of the target on which the virtual object is to be displayed (step S11). The amount of movement is calculated by one or more of the first to fourth methods described above. The processing device 150 determines whether the amount of movement exceeds a threshold (step S12). If the amount of movement exceeds the threshold, the processing device 150 moves the base point according to the amount of movement (step S13). For example, the processing device 150 translates the base point by the same amount as the amount of translational movement of the article 200, and rotates the base point by the same amount as the amount of rotational movement of the article 200. The processing device 150 redisplays the virtual object based on the moved base point (step S14). If the amount of movement does not exceed the threshold in step S12, or after step S14, the processing device 150 determines whether step S6 shown in FIG. 11 has been executed (step S15). If step S6 has not been executed, step S11 is executed again.

[0053] It should be noted that, if the marker 210 is photographed after the base point has been moved, the base point is reset using the marker 210. The display position of the virtual object is set based on the base point using the marker 210. Therefore, by resetting the base point using the marker 210, the virtual object can be displayed at the most desirable position.

[0054] Advantages of the embodiment will be described. MR devices are effective for improving work efficiency. For example, as shown in FIG. 5, by displaying a virtual object, a worker can easily understand which fastening point to turn a screw into. Also, when turning a screw into a fastening point, the worker can intuitively understand the positions of the tool and hand to be placed. This can improve work efficiency. Furthermore, it is also possible to create a work record based on the proximity or contact between a specified object and a virtual object. By automatically creating a work record, there is no need for the worker to create the work record. It is possible to prevent an erroneous work record from being created due to a worker's mistake.

[0055] These advantages are obtained by displaying the virtual object at a specific position corresponding to the item in the real space. The display position of the virtual object in the three-dimensional coordinate system is set in advance. The origin of this three-dimensional coordinate system is set using a marker provided in the real space. Therefore, when the positional relationship between the marker recognized by the MR device 100 and the item in the real space changes, the display position of the virtual object shifts with respect to the item in the real space. As a result, the worker may become confused or a work record may be created for a place other than the place where the work was actually performed.

[0056] As an example, a worker rotates the article 200 and the marker 210 from the state shown in Fig. 7, and the marker 210 after the rotation is not photographed (not recognized). In this case, the positional relationship between the recognized marker 210 and the article 200 changes due to the rotation. As a result, as shown in Fig. 8(a), the virtual objects 303 and 313 are displayed shifted from the fastening point 203.

[0057] Regarding this problem, in the embodiment of the present invention, the processing device 150 moves the base point according to the movement amount. By moving the base point, it is possible to prevent the virtual object from being displayed at a position displaced from the desired position. As a result, as shown in FIG. 8(b), even if the article 200 moves, the display positions of the virtual objects 303 and 313 can be moved according to the movement of the article 200. By displaying the virtual object at an appropriate position, it becomes easier for the worker to work. In addition, it becomes possible to create a work record more appropriately. According to the embodiment of the present invention, it is possible to improve the convenience of the MR device 100.

[0058] When calculating the amount of movement, two or more of the first to fourth methods may be used. For example, the first method and either the third or fourth method are executed. The first method requires a small amount of calculation and has good accuracy. For this reason, the amount of movement calculated by the first method is prioritized over the amount of movement calculated by other methods. On the other hand, if the marker 215 is hidden by another object, the amount of calculation cannot be obtained by the first method. In this case, the amount of movement calculated by the third or fourth method is used.

[0059] In addition, in order to suppress misalignment between the virtual object and the article, there is also a method of recognizing the article and its details (fastening points, etc.) from an image. In this case, the virtual object can be displayed according to the recognized details. Therefore, even if the worker moves the article, the article after the movement is recognized and the virtual object is displayed. However, a large amount of calculation is required to accurately recognize the article and its details from an image. It takes a longer time from when the worker moves the article until the virtual object is displayed at the desired position.

[0060] In the embodiment, a marker is used to set the base point, and a virtual object is displayed at a predetermined position. This method can reduce the amount of calculation required. In addition, the amount of calculation required can be further reduced by calculating the amount of movement using an image of the marker 215 or the detection result of the sensor 223. According to the embodiment of the present invention, the amount of calculation can be reduced, and the display position of the virtual object can be updated at shorter intervals. For example, the display position of the virtual object can be updated in real time according to the movement of an article.

[0061] (Modification) Fig. 13 is a flowchart showing a method for controlling the operation of the MR device Fig. 14 is a schematic diagram showing a display example by a mixed reality device according to a modified example. Instead of the correction method M1 shown in Fig. 12, the control method M2 shown in Fig. 13 may be executed. In the control method M2, in steps S21 and S22, the movement amount is acquired and compared with a threshold value, similarly to steps S11 and S12 of the correction method M1. If the movement amount exceeds the threshold value, the processing device 150 stops the processing of the processing method M shown in Fig. 11 (step S23). As a result, contact between the predetermined object and the virtual object is no longer determined. The processing device 150 may stop displaying the virtual object. As a result, the creation of the work record is also stopped.

[0062] The processing device 150 outputs an alert (step S24). For example, as shown in FIG. 14, a message 320 is displayed to the worker to re-determine the display position of the virtual object. When the message 320 is displayed, the worker faces the marker 210. The image camera 131 captures the image of the marker 210, and the base point is reset based on the marker 210. In addition to the message 320, sound, vibration, light, or the like may be output as an alert.

[0063] The processing device 150 determines whether the base point has been reset (step S25). When the base point has been reset, the processing device 150 redisplays the virtual object based on the reset base point (step S26). The processing device 150 resumes the processing method M (step S27). This allows contact between a predetermined object and the virtual object to be determined. As a result, a work record is also automatically created.

[0064] If the movement amount does not exceed the threshold in step S22, or after step S27, the processing device 150 determines whether step S6 shown in Fig. 11 has been executed (step S28). If step S6 has not been executed, step S21 is executed again.

[0065] According to the modified embodiment, when the amount of movement exceeds the threshold, the processing of the processing method M shown in FIG. 11 is stopped. As a result, contact between the predetermined object and the virtual object is not determined, and the estimation of the fastening point being worked on is also stopped. The creation of the work record is also stopped. As a result, it is possible to prevent a different fastening point from being erroneously estimated as being worked on.

[0066] According to the modified example, it is possible to prevent an erroneous work record from being created due to the movement of the article 200. This makes it possible to improve the convenience of the MR device 100.

[0067] When the processing is stopped, it is preferable that the processing device 150 outputs an alert to prompt the worker to reset the base point. The sooner the base point is reset, the sooner the work can be resumed. This makes it possible to suppress a decrease in work efficiency.

[0068] Here, an example has been described in which the embodiment of the present invention is applied mainly to the task of tightening a screw. The embodiment of the present invention may also be applied to the task of loosening a screw. When loosening a screw, a tool is used to turn the screw, as shown in Fig. 5. In this case as well, the task can be performed efficiently by displaying a virtual object.

[0069] The embodiment of the present invention may be applied to tasks other than turning a screw. For example, the task may be assembling an item, dismantling an item, or transporting an item. In any task, the task can be performed efficiently by displaying a virtual object. In addition, when the item being worked on moves, the display position of the virtual object is changed according to the amount of movement, or estimation of the location being worked on is stopped, thereby improving the convenience of the MR device.

[0070] FIG. 15 is a schematic diagram showing a hardware configuration. 15 is used as each of the processing devices 2 and 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0071] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0072] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0073] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) or the like.

[0074] The input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95.

[0075] An output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to the output device via the output I / F 96 and cause the output device to output information.

[0076] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0077] Data processing by each of the processing device 2 and the processing device 150 may be executed by only one computer 90. Part of the data processing may be executed by a server or the like via the communication I / F 97. Processing by the processing device 2 may be executed by the processing device 150. Part of the processing executed by the processing device 150 in the above example may be executed by the processing device 2.

[0078] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0079] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0080] Embodiments of the invention include the following features. (Feature 1) A base point in the virtual space is set using a marker provided in the real space; A mixed reality device that displays a virtual object at a preset position in a three-dimensional coordinate system based on the base point, acquiring a movement amount of the marker relative to a reference position of the marker when the base point is set; A mixed reality device that moves the base point in accordance with the amount of movement when the amount of movement exceeds a preset threshold. (Feature 2) Recognizing the marker from an image in which the marker is captured; The mixed reality device according to feature 1, wherein the base point is set based on the position and orientation of the marker. (Feature 3) The mixed reality device of feature 2, wherein when the marker is re-photographed after the base point is moved, the base point is reset based on the position and orientation of the marker recognized from the re-photographed image. (Feature 4) Repeatedly acquiring the amount of movement; The mixed reality device according to any one of features 1 to 3, wherein each time the amount of movement is acquired, it is determined whether the amount of movement exceeds the threshold value. (Feature 5) The mixed reality device according to any one of features 1 to 4, wherein the amount of movement is calculated using an image of an item to be worked on or a detection result from a sensor fixed to the item. (Feature 6) 6. The mixed reality device according to any one of features 1 to 5, wherein the virtual object is displayed corresponding to a fastening location of an article to be worked on. (Feature 7) Calculate the distance between a predetermined object and the virtual object; The mixed reality device of feature 6, wherein if the distance is less than a preset threshold, it is estimated that an operation is being performed on the fastening point. (Feature 8) A base point in the virtual space is set using a marker provided in the real space; A mixed reality device that displays a virtual object corresponding to a fastening point of an article, When a distance between a predetermined object and the virtual object is less than a preset threshold, it is estimated that an operation is being performed on the fastening portion; A mixed reality device that stops estimating the fastening point being worked on when the amount of movement of the marker relative to the reference position of the marker when the base point was set exceeds a preset threshold. (Feature 9) The mixed reality device of feature 8, further comprising: a device that resumes estimation of the fastening point being worked on if the base point is reset after estimation has stopped. (Feature 10) Repeatedly acquiring the amount of movement; The mixed reality device of feature 8 or 9, wherein each time the amount of movement is acquired, it is determined whether the amount of movement exceeds the threshold value. (Feature 11) Mixed reality devices A base point in the virtual space is set using a marker provided in the real space; Displaying a virtual object at a preset position in a three-dimensional coordinate system based on the base point; acquiring a movement amount of the marker relative to a reference position of the marker when the base point is set; When the amount of movement exceeds a preset threshold, the base point is moved according to the amount of movement. Processing methods. (Feature 12) A program for causing a mixed reality device to execute the processing method according to feature 11. (Feature 13) A storage medium storing the program according to feature 12.

[0081] According to the embodiments described above, a mixed reality device having better usability is provided. Also, a processing method, a program, and a storage medium capable of further improving the usability of the mixed reality device are provided.

[0082] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0083] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0084] 1: imaging device, 2: processing device, 90: computer, 91: CPU, 92: ROM, 93: RAM, 94: storage device, 95: input interface, 96: output interface, 97: communication interface, 98: system bus, 100: MR device, 101: frame, 111, 112: lens, 121, 122: projection device, 131: image camera, 132: depth camera, 140: sensor, 141: microphone, 150: processing device, 160: battery, 170: storage device, 170a: work master data, 170b: base point master data, 170c: tool master data, 170d: fastening point master data, 170e: history data, 200: item, 201-208: fastening point, 210, 215: marker, 221: Cart, 222: Caster, 223: Sensor, 251: Left hand, 252: Right hand, 261, 262: Virtual object, 280: Wrench, 281: Marker, 282: Head, 290: Extension bar, 301-308, 311-318: Virtual object, 320: Message, M: Processing method, M1: Correction method, M2: Control method

Claims

1. A base point in the virtual space is set using a marker provided in the real space; A mixed reality device that displays a virtual object at a preset position in a three-dimensional coordinate system based on the base point, acquiring a movement amount of the marker relative to a reference position of the marker when the base point is set; A mixed reality device that moves the base point in accordance with the amount of movement when the amount of movement exceeds a preset threshold.

2. Recognizing the marker from an image in which the marker is captured; The mixed reality device according to claim 1 , wherein the base point is set based on the position and orientation of the marker.

3. The mixed reality device according to claim 2 , wherein when the marker is photographed again after the base point is moved, the base point is reset based on the position and orientation of the marker recognized from the re-photographed image.

4. Repeatedly acquiring the amount of movement; The mixed reality device according to claim 1 , further comprising: a determination unit configured to determine whether the amount of movement exceeds the threshold each time the amount of movement is acquired.

5. The mixed reality device of claim 1 , wherein the amount of movement is calculated using an image of an item being worked on or detection results from a sensor fixed relative to the item.

6. The mixed reality device according to claim 1 , wherein the virtual object is displayed corresponding to a fastening point of an article to be worked on.

7. Calculate the distance between a predetermined object and the virtual object; The mixed reality device of claim 6 , further comprising: a device configured to estimate that an operation is being performed on the fastening point if the distance is less than a preset threshold value.

8. A base point in the virtual space is set using a marker provided in the real space; A mixed reality device that displays a virtual object corresponding to a fastening point of an article, When a distance between a predetermined object and the virtual object is less than a preset threshold, it is estimated that an operation is being performed on the fastening portion; A mixed reality device that stops estimating the fastening point being worked on when the amount of movement of the marker relative to the reference position of the marker when the base point was set exceeds a preset threshold.

9. The mixed reality device of claim 8 , further comprising: a first base point for receiving a first input from a second input of the first input;

10. Repeatedly acquiring the amount of movement; The mixed reality device according to claim 8 or 9, further comprising: a determination as to whether the amount of movement exceeds the threshold each time the amount of movement is acquired.

11. Mixed reality devices A base point in the virtual space is set using a marker provided in the real space; Displaying a virtual object at a preset position in a three-dimensional coordinate system based on the base point; acquiring a movement amount of the marker relative to a reference position of the marker when the base point is set; When the amount of movement exceeds a preset threshold, the base point is moved according to the amount of movement. Processing methods.

12. A program for causing a mixed reality device to execute the processing method according to claim 11.

13. A storage medium storing the program according to claim 12.

Images