Mixed reality device, processing device, processing method, program and storage medium

Abstract

To provide a mixed reality device, a processing device, a processing method, a program, and a storage medium configured to allow users to more easily grasp positional relationship between an object in the real space and a virtual object in a virtual space.SOLUTION: A mixed reality device superimposes and displays a virtual object in a virtual space, on a real space. The mixed reality device is configured to obtain a position of an object in the real space and a position of the virtual object. The mixed reality device is configured to change outer appearance of the virtual object when the virtual object is positioned on a far side of the object relative to the mixed reality device and overlaps the object.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] An embodiment of the present invention relates to a mixed reality device, a processing device, a processing method, a program, and a storage medium. [Background technology]

[0002] There are devices that display a virtual space overlaid on a real space. For these devices, there is a demand for technology that allows users to more easily grasp the positional relationship between an object in the real space and a virtual object in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that the present invention aims to solve is to provide a mixed reality device, a processing device, a processing method, a program, and a storage medium that enable a user of a mixed reality device to more easily grasp the positional relationship between an object in real space and a virtual object in virtual space. [Means for solving the problem]

[0005] A mixed reality device according to an embodiment displays a virtual object in a virtual space by overlaying it on a real space. The mixed reality device acquires a position of an object and a position of the virtual object in the real space. When the virtual object is located behind the object with respect to the mixed reality device and overlaps with the object, the mixed reality device changes an appearance of the virtual object. [Brief description of the drawings]

[0006] [Figure 1]FIG. 1 is a schematic diagram showing the configuration of a processing system according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 3] FIG. 3 is a schematic diagram illustrating an example of an object to be worked on. [Figure 4] FIG. 4 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Diagram 5] FIG. 5 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 8] FIG. 8 is a schematic plan view for explaining the embodiment. [Figure 9] FIG. 9 is an example of a depth image. [Figure 10] FIG. 10 is a schematic plan view for explaining the embodiment. [Figure 11] FIG. 11 is a schematic plan view for explaining the embodiment. [Figure 12] FIG. 12 is a schematic plan view for explaining the embodiment. [Figure 13] FIG. 13 is a schematic plan view for explaining the embodiment. [Figure 14] FIG. 14 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 15] 15(a) and 15(b) are schematic diagrams for explaining a display example by the processing system according to the embodiment. [Figure 16] 16(a) and 16(b) are schematic diagrams for explaining a display example by the processing system according to the embodiment. [Figure 17] FIG. 17 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 18] 18(a) and 18(b) are schematic diagrams for explaining a display example by the processing system according to the embodiment. [Figure 19] FIG. 19 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 21] 21(a) and 21(b) are schematic diagrams for explaining a display example by the processing system according to the embodiment. [Figure 22] FIG. 22 is a schematic diagram illustrating the state of work. [Diagram 23] FIG. 23 is a schematic diagram for explaining a display example by the processing system according to the embodiment. [Figure 24] 24(a) and 24(b) are schematic diagrams for explaining a display example by the processing system according to the embodiment. [Diagram 25] FIG. 25 is a flowchart illustrating an example of a processing method according to an embodiment. [Figure 26] FIG. 26 is a schematic diagram illustrating data to be referred to in the embodiment. [Figure 27] FIG. 27 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a processing system according to the first embodiment. The invention according to the first embodiment is applicable to a task of turning a screw using a tool. As shown in FIG. 1, a processing system 10 includes an imaging device 1, a processing device 2, an input device 3, a display device 4, and a storage device 5.

[0009] The imaging device 1 captures images of the work being performed and acquires images. In the work, a fastener such as a screw is fastened to an article using a tool. Alternatively, a screw fastened to the article is loosened using a tool. The article is a part, a unit, or a semi-finished product for producing a product. The tool is a wrench, a screwdriver, or the like. Here, an example in which the embodiment of the present invention is applied to a fastening work of tightening a screw will be mainly described.

[0010] When assembling an item, a worker holds a tool in his / her hand and tightens a screw. The image capturing device 1 captures the left or right hand of the worker holding the tool, the item being worked on, and the like. For example, the image capturing device 1 includes a camera that captures a color image and a depth image. The color image is an image based on visible light, expressed in RGB or HSV. The depth image is an image based on reflected infrared light, and indicates the distance between the image capturing device 1 and the subject. The image capturing device 1 that captures the color image and the image capturing device 1 that captures the depth image may be provided separately. A distance measuring sensor such as a laser range finder (LRF) may be used to capture the depth image.

[0011] The processing device 2 receives successive images (video) captured by the photographing device 1. The processing device 2 recognizes the left or right hand in the image. Hand tracking is used to recognize the left or right hand. Hereinafter, when there is no particular distinction between the left hand and the right hand, the term "hand" will simply refer to at least either the left hand or the right hand.

[0012] The input device 3 is used by the worker to input information to the processing device 2. The input device 3 includes a microphone. The worker can input information to the processing device 2 by speaking into the input device 3. For example, a voice corresponding to a voice command is input to the input device 3. In addition to the input device 3, the worker can input information to the processing device 2 by hand gestures or the like.

[0013] The display device 4 displays information for the worker. The processing device 2 causes the display device 4 to display information for supporting the fastening work. Hereinafter, the processing device causing the display device to display information is also simply referred to as "the processing device displays information." For example, the processing device 2 displays a virtual object indicating the position where the worker's hand should be placed, the hand recognition result, etc. during the fastening work.

[0014] The storage device 5 stores data necessary for processing by the processing device 2, data obtained by the processing by the processing device 2, etc. For example, data on tools used in work, data necessary for calculating the position of the tools described later, etc. are registered in the storage device 5.

[0015] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. The processing system 10 shown in Fig. 1 is realized as, for example, a mixed reality (MR) device. The MR device 100 shown in Fig. 2 includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0016] The image camera 131 and the depth camera 132 are an example of an imaging device 1. The processing device 150 is an example of a processing device 2. The microphone 141 is an example of an input device 3. The projection device 121 and the projection device 122 are an example of a display device 4. The storage device 170 is an example of a storage device 5.

[0017] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0018] The projection device 121 and the projection device 122 display the recognition result of the body of the worker, a virtual object, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0019] The lenses 111 and 112 are optically transparent. The worker can visually recognize the real state through the lenses 111 and 112. The worker can also visually recognize information projected onto the lenses 111 and 112 by the projection devices 121 and 122. The information is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0020] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. More specifically, the depth camera 132 measures the time of flight (ToF) of light from irradiating infrared light to receiving the reflected light. The longer the distance between the depth camera 132 and the object that reflects the infrared light, the longer the ToF. Based on the ToF, a depth image in the real space in front of the MR device 100 is generated. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0021] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0022] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0023] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0024] FIG. 3 is a schematic diagram illustrating an example of an object to be worked on. For example, a fastening operation is performed on an article 200 shown in Fig. 3. The article 200 is a cylindrical hollow member, and has fastening points 211 to 214. A worker uses a wrench to fasten screws into each of the fastening points 211 to 214.

[0025] A marker 220 is provided near the work target. In the illustrated example, the marker 220 is an AR marker. As described below, the marker 220 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 220. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 150 sets a three-dimensional coordinate system based on a plurality of points indicated by a hand gesture.

[0026] 4 to 7 are schematic diagrams for explaining display examples by the processing system according to the embodiment. When the fastening operation is started, the image camera 131 and the depth camera 132 capture an image of the marker 220. The processing device 150 recognizes the marker 220 from the captured image. The processing device 150 sets a three-dimensional coordinate system based on the position and orientation of the marker 220.

[0027] The image camera 131 and the depth camera 132 capture images of the article 200, the worker's left hand, and the worker's right hand. The processing device 150 recognizes the left hand and the right hand from the captured images. When the processing device 150 recognizes the left hand 251 and the right hand 252, it measures the position of each hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may be used as the position of the hand. Or, the position of the center of the entire hand may be used as the position of the hand.

[0028] The processing device 150 causes the projection device 121 and the projection device 122 to display the recognition result on the lens 111 and the lens 112. Hereinafter, the processing device 150 causing the lenses 111 and 112 to display information using the projection devices 121 and 122, respectively, will also be simply referred to as "the processing device displays information."

[0029] 4, for example, the processing device 150 displays the recognition results of the left hand 251 and the right hand 252 superimposed on the hands in real space. In the illustrated example, a plurality of virtual objects 261 and a plurality of virtual objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The plurality of virtual objects 261 respectively indicate a plurality of joints of the left hand 251. The plurality of virtual objects 262 respectively indicate a plurality of joints of the right hand 252. Instead of the joints, virtual objects (meshes) respectively indicating the surface shapes of the left hand 251 and the right hand 252 may be displayed.

[0030] Furthermore, the processing device 150 displays virtual objects 311-314 corresponding to the fastening points 211-214 as shown in Fig. 5. The virtual objects 311-314 are adjacent to the fastening points 211-214, respectively. The virtual objects 311-314 indicate positions where the worker's hands should be placed when fastening the fastening points 211-214, respectively. The positions where the virtual objects 311-314 are displayed are registered in advance using a three-dimensional coordinate system based on the base point of the marker 220. In the illustrated example, the virtual objects 311-314 are spheres. The shape, color, etc. of each virtual object are arbitrary as long as the worker can visually distinguish the virtual object from the real space.

[0031] As an example, a worker fastens a screw into fastening point 211. In this case, the worker places screw 225 in a screw hole of fastening point 211 as shown in FIG. 6. Next, the worker holds the grip of wrench 400 with his right hand. The worker fits the tip (head) of wrench 400 to which a socket is attached into screw 225. After that, the worker rotates wrench 400 with right hand 252 while holding the head of wrench 400 with left hand 251 as shown in FIG. 7. This causes screw 225 to be fastened.

[0032] In the series of operations described above, the processing device 150 may determine whether the left hand 251 or the right hand 252 has come into contact with any of the virtual objects. Specifically, the processing device 150 calculates the distance between the position of the hand and the virtual object. If either distance is less than a preset threshold, the processing device 150 determines that the hand has come into contact with that virtual object. As an example, in FIG. 7, the diameter of the virtual object 311 (sphere) corresponds to the threshold. The sphere indicates the range within which it is determined that the hand has come into contact with the virtual object.

[0033] In the above example of work, when tightening a screw, the left hand 251 holding the tip of the wrench comes into contact with the virtual object 311. When the hand comes into contact with a virtual object, it can be estimated that the screw is being turned into the fastening point corresponding to that virtual object. Here, the fastening point corresponding to the virtual object contacted by the hand and estimated to be the point where the screw is being turned is referred to as the "estimated point."

[0034] When the processing device 150 estimates that a screw has been turned into a specific fastening point, the processing device 150 creates a work record. For example, the processing device 150 associates data regarding the fastening point 211 with data indicating that a screw has been turned, and stores the data.

[0035] Contact between the tip of a tool and a virtual object may be determined instead of a hand. For example, a template image of the tool to be used is prepared in advance. The processing device 150 detects the position of the tool from the captured image using the template image. Alternatively, as shown in Figs. 6 and 7, a plurality of markers 420 may be provided on the tool. The processing device 150 detects the plurality of markers 420 from the captured image. The processing device 150 calculates the position of the tool based on the positions of the markers 420. The processing device 150 uses the obtained position to determine whether the tip of the tool is in contact with the virtual object.

[0036] When a digital tool is used for the work, the processing device 150 may receive a detection value from the digital tool. The digital tool includes a sensor that detects a signal generated during the work. When it is determined that a screw is fastened to an estimated location, the processing device 150 links the detection value to data (e.g., an ID) related to the fastening location 211. The processing device 150 stores the linked data in the storage device 170. This makes it possible to automatically create a work record for each fastening location.

[0037] When a digital torque wrench or digital torque driver is used as the tool, the detected value is torque. Alternatively, a threshold value may be set in advance for the torque. The digital torque wrench or digital torque driver may determine whether or not a torque exceeding the threshold value is detected, and output the determination result as the detected value.

[0038] As described above, by displaying the virtual object, the worker can easily understand which part he or she should work on. In the embodiment, the processing device 150 further detects an overlap between an object in the real space and a virtual object in the virtual space. The processing device 150 appropriately changes the appearance or position of the virtual object depending on the overlap.

[0039] First, the processing device 150 acquires the position of an object in real space and the position of a virtual object in virtual space. The position of the object in real space is included in a depth image acquired by the depth camera 132. The processing device 150 acquires the position of the object from the depth image. Alternatively, the position of the object in real space may be registered in advance. In this case, the position is represented using a three-dimensional coordinate system with the marker 220 as the base point.

[0040] Fig. 8 is a schematic plan view for explaining the embodiment, Fig. 9 is an example of a depth image, Figs. 10 to 13 are schematic plan views for explaining the embodiment. 8, the depth camera 132 acquires a depth image within the range of the field of view 132a. Specifically, an area 200a of the article 200 located on the side of the depth camera 132 is visible to the worker (depth camera 132). The distance between the depth camera 132 and each part of the area 200a is measured, and a depth image showing the shape and position of the surface of the area 200a is acquired.

[0041] 9 is a depth image showing the vicinity of the fastening point 211 of the article 200. For example, the depth camera 132 obtains mesh data indicating the position of each photographed object as the depth image. The mesh data is generated by dividing the photographed range into multiple sections (meshes) and assigning a value (position) to each mesh.

[0042] The processing device 150 acquires the position of each part of the region 200a from the mesh data. Specifically, the processing device 150 calculates the center point of each mesh. The position of the center point of each mesh is used as the position of each part of the region 200a. In addition, the position of the table on which the article 200 is placed, the position of the wall on the back side of the article 200, etc. may be further acquired by the depth camera 132.

[0043] The processing device 150 acquires the positions of the virtual objects 311 to 314. The processing device 150 calculates the distance between the center point of each mesh and the reference point, and the distance between each virtual object and the reference point. The position of the MR device 100 is used as the reference point. For example, the position of the image camera 131 or the position of the depth camera 132 is used as the reference point. When the viewpoint of the operator is measured in the MR device 100, the position of the viewpoint may be used as the reference point.

[0044] The processing device 150 determines whether or not there is a mesh longer than the distance between the virtual object and the reference point for each of the virtual objects 311 to 314. If there is a mesh longer than the distance between the virtual object and the reference point, the virtual object is determined to be located in front of the object in real space. If there is no mesh longer than the distance between the virtual object and the reference point, the virtual object is determined to be located behind the object in real space.

[0045] Next, the processing device 150 sets a virtual plane 350 as shown in Fig. 10. The virtual plane 350 is set in front of the article 200 and the virtual objects 311 to 314. That is, the virtual plane 350 is located between the reference point and the article 200, and between the reference point and each of the virtual objects 311 to 314. The processing device 150 projects the article 200 onto the virtual plane 350. Next, the processing device 150 projects, among the virtual objects 311 to 314, a virtual object that is determined to exist on the rear side of the article 200 onto the virtual plane 350. In the examples shown in Figs. 8 and 10, the virtual object 311 and the virtual object 314 are determined to be located on the rear side of the article 200.

[0046] 11, a plurality of regions 351 are set on a virtual plane 350. The plurality of regions 351 are arranged along a first direction D1 and a second direction D2. The first direction D1 intersects with a direction from a reference point toward the article 200 or any of the virtual objects 311 to 314. The second direction D2 is a direction intersecting with a horizontal plane. For example, the first direction D1 is parallel to the horizontal plane, and the second direction D2 is parallel to the vertical direction.

[0047] The processing device 150 generates a plurality of straight lines L connecting the reference point P0 and points on the outer edge of the region 200a, as shown in Fig. 12. The processing device 150 calculates a point P1 at which each straight line L intersects with the virtual surface 350. The range surrounded by the points P1 is the region 200a projected onto the virtual surface 350. The processing device 150 identifies a region 351 that overlaps with the article 200 among the plurality of regions 351, as shown in Fig. 13(a). The processing device 150 assigns a label La1 to the region 351 that overlaps with the article 200.

[0048] In a similar manner, the processing device 150 also projects the virtual objects 311 and 314 onto the virtual plane 350. As shown in Fig. 13(b) , the processing device 150 specifies the regions 351 that overlap with the virtual objects 311 and 314 among a plurality of regions 351. The processing device 150 assigns labels La2 and La3 to the regions 351 that overlap with the virtual objects 311 and 314, respectively.

[0049] At this time, the processing device 150 determines whether the area 351 to which the label La1 is assigned overlaps with the area 351 to which the label La2 or the label La3 is assigned. If an area 351 to which both the label La1 and the label La2 are assigned exists, or if an area 351 to which both the label La1 and the label La3 are assigned exists, it means that the article 200 overlaps with any of the virtual objects when viewed from the reference point.

[0050] In this case, the processing device 150 changes the appearance of the virtual object 311, and also changes the appearance of the virtual object 314. As a result, it becomes easier for the worker to understand that the virtual objects 311 and 314 are present on the rear side of the article 200.

[0051] Hereinafter, the above-described process of determining whether a virtual object in a virtual space overlaps with an object in a real space will be referred to as an “overlapping determination.” Also, a virtual object that is located behind an object in a real space and overlaps with the object as a result of the overlapping determination will be referred to as an “overlapping object.”

[0052] When performing the overlap determination, the processing device 150 may generate a straight line connecting the reference point and the virtual object, and calculate the distance between each mesh and the straight line. The processing device 150 extracts meshes whose distance is less than a preset threshold from all meshes. For example, meshes of other objects located behind the article 200 or meshes of walls are excluded. This allows only meshes that may overlap with the virtual object to be extracted. The processing device 150 performs the overlap determination using the extracted meshes. This allows the amount of calculation required for the overlap determination to be reduced.

[0053] 14, 15(a), 15(b), 16(a), 16(b), and 17 are schematic views for explaining display examples by the processing system according to the embodiment. 14 shows a display example in the case where overlap determination is not performed. In this case, all virtual objects are displayed in the same manner, regardless of whether the virtual objects overlap with objects in real space. Therefore, virtual object 311 and virtual object 314 are displayed in the same manner as virtual object 312 and virtual object 313. For example, the appearance (color, shape, etc.) of virtual object 311 and the appearance (color, shape, etc.) of virtual object 314 are the same as the appearance of virtual object 312 and the appearance of virtual object 313. The relative position of virtual object 311 with respect to fastening point 211 and the relative position of virtual object 314 with respect to fastening point 214 are the same as the relative position of virtual object 312 with respect to fastening point 212 and the relative position of virtual object 313 with respect to fastening point 213.

[0054] 15(a) and 15(b) show display examples when overlap determination is performed. As a result of overlap determination being performed, the appearance of virtual object 311 and virtual object 314 that overlap with article 200 changes, as shown in Fig. 15(a). In the illustrated example, virtual object 311 and virtual object 314 are displayed lighter than virtual object 312 and virtual object 313.

[0055] In addition to the examples shown in the drawings, the sizes of virtual objects 311 and 314 may be displayed smaller than the sizes of virtual objects 312 and 313. The colors of virtual objects 311 and 314 may be different from the colors of virtual objects 312 and 313. For example, virtual objects 311 and 314 may be displayed in relatively light colors, and virtual objects 312 and 313 may be displayed in relatively dark colors.

[0056] 15(b), the virtual object 311 and the virtual object 314 may be moved to positions where they do not overlap with the article 200. In this case, the distance between the virtual object 311 and the fastening point 211 becomes longer, and the distance between the virtual object 314 and the fastening point 214 becomes longer. In other words, the distance between the fastening point 211 and the virtual object 311 when the virtual object 311 is determined to be an overlapping object is longer than the distance between the fastening point 211 and the virtual object 311 when the virtual object 311 is not determined to be an overlapping object. The distance between the fastening point 214 and the virtual object 314 when the virtual object 314 is determined to be an overlapping object is longer than the distance between the fastening point 214 and the virtual object 314 when the virtual object 314 is not determined to be an overlapping object.

[0057] When moving the overlapping object, the processing device 150 searches for the position after the movement. Specifically, the processing device 150 calculates the center of gravity of the article 200 projected onto the virtual surface 350. The processing device 150 calculates the center of gravity of the overlapping object projected onto the virtual surface 350. The processing device 150 changes the position of the overlapping object in a direction from the center of gravity of the article 200 toward the center of gravity of the overlapping object. When the position of the overlapping object changes, the processing device 150 performs the overlap determination again using the position after the movement.

[0058] The search for the position is repeated until a termination condition is satisfied. The termination condition is set to be that the virtual object moves to a position where it does not overlap with the article 200. Alternatively, the termination condition may be set to be that the movement of the virtual object and the execution of the overlap determination are repeated a specified number of times. If a position where the virtual object does not overlap with the article 200 is not found even after the movement of the virtual object and the execution of the overlap determination are repeated a specified number of times, the processing device 150 determines the position of the virtual object where the overlapping area between the article 200 and the virtual object is the smallest as the position of the virtual object after the movement. The overlapping area between the article 200 and the virtual object corresponds to the number of regions 351 to which both the label of the article 200 and the label of the virtual object are assigned.

[0059] As a result of the movement, a portion of the overlapping object may overlap the item 200, and another portion of the overlapping object may not overlap the item 200. In this case, the processing device 150 may cause the appearance of the portion of the overlapping object to differ from the appearance of the other portion of the overlapping object. For example, the portion of the overlapping object may be displayed lighter than the other portion of the overlapping object.

[0060] It is preferable that virtual objects 311a and 314a are displayed as virtual objects 311 and 314 move away from fastening point 211 and fastening point 214, respectively. Virtual object 311a indicates the correspondence between virtual object 311 and fastening point 211. Virtual object 314a indicates the correspondence between virtual object 314 and fastening point 214. By displaying virtual objects 311a and 314a, the worker can easily understand which fastening point virtual object 311 and virtual object 314 correspond to.

[0061] The overlap determination is repeatedly performed. For example, the overlap determination is repeatedly performed at a predetermined interval. Alternatively, the overlap determination may be performed again when the orientation or position of the reference point changes. The change in orientation or position is determined based on the detection result of the sensor 140.

[0062] As an example, a worker moves around the article 200 and views the article 200 from the direction of the arrow A1 shown in FIG. 14. In this case, similarly to the above example, the distance between the reference point and the article 200 and the distance between the reference point and each of the virtual objects 311 to 314 are calculated. It is determined whether each of the virtual objects 311 to 314 is located on the back side or the front side of the article 200. In addition, a virtual plane 350 is set, and an overlap between the article 200 and the virtual object located on the back side on the virtual plane 350 is determined. As a result, it is determined that the virtual object 311 and the virtual object 312 are located on the front side of the article 200. It is determined that the virtual object 313 and the virtual object 314 are located on the back side of the article 200. In addition, it is determined that the virtual object 313 and the virtual object 314 overlap with the article 200.

[0063] The processing device 150 makes the appearance of the virtual object 313 and the appearance of the virtual object 314 different from the appearance of the virtual object 311 and the appearance of the virtual object 312, as shown in, for example, FIG. 16(a). That is, the appearance of the virtual object 313 changes from the state shown in FIG. 15(a) to the state shown in FIG. 16(a) in response to the movement of the worker. The virtual object 314 overlaps with the article 200 in both the state shown in FIG. 15(a) and the state shown in FIG. 16(a). Therefore, the appearance of the virtual object 314 does not change during the transition from the state shown in FIG. 15(a) to the state shown in FIG. 16(a). Moreover, the virtual object 311 is located behind the article 200 and overlaps with the article 200 in the state shown in FIG. 15(a). In the state shown in FIG. 16(a), the virtual object 311 is located in front of the article 200. Therefore, the appearance of the virtual object 311 changes in the same manner as the appearance of the virtual object 312.

[0064] Alternatively, the processing device 150 may change the position of the virtual object 313 and the position of the virtual object 314 as shown in Fig. 16(b). In the state shown in Fig. 15(b), the virtual object 311 and the virtual object 314 are displayed at positions away from the fastening point 211 and the fastening point 214, respectively. In the state shown in Fig. 16(b), the virtual object 313 and the virtual object 314 are displayed at positions away from the fastening point 213 and the fastening point 214, respectively. For example, while the worker is moving, the position of the virtual object 313 changes so that the virtual object 313 does not overlap with the article 200. Also, while the worker is moving, the position of the virtual object 311 approaches the fastening point 211 within a range where it does not overlap with the article 200.

[0065] A virtual object 313a indicating the correspondence between the fastening point 213 and the virtual object 313 is displayed between the fastening point 213 and the virtual object 313. A virtual object 314a indicating the correspondence between the fastening point 214 and the virtual object 314 is displayed between the fastening point 214 and the virtual object 314.

[0066] The processing device 150 may display an image of the article 200 captured by the image camera 131 superimposed on the position of the article 200. In this case, a virtual object located behind the article 200 and overlapping with the article 200 is not displayed. In the example shown in Fig. 17, a part of the virtual object 311 overlaps with the article 200, and the entire virtual object 314 overlaps with the article 200. In this case, the part of the virtual object 311 and the entire virtual object 314 are not visible to the worker.

[0067] Advantages of embodiments of the present invention will now be described. According to an embodiment of the present invention, a virtual object corresponding to a fastening location is displayed. The virtual object is displayed at a position where a worker's hands should be placed when performing a task. By displaying the virtual object, the worker can easily visually understand where to position his / her hands and which fastening location to turn the screw for. Furthermore, in this method, by linking data on the estimated location to data indicating that a screw has been turned, it is also possible to automatically create a record of the task.

[0068] Incidentally, virtual objects are displayed superimposed on the real space. When the positional relationship between the virtual objects and the objects in the real space is not taken into consideration, all virtual objects are displayed superimposed on the real space as shown in FIG. 14. This may lead to the worker mistaking a virtual object that is actually located behind an object in the real space for being located in front of the object. That is, it may be difficult for the worker to recognize where the virtual object is located relative to the object in the real space. In addition, the worker may reach out to the virtual object behind due to the misunderstanding, and hit his / her hand against the object in the real space, resulting in injury to the worker.

[0069] Regarding this problem, in the embodiment, an overlap determination is performed. That is, it is determined whether a virtual object is located behind an object in real space with respect to a reference point and overlaps with the object. If it is determined that an overlapping object exists, the processing device 150 changes the appearance of the overlapping object. By changing the appearance of the overlapping object, it becomes easier for the worker to understand which virtual object is actually located behind the object.

[0070] Alternatively, as a result of the overlap determination, the processing device 150 may change the positions of the overlapping objects. Changing the positions of the overlapping objects makes it easier for the worker to grasp which virtual object is actually located behind the article.

[0071] According to the embodiment, the user of the MR device 100 can more easily grasp the positional relationship between an object in real space and a virtual object in virtual space. By using the MR device 100 according to the embodiment for work, it becomes possible to more efficiently support the work.

[0072] The processing device 150 repeatedly executes the overlap determination. When the worker moves, the overlap determination is executed again based on the reference point after the movement. For example, as shown in Fig. 15(a) to Fig. 16(b), the appearance or position of each virtual object changes according to the movement of the reference point. This makes it possible to support the worker in real time while working.

[0073] 18(a), 18(b), 19, 20, 21(a), and 21(b) are schematic views for explaining display examples by the processing system according to the embodiment. The processing device 150 may display information related to the work on the virtual object. For example, as shown in Fig. 18(a), the processing device 150 displays numbers indicating the order of work for fastening points 211-214 on virtual objects 311-314, respectively. In Fig. 18(a), the numbers displayed on the virtual objects are written in speech bubbles.

[0074] When information is displayed on a virtual object, the processing device 150 may change the orientation of the displayed information depending on the positional relationship between the object and the virtual object in real space. In the example shown in FIG. 18(a), the virtual object 311 displaying the number "1" is located in front of the article 200 as seen by the worker. The processing device 150 displays the number "1" toward the worker. The worker can see the normal number "1".

[0075] In contrast, virtual object 313 displaying "2" is located behind article 200 as viewed by the worker, and overlaps with article 200. Processing device 150 displays number "2" facing away from the worker. As a result, the worker sees number "2" inverted left to right. Virtual object 312 displaying "4" and virtual object 314 displaying "3" are located to the side of article 200 as viewed by the worker. Processing device 150 displays numbers "3" and "4" facing toward the side of article 200. As a result, the worker sees numbers "3" and "4" that are narrower than usual.

[0076] The orientation of the information is determined according to the positional relationship between the article 200 and each virtual object. For example, the information is displayed in a direction from the article 200 to the virtual object. More specifically, the processing device 150 calculates the center of gravity of the article 200. The processing device 150 displays the information in a direction from the center of gravity of the article 200 to the virtual object.

[0077] The orientation of the information is calculated repeatedly. When the reference point moves, the orientation of the information also changes. For example, as shown in FIG. 18(b), when the worker moves to the opposite side of the article 200, the orientation of the information is also reversed from the state shown in FIG. 18(a) to the left and right.

[0078] 19, virtual objects 321 to 324 may be displayed in addition to virtual objects 311 to 314. Virtual objects 321 to 324 are adjacent to virtual objects 311 to 314, respectively. Virtual objects 321 to 324 include information regarding fastening screws into fastening points 211 to 214, respectively.

[0079] 20, for example, the virtual object 321 includes task information including an identification number 321a, a specified torque value 321b, a detected value 321c, a meter 321d, a ratio 321e, and a count 321f. The identification number 321a is the identification number of the fastening point 211. The specified torque value 321b is a torque value specified for fastening a screw into the fastening point 211. The detected value 321c is a torque value detected by a tool. The meter 321d indicates the specified torque value and the detected torque value. The ratio 321e indicates the ratio of the detected value to the specified torque value.

[0080] Depending on the work, it is required to fasten a screw multiple times into one fastening point. As an example, for fastening points 211 to 214, it is specified that the screws are to be fastened in the following order: fastening point 211, fastening point 213, fastening point 214, and fastening point 212. Furthermore, it is specified that after fastening the screws into fastening points 211 to 214, the screws are to be tightened again in the same order. In this case, the number of times 321f indicates the number of times the screw has been fastened into fastening point 211.

[0081] As with the virtual objects 311 to 314, the overlap of the virtual objects 321 to 324 with the article 200 is determined. In the example shown in Fig. 21(a) , among the virtual objects 321 to 324, the virtual objects 321 and 324 are determined to be overlapping objects. As a result, as with the virtual objects 311 to 314, the appearances of the virtual objects 321 and 324 are displayed differently from the appearances of the virtual objects 322 and 323, as shown in Fig. 21(a) . Alternatively, the positions of the virtual objects 321 and 324 change, as shown in Fig. 21(b) .

[0082] FIG. 22 is a schematic diagram illustrating the state of work. The positions of the virtual objects 311-314 are not limited to positions adjacent to the fastening points 211-214, and may be set appropriately depending on the tools used. For example, as shown in Fig. 22, a wrench 400 and an extension bar 450 may be used for work. The wrench 400 fastens a screw into the fastening point via the extension bar 450.

[0083] 23, 24(a), and 24(b) are schematic diagrams for explaining display examples by the processing system according to the embodiment. When the work shown in FIG. 22 is performed, the processing device 150 displays the virtual objects 311-314 at positions away from the fastening points 211-214 as shown in FIG. 23. The fastening points 211-214 are each away from the virtual objects 311-314 by the length of the extension bar 450. In this case, it is preferable to display virtual objects 311a-314a indicating that the virtual objects 311-314 correspond to the fastening points 211-214, respectively. The virtual objects 311a-314a are displayed between the fastening points 211-214 and the virtual objects 311-314, respectively. By displaying the virtual objects 311a-314a, the worker can easily understand which fastening points the virtual objects 311-314 correspond to.

[0084] The processing device 150 executes overlap determination for each of the virtual objects 311-314 and the virtual objects 311a-314a. As a result, the virtual objects 313, 313a, 314, and 314a are determined to be overlapping objects. As shown in Fig. 24(a), the appearances of the virtual objects 313, 313a, 314, and 314a change. Alternatively, as shown in Fig. 24(b), the positions of the virtual objects 313, 313a, 314, and 314a change.

[0085] FIG. 25 is a flowchart illustrating an example of a processing method according to an embodiment. In the processing method shown in Fig. 25, first, the processing device 150 accepts the selection of an operation and a tool (step S0). The operation and the tool are selected by the worker. The operation to be performed and the tool to be used may be instructed by a higher-level system, and the processing device 150 may accept the selection by the higher-level system. Based on data obtained from the image camera 131 or other sensors, the processing device 150 may determine the operation to be performed and the tool to be used. The processing device 150 accepts the selection according to the determination result.

[0086] Next, the processing device 150 sets a base point (step S1). The base point is set based on a marker or a hand gesture. The processing device 150 sets a three-dimensional coordinate system based on the set base point, and displays one or more virtual objects in the virtual space (step S2). The processing device 150 acquires the position of each object in the real space (step S3). The object position is acquired from a depth image. The object position may be registered in advance. The processing device 150 acquires the position of each virtual object in the virtual space (step S4). The processing device 150 executes overlap determination using the acquired positions (step S5). That is, it is determined whether any virtual object is located behind an object in the real space and overlaps with the object.

[0087] The processing device 150 determines whether or not there are overlapping objects as a result of the overlap determination (step S6). If there are no overlapping objects, the processing device 150 maintains the appearance and position of each virtual object as they were when they were displayed in step S1. If there are overlapping objects, the processing device 150 changes the appearance or position of the overlapping objects (step S7).

[0088] The processing device 150 determines whether the hand has come into contact with any of the virtual objects (step S8). If the hand has not come into contact with any of the virtual objects, the processing device 150 executes step S3 again. If the hand has come into contact with a virtual object, the processing device 150 estimates that a screw is being turned into the fastening point corresponding to that virtual object (step S9). The processing device 150 receives a detection value from the tool. The processing device 150 associates the detection value with data related to the fastening point, and stores the associated data (step S10).

[0089] A mark indicating the completion of the work may be attached to the fastened screw. When the screw tightening is completed, the worker puts a mark on the screw or its vicinity. The screw tightening tool may automatically put the mark on completion. The processing device 150 may detect the mark from the image (step S11). When the processing device 150 determines that the screw tightening is completed, it refers to the color of the mark used in the screw tightening. The processing device 150 determines whether the number of pixels of the mark color in the image exceeds a preset threshold value. If the number of pixels exceeds the threshold value, the processing device 150 determines that a mark has been attached to the screw. The processing device 150 further links the detection result indicating that the mark has been detected to the data regarding the fastening location. The processing device 150 determines whether the work is completed (step S12). If the work is not completed, step S3 is executed again.

[0090] The order of the processes shown in Fig. 25 can be changed as appropriate. For example, steps S3 to S7 and steps S8 to S11 may be executed in parallel. Steps S8 to S11 may be executed before steps S3 to S7.

[0091] During the work, the worker can also move the base point. The worker instructs the processing device 150 to move the base point. The instruction may be input using the input device 3, or may be input by a hand gesture indicating the movement of the base point. When the processing device 150 receives the input, it interrupts the processing being executed and proceeds to step S1. Each virtual object is displayed based on the reset base point.

[0092] FIG. 26 is a schematic diagram illustrating data to be referred to in the embodiment. The storage device 5 stores master data 51 and history data 52. The master data 51 includes operation master data 51a, tool master data 51b, base point master data 51c, and fastening point master data 51d. The master data 51 is prepared in advance before the screw tightening.

[0093] The task master data 51a stores a task ID, a task name, an item ID, and an item name. The processing device 150 can accept any one of the task ID, task name, item ID, or item name as a task selection.

[0094] The tool master data 51b stores a tool type and a tool ID. The tool type indicates a classification of a tool based on a structure, an external shape, or performance. The processing device 150 can accept either the tool type or the tool ID as a selection of a tool.

[0095] The base point master data 51c stores a method for setting a base point for each task. The processor 150 sets a base point according to the method for setting a base point for each task.

[0096] The fastening point master data 51d stores the fastening point ID, and for each fastening point ID, the fastening position, angle, extension type, torque value, number of times of screw tightening, mark color, object shape, and display mode are stored. The fastening position indicates the position of each fastening point. The angle indicates the angle of the tool or extension bar when tightening a screw at each fastening point. The extension type indicates the classification of the extension bar according to the structure, external shape, performance, etc. The torque value indicates the magnitude of the torque required to tighten a screw at each fastening point. The number of times of screw tightening indicates the number of times a screw is tightened at each fastening point. The mark color is the color of a mark indicating the completion of screw tightening. The virtual object shape indicates the shape of a virtual object displayed corresponding to each fastening point. The display mode indicates the mode of each virtual object displayed. The display mode may be set for each number of times of screw tightening.

[0097] The position where each virtual object is displayed is calculated based on the base point, fastening position, angle, extension type, etc. For example, the length of the extension bar is specified by the extension type. A first virtual object is displayed at a position away from the fastening position by the length of the extension bar at a set angle. A virtual object is displayed in an area of ​​the length of the extension bar at a set angle to the fastening position.

[0098] Here, an example in which the embodiment of the present invention is applied to the task of tightening a screw has been mainly described. The embodiment of the present invention may also be applied to the task of loosening a screw. When loosening a screw, a tool is used to turn the screw as shown in FIG. 7. In this case, too, the task can be performed efficiently by displaying a virtual object. In addition, by performing overlap determination, the user of the MR device 100 can more easily grasp the positional relationship between an object in real space and a virtual object in virtual space.

[0099] FIG. 27 is a schematic diagram showing a hardware configuration. 27 is used as the processing device 2 or the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0100] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0101] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0102] The storage device 94 stores data necessary for executing the program and data obtained by executing the program. The storage device 94 includes a solid state drive (SSD) and the like. The storage device 94 may be used as the storage device 5 or the storage device 170.

[0103] An input interface (I / F) 95 can connect the computer 90 to the input device 3. The CPU 91 can read various data from the input device 3 via the input I / F 95.

[0104] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to the display device 4 via the output I / F 96 and cause the display device 4 to display information.

[0105] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0106] The data processing by the processing device 2 or the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0107] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0108] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0109] In the above description of the embodiment, an example in which the processing system 10 is realized as an MR device has been mainly described. The invention according to the embodiment is not limited to this example. The processing system 10 may be realized by a general-purpose personal computer. In this case, a keyboard, a microphone, a touch pad, or the like is used as the input device 3. A monitor is used as the display device 4.

[0110] In the above embodiment, an example has been described in which the processing system 10 and the MR device 100 are used for work. However, the present invention is not limited to this example, and the processing system 10 and the MR device 100 may be used for purposes other than work. Even in this case, the overlap determination is performed, so that a person using the MR device 100 can more easily grasp the positional relationship between an object in real space and a virtual object in virtual space.

[0111] Embodiments of the invention include the following features. (Feature 1) A mixed reality device that displays a virtual object in a virtual space by overlaying it on a real space, acquiring a position of the object in the real space and a position of the virtual object; A mixed reality device that changes an appearance of the virtual object when the virtual object is located behind the object with respect to the mixed reality device and overlaps with the object. (Feature 2) The mixed reality device of feature 1, further comprising: acquiring a depth image in the real space; and acquiring the position of the object from the depth image. (Feature 3) A virtual plane is set in front of the object and the virtual object; The mixed reality device of feature 2, wherein the device projects the object and the virtual object on the virtual surface to determine whether the virtual object overlaps with the object. (Feature 4) the depth image is mesh data including a plurality of meshes; extracting, from the plurality of meshes, a mesh in which a distance between the mixed reality device and a straight line connecting the virtual object is less than a threshold; The mixed reality device according to feature 2 or 3, further comprising: determining whether the virtual object is located behind the extracted mesh and overlaps with the extracted mesh. (Feature 5) The mixed reality device according to any one of features 1 to 4, wherein the color or size of the virtual object is changed when the virtual object is located behind the object and overlaps with the object. (Feature 6) A mixed reality device described in any one of features 1 to 4, wherein when the virtual object is located behind the object and overlaps with the object, the mixed reality device displays an image of the object superimposed on the object. (Feature 7) A three-dimensional coordinate system is set with a marker provided in the real space as a base point; The mixed reality device according to any one of features 1 to 6, wherein the virtual object is displayed at a position registered in advance in the three-dimensional coordinate system. (Feature 8) The mixed reality device according to any one of features 1 to 7, wherein the virtual object is displayed corresponding to a fastening point of an article in the real space. (Feature 9) receiving, from a tool for tightening a screw into the fastening portion, a detection value detected by the tool; The mixed reality device of feature 8, wherein when a hand or the tool comes into contact with the virtual object, the detection value is linked to data regarding the fastening point. (Feature 10) Displaying information regarding the work to be performed on the fastening portion on the virtual object; 10. The mixed reality device of feature 8 or 9, wherein the information is displayed in a direction from the item to the virtual object. (Feature 11) Acquire a position of an object in a real space and a position of a virtual object in a virtual space that is displayed superimposed on the real space; a processing device that changes an appearance of the virtual object in the virtual space when the virtual object is located behind the physical object with respect to a reference point and overlaps with the physical object. (Feature 12) The processing device includes: acquiring a position of an object in a real space and a position of a virtual object in a virtual space that is displayed superimposed on the real space; changing an appearance of the virtual object in the virtual space when the virtual object is located behind the object with respect to a reference point and overlaps with the object; Processing methods. (Feature 13) A program for causing the processing device to execute the processing method according to feature 12. (Feature 14) A storage medium storing the program according to feature 13.

[0112] According to the above-described embodiment, a mixed reality device, a processing device, a processing method, a program, and a storage medium are provided that allow a user of an MR device to more easily grasp the positional relationship between an object in real space and a virtual object in virtual space. According to the embodiment, it is possible to more appropriately support the tightening work of a screw.

[0113] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0114] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0115] 1: Photographing device, 2: Processing device, 3: Input device, 4: Display device, 5: Storage device, 10: Processing system, 51: Master data, 51a: Work master data, 51b: Tool master data, 51c: Base point master data, 51d: Fastening point master data, 52: History data, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: MR device, 101: Frame, 111, 112: Lens, 121, 122: Projection device, 131: Image camera, 132: Depth camera, 132a: Field of view, 140: Sensor, 141: Microphone, 150: Processing device, 160: Battery, 170: Storage device, 200: object, 200a: area, 211-214: fastening point, 220: marker, 225: screw, 251: left hand, 252: right hand, 261, 262: virtual object, 311-314, 311a-314a, 321-324: virtual object, 321a: identification number, 321b: specified torque value, 321c: detection value, 321d: meter, 321e: ratio, 321f: number of times, 350: virtual surface, 351: area, 400: wrench, 420: marker, 450: extension bar, A1: arrow, D1: first direction, D2: second direction, La1-La3: label

Claims

1. A mixed reality device that displays a virtual object in a virtual space by overlaying it on a real space, acquiring a position of the object in the real space and a position of the virtual object; A mixed reality device that changes an appearance of the virtual object when the virtual object is located behind the object with respect to the mixed reality device and overlaps with the object.

2. The mixed reality device of claim 1 , further comprising: a depth image in the real space; and a position of the object from the depth image.

3. A virtual plane is set in front of the object and the virtual object; The mixed reality device according to claim 2 , wherein the device determines whether the virtual object overlaps with the object by projecting the object and the virtual object on the virtual surface.

4. the depth image is mesh data including a plurality of meshes; extracting, from the plurality of meshes, a mesh in which a distance between the mixed reality device and a straight line connecting the virtual object is less than a threshold; The mixed reality device according to claim 2 , further comprising: determining whether the virtual object is located behind the extracted mesh and overlaps with the extracted mesh.

5. The mixed reality device according to claim 1 , wherein a color or a size of the virtual object is changed when the virtual object is located behind the object and overlaps with the object with respect to the mixed reality device.

6. The mixed reality device according to claim 1 , wherein, when the virtual object is located behind the object and overlaps with the object, the mixed reality device displays an image of the object superimposed on the object.

7. A three-dimensional coordinate system is set with a marker provided in the real space as a base point; The mixed reality device according to claim 1 , wherein the virtual object is displayed at a position registered in advance in the three-dimensional coordinate system.

8. The mixed reality device according to claim 1 , wherein the virtual object is displayed corresponding to a fastening point of an article in the real space.

9. receiving, from a tool for tightening a screw into the fastening portion, a detection value detected by the tool; The mixed reality device according to claim 8 , wherein when a hand or the tool comes into contact with the virtual object, the detection value is linked to data relating to the fastening location.

10. Displaying information regarding the work to be performed on the fastening portion on the virtual object; The mixed reality device of claim 8 , wherein the information is displayed in a direction from the article to the virtual object.

11. Acquire a position of an object in a real space and a position of a virtual object in a virtual space that is displayed superimposed on the real space; a processing device that changes an appearance of the virtual object in the virtual space when the virtual object is located behind the physical object with respect to a reference point and overlaps with the physical object.

12. The processing device includes: acquiring a position of an object in a real space and a position of a virtual object in a virtual space that is displayed superimposed on the real space; changing an appearance of the virtual object in the virtual space when the virtual object is located behind the object with respect to a reference point and overlaps with the object; Processing methods.

13. A program for causing the processing device to execute the processing method according to claim 12.

14. A storage medium storing the program according to claim 13.

Images