Mixed reality device, processing method, program and storage medium

Abstract

To provide a mixed reality device with improved convenience, and provide a processing method, a program and a storage medium for improving the convenience of the mixed reality device.SOLUTION: A mixed reality device is configured to display a virtual space to overlap a real space. The mixed reality device is configured to set an origin of the virtual space by using a predetermined object imaged in the real space. The mixed reality device is configured to acquire a display position of a virtual object in a three-dimensional coordinate system based on the origin. The mixed reality device is configured to determine whether or not the display position is included in a visible region set in front of the mixed reality device. The mixed reality device is configured to display a guide indicating a direction of the display position when the display position is not included in the visible region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] SUMMARY OF THE PRESENT EMBODIMENTS An embodiment of the present invention relates to a mixed reality device, a processing method, a program, and a storage medium. [Background technology]

[0002] Conventionally, mixed reality devices have been used for efficient work. Mixed reality devices can display a virtual space overlaid on a real space, and can provide various information to workers. Workers can perform work more efficiently by referring to the information displayed by the mixed reality device. There is a demand for technology that can further improve the convenience of mixed reality devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a mixed reality device having better convenience. Also, the problem to be solved by the present invention is to provide a processing method, a program, and a storage medium capable of further improving the convenience of the mixed reality device. [Means for solving the problem]

[0005] A mixed reality device according to an embodiment is capable of displaying a virtual space superimposed on a real space. The mixed reality device sets a base point for the virtual space using a predetermined object photographed in the real space. The mixed reality device obtains a display position of a virtual object in a three-dimensional coordinate system based on the base point. The mixed reality device determines whether the display position is within a viewing area set in front of the mixed reality device. If the display position is not within the viewing area, the mixed reality device displays a guide indicating the direction of the display position. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing a mixed reality device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. [Diagram 3] FIG. 3 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Diagram 5] FIG. 5 is a schematic diagram illustrating the state of work. [Figure 6] FIG. 6 is a schematic diagram showing the field of view of an operator. [Figure 7] FIG. 7 is a schematic plan view showing the field of vision of an operator. [Figure 8] FIG. 8 is a schematic diagram showing the field of view of an operator. [Figure 9] FIG. 9 is a schematic plan view showing the field of vision of an operator. [Figure 10] FIG. 10 is a flowchart illustrating a processing method according to an embodiment. [Figure 11] FIG. 11 is a flowchart showing a method for displaying a guide. [Figure 12] 12(a) and 12(b) are schematic diagrams showing an example of output from the MR device according to the embodiment. [Figure 13]FIG. 13 is a schematic diagram showing an output example of the MR device according to the embodiment. [Figure 14] FIG. 14 is a flowchart showing a method for calculating a route. [Figure 15] 15(a) and 15(b) are schematic diagrams for explaining a route calculation method. [Figure 16] 16(a) and 16(b) are schematic diagrams for explaining a route calculation method. [Figure 17] 17(a) and 17(b) are schematic diagrams for explaining a route calculation method. [Figure 18] 18(a) and 18(b) are schematic diagrams for explaining a route calculation method. [Figure 19] FIG. 19 is a schematic diagram for explaining another method of calculating a route. [Figure 20] FIG. 20 is a schematic diagram for explaining another method of calculating a route. [Figure 21] 21(a) and 21(b) are schematic diagrams showing an example of output from a mixed reality device according to the embodiment. [Figure 22] FIG. 22 is a flowchart showing processing by the mixed reality device according to the embodiment. [Diagram 23] FIG. 23 is a schematic diagram for explaining the process of the flowchart shown in FIG. [Figure 24] 24(a) and 24(b) are schematic diagrams showing an example of output by a mixed reality device according to an embodiment. [Diagram 25] FIG. 25 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be shown differently depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing a mixed reality device according to an embodiment. The embodiment of the present invention relates to a mixed reality device (MR device). For example, as shown in Fig. 1, the MR device 100 according to the embodiment includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a light source 133, an eye tracking camera 134, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0009] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0010] The projection device 121 and the projection device 122 display the recognition result of the body of the worker, a virtual object, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0011] The lenses 111 and 112 are optically transparent. The worker can visually recognize the real state through the lenses 111 and 112. The worker can also visually recognize information projected onto the lenses 111 and 112 by the projection devices 121 and 122. Information (virtual space) is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0012] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 emits infrared light and obtains a depth image based on the reflected infrared light. The light source 133 emits light (e.g., infrared light) toward the wearer's eyeball. The eye tracking camera 134 detects the light reflected by the wearer's eyeball. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0013] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the projection device 121 and the projection device 122 to display information on the lenses 111 and 112. Hereinafter, the processing device 150 causing the lenses 111 and 112 to display information using the projection devices 121 and 122 is also simply referred to as "the processing device displays information." In addition, the processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view.

[0014] In addition, the processing device 150 can execute various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc. For example, the processing device 150 recognizes a preset object from the image obtained by the image camera 131. The processing device 150 recognizes the surface shape of the object from the image obtained by the depth camera 132. The processing device 150 calculates the viewpoint and line of sight of the worker's eyes from the detection result obtained by the eye tracking camera 134.

[0015] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0016] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0017] FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. For example, a worker wearing the MR device 100 works on an article 200 shown in Fig. 2. The article 200 is a cylindrical member and has fastening points 201 to 208. The worker uses a wrench and an extension bar to turn the screws into the fastening points 201 to 208 in order.

[0018] When performing work using the MR device 100, a base point of the virtual space is set using a specific object. When the base point is set, a three-dimensional coordinate system of the virtual space is determined. In the example shown in FIG. 2, a marker 210 is provided near the work object. The marker 210 is an AR marker. As described later, the marker 210 is provided to set the base point of the three-dimensional coordinate system. Instead of the AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, the base point may be indicated by the worker's hand (hand gesture). The processing device 150 sets the base point of the three-dimensional coordinate system using multiple points indicated by the hand gesture.

[0019] As long as the base point of the three-dimensional coordinate system can be set, any object can be used for the setting. Here, an example in which the base point is set using the marker 210 will be described.

[0020] At the start of the work, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a base point in the virtual space based on the position and orientation of the marker 210. By setting the base point based on a predetermined object that exists in the real space, a virtual object is displayed corresponding to the object in the real space.

[0021] The processing device 150 also calculates the position of the MR device 100. As an example, the processing device 150 calculates the position and orientation of the MR device 100 by a function of spatial mapping. In the MR device 100, the depth camera 132 measures the distance between the MR device 100 and the surrounding objects. Surface information of the surrounding objects is obtained from the measurement results by the depth camera 132. The surface information includes the position and orientation of the object's surface. For example, the surface of each object is represented by a plurality of meshes, and the position and orientation are calculated for each mesh. The processing device 150 calculates the relative position and orientation of the MR device 100 with respect to the surfaces of the surrounding objects from the surface information. When the marker 210 is recognized, the position of each surface is also represented in a three-dimensional coordinate system with the marker 210 as the base point. The position and orientation of the MR device 100 in the three-dimensional coordinate system are calculated from the positional relationship between the surface of the object and the MR device 100. The spatial mapping is repeatedly performed at a predetermined interval.

[0022] FIG. 3 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. During work, the image camera 131 and the depth camera 132 capture images of the article 200, the worker's left hand 251, and the worker's right hand 252. The processing device 150 recognizes the left hand 251 and the right hand 252 from the captured images by hand tracking. The processing device 150 may cause the projection device 121 and the projection device 122 to display the recognition results on the lens 111 and the lens 112. Hereinafter, the processing device causing the projection device to display information on the lens is also simply referred to as "the processing device displays information."

[0023] 3, the processing device 150 displays the recognition results of the left hand 251 and the right hand 252 superimposed on the hands in real space. In the illustrated example, a plurality of virtual objects 261 and a plurality of virtual virtual objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The plurality of virtual objects 261 respectively represent the plurality of joints of the left hand 251. The plurality of virtual objects 262 respectively represent the plurality of joints of the right hand 252. Instead of the joints, virtual objects (meshes) respectively representing the surface shapes of the left hand 251 and the right hand 252 may be displayed.

[0024] When the processing device 150 recognizes the left hand 251 and the right hand 252, it measures the position of each hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may be used as the position of the hand. Or, the position of the center of the entire hand may be used as the position of the hand.

[0025] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. The processing device 150 displays virtual objects 301-305 and virtual objects 311-315 as shown in Fig. 4. The virtual objects 311-315 are displayed between the fastening points 201-205 and the virtual objects 301-305, respectively. The virtual objects 311-315 indicate to which fastening points the virtual objects 301-305 correspond, respectively.

[0026] In the illustrated example, the virtual objects 301 to 305 are spherical, and the virtual objects 311 to 315 are rod-shaped. As long as the worker can visually recognize each virtual object, the shape of each object is not limited to this example. For example, the virtual objects 301 to 305 may be cubes, and the virtual objects 311 to 315 may be linear. Also, virtual objects similar to the virtual objects 301 to 305 and the virtual objects 311 to 315 are displayed for the fastening points 206 to 208 not shown in FIG. 4.

[0027] FIG. 5 is a schematic diagram illustrating the state of work. For example, as shown in Fig. 5, a wrench 280 and an extension bar 290 are used to turn a screw into a fastening point. As an example, when fastening a screw into fastening point 203, an operator places a screw into a threaded hole of fastening point 203. The operator fits one end of extension bar 290 onto the screw. The operator fits the head of wrench 280 onto the other end of extension bar 290. The operator holds the head of wrench 280 with one hand and holds the grip of wrench 280 with the other hand. By turning wrench 280, the screw is tightened into fastening point 203 via extension bar 290.

[0028] At this time, the worker positions the extension bar 290 so that the extension bar 290 is close to or in contact with the virtual object 313. The worker also grips the head of the wrench 280 so that his or her hand comes into contact with the virtual object 303. By displaying the virtual object, the worker can easily grasp the positions of the tool and hand when turning the screw into the fastening point 203. This can improve the work efficiency.

[0029] The virtual object is displayed at a preset position in a three-dimensional coordinate system based on the marker 210. The position of the fastening point in real space may be registered in advance, and the virtual object may be displayed based on the position of the fastening point. Specifically, for the example shown in FIG. 5, the position of the fastening point 203, the length of the extension bar 290, and the like are registered in advance. The virtual object 313 is displayed between the position of the fastening point 203 and a position spaced from the fastening point 203 by the length of the extension bar 290. The virtual object 303 is displayed at a position spaced from the fastening point 203 by the length of the extension bar 290. As a result, the virtual objects 303 and 313 are displayed corresponding to the fastening point 203.

[0030] Figures 6 and 8 are schematic diagrams showing the field of vision of an operator, and Figures 7 and 9 are schematic plan views showing the field of vision of an operator. 6 and 7, an item 200 and a shelf 400 are present within the field of view V of a worker. In the illustrated example, the processing device 150 displays virtual objects 303 and 313 at the fastening point 203. As shown in FIG. 4, a virtual object may be displayed for each fastening point. The worker can visually recognize the virtual objects 303 and 313 and understand that the worker needs to perform work at the fastening point 203.

[0031] 8 and 9, the item 200 is present outside the field of view V of the worker. In this case, the virtual objects 303 and 313 are not displayed in the field of view V of the worker. The processing device 150 displays a guide 350 indicating the display positions of the virtual objects 303 and 313. When the worker moves or changes his / her direction according to the guide 350, the display positions of the virtual objects 303 and 313 come into the field of view V. As a result, the virtual objects 303 and 313 are displayed, and the worker can visually recognize the virtual objects 303 and 313. When the virtual objects 303 and 313 are displayed, the display of the guide 350 disappears.

[0032] In the illustrated example, the guide 350 is an arrow. The guide 350 may be expressed by something other than an arrow. For example, the virtual object may be displayed on the side where the virtual object exists as seen from the worker. The virtual object may be displayed in a flowing manner from in front of the worker toward the display position of the virtual object.

[0033] The display of the guide 350 is based on the determination result by the processing device 150. Specifically, the processing device 150 sets a visual recognition area R in front of the MR device 100, as shown in Figs. 7 and 9. The visual recognition area R is at least a part of the range visible to the worker when the worker wears the MR device 100 in a normal manner. When the processing device 150 acquires the display position of the virtual object, it determines whether or not the display position is present within the visual recognition area R. If the display position of the virtual object is not present within the visual recognition area R, the processing device 150 displays the guide 350. If the display position is present within the visual recognition area R, the processing device 150 does not display the guide 350.

[0034] The visual recognition area is set in a predetermined range with respect to a reference position and a reference direction. The position of the MR device 100 is used as the reference position. The reference direction is parallel to the front of the MR device 100. The position and direction of the MR device 100 are calculated by spatial mapping as described above. Alternatively, the processing device 150 may calculate the position and direction of the viewpoint from the position of the MR device 100 and the detection result by the eye tracking camera 134. In that case, the processing device 150 may use the position of the viewpoint as the reference position and the direction of the viewpoint as the reference direction. Here, an example in which the visual recognition area is set based on the position and direction of the MR device 100 will be described.

[0035] As an example, the visual recognition area is set in a range of 15 cm to 1 m in front of the reference position. The visual recognition area is also set in a range of 43 to 120 degrees up, down, left, and right with respect to the reference direction. More preferably, the visual recognition area is set in a range of 50 to 70 degrees up, down, left, and right with respect to the reference direction so that the visual recognition area is set only in a range that is easy for the operator to focus on. The size of the visual recognition area is set appropriately according to the field of vision of the wearer wearing the MR device 100.

[0036] The guide is displayed at a preset position relative to the reference position and reference direction. Specifically, a position a predetermined distance from the reference position in the reference direction is calculated as the display position of the guide. As an example, the predetermined distance is set to about 50 cm. After the display position of the guide is calculated, the processing device 150 determines whether the display position of the virtual object exists within the visual recognition area. If the display position of the virtual object does not exist within the visual recognition area, the processing device 150 displays the guide at the calculated display position. The guide is displayed to point in the direction from the display position of the guide to the display position of the virtual object.

[0037] FIG. 10 is a flowchart illustrating a processing method according to an embodiment. 10 is executed, task master data 170a, base point master data 170b, tool master data 170c, and fastening point master data 170d are prepared in advance. Each master data is stored in the storage device 170.

[0038] First, a task to be performed is selected (step S1). A task ID, a task name, an item ID, and an item name are registered in the task master data 170a. A task is specified by a task ID, a task name, an ID of an item to be worked on, or an item name. The processing device 150 accepts the selection of a task. For example, the task to be performed is selected by a worker. The task to be performed may be selected by a higher-level system, and the processing device 150 may accept the selection. The processing device 150 may determine the task to be performed based on data obtained from the image camera 131 or another sensor. The processing device 150 selects a task based on the result of the determination.

[0039] Next, the image camera 131 captures the marker 210. The processing device 150 sets a base point of a three-dimensional coordinate system based on the position and orientation of the marker 210 (step S2). At this time, the processing device 150 refers to the base point master data 170b. The base point master data 170b stores a method for setting a base point for each task. The processing device 150 acquires the method for setting a base point for the selected task, and sets the base point according to the setting method.

[0040] After setting the base point, the processing device 150 displays a virtual object (step S3). When the virtual object is displayed, the tool master data 170c and the fastening point master data 170d are appropriately referred to.

[0041] The tool master data 170c stores the ID, tool type, tool length, socket type, socket length, etc. of the tool used for each task. The tool type indicates the classification of the tool based on the structure, external shape, or performance. The tool length is the length from the center of rotation to the grip when the tool is used for screw tightening. The socket type indicates the classification of the socket based on the structure or external shape. The socket length indicates the length of the socket in the direction connecting the tool and the screw when the screw is tightened. The processing device 150 acquires data of the tool used in the task selected in step S1 from the tool master data 170c. When an extension bar is used, the type, length, etc. of the extension bar are also stored in the tool master data 170c. The processing device 150 further acquires data related to the extension bar from the tool master data 170c.

[0042] The fastening point master data 170d stores, for each fastening point, the ID of the fastening point, the position of the fastening point, the required torque value, and the number of times the screw is tightened. The fastening position indicates the position where the fastening point is located, and is specified by the position of the three-dimensional coordinate system set in step S2. The number of times the screw is tightened indicates the number of times the screw is tightened that is required for each fastening point. If a mark is applied to the screw after tightening, the color of the mark is also registered.

[0043] The processing device 150 recognizes a predetermined object from the image captured by the image camera 131 (step S4). The "predetermined object" is an object that is determined to have come into contact with a virtual object. The processing device 150 calculates the position of the predetermined object, and determines whether the distance between the object and the virtual object is less than a threshold (step S5). For example, if the distance is equal to the threshold, it is determined that the predetermined object has come into contact with the virtual object.

[0044] If the distance is equal to or greater than the threshold, step S4 is executed again. If the distance is less than the threshold, the processing device 150 presumes that an operation is being performed on the location corresponding to the virtual object.

[0045] The processing device 150 records the work related to the estimated location and saves it in the history data 170e (step S6). For example, the torque value detected by the tool is linked to the ID of the work and the ID of the estimated location. As shown in the figure, the processing device 150 may further link the type and ID of the tool used, the number of times the screw was tightened, and the mark recognition result to the ID of the fastening location. The mark is recognized by the processing device 150 from an image taken by the image camera 131. The processing device 150 extracts a block of pixels of the mark color from the image and counts the number of pixels in that block. If the number of pixels exceeds a preset threshold, it is determined that a mark has been attached.

[0046] Once the work is recorded, the display of the virtual object disappears. The processing device 150 determines whether the work is complete (step S7). If the work is not complete, step S3 is executed again.

[0047] The work master data 170a, the base point master data 170b, the tool master data 170c, the fastening point master data 170d, and the history data 170e may be stored in the storage device 170, or in a computer capable of communicating with the MR device 100 or a server on a network.

[0048] FIG. 11 is a flowchart showing a method for displaying a guide. The display method M1 shown in Fig. 11 is executed after step S3 shown in Fig. 11. The display method M1 is repeated until, for example, step S6 is executed.

[0049] First, the processing device 150 acquires a reference position and a reference direction (step S11). When the position and direction of the MR device 100 are used as the reference position and the reference direction, the data is acquired from the calculation result of the spatial mapping. The processing device 150 sets a visual recognition area based on the reference position and the reference direction (step S12). The processing device 150 calculates a display position of a guide based on the reference position and the reference direction (step S13). The processing device 150 determines whether the display position of the virtual object is within the visual recognition area (step S14). If the display position of the virtual object is outside the visual recognition area, the processing device 150 displays a guide at the display position calculated in step S13 (step S15). If the display position of the virtual object is within the visual recognition area or after step S15, the processing device 150 determines whether step S6 shown in FIG. 10 is completed (step S16). If step S6 is not completed, step S11 is executed again. In that case, in the next step S14, when it is determined that the display position of the virtual object is within the visual recognition area, the processing device 150 makes the guide invisible.

[0050] Advantages of the embodiment will be described. The MR device can provide various information to the wearer. For example, when used at a work site, the virtual object is displayed as shown in FIG. 4 or FIG. 5, allowing the worker to perform the work efficiently. On the other hand, when the virtual object is not present in the field of view, the wearer moves or turns around to search for the virtual object. At this time, since the wearer is concentrating on searching for the virtual object, the wearer's attention is not easily directed to the surrounding objects. As a result, while searching for the virtual object, the wearer may not notice the surrounding objects and may come into contact with the objects. In addition, while searching for the virtual object, the work does not progress, and the work efficiency also decreases.

[0051] Regarding this problem, according to the MR device 100 of the embodiment, after the display position of the virtual object is acquired, the processing device 150 determines whether or not the virtual object exists in the visual recognition area. When the virtual object does not exist in the visual recognition area, the processing device 150 displays a guide indicating the display position of the virtual object. By displaying the guide, the wearer can easily grasp the direction of the virtual object. Therefore, the effort of the wearer to search for the virtual object can be reduced. As a result, the possibility that the wearer will come into contact with surrounding objects while searching for the virtual object can be reduced. During work, the virtual object can be found more quickly, and work efficiency can be improved. According to the embodiment of the present invention, the convenience of the mixed reality device can be improved.

[0052] The processing device 150 may generate an environmental map showing the environment around the MR device 100. As described above, surface information of objects around the MR device 100 is obtained by spatial mapping. The processing device 150 refers to a database in which the surface information is stored. The processing device 150 determines whether the obtained surface information has already been stored in the database. If the surface information has not yet been stored in the database, the processing device 150 stores the obtained surface information in the database. In this way, an environmental map showing the environment around the MR device 100 is generated.

[0053] The processing device 150 may use the surface information to display the guide. For example, after the display position of the guide is calculated, the processing device 150 determines whether the surface of an object exists between the guide and the virtual object. If a surface exists between the guide and the virtual object, the processing device 150 indicates the guide in a direction in which the surface does not exist.

[0054] 12(a), 12(b), and 13 are schematic diagrams showing examples of outputs from the MR device according to the embodiment. As shown in FIG. 12(a), the processing device 150 calculates a reference position P1 and a reference direction D1. The processing device 150 calculates a display position P2 of a guide based on the reference position P1 and the reference direction D1. The processing device 150 calculates a vector V1 from the display position P2 to a display position P3 of a virtual object. The processing device 150 determines whether the vector V1 intersects with a recognized surface. In the example shown in FIG. 12(a), no surface exists between the display position P2 and the display position P3. The processing device 150 displays a guide 350 indicating a direction parallel to the vector V1.

[0055] On the other hand, in the example shown in FIG. 12(b), the vector V1 intersects with the surface Su of the object. In this case, the processing device 150 searches for a vector V2 that does not intersect with the surface Su and has a small angle with the vector V1. The processing device 150 displays a guide 350 indicating a direction parallel to the vector V2. This can reduce the possibility that the wearer will come into contact with an object when the wearer moves according to the guide 350. This can improve the safety of the wearer.

[0056] In the example shown in FIG. 13, when the MR device 100 is at position P1a, the display position of the guide is determined to be position P2a. The display position P3 of the virtual object is not included in the visual recognition area. Furthermore, there is no surface between the guide position P2a and the virtual object display position P3. Therefore, the guide indicates the direction from position P2a to display position P3. When the MR device 100 is at position P1b, the display position of the guide is determined to be position P2b. The display position P3 is not included in the visual recognition area. There is no surface between the guide position P2b and display position P3. Therefore, the guide indicates the direction from position P2b to display position P3.

[0057] When the MR device 100 is at position P1c, the display position of the guide is determined to be position P2c. The display position P3 of the virtual object is not included in the visual recognition area. The surface of the object OBJ1 in the real space exists between the guide position P2c and the display position P3 of the virtual object. In this case, the guide indicates a direction toward the display position P3 while avoiding the surface of the object OBJ1. For example, the guide is displayed so as to indicate the path indicated by the dashed line in FIG. 13.

[0058] When the MR device 100 is located at position P1d, the display position of the guide is determined to be position P2d. The display position P3 of the virtual object is not included in the visual recognition area. The surface of the object OBJ2 exists between the guide position P2d and the display position P3 of the virtual object. In this case, the guide indicates a direction toward the display position P3 while avoiding the surface of the object OBJ2.

[0059] When the MR device 100 is at position P1c and facing the object OBJ1, the object OBJ2 is not visible to the MR device 100. Therefore, the MR device 100 cannot acquire the surface information of the object OBJ2 by ​​spatial mapping. However, when the surface information of the object OBJ2 is registered in the environmental map, the MR device 100 can calculate the route from position P1c to the display position P3 using the registered surface information. When the surface information of the object OBJ2 is not registered in the environmental map, the MR device 100 moves according to the guide, and the route to the display position P3 is recalculated at the timing when the surface information of the object OBJ2 is acquired.

[0060] Fig. 14 is a flowchart showing a route calculation method. Fig. 15(a) to Fig. 18(b) are schematic diagrams for explaining the route calculation method. If the display position of the guide is not included in the visual recognition area, the processing device 150 executes the calculation method shown in Fig. 14. This calculates a path from the display position of the guide to the display position of the virtual object. The guide is displayed to show that path.

[0061] First, the processing device 150 acquires the display position P of the guide (step S21). The processing device 150 arranges the shortest virtual route connecting the display position P of the guide to the display position of the virtual object (step S22). For example, as shown in FIG. 15(a), a straight virtual route R1 connecting the display position P and the display position P3 is arranged. The processing device 150 determines whether a surface recognized by spatial mapping exists on the arranged virtual route (step S23).

[0062] In the example shown in FIG. 15(a), a surface Su1 of an object OBJ1 exists on a virtual route R1. When it is determined that a surface exists, the processing device 150 identifies a point on the virtual route from the display position P toward the display position of the virtual object just before contacting the surface. For example, as shown in FIG. 15(a), a point p1 is identified. The processing device 150 identifies a portion of the virtual route up to the point where the surface is contacted (step S24). That is, a portion between the display position P and the position p1 is identified. As shown in FIG. 15(b), the processing device 150 leaves only the portion of the virtual route R1 that does not contact the surface, and deletes the portion between the point where the surface is contacted and the display position P3.

[0063] The processing device 150 uses the previous point as a starting point and arranges a plurality of candidates for the via points of the virtual route around the previous point (step S25). For example, as shown in FIG. 16(a), a plurality of candidates c1 to c5 are arranged around the point p1. The processing device 150 adopts one candidate from the plurality of candidates and arranges the virtual route between the adopted point and the display position of the virtual object (step S26). The candidate that does not contact the surface and has the shortest distance to the display position of the virtual object is adopted as the via position. For example, as shown in FIG. 16(b), the candidate c4 is selected and adopted as the via point p2. The processing device 150 arranges the virtual route R2 from the position p2 toward the display position P3 as shown in FIG. 17(a).

[0064] After the virtual route is placed, step S23 is executed again. In the illustrated example, surface Su1 still exists on virtual route R2. Therefore, steps S24 to S27 are executed again. By repeating steps S24 to S27, positions p2 to p5 are calculated as shown in FIG. 17(b). Virtual route R3 from position p5 to display position P3 does not contact surface Su1. Therefore, in step S23 after virtual route R3 is placed, it is determined that no surface exists on the virtual route.

[0065] If it is determined that no surface exists on the virtual route, the processing device 150 assigns "s(k)" to P and assigns "0" to k (step S31). The processing device 150 determines whether or not a detouring virtual route exists between the position P=s(k) and the display position of the virtual object (step S32). If steps S24 to S26 have been executed once or more, it is determined that a detouring virtual route exists. If it is determined that a detouring virtual route exists, the processing device 150 searches for the points used to place the virtual route in the immediately preceding step S26 (step S33). In the example shown in FIG. 17(b), position p5 is found by the search.

[0066] The processing device 150 acquires the discovered position as s(k+1) (step S34). The processing device 150 rearranges a virtual route that connects the position P=s(k) to the discovered position s(k+1) with a straight line (step S35). The processing device 150 assigns k+1 to k (step S36). For example, as shown in FIG. 18(a), the position p5 is acquired as position s(k), k=1. As shown in FIG. 18(b), a virtual route R4 from the position P to the position p5 is arranged. After that, step S32 is executed again with P=s(1) as the reference. If it is determined in step S32 that there is no detour route, the calculation ends.

[0067] The path search calculates the shortest path from the display position of the guide to the display position of the virtual object. The guide is displayed to indicate a direction along that path. This allows the wearer of the MR device 100 to be presented with the shortest path to the virtual object while avoiding objects in the real world.

[0068] 14 is repeatedly executed at a predetermined interval. As a result, the direction indicated by the guide is updated according to the movement of the wearer. If unregistered surface information is obtained after the movement of the MR device 100, the surface information is used to calculate the route.

[0069] 19 and 20 are schematic diagrams for explaining another method of calculating a route. If the structure of the article 200 or the wearer is complex, the above-mentioned calculation method may not be able to find a route. For example, as shown in Fig. 19, if the wearer is surrounded by walls W, a route may be searched for as shown by a dotted circle. In this case, a route that is not the shortest is calculated from the display position P2 of the guide to the display position P3 of the virtual object.

[0070] In this case, the processing device 150 may use an algorithm for searching a path, such as the Dijkstra method or the Laplace potential method. The Dijkstra method is an algorithm for searching the shortest path between two points, and requires a small amount of calculation. The Laplace potential method is also an algorithm for searching a moving path, and enables smooth path generation. Specifically, as shown in FIG. 20, the processing device 150 sets three-dimensional path candidates in a space including the display position P2 of the guide and the display position P3 of the virtual object. For example, virtual grids are set along two mutually orthogonal horizontal directions and a vertical direction, and the intersections of the grids are used as path candidates. The processing device 150 calculates the path from the display position P2 to the display position P3 using the Dijkstra method or the Laplace potential method. For example, when the Dijkstra method is used, a path is set on the intersections of the grid, and as a result, the path shown by the dotted circle in FIG. 20 is found.

[0071] 20, an area r where entry is prohibited may be set within the space. In this case, the processing device 150 searches for a route from display position P2 to display position P3 so as to avoid area r.

[0072] When a route is calculated using the Dijkstra algorithm or the Laplace potential method, two-dimensional route candidates may be simply set in space. In this case, virtual grids are set along two mutually orthogonal horizontal directions, and the intersections of the grids are used as route candidates.

[0073] The calculation method of the path shown in Fig. 14 requires a smaller amount of calculation than the calculation method using the Dijkstra algorithm or the Laplace potential method. Therefore, the calculation method shown in Fig. 14 can be repeatedly executed at shorter intervals. As a result, the display of the guide can be updated at shorter intervals according to the movement of the MR device 100. On the other hand, the calculation method using the Dijkstra algorithm or the Laplace potential method may be able to obtain a shorter and more accurate path than the calculation method shown in Fig. 14, as shown in Figs. 19 and 20. The calculation method used is appropriately selected according to the performance of the processing device 150, the environment around the wearer, and the like.

[0074] An additional object may be set to more appropriately display the guide direction. This additional object functions as a virtual surface. That is, when an additional object exists between the guide and the virtual object, the processing device 150 uses the guide to indicate a direction in which the additional object does not exist.

[0075] 21(a) and 21(b) are schematic diagrams showing an example of output from a mixed reality device according to the embodiment. An article 220 shown in FIG. 21(a) has a fastening point 221. The article 220 has cylindrical members 220a to 220d extending in the vertical direction. Virtual objects 321 and 322 are displayed at the fastening point 221. The members 220a to 220d are separated from one another. In this case, the spaces between the members 220a to 220d are treated as movable areas. Although it is actually not possible to pass through the spaces between the members 220a to 220d, there is a possibility that a virtual route may be set in the spaces.

[0076] For this problem, an additional object 330 is registered as shown in Fig. 21(b). The additional object 330 is set around the members 220a to 220d. The processing device 150 adds the surface information of the additional object 330 to the surface information of the object obtained by spatial mapping. As a result, a path is calculated so as to avoid the additional object 330.

[0077] The processing device 150 may display the additional object 330, or may not display the additional object 330. By displaying the additional object 330, the wearer can understand that the displayed area is impassable. Furthermore, the processing device 150 may issue an alert when the MR device 100 comes into contact with the additional object 330. As the alert, a message may be displayed, or a sound or light may be output. This notifies the wearer that the area is impassable.

[0078] Fig. 22 is a flowchart showing processing by a mixed reality device according to an embodiment. Fig. 23 is a schematic diagram for explaining the processing of the flowchart shown in Fig. 22. Fig. 24(a) and Fig. 24(b) are schematic diagrams showing an example of output by a mixed reality device according to an embodiment. The MR device 100 may execute the process of the flowchart shown in Fig. 22. For example, the process shown in Fig. 22 is executed in parallel with the processes of steps S4 and S5 after step S3 shown in Fig. 10.

[0079] First, the processing device 150 obtains data on the location where work will be performed (step S41). The data on the location where work will be performed is obtained, for example, from the fastening location master data 170d shown in Fig. 10. The processing device 150 calculates a workable area in which work is permitted from the work location (step S42). For example, a range up to a preset distance in a preset direction based on the work location is calculated as the workable area.

[0080] The processing device 150 acquires the position of the MR device 100 and controls the display of the guide (step S43). In step S43, the process shown in Fig. 11 is executed. For example, if the display position of a virtual object to be displayed at the next work location is outside the visual recognition area, a route from the position of the MR device 100 to the display position of the virtual object is searched for. A guide showing the route is displayed.

[0081] The processing device 150 determines whether the work location is within the workable area (step S44). If it is determined that the work location is not within the workable area, the processing device 150 outputs an alert (step S45). As the alert, a message may be displayed, or a sound or light may be output. After the alert is output, step S43 is executed again.

[0082] If it is determined in step S44 that the work location is within the workable area, the processor 150 displays a message indicating that work is permitted (step S45).

[0083] 23 , for example, work is performed on an article 230. The article 230 has fastening points 231 and 232. The worker uses a lifter 240 to sequentially tighten screws into the fastening points 231 and 232. A virtual object 341 is displayed at the fastening point 231, and a virtual object 342 is displayed at the fastening point 232.

[0084] After the worker tightens the screw into the fastening point 231, the processing device 150 acquires the position of the fastening point 232. The processing device 150 calculates a workable area r1 based on the position of the fastening point 232. In the illustrated example, the position of the MR device 100 is not included in the workable area r1. Therefore, the processing device 150 issues an alert.

[0085] When the work shown in Fig. 23 is performed, for example, the work environment master data 170f shown in Fig. 22 is referenced. The work environment master data 170f stores the shape of the surroundings of the item 230, the shape of the lifter 240, the location where the work is performed, and display contents such as guides and alerts. The processing device 150 calculates the workable area r1 using the data registered in the work environment master data 170f. The processing device 150 also displays alerts and work permissions using the data registered in the work environment master data 170f.

[0086] As an example, after work on fastening point 231 is completed, alert 351 and message 352 are displayed as shown in Fig. 24(a). Alert 351 instructs the worker to move with lifter 240 to work on fastening point 232. Message 352 indicates the distance to be moved by lifter 240. Guide 350 indicates the direction to move by the lifter.

[0087] 24(b), when the wearer of the MR device 100 moves close to the fastening point 232, a message 353 is displayed. The message 353 indicates that it is OK to carry out the work.

[0088] Here, an example has been described in which the embodiment of the present invention is applied mainly to the task of tightening a screw. The embodiment of the present invention may also be applied to the task of loosening a screw. When loosening a screw, a tool is used to turn the screw, as shown in Fig. 5. In this case as well, the task can be performed efficiently by displaying a virtual object.

[0089] The embodiment of the present invention may be applied to tasks other than turning a screw. For example, the task may be assembling an item, dismantling an item, or transporting an item. In any task, the task can be performed efficiently by displaying a virtual object. In addition, when the item being worked on moves, the display position of the virtual object is changed according to the amount of movement, or estimation of the location being worked on is stopped, thereby improving the convenience of the MR device.

[0090] FIG. 25 is a schematic diagram showing a hardware configuration. 25 is used as the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0091] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0092] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0093] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) or the like.

[0094] The input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95.

[0095] An output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to the output device via the output I / F 96 and cause the output device to output information.

[0096] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0097] Data processing by each of the processing devices 150 may be executed by only one computer 90. A part of the data processing may be executed by another computer or a server via the communication I / F 97.

[0098] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0099] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0100] Embodiments of the invention include the following features. (Feature 1) A mixed reality device capable of displaying a virtual space overlaid on a real space, A base point of the virtual space is set using a predetermined object photographed in the real space; Obtaining a display position of a virtual object in a three-dimensional coordinate system based on the base point; A mixed reality device that determines whether the display position is within a viewing area set in front of the mixed reality device, and if the display position is not within the viewing area, displays a guide indicating the direction of the display position. (Feature 2) Recognize the surfaces of surrounding objects, 2. The mixed reality device of feature 1, wherein the visibility area is set between the mixed reality device and the recognized surface. (Feature 3) The mixed reality device of feature 2, wherein, when the surface is between the guide and the display position, the guide indicates a direction in which the surface is not present. (Feature 4) reading an additional object set at a predetermined position in the virtual space; A mixed reality device according to feature 2 or 3, wherein, when the additional object is present between the guide and the display position, the guide indicates a direction in which the additional object is not present. (Feature 5) If the surface is between the guide and the display position, searching for a path from the guide to the display position; 5. The mixed reality device according to any one of features 2 to 4, wherein the guide indicates a direction along a searched route. (Feature 6) 6. The mixed reality device according to any one of features 1 to 5, wherein the virtual object is displayed corresponding to a fastening location where a screw is turned. (Feature 7) The mixed reality device of feature 6, wherein when a hand or a tool comes into contact with the virtual object, it is estimated that work is being performed on the fastening point corresponding to the virtual object. (Feature 8) The mixed reality device described in feature 7, wherein when data is received from a tool after a hand or a tool comes into contact with the virtual object, the data of the corresponding fastening point is linked to the data received from the tool and stored. (Feature 9) A mixed reality device that can overlay virtual space on real space. setting a base point of the virtual space using a predetermined object photographed in the real space; acquiring a display position of a virtual object in a three-dimensional coordinate system based on the base point; determining whether the display position is present within a visual recognition area set in front of the mixed reality device, and if the display position is not present within the visual recognition area, displaying a guide indicating a direction of the display position; Processing methods. (Feature 10) A program for causing a mixed reality device to execute the processing method according to feature 9. (Feature 11) A storage medium storing the program according to feature 10.

[0101] According to the embodiment described above, a mixed reality device having better usability is provided. Also, a processing method, a program, and a storage medium capable of further improving the usability of the mixed reality device are provided.

[0102] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0103] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0104] 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: MR device, 101: Frame, 111, 112: Lens, 121, 122: Projection device, 131: Image camera, 132: Depth camera, 133: Light source, 134: Eye tracking camera, 140: Sensor, 141: Microphone, 150: Processing device, 160: Battery, 170: Storage device, 170a: Work master data, 170b: Base point master data, 170c: Tool master data, 170d: Fastening point master data, 170e: History data, 170f: Work environment master data, 200: Item, 201-208: Fastening point, 210: marker, 220: item, 220a to 220d: parts, 221: fastening point, 230: item, 231, 232: fastening point, 240: lifter, 251: left hand, 252: right hand, 261, 262: virtual object, 280: wrench, 290: extension bar, 301 to 305, 311 to 315, 321: virtual object, 330: additional object, 341, 342: virtual object, 350: guide, 351: alert, 352, 353: message, 400: shelf, c1 to c5: candidate, D1: reference direction, M: processing method, M1: display method, OBJ1, OBJ2: object, P: display position, P1: reference position, P2, P3: display position, p1~p5: position, R: visible area, R1~R4: virtual path, r: area, r1: workable area, Su,Su1: surface, V: field of view, V1,V2: vector, W: wall

Claims

1. A mixed reality device capable of displaying a virtual space overlaid on a real space, A base point of the virtual space is set using a predetermined object photographed in the real space; Obtaining a display position of a virtual object in a three-dimensional coordinate system based on the base point; A mixed reality device that determines whether the display position is within a viewing area set in front of the mixed reality device, and if the display position is not within the viewing area, displays a guide indicating the direction of the display position.

2. Recognize the surfaces of surrounding objects, The mixed reality device of claim 1 , wherein the visibility region is set between the mixed reality device and the recognized surface.

3. The mixed reality device of claim 2 , wherein, when the surface exists between the guide and the display position, the guide indicates a direction in which the surface does not exist.

4. reading an additional object set at a predetermined position in the virtual space; The mixed reality device according to claim 2 , wherein, when the additional object is present between the guide and the display position, the guide indicates a direction in which the additional object is not present.

5. If the surface is between the guide and the display position, searching for a path from the guide to the display position; The mixed reality device of claim 2 , wherein the guide indicates directions along a explored path.

6. The mixed reality device according to claim 1 , wherein the virtual object is displayed corresponding to a fastening point where a screw is turned.

7. The mixed reality device according to claim 6 , wherein when a hand or a tool comes into contact with the virtual object, it is estimated that work is being performed on the fastening point corresponding to the virtual object.

8. The mixed reality device according to claim 7, wherein when data is received from a tool after a hand or a tool comes into contact with the virtual object, the data of the corresponding fastening point and the data received from the tool are linked and stored.

9. A mixed reality device that can overlay virtual space on real space. setting a base point of the virtual space using a predetermined object photographed in the real space; acquiring a display position of a virtual object in a three-dimensional coordinate system based on the base point; determining whether the display position is present within a visual recognition area set in front of the mixed reality device, and if the display position is not present within the visual recognition area, displaying a guide indicating a direction of the display position; Processing methods.

10. A program for causing a mixed reality device to execute the processing method according to claim 9.

11. A storage medium storing the program according to claim 10.

Images