Mixed reality device, processing method, program, and storage medium

Abstract

To provide a mixed reality device, a processing method, a program, and a storage medium with which the safety of a wearer can be further improved.SOLUTION: A mixed reality device is configured to superimpose a virtual space on a real space. The mixed reality device measures first coordinates of the mixed reality device and calculates a first region based on the first coordinates. The mixed reality device detects an object present around the mixed reality device in the real space and calculates a second region based on a second coordinate of the object. The mixed reality device outputs an alert when the first region and the second region overlap each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] SUMMARY OF THE PRESENT EMBODIMENTS An embodiment of the present invention relates to a mixed reality device, a processing method, a program, and a storage medium. [Background technology]

[0002] Conventionally, mixed reality devices have been used for efficient work. The mixed reality device can display a virtual space overlaid on a real space and provide various information to a worker. The worker can perform the work more efficiently by referring to the information displayed by the mixed reality device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a mixed reality device, a processing method, a program, and a storage medium that can further improve the safety of the wearer. [Means for solving the problem]

[0005] A mixed reality device according to an embodiment is capable of displaying a virtual space superimposed on a real space. The mixed reality device measures a first coordinate of the mixed reality device and calculates a first area based on the first coordinate. The mixed reality device detects an object present in the real space around the mixed reality device and calculates a second area based on second coordinate of the object. The mixed reality device outputs an alert when the first area and the second area overlap. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating a work site. [Diagram 3] FIG. 3 is a schematic diagram illustrating a work site. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Diagram 5] FIG. 5 is a flowchart showing processing by the mixed reality device according to the embodiment. [Figure 6] 6(a) is a flow chart showing a method for calculating the first region, and FIG 6(b) is a flow chart showing a method for calculating the second region. [Figure 7] 7(a) and 7(b) are schematic diagrams for explaining processing by the mixed reality device according to the embodiment. [Figure 8] 8(a) to 8(d) are schematic diagrams for explaining processing by the mixed reality device according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Figure 10] 10(a) to 10(c) are schematic diagrams showing a calculation method for the first region. [Figure 11] FIG. 11 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Figure 12] FIG. 12 is a schematic diagram illustrating the state of work. [Figure 13] FIG. 13 is a schematic diagram showing an example of a tool. [Figure 14] FIG. 14 is a flowchart illustrating a processing method according to an embodiment. [Figure 15] FIG. 15 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be shown differently depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram illustrating a mixed reality device according to an embodiment. The embodiment of the present invention relates to a mixed reality device (MR device). For example, as shown in Fig. 1, the MR device 100 according to the embodiment includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0009] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0010] The projection device 121 and the projection device 122 display the recognition result of the body of the worker, a virtual object, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0011] The lenses 111 and 112 are optically transparent. The worker can visually recognize the real state through the lenses 111 and 112. The worker can also visually recognize information projected onto the lenses 111 and 112 by the projection devices 121 and 122. The information is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0012] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0013] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0014] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0015] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0016] 2 and 3 are schematic diagrams illustrating a work site. In the example shown in Fig. 2, workers W1 to W5 are working. Workers W1 and W2 are working on an article 200. Workers W3 and W4 are transporting the article using a transport device. Worker W5 is working on a robot. For example, workers W1 and W2 are wearing MR devices 100 and tightening screws into the article 200.

[0017] 3, the article 200 is a cylindrical member and has fastening points 201 to 208. Workers W1 and W2 use a wrench and an extension bar to fasten screws into the fastening points 201 to 208, respectively.

[0018] A marker 210 is provided near the work target. The marker 210 is an AR marker. As described below, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 150 sets a three-dimensional coordinate system based on a plurality of points indicated by a hand gesture.

[0019] At the start of the task, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a three-dimensional coordinate system based on the position and orientation of the marker 210.

[0020] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. During work, the image camera 131 and the depth camera 132 capture images of the article 200, the worker's left hand 251, and the worker's right hand 252. The processing device 150 recognizes the left hand 251 and the right hand 252 from the captured images by hand tracking. The processing device 150 may cause the projection device 121 and the projection device 122 to display the recognition results on the lens 111 and the lens 112. Hereinafter, the processing device causing the projection device to display information on the lens is also simply referred to as "the processing device displays information."

[0021] 4, for example, the processing device 150 displays the recognition results of the left hand 251 and the right hand 252 superimposed on the hands in the real space. In the illustrated example, a plurality of virtual objects 261 and a plurality of virtual virtual objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The plurality of virtual objects 261 respectively indicate a plurality of joints of the left hand 251. The plurality of virtual objects 262 respectively indicate a plurality of joints of the right hand 252. Instead of the joints, virtual objects (meshes) respectively indicating the surface shapes of the left hand 251 and the right hand 252 may be displayed.

[0022] When the processing device 150 recognizes the left hand 251 and the right hand 252, it measures the coordinates of each hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The coordinates of any of these joints are used as the coordinates of the hand. The center of gravity of the multiple joints may be used as the coordinates of the hand. Alternatively, the center coordinates of the entire hand may be used as the coordinates of the hand.

[0023] FIG. 5 is a flowchart showing processing by the mixed reality device according to the embodiment. In the processing method M1 shown in FIG. 5, the processing device 150 receives designation of an MR device to be used in a task, a worker who performs the task, the task to be performed, a tool to be used, and the like (step S1). The designation may be input by the worker, or may be input by a higher-level computer or the like. The worker, tool, and the like may be automatically designated according to a schedule created in advance. Information on the MR device, worker, task, tool, and the like may be linked to one another. For example, the MR device is linked in advance to the worker. The task is linked in advance to the worker and the tool. In this case, it is sufficient to designate any one of the task, worker, or MR device.

[0024] The processing device 150 reads the task data 171 (step S2) and acquires data related to the specified target. The task data 171 includes the ID of the MR device, the ID of the worker, the worker's skill level, the task ID, the tool ID, and the like. At the start of the task, the image camera 131 captures the marker 210 (step S3). The processing device 150 recognizes the marker 210 from the image and sets a three-dimensional coordinate system with the marker 210 as the base point (step S4). The processing device 150 performs spatial mapping and acquires a spatial map (step S5). In the spatial map, the surfaces of objects around the MR device 100 are represented by meshes. The spatial map includes the coordinates of each mesh.

[0025] The spatial mapping is repeatedly performed during the operation. After at least one spatial mapping is performed, the processing device 150 performs a calculation of the first region (step S10) and a calculation of the second region (step S20).

[0026] Fig. 6(a) is a flowchart showing a method for calculating a first region. Fig. 6(b) is a flowchart showing a method for calculating a second region. Fig. 7(a) and Fig. 7(b) are schematic diagrams for explaining processing by a mixed reality device according to an embodiment. As shown in FIG. 6(a), in step S10, the processing device 150 acquires the coordinates (first coordinates) of the MR device 100 (step S11). As shown in FIG. 7(a), the processing device 150 calculates a first region R1 based on the coordinates of the MR device 100 (step S12). In the illustrated example, the first region R1 is spherical with the coordinates of the MR device 100 as the center. The diameter of the sphere is set in advance and is determined with safety in mind. Not limited to this example, the first region R1 may be cylindrical. In that case, the central axis of the cylinder passes through the coordinates of the MR device 100 and is set parallel to the vertical direction. The processing device 150 stores the calculated first region in memory (step S13).

[0027] As shown in FIG. 6(b), in step S20, the processing device 150 reads object data 172 (step S21). The object data 172 includes information on physical objects that may exist in real space. For example, information on production equipment, transport equipment, etc. is registered in the object data 172. The production equipment is a work processing device, an industrial robot, etc. The transport equipment is an AGV, a forklift, a crane, etc. The object data 172 registers the ID of each object, the coordinates where the object is placed, the movable area of ​​the object, etc. The coordinates of a person wearing another MR device 100 may be registered in the object data 172 by the other MR device 100.

[0028] The processing device 150 recognizes the surfaces of surrounding objects in real space based on the information obtained by the depth camera 132, and acquires a space map (step S22). For example, the surfaces of surrounding facilities, equipment, people, etc. are recognized. The processing device 150 detects individual objects existing around the MR device 100 from the space map (step S23). For example, when the distance between surfaces is longer than a preset threshold in the depth direction, the processing device 150 determines that those surfaces are surfaces of separate objects.

[0029] The processing device 150 determines whether the detected object is registered in the object data 172 read in step S21 (step S24). For example, the object coordinates and surface information (surface coordinates) of the object are registered in the object data 172. The processing device 150 compares the surface information of the detected object with the surface information of the registered object. For example, the processing device 150 calculates the position error between the surface coordinates of the detected object and the surface coordinates of the registered object. The processing device 150 also calculates the error between the surface size of the detected object and the surface size of the registered object. If the position error and size error are each less than a preset threshold, the processing device 150 determines that the detected object is a registered object.

[0030] If the detected object is registered in the object data 172, the processing device 150 sets the detected object among the loaded objects as "valid" (step S25). As shown in FIG. 7(b), the processing device 150 calculates a second region R2 based on the coordinates (second coordinates) of the detected object (step S26). At this time, the second region is calculated so as to include the movable region registered in the object data 172. For example, the second region is calculated by adding a safety region around the movable region.

[0031] If the detected object is not registered in the object data 172, the processing device 150 newly registers the object in the object data 172 as an unregistered object (step S27). The processing device 150 calculates a second region based on the coordinates of the detected object (step S28). When the second region is calculated in step S26 or S28, the processing device 150 stores the calculated second region in memory (step S29).

[0032] The processing device 150 executes the determination in step S24 for each object detected in step S23, thereby calculating a second region for each detected object.

[0033] The size of the second region for the unregistered object may be registered in advance or may be calculated according to the size of the detected object. For example, the larger the size of the detected object, the larger the size of the second region is calculated to be. Also, the second region R2 for the unregistered object may be spherical or cylindrical with the coordinates of the object as the center. The central axis of the cylinder passes through the central coordinates of the unregistered object and is set parallel to the vertical direction.

[0034] As shown in Fig. 5, after steps S10 and S20, the processing device 150 retrieves the calculated first and second regions from memory (step S31). The processing device 150 determines whether there is an overlapping region between the first and second regions (step S32). The processing device 150 outputs a notification according to the determination result (step S33). For example, the determination result is notified to the worker by display, sound, or vibration. The sound may be a notification sound or the like, or may be a voice message.

[0035] When the first area and the second area overlap, the processing device 150 determines that there is a possibility that the worker may come into contact with a surrounding object. The processing device 150 outputs an alert notifying the worker of the danger. The alert enables the worker to recognize that there is an object around the worker that the worker may come into contact with.

[0036] If the first and second regions do not overlap, the processing device 150 determines that there is little or no possibility of contact between the worker and the object. In this case, the processing device 150 outputs a notification that there is no danger. Alternatively, the processing device 150 may not output a notification so that the notification does not impede the work.

[0037] The size of the first region and the size of the second region are arbitrary, but it is preferable to set these sizes to values ​​with a sufficient margin in consideration of safety.

[0038] Steps S5, S10, S20, and S31 to S33 are repeated at predetermined intervals. That is, spatial mapping, calculation of the first area, calculation of the second area, and determination of overlap of each area are repeated. This reduces the possibility that the worker will come into contact with surrounding objects even if the positions of the objects around the worker change over time.

[0039] Advantages of the embodiment will be described. At a work site, various things may exist around a worker. For example, as shown in FIG. 2, there are production equipment, transport equipment, and the like around the worker. If the worker comes into contact with the production equipment or transport equipment, the worker may be injured. In particular, when a worker wears an MR device, the field of view is narrower than when the worker does not wear an MR device. In addition, the MR device displays various information to improve workability or safety. When information overlaps with an object in the real space, the object in the real space becomes difficult to see. Therefore, the worker may not notice the approach of an object in the real space or may notice the object late. Therefore, there is an increased risk that the worker may come into contact with an object such as a production equipment or transport equipment without noticing it.

[0040] In an embodiment of the present invention, the MR device 100 measures its own first coordinates and calculates a first region based on the first coordinates. The MR device 100 can also recognize objects existing in the real space and acquire a space map. The MR device 100 calculates a second region based on second coordinates of the recognized object. If the first region and the second region overlap, there is a possibility that the worker wearing the MR device 100 may come into contact with surrounding objects. In that case, the MR device 100 outputs an alert.

[0041] By outputting the alert, the worker can be notified that there is a possibility of contact with a surrounding object. The alert allows the worker to recognize that there is an object around the worker that may be contacted. According to the embodiment of the present invention, it is possible to further improve the safety of the wearer of the MR device 100.

[0042] 8(a) to 8(d) are schematic diagrams for explaining processing by the mixed reality device according to the embodiment. When the first and second regions overlap, the processing device 150 may calculate the direction in which they may come into contact. For example, as shown in FIG. 8(a), the processing device 150 sets a first surface Su1 perpendicular to the front-rear direction (first direction), a second surface Su2 perpendicular to the left-right direction (second direction), and a third surface Su3 perpendicular to the up-down direction (third direction). Each direction is determined based on the orientation of the MR device 100. The processing device 150 projects the first and second regions onto the first to third surfaces Su1 to Su3, respectively. FIG. 8(b) to FIG. 8(d) show the results of projecting the first and second regions onto the first surface Su1, the second surface Su2, and the third surface Su3, respectively.

[0043] By projecting each region onto the first surface Su1, the amount of overlap between the first region R1 and the second region R2 in the front-to-back direction can be calculated. By projecting each region onto the second surface Su2, the amount of overlap between the first region R1 and the second region R2 in the left-to-right direction can be calculated. By projecting each region onto the third surface Su3, the amount of overlap between the first region R1 and the second region R2 in the up-to-down direction can be calculated. The processing device 150 calculates the amount of overlap in each direction and compares these amounts of overlap. It is determined that the object exists in the direction with the largest amount of overlap as viewed from the MR device 100.

[0044] In the object data 172, a danger level may be registered for each object. For example, the larger the object is or the faster the object moves, the higher the danger level is set. When it is determined that the first area and the second area overlap, the processing device 150 refers to the danger level of the object corresponding to the second area. The processing device 150 changes the notification according to the danger level. The higher the danger level, the more dangerous the notification is output so that the worker can recognize that it is more dangerous. For example, the processing device 150 may display an alert indicating the danger level. The processing device 150 may output a louder sound or a larger vibration as the danger level increases.

[0045] If the object is unregistered, the risk level may be set to the maximum. The risk level may be calculated according to the size or movement of the object. The processing device 150 may change the size of the second area according to the risk level. The higher the risk level of the object, the greater the risk if the worker comes into contact with the object. The processing device 150 calculates the size of the second area to be larger the higher the risk level. This can further reduce the risk of contact between the worker and the object.

[0046] FIG. 9 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. When the first area and the second area overlap, the processing device 150 displays an alert 270 as shown in Fig. 9. In the illustrated example, the alert 270 indicates the possibility of contact with an object, the direction of the object, the name of the object, and the degree of danger of the object. The possibility of contact with the object is based on the determination result of the overlap between the first area and the second area. The direction of the object is based on the calculation result of the direction in which the first area and the second area overlap. The name of the object and the degree of danger of the object are registered in the object data 172. Instead of or in addition to displaying the alert 270, this information may be output by voice.

[0047] When calculating the first area, the processing device 150 may change the size of the first area according to the worker's skill level. "Skill level" indicates the degree to which the worker is familiar with the work. The higher the skill level, the more familiar the worker is with the work. A highly skilled worker is less likely to come into contact with an object compared to a less skilled worker. Therefore, the processing device 150 calculates the first area to be smaller the higher the worker's skill level. As a result, the frequency of outputting a notification informing the worker of the possibility of contact between the worker and an object decreases. A highly skilled worker can more easily concentrate on the work.

[0048] 10(a) to 10(c) are schematic diagrams showing a calculation method for the first region. 10(a), the processing device 150 may calculate the first region R1 so that the tool held by the worker is included in the first region R1. For example, the processing device 150 calculates the region covering the tool from the data of the tool registered in the work data 171.

[0049] Specifically, first, the processing device 150 recognizes the tool from the image. For example, a marker is attached to the tool, and the processing device 150 recognizes the marker. An image of the tool (template image) may be prepared in advance. The processing device 150 recognizes the image of the tool from the entire image by template matching.

[0050] When the tool is recognized, as shown in Fig. 10(b), the processing device 150 calculates the position p of the tool. The processing device 150 sets a tool area r including an area corresponding to the size of the tool and an area surrounding the tool, based on the position p. The size of the tool and the size of the surrounding area are registered in advance in the work data 171.

[0051] 10(c), the tool region r may be set taking into consideration the range in which the tool is moved. The processing device 150 calculates the first region by adding the tool region calculated by any method to the region calculated based on the coordinates of the MR device 100.

[0052] When a large tool is used, the tool may come into contact with other objects. If the tool comes into contact with other objects, the worker holding the tool may be injured or other objects may be damaged. The first area is calculated to cover the tool, reducing the possibility of contact between the tool and other objects and improving safety.

[0053] In addition to the above-mentioned information regarding safety, the MR device 100 can also output information for assisting in the work. In the following, a specific example in which the MR device 100 is used for the work of turning a screw will be described.

[0054] FIG. 11 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. The processing device 150 displays virtual objects 301-305 and virtual objects 311-315 as shown in Fig. 11. The virtual objects 311-315 are displayed between the fastening points 201-205 and the virtual objects 301-305, respectively. The virtual objects 311-315 indicate to which fastening points the virtual objects 301-305 correspond, respectively.

[0055] FIG. 12 is a schematic diagram illustrating the state of work. For example, as shown in FIG. 12 , a wrench 280 and an extension bar 290 are used to turn a screw into a fastening point. As an example, when fastening a screw into fastening point 203, an operator places a screw into a threaded hole of fastening point 203. The operator fits one end of extension bar 290 onto the screw. The operator fits the head of wrench 280 into the other end of extension bar 290. The operator holds the head of wrench 280 with one hand and holds the grip of wrench 280 with the other hand. By turning wrench 280, the screw is tightened into fastening point 203 via extension bar 290.

[0056] At this time, the worker positions the extension bar 290 so that the extension bar 290 is close to or in contact with the virtual object 313. The worker also grips the head of the wrench 280 so that his or her hand comes into contact with the virtual object 303. By displaying the virtual object, the worker can easily grasp the positions of the tool and hand when turning the screw into the fastening point 203. This can improve the work efficiency.

[0057] In the illustrated example, the virtual objects 301 to 305 are spherical, and the virtual objects 311 to 315 are rod-shaped. As long as the worker can visually recognize each virtual object, the shape of each virtual object is not limited to this example. For example, the virtual objects 301 to 305 may be cubes, and the virtual objects 311 to 315 may be linear. Also, virtual objects similar to the virtual objects 301 to 305 and the virtual objects 311 to 315 are displayed for the fastening points 206 to 208 not shown in FIG. 11 .

[0058] After the virtual objects are displayed, the processing device 150 may determine whether a predetermined object has come into contact with the virtual objects 301-305. For example, the processing device 150 determines whether a hand has come into contact with a virtual object. Specifically, the processing device 150 calculates the distance between the coordinates of the hand and each of the virtual objects 301-305. If any of the distances is less than a preset threshold, the processing device 150 determines that the hand has come into contact with that virtual object. As an example, in FIG. 4, the diameter of the virtual objects 301-305 (spheres) corresponds to the threshold. The spheres indicate the range within which it is determined that the hand has come into contact with the virtual object.

[0059] FIG. 13 is a schematic diagram showing an example of a tool. The processing device 150 may determine whether the tool has come into contact with the virtual objects 301 to 305. For example, as shown in FIG. 13, a plurality of markers 281 are attached to the wrench 280. The processing device 150 recognizes the plurality of markers 281 from an image captured by the image camera 131. The processing device 150 measures the coordinates of each of the markers 281. A positional relationship between the plurality of markers 281 and the head 282 of the wrench 280 is registered in advance. The processing device 150 calculates the coordinates of the head 282 based on the coordinates of at least three recognized markers 281 and the positional relationship registered in advance. The processing device 150 calculates the distance between the coordinates of the head 282 and each of the virtual objects 301 to 305. If any of the distances is less than a preset threshold, the processing device 150 determines that the wrench 280 has come into contact with that virtual object.

[0060] When a hand or a tool comes into contact with one of the virtual objects 301 to 305, it can be estimated that a screw has been turned into a fastening point corresponding to the one of the virtual objects 301 to 305. In the example shown in FIG. 12, it is estimated that a screw has been turned into a fastening point 203 corresponding to the virtual object 303 due to contact between the hand and the virtual object 303. Hereinafter, a fastening point that is estimated to be working on based on contact between a predetermined object and a virtual object among one or more fastening points is referred to as a "work point." When a work point is estimated, the processing device 150 may record that a screw has been turned into the work point. This allows a work record to be automatically created.

[0061] The tool used may be a digital tool capable of detecting torque. In that case, the processing device 150 receives the detected torque from the tool. The torque required for fastening may be set in advance, and the tool may refer to the required torque. The tool determines whether the required torque has been detected, and transmits the determination result to the processing device 150. In addition, the tool can transmit the rotation angle, the time when the torque was detected, and the like to the processing device 150. The processing device 150 links the data received from the tool with data related to the work location. This allows a more detailed work record to be automatically created.

[0062] FIG. 14 is a flowchart illustrating a processing method according to an embodiment. In processing method M2 shown in Fig. 14, the processing device 150 receives designation of the MR device to be used in the work, the worker who will perform the work, the work to be performed, the tools to be used, etc. (step S1). Upon receiving the designation, the processing device 150 reads the work data 171 and fastening point data 173 (step S2). The fastening point data 173 stores the method of identifying the base point, the ID of each fastening point, the position of each fastening point, the type of tool to be used, the angle of the extension bar, the number of fastenings required, the required torque, the color of the mark, and the ID of each virtual object. The processing device 150 obtains data related to the work to be performed from the fastening point data 173.

[0063] Methods for identifying the base point include a method using a marker or a method using hand gestures. The ID of the fastening point is a unique string for identifying each fastening point. The coordinates of a three-dimensional coordinate system based on the base point are registered as the position of the fastening point. The tool type indicates the classification of the tool based on the structure, external shape, performance, etc. For example, the length of the extension bar is specified from the extension bar type. The angle indicates the limit value of the angle at which the extension bar can engage with the screw when turning the screw into the fastening point.

[0064] In work, a mark may be made when a screw is turned into a fastening point. "Mark color" indicates the color of the mark made at each fastening point. When marks of different colors are made depending on the number of times a screw is turned, the color of the mark for each number of times is registered. The virtual object ID is a character string for identifying data of a virtual object registered in advance, and a virtual object ID is linked to each fastening point.

[0065] After step S2, the processing device 150 executes steps S3 to S5 in the same manner as the processing method M1 shown in Fig. 5. Thereafter, the processing device 150 displays a virtual object corresponding to the fastening location (step S41). The processing device 150 determines whether a predetermined physical object has come into contact with the virtual object (step S42). If it is determined that the predetermined object has come into contact with the virtual object, the work location is estimated (step S43). That is, the processing device 150 estimates that a screw is being turned into the fastening location corresponding to the virtual object. The processing device 150 saves a record of the work performed on the work location (step S44).

[0066] In step S44, the processing device 150 associates the torque detected by the tool with the ID of the fastening point where it is estimated that the screw is being turned, and stores it in the history data 174. As shown in FIG. 14, the processing device 150 may further associate the type and ID of the tool used, the number of times the screw is fastened, and the mark recognition result with the ID of the fastening point. The mark is recognized by the processing device 150 from an image captured by the image camera 131. The processing device 150 extracts a block of pixels of the mark color from the image, and counts the number of pixels in that block. If the number of pixels exceeds a preset threshold, it is determined that a mark has been attached.

[0067] In parallel with steps S41 to S44, the processing device 150 executes steps S10, S20, and S31 to S33 shown in Fig. 5. This makes it easier for the worker to recognize the possibility of contact with surrounding objects even when the worker is performing a task while gazing at a virtual object.

[0068] The task data 171, the object data 172, and the fastening location data 173 are stored in the storage device 170 of the MR device 100. Alternatively, the task data 171, the object data 172, and the fastening location data 173 may be stored in a storage area other than the MR device 100. In this case, the processing device 150 accesses the task data 171, the object data 172, and the fastening location data 173 via wireless communication or a network.

[0069] Here, an example in which a screw is tightened at a fastening point has been mainly described. The embodiments of the present invention are applicable not only to cases in which a screw is tightened at a fastening point, but also to cases in which a screw at a fastening point is loosened. For example, a screw at a fastening point is loosened when performing maintenance, inspection, or repair of a product. For example, according to the embodiments of the present invention, when a screw is loosened, the location where the screw is being turned is estimated, and a work record related to that work location is automatically created.

[0070] FIG. 15 is a schematic diagram showing a hardware configuration. 15 is used as the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0071] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0072] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0073] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) etc. The storage device 94 may be used as the storage device 170.

[0074] The input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95. The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to an output device (e.g., projection devices 121 and 122) via the output I / F 96 and cause information to be displayed on the output device.

[0075] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0076] Data processing by the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0077] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0078] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0079] Embodiments of the invention include the following features. (Feature 1) A mixed reality device capable of displaying a virtual space overlaid on a real space, measuring a first coordinate of the mixed reality device and calculating a first area based on the first coordinate; Detecting an object present in a real space around the mixed reality device, and calculating a second area based on second coordinates of the object; The mixed reality device outputs an alert when the first region and the second region overlap. (Feature 2) obtaining a spatial map of an area surrounding the mixed reality device; 2. The mixed reality device of claim 1, further comprising: (Feature 3) Read the data of the tools used in the work, 3. The mixed reality device of feature 1 or 2, wherein the first region is calculated to cover the tool. (Feature 4) setting a first direction, a second direction, and a third direction perpendicular to each other with respect to the mixed reality device; The mixed reality device according to any one of features 1 to 3, wherein an overlap between the first region and the second region is detected in each of the first direction, the second direction, and the third direction. (Feature 5) The mixed reality device according to feature 4, further comprising: when an overlap between the first region and the second region is detected, outputting information indicating the direction of the overlap. (Feature 6) A three-dimensional coordinate system is set using a preset base point; The mixed reality device according to any one of features 1 to 5, wherein the first coordinate and the second coordinate in the three-dimensional coordinate system are repeatedly measured. (Feature 7) Recognizing a marker from an image including the marker provided in a real space; 7. The mixed reality device of feature 6, wherein the three-dimensional coordinate system is set using the marker as the base point. (Feature 8) The mixed reality device according to feature 6 or 7, which displays a virtual object based on the coordinates of a pre-registered fastening point in the three-dimensional coordinate system. (Feature 9) The mixed reality device of feature 8, wherein when a predetermined object comes into contact with the virtual object, it is estimated that a screw is being turned into the fastening point. (Feature 10) A mixed reality device that can display virtual space over real space measuring a first coordinate of the mixed reality device and calculating a first area based on the first coordinate; Detecting an object present in a real space around the mixed reality device, and calculating a second area based on second coordinates of the object; A processing method, comprising: outputting an alert when the first area and the second area overlap. (Feature 11) A mixed reality device that can display virtual space over real space. Measuring a first coordinate of the mixed reality device and calculating a first area based on the first coordinate; detecting an object present in a real space around the mixed reality device, and calculating a second area based on second coordinates of the object; outputting an alert when the first area and the second area overlap; program. (Feature 12) A storage medium storing the program according to feature 11.

[0080] According to the embodiments described above, a mixed reality device, a processing method, a program, and a storage medium are provided that can further improve the safety of the wearer.

[0081] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0082] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0083] 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: MR device, 101: Frame, 111, 112: Lens, 121, 122: Projection device, 131: Image camera, 132: Depth camera, 140: Sensor, 141: Microphone, 150: Processing device, 160: Battery, 170: Storage device, 171: Work data, 172: Object data, 173: Fastening point data, 174: History data, 200: Item, 201-208: Fastening point, 210: Marker, 251: Left hand, 252: Right hand, 261, 262: Virtual object, 270: Alert, 280: wrench, 281: marker, 282: head, 290: extension bar, 301 to 305, 311 to 315: virtual objects, M1, M2: processing method, R1: first area, R2: second area, Su1: first surface, Su2: second surface, Su3: third surface, W1 to W5: workers

Claims

1. A mixed reality device capable of displaying a virtual space overlaid on a real space, measuring a first coordinate of the mixed reality device and calculating a first area based on the first coordinate; detecting an object present in a real space around the mixed reality device, and calculating a second area based on second coordinates of the object; A mixed reality device that outputs an alert when the first region and the second region overlap.

2. obtaining a spatial map of an area surrounding the mixed reality device; The mixed reality device of claim 1 , further comprising: a spatial map for detecting the object;

3. Read the data of the tools used in the work, The mixed reality device of claim 1 , wherein the first region is calculated to cover the tool.

4. setting a first direction, a second direction, and a third direction perpendicular to each other with respect to the mixed reality device; The mixed reality device of claim 1 , further comprising: a display unit configured to display a display of the first region and the second region in a first direction, a second direction, and a third direction;

5. The mixed reality device according to claim 4 , further comprising: when an overlap between the first region and the second region is detected, information indicating a direction of the overlap is output.

6. A three-dimensional coordinate system is set using a preset base point; A mixed reality device according to claim 1 , further comprising: a first coordinate system that is a first coordinate position and a second coordinate system that is a second coordinate position.

7. Recognizing a marker from an image including the marker provided in a real space; The mixed reality device of claim 6 , wherein the three-dimensional coordinate system is established using the marker as the base point.

8. The mixed reality device according to claim 6 , wherein a virtual object is displayed based on coordinates of a fastening point registered in advance in the three-dimensional coordinate system.

9. The mixed reality device of claim 8 , wherein when a predetermined object comes into contact with the virtual object, it is estimated that a screw is being turned into the fastening point.

10. A mixed reality device that can display virtual space over real space measuring a first coordinate of the mixed reality device and calculating a first area based on the first coordinate; detecting an object present in a real space around the mixed reality device, and calculating a second area based on second coordinates of the object; A processing method comprising: outputting an alert when the first area and the second area overlap.

11. A mixed reality device that can display virtual space over real space. Measuring a first coordinate of the mixed reality device and calculating a first area based on the first coordinate; detecting an object present in a real space around the mixed reality device, and calculating a second area based on second coordinates of the object; outputting an alert when the first area and the second area overlap; program.

12. A storage medium storing the program according to claim 11.

Images