Mixed reality device, acquisition system, processing method, program and storage medium

Abstract

To provide a mixed reality device, a processing method, a program, and a storage medium for improving work efficiency, and to provide an acquisition system for automatically acquiring physique data.SOLUTION: A mixed reality device is configured to display a virtual object, corresponding to a fastening location where a screw is turned, in a position separated from the fastening location. The mixed reality device is configured to change a display position of the virtual object with respect to the fastening location, according to the physique of a wearer. Preferably, the mixed reality device recognizes a predetermined marker, and sets a three-dimensional coordinate system based on the marker. The position of the fastening location is set in advance in the three-dimensional coordinate system.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] SUMMARY OF THE PRESENT EMBODIMENTS The present invention relates to a mixed reality device, an acquisition system, a processing method, a program, and a storage medium. [Background technology]

[0002] When working on an object, screws are sometimes turned. There is a demand for technology that can improve the efficiency of turning screws. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a mixed reality device, a processing method, a program, and a storage medium that can improve work efficiency, and to provide an acquisition system that can automatically acquire physique data that is referenced when using the mixed reality device, the processing method, the program, and the storage medium. [Means for solving the problem]

[0005] A mixed reality device according to an embodiment displays a virtual object at a position away from a fastening point where a screw is turned, and changes a display position of the virtual object relative to the fastening point according to a physique of a wearer. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 2]FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. [Diagram 3] FIG. 3 is a schematic diagram showing an example of output by the processing device according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. [Diagram 5] FIG. 5 is a schematic diagram illustrating the state of work. [Figure 6] FIG. 6 is a schematic diagram for explaining a display example by the mixed reality device according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram for explaining a display example by the mixed reality device according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram for explaining a display example by the mixed reality device according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram for explaining a display example by the mixed reality device according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram for explaining processing by the mixed reality device according to the embodiment. [Figure 11] FIG. 11 is a schematic diagram for explaining processing by the mixed reality device according to the embodiment. [Figure 12] FIG. 12 is a schematic diagram showing an example of a tool. [Figure 13] FIG. 13 is a flowchart illustrating a processing method according to an embodiment. [Figure 14] 14A and 14B are schematic diagrams for explaining a display example by the mixed reality device according to the embodiment. [Figure 15] 15(a) and 15(b) are schematic diagrams for explaining a display example by the mixed reality device according to the embodiment. [Figure 16] 16(a) and 16(b) are schematic diagrams for explaining a display example by the mixed reality device according to the embodiment. [Figure 17] 17(a) and 17(b) are schematic diagrams for explaining a display example by the mixed reality device according to the embodiment. [Figure 18] FIG. 18 is a schematic diagram for explaining a display example by the mixed reality device according to the embodiment. [Figure 19] FIG. 19 is a flow chart showing a method for checking work. [Figure 20] FIG. 20 is a schematic diagram showing a display example by the mixed reality device according to the embodiment. [Figure 21] FIG. 21 is a schematic diagram showing the configuration of an acquisition system according to an embodiment. [Figure 22] FIG. 22 is a schematic diagram illustrating a display example by a mixed reality device according to a modified example of the embodiment. [Diagram 23] FIG. 23 is a flowchart showing a processing method according to a modified example of the embodiment. [Figure 24] 24(a) to 24(c) are schematic diagrams showing a display example by a mixed reality device according to a modified example of the embodiment. [Diagram 25] FIG. 25 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram illustrating a mixed reality device according to an embodiment. The embodiment of the present invention relates to a mixed reality (MR) device. For example, as shown in Fig. 1, the MR device 100 according to the embodiment includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0009] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0010] The projection device 121 and the projection device 122 display the recognition result of the body of the worker, a virtual object, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0011] The lenses 111 and 112 are optically transparent. The worker can visually recognize the real state through the lenses 111 and 112. The worker can also visually recognize information projected onto the lenses 111 and 112 by the projection devices 121 and 122. The information is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0012] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0013] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0014] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0015] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0016] FIG. 2 is a schematic diagram illustrating an example of an object to be worked on. For example, a screw fastening operation is performed on an article 200 shown in Fig. 2. The article 200 is a cylindrical hollow member, and has fastening points 201 to 208. A worker uses a wrench and an extension bar to fasten screws into each of the fastening points 201 to 208.

[0017] A marker 210 is provided near the work target. In the illustrated example, the marker 210 is an AR marker. As described below, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 150 sets a three-dimensional coordinate system based on a plurality of points indicated by a hand gesture.

[0018] FIG. 3 is a schematic diagram showing an example of output by the processing device according to the embodiment. Here, an example will be described in which a screw is fastened using the MR device 100 shown in Fig. 1. At the start of a fastening operation, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a three-dimensional coordinate system based on the position of the marker 210.

[0019] The image camera 131 and the depth camera 132 capture images of the article 200, the worker's left hand 251, and the worker's right hand 252. The processing device 150 recognizes the left hand 251 and the right hand 252 from the captured images. The processing device 150 may cause the projection device 121 and the projection device 122 to display the recognition results on the lens 111 and the lens 112. Hereinafter, the processing device causing the projection device to display information on the lens will also be simply referred to as "the processing device displays information."

[0020] 3, the processing device 150 displays the recognition results of the left hand 251 and the right hand 252 superimposed on the hands in the real space. In the illustrated example, a plurality of virtual objects 261 and a plurality of virtual virtual objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The plurality of virtual objects 261 respectively represent the plurality of joints of the left hand 251. The plurality of virtual objects 262 respectively represent the plurality of joints of the right hand 252. Instead of the joints, virtual objects (meshes) respectively representing the surface shapes of the left hand 251 and the right hand 252 may be displayed.

[0021] When the processing device 150 recognizes the left hand 251 and the right hand 252, it measures the coordinates of each hand. Specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The coordinates of any of these joints are used as the coordinates of the hand. The center of gravity of the multiple joints may be used as the coordinates of the hand. Alternatively, the center coordinates of the entire hand may be used as the coordinates of the hand.

[0022] FIG. 4 is a schematic diagram illustrating an example of output by the mixed reality device according to the embodiment. The processing device 150 displays virtual objects 301-308 and virtual objects 311-318 as shown in Fig. 4. The virtual objects 301-308 are displayed at positions away from the fastening points 201-208, respectively. The virtual objects 311-318 are displayed between the fastening points 201-208 and the virtual objects 301-308, respectively. The virtual objects 311-318 indicate to which fastening points the virtual objects 301-308 correspond, respectively.

[0023] Virtual objects 301-308 indicate positions where hands should be placed when tightening screws into fastening points 201-208, respectively. Virtual objects 311-318 indicate positions where extension bars should be placed when tightening screws into fastening points 201-208, respectively. For example, the distance between fastening points 201-208 and virtual objects 301-308 corresponds to the length of the extension bars.

[0024] FIG. 5 is a schematic diagram illustrating the state of work. For example, as shown in FIG. 5, a wrench 280 and an extension bar 290 are used to turn the screws into each of the fastening points 201-208. As an example, when fastening a screw into the fastening point 204, the worker places the screw in a threaded hole of the fastening point 204. The worker fits one end of the extension bar 290 onto the screw. The worker fits the head of the wrench 280 into the other end of the extension bar 290. The worker holds the head of the wrench 280 with one hand and holds the grip of the wrench 280 with the other hand. By turning the wrench 280, the screw is fastened into the fastening point 204 via the extension bar 290.

[0025] At this time, the worker positions the extension bar 290 so that the extension bar 290 is close to or in contact with the virtual object 314. The worker also grips the head of the wrench 280 so that his or her hand comes into contact with the virtual object 304. By displaying the virtual object, the worker can easily grasp the positions where the tool and hand are placed when turning the screw into the fastening point 204. This can improve the work efficiency.

[0026] In the illustrated example, the virtual objects 301 to 308 are spherical, and the virtual objects 311 to 318 are rod-shaped. As long as the worker can visually recognize each virtual object, the shape of each object is not limited to this example. For example, the virtual objects 301 to 308 may be cubes, and the virtual objects 311 to 318 may be linear.

[0027] The processing device 150 may change the display positions of the virtual objects 301-308 and the virtual objects 311-318 depending on the physique of the person wearing the MR device 100. The person wearing the MR device 100 is an operator. The "physique" referred to in adjusting the display position is the arm length. The "arm length" is the length from the shoulder to the hand when the hand is outstretched. Data other than the arm length may also be referred to as the physique.

[0028] 6 and 7 are schematic diagrams for explaining a display example by the mixed reality device according to the embodiment. Fig. 6 shows a state where a first worker 250a works. Fig. 7 shows a state where a second worker 250b works. The height of the first worker 250a is taller than the height of the second worker 250b. The arms of the first worker 250a are longer than the arms of the second worker 250b. In the illustrated example, the first worker 250a and the second worker 250b stand on a platform at the same height and turn the screw into the fastening point 206.

[0029] The first worker 250a is tall and has long arms. Therefore, as shown in Fig. 6, the first worker 250a can place his / her hand directly above the fastening point 204 without stretching his / her arm to its limit. The first worker 250a uses a wrench 280 and an extension bar 290a to turn a screw into the fastening point 206. In this case, the processing device 150 displays a virtual object 306 directly above the fastening point 206. The processing device 150 also displays a virtual object 316 along the vertical direction.

[0030] On the other hand, the arms of the second worker 250b are shorter than those of the first worker 250a. In order to place the hands directly above the fastening point 204, the arms need to be stretched to their limit. However, if the arms are stretched to their limit, it becomes difficult to turn the screws. During the work, the posture may become unstable, and the second worker 250b may tumble or fall. For this reason, when the second worker 250b works, the hands of the second worker 250b are positioned closer to the worker than the hands of the first worker 250a.

[0031] In addition, the second worker 250b is shorter than the first worker 250a. Therefore, the hands of the second worker 250b are positioned lower than the hands of the first worker 250a. The second worker 250b uses an extension bar 290b that is shorter than the extension bar 290a. The second worker 250b uses the extension bar 290b by tilting it within a range where the extension bar 290b can be fitted into the screw.

[0032] In this case, the processing device 150 displays the virtual object 306 diagonally above the fastening point 206. Moreover, the processing device 150 displays the virtual object 316 tilted with respect to the vertical direction. The display position (height) of the virtual object 306 shown in FIG. 7 is lower than the display position of the virtual object 306 shown in FIG. 6. In other words, the distance between the fastening point 206 and the virtual object 306 shown in FIG. 7 is shorter than the distance between the fastening point 206 and the virtual object 306 shown in FIG. 6. This is because the extension bar 290b is shorter than the extension bar 290a. Moreover, the virtual object 316 shown in FIG. 7 is tilted more toward the horizontal direction than the virtual object 316 shown in FIG. 6.

[0033] 6 and 7, the processing device 150 changes the display position of the virtual object in accordance with the physique of the worker, thereby enabling each worker to turn the screw in a posture that is more comfortable for the worker.

[0034] The processing device 150 may change the display positions of the virtual objects 301-308 and the virtual objects 311-318 according to the positional relationship between the fastening points 201-208 and the MR device 100. After the three-dimensional coordinate system is set based on the marker 210, the processing device 150 continuously calculates the position of the MR device 100 in the three-dimensional coordinate system. In addition, the positions of the fastening points 201-208 in the three-dimensional coordinate system are registered in advance. Using these data, the processing device 150 calculates the positional relationship between each of the fastening points 201-208 and the MR device 100.

[0035] The calculation method of the position of the MR device 100 is arbitrary, and an existing positioning method may be used. As an example, the processing device 150 calculates the position and direction of the MR device 100 by a function of spatial mapping. In the MR device 100, the depth camera 132 measures the distance between the MR device 100 and the surrounding objects. From the measurement result by the depth camera 132, surface information of the surrounding objects is obtained. The surface information includes the position and direction of the object's surface. For example, the surface of each object is represented by a plurality of meshes, and the position and direction are calculated for each mesh. The processing device 150 calculates the relative position and direction of the MR device 100 with respect to the surfaces of the surrounding objects from the surface information. When the marker 210 is recognized, the position of each surface is also represented in a three-dimensional coordinate system with the marker 210 as the base point. From the positional relationship between the surface of the object and 100, the position and direction of the MR device 100 are calculated.

[0036] The spatial mapping is repeatedly performed at a predetermined interval. Each time spatial mapping is performed, surface information of the surrounding objects is obtained. The processing device 150 calculates the change in the position and orientation of the surface between the latest spatial mapping result and the immediately previous spatial mapping result. When the surrounding objects are not moving, the change in the position and orientation of the surface corresponds to the change in the position and orientation of the MR device 100. The processing device 150 calculates the amount of change in the position and orientation of the MR device 100 from the change in the surface position, the change in the surface orientation, the detection result of the sensor 140, etc. The processing device 150 updates the position and orientation of the MR device 100 from the obtained amount of change.

[0037] 8 and 9 are schematic diagrams for explaining a display example by the mixed reality device according to the embodiment. 8, the second worker 250b is located to the left of the article 200 in the drawing (first direction). In Fig. 9, the second worker 250b is located to the right of the article 200 in the drawing (second direction). The positional relationship between the MR device 100 and the article 200 in Fig. 8 is different from the positional relationship between the MR device 100 and the article 200 in Fig. 9.

[0038] In the state shown in FIG. 8, the processing device 150 changes the display positions of the virtual objects 301-308 to the left in the drawing within a range in which the extension bar 290b can be fitted to the screw. The virtual objects 311-318 are tilted to the left in the drawing with respect to the vertical direction. In the state shown in FIG. 9, the processing device 150 changes the display positions of the virtual objects 301-308 to the right in the drawing within a range in which the extension bar 290b can be fitted to the screw. The virtual objects 311-318 are tilted to the right in the drawing with respect to the vertical direction. By changing the display positions of the virtual objects, the virtual objects are displayed at positions closer to the MR device 100.

[0039] 8 and 9, the processing device 150 changes the display position of the virtual object according to the positional relationship between each fastening point and the MR device 100. This allows each worker to turn the screws in a posture that is more comfortable for the worker.

[0040] One advantage of the embodiment will now be described. As shown in FIG. 4, by displaying the virtual objects 301 to 308, the worker can easily understand the positions where the hands should be placed when turning the screws into the fastening points 201 to 208. This can improve the work efficiency. On the other hand, when the work is actually performed, the optimal position of the hands may differ depending on the physique of the worker. For example, the standard position of each virtual object may be suitable for a tall worker, but may not be suitable for a short worker. In this case, the short worker is forced to assume an awkward posture in order to place the hands at the display positions of the virtual objects. As a result, the work efficiency may decrease, or the worker may fall.

[0041] Regarding this problem, in an embodiment of the present invention, the display position of a virtual object relative to a fastening point changes depending on the physique of a worker (a person wearing the MR device 100). Changing the display position of the virtual object allows each worker to work in a more appropriate posture. This can further improve work efficiency. Also, the worker can work more safely.

[0042] As a specific example, as shown in FIG. 6 and FIG. 7, a first worker 250a having a first height and a second worker 250b having a second height perform the same work. The height of the second worker 250b is shorter than that of the first worker 250a. The display position of the virtual object 306 during the work of the second worker 250b is lower than the display position of the virtual object 306 during the work of the first worker 250a. That is, the distance between the fastening point 206 and the virtual object 306 during the work of the second worker 250b is shorter than the distance between the fastening point 206 and the virtual object 306 during the work of the first worker 250a. In addition, the inclination of the virtual object 316 during the work of the first worker 250a is different from the inclination of the virtual object 316 during the work of the second worker 250b.

[0043] Another advantage of the embodiment will now be described. Depending on the size or shape of the object, the fastening point may be in a position that is difficult to reach. In this case, it may be difficult to place the hand at the position where the virtual object is displayed. If a worker places the hand at the display position of the virtual object in an awkward posture, the work efficiency may decrease or the worker may fall.

[0044] Regarding this problem, in an embodiment of the present invention, the display position of a virtual object relative to a fastening point changes according to the positional relationship between the fastening point and the MR device 100. By changing the display position of the virtual object according to the positional relationship, each worker can work in a more appropriate posture. This allows the worker to work more safely. Also, the work efficiency can be improved.

[0045] As a specific example, in the state shown in FIG. 8, the second worker 250b (MR device 100) is located in a first direction relative to the article 200. In this case, the processing device 150 changes the display positions of the virtual objects 301-308 in the first direction. In the state shown in FIG. 9, the second worker 250b (MR device 100) is located in a second direction opposite to the first direction relative to the article 200. In this case, the processing device 150 changes the display positions of the virtual objects 301-308 in the second direction. In addition, the processing device 150 changes the inclinations of the virtual objects 311-318 in response to the change in the display positions of the virtual objects 301-308.

[0046] The method of changing the display position is arbitrary. For example, a standard arm length is registered in advance, and the horizontal position of the virtual object changes according to the difference between the standard arm length and the arm length of the worker. Also, the greater the distance between the standard position of the virtual object and the MR device 100, the greater the change in the display position of the virtual object.

[0047] 10 and 11 are schematic diagrams for explaining processing by the mixed reality device according to the embodiment. In order to display the virtual object at a more desirable position, the following process may be executed. First, the processing device 150 calculates a first range in which work can be performed for the fastening point based on the position of the fastening point, the length of the extension bar used, and the allowable angle of the extension bar. As a specific example, first ranges R1a to R1d are calculated as shown in Fig. 10. The first ranges R1a to R1d indicate the range in which work can be performed for the fastening points 201 to 204, respectively.

[0048] Next, a second range R2 in which the worker can work is calculated based on the position of the MR device 100 and the length of the worker's arm. The processing device 150 calculates the range in which each of the first ranges R1a to R1d overlaps with the second range R2. The processing device 150 adopts the position closest to the MR device 100 within the overlapping range as the display position of the virtual object. The processing device 150 displays the virtual objects 301 to 304 at the adopted position, as shown in Fig. 11. Although the virtual objects 301 to 304 are omitted in Figs. 10 and 11, the display positions of these virtual objects can also be changed by a similar method.

[0049] It should be noted that if the first range and the second range do not overlap, the processing device 150 does not display a virtual object. If the first range and the second range do not overlap, this indicates that the worker is in an inappropriate position relative to the fastening location to be worked on. If a worker works in an inappropriate position, work efficiency or safety may decrease. If the first range and the second range do not overlap, the processing device 150 may output an alert to encourage the worker to move to a more appropriate position.

[0050] After the virtual objects are displayed, the processing device 150 may determine whether a predetermined object has come into contact with the virtual objects 301-308. For example, the processing device 150 determines whether a hand has come into contact with a virtual object. Specifically, the processing device 150 calculates the distance between the coordinates of the hand and each of the virtual objects 301-308. If any of the distances is less than a preset threshold, the processing device 150 determines that the hand has come into contact with that virtual object. As an example, in FIG. 4, the diameter of the virtual objects 301-308 (spheres) corresponds to the threshold. The spheres indicate the range within which it is determined that the hand has come into contact with the virtual object.

[0051] FIG. 12 is a schematic diagram showing an example of a tool. The processing device 150 may determine whether the tool has come into contact with the virtual objects 301 to 308. For example, as shown in FIG. 12, a plurality of markers 281 are attached to the wrench 280. The processing device 150 recognizes the plurality of markers 281 from an image captured by the image camera 131. The processing device 150 measures the coordinates of each of the markers 281. A positional relationship between the plurality of markers 281 and the head 282 of the wrench 280 is registered in advance. The processing device 150 calculates the coordinates of the head 282 based on the coordinates of at least three recognized markers 281 and the positional relationship registered in advance. The processing device 150 calculates the distance between the coordinates of the head 282 and each of the virtual objects 301 to 308. If any of the distances is less than a preset threshold, the processing device 150 determines that the wrench 280 has come into contact with that virtual object.

[0052] When a predetermined object comes into contact with one of the virtual objects 301-308, it can be estimated that a screw has been turned into a fastening point corresponding to that one of the virtual objects 301-308. Hereinafter, a fastening point that is estimated to be working on based on the contact between a predetermined object and a virtual object among one or more fastening points will be referred to as a "work point." When a work point is estimated, the processing device 150 may record that a screw has been turned into that work point. This allows a work record to be automatically created.

[0053] The tool used may be a digital tool capable of detecting torque. In this case, the processing device 150 receives the detected torque from the tool. The torque required for fastening may be set in advance, and the tool may determine whether the required torque has been detected and transmit the determination result to the processing device 150. In addition, the tool transmits the rotation angle, the time when the torque was detected, and the like to the processing device 150. The processing device 150 links the data received from the tool with data related to the work location. This allows a more detailed work record to be automatically created.

[0054] FIG. 13 is a flowchart illustrating a processing method according to an embodiment. 13, task data 171, worker data 172, and fastening location data 173 are referenced. The task data 171, worker data 172, and fastening location data 173 are stored in advance in the storage device 170 before the task. The task data 171, worker data 172, and fastening location data 173 may be stored in a storage area other than the MR device 100. In this case, the processing device 150 accesses the task data 171, worker data 172, and fastening location data 173 via wireless communication or a network.

[0055] First, the processing device 150 accepts the selection of a task and a worker (step S1). When selecting a task, task data 171 is read. When selecting a worker, worker data 172 is read. A task ID, a task name, an item ID, and an item name are registered in the task data 171. The processing device 150 can accept any one of the task ID, task name, item ID, or item name as a task selection. The worker data 172 registers the worker ID, worker name, and physical build. The processing device 150 can accept a worker ID or a worker name as a worker selection.

[0056] For example, the work and the worker may be selected by the worker, or may be selected by a higher-level system. The processing device 150 may determine the work and the worker based on data obtained from a sensor or a reader installed in the work area. The work and the worker may be automatically selected based on a schedule prepared in advance.

[0057] When the work and the worker are selected, the processing device 150 reads the data stored in the fastening point data 173 (step S2). The fastening point data 173 stores the method of identifying the base point, the ID of each fastening point, the position of each fastening point, the type of tool used, the angle of the extension bar, the required number of fastenings, the required torque, the color of the mark, and the ID of each virtual object.

[0058] Methods for identifying the base point include a method using a marker or a method using hand gestures. The ID of the fastening point is a unique string for identifying each fastening point. The coordinates of a three-dimensional coordinate system based on the base point are registered as the position of the fastening point. The tool type indicates the classification of the tool based on the structure, external shape, performance, etc. For example, the length of the extension bar is specified from the extension bar type. The angle indicates the limit value of the angle at which the extension bar can engage with the screw when turning the screw into the fastening point.

[0059] In work, a mark may be made when a screw is turned into a fastening point. "Mark color" indicates the color of the mark made at each fastening point. When marks of different colors are made depending on the number of times a screw is turned, the color of the mark for each number of times is registered. The virtual object ID is a character string for identifying data of a virtual object registered in advance, and a virtual object ID is linked to each fastening point.

[0060] The processing device 150 calculates a first range based on the position of the fastening point, the length of the extension bar used, and the allowable angle of the extension bar (step S3). The first range indicates the hand position at which the screw can be turned relative to the fastening point when the worker's physique is not taken into consideration. The processing device 150 calculates a second range based on the position of the MR device 100 and the worker's arm length (step S4). The second range indicates the hand position at which the worker can work. The arm length is pre-stored in the worker data 172 as physique data.

[0061] Data other than the arm length may be further referred to as the physique. For example, three pieces of data, the arm length, the distance from the eye to the chin, and the neck length, are referred to. The sum of the distance from the eye to the chin and the neck length is approximately equal to the vertical distance between the shoulder and the MR device 100 when the person stretches his / her arm straight in front. Consider a virtual triangle with the hand, the base of the neck, and the eye as vertices. In this case, the distance between the hand and the MR device 100 can be calculated from the arm length, the distance from the eye to the chin, and the neck length by trigonometry. The position of the MR device 100 is repeatedly calculated by spatial mapping. The processing device 150 can calculate the second range based on the position of the MR device 100 and the distance between the hand and the MR device 100.

[0062] As an example of physical build, shoulder width may be referred to. Shoulder width is the distance between the left shoulder and the right shoulder. When the hand is stretched straight out in front, the distance in the left-right direction between the eye (MR device 100) and the hand is approximately half the shoulder width. Consider a virtual triangle with the hand, eye (MR device 100), and shoulder as vertices. In this case, the distance between the hand and the MR device 100 can be calculated from the arm length and half the shoulder width by trigonometry. The processing device 150 can calculate the second range based on the position of the MR device 100 and the distance between the hand and the MR device 100. The second range may be calculated using four pieces of data: the arm length, the distance from the eye to the chin, the neck length, and the shoulder width.

[0063] The processing device 150 calculates the overlapping range of the first range and the second range (step S5). The processing device 150 calculates the closest position to the MR device 100 from the overlapping range (step S6). The processing device 150 displays the virtual object at the closest position (step S7).

[0064] When multiple virtual objects are displayed, steps S3 to S7 are executed for each virtual object. After a virtual object is displayed, steps S3 to S7 are repeated. This updates the display position of the virtual object according to the positional relationship between the fastening point and the MR device 100.

[0065] After displaying the virtual object, the processing device 150 determines whether a predetermined object has come into contact with the virtual object (step S8). If the predetermined object has come into contact with the virtual object, the processing device 150 estimates the work location based on the determination result (step S9). That is, it is estimated that a screw is being turned into the fastening location corresponding to the virtual object. The processing device 150 saves a record of the work performed on the work location (step S10).

[0066] In step S10, the processing device 150 associates the torque detected by the tool with the ID of the fastening point where it is estimated that the screw is being turned, and stores it in the history data 174. As shown in FIG. 13, the processing device 150 may further associate the type and ID of the tool used, the number of times the screw has been tightened, and the mark recognition result with the ID of the fastening point. The mark is recognized by the processing device 150 from an image captured by the image camera 131. The processing device 150 extracts a block of pixels of the mark color from the image, and counts the number of pixels in that block. If the number of pixels exceeds a preset threshold, it is determined that a mark has been attached.

[0067] Figures 14(a), 14(b), 15(a), 15(b), 16(a), 16(b), 17(a), 17(b), and 18 are schematic diagrams for explaining display examples by a mixed reality device relating to an embodiment. The processing device 150 may display information indicating the order of tasks on the virtual objects 301 to 308. For example, as shown in Fig. 14(a), numbers indicating the order are displayed superimposed on the virtual objects 301 to 308. As shown in Fig. 14(b), numbers indicating the order may be displayed near each of the virtual objects 301 to 308.

[0068] The display of a virtual object corresponding to a fastening point where a screw has been turned may be different from the display of a virtual object corresponding to a fastening point where a screw has not been turned. In the example shown in Fig. 15(a), screws have been turned into fastening point 201 and fastening point 205, and no screws have been turned into other fastening points. The color of virtual object 301 corresponding to fastening point 201 and the color of virtual object 305 corresponding to fastening point 205 are different from the colors of virtual objects 302 to 304 and the colors of virtual objects 306 to 308. Instead of the color, the size or shape of the virtual object may be changed.

[0069] When a screw is turned multiple times for one fastening location, the display of the virtual object may change depending on the number of times the screw is turned. For example, the color of virtual object 301 and the color of virtual object 305 shown in Fig. 15(a) indicate that the first screw has been turned. The color of virtual object 301 and the color of virtual object 305 shown in Fig. 15(b) indicate that the second screw has been turned. Instead of the color, the size or shape of the virtual object may change depending on the number of times.

[0070] The order of operations may change depending on the number of times the screw is turned. For example, FIG. 16(a) shows a state in which no screw is turned at any of the fastening points. In this state, the screws are tightened at each fastening point in a diagonal order. That is, the screws are tightened at the fastening point 201, the fastening point 205, the fastening point 202, the fastening point 206, the fastening point 203, the fastening point 207, the fastening point 204, and the fastening point 208 in this order. FIG. 16(b) shows a state after the screws are tightened at the diagonal order. In this state, the screws are tightened at the fastening points 201 to 208 in a clockwise or counterclockwise direction. The order displayed on the virtual objects 301 to 308 is different between the state shown in FIG. 16(a) and the state shown in FIG. 16(b).

[0071] The number of times the screw is turned is counted based on the data stored in the history data 174. When a screw is turned into one fastening point, a work record indicating this is saved in the history data 174. The processing device 150 counts the number of times the screw is turned into each fastening point from the records stored in the history data 174, and controls the display of each virtual object based on the counted number of times.

[0072] The processing device 150 may display a virtual panel including information related to the work. In the example shown in Fig. 17(a), a virtual panel 321 is displayed. The panel 321 is displayed near a virtual object 301 corresponding to the fastening point 201. As shown in Fig. 17(b), panels 321 to 328 may be displayed near virtual objects 301 to 308, respectively.

[0073] As shown in FIG. 17(a), among the multiple virtual objects, only a virtual object corresponding to a fastening point 201 where the screw is to be turned next may be displayed, or as shown in FIG. 17(b), more virtual objects may be displayed. By displaying only the virtual object corresponding to the fastening point where the screw is to be turned, the information visible to the worker is reduced, making it easier for the worker to understand the next task. On the other hand, by displaying more virtual objects, it becomes easier for the worker to confirm the flow of the work. The worker may be able to switch between the state shown in FIG. 17(a) and the state shown in FIG. 17(b). For example, the processing device 150 receives an instruction to switch by a voice command or a hand gesture.

[0074] As an example, as shown in Fig. 18, the panel 321 includes work information including an identification number 321a, a specified torque value 321b, a detected value 321c, a meter 321d, a ratio 321e, and a number of times 321f. The identification number 321a is an identification number of the fastening point 201. The specified torque value 321b is a torque value specified for fastening a screw into the fastening point 201. The detected value 321c is a torque value detected by a tool. The meter 321d indicates the specified torque value and the detected torque value. The ratio 321e indicates the ratio of the detected value to the specified torque value. The number of times 321f indicates the number of times the screw has been fastened into the fastening point 201.

[0075] Fig. 19 is a flowchart showing a method for checking an operation. Fig. 20 is a schematic diagram showing a display example by the mixed reality device according to the embodiment. After the work is completed, the processing device 150 may check whether the screws have been properly turned in all fastening points. Specifically, the processing device 150 receives a check instruction (step S11). The check instruction may be input by the worker or may be input from a higher-level system. Upon receiving the instruction, the processing device 150 reads the fastening point data 173 (step S12) and reads the history data 174 (step S13).

[0076] The processing device 150 checks whether the screw has been properly turned for the fastening portion to be worked on (step S14). Specifically, the processing device 150 checks whether the screw has been turned the required number of times for each fastening portion. The processing device 150 also checks whether the required torque has been detected for each screw turning.

[0077] The processing device 150 determines whether an error has occurred as a result of the check (step S15). If the torque detected in any screw turning is less than the required torque registered in advance, or if the number of times a screw has been turned in any fastening location is less than the required number registered in advance, the processing device 150 determines that an error has occurred in the work.

[0078] If there is no error, the processing device 150 ends the process. If there is an error, the processing device 150 displays the fastening point where the error was detected to the worker (step S16). For example, as shown in FIG. 20, the processing device 150 makes the color of the panel 328 corresponding to the fastening point where the error was detected different from the colors of the other panels 321 to 327. The shape or size of the panel 328 may be different from the shapes or sizes of the panels 321 to 327. A new virtual object may be displayed to highlight the panel 328. Information included in the panel 328 may be displayed with an emphasis. For example, the panel 328 may be highlighted by making the color or size of the characters included in the panel 328 different from the color or size of the characters included in the panels 321 to 327.

[0079] Thereafter, when a screw is turned into the fastening location where the error was detected, the processing device 150 checks the work (step S17). When it is confirmed that the screw has been turned properly, the processing device 150 ends the process. By carrying out the check, it is possible to prevent the production of an article 200 that has been improperly worked on. For example, it is possible to improve the quality of the article 200.

[0080] FIG. 21 is a schematic diagram showing the configuration of an acquisition system according to an embodiment. 21 acquires physique data referenced by a processing device 150. The acquisition system 1 includes an imaging device 2 and a processing device 3.

[0081] The imaging device 2 photographs the worker and acquires an image. An image showing at least the upper body is acquired. The imaging device 2 includes a camera. The processing device 3 receives the image acquired by the imaging device 2. The processing device 3 inputs the image into a posture estimation model. When an image is input, the posture estimation model estimates the posture of the person depicted in the image. The posture is represented by the joints of the human body and the skeleton that connects the joints. The joints include the head, neck, shoulders, elbows, wrists, hips, knees, and ankles.

[0082] The pose estimation model preferably includes a neural network. More preferably, the pose estimation model includes a convolutional neural network (CNN). As the pose estimation model, OpenPose, DarkPose, CenterNet, or the like can be used.

[0083] The processing device 3 acquires the estimation result output from the posture estimation model. The processing device 3 calculates physique data such as the worker's height from the estimated posture. Other physique data may also be calculated, such as the worker's height, neck position, and shoulder position. The processing device 3 stores the calculated physique data in the worker data 172. By using the acquisition system 1, the physique data used by the processing device 150 can be automatically acquired.

[0084] The MR device 100 may have a function as the acquisition system 1. For example, the image camera 131 captures the worker's arm when the worker stretches his / her arm out in front of him / her to the limit. The processing device 150 measures the coordinates of the worker's hand and calculates the distance between the MR device 100 and the coordinates of the hand. The distance is proportional to the length of the worker's arm and indicates the limit of the range in which the worker can work. The processing device 150 stores the distance in the worker data 172 as data on the length of the worker's arm.

[0085] (Modification) The processing device 150 may determine whether the posture of the worker is appropriate, instead of controlling the display position of the virtual object. Alternatively, the processing device 150 may execute posture determination in addition to controlling the display position of the virtual object.

[0086] Specifically, the processing device 150 estimates the posture of the worker based on the position of the MR device 100, the positions of the worker's hands, and the physique of the worker. The processing device 150 determines whether the estimated posture is appropriate. For example, if the posture is inappropriate for the work or is unsafe, the processing device 150 determines that the posture is inappropriate.

[0087] In estimating the posture, the processing device 150 measures the position of the MR device 100 and the position of each hand of the worker. The processing device 150 calculates the distance between the MR device 100 and each hand. The processing device 150 also refers to the arm length stored in the worker data 172. The processing device 150 sets a first range based on the arm length. The first range may be set using the distance from the eye to the chin, the neck length, the shoulder width, etc. in addition to the arm length. The processing device 150 compares the calculated distance with the first range.

[0088] The distance between the MR device 100 and the hand becomes longer as the worker extends his / her arm. The closer the distance is to the preregistered arm length, the more the worker extends his / her arm. If the distance is too short compared to the preregistered arm length, the hand of the worker is located close to the body. For example, the first range is calculated by multiplying the preregistered arm length by a predetermined ratio. The predetermined ratio is appropriately set from the viewpoint of safety or work efficiency. For example, a value 0.5 times the arm length is set as the lower limit of the first range. A value 0.8 times the arm length is set as the upper limit of the first range. If the distance is outside the first range, it indicates that the range of motion of the worker's arm is narrow and the posture is inappropriate for work. Therefore, if any of the distances is outside the first range, the processing device 150 determines that the posture of the worker is not appropriate.

[0089] FIG. 22 is a schematic diagram illustrating a display example by a mixed reality device according to a modified example of the embodiment. 22 shows a state in which a worker is performing a task with his / her arms outstretched. The processing device 150 measures the position of the MR device 100, the position of the left hand 251, and the position of the right hand 252. The processing device 150 calculates the distance between the MR device 100 and the left hand 251 and the distance between the MR device 100 and the right hand 252. The processing device 150 compares each distance with a first range.

[0090] When the worker stretches his / her arms almost to their limit, it is determined that both distances are outside the first range. When the processing device 150 determines that the work is inappropriate, it displays an alert 350. As an example, the alert 350 includes a reason 351 why the posture was determined to be inappropriate, and an instruction 352 for the worker. By displaying the alert 350, it is possible to encourage the worker to work in a more appropriate posture.

[0091] The processing device 150 may further use the inclination of the MR device 100 to determine whether the posture is appropriate. The inclination of the MR device 100 is detected by the sensor 140. Here, the inclination refers to the orientation (angle) of the MR device 100 with respect to a reference direction. The reference direction is the orientation of the MR device 100 when the worker faces the horizontal direction. The processing device 150 determines that the work is inappropriate when the distance between the MR device 100 and the hand is out of a first range or when the inclination exceeds a first threshold. The first threshold is appropriately set from the viewpoint of work safety. For example, any value within a range of 15 degrees to 45 degrees is set as the first threshold. When the inclination exceeds the first threshold, it indicates that the worker is leaning forward. If the worker is leaning forward, there is a possibility that the worker will fall. When the work is performed at a high place, there is a possibility that the worker will fall.

[0092] If the operation is appropriate, the processing device 150 may not display any information, or may display information indicating that the operation is appropriate.

[0093] FIG. 23 is a flowchart showing a processing method according to a modified example of the embodiment. In the processing method M2 shown in Fig. 23, like the processing method M1 shown in Fig. 13, task data 171, worker data 172, and fastening point data 173 are referenced. In Fig. 23, illustration of these data is omitted.

[0094] First, the processing device 150 executes steps S1 and S2 in the same manner as in the processing method M1. Next, the processing device 150 displays a virtual object based on the data stored in the fastening point data 173 (step S7). The processing device 150 determines whether a predetermined object has come into contact with the virtual object (step S8).

[0095] When a predetermined object comes into contact with the virtual object, the processing device 150 measures the position of the hand. The processing device 150 estimates the posture of the worker based on the position of the MR device 100, the position of the worker's hand, and the physique of the worker (step S21). The processing device 150 determines whether the estimated posture is appropriate (step S22). Specifically, a first determination is made as to whether the distance between the MR device 100 and the hand falls outside a first range, and a second determination is made as to whether the inclination exceeds a first threshold.

[0096] If it is determined as a result of the first and second determinations that the posture is not appropriate, the processing device 150 prohibits the execution of the work (step S23). The processing device 150 determines the display content (step S24). The display content includes, for example, as shown in FIG. 22, the danger of an inappropriate posture, instructions for making the posture more appropriate, and the like. Once the display content is determined, the processing device 150 displays an alert (step S25).

[0097] Two or more ranges to be compared with the distance may be set. Two or more thresholds to be compared with the inclination may be set. In this case, the display content may be changed according to the results of comparing the distance with each range or the results of comparing the inclination with each threshold. For example, the wider the range the distance deviates from, or the larger the threshold the inclination exceeds, the more an alert that emphasizes the danger is output.

[0098] 24(a) to 24(c) are schematic diagrams showing a display example by a mixed reality device according to a modified example of the embodiment. As an example, one range to be compared with the distance is set, and two thresholds (first and second thresholds) to be compared with the inclination are set. The second threshold is greater than the first threshold. In step S24, the processing device 150 determines the display content according to the results of the first and second judgments. For example, when the distance exceeds the upper limit of the first range and the inclination exceeds the first threshold, the processing device 150 displays an alert 350a indicating that the worker is too far away, as shown in FIG. 24(a). When the distance is below the lower limit of the first range and the inclination does not exceed the first threshold, the processing device 150 displays an alert 350b indicating that the worker is too close, as shown in FIG. 24(b). When the distance is within the first range and the inclination exceeds the second threshold, the processing device 150 displays an alert 350c indicating that there is a risk of falling, as shown in FIG. 24(c).

[0099] If the posture is determined to be appropriate in step S22, the processing device 150 permits the work (step S26). After step S25 or S26, the processing device 150 estimates the work location (step S9). That is, it is estimated that a screw is being turned into a fastening location corresponding to a virtual object with which a predetermined object has come into contact. Thereafter, step S10 is executed as in processing method M1. By determining whether the working posture is appropriate, it is possible to encourage the worker to work in a safer or more efficient posture. As a result, it is possible to improve the safety or efficiency of the work.

[0100] The processing method M1 shown in Fig. 13 may be combined with the processing method M2 shown in Fig. 23. Specifically, steps S3 to S6 of the processing method M1 may be performed after step S2. By combining the processing methods M1 and M2, it is possible to further improve safety and work efficiency.

[0101] Here, an example has been described in which the embodiment of the present invention is applied mainly to the task of tightening a screw. The embodiment of the present invention may also be applied to the task of loosening a screw. When loosening a screw, a tool is used to turn the screw, as shown in FIG. 5. In this case, too, the task can be performed efficiently by displaying a virtual object. Furthermore, the task can be performed more efficiently and safely.

[0102] FIG. 25 is a schematic diagram showing a hardware configuration. 25 is used as the processing device 3 and the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0103] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0104] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0105] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) etc. The storage device 94 may be used as the storage device 170.

[0106] The input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95. The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to an output device (e.g., projection devices 121 and 122) via the output I / F 96 and cause information to be displayed on the output device.

[0107] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0108] The data processing by the processing device 3 or the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0109] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0110] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0111] Embodiments of the invention include the following features. (Feature 1) A mixed reality device that displays a virtual object at a position away from a fastening point where a screw is turned, A mixed reality device that changes the display position of the virtual object relative to the fastening point depending on the wearer's physique. (Feature 2) The mixed reality device of feature 1, wherein the height at which the virtual object is displayed is changed using the position of the fastening point and the length of a pre-registered tool. (Feature 3) Calculating a first range within which work can be performed on the fastening location using the position of the fastening location and a previously registered tool length; Calculating a second range in which the wearer can work based on a position of the mixed reality device and the body size; 2. The mixed reality device of feature 1, wherein the virtual object is displayed within a range where the first range and the second range overlap. (Feature 4) A mixed reality device that displays a virtual object at a position away from a fastening point where a screw is turned, A mixed reality device that changes the display position of the virtual object relative to the fastening point depending on the positional relationship between the fastening point and the mixed reality device. (Feature 5) When the mixed reality device is located in a first direction with respect to the fastening point, the display position of the virtual object is changed toward the first direction; The mixed reality device of feature 4, wherein when the mixed reality device is positioned in a second direction opposite to the first direction relative to the fastening point, the display position of the virtual object is changed toward the second direction. (Feature 6) A mixed reality device that displays a virtual object at a position away from a fastening point where a screw is turned, measuring a distance between the worker's hand and the mixed reality device when a predetermined object touches the virtual object; A mixed reality device that displays an alert if the distance is outside a first range set based on a wearer's body size or if the tilt of the mixed reality device exceeds a first threshold. (Feature 7) Recognizing a predetermined marker, A three-dimensional coordinate system is set with the marker as a base point; The mixed reality device according to any one of features 1 to 6, wherein the position of the fastening point is preset in the three-dimensional coordinate system. (Feature 8) Displaying a plurality of the virtual objects corresponding to the plurality of fastening points, respectively; The mixed reality device according to any one of features 1 to 7, wherein information indicating an order in which the screws are turned is displayed on each of the plurality of virtual objects. (Feature 9) determining whether the virtual object has come into contact with a predetermined object; The mixed reality device according to any one of features 1 to 8, wherein, when the virtual object comes into contact with the specified object, it is estimated that a screw is being turned into the fastening point. (Feature 10) The mixed reality device of feature 9, wherein work records are linked to data regarding the estimated fastening location. (Feature 11) The work record includes a torque detected when the screw is turned into the fastening portion and a number of times the screw is turned into the fastening portion, The torque and the number of times are compared with a required torque and a required number of times that are preset for the fastening portion, respectively; 11. The mixed reality device of feature 10, wherein an error is determined to exist if the torque is insufficient to the required torque or if the number of times is less than the required number of times. (Feature 12) An acquisition system for acquiring the physique referenced by the mixed reality device according to feature 1 or 6, An acquisition system that calculates the body size from the distance between the mixed reality device and the wearer's hands or from an image of the wearer's body. (Feature 13) displaying a virtual object on the mixed reality device at a position away from the fastening point where the screw is turned, in correspondence with the fastening point; A processing method in which the display position of the virtual object relative to the fastening point is changed depending on the wearer's physique or the positional relationship between the fastening point and a mixed reality device. (Feature 14) A program for causing a computer to execute the processing method according to feature 13. (Feature 15) A storage medium storing the program according to feature 14.

[0112] According to the above-described embodiment, a mixed reality device, a processing method, a program, and a storage medium that can improve work efficiency are provided. Also, an acquisition system that can automatically acquire physique data that is referenced when using the mixed reality device, the processing method, the program, and the storage medium is provided.

[0113] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0114] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0115] 1: Acquisition system, 2: Imaging device, 3: Processing device, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: MR device, 101: Frame, 111, 112: Lens, 121, 122: Projection device, 131: Image camera, 132: Depth camera, 140: Sensor, 141: Microphone, 150: Processing device, 160: Battery, 170: Storage device, 171: Work data, 172: Worker data, 173: Fastening point data, 174: History data, 200: Article, 201-208: Fastening point, 210: Marker, 250a: First worker, 250b: second operator, 251: left hand, 252: right hand, 261, 262: virtual object, 280: wrench, 281: marker, 282: head, 290: extension bar, 290a: extension bar, 290b: extension bar, 301-308: virtual object, 311-318: virtual object, 321: panel, 321a: identification number, 321b: specified torque value, 321c: detected value, 321d: meter, 321e: ratio, 321f: number of times, 328: panel, 350, 350a, 350b: alert, 351: cause, 352: instruction, M1, M2: processing method, R1a-R1e: first range, R2: second range

Claims

1. A mixed reality device that displays a virtual object at a position away from a fastening point where a screw is turned, A mixed reality device that changes the display position of the virtual object relative to the fastening point depending on the wearer's physique.

2. The mixed reality device according to claim 1 , wherein the height at which the virtual object is displayed is changed using the position of the fastening point and a length of a preregistered tool.

3. Calculating a first range within which work can be performed on the fastening location using the position of the fastening location and a length of a tool that has been registered in advance; Calculating a second range in which the wearer can work based on a position of the mixed reality device and the body size; The mixed reality device of claim 1 , wherein the virtual object is displayed within a range where the first range and the second range overlap.

4. A mixed reality device that displays a virtual object at a position away from a fastening point where a screw is turned, A mixed reality device that changes the display position of the virtual object relative to the fastening point depending on the positional relationship between the fastening point and the mixed reality device.

5. When a mixed reality device is located in a first direction with respect to the fastening point, the display position of the virtual object is changed toward the first direction; The mixed reality device of claim 4 , wherein when the mixed reality device is positioned in a second direction opposite to the first direction with respect to the fastening point, the display position of the virtual object is changed toward the second direction.

6. A mixed reality device that displays a virtual object at a position away from a fastening point where a screw is turned, measuring a distance between the worker's hand and the mixed reality device when a predetermined object touches the virtual object; A mixed reality device that displays an alert if the distance is outside a first range set based on the wearer's body size or if the tilt of the mixed reality device exceeds a first threshold.

7. Recognizing a predetermined marker, A three-dimensional coordinate system is set with the marker as a base point; The mixed reality device according to claim 1 , wherein the position of the fastening point is preset in the three-dimensional coordinate system.

8. Displaying a plurality of the virtual objects corresponding to the plurality of fastening points, respectively; The mixed reality device according to claim 1 , wherein information indicating an order in which screws are turned is displayed on each of the plurality of virtual objects.

9. determining whether the virtual object has come into contact with a predetermined object; The mixed reality device according to claim 1 , wherein, when the virtual object comes into contact with the specified object, it is estimated that a screw is being turned into the fastening point.

10. The mixed reality device according to claim 9 , wherein a work record is linked to data relating to the estimated fastening location.

11. The work record includes a torque detected when the screw is turned into the fastening portion and a number of times the screw is turned into the fastening portion, The torque and the number of times are compared with a required torque and a required number of times that are preset for the fastening portion, respectively; The mixed reality device of claim 10 , further comprising: determining that there is an error if the torque is less than the required torque or if the number of times is less than the required number of times.

12. 10. A system for acquiring the physique referenced by a mixed reality device according to claim 1, comprising: An acquisition system that calculates the body size from the distance between the mixed reality device and the wearer's hands or from an image of the wearer's body.

13. displaying a virtual object on the mixed reality device at a position away from the fastening point where the screw is turned; A processing method in which the display position of the virtual object relative to the fastening point is changed depending on the wearer's physique or the positional relationship between the fastening point and a mixed reality device.

14. A program for causing a computer to execute the processing method according to claim 13.

15. A storage medium storing the program according to claim 14.

Images