Mixed reality device, processing method, processing device, program and storage medium

Abstract

To provide a mixed reality device, a processing method, a processing device, a program, and a storage medium capable of suppressing inappropriate work from being performed.SOLUTION: A mixed reality device includes an imaging part and a processing part. The imaging part is configured to acquire an image. The processing part is configured to recognize, from the image, a hand of a worker performing fastening work. The processing part is further configured to determine whether the fastening work is appropriate or not, based on a first distance between a first position corresponding to a fastening location of a screw and a second position of the hand holding a first tool.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a mixed reality device, a processing method, a processing device, a program, and a storage medium. [Background technology]

[0002] In some cases, mixed reality devices are used in the manufacture of goods. There is a demand for a technology that can prevent inappropriate work from being performed in fastening work using a mixed reality device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a mixed reality device, a processing method, a processing device, a program, and a storage medium that are capable of suppressing the execution of inappropriate tasks. [Means for solving the problem]

[0005] A mixed reality device according to an embodiment includes an image capturing unit and a processing unit. The image capturing unit captures an image. The processing unit recognizes the hand of a worker performing a fastening operation from the image. The processing unit determines whether the fastening operation is appropriate based on a first distance between a first position corresponding to a fastening location of a screw and a second position of the hand holding a first tool. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a processing system according to the first embodiment. [Diagram 2]FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 3] FIG. 3 is a schematic diagram illustrating an example of an object to be worked on. [Figure 4] FIG. 4 is a schematic diagram for explaining the first embodiment of the present invention. [Diagram 5] 5(a) and 5(b) are schematic diagrams for explaining a first embodiment of the present invention. [Figure 6] 6(a) and 6(b) are schematic diagrams for explaining a first embodiment of the present invention. [Figure 7] 7(a) and 7(b) are schematic diagrams for explaining the first embodiment of the present invention. [Figure 8] FIG. 8 is a schematic diagram for explaining the first embodiment of the present invention. [Figure 9] FIG. 9 is a flowchart illustrating the processing method according to the first embodiment. [Figure 10] 10(a) and 10(b) are schematic diagrams for explaining a second embodiment of the present invention. [Figure 11] 11(a) and 11(b) are schematic diagrams showing an output example in the second embodiment of the present invention. [Figure 12] FIG. 12 is a flowchart illustrating a processing method according to the second embodiment. [Figure 13] FIG. 13 is a flowchart illustrating a processing method according to a first modified example of the second embodiment. [Figure 14] FIG. 14 is a schematic diagram for explaining a second modified example of the second embodiment. [Figure 15] FIG. 15 is a flowchart illustrating a processing method according to a second modification of the second embodiment. [Figure 16] FIG. 16 is a schematic diagram for explaining the third embodiment of the present invention. [Figure 17] FIG. 17 is a schematic diagram for explaining the third embodiment of the present invention. [Figure 18] FIG. 18 is a flowchart showing a processing method according to the third embodiment. [Figure 19] FIG. 19 is a flowchart showing an outline of data processing in the processing method according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals and detailed explanations are omitted as appropriate.

[0008] (First embodiment) FIG. 1 is a schematic diagram showing the configuration of a processing system according to the first embodiment. The first embodiment of the present invention is applicable to work using a tool. A processing system 10 includes an imaging unit 1, a processing unit 2, an input unit 3, a display unit 4, and a storage unit 5, as shown in FIG.

[0009] The photographing unit 1 photographs the state of work. Specifically, in the work, a fastener such as a screw is fastened to an object using a tool. The object is a part, a unit, or a semi-finished product for manufacturing a product. The tool is a wrench, a screwdriver, or the like.

[0010] When assembling an article, a worker holds a tool in his / her hand and tightens a screw. The image capturing unit 1 captures an image of the left or right hand of the worker holding the tool. For example, the image capturing unit 1 includes a camera that captures an RGB image and a depth image.

[0011] The processing unit 2 receives successive images (video) captured by the imaging unit 1. The processing unit 2 recognizes the left or right hand in the image. Hand tracking is used to recognize the left or right hand. Hereinafter, when there is no particular distinction between the left hand and the right hand, the term "hand" will simply refer to at least either the left hand or the right hand.

[0012] The processing unit 2 receives designation of the fastening point of the screw through the video. As a specific example, the fastening point is indicated by a hand gesture using the worker's hand (e.g., finger). When the worker points to the fastening point, the processing unit 2 recognizes the hand movement and recognizes a first position of the finger tip. The processing unit 2 receives the first position as the fastening point of the screw and measures the first position.

[0013] The processing unit 2 further recognizes a second position of the hand holding the tool (first tool) and measures the second position. For example, the hand holding the tool or the dominant hand of the worker is registered in advance. The processing unit 2 recognizes the registered hand and measures its position as the second position. More specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may be used as the position of the hand. Or, the position of the center of the entire hand may be used as the position of the hand.

[0014] The first position and the second position are each represented by coordinates in a predetermined three-dimensional coordinate system. The order of measuring the first position and the second position is arbitrary. The first position may be measured after the second position, or the second position may be measured after the first position.

[0015] The processing unit 2 calculates a first distance between the first position and the second position. The first distance when the screw is fastened corresponds to the length of a tool used to fasten the screw. The processing unit 2 determines whether the fastening operation being performed is appropriate based on the first distance.

[0016] The input unit 3 is used by the worker to input information to the processing unit 2. The input unit 3 includes a microphone. The worker can input information to the processing unit 2 by speaking into the input unit 3. For example, a voice corresponding to a voice command is input to the input unit 3 at a timing when the processing unit 2 is to measure a first position, a timing when the processing unit 2 is to measure a second position, etc. In addition to the input unit 3, the worker can input information to the processing unit 2 by hand gestures, etc.

[0017] The display unit 4 displays information to the worker. For example, the display unit 4 displays the left hand and right hand recognized by the processing unit 2, a virtual object indicating a first position, a virtual object indicating a second position, etc. When the processing unit 2 determines that the fastening work is inappropriate, information indicating that the fastening work is inappropriate may be displayed on the display unit 4.

[0018] The memory unit 5 stores data necessary for the processing of the processing unit 2, data obtained by the processing of the processing unit 2, etc. For example, data on tools used in the work is registered in the memory unit 5. The registered data includes information necessary to identify the length of the tool, such as the tool model. The length of the tool may also be registered. The processing unit 2 compares the tool length obtained from the data in the memory unit 5 with the first distance to determine whether the fastening work is appropriate.

[0019] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. The processing system 10 shown in Fig. 1 is realized as, for example, a mixed reality (MR) device. The MR device 100 shown in Fig. 2 includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0020] The image camera 131 and the depth camera 132 are an example of the imaging unit 1. The processing device 150 is an example of the processing unit 2. The microphone 141 is an example of the input unit 3. The projection device 121 and the projection device 122 are an example of the display unit 4. The storage device 170 is an example of the storage unit 5.

[0021] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0022] The projection device 121 and the projection device 122 display the recognition result of the body of the worker, a virtual object indicating a first position, a virtual object indicating a second position, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0023] The lenses 111 and 112 are optically transparent. The worker can view the real situation through the lenses 111 and 112. The worker can also view information projected onto the lenses 111 and 112 by the projection devices 121 and 122. Information (virtual space) is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0024] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0025] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0026] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0027] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0028] FIG. 3 is a schematic diagram illustrating an example of an object to be worked on. For example, a fastening operation is performed on an article 200 shown in Fig. 3. The article 200 is a cylindrical hollow member, and has fastening points 201 to 208. A worker uses a wrench to fasten screws into each of the fastening points 201 to 208.

[0029] A marker 210 is provided near the work target. In the illustrated example, the marker 210 is an AR marker. As described below, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 150 sets a three-dimensional coordinate system based on a plurality of points indicated by a hand gesture.

[0030] FIGS. 4, 5(a), 5(b), 6(a), 6(b), 7(a), 7(b), and 8 are schematic views for explaining the first embodiment of the present invention. For example, a worker wearing the MR device 100 performs a fastening operation on an article 200. The worker can view the article 200 and his / her own hands through the lenses 111 and 112.

[0031] At the start of the fastening operation, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a three-dimensional coordinate system based on the position and orientation of the marker 210.

[0032] The image camera 131 and the depth camera 132 capture images of the article 200, the left hand 301, and the right hand 302. The processing device 150 recognizes the left hand 301 and the right hand 302 from the captured images. The processing device 150 causes the projection device 121 and the projection device 122 to display the recognition results on the lens 111 and the lens 112. Hereinafter, the processing device causing the projection device to display information on the lens is also simply referred to as "the processing device displays information."

[0033] 4, for example, the processing device 150 displays the recognition results of the left hand 301 and the right hand 302 superimposed on the hands in real space. In the illustrated example, a plurality of virtual objects 401 and a plurality of virtual objects 402 are displayed as the recognition results of the left hand 301 and the right hand 302. The plurality of virtual objects 401 respectively represent the plurality of joints of the left hand 301. The plurality of virtual objects 402 respectively represent the plurality of joints of the right hand 302. Instead of the joints, virtual objects (meshes) respectively representing the surface shapes of the left hand 301 and the right hand 302 may be displayed.

[0034] 5(a) to 8(b) are schematic diagrams showing a part of the field of vision of a worker. The worker places a screw 215 at one of the fastening points 201 to 208. As shown in FIG. 5(a), the worker points with a finger at the head of the placed screw 215. If the screw 215 is short, the fastening point may be pointed to before placing the screw 215. The processing device 150 recognizes the hand gesture by pointing. The processing device 150 accepts a first position of the tip of the finger when pointing as a position indicating the fastening point, and measures the first position. The worker may specify the timing for measuring the first position by a voice command. When the processing device 150 accepts the voice command, it measures the first position at that timing.

[0035] As shown in Fig. 5(b), the processing device 150 displays a first virtual object 411 at the measured first position. In the illustrated example, the first virtual object 411 is a sphere. The shape, color, and the like of the first virtual object 411 are arbitrary as long as the worker can visually distinguish it from the real space. The processing device 150 may display coordinates 421 of the first virtual object 411.

[0036] As shown in FIG. 6(a), the worker moves the right hand 302 holding the wrench 310 to a working position for fastening the screw 215. The processing device 150 measures the second position of the right hand 302. The worker may specify the timing for measuring the second position by voice. When the processing device 150 receives the voice command, it measures the second position at that timing. Alternatively, the worker may indicate the timing for measuring the second position to the processing device 150 by a hand gesture. For example, the worker opens the right hand 302 holding the wrench 322 toward the image camera 131 and the depth camera 132 and then grips it. When the processing device 150 recognizes this hand gesture, it measures the second position at that timing.

[0037] As shown in Fig. 6(b), the processing device 150 displays a second virtual object 412 at the measured second position. In the illustrated example, the second virtual object 412 is a sphere. The shape, color, and the like of the second virtual object 412 are arbitrary as long as the worker can visually distinguish it from the real space. The processing device 150 may display coordinates 422 of the second virtual object 412.

[0038] The processing device 150 calculates a first distance between the first position and the second position. The first distance corresponds to the length of the wrench 322 held by the operator. As shown in FIG. 7(a), the processing device 150 may display the first distance 424.

[0039] For example, the processing device 150 calculates a first error between a first length registered in advance and a first distance. If the first error is greater than a first threshold value set in advance, the processing device 150 determines that the fastening work performed by the worker is inappropriate. If the first error is equal to or smaller than the first threshold value, the processing device 150 determines that the fastening work performed by the worker is appropriate.

[0040] Examples of cases in which the work is inappropriate include a case in which the worker holds an inappropriate tool, and a case in which the worker is trying to fasten a fastening point other than the one pointed to. If the first error is greater than the first threshold, the processing device 150 may display a message indicating that the fastening work is inappropriate. For example, as shown in FIG. 7(b), the processing device 150 displays a message 430 prompting the user to check the tool or the fastening point. Instead of the message, an error code or the like may be displayed.

[0041] The tool length compared with the first distance may be another value calculated using the registered tool length. For example, the registered tool length may be multiplied by a predetermined ratio taking into account the grip position of the tool, etc. The processing device 150 compares the value obtained by the multiplication with the first distance.

[0042] The first threshold value is set in advance by an administrator of the processing system 10 or the like. Alternatively, the first threshold value may be calculated based on the first length of the referenced tool by the processing device 150. For example, a value obtained by multiplying the first length by a predetermined ratio may be used as the first threshold value.

[0043] When the order of fastening the screws for the fastening points 201 to 208 is determined in advance, the processing device 150 may determine whether the pointed fastening point is appropriate. For example, the storage device 170 stores a work procedure. The work procedure includes the order of fastening the screws for the fastening points 201 to 208, the coordinates of each of the fastening points 201 to 208, and the like. The coordinates of the fastening points 201 to 208 are expressed in a three-dimensional coordinate system based on the marker 210. The three-dimensional coordinate system used for the coordinates of the fastening points 201 to 208 is the same as the three-dimensional coordinate system set during work.

[0044] After measuring the first position, the processing device 150 refers to the coordinates of the fastening point to be fastened. The processing device 150 calculates an error between the first position and the referred coordinates. If the error is equal to or less than a preset threshold, the processing device 150 determines that the pointed fastening point is appropriate. If the error is greater than the threshold, the processing device 150 determines that the pointed fastening point is inappropriate. The threshold may be preset by a person, or may be calculated by the processing device 150 based on the mutual distances of the fastening points 201 to 208.

[0045] For example, if the processing device 150 determines that the pointed fastening location is appropriate, it accepts the measured first position and displays the first virtual object at the first position. If the processing device 150 determines that the fastening location is inappropriate, it does not accept the measured first position and does not display the first virtual object.

[0046] When the pointed fastening location is not appropriate, the processing device 150 may display a message. Fig. 8(a) shows an example in which the worker points to fastening location 204 when fastening location 203 should be performed. As shown in Fig. 8(a), the processing device 150 may display a message 431 prompting the worker to confirm whether the fastening location to be worked on is appropriate. The processing device 150 may display a virtual object 415 at the fastening location to be fastened. The color or shape of the virtual object 415 may be different from the color or shape of the other virtual objects so that the virtual object 415 can be easily distinguished from the first virtual object 411 or the second virtual object 412.

[0047] When the pointed fastening point is suitable, the processing device 150 may display a message 432 indicating that the fastening point is suitable, as shown in Fig. 8(b). When the fastening operation is suitable, the processing device 150 may display a message 433 indicating that the tool is suitable.

[0048] FIG. 9 is a flowchart illustrating the processing method according to the first embodiment. 9, when a fastening operation is started, a worker indicates a fastening location by a hand gesture. The processing unit 2 recognizes the hand gesture (step S1). The processing unit 2 measures a first position indicated by the hand gesture (step S2).

[0049] The processing unit 2 refers to a preregistered work procedure and acquires the coordinates of the location to be fastened. The processing unit 2 determines whether the error between the first position and the referenced coordinates of the fastening location is equal to or smaller than a threshold (step S3). If the error is greater than the threshold, the processing unit 2 displays a message on the display unit 4 prompting the user to confirm the fastening location (step S4).

[0050] If the error is equal to or less than the threshold, the worker places the hand holding the tool at the working position. The worker indicates by a voice command that the hand is at the working position. The processing unit 2 accepts the voice command (step S5) and measures the second position (step S6). The processing unit 2 calculates a first distance between the first position and the second position (step S7). The processing unit 2 determines whether a first error between a first length of the tool registered in advance and the first distance is equal to or less than a first threshold (step S8). If the first error is greater than the first threshold, the processing unit 2 displays a message indicating that the work is not appropriate on the display unit 4 (step S9). If the first error is equal to or less than the first threshold, the processing unit 2 displays on the display unit 4 that the work is appropriate (step S10).

[0051] The advantages of the first embodiment will be described. When manufacturing an article, a screw tightening operation may be performed. It is desirable that the screw is tightened with an appropriate tool for each tightening location. If an inappropriate tool is used to tighten a screw, a defect may occur. For example, if a short tool is used when a long tool should be used, the torque applied to the screw may be small, and the tightening force may be insufficient. The insufficient tightening force reduces the reliability of the article. If a long tool is used when a short tool should be used, it may be difficult to tighten the screw. In addition, the tool may interfere with other objects, hindering the work or damaging other objects. There is also a possibility that the work efficiency may decrease. Furthermore, depending on the article, the order of tightening the screws may be determined in advance. In such a case, a worker may inadvertently tighten a screw in a different location from the intended one. Conventionally, the tightening operation of the screw is confirmed between multiple workers using a checklist or the like, but oversights may occur. In addition, since it takes time for the worker to check, the lead time for manufacturing the article is long.

[0052] Regarding this problem, in the first embodiment, after an image of a worker's hand is captured, the processing unit 2 measures a first position and a second position. The first position corresponds to a position where the screw is to be fastened. The second position corresponds to a position of the hand holding the tool. The processing unit 2 determines whether the fastening work is appropriate based on a first distance between the first position and the second position. The processing by the processing unit 2 makes it possible to prevent a screw from being fastened using a tool other than a predetermined tool. Also, it makes it possible to prevent a screw from being erroneously fastened at a fastening position other than the fastening position corresponding to the first position.

[0053] Furthermore, the processing of the processing unit 2 makes it possible to determine whether the work is appropriate before the worker tightens the screw. This makes it possible to prevent the worker from having to tighten the screw again. Furthermore, the processing of the processing unit 2 can be performed in real time according to the worker's movements. This makes it possible to reduce the impact on the lead time. According to the first embodiment, it is possible to prevent inappropriate work from being performed while preventing an extension of the lead time.

[0054] As a reference example, in order to determine whether a tool is appropriate, a method of recognizing a tool from an image and determining whether the recognized tool is appropriate can be considered. However, it is difficult to calculate the length of the tool in three-dimensional space using image recognition. In addition, since the tool is held by a person during work, part of the tool is hidden by the hand and cannot be recognized. For this reason, it is difficult to calculate the exact length of the tool. According to the embodiment, it is possible to more accurately determine whether the tool held by the worker is appropriate.

[0055] Furthermore, image recognition requires a large amount of calculations. This places a large load on the processing unit 2, and the processing unit 2 and the MR device may become large. Since the MR device is worn by a person, it is desirable that it is small and lightweight. According to the first embodiment of the present invention, it is possible to determine whether the task is appropriate with a smaller amount of calculations, and it is possible to suppress an increase in the size of the processing unit 2. For this reason, the first embodiment of the present invention is particularly suitable for MR devices that require miniaturization.

[0056] Preferably, the processing unit 2 compares the first position with a preregistered position of the fastening point of the screw. The processing unit 2 determines whether the point where the worker is trying to fasten the screw is appropriate. This makes it possible to prevent the screw from being fastened at an incorrect position. For example, there are cases where screws are fastened at multiple fastening points in a predetermined order. Even in such a case, the processing unit 2 determines whether the fastening point is appropriate, making it possible to prevent the screw from being fastened at an incorrect position.

[0057] The larger the item, the more points the screws need to be fastened. In particular, for indented products (made-to-order products), the parts used and manufacturing procedures may differ for each item. This makes it more likely that exceptional work not listed on the checklist will occur or that something will be overlooked. The first embodiment of the present invention is particularly suitable for manufacturing indented products.

[0058] If the work is judged to be inappropriate, or if the fastening location is judged to be inappropriate, the processing unit 2 outputs information (message) indicating the content. The worker can recognize from the message that the tool or the fastening location is inappropriate.

[0059] The first position is indicated by, for example, a hand movement (hand gesture). The processing unit 2 measures the position indicated by the operator's hand gesture as the first position. By indicating the position to be measured, it is possible to prevent a position unintended by the operator from being measured as the first position. This improves the usability of the MR device.

[0060] Furthermore, the timing for measuring the first position may be indicated by voice. The processing unit 2 measures the first position at the indicated timing. By indicating the timing for measurement, it is possible to prevent the first position from being measured at a timing unintended by the operator. This can improve the usability of the processing system 10.

[0061] It is preferable that the timing for measuring the second position is also indicated by a hand movement or a voice. The processing unit 2 measures the second position at the indicated timing. By indicating the timing for measurement, it is possible to prevent the second position from being measured at a timing unintended by the operator. This can improve the usability of the MR device.

[0062] The tool may be a digital tool. The digital tool includes a sensor that detects a signal generated by the fastening operation. The processing unit 2 receives a detection value from the digital tool when the fastening operation is performed. If the fastening operation is appropriate, the processing unit 2 associates the detection value with the pointed fastening location and stores it in the storage unit 5. This makes it possible to automatically create a work record for each fastening location. When an operator records a detection value, it is possible that the detection value is erroneously recorded for a fastening location other than the fastening location actually worked on. According to the first embodiment, it is possible to automatically create a more accurate work record.

[0063] For example, a digital torque wrench or a digital torque driver is used as the digital tool. When such a digital tool is used, the detected value is torque. Alternatively, a threshold value may be set in advance for the torque. The digital torque wrench or the digital torque driver may determine whether or not a torque exceeding the threshold value is detected, and output the determination result as the detected value.

[0064] Second embodiment The second embodiment of the present invention is applied to a fastening operation in which a plurality of tools are used. For example, when fastening a screw, an extension bar and a wrench are used. The wrench fastens the screw via the extension bar.

[0065] 10(a) and 10(b) are schematic diagrams for explaining a second embodiment of the present invention. Here, an example will be described in which a worker wearing the MR device 100 performs fastening work on an article 200 using an extension bar and a wrench.

[0066] First, the hand of the worker is recognized in the same manner as in the example shown in Fig. 4. As in the examples shown in Fig. 5(a) and Fig. 5(b), a first position pointed to by the worker is measured, and a first virtual object 411 indicating the first position is displayed.

[0067] 10(a), the worker places one end of the extension bar 321 on the fastening point that he or she has pointed to. The worker places a wrench 322 on the other end of the extension bar 321. In the illustrated example, a first virtual object 411 is displayed at the fastening point 204. Also, one end of the wrench 322 that is in contact with the extension bar 321 is held by the right hand 302. The other end of the wrench 322 is held by the left hand 301.

[0068] The processing device 150 measures the second position of the right hand 302 and the third position of the left hand 301. For example, the timing of measuring the second position and the third position is indicated by a voice command or a hand gesture. A voice command for causing the processing device 150 to measure the second position and a voice command for causing the processing device 150 to measure the third position may be prepared separately.

[0069] 10(b), the processing unit 2 displays a first virtual object 411 to a third virtual object 413 at a first position to a third position, respectively. The colors or shapes of the first virtual object 411 to the third virtual object 413 are not limited to the illustrated example and may be arbitrary. The processing device 150 may display coordinates 421 to 423 of the first virtual object 411 to the third virtual object 413.

[0070] It is specified in advance whether the left hand 301 and the right hand 302 correspond to the second position or the third position, respectively. Alternatively, the processing device 150 may set it based on the worker's handedness. For example, when a fastening operation is performed, the processing device 150 refers to a database in which data of the worker is recorded. The processing device 150 acquires data of the worker performing the fastening operation. The acquired data includes the worker's handedness. The processing device 150 measures the position of the dominant hand as the third position, and measures the position of the non-dominant hand as the second position.

[0071] Alternatively, the processing device 150 may determine the second and third positions based on the positional relationship between the first position, the position of the left hand 301, and the position of the right hand 302. As an example, the processing device 150 compares the distance between the first position and the left hand 301 with the distance between the first position and the right hand 302. The processing device 150 determines the hand position with the shorter distance as the second position, and the hand position with the longer distance as the third position. As another example, the extension bar 321 is often used upright. The processing device 150 compares the horizontal distance between the first position and the left hand 301 with the horizontal distance between the first position and the right hand 302. The processing device 150 determines the hand position with the shorter distance as the second position, and the hand position with the longer distance as the third position.

[0072] The processing device 150 calculates a first distance between the first position and the second position. Also, the processing device 150 calculates a second distance between the second position and the third position. The first distance corresponds to the length of the extension bar 321. The second distance corresponds to the length of the wrench 322. As shown in FIG. 10(b), the processing device 150 may display the first distance 424 and the second distance 425.

[0073] The processing device 150 calculates a first error between a first length registered in advance and the first distance. The first length is the length of the extension bar to be used in the fastening work. Another value calculated using the first length may be used as the first length.

[0074] The processing device 150 also calculates a second error between a second length registered in advance and the second distance. The second length is the length of a wrench to be used in the fastening operation. Another value calculated using the second length may be used as the second length.

[0075] When the first error is greater than the first threshold value or when the second error is greater than the second threshold value, the processing device 150 determines that the fastening work by the worker is inappropriate. When the first error is equal to or less than the first threshold value and the second error is equal to or less than the second threshold value, the processing device 150 determines that the fastening work is appropriate. The second threshold value may be the same as the first threshold value or may be different from the first threshold value.

[0076] If the first error is greater than the first threshold or if the second error is greater than the second threshold, the processing unit 150 may display a message. The message displayed when the first error is greater than the first threshold may be different from the message displayed when the second error is greater than the second threshold.

[0077] 11(a) and 11(b) are schematic diagrams showing an output example in the second embodiment of the present invention. For example, when the first error is greater than the first threshold, the processing device 150 displays a message 435 prompting the user to check the extension bar 321 or the fastening point as shown in Fig. 11(a). When the second error is greater than the second threshold, the processing device 150 displays a message 436 prompting the user to check the wrench 322 as shown in Fig. 11(b).

[0078] FIG. 12 is a flowchart illustrating a processing method according to the second embodiment. A processing method M2 shown in Fig. 12 further includes steps S11 to S14 compared to the processing method M1 shown in Fig. 9. First, steps S1 to S3 are executed similarly to the processing method M1. In step S3, if the error is equal to or smaller than the threshold, the worker moves the first tool and the second tool to the working positions for fastening the screw. The worker indicates by a voice command that both hands are at the working positions. The processing unit 2 accepts the voice command (step S5).

[0079] The processing unit 2 recognizes a second position of one of the left hand 301 and the right hand 302 (step S6). The processing unit 2 recognizes a third position of the other of the left hand 301 and the right hand 302 (step S11). The processing unit 2 calculates a first distance between the first position and the second position (step S7). The processing unit 2 calculates a second distance between the second position and the third position (step S12).

[0080] The processing unit 2 determines whether a first error between a first length of a first tool registered in advance and the first distance is equal to or less than a first threshold (step S8). The processing unit 2 determines whether a second error between a second length of a second tool registered in advance and the second distance is equal to or less than a second threshold (step S13). If the first error is greater than the first threshold or if the second error is greater than the second threshold, the processing unit 2 displays a message on the display unit 4 (step S9 or S14). If the first error is equal to or less than the first threshold and the second error is equal to or less than the second threshold, the processing unit 2 displays on the display unit 4 that the fastening operation (fastening location and tool) is appropriate (step S10).

[0081] According to the second embodiment, even if a plurality of tools are used in the fastening work, it is possible to determine whether the fastening work is appropriate. According to the second embodiment, like the first embodiment, it is possible to prevent the lead time from being extended and prevent the performance of inappropriate work.

[0082] The timings for measuring the first to third positions are preferably indicated by hand movements or voice. The processing unit 2 measures each position at the indicated timings. This makes it possible to prevent each position from being measured at a timing unintended by the operator, thereby improving the usability of the processing system 10.

[0083] (First Modification) FIG. 13 is a flowchart illustrating a processing method according to a first modified example of the second embodiment. In the processing method M2 shown in Fig. 12, the determination of the fastening point may be omitted. For example, as in the processing method M2a shown in Fig. 13, steps S1 to S4 may be omitted and the processing may start from step S5. In this case, it is determined whether the first tool and the second tool are each appropriate.

[0084] (Second Modification) FIG. 14 is a schematic diagram for explaining a second modified example of the second embodiment. There are cases where the fastening point of a screw is located in a position that is difficult to see. In the example shown in Fig. 14, a screw is fastened to a cylindrical object 220. The inside of the object 220 is hollow. There are multiple fastening points 221-226 at the bottom of the cavity. The worker places a screw in one of the fastening points and inserts an extension bar 321 into the cavity. The worker uses the extension bar 321 and a wrench 322 to fasten the screw.

[0085] FIG. 15 is a flowchart illustrating a processing method according to a second modification of the second embodiment. A part of the extension bar 321 and the fastening point where the screw is fastened are not captured by the imaging unit 1. For this reason, similarly to the first modified example, steps S1 to S4 are omitted in the processing method M2b shown in Fig. 15. Moreover, compared to the processing method M2a shown in Fig. 13, the processing method M2b further includes steps S15 and S16.

[0086] After steps S6 and S11 are executed, the processing unit 2 calculates the error between the height of the second position and the height of the third position. The processing unit 2 determines whether the height error is equal to or less than a preset threshold value (step S15). As described above, the extension bar 321 is often used standing vertically. The wrench 322 is applied vertically to the extension bar 321. Therefore, when the posture of each tool is appropriate, the height of the hand holding one end of the wrench 322 and the height of the hand holding the other end of the wrench 322 are approximately the same.

[0087] If the height error is greater than the threshold, the processor 2 displays a message indicating that the work is not appropriate (step S16). The processor 2 may also display a message indicating that the tool posture is not appropriate. If the height error is equal to or less than the threshold, steps S7 and after are executed as in the processing method M2.

[0088] According to the second modification of the second embodiment, it is possible to determine whether the worker uses the tool appropriately. In the processing method M2 shown in Fig. 12, steps S15 and S16 may be further executed. This makes it possible to determine whether the fastening point is appropriate or not, as well as whether the tool is used appropriately or not. It is possible to encourage the worker to perform a more appropriate fastening operation.

[0089] Third embodiment 16 and 17 are schematic diagrams for explaining a third embodiment of the present invention. The third embodiment of the present invention is used after the start of a fastening operation. First, the first position to the third position are measured by the second embodiment of the present invention. If the operation is judged to be appropriate, the fastening of the screw is started. In the fastening operation using the extension bar 321 and the wrench 322, as shown in FIG. 16, the wrench 322 rotates around the extension bar 321 along a horizontal plane.

[0090] When the first to third positions are measured and the work is determined to be appropriate, the processing unit 2 sets one or more pass points. The pass points are set at positions through which the hand holding the wrench 322 passes when the fastening work is performed appropriately. The rotation direction of the wrench 322 is determined based on the direction of the thread groove at the fastening location.

[0091] In the illustrated example, the screw groove at the fastening portion is oriented in the opposite direction to a general screw groove. Therefore, the wrench 322 is rotated counterclockwise when fastening the screw. For example, the processing unit 2 sets two passing points. As shown in Fig. 17, the processing unit 2 may display a virtual object 441 and a virtual object 442 at the positions of the set passing points.

[0092] When the fastening operation is started, the processing unit 2 judges whether the hand holding one end of the wrench 322 has passed through the passing point. When the hand has passed through the passing point, the processing unit 2 judges that the operation is being performed properly. The processing unit 2 may display that the hand has passed through the passing point or that the operation is being performed properly.

[0093] FIG. 18 is a flowchart showing a processing method according to the third embodiment. In the processing method M3 shown in Fig. 18, first, the processing unit 2 measures the third position (step S21). The processing unit 2 sets a plurality of passing points (step S22). For example, the processing unit 2 sets a first passing point and a second passing point. More passing points may be set.

[0094] The first passing point is located between the third position and the second passing point. The processing unit 2 calculates a first interval between the first passing point and the third position (step S23). The processing unit 2 determines whether the first interval is equal to or smaller than a preset threshold (step S24). If the first interval is greater than the threshold, step S21 is executed again, and the third position is measured again.

[0095] If the first interval is equal to or less than the threshold, the processing unit 2 displays that the wrench 322 has passed the first passing point (step S25). The processing unit 2 measures the third position (step S26). The processing unit 2 calculates a second interval between the second passing point and the third position (step S27). The processing unit 2 determines whether the second interval is equal to or less than a preset threshold (step S28). If the second interval is greater than the threshold, step S26 is executed again, and the third position is measured again. If the second interval is equal to or less than the threshold, the processing unit 2 displays that the wrench 322 has passed the second passing point (step S29).

[0096] When three or more passing points are set, the same process is executed after step S29. According to the third embodiment, it is possible to prompt the worker to use the tool appropriately.

[0097] FIG. 19 is a flowchart showing an outline of data processing in the processing method according to the embodiment. The storage unit 5 stores master data 51 and history data 52. The master data 51 includes operation master data 51a, tool master data 51b, base point master data 51c, and fastening point master data 51d. The master data 51 is prepared in advance before the screw tightening.

[0098] The processing unit 2 refers to the fastening work to be performed and the tool to be used (step S31). The fastening work and the tool may be selected by the worker or may be instructed by a higher-level system. The work master data 51a stores the work ID, work name, item ID, and item name related to the fastening work. The tool master data 51b stores the tool type and tool ID. The tool type indicates the classification of the tool based on the structure, external shape, performance, etc. By identifying the tool type, the length of the tool is identified.

[0099] The processing unit 2 refers to the base point master data 51c. The base point master data 51c stores a method for setting a base point for each task. The processing unit 2 acquires the method for setting a base point for the task selected in step S31, and sets a base point according to the method (step S32). A marker or a hand gesture is used to set the base point.

[0100] Any of the processing methods described above is executed (step S33). During execution of the processing method, the processing unit 2 refers to the fastening point master data 51d. The fastening point master data 51d stores fastening point IDs, and stores fastening positions and torque values ​​for each fastening point ID. The fastening positions indicate the positions of each fastening point. The torque values ​​indicate the magnitude of torque required to screw-tighten each fastening point. The tool length identified by the tool type registered in the tool master data 51b is compared with each distance. This allows the processing unit 2 to determine whether the work is appropriate.

[0101] When a digital tool is used, the processing unit 2 receives data from the digital tool. The processing unit 2 links the received data with a job ID, data related to the fastening location, data related to the tool, and the like, and records the data in the history data 52 (step S34). For example, a torque value is transmitted from the digital tool. The processing unit 2 links the torque value with the job ID, fastening location ID, the model of the tool used, and the tool ID, and records it.

[0102] The processor 2 determines whether the task selected in step S31 has been completed. If the task has not been completed, steps S33 and S34 are executed again.

[0103] FIG. 20 is a schematic diagram showing a hardware configuration. 20 is used as the processing unit 2. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0104] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0105] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0106] The storage device 94 stores data necessary for executing the program and data obtained by executing the program. The storage device 94 includes a solid state drive (SSD) and the like. The storage device 94 may be used as the storage unit 5 shown in FIG. 1.

[0107] An input interface (I / F) 95 can connect the computer 90 to the input unit 3. The CPU 91 can read various data from the input unit 3 via the input I / F 95.

[0108] An output interface (I / F) 96 can connect the computer 90 to the display unit 4. The CPU 91 can transmit data to the display unit 4 via the output I / F 96 and cause the display unit 4 to display information.

[0109] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0110] Data processing by the processing unit 2 or the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or another computer via the communication I / F 97.

[0111] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0112] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0113] In the above description of the embodiment, the processing system 10 is mainly realized as an MR device. The embodiment of the present invention is not limited to this example. The processing system 10 may be realized using a general-purpose personal computer. In this case, a keyboard, a microphone, a touchpad, etc. are used as the input unit 3. A monitor is used as the display unit 4. A general-purpose camera is used as the imaging unit 1. The camera captures the worker from the side or above so as to capture the worker's hands. The processing unit 2 executes each of the above-mentioned processes using the video captured by the camera.

[0114] Embodiments of the invention include the following features. (Feature 1) An imaging unit for acquiring an image; a processing unit that recognizes a hand of a worker performing a fastening operation from the image, and determines whether the fastening operation is appropriate based on a first distance between a first position corresponding to a fastening location of a screw and a second position of the hand holding a first tool; A mixed reality device with (Feature 2) The processing unit includes: Calculating a first error between a first length of the first tool, which is registered in advance, and the first distance; When the first error is greater than a first threshold value, it is determined that the fastening operation is not appropriate. 2. The mixed reality device according to feature 1. (Feature 3) The mixed reality device according to feature 1 or 2, wherein the processing unit determines whether the first position is appropriate by comparing the first position with a previously registered position of the fastening point. (Feature 4) When the screw is fastened by the second tool via the first tool, the processing unit: Recognizing the two hands of the worker; determining whether the second tool is appropriate based on a second distance between the second position of one of the two hands and a third position of the other of the two hands; A mixed reality device according to any one of features 1 to 3. (Feature 5) The mixed reality device of feature 4, wherein the processing unit determines that the fastening operation is inappropriate when a difference between the height of the second position and the height of the third position exceeds a threshold value. (Feature 6) The mixed reality device according to any one of features 1 to 5, wherein the processing unit measures the first position at a timing indicated by a hand gesture or a voice command. (Feature 7) Further comprising a display unit, The mixed reality device according to any one of features 1 to 6, wherein the processing unit causes the display unit to display a first object indicating the first position and a second object indicating the second position. (Feature 8) A mixed reality device described in any one of features 1 to 7, wherein when a marker is photographed by the photographing unit, the processing unit sets a three-dimensional coordinate system with the marker as a base point and measures the first position and the second position in the coordinate system. (Feature 9) The mixed reality device according to any one of features 1 to 8, wherein when the processing unit receives a detection value when the screw is tightened, the processing unit associates the detection value with data regarding the tightening location and stores the received detection value. (Feature 10) displaying, on a display unit, a recognition result of the worker's left hand, a recognition result of the worker's right hand, a first object indicating a first position corresponding to a fastening location of a screw, and a second object indicating a second position of one of the left hand and the right hand, superimposed on a real space; A mixed reality device that displays a message on the display unit when the tool held in one of the left hand and the right hand is not appropriate. (Feature 11) The processing device includes: A first position corresponding to a fastening point of the screw is measured; Recognizing a left hand and a right hand of a worker from an image of the worker holding a first tool; determining whether a fastening operation performed by the worker is appropriate based on a first distance between the first position and a second position of one of the left hand and the right hand; Processing methods. (Feature 12) 12. A processing device for carrying out the processing method according to feature 11. (Feature 13) A program for causing the processing device to execute the processing method according to feature 11. (Feature 14) A storage medium storing the program according to feature 13.

[0115] According to the embodiments described above, a mixed reality device, a processing method, a processing device, a program, and a storage medium are provided that are capable of suppressing the execution of inappropriate tasks.

[0116] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0117] 1: imaging unit, 2: processing unit, 3: input unit, 4: display unit, 5: memory unit, 10: processing system, 51: master data, 51a: work master data, 51b: tool master data, 51c: base point master data, 51d: fastening point master data, 52: history data, 90: computer, 91: CPU, 92: ROM, 93: RAM, 94: memory device, 95: input interface, 96: output interface, 97: communication interface, 98: system bus, 100: MR device, 101: frame, 111, 112: lens, 121, 122: projection device, 131: image camera, 132: depth camera, 140: sensor, 141: microphone, 150: processing device, 160: battery, 170: memory device, 200: item, 201-208: fastening point, 210: marker, 215: screw, 220: object, 221-226: fastening point, 301: left hand, 302: right hand, 310: wrench, 321: extension bar, 322: wrench, 401, 402: virtual object, 411: first virtual object, 412: second virtual object, 413: third virtual object, 415: virtual object, 421-423: coordinates, 424: first distance, 425: second distance, 430-433, 435, 436: message, 441, 442: virtual object, M1, M2, M2a, M2b, M3: processing method

Claims

1. An imaging unit for acquiring an image; a processing unit that recognizes a hand of a worker performing a fastening operation from the image, and determines whether the fastening operation is appropriate based on a first distance between a first position corresponding to a fastening location of a screw and a second position of the hand holding a first tool; A mixed reality device with

2. The processing unit includes: Calculating a first error between a first length of the first tool, which is registered in advance, and the first distance; When the first error is greater than a first threshold value, it is determined that the fastening operation is not appropriate. The mixed reality device of claim 1 .

3. The mixed reality device according to claim 1 , wherein the processing unit determines whether the first position is appropriate by comparing the first position with pre-registered positions of the fastening points.

4. When the screw is fastened by a second tool via the first tool, the processing unit: Recognizing the two hands of the worker; determining whether the second tool is appropriate based on a second distance between the second position of one of the two hands and a third position of the other of the two hands; A mixed reality device according to any one of claims 1 to 3.

5. The mixed reality device according to claim 4 , wherein the processing unit determines that the fastening operation is inappropriate when a difference between a height of the second position and a height of the third position exceeds a threshold value.

6. The mixed reality device according to claim 1 , wherein the processing unit measures the first position at a timing indicated by a hand gesture or a voice command.

7. Further comprising a display unit, The mixed reality device according to any one of claims 1 to 3, wherein the processing unit displays a first object indicating the first position and a second object indicating the second position on the display unit.

8. A mixed reality device described in any one of claims 1 to 3, wherein when a marker is photographed by the photographing unit, the processing unit sets a three-dimensional coordinate system with the marker as a base point and measures the first position and the second position in the coordinate system.

9. The mixed reality device according to any one of claims 1 to 3, wherein when the processing unit receives a detection value when the screw is tightened, the processing unit links the detection value to data regarding the tightening location and stores the data.

10. displaying, on a display unit, a recognition result of the worker's left hand, a recognition result of the worker's right hand, a first object indicating a first position corresponding to a fastening location of a screw, and a second object indicating a second position of one of the left hand and the right hand, superimposed on a real space; A mixed reality device that displays a message on the display unit when the tool held in one of the left hand and the right hand is not appropriate.

11. The processing device includes: A first position corresponding to a fastening point of the screw is measured; Recognizing a left hand and a right hand of a worker from an image of the worker holding a first tool; determining whether a fastening operation performed by the worker is appropriate based on a first distance between the first position and a second position of one of the left hand and the right hand; Processing methods.

12. A processing device for carrying out the processing method according to claim 11.

13. A program for causing the processing device to execute the processing method according to claim 11.

14. A storage medium storing the program according to claim 13.

Images