Processing device, mixed reality device, processing method, program, and storage medium

Abstract

The invention provides a processing device, a mixed reality device, a processing method, a program, and a storage medium capable of assisting screw screwing work. A processing device according to an embodiment of the present invention measures coordinates of a hand, which screws a screw using a first tool, on the basis of a plurality of images obtained by continuously capturing images of the hand. The processing device calculates a center coordinate of rotation of the first tool using the plurality of coordinates. Preferably, the processing device estimates the position of the screw on the basis of the center coordinates, and refers to the pre-registered position of the fastening portion where the screw is screwed. When the distance between the position of the fastening portion and the estimated position of the screw is equal to or less than a threshold value, the processing device estimates that the screw is screwed with respect to the fastening portion.

Description

Technical Field

[0001] Embodiments of the present invention relate to a processing device, a mixed reality device, a processing method, a program, and a storage medium. Background Art

[0002] When manufacturing an object, screws are sometimes tightened or loosened, and technology is required to assist in the work of tightening these screws.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2021 / 176645 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The problem to be solved by the present invention is to provide a processing device, a mixed reality device, a processing method, a program and a storage medium that can assist the operation of tightening a screw.

[0008] Means for solving problems

[0009] The processing device of the embodiment measures the coordinates of the hand based on a plurality of images obtained by continuously photographing the hand tightening a screw using the first tool. The processing device calculates the center coordinates of the rotation of the first tool using the plurality of the coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram showing the configuration of the processing system according to the first embodiment.

[0011] Figure 2 is a schematic diagram of a mixed reality device of an exemplary embodiment.

[0012] Figure 3 It is a schematic diagram which illustrates the object of a work.

[0013] Figure 4 It is a schematic diagram showing an output example of the processing device according to the embodiment.

[0014] Figure 5 This is a diagram showing what the work looks like.

[0015] Figure 6 This is a schematic diagram showing what the work looks like.

[0016] Figure 7 This is a diagram showing what the work looks like.

[0017] Figure 8 This is a diagram showing what the work looks like.

[0018] Fig. 9 This is a diagram for explaining the method of calculating the center coordinates.

[0019] Fig.10 This is a diagram for explaining the method of calculating the center coordinates.

[0020] Fig.11 This is a diagram for explaining the method of calculating the center coordinates.

[0021] Fig.12 It is a schematic diagram showing an output example of the processing device according to the embodiment.

[0022] Fig.13 It is a schematic diagram showing an output example of the processing device according to the embodiment.

[0023] Fig.14 This is an example of a database related to fastening locations.

[0024] Fig.15 (a) and Fig.15 (b) is a diagram for explaining the method of calculating the center coordinates.

[0025] Fig.16 This is a diagram for explaining the method of calculating the center coordinates.

[0026] Fig.17 This is a schematic diagram illustrating an example of a task.

[0027] Fig.18 This is a schematic diagram illustrating an example of a task.

[0028] Fig.19 This is a schematic diagram illustrating an example of a task.

[0029] Fig. 20 It is a flowchart showing the processing method of the implementation mode.

[0030] Fig.21 It is a schematic diagram showing an output example of the processing device according to the first modified example of the embodiment.

[0031] Fig. 22 This is a diagram showing what the work looks like.

[0032] Fig.23 This is a diagram showing what the work looks like.

[0033] Fig.24 It is a schematic diagram showing an output example of the processing device according to the first modified example of the embodiment.

[0034] Fig.25 It is a schematic diagram showing an output example of the processing device according to the first modified example of the embodiment.

[0035] Fig.26 It is a schematic diagram showing an output example of the processing device according to the first modified example of the embodiment.

[0036] Fig. 27 This is a flowchart showing a processing method according to a first variation of the embodiment.

[0037] Fig.28 It is a flowchart showing the processing method of the second modified example of the embodiment.

[0038] Fig.29 It is a schematic diagram for explaining the processing method of the second modified example of the embodiment.

[0039] Fig.30 This is a schematic diagram showing the hardware configuration. DETAILED DESCRIPTION

[0040] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. The accompanying drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the size between the parts, etc. may not be the same as in reality. Even when representing the same part, the size and ratio of each other are sometimes represented differently according to the accompanying drawings. In the specification of this application and each figure, the same symbols are marked for the same elements as those already described, and the detailed description is appropriately omitted.

[0041] Figure 1 It is a schematic diagram showing the configuration of the processing system according to the first embodiment.

[0042] The embodiments of the present invention are applicable to operations of tightening screws using tools. Figure 1 As shown, the processing system 10 of the embodiment includes a photographing device 1 , a processing device 2 , an input device 3 , a display device 4 , and a storage device 5 .

[0043] The camera 1 captures the work. In the work, a tool is used to tighten a fastener such as a screw on an object. The object is a component, unit or semi-finished product used to make a product. The tool is a wrench for tightening a screw. The wrench is a ratchet wrench, a socket wrench, a glasses wrench, a hexagonal wrench, etc. The specific type of wrench is appropriately selected according to the work.

[0044] When assembling an item, the operator holds a tool in his hand. The operator twists the tool to tighten the screw. Thus, the screw is tightened to a predetermined position. The camera 1 captures the left or right hand of the operator twisting the tool. For example, the camera 1 includes a camera that acquires an RGB image and a depth image.

[0045] The processing device 2 receives the continuous images (dynamic images) captured by the camera 1. The processing device 2 recognizes the left hand or the right hand in the image. In the recognition of the left hand or the right hand, hand tracking is used. Hereinafter, when the left hand and the right hand are not particularly distinguished, at least one of the left hand and the right hand is referred to as "hand".

[0046] The input device 3 is used for the operator to input information to the processing device 2. For example, the input device 3 includes a microphone. The operator can input information to the processing device 2 by speaking toward the input device 3. As an example, a sound corresponding to a voice command is input to the input device 3. In addition to the input device 3, the operator can also input information to the processing device 2 through gestures, etc.

[0047] The display device 4 displays information to the operator. For example, the processing device 2 causes the display device 4 to display information for assisting the work. Hereinafter, the case where the processing device causes the display device to display information is also referred to as "the processing device displays information".

[0048] The storage device 5 stores data required for processing by the processing device 2, data obtained by processing by the processing device 2, etc. For example, data of a tool used in work, data required for calculating the position of a tool described later, etc. are registered in the storage device 5.

[0049] When the processing device 2 recognizes the hand from the image obtained by the photographing device 1, the coordinates of the recognized hand are measured. The processing device 2 performs hand recognition and coordinate measurement every time an image is obtained. When tightening a screw, the screw is turned by a tool. The hand moves along a circle centered on a part of the tool. Therefore, the coordinates of the hand are located on the circle centered on a part of the tool. The processing device 2 calculates the center coordinates of the rotation of the tool based on the measured multiple coordinates.

[0050] The center coordinates correspond to the position of the screw tightened by the tool. Therefore, the position of the screw can be estimated using the calculated center coordinates. When the position of the screw is estimated, the position where the screw is tightened can be estimated. For example, by estimating the position where the screw is tightened, a work record indicating which position the screw is tightened can be automatically created. Alternatively, when the tightening order of multiple positions is determined, it can be automatically determined whether the position where the screw is tightened is appropriate.

[0051] Figure 2 is a schematic diagram of a mixed reality device of an exemplary embodiment.

[0052] Figure 1 The illustrated processing system 10 is implemented, for example, as a mixed reality (MR) device. Figure 2The MR device 100 shown includes a frame 101 , a lens 111 , a lens 112 , a projection device 121 , a projection device 122 , an image camera 131 , a depth camera 132 , a sensor 140 , a microphone 141 , a processing device 150 , a battery 160 , and a storage device 170 .

[0053] The image camera 131 and the depth camera 132 are examples of the imaging device 1. The processing device 150 is an example of the processing device 2. The microphone 141 is an example of the input device 3. The projection device 121 and the projection device 122 are examples of the display device 4. The storage device 170 is an example of the storage device 5.

[0054] In the example shown in the figure, the MR device 100 is a binocular head mounted display. Two lenses 111 and 112 are embedded in a frame 101. The projection device 121 and the projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0055] Projection devices 121 and 122 display the recognition result of the operator's body, virtual objects, etc. on lenses 111 and 112. Only one of projection devices 121 and 122 may be provided, and information may be displayed on only one of lenses 111 and 112.

[0056] The lenses 111 and 112 are light-transmissive. The operator visually confirms the reality through the lenses 111 and 112. In addition, the operator can also visually confirm the information projected onto the lenses 111 and 112 by the projection devices 121 and 122. The information is displayed in a superimposed manner in the real space through the projection of the projection devices 121 and 122.

[0057] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor that can detect three-axis angular velocity and three-axis acceleration. The microphone 141 receives sound input.

[0058] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display of the projection device 121 and the projection device 122. The processing device 150 detects the movement of the field of view based on the detection result of the sensor 140. The processing device 150 changes the display of the projection device 121 and the projection device 122 according to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, and the like.

[0059] The battery 160 supplies power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, and the like. The storage device 170 may be provided outside the MR device 100 and communicate with the processing device 150.

[0060] The MR device of the embodiment is not limited to the example shown in the figure, and may be a single-eye head mounted display. The MR device may be a glasses type as shown in the figure, or a helmet type.

[0061] Figure 3 It is a schematic diagram which illustrates the object of a work.

[0062] For example, Figure 3 The article 200 shown is being fastened. The article 200 is a hollow cylindrical member having fastening locations 201 to 208. The worker uses a wrench to fasten screws at the fastening locations 201 to 208, respectively.

[0063] A marker 210 is provided near the work object. In the illustrated example, the marker 210 is an AR marker. As described later, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), etc. may be used as the marker 210. Alternatively, instead of a marker, a base point may be indicated by a gesture. The processing device 150 sets a three-dimensional coordinate system based on a plurality of points indicated by gestures.

[0064] Figure 4 It is a schematic diagram showing an output example of the processing device according to the embodiment.

[0065] Here, use Figure 2 The example of tightening a screw with the MR device 100 shown will be described. When the tightening work starts, the image camera 131 and the depth camera 132 capture the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a three-dimensional coordinate system based on the position and orientation of the marker 210.

[0066] The object 200, the left hand 251 of the operator, and the right hand 252 of the operator are photographed by the image camera 131 and the depth camera 132. The processing device 150 recognizes the left hand 251 and the right hand 252 from the photographed images. The processing device 150 may also display the recognition result to the lens 111 and the lens 112 through the projection device 121 and the projection device 122. Hereinafter, the case where the processing device displays information to the lens using the projection device may also be referred to as "the processing device displays information".

[0067] For example, Figure 4As shown, the processing device 150 displays the recognition results of the left hand 251 and the recognition results of the right hand 252 superimposed on the hands in the real space. In the example shown in the figure, multiple virtual objects 261 and multiple virtual objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The multiple objects 261 respectively represent multiple joints of the left hand 251. The multiple objects 262 respectively represent multiple joints of the right hand 252. Instead of the joints, objects representing the surface shapes of the left hand 251 and the surface shapes of the right hand 252 may be displayed.

[0068] In addition, when the processing device 150 recognizes the left hand 251 and the right hand 252, it measures the coordinates of each hand. Specifically, the hand includes multiple joints such as the DIP joint, the PIP joint, the MP joint, and the CM joint. The coordinates of any of these joints are used as the coordinates of the hand. The center of gravity positions of multiple joints can also be used as the coordinates of the hand. Alternatively, the center coordinates of the entire hand can also be used as the coordinates of the hand.

[0069] Figures 5 to 8 This is a diagram showing what the work looks like.

[0070] For example, the operator tightens the screw to the tightening portion 203. In this case, Figure 5 As shown, the operator places the screw 215 in the threaded hole of the fastening part 203. The operator holds the handle of the wrench 280 with the right hand. The wrench 280 is an example of a first tool. The operator inserts the front end (head) of the wrench 280 with a socket into the screw 215.

[0071] Such as Figure 6 As shown, the operator presses the head of the wrench 280 with the left hand 251. Then, as shown in FIG. Figure 7 As shown, the wrench 280 is rotated with the right hand 252. As the wrench 280 is rotated, the screw is screwed. The screw 215 is tightened by repeatedly rotating the wrench 280.

[0072] While the operator is turning the wrench 280, the processing device 150 repeatedly measures the coordinates of the hand. Figure 8 As shown, the hand is located on a circle centered on a portion of the wrench 280. The hand moves in a circular arc. The processing device 150 uses this to calculate the center coordinates of the tool rotation. In calculating the center coordinates, any of the following first to third methods can be used.

[0073] Figures 9 to 11 This is a diagram for explaining the method of calculating the center coordinates.

[0074] In the first method, the processing device 150 extracts four different coordinates from the measured multiple coordinates. The processing device 150 calculates a sphere passing through the four coordinates. The sphere is represented by a spherical formula. Here, Fig. 9 As shown, the four coordinates are set to P1 (x1, y1, z1), P2 (x2, y2, z2), P3 (x3, y3, z3) and P4 (x4, y4, z4). The coordinates of the center of the sphere are set to P0 (x0, y0, z0). The radius of the sphere is set to r. r is equivalent to the distance from the head of the wrench 280 to the hand (handle), which is pre-registered. In this case, the spherical form is represented by the following mathematical formula (1). In mathematical formula (1), k, l, m and n are constants.

[0075] [Mathematical formula 1]

[0076] (x-x0) 2 +(y-y0) 2 +(z-z0) 2 =r 2

[0077] Substituting the coordinate P1 into the equation (1), the following equation (2) is obtained. The equation (2) is rewritten as the equation (3).

[0078] [Mathematical formula 2]

[0079] (x1-x0) 2 +(y1-y0) 2 +(z1-z0) 2 =r 2

[0080] [Mathematical formula 3]

[0081] x0 2 +y0 2 +z0 2 -2x1x0-2y1y0-2z1z0+x1 2 +y1 2 +z1 2 =0

[0082] Similarly, by substituting the coordinates P2 to P4 into the mathematical formula (1), the following mathematical formulas (4) to (6) are obtained.

[0083] [Formula 4]

[0084] x0 2 +y0 2 +z0 2 -2x2x0-2y2y0-2z2z0+x2 2 +y2 2 +z2 2 =0

[0085] [Formula 5]

[0086] x0 2+y0 2 +z0 2 -2x3x0-2y3y0-2z3z0+x3 2 +y3 2 +z3 2 =0

[0087] [Mathematical formula 6]

[0088] x0 2 +y0 2 +z0 2 -2x4x0-2y4y0-2z4z0+x4 2 +y4 2 +z4 2 =0

[0089] Solve the simultaneous equations of equations (3) to (6) to calculate the coordinates P0 (x0, y0, z0). Thus, the center of the ball passing through the four coordinates P1 (x1, y1, z1), P2 (x2, y2, z2), P3 (x3, y3, z3), and P4 (x4, y4, z4) is obtained. The processing device 150 uses the coordinates P0 (x0, y0, z0) of the center of the ball as the center coordinates of the rotation of the wrench 280.

[0090] The processing device 150 may also extract a combination of multiple four coordinates. The processing device 150 calculates the coordinates of the center of the ball for each combination. Thus, the coordinates of the centers of the multiple balls are calculated. There are errors in the measured coordinates of the hand. Therefore, the coordinates of the multiple centers calculated are also different from each other. The processing device 150 uses multiple coordinates to determine the coordinates used as the rotation center of the wrench 280. For example, the processing device 150 calculates the central value of the multiple coordinates as the center coordinate of the rotation of the wrench 280. The processing device 150 may also calculate the average value or the most frequent value of the multiple coordinates as the center coordinate of the rotation of the wrench 280. Thus, the accuracy of the calculated center coordinate can be improved.

[0091] In the second method, the processing device 150 extracts three coordinate pairs from the measured multiple coordinates. Each pair consists of two coordinates. Fig.10 In the example shown, a pair of coordinates P1 and P2, a pair of coordinates P3 and P4, and a pair of coordinates P5 and P6 are extracted. The six coordinates are respectively set to P1 (x1, y1, z1), P2 (x2, y2, z2), P3 (x3, y3, z3), P4 (x4, y4, z4), P5 (x5, y5, z5), and P6 (x6, y6, z6). The processing device 150 calculates the midpoint of the two coordinates for each pair. The coordinates of the midpoint of each pair are respectively set to P 12 , P 34 and P 56 .

[0092] As mentioned above, the hand moves in an arc shape. Therefore, the perpendicular bisector of each coordinate pair passing through each midpoint passes through the center of the circle. The center of the circle is set to coordinate P0 (x0, y0, z0). In this case, as shown in mathematical formula (7), the vector from coordinate P1 to coordinate P2 is the same as the vector from coordinate P 12 The inner product of the vector to the coordinate P0 is 0. The mathematical formula (7) can be rewritten as the mathematical formula (8).

[0093] [Formula 7]

[0094] (P2-P1)·(P0-P 12 )=0

[0095] [Mathematical formula 8]

[0096] (x2-x1)(x0-(x1+x2) / 2)+(y2-y1)(y0-(y1+y2) / 2)+(z2-z1)(z0-(z1+z2) / 2)=0

[0097] The same applies to other coordinate pairs, resulting in the following mathematical expressions (9) and (10).

[0098] [Mathematical formula 9]

[0099] (x4-x3)(x0-(x3+x4) / 2)+(y4-y3)(y0-(y3+y4) / 2)+(z4-z3)(z0-(z3+z4) / 2)=0

[0100] [Formula 10]

[0101] (x6-x5)(x0-(x5+x6) / 2)+(y6-y5)(y0-(y5+y6) / 2)+(z6-z5)(z0-(z5+z6) / 2)=0

[0102] These simultaneous equations are solved to calculate the coordinates P0 (x0, y0, z0). The processing device 150 calculates the coordinates P0 (x0, y0, z0) as the center coordinates of the rotation of the wrench 280.

[0103] The processing device 150 may also extract four or more coordinate pairs. The processing device 150 selects three coordinate pairs from the four or more coordinate pairs, and calculates the coordinate P0 for each combination of the selected coordinate pairs. The processing device 150 calculates the median value, average value, or mode value of the plurality of coordinates P0 as the center coordinate of the rotation of the wrench 280. Thus, the accuracy of the calculated center coordinate can be improved.

[0104] In the third method, the processing device 150 extracts three different coordinates from the measured multiple coordinates. The processing device 150 calculates the circumcenter O of the three coordinates. Fig.11 As shown, the three coordinates are set to P1 (x1, y1, z1), P2 (x2, y2, z2) and P3 (x3, y3, z3) respectively. The coordinates of the circumcenter O are set to P0 (x0, y0, z0). In addition, for the triangle obtained by connecting the coordinates P1 to P3, the length of the opposite side of the coordinate P1 is set to L1. The length of the opposite side of the coordinate P2 is set to L2. The length of the opposite side of the coordinate P3 is set to L3. The angle of the coordinate P1 is set to α. The angle of the coordinate P2 is set to β. The angle of the coordinate P3 is set to γ. In this case, the coordinates of the circumcenter O are expressed by the following mathematical formula (11). In addition, in mathematical formula (11), the symbol with an arrow attached represents the position vector. Mathematical formula (11) can be rewritten as mathematical formula (12). Mathematical formula (12) can be decomposed into mathematical formulas (13) to (15).

[0105]

Mathematical formula 11

[0106]

[0107]

Mathematical formula 12

[0108]

[0109]

Mathematical formula 13

[0110]

[0111]

Mathematical formula 14

[0112]

[0113]

Mathematical formula 15

[0114]

[0115] Based on equations (13) to (15), x0, y0, and z0 are calculated respectively. The processing device 150 calculates the coordinates P0 (x0, y0, z0) of the circumcenter O as the center coordinates of the rotation of the wrench 280.

[0116] The processing device 150 may also extract a plurality of combinations of three coordinates. The processing device 150 calculates the coordinates of the circumcenter for each combination. The processing device 150 may also calculate the median, average, or mode of the plurality of coordinates as the center coordinate of the rotation of the wrench 280. Thus, the accuracy of the calculated center coordinate can be improved.

[0117] Before executing any of the first to third methods, a portion of the coordinates of the hand may be selected from the plurality of coordinates. Based on the selected coordinates, any of the first to third methods is executed. For example, only the coordinates estimated to be when the hand is in an arc shape are selected, and the coordinates used in the first to third methods are extracted from these coordinates. By using only the coordinates when the hand is in an arc shape, the accuracy of the center coordinates can be further improved.

[0118] As an example, when using a digital tool, the processing device 150 can select the coordinates of the hand obtained at the timing (moment) when data is received from the digital tool. When the screw is tightened by the digital tool, the torque value, the screwing angle, etc. are detected by the digital tool. The processing device 150 receives these detection values ​​detected by the digital tool. The reception of the detection value indicates that the hand moves in an arc shape. Therefore, the coordinates of the hand obtained at the timing (moment) when the detection value is received are selected and used for the calculation of the center coordinates, thereby being able to calculate the center coordinates with higher accuracy.

[0119] In addition, even when the coordinates are not selected, the center coordinates can be calculated by calculating a plurality of center coordinate candidates in any of the first to third methods. For example, the processing device 150 removes the deviation value from the plurality of candidates and calculates the median value, average value, or mode value of the remaining candidates as the center coordinates.

[0120] Fig.12 as well as Fig.13 It is a schematic diagram showing an output example of the processing device according to the embodiment.

[0121] The center coordinate of the rotation of the wrench 280 can be regarded as the position of the screw tightened by the wrench 280. Since the screw is tightened at a predetermined position of the article, by estimating the position of the screw, the position where the screw is tightened can also be estimated. For example, when the position of the tightening part is pre-registered, it can be estimated that the screw is tightened at the tightening part closest to the center coordinate. The processing device 150 can also record that the screw is tightened at the position where the screw is estimated to be tightened. In this way, the operation record can be automatically created.

[0122] Alternatively, when the order of tightening with respect to a plurality of parts is determined, the processing device 150 can determine whether the part where the screw is tightened is appropriate. For example, the processing device 150 obtains the coordinates of the part where the screw is to be tightened. The processing device 150 calculates the center coordinates of the rotation of the wrench 280 based on the coordinates of the hand measured while the operator is tightening the screw. The processing device 150 calculates the distance between the coordinates of the part where the screw is to be tightened and the center coordinates.

[0123] When the calculated distance exceeds a preset threshold, the processing device 150 determines that the screw is tightened at an incorrect position. Fig.12As shown, the processing device 150 outputs an alarm 265. In the example shown in the figure, a message for the operator is displayed as the alarm. The processing device 150 may output sound, voice, vibration, light, etc. as the alarm to a predetermined output device instead of displaying.

[0124] When the calculated distance is less than the threshold value, the processing device 150 may output a message, sound, voice, vibration, light, etc. These outputs are different from an alarm. That is, the processing device 150 may output a notification to the operator indicating that the operation is appropriate.

[0125] When the operation is appropriate, the mode of outputting notification and the mode of not outputting notification can also be switched. The processing device 150 receives the mode selection. The operator can select the mode by voice command or gesture. The operator can adjust the amount of notification by selecting the mode, so the convenience of use of the MR device 100 is improved.

[0126] like Fig.13 As shown, the processing device 150 can also display the calculated center coordinates. Fig.13 In the example shown, the center coordinates are represented by a virtual object 270. By displaying the object 270, the operator can easily understand which position is calculated as the center coordinates. For example, the center coordinates may be calculated incorrectly. In the case where an alarm is output, the operator can easily understand the cause of the alarm by displaying the object 270. That is, the operator can easily understand whether the alarm is output due to an operation error or due to an error in the center coordinates. As a result, the convenience of the processing system 10 or the MR device 100 can be improved.

[0127] As described above, when the wrench 280 is a digital tool, the processing device 150 receives the detection value from the wrench 280. In this case, the processing device 150 may also associate the detection value such as the torque value with the data of the part where the screw is estimated to be tightened. In the case where the torque required for tightening the screw is pre-registered, the processing device 150 may also determine whether the required torque is detected. In addition to the torque value, the processing device 150 also registers the determination result in the operation record.

[0128] Fig.14 This is an example of a database related to fastening locations.

[0129] For example, Fig.14 As shown, data related to the fastening location is registered in advance. Fig.14The database 400 shown includes a job ID 401, a fastening part ID 402, a position 403, a torque value 404, a sequence 405, a record 406, and a detection value 407. The job ID 401 records the job ID of the job in which the screw is fastened. The fastening part ID 402 records the ID of the part where the screw is fastened in each job. The position 403 indicates the coordinates of the fastening part determined by the fastening part ID. In addition, the coordinates in the three-dimensional coordinate system during the fastening job are registered in the position 403. Figure 3 to Figure 7 In the example shown, the coordinates of the hand are measured in a three-dimensional coordinate system with the marker 210 as a base point. The position 403 of each part is represented by the coordinates in the same three-dimensional coordinate system.

[0130] Torque value 404 indicates the torque value required for tightening the screws to each part. Sequence 405 indicates the order of tightening for multiple parts tightened in one operation. If the order is not determined, the column of sequence 405 can be omitted. Record 406 indicates whether the tightening of the screws to each part is completed. When tightening the screws, the data indicating completion is registered in record 406. Detection value 407 indicates the value detected by the digital tool. For example, with respect to a part where the screw is presumed to be tightened, when the tightening order is appropriate and the required torque value is detected, the processing device 150 registers the data indicating that the tightening of the screw is completed in record 406. At the same time, the processing device 150 registers the torque value at this time in detection value 407. Thus, the record of the operation and the detection value are automatically associated with the data of each tightening part.

[0131] In order to estimate the position of the screw more accurately, the processing device 150 may use the length of the tool sandwiched between the wrench 280 and the screw to estimate the position where the screw exists. Figure 5 to Figure 7 In the example shown, a sleeve (not shown) is embedded in the wrench 280. That is, the center coordinates of the rotation of the wrench 280 are separated from the position of the screw by the length of the sleeve. When the length of the sleeve is pre-registered, the processing device 150 can use the center coordinates and the length of the sleeve to more accurately estimate the position of the screw.

[0132] Fig.15 (a) Fig.15 (b) and Fig.16 This is a diagram for explaining the method of calculating the center coordinates.

[0133] When estimating the position of the screw using the length of the sleeve, it is necessary to determine on which side the screw is located relative to the rotation plane of the wrench 280. Fig.15 In the example shown in (a), the wrench 280 is rotated in the rotation direction RD1. The screw 215 and the sleeve 285 are located at the bottom. Fig.15In the example shown in (b), the wrench 280 rotates in the rotation direction RD2. The rotation direction RD2 is opposite to the rotation direction RD1. The screw 215 and the sleeve 285 are located on the upper side with respect to the plane parallel to the rotation direction RD2.

[0134] In order to determine the side where the screw 215 is located, the processing device 150 uses the center coordinates, the two coordinates of the hand, the time series information of the two coordinates, and the tightness information of the screw. Fig.16 As shown, the two coordinates are set to P1 (x1, y1, z1) and P2 (x2, y2, z2) respectively. The center coordinate is set to P0 (x0, y0, z0). The time when the hand is at the coordinate P1 and the time when the hand is at the coordinate P2 are known. That is, the processing device 150 maintains the time series information of the coordinates P1 and P2. In this example, the time when the hand is at the coordinate P1 is before the time when the hand is at the coordinate P2.

[0135] The tightness information indicates whether the screw is tightened or loosened. In the case where the wrench 280 is a digital tool, the wrench 280 determines whether the screw is tightened or loosened according to the detected torque value and generates the tightness information. Alternatively, the processing device 150 determines whether the screw is tightened or loosened according to the time series data of the torque value received from the wrench 280 and generates the tightness information.

[0136] The plane passing through the coordinates P0 to P2 is represented by the following mathematical formula (16). In mathematical formula (16), k, l, m, and n are constants.

[0137] [Formula 16]

[0138] kx+ly+mz+n=0

[0139] By substituting the coordinates of P0 to P2 into equation (16), the following equations (17) to (19) are obtained. Based on equations (17) to (19), constants k, l, m, and n are calculated.

[0140] [Mathematical formula 17]

[0141] kx0+ly0+mz0+n=0

[0142] [Mathematical formula 18]

[0143] kx1+ly1+mz1+n=0

[0144] [Mathematical formula 19]

[0145] kx2+ly2+mz2+n=0

[0146] Here, the processing device 150 calculates the vector from the center coordinate P0 to the coordinate P1 at one moment forward. In addition, the processing device 150 calculates the vector from the center coordinate P0 to the coordinate P2 at one moment backward. The screw 215 is located at a position P separated from the center coordinate P0 by a length L0 of the sleeve on the normal vector P0P1×P0P2. Q When the screw is tightened and the time of the coordinate P1 is earlier than the time of the coordinate P2, the processing device 150 calculates the normal vector P0P1×P0P2 of the vector P0P1 and the vector P0P2.

[0147] From coordinate P0 to coordinate P where the wrench and the socket act on the screw Q The length of is represented by the following mathematical formula (20). In the following mathematical formula, an arrow with a symbol attached indicates that the value indicated by the symbol is a vector.

[0148]

Mathematical formula 20

[0149]

[0150] On the other hand, from coordinate P0 to coordinate P Q The vector can also be expressed by the following mathematical formula (21) using vector P0P1 and vector P0P2. In mathematical formula (21), t is a constant.

[0151]

Mathematical formula 21

[0152]

[0153] By substituting equation (21) into equation (20), the following equation (22) is obtained. The length L0 in equation (22) is pre-registered. By solving equation (22), t is calculated.

[0154]

Mathematical formula 22

[0155] √{({(y1-y0)(z2-z0)-(y2-y0)(z2-z0)}} 2 *t 2 +{(z1-z0)(x2-x0)-(z2-z0)(x1-x0)}*t 2 +{(x1-x0)(y2-y0)-(X2-x0)(y1-y0)}*t 2 =L0

[0156] When t is calculated, use the coordinate P0, constants k, l, m, n, and t to calculate the coordinate P Q . That is, the position of the screw is obtained.

[0157] Figure 17 to Figure 19 This is a diagram showing what the work looks like.

[0158] like Fig.17 As shown, the screw is sometimes tightened by means of an extension rod 290. The extension rod 290 is installed between the wrench 280 and the screw. In this case, the position of the screw can be estimated using the length of the extension rod 290, as in the above method.

[0159] That is, the screw is located at a position on the normal vector P0P1×P0P2 that is separated from the center coordinate P0 by the sum of the length of the sleeve 285 and the length of the extension rod 290. The position P of the screw is estimated using the center coordinate P0, the length of the sleeve 285, and the length of the extension rod 290. Q By taking into account the length of other tools interposed between the screw and the wrench 280, the position of the screw can be estimated with higher accuracy.

[0160] like Fig.18 As shown, sometimes the tool is held with both hands. Fig.18 In the example shown, the object 220 is located at a high position. The operator holds the wrench 280 with both hands to stabilize the wrench 280. Fig.19 As shown, the wrench 280 is turned with both hands to tighten the screws at the fastening locations 221 to 225 respectively.

[0161] In this case, the left hand 251 moves in an arc shape at a position separated from the head of the wrench 280 by a distance r1. The right hand 252 moves in an arc shape at a position separated from the head of the wrench 280 by a distance r2. The processing device 150 can calculate the center coordinates of the rotation of the tool based on both the multiple coordinates of the left hand 251 and the multiple coordinates of the right hand 252. In calculating the center coordinates, any of the first to third methods described above can be used.

[0162] For example, the processing device 150 calculates the first center coordinates based on the multiple coordinates of the left hand 251. The processing device 150 calculates the second center coordinates based on the multiple coordinates of the right hand 252. The processing device 150 uses the intermediate value between the first center coordinate and the second center coordinate as the final center coordinate.

[0163] The processing device 150 may also extract multiple combinations from the multiple coordinates of the left hand 251 and calculate multiple first center coordinates. Similarly, the processing device 150 may also extract multiple combinations from the multiple coordinates of the right hand 252 and calculate multiple second center coordinates. The processing device 150 uses the median, average, or mode of the multiple first center coordinates and the multiple second center coordinates as the final center coordinate.

[0164] The processing device 150 may also weight the first center coordinate and the second center coordinate. The farther the hand is from the rotation center of the tool, the more accurately the center coordinate can be calculated. Fig.19In the example shown, the distance r2 between the head of the wrench 280 and the right hand 252 is longer than the distance r1 between the head of the wrench 280 and the left hand 251. Therefore, it is preferable to weight the second center coordinate calculated based on the movement of the right hand 252 more than the first center coordinate calculated based on the movement of the left hand 251.

[0165] As an example, the processing device 150 uses the weighted average of the first center coordinate and the second center coordinate as the final center coordinate. The ratio of the weight for the first center coordinate to the weight for the second center coordinate can be the ratio of the distance r1 to the distance r2.

[0166] When calculating a plurality of first center coordinates and a plurality of second center coordinates, a part of the plurality of first center coordinates and a part of the plurality of second center coordinates may be extracted. In this case, by making the number of extracted second center coordinates greater than the number of extracted first center coordinates, the second center coordinates are weighted more heavily than the first center coordinates. For the ratio of the number of extracted second center coordinates to the number of extracted first center coordinates, the ratio of distance r1 to distance r2 may be used. The processing device 150 uses the median, average, or mode of the extracted first center coordinates and the extracted second center coordinates as the final center coordinates.

[0167] Fig. 20 It is a flowchart showing the processing method of the implementation mode.

[0168] In execution Fig. 20 Before the processing method M1 shown, the work master data 5 a , the base point master data 5 b , the tool master data 5 c , and the fastening location master data 5 d are prepared in advance.

[0169] First, select the job to be performed (step S0). The job is determined by the job ID, job name, ID of the object to be performed, or the name of the object. The processing device 2 accepts the selection of the job. For example, the job to be performed is selected by the operator. Alternatively, the job to be performed may be selected by the upper system, and the processing device 2 accepts the selection. The processing device 2 may also determine the next job to be performed based on the data obtained from the camera 1 or other sensors. The processing device 2 selects the job based on the determination result.

[0170] Next, the imaging device 1 images the mark 210. The processing device 2 sets the three-dimensional coordinate system based on the position and direction of the mark 210 (step S1). At this time, the processing device 2 refers to the base point master data 5b. The base point master data 5b stores the method for setting the base point for each operation. The processing device 2 obtains the method for setting the base point in the selected operation and sets the base point according to the method.

[0171] Then, the operation starts. The imaging device 1 captures the hand (step S2) and acquires an image. The processing device 2 recognizes the hand from the image and measures the coordinates of the hand (step S3). The acquisition of the image, recognition of the hand, and measurement of the coordinates are repeated continuously during the operation.

[0172] The processing device 2 uses the plurality of coordinates obtained by repeatedly performing step S3 to calculate the center coordinates of the rotation of the tool by any of the first to third methods (step S4). At this time, the processing device 2 refers to the tool master data 5c.

[0173] In the tool master data 5c, the ID of the tool used, the model of the tool, the length of the tool, the model of the socket, the length of the socket, etc. are stored for each operation. The model of the tool indicates the classification of the tool based on the structure, appearance or performance. The length of the tool is the length from the rotation center to the handle when the tool is used for screw tightening. The model of the socket indicates the classification of the socket based on the structure or appearance. The length of the socket indicates the length of the socket in the direction of connecting the tool to the screw when tightening the screw. The processing device 2 obtains the data of the tool used in the operation selected in step S0 from the tool master data 5c. Fig.17 As shown, when an extension rod is used, the model and length of the extension rod are also stored in the tool master data 5c. The processing device 2 also obtains data related to the extension rod from the tool master data 5c.

[0174] The processing device 2 estimates the position of the screw based on the center coordinates calculated in step S4 (step S5). The processing device 2 estimates the location where the screw is tightened based on the estimated position of the screw (step S6). At this time, the processing device 2 refers to the tightening location master data 5d.

[0175] In the fastening part master data 5d, the fastening part ID, the position of the fastening part, the required torque value, and the number of screw tightening times are stored for each fastening part. The fastening position indicates the position where the fastening part exists, and is determined by the coordinates of the three-dimensional coordinate system set in step S1. The number of screw tightening times indicates the number of screw tightening times required for each fastening part. When a mark is added to the screw after tightening, the color of the mark is also registered.

[0176] The processing device 2 determines whether the operation is appropriate based on the estimated result (step S7). For example, when the tightening order of the screws is predetermined, the processing device 2 determines whether the screws are tightened to the estimated parts in the determined order. When the torque value required for tightening is predetermined, the processing device 2 determines whether the torque value required for tightening the screws to the estimated parts is detected. When the operation is inappropriate, the processing device 2 outputs an alarm (step S8).

[0177] If the operation is appropriate, the processing device 2 determines whether the screw tightening to the estimated part is completed (step S9). For example, the processing device 2 refers to the torque value required for screw tightening to the estimated part from the tightening part master data 5d. If the torque value received from the digital tool exceeds the required torque value, the processing device 2 determines that the screw tightening to the estimated part is completed. If it is determined that the screw tightening is not completed, step S4 is executed again.

[0178] When it is determined that the screw tightening is completed, the processing device 2 records the completion of the screw tightening to the estimated position in the history data 5e (step S10). For example, the torque value detected by the tool is associated with the ID of the operation and the ID of the estimated position. As shown in the figure, the processing device 2 can also associate the model and ID of the tool used, the number of screw tightenings, and the recognition result of the mark with the ID of the tightening position. The mark is identified from the image captured by the camera 1 by the processing device 150. The processing device 150 extracts a block of pixels of the mark color from the image and counts the number of pixels of the block. When the number of pixels exceeds a preset threshold, it is determined that the mark is attached.

[0179] Advantages of the embodiment will be described.

[0180] When manufacturing an item, screws are sometimes tightened. In the past, in a manufacturing plant, in order to prevent forgetting to tighten the screws and to comply with the tightening order of the screws, a checklist was made and multiple operators checked with each other. However, in the checklist made by the operator, there may be omissions in the record, errors in the record, etc. The confirmation by multiple operators requires labor and time, and the cost increases. The time required for the manufacture of the item also becomes longer. In addition, if an error is found in the confirmation of the item after the operation, the operation will be reworked.

[0181] Regarding these issues, in an embodiment of the present invention, the camera 1 continuously and repeatedly photographs the operator's hand. The processing device 2 measures the coordinates of the hand based on each image. When tightening a screw, the operator's hand moves in an arc shape with a part of the tool as the center. The present invention focuses on this point and uses the measured multiple coordinates to calculate the center coordinates of the rotation of the tool. The position of the screw being turned corresponds to the center coordinate. Therefore, the center coordinate can be used to confirm the tightening operation of the screw.

[0182] For example, it is possible to estimate the location where the screw is tightened based on the center coordinates. Based on the estimated results, a record of the screw tightening can be automatically made. If the tightening order of the screws is determined, it is possible to automatically confirm whether the locations tightened by the screws are in the determined order. When using digital tools, it is possible to automatically confirm whether the screws are tightened with the required torque value.

[0183] Furthermore, according to an embodiment of the present invention, the center coordinates can be calculated during the execution of the fastening operation. Therefore, it is possible to detect errors in the operation in real time. In the case of an erroneous operation being performed, by detecting the situation as early as possible, it is possible to reduce rework of the operation. Thus, it is possible to improve the operation efficiency.

[0184] In the above, an example of implementing the embodiment using the MR device 100 is described. The embodiment can be implemented even if a device other than the MR device 100 is used. For example, the imaging device 1 can also capture the operator from above at a position away from the operator. The processing device 2 recognizes the operator's hand from the image and measures the coordinates of the hand. Hereinafter, as in the above example, the center coordinates of the rotation of the tool are calculated using the coordinates of the hand. In this case, a general-purpose personal computer can be used as the processing device 2. As the input device 3, a mouse or a keyboard can be used. As the display device 4, a monitor can be used.

[0185] When the MR device 100 is used, the image camera 131 and the depth camera 132 also move with the operator. Therefore, during the operation, the operator's hands are unlikely to be blocked by other objects. The operator's hands can be photographed more reliably. Therefore, it is difficult to produce a state where the hands cannot be photographed and the center coordinates cannot be calculated. In addition, since the hands are unlikely to be blocked, the accuracy of the calculated center coordinates can also be improved.

[0186] Furthermore, according to the MR device 100, data output from the processing device 150 is displayed on the lenses 111 and 112. Therefore, the operator can check the alarm etc. issued from the processing device 150 while performing the work. By using the MR device 100, the efficiency of the work can be improved.

[0187] (First Modification)

[0188] Fig.21 It is a schematic diagram showing an output example of the processing device according to the first modified example of the embodiment. Fig. 22 as well as Fig.23 This is a schematic diagram illustrating an example of a task.

[0189] like Fig.21 As shown, the processing device 150 may also display virtual objects 301 to 308 (first objects) during the tightening operation. The objects 301 to 308 are displayed corresponding to the tightening parts 201 to 208. The objects 301 to 308 respectively represent the positions of the tightening parts 201 to 208. For example, the distance between the object 301 and the corresponding tightening part 201 is shorter than the distance between the object 301 and other tightening parts.

[0190] The processing device 150 detects contact between the objects 301 to 308 and a predetermined object. For example, the processing device 150 detects contact between the hand and the objects 301 to 308. More specifically, the processing device 150 repeatedly calculates the distance between the hand and each of the objects 301 to 308. When the distance to any object is lower than a preset threshold, the processing device 150 determines that the hand is in contact with the object.

[0191] When the hand contacts any object, the processing device 150 infers that the screw is tightened at the position corresponding to the object. Fig. 22 As shown, in the case where the fastening part 203 is tightening a screw, the wrench 280 is rotated with one hand and the head of the wrench 280 is held with the other hand. At this time, the hand contacts the object 303. The processing device 150 detects the contact.

[0192] In addition to the contact between the object and the hand, the processing device 150 also estimates the part where the screw is tightened based on the movement of the hand that twists the tool. By using two methods to estimate the part that is tightened by the screw, even if the estimation fails in one method, the estimation may succeed in the other method. According to the first variant, the part that is tightened by the screw can be estimated more reliably. In addition, by displaying objects 301 to 308, the operator can easily understand where the part to be tightened is. As a result, the work efficiency can be improved.

[0193] The positions of the display objects 301 to 308 are shown by Figure 3 The mark 210 shown is a pre-registered coordinate of the three-dimensional coordinate system of the base point. Fig. 22 In the example shown, objects 301 to 308 are displayed close to fastening locations 201 to 208. That is, the positions of objects 301 to 308 are close to the center coordinates when the screws are fastened at the fastening locations 201 to 208. Therefore, the positions of objects 301 to 308 can also be used to calculate or correct the center coordinates of the rotation.

[0194] In addition, the detection of the contact between the object and the hand requires less calculation than the calculation of the center coordinates. Therefore, by detecting the contact between the object and the hand, the location where the screw is tightened can be estimated earlier. For example, in the case of a work error, by outputting an alarm earlier, the rework of the work can be reduced.

[0195] On the other hand, Fig.23As shown in the figure, the fastening part 231 is sometimes present in a narrow place. It is difficult for the operator to visually confirm the fastening part 231. Furthermore, in this example, the left hand cannot be inserted into the place of the fastening part, and the operator cannot hold the head of the wrench 280 with the left hand. That is, the hand cannot be in contact with the object. In addition, even if the hand is in contact with the object, the image camera 131 and the depth camera 132 cannot capture it. Therefore, the contact between the hand and the object cannot be detected. In such a case, by calculating the center coordinates of the rotation of the tool based on the movement of the hand, the part where the screw is fastened can be inferred.

[0196] Fig.24 It is a schematic diagram showing an output example of the processing device according to the first modified example of the embodiment.

[0197] When the fastening position estimated based on the contact between the object and the hand (first fastening position) is different from the fastening position estimated based on the center coordinates of the rotation of the tool (second fastening position), the processing device 150 may output an alarm. Fig.24 As shown, when the left hand 251 is in contact with the object 304, the processing device 150 estimates that the screw is tightened at the tightening portion 204 corresponding to the object 304. On the other hand, the right hand 252 is tightening the screw at the tightening portion 203 (not shown) using the wrench 280. The processing device 150 estimates that the screw is tightened at the tightening portion 203 based on the movement of the right hand 252. The tightening portion 203 estimated based on the movement of the hand is different from the tightening portion 204 estimated based on the contact between the hand and the object. Therefore, the processing device 150 outputs an alarm.

[0198] When tightening a screw, in addition to holding the handle of the tool with both hands, it is preferred to hold the handle and the head of the tool separately. By pressing the head of the tool and tightening the screw, the head of the tool can be prevented from moving in an undesirable direction. For example, the possibility of the tool moving in an undesirable direction and damaging an object can be reduced.

[0199] According to the first variant of the implementation method, Fig.24 As shown in FIG. 1 , even though the head of the tool can be gripped, an alarm is output when the work is performed without gripping the head. Thus, the operator can be urged to perform appropriate work.

[0200] In the above example, an example of detecting the contact between a hand and an object is described. Instead of the hand, the contact between a tool and an object may be detected. Alternatively, both the contact between the hand and the object and the contact between the tool and the object may be detected. Even in the case where the tool and the object are in contact, it can be inferred that the screw is tightened at the tightening position corresponding to the object, as in the above example.

[0201] Various methods can be used to estimate the position of the tool. For example, a sensor is provided on the wrench 280, and the position of the wrench 280 is estimated using the detection value of the sensor. The sensor is a tilt sensor, an acceleration sensor, a gyro sensor, etc. The position of the wrench 280 can also be estimated by combining the detection value of the sensor with the detection result of the hand.

[0202] Alternatively, the processing device 150 may estimate the position of the wrench 280 by image processing. For example, an image of the tool to be used (template image) is prepared in advance. The processing device 150 performs template matching to determine whether the tool of the template image is captured in the image obtained during the operation. The processing device 150 uses the position of the tool determined to be captured by the template image as the position of the tool.

[0203] Alternatively, a mark for estimating the position of the tool may be installed on the tool. Fig. 22 In the example shown, a plurality of markers 281 are attached to the wrench 280. The processing device 150 recognizes one or more markers 281 in the captured image. The processing device 150 calculates the position of a specific part (head) of the tool based on the recognized markers 281 and their positional relationship.

[0204] Fig.25 as well as Fig.26 It is a schematic diagram showing an output example of the processing device according to the first modified example of the embodiment.

[0205] exist Fig. 22 In the example shown, objects 301 to 308 are displayed near the fastening parts 201 to 208. When using an extension rod, the hand is placed away from the fastening part during operation. In this case, the object can also be displayed at a position away from the fastening part. Fig.25 In the illustrated example, objects 303 to 306 corresponding to the fastening locations 203 to 206 , respectively, are displayed at positions away from the fastening locations 203 to 206 .

[0206] like Fig.25 As shown, processing device 150 may further display virtual objects 313 to 316 (second objects). Objects 313 to 316 are displayed between fastening portion 203 and object 303, between fastening portion 204 and object 304, between fastening portion 205 and object 305, and between fastening portion 206 and object 306, respectively.

[0207] Objects 303 to 306 respectively indicate where the hand should be placed when tightening the screws at the fastening locations 203 to 206. Objects 313 to 316 respectively indicate where the extension rod should be placed when tightening the screws at the fastening locations 203 to 206. For example, when tightening the screws at the fastening location 204, Fig.26As shown, the hand is in contact with the object 304. The extension rod 290 overlaps with the object 304. The processing device 150 detects the contact between the hand and the object 304. The processing device 150 infers that the screw is tightened at the tightening portion 204 corresponding to the object 304.

[0208] By displaying the objects 303 to 306 and the objects 313 to 316 , the operator can easily understand where the hand and the extension rod should be positioned when tightening the screw, thereby improving the work efficiency.

[0209] In the example shown in the figure, objects 303 to 306 are spherical, and objects 313 to 316 are rod-shaped. The shape of each object is not limited to this example as long as each object can be easily visually recognized.

[0210] Fig. 27 This is a flowchart showing a processing method according to a first variation of the embodiment.

[0211] Fig. 27 The processing method M2 shown is the same as Fig. 20 Compared with the processing method M1 shown in the figure, it also includes step S11. In addition, the processing method M2 includes step S12 instead of steps S4 to S6. The extension part model, angle, object shape and display method are also registered in the fastening part master data 5d. The angle indicates the angle of the tool or extension rod when tightening the screw at each fastening part. The extension part model indicates the classification of the extension rod based on structure, appearance or performance. The object shape indicates the shape of the object displayed corresponding to each fastening part. The display method indicates the way of displaying each object. The display method is set according to the number of screw tightening times.

[0212] The processing device 2 displays the first object and the second object based on the set base point and the reference data (step S11). The positions where the first object and the second object are displayed are calculated based on the base point, the fastening position, the angle, the extension part model, the tool model, etc. For example, the length of the extension rod is determined by the extension part model. The first object is displayed at a position separated by the length of the extension rod at a set angle relative to the fastening position. The second object is displayed in an area separated by the length of the extension rod at a set angle relative to the fastening position.

[0213] The processing device 2 estimates the fastening position (step S12). Specifically, when the contact between the first object and the hand is detected, the processing device 2 estimates that the screw is tightened at the fastening position corresponding to the first object. At the same time, the processing device 2 executes steps S4 to S6 of the processing method M1 and also estimates the fastening position based on the center coordinates.

[0214] When the tightening position of the screw is estimated, steps S7 to S10 are performed as in processing method M1. In step S7, as described above, if the tightening position estimated based on the contact between the object and the hand is different from the tightening position estimated based on the center coordinates of the rotation of the tool, it can be determined that the operation is inappropriate.

[0215] (Second Modification)

[0216] In the example described above, the center coordinates of the rotation of the tool are calculated based on the coordinates of multiple hands. Instead of this method, the center coordinates of the rotation of the tool can be pre-registered. For example, when tightening a screw at a fastening position, the position where the front end of the tool is configured is registered as the center coordinate. Alternatively, when tightening a screw at a fastening position, the position of the screw can be registered as the center coordinate. The position of the threaded hole (fastening position) can also be registered as the center coordinate. The corresponding fastening position is associated with the center coordinate. The processing device 2 uses the center coordinate to infer the position where the screw is tightened.

[0217] Fig.28 It is a flowchart showing the processing method of the second modified example of the embodiment.

[0218] In the processing method M3 of the second variant, the processing device 2 and Fig. 20 The processing method M1 shown executes steps S0 to S3 in the same manner. The processing device 2 refers to the pre-registered center coordinates (step S21). For example, the center coordinates are stored in the fastening part master data 5d in association with the fastening part ID. The center coordinates are registered for each fastening part. The processing device 2 calculates the distance between the center coordinate and multiple coordinates of the hand for each center coordinate (step S22). The processing device 2 calculates the deviation of each distance between the center coordinate and multiple coordinates for each center coordinate (step S23). For any center coordinate, when the deviation is less than a preset threshold, the processing device 2 infers that the screw is tightened at the fastening part associated with the center coordinate (step S24). When the fastening part where the screw is tightened is inferred, steps S7 to S10 are executed in the same manner as the processing method M1.

[0219] Fig.29 It is a schematic diagram for explaining the processing method of the second modified example of the embodiment.

[0220] For example, Fig.29As shown, three center coordinates C1 to C3 are pre-registered corresponding to three fastening locations. Based on the image of the hand tightening the screw, three coordinates P1 to P3 are calculated. The processing device 2 calculates the distances d1 to d3 between the center coordinate C1 and the coordinates P1 to P3. Similarly, the processing device 2 calculates the distances between the center coordinate C2 and the coordinates P1 to P3, and the distances between the center coordinate C3 and the coordinates P1 to P3. The processing device 2 calculates the deviation of the distances between the center coordinate C1 and the coordinates P1 to P3, the deviation of the distances between the center coordinate C2 and the coordinates P1 to P3, and the deviation of the distances between the center coordinate C3 and the coordinates P1 to P3. When any one of the deviations is less than the threshold value, the processing device 2 infers that the screw is tightened at the fastening location associated with the center coordinate.

[0221] As the deviation, the sum of the average value of a plurality of distances and the difference between each distance, the dispersion of a plurality of distances, or the standard deviation of a plurality of distances is used. Fig.29 In the example shown, the coordinates of the hand are substantially equidistant from the center coordinate C1. Therefore, it is estimated that the screw is tightened at the tightening position associated with the center coordinate C1.

[0222] According to the processing method of the second modification, the fastening location where the screw is tightened can be estimated with less calculation amount than when the center coordinates are calculated.

[0223] It is also possible to extract a portion of the coordinates of the hand from the multiple coordinates repeatedly measured by hand tracking. For example, when a digital tool is used, data is sent from the digital tool to the processing device 2. The processing device 2 extracts the coordinates of the hand measured at the timing when the data is received. At the timing when the data is received from the digital tool, it is highly likely that the screw is actually being tightened. By using the coordinates of the hand measured at this timing for estimation, the accuracy of the estimation can be further improved.

[0224] The second variant of the embodiment can also be combined with the first variant of the embodiment. For example, the processing device 150 can estimate the location where the screw is tightened based on the distance between the center coordinates and the coordinates of the hand in addition to the contact between the object and the hand. The location tightened by the screw is estimated by two methods. In addition, in this case, the position of the displayed virtual object can also be used as the center coordinate.

[0225] In the example described above, the case where the present invention is applied to the operation of tightening a screw is described. The embodiment of the present invention is not limited to this example. For example, the present invention can also be applied to the operation of loosening a screw. When loosening a screw, the tool is also rotated by hand with a part of the tool as the center. Therefore, the center coordinates of the rotation of the tool can be calculated based on the movement of the hand. For example, the fastening part where the screw is loosened can be estimated using the center coordinates. When the order of the operation is determined for multiple parts, it can be automatically determined whether the part where the screw is loosened is appropriate.

[0226] Fig.30 This is a schematic diagram showing the hardware configuration.

[0227] As the processing system 10, for example, Fig.30 The computer 90 shown in FIG. 9 includes a CPU 91 , a ROM 92 , a RAM 93 , a storage device 94 , an input interface 95 , an output interface 96 , and a communication interface 97 .

[0228] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for the computer 90 to realize the above-mentioned processing. The RAM 93 functions as a storage area for developing the program stored in the ROM 92.

[0229] The CPU 91 includes a processing circuit and uses the RAM 93 as a working memory to execute a program stored in at least one of the ROM 92 and the storage device 94. During the execution of the program, the CPU 91 controls each component via the system bus 98 and executes various processes.

[0230] The storage device 94 stores data required for executing the program and data obtained by executing the program. The storage device 94 includes a solid state drive (SSD) etc. The storage device 94 can also be used as the storage device 5 or the storage device 170.

[0231] An input interface (I / F) 95 can connect the computer 90 to the input device 3 . The CPU 91 can read various data from the input device 3 via the input I / F 95 .

[0232] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can send data to the display device 4 via the output I / F 96 and display information on the display device 4.

[0233] The communication interface (I / F) 97 can connect a device outside the computer 90 to the computer 90. The communication I / F 97 connects the digital tool to the computer 90 through, for example, Bluetooth (registered trademark) communication.

[0234] The data processing performed by the processing device 2 or the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0235] The processing of the above-mentioned various data can also be recorded as a program that can be executed by a computer, on a disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory or other non-transitory computer-readable storage medium.

[0236] For example, the information recorded in the recording medium can be read out by a computer (or embedded system). In the recording medium, the recording format (storage format) is arbitrary. For example, the computer reads out the program from the recording medium and causes the CPU to execute the instructions recorded in the program based on the program. In the computer, the acquisition (or reading) of the program can also be performed through a network.

[0237] The embodiments of the present invention include the following features.

[0238] (Feature 1)

[0239] A processing device, wherein

[0240] The coordinates of the hand are measured based on a plurality of images obtained by continuously photographing the hand that is tightening the screw using the first tool.

[0241] The center coordinates of the rotation of the first tool are calculated using the plurality of coordinates.

[0242] (Feature 2)

[0243] In the processing device described in feature 1,

[0244] The position of the screw is estimated based on the center coordinates.

[0245] Referring to the pre-registered position of the fastening part where the above screw is tightened,

[0246] When the distance between the position of the fastened portion and the estimated position of the screw is equal to or smaller than a threshold value, it is estimated that the screw is tightened relative to the fastened portion.

[0247] (Feature 3)

[0248] In the processing device described in feature 2,

[0249] When the distance is greater than the threshold, an alarm is output.

[0250] (Feature 4)

[0251] In the processing device described in feature 2 or 3,

[0252] When the distance is equal to or smaller than the threshold value, a notification indicating that the work on the fastening location is appropriate is output.

[0253] (Feature 5)

[0254] In the processing device described in feature 4,

[0255] It is possible to accept selection of a mode for outputting the above notification and a mode for not outputting the above notification.

[0256] (Feature 6)

[0257] In the processing device according to any one of features 2 to 5,

[0258] receiving the detection value detected by the first tool,

[0259] The detection value is associated with the data related to the fastened portion.

[0260] (Feature 7)

[0261] In the processing device according to any one of features 2 to 5,

[0262] receiving the detection value detected by the first tool,

[0263] selecting one or more of the above-mentioned coordinates obtained at the timing of receiving the above-mentioned detection value from the above-mentioned multiple coordinates,

[0264] Using the one or more coordinates, the center coordinates are calculated.

[0265] (Feature 8)

[0266] In the processing device according to any one of features 1 to 7,

[0267] Extract 4 coordinates from the above multiple coordinates,

[0268] The coordinates of the center of the sphere passing through the above four coordinates are calculated as the above center coordinates.

[0269] (Feature 9)

[0270] In the processing device according to any one of features 1 to 7,

[0271] Extract multiple combinations from the multiple coordinates, where the combination is a combination of two coordinates;

[0272] For each of the above combinations, generate the perpendicular bisector between the above two coordinates,

[0273] The coordinates of the intersection points of the plurality of perpendicular bisectors are calculated as the center coordinates.

[0274] (Feature 10)

[0275] In the processing device according to any one of features 1 to 7,

[0276] Extract 3 coordinates from the above multiple coordinates,

[0277] The coordinates of the circumcenter of the above three coordinates are calculated as the above center coordinates.

[0278] (Feature 11)

[0279] In the processing device according to any one of features 1 to 9,

[0280] obtaining the length of the tool sandwiched between the first tool and the screw,

[0281] The normal vector is calculated using the center coordinate, two coordinates extracted from the plurality of coordinates, time series information of the two coordinates, and tightness information indicating whether the screw is tightened or loosened by the first tool.

[0282] It is estimated that the screw exists at a position separated from the center coordinate by the tool length on the normal vector.

[0283] (Feature 12)

[0284] In the processing device according to any one of features 1 to 11,

[0285] When the first tool is rotated by the two hands, the coordinates of each of the two hands are measured based on each of the plurality of images.

[0286] Using the plurality of coordinates of one hand, calculating the first center coordinates of the rotation of the first tool,

[0287] Using the plurality of coordinates of the other hand, the second center coordinates of the rotation of the first tool are calculated,

[0288] The center coordinates are calculated using the first center coordinates and the second center coordinates.

[0289] (Feature 13)

[0290] In the processing device described in feature 12,

[0291] When the distance between the rotation center of the first tool and the other hand is longer than the distance between the center of the first tool and the one hand, the second center coordinate is weighted larger than the first center coordinate to calculate the center coordinate.

[0292] (Feature 14)

[0293] A processing device, wherein

[0294] A virtual first object corresponding to the fastening portion where the screw is tightened is displayed,

[0295] The center coordinates of the rotation of the first tool are calculated based on a plurality of images obtained by continuously photographing the hand holding the first tool.

[0296] Calculate the distance between the coordinates of the fastening part where the screw is turned and the center coordinates,

[0297] When a predetermined object is in contact with the first target or the distance is smaller than a preset threshold, it is estimated that the screw is tightened with respect to the fastening portion.

[0298] (Feature 15)

[0299] In the processing device described in feature 14,

[0300] When a first fastening location where the screw is estimated to be tightened based on the contact between the first object and the predetermined object is different from a second fastening location where the screw is estimated to be tightened based on the distance, an alarm is output.

[0301] (Feature 16)

[0302] A processing device, wherein

[0303] The coordinates of the hand are measured based on a plurality of images obtained by continuously photographing the hand that is tightening the screw using the first tool.

[0304] calculating the distances between the plurality of coordinates and the pre-registered center coordinates of the rotation of the first tool, respectively;

[0305] When the deviation of the plurality of distances is smaller than a preset threshold value, it is estimated that the screw is tightened relative to the fastening location associated with the center coordinates.

[0306] (Feature 17)

[0307] A mixed reality device having:

[0308] The processing device according to any one of features 1 to 16;

[0309] A photographing device for acquiring the plurality of images; and

[0310] The display device displays the information output from the processing device.

[0311] (Feature 18)

[0312] A processing method, wherein

[0313] The processing device measures the coordinates of the hand based on a plurality of images obtained by continuously photographing the hand that is tightening the screw using the first tool,

[0314] The processing device is caused to calculate the center coordinates of the rotation of the first tool using the plurality of coordinates.

[0315] (Feature 19)

[0316] A procedure in which

[0317] The processing device is caused to execute the processing method described in feature 18.

[0318] (Feature 20)

[0319] A storage medium, wherein

[0320] The program described in feature 19 is stored.

[0321] According to the above-described embodiments, a processing device, a mixed reality device, a processing method, a program, and a storage medium are provided that can assist in the operation of tightening a screw.

[0322] Several embodiments of the present invention are illustrated above, and these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments and their variations are included in the scope and main purpose of the invention, and are included in the invention described in the scope of the patent claim and the scope equivalent thereto. In addition, the above-mentioned embodiments can be implemented in combination with each other.

[0323] Explanation of symbols:

[0324] 1: Photographic device; 2: Processing device; 3: Input device; 4: Display device; 5: Storage device; 5a: Operation master data; 5b: Base point master data; 5c: Tool master data; 5d: Fastening part master data; 5e: History data; 10: Processing system; 10: Processing system; 90: Computer; 91: CPU; 92: ROM; 93: RAM; 94: Storage device; 95: Input interface; 96: Output interface; 97: Communication interface; 98: System bus; 100: MR device; 101: Frame; 111, 112: Lens; 121, 122: Projection device; 131: Image camera; 132: Depth camera; 140: Sensor; 141: Microphone ; 150: processing device; 160: battery; 170: storage device; 200: item; 201-208: fastening part; 210: mark; 215: screw; 220: item; 221-225: fastening part; 230: item; 231: fastening part; 251: left hand; 252: right hand; 261, 262: object; 265: alarm; 270: object; 280: wrench; 281: mark; 285: socket; 290: extension rod; 301-308, 313-316: object; 400: database; 404: torque value; 405: sequence; 406: record; 407: detection value; M1, M2: processing method; O: circumcenter.

Claims

1. A processing device, wherein: The coordinates of the hand are measured based on a plurality of images obtained by continuously photographing the hand that is tightening the screw using the first tool. The center coordinates of the rotation of the first tool are calculated using the plurality of coordinates.

2. The processing device according to claim 1, wherein: The position of the screw is estimated based on the center coordinates. Referring to the pre-registered position of the fastening part where the above screw is tightened, When the distance between the position of the fastened portion and the estimated position of the screw is equal to or smaller than a threshold value, it is estimated that the screw is tightened relative to the fastened portion.

3. The processing device according to claim 2, wherein: When the distance is greater than the threshold, an alarm is output.

4. The processing device according to claim 2, wherein: When the distance is equal to or smaller than the threshold value, a notification indicating that the work on the fastening location is appropriate is output.

5. The processing device according to claim 4, wherein: It is possible to accept selection of a mode for outputting the above notification and a mode for not outputting the above notification.

6. The processing device according to claim 2, wherein: receiving the detection value detected by the first tool, The detection value is associated with the data related to the fastened portion.

7. The processing device according to claim 2, wherein: receiving the detection value detected by the first tool, selecting one or more of the above-mentioned coordinates obtained at the timing of receiving the above-mentioned detection value from the above-mentioned multiple coordinates, Using the one or more coordinates, the center coordinates are calculated.

8. The processing device according to claim 1, wherein: Extract 4 coordinates from the above multiple coordinates, The coordinates of the center of the sphere passing through the above four coordinates are calculated as the above center coordinates.

9. The processing device according to claim 1, wherein: Extract multiple combinations from the multiple coordinates, where the combination is a combination of two coordinates; For each of the above combinations, generate the perpendicular bisector between the above two coordinates, The coordinates of the intersection points of the plurality of perpendicular bisectors are calculated as the center coordinates.

10. The processing device according to claim 1, wherein: Extract 3 coordinates from the above multiple coordinates, The coordinates of the circumcenter of the above three coordinates are calculated as the above center coordinates.

11. The processing device according to claim 1, wherein: obtaining the length of the tool sandwiched between the first tool and the screw, The normal vector is calculated using the center coordinate, two coordinates extracted from the plurality of coordinates, time series information of the two coordinates, and tightness information indicating whether the screw is tightened or loosened by the first tool. It is estimated that the screw exists at a position separated from the center coordinate by the tool length on the normal vector.

12. The processing device according to claim 1, wherein: When the first tool is rotated by the two hands, the coordinates of each of the two hands are measured based on each of the plurality of images. Using the plurality of coordinates of one hand, calculating the first center coordinates of the rotation of the first tool, Using the plurality of coordinates of the other hand, the second center coordinates of the rotation of the first tool are calculated, The center coordinates are calculated using the first center coordinates and the second center coordinates.

13. The processing device according to claim 12, wherein: When the distance between the rotation center of the first tool and the other hand is longer than the distance between the center of the first tool and the one hand, the second center coordinate is weighted larger than the first center coordinate to calculate the center coordinate.

14. A processing device, wherein: A virtual first object corresponding to the fastening portion where the screw is tightened is displayed, The center coordinates of the rotation of the first tool are calculated based on a plurality of images obtained by continuously photographing the hand holding the first tool. Calculate the distance between the coordinates of the fastening part where the screw is turned and the center coordinates, When a predetermined object is in contact with the first target or the distance is smaller than a preset threshold, it is estimated that the screw is tightened with respect to the fastening portion.

15. The processing device according to claim 14, wherein: When a first fastening location where the screw is estimated to be tightened based on the contact between the first object and the predetermined object is different from a second fastening location where the screw is estimated to be tightened based on the distance, an alarm is output.

16. A processing device, wherein: The coordinates of the hand are measured based on a plurality of images obtained by continuously photographing the hand that is tightening the screw using the first tool. calculating the distances between the plurality of coordinates and the pre-registered center coordinates of the rotation of the first tool, respectively; When the deviation of the plurality of distances is smaller than a preset threshold value, it is estimated that the screw is tightened relative to the fastening location associated with the center coordinates.

17. A mixed reality device comprising: The processing device according to any one of claims 1 to 16; A photographing device for acquiring the plurality of images; and The display device displays the information output from the processing device.

18. A method of treatment, wherein: The processing device measures the coordinates of the hand based on a plurality of images obtained by continuously photographing the hand that is tightening the screw using the first tool, The processing device is caused to calculate the center coordinates of the rotation of the first tool using the plurality of coordinates.

19. A program, wherein: The processing device is caused to execute the processing method according to claim 18.

20. A storage medium, wherein: The program according to claim 19 is stored.

Images