Processing device, processing system, composite reality device, processing method, program, and storage medium

Abstract

To provide a processing device, a processing system, a composite reality device, a processing method, a program, and a storage medium capable of supporting work using a screw more appropriately.SOLUTION: A processing device acquires an imaging showing a hand of an operator and a plurality of markers attached to a first tool for turning a screw. When the hand comes into contact with a virtual first object, the processing device calculates a first position of the first tool from the positions of the plurality of markers. On the basis of a first distance between the first object and the first position, the processing device determines whether or not the screw is turned to a fastening position corresponding to the first object.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] An embodiment of the present invention relates to a processing device, a processing system, a mixed reality device, a processing method, a program, and a storage medium. [Background technology]

[0002] When manufacturing an article, a screw may be tightened. Or, when maintaining, inspecting, or repairing an article, a screw may be loosened. There is a demand for a technology that can more appropriately support such work using screws. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a processing device, a processing system, a mixed reality device, a processing method, a program, and a storage medium that can more appropriately assist work using screws. [Means for solving the problem]

[0005] A processing device according to an embodiment acquires an image of a worker's hand and a plurality of markers attached to a first tool for turning a screw. When the hand comes into contact with a virtual first object, the processing device calculates a first position of the first tool from positions of the plurality of markers. The processing device determines whether the screw is turned into a fastening point corresponding to the first object based on a first distance between the first object and the first position. [Brief description of the drawings]

[0006] [Figure 1]FIG. 1 is a schematic diagram showing the configuration of a processing system according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 3] FIG. 3 is a schematic diagram illustrating an example of an object to be worked on. [Figure 4] FIG. 4 is a schematic diagram for explaining the first embodiment of the present invention. [Diagram 5] FIG. 5 is a schematic diagram for explaining the first embodiment of the present invention. [Figure 6] FIG. 6 is a schematic diagram for explaining the first embodiment of the present invention. [Figure 7] FIG. 7 is a schematic diagram for explaining the first embodiment of the present invention. [Figure 8] FIG. 8 is a schematic diagram for explaining the first embodiment of the present invention. [Figure 9] FIG. 9 is a schematic diagram illustrating the state of work. [Figure 10] 10(a) and 10(b) are schematic diagrams illustrating the state of work. [Figure 11] 11(a) and 11(b) are schematic diagrams showing an example of output in the embodiment of the present invention. [Figure 12] FIG. 12 is a flowchart illustrating an example of a processing method according to the embodiment. [Figure 13] Fig. 13(a) is a schematic diagram showing an example of a tool, and Figs. 13(b) to 13(d) are schematic diagrams showing an example of a marker. [Figure 14] 14(a) to 14(h) are schematic views for explaining the processing in the processing system according to the embodiment. [Figure 15] FIG. 15 is a schematic diagram showing the movement of the tool. [Figure 16] FIG. 16 shows simultaneous equations that represent the relationship between the position of each marker before movement and the position of each marker after movement. [Figure 17] FIG. 17 is a determinant that represents the relationship between the position of the tool before the movement and the position of the tool after the movement. [Figure 18] FIG. 18 is a flowchart showing a process for calculating the tool position. [Figure 19] FIG. 19 is a flowchart showing a process for calculating the tool position. [Figure 20] 20(a) and 20(b) are schematic diagrams for explaining the filter. [Figure 21] 21(a) and 21(b) are schematic diagrams for explaining the filter. [Figure 22] FIG. 22 is a schematic diagram for explaining a first modified example of the embodiment. [Figure 23] FIG. 23 is a schematic diagram for explaining a first modified example of the embodiment. [Figure 24] FIG. 24 is a schematic diagram for explaining a first modified example of the embodiment. [Diagram 25] Figure 25(a) is a schematic diagram showing an example of a first tool, and Figure 25(b) is a schematic diagram showing an example of a second tool. [Figure 26] 26(a) and 26(b) are schematic diagrams for explaining a second modified example of the embodiment. [Figure 27] 27(a) and 27(b) are schematic diagrams for explaining a second modified example of the embodiment. [Figure 28] FIG. 28 is a flowchart showing a processing method according to the second modified example of the embodiment. [Figure 29] FIG. 29 is a flowchart showing the flow of data in the processing method according to the embodiment. [Diagram 30] FIG. 30 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a processing system according to the first embodiment. The first embodiment of the present invention is applicable to a task of turning a screw using a tool. A processing system 10 includes an imaging device 1, a processing device 2, an input device 3, a display device 4, and a storage device 5, as shown in FIG.

[0009] The imaging device 1 captures images of the state of work. For example, in the work, a fastener such as a screw is fastened to an article using a tool. Or, a screw fastened to the article is loosened using a tool. The article is a part, a unit, or a semi-finished product for producing a product. The tool is a wrench, a screwdriver, or the like. Here, an example in which the embodiment of the present invention is applied to a fastening work of tightening a screw will be mainly described.

[0010] When assembling an article, a worker holds a tool in his / her hand and tightens a screw. The image capturing device 1 captures an image of the worker's left hand or right hand holding the tool. For example, the image capturing device 1 includes a camera that captures an RGB image and a depth image.

[0011] The processing device 2 receives successive images (video) captured by the photographing device 1. The processing device 2 recognizes the left or right hand in the image. Hand tracking is used to recognize the left or right hand. Hereinafter, when there is no particular distinction between the left hand and the right hand, the term "hand" will simply refer to at least either the left hand or the right hand.

[0012] The input device 3 is used by the worker to input information to the processing device 2. The input device 3 includes a microphone. The worker can input information to the processing device 2 by speaking into the input device 3. For example, a voice corresponding to a voice command is input to the input device 3. In addition to the input device 3, the worker can input information to the processing device 2 by hand gestures or the like.

[0013] The display device 4 displays information for the worker. The processing device 2 causes the display device 4 to display information for supporting the fastening work. Hereinafter, the processing device causing the display device to display information is also simply referred to as "the processing device displays information." For example, the processing device 2 displays a virtual object indicating the position where the worker's hand should be placed, a hand recognition result, etc. during the fastening work.

[0014] The storage device 5 stores data necessary for processing by the processing device 2, data obtained by the processing by the processing device 2, etc. For example, data on tools used in work, data necessary for calculating the position of the tools described later, etc. are registered in the storage device 5.

[0015] The processing device 2 judges whether inappropriate work is being performed from the image captured by the photographing device 1. Specifically, the processing device 2 judges whether a hand has touched an object from the image. The processing device 2 also calculates the position of a tool from the image. A number of markers are attached to the tool, and the processing device 2 calculates the position of the tool from the positions of the number of markers. The processing device 2 calculates the distance between the position of the tool and the position of the object, and judges whether the work is appropriate based on the distance.

[0016] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. The processing system 10 shown in Fig. 1 is realized as, for example, a mixed reality (MR) device. The MR device 100 shown in Fig. 2 includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0017] The image camera 131 and the depth camera 132 are an example of an imaging device 1. The processing device 150 is an example of a processing device 2. The microphone 141 is an example of an input device 3. The projection device 121 and the projection device 122 are an example of a display device 4. The storage device 170 is an example of a storage device 5.

[0018] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, a lens 111 and a lens 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0019] The projection device 121 and the projection device 122 display the recognition result of the body of the worker, a virtual object, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0020] The lenses 111 and 112 are optically transparent. The worker can visually recognize the real state through the lenses 111 and 112. The worker can also visually recognize information projected onto the lenses 111 and 112 by the projection devices 121 and 122. The information is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0021] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0022] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0023] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0024] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0025] FIG. 3 is a schematic diagram illustrating an example of an object to be worked on. For example, a fastening operation is performed on an article 200 shown in Fig. 3. The article 200 is a cylindrical hollow member, and has fastening points 201 to 208. A worker uses a wrench to fasten screws into each of the fastening points 201 to 208.

[0026] A marker 210 is provided near the work target. In the illustrated example, the marker 210 is an AR marker. As described below, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 150 sets a three-dimensional coordinate system based on a plurality of points indicated by a hand gesture.

[0027] 4 to 8 are schematic diagrams illustrating the first embodiment of the present invention. At the start of the fastening operation, the image camera 131 and the depth camera 132 capture an image of the marker 210. The processing device 150 recognizes the marker 210 from the captured image. The processing device 150 sets a three-dimensional coordinate system based on the position and orientation of the marker 210.

[0028] The image camera 131 and the depth camera 132 capture images of the article 200, the worker's left hand, and the worker's right hand. The processing device 150 recognizes the left hand and the right hand from the captured images. The processing device 150 measures the position of the recognized hand. The hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The center of gravity of multiple joints may be used as the position of the hand. Alternatively, the overall center position of the hand may be used as the position of the hand. The processing device 150 causes the projection device 121 and the projection device 122 to display the recognition result on the lens 111 and the lens 112. Hereinafter, the processing device causing the lens to display information using the projection device is also simply referred to as "the processing device displays information."

[0029] 4, for example, the processing device 150 displays the recognition results of the left hand 251 and the right hand 252 superimposed on the hands in real space. In the illustrated example, a plurality of objects 261 and a plurality of objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The plurality of objects 261 respectively indicate a plurality of joints of the left hand 251. The plurality of objects 262 respectively indicate a plurality of joints of the right hand 252. Instead of the joints, objects respectively indicating the surface shapes of the left hand 251 and the right hand 252 may be displayed.

[0030] Furthermore, the processing device 150 displays virtual objects 301-308 (examples of first objects) corresponding to the fastening points 201-208, as shown in Fig. 5. The objects 301-308 indicate positions where the worker's hands should be placed when fastening the fastening points 201-208, respectively. In the illustrated example, the objects 301-308 are spheres. The shape, color, etc. of each object may be arbitrary as long as the worker can visually distinguish it from the real space.

[0031] The positions at which the objects 301-308 are displayed are registered in advance using a coordinate system based on the base point of the marker 210. The three-dimensional coordinate system used in registering the display positions of the objects 301-308 is the same as the three-dimensional coordinate system set during work. The positional relationship between the article 200 and the marker 210 when the object is prepared is adjusted to be the same as the positional relationship between the article 200 and the marker 210 when the work is performed. This allows the prepared object to be displayed overlapping the article 200 at an appropriate position during work.

[0032] As an example, a worker fastens a screw into fastening point 203. In this case, the worker places screw 215 in a screw hole of fastening point 203 as shown in FIG. 6. Next, the worker holds the grip of wrench 400 with his right hand. Wrench 400 is an example of a first tool. The worker fits the tip (head) of wrench 400 to which a socket is attached into screw 215. After that, the worker rotates wrench 400 with right hand 252 while holding the head of wrench 400 with left hand 251. This causes screw 215 to be fastened.

[0033] In the above-mentioned series of operations, the processing device 150 judges whether the left hand 251 or the right hand 252 has come into contact with any object. More specifically, the processing device 150 calculates the distance between the position of the hand and the object. If the distance is less than a preset threshold, the processing device 150 judges that the hand has come into contact with the object. As an example, in FIG. 5, the diameter of each of the objects 301 to 308 corresponds to the threshold. The sphere indicates the range in which the hand is judged to have come into contact with the virtual object. In the above-mentioned work example, when tightening a screw, the left hand 251 holding the tip of the wrench comes into contact with the object 303. When the processing device 150 judges that the left hand 251 has come into contact with the object 303, it calculates the position of the wrench 400.

[0034] 7, a plurality of markers 430 are attached to a wrench 400. In the illustrated example, a jig 420 is attached to the wrench 400, and a plurality of markers 430 are attached to the jig 420. The jig 420 has a plurality of flat surfaces. Three markers 430 are attached to each flat surface.

[0035] For three markers 430 on one plane, the distance between one marker 430 and another marker 430 is different from the distance between the one marker 430 and yet another marker 430. In other words, the three markers 430 are attached so that when a virtual triangle connecting the three markers 430 is generated, the triangle is not an equilateral triangle.

[0036] Preferably, the multiple markers 430 are arranged such that one side of the triangle is parallel to the extension direction of the tool. In the example shown in Fig. 7, the wrench 400 extends in the Z direction (first direction). The Z direction corresponds to the direction connecting the grip 411 of the wrench 400 and the head 412 of the wrench 400. The direction from the marker 430a to the marker 430c is parallel to the Z direction. A virtual triangle 441 based on the markers 430a to 430c has a side parallel to the Z direction.

[0037] The processing device 150 detects a plurality of markers 430 from images captured by the image camera 131 and the depth camera 132. The processing device 150 calculates the position of the wrench 400 from the positions of at least three markers 430. When a proper fastening operation is performed, the position of the part that overlaps or is close to the object is used as the position of the wrench 400. For example, the processing device 150 calculates the position of the head 412 from the positions of the at least three markers 430.

[0038] As shown in Fig. 8, the worker holds the head 412 with the left hand 251 and grasps the grip 411 with the right hand (not shown). At this time, the left hand 251 comes into contact with the object 303. When the hand comes into contact with the object, it can be estimated that a screw is being turned into the fastening point corresponding to the object. Here, the fastening point corresponding to the object contacted by the hand and estimated to be where the screw is being turned is referred to as the "estimated point."

[0039] When the hand touches an object, the processing device 150 calculates a first position p1 of the head 412 from the positions of the multiple markers 430. The first position p1 is represented by coordinates in a three-dimensional coordinate system set using the markers 210. The processing device 150 calculates a first distance d1 between the first position p1 and the object 303 touched by the hand.

[0040] The processing device 150 compares the first distance d1 with a preset first threshold value. If the first distance d1 is greater than the first threshold value, the processing device 150 determines that the screw is not turned into the estimated location. If the first distance d1 is equal to or less than the first threshold value, the processing device 150 determines that the screw is turned into the estimated location.

[0041] When the processing device 150 determines that a screw has been fastened to the estimated location, the processing device 150 creates a work record. For example, the processing device 150 associates data regarding the fastening location 203 with data indicating that the screw has been fastened, and stores the data.

[0042] When a digital tool is used in the fastening operation, the processing device 150 may receive a detection value from the digital tool. The digital tool includes a sensor that detects a signal generated during the fastening operation. When it is determined that a screw is fastened at an estimated location, the processing device 150 associates the detection value with the ID of the fastening location 203 and stores the detected value in the storage device 170. This makes it possible to automatically create a work record for each fastening location.

[0043] When a digital torque wrench or digital torque driver is used as the tool, the detected value is torque. Alternatively, a threshold value may be set in advance for the torque. The digital torque wrench or digital torque driver may determine whether or not a torque exceeding the threshold value is detected, and output the determination result as the detected value. The digital tool may further detect a rotation angle, etc.

[0044] Advantages of the embodiment will be described. To assist with the work, a method according to the following reference example is considered. In the method according to the reference example, a virtual object corresponding to a fastening location is displayed. The object is displayed at a position where the worker's hands should be placed when performing the work. By displaying the object, the worker can easily visually understand where to position his / her hands and which fastening location to turn the screw at. Furthermore, in this method, by linking data on the estimated location to data indicating that the work has been performed, it is also possible to automatically create a work record.

[0045] 9, 10(a) and 10(b) are schematic diagrams illustrating the state of the work. In the example shown in Fig. 9, a left hand 251 is in contact with an object 304 corresponding to a fastening point 204. A right hand 252 is fastening a screw into a fastening point 203 with a wrench 400. In this case, since the hand is in contact with the object 304, it is determined that a screw has been fastened into the fastening point 204 corresponding to the object 304. According to the method of the reference example, even though a screw has actually been fastened into the fastening point 203, a fastening record for the fastening point 204 is created. For this reason, there is a demand for a technology that can more accurately determine the location where a screw is fastened in the method of the reference example.

[0046] In an embodiment of the present invention, a first distance between an object contacted by a hand and a position of a tool is calculated. When a screw is actually being turned at an estimated location, the tool is located near the estimated location. Therefore, the calculated first distance is also shorter. In the example shown in FIG. 10(a), the fastening location where the screw is fastened by the tool is different from the fastening location where the hand is placed. In the example shown in FIG. 10(b), the screw is fastened at the fastening location where the hand is placed. In the example shown in FIG. 10(b), the first distance d1 is shorter than that in the example shown in FIG. 10(a).

[0047] When the first distance d1 is greater than the first threshold, the processing device 2 determines that a screw has not actually been turned into the estimated location. When the first distance d1 is equal to or less than the first threshold, the processing device 2 determines that a screw has been turned into the estimated location. According to the embodiment, it is possible to more accurately determine whether or not a screw has actually been turned into a fastening location corresponding to an object contacted by a hand. As a result, it is possible to create a more accurate work record. According to the embodiment, it is possible to more appropriately support work using screws.

[0048] In addition, when turning a screw with a tool, it is desirable to rotate the tool while holding the grip and head of the tool with your hands. It is possible to handle the tool by holding only the grip, but there is a possibility that the object or the screw may be damaged due to unintended movement of the tool during the operation. According to the embodiment, it is easy to prevent the execution of inappropriate work as shown in Fig. 10(a).

[0049] 11(a) and 11(b) are schematic diagrams showing an example of output in the embodiment of the present invention. When the first distance d1 is greater than the first threshold, the processing device 2 may display a message 311 (alert) prompting the worker to check the work, as shown in Fig. 11(a). The content of the message may be an error code or the like. By displaying the message, the worker can easily recognize that an inappropriate work is being performed.

[0050] When the first distance d1 is equal to or less than the first threshold, the processing device 2 may display a message 312 indicating that the work is appropriate, as shown in Fig. 11(b), or may not display any message.

[0051] The first threshold value to be compared with the first distance d1 is set in advance by an administrator of the processing system 10. Alternatively, the first threshold value may be automatically calculated by the processing device 2 from the interval between the fastening points or the length of the tool, etc.

[0052] When the tightening of the screw is completed, the worker inputs completion information indicating the completion of tightening using the input device 3. When it is determined that the screw has been tightened at the estimated location, the processing device 2 accepts the input. The processing device 2 links the completion information to data regarding the estimated location and stores it in the storage device 5.

[0053] When a digital tool is used, the processing device 2 receives a detection value from the digital tool. When it is determined that a screw is fastened to the estimated location, the processing device 2 associates the detection value with data related to the estimated location and stores the data in the storage device 5. This makes it possible to automatically generate an accurate work record.

[0054] For a fastening point where a screw has been fastened and completion information or a detection value has been received, the processing device 2 hides the object corresponding to that fastening point. Alternatively, the processing device 2 may display the object corresponding to that fastening point differently from the display of other objects. This allows the worker to easily visually identify the points where a screw has not yet been fastened.

[0055] FIG. 12 is a flowchart illustrating an example of a processing method according to the embodiment. In the processing method M1 shown in Fig. 12, the processing device 2 sets a base point (step S0). The base point is set based on a marker or a hand gesture. The processing device 2 displays one or more first objects respectively corresponding to one or more fastening points (step S1). The processing device 2 determines whether a hand has contacted any of the first objects (step S2).

[0056] When the hand touches any of the first objects, the processing device 2 calculates a first position of the first tool from a plurality of markers attached to the first tool (step S3). The processing device 2 calculates a first distance between the first position and the first object touched by the hand (step S4). The processing device 2 determines whether the first distance is equal to or smaller than a first threshold (step S5). When the first distance is greater than the first threshold, the processing device 2 displays a message prompting the worker to confirm (step S6). After that, the first position is calculated again (step S3).

[0057] If the first distance is equal to or less than the first threshold, the processing device 2 accepts the detection value transmitted from the digital tool (step S7). The processing device 2 associates the detection value with data related to the fastening points and stores it in the storage device 5 (step S8). The processing device 2 determines whether the work is completed (step S9). That is, the processing device 2 determines whether the screws have been turned for all fastening points where the objects are displayed. If the work is not completed, the determination of step S2 is executed again. If it is determined that the work is completed, the processing device 2 ends the processing method M1 shown in FIG. 12.

[0058] An example of a specific process according to an embodiment of the present invention will be described below.

[0059] (Tool and marker registration) Fig. 13(a) is a schematic diagram showing an example of a tool, and Figs. 13(b) to 13(d) are schematic diagrams showing an example of a marker. The ID of the marker 430, the position of the marker 430, the position of the tool calculated by the marker 430, etc. are registered in advance before the work. Fig. 13(a) to Fig. 13(d) show the state when the jig 420 is viewed from different directions. The external shape of the jig 420 is a rectangular parallelepiped, and the jig 420 has four planes 421 to 424. Markers 430a to 430c are affixed to the plane 421. Markers 430e to 430g are affixed to the plane 422. Markers 430i to 430k are affixed to the plane 423. Markers 430m to 430o are affixed to the plane 424.

[0060] Markers 430a-430c are attached so that virtual triangle 441 obtained by connecting these markers is an isosceles triangle. Similarly, markers 430e-430g, markers 430i-430k, and markers 430m-430o are also attached so that virtual triangles 442-444 obtained by connecting a plurality of markers are isosceles triangles. Each isosceles triangle has sides parallel to the Z direction. In addition, each marker is provided at a position where triangles 441-444 are rotationally symmetrical with respect to the center of wrench 400 in the XY plane. The XY plane is a plane perpendicular to the Z direction in which wrench 400 extends.

[0061] In advance preparation, the ID of each marker 430 and the position of each marker 430 in an arbitrary spatial coordinate system are registered. In addition, a part of the position of the wrench 400 is registered for each combination of markers 430a to 430c, markers 430e to 430g, markers 430i to 430k, and markers 430m to 430o. A different position may be registered for each combination of markers, or a common position may be registered for each combination. As an example, position p0 shown in FIG. 13(a) is registered for each combination of markers. The registered position is the position where the worker holds the tool when the tool is used for work.

[0062] In addition, a position-related attribute is further registered for each marker 430. The attribute indicates where the marker 430 exists among the areas 451 to 453 shown in Fig. 13(a). The areas 451 to 453 are aligned in the Z direction. This attribute is used to improve the estimation accuracy of the position of the tool during work, as will be described later.

[0063] The ID of the marker 430, the position of the marker 430, the position of the tool corresponding to the marker, and the position attributes are stored in association with the ID of the tool. This completes the advance preparation for the marker. Hereinafter, the marker position and the tool position registered in the advance preparation are referred to as the advance marker position and the advance tool position, respectively.

[0064] (Calculation of tool position) The processing device 2 detects the markers 430 appearing in the image. When four or more markers 430 are recognized, the processing device 2 extracts three markers 430. The processing device 2 calculates the position of the tool from the positions of the three extracted markers 430.

[0065] 14(a) to 14(h) are schematic views for explaining the processing in the processing system according to the embodiment. For example, the photographing device 1 obtains an image IMG shown in Fig. 14(a). Markers 430a to 430c and markers 430e to 430g are captured in the image IMG. The processing device 2 extracts combinations of three markers 430 from the markers 430a to 430c and markers 430e to 430g. The processing device 2 generates a triangle connecting the three markers 430 for each combination. Figs. 14(b) to 14(h) show examples of the generated triangles.

[0066] Next, the processing device 2 refers to the data of each recognized marker 430. As described above, position attributes are registered for each marker 430. The processing device 2 extracts triangles in which the marker 430 exists in each of the regions 451 to 453 from the generated triangles. That is, the processing device 2 extracts triangles having sides along the Z direction. As a result of the processing, the triangles shown in Figs. 14(b) to 14(e) are extracted, and the triangles shown in Figs. 14(f) to 14(h) are not extracted.

[0067] The processing device 2 calculates a first vector parallel to the normal of the extracted triangle. The processing device 2 also calculates a second vector connecting the observation point (the image capture device 1) and the center point of the triangle. The processing device 2 calculates the angle between the first vector and the second vector. The processing device 2 calculates the angle between the first vector and the second vector for each of the extracted multiple triangles.

[0068] The processing device 2 extracts one triangle that has the smallest angle from the extracted triangles. The processing device 2 determines the marker 430 corresponding to the extracted triangle as the marker 430 to be used in calculating the tool position. In the illustrated example, the markers 430a to 430c are determined as the markers to be used in calculating the tool position. Hereinafter, the three determined markers are also referred to as the first marker, the second marker, and the third marker.

[0069] The processing device 2 measures the position of each of the first to third markers. Furthermore, the processing device 2 refers to the IDs of each of the first to third markers. The processing device 2 acquires the prior positions of the first to third markers and the prior position of the tool corresponding to the first to third markers. The processing device 2 calculates the position of the tool during work using the positions of the first to third markers obtained during work, the prior positions of the first to third markers, and the prior position of the tool.

[0070] FIG. 15 is a schematic diagram showing the movement of the tool. Even if a tool is used in a task, the relative position of the tool to the positions of the first to third markers does not change. Here, as shown in FIG. 15, the prior position of the first marker is (x1, y1, z1). The prior position of the second marker is (x2, y2, z2). The prior position of the third marker is (x3, y3, z3). The prior position of the tool is (x0, y0, z0). The positions of the first to third markers measured during the task are (x1', y1', z1'), (x2', y2', z2'), and (x3', y3', z3'), respectively. The position of the tool during the task is (x0', y0', z0').

[0071] FIG. 16 shows simultaneous equations that represent the relationship between the position of each marker before movement and the position of each marker after movement. The relationship between the positions of the first to third markers beforehand and the positions of the first to third markers during work is expressed by the equation in FIG. 16. Each position is substituted into the relational equation shown in FIG. 16 to solve the simultaneous equations. At this time, coefficients b1 to b3 are set to zero. Coefficients b1 to b3 correspond to the amount of translation from the positions of the first to third markers beforehand to the positions of the first to third markers during work. Coefficients a11 to a33 are calculated by solving the simultaneous equations. Coefficients a11 to a33 indicate the amount of rotation and the amount of deformation from the positions of the first to third markers beforehand to the positions of the first to third markers during work.

[0072] Then, the difference between the previous position of the midpoint of the first to third markers and the position of the midpoint of the first to third markers during the work is calculated. The difference in the X direction, the difference in the Y direction, and the difference in the Z direction are used as coefficients b1 to b3, respectively. Note that the spatial coordinate system used when registering the positions in advance may be different from the spatial coordinate system used during the work. In that case, the change in the base point of those spatial coordinate systems is also expressed as the rotation, deformation, and translation of each marker.

[0073] FIG. 17 is a determinant that represents the relationship between the position of the tool before the movement and the position of the tool after the movement. The processor 2 uses the coefficients a11 to a33 and the coefficients b1 to b3 as variables of an affine transformation matrix. As shown in Fig. 17, the position (x0', y0', z0') of the tool during work is calculated by multiplying the previous position (x0, y0, z0) of the tool by the affine transformation matrix.

[0074] The rotation and translation may be calculated by a method other than the affine transformation matrix. For example, the processing device 2 calculates the difference between the center points of the first to third markers in advance and the center points of the first to third markers during work as the translation distance. The processing device 2 also calculates normal vectors of the first to third markers in advance. Here, the normal vector is called the previous normal vector. The processing device 2 calculates normal vectors of the first to third markers during work. Here, the normal vector is called the current normal vector. The processing device 2 calculates the direction of rotation and the rotation angle for matching the previous vector with the current vector. In this way, the translation and rotation of the first to third markers are calculated. The processing device 2 calculates the position of the tool during work by adding the calculated translation and rotation to the position of the tool registered in advance.

[0075] 18 and 19 are flowcharts showing the process for calculating the tool position. First, as a preliminary preparation, the process shown in Fig. 18 is executed. Specifically, an operator attaches a plurality of markers 430 to a tool (step S20). The operator who attaches the markers 430 may be different from the operator who performs the fastening work. Furthermore, the markers 430 may be attached by a person other than the operator, such as a manager of the processing system 10 or a manager of the tools.

[0076] The photographing device 1 photographs each marker 430, and the processing device 2 recognizes each marker 430 from the image (step S21). The worker registers the ID of each marker (step S22). The processing device 2 calculates the position of each marker and registers it in association with the marker ID (step S23). The worker specifies the position of the tool for the combination of markers. The processing device 2 calculates the specified position and registers the tool position (step S24). The worker registers the tool ID in association with the marker ID, marker position, and tool position (step S25).

[0077] Next, during work, the process shown in Fig. 19 is executed. Specifically, the processing device 2 recognizes the markers 430 from the image (step S30). The processing device 2 extracts the first to third markers used to calculate the position of the tool from the three or more markers 430 (step S31). The processing device 2 acquires the positions of the first to third markers during work (step S32). The processing device 2 acquires the positions of the first to third markers in advance (step S33). The processing device 2 calculates the translation and rotation of the tool (step S34). In addition to the translation and rotation, deformation may also be calculated.

[0078] The processing device 2 acquires the previous tool positions corresponding to the first to third markers (step S35). The processing device 2 calculates the position of the tool during work using the calculated translation and rotation and the acquired previous tool position (step S36). Using the calculated tool position, the processing from step S4 onwards shown in FIG. 12 is executed.

[0079] (filter) In the above example, when a triangle is generated, the vertices of the triangle are set to the positions of the markers 430. Instead of this method, a triangle may be generated based on three points whose relative positional relationship is fixed. Here, the points whose positional relationship is fixed are called "fixed points." Also, a triangle generated based on three fixed points is called a "filter." By using a filter, it is possible to reduce errors in the rotation, deformation, and translation of the triangle caused by errors in the position of the markers 430.

[0080] FIGS. 20(a), 20(b), 21(a), and 21(b) are schematic diagrams for explaining filters. The processing device 2 sets three fixed points for a combination of three markers 430 according to a predetermined method. The processing device 2 connects the three fixed points to generate a filter. For example, as shown in FIG. 20(a), the processing device 2 sets a filter 460 for the positions of the first to third markers in advance. As shown in FIG. 20(b), the processing device 2 sets a filter 461 for the positions of the first to third markers during work. The shape and size of the filter 460 are the same as those of the filter 461.

[0081] In the illustrated example, a center point C2 of the filter is located at a center point C1 of the positions of the three markers 430. The orientation of the filter is set so that the angle between the normal vector of a triangle 441 based on the three markers 430 and the normal vector of the filter is minimized. An angle around the normal vector is set using one or more markers 430, the center points C1 and C2, and one or more vertices of the filter. One of the following methods is used to set the angle around the normal vector.

[0082] As shown in FIG. 21(a), the filter 460 has vertices 460a to 460c. The vertices 460a to 460c are individually identifiable. In the first method, the angle of the filter 460 around the normal vector is adjusted so that any one of the markers 430, the center point C1, the center point C2, and any one of the vertices 460a to 460c are aligned on a straight line. More preferably, the filter 460 is rotated so that the marker 430 farthest from the center point C1 or C2 is positioned on a straight line. By using the marker 430 farthest from the center point, the error can be further reduced. The angle of the filter 461 around the normal vector is also set by the same method.

[0083] In the second method, as shown in Fig. 21(b), the angle of the filter 460 may be adjusted so that the sum of the distance Da between the marker 430a and the vertex 460a, the distance Db between the marker 430b and the vertex 460b, and the distance Dc between the marker 430c and the vertex 460c is minimized. In the example shown in Fig. 20(a) and Fig. 20(b), the angle of the filter 460 and the angle of the filter 461 are set by the second method.

[0084] The processing device 2 uses the positions of the three fixed points of the filter 460a as the positions of the first to third markers beforehand. The processing device 2 uses the positions of the three fixed points of the filter 460b as the positions of the first to third markers during work. Thereafter, the rotation, deformation, and translation of the tool are calculated in the same manner as in the above example, and the position of the tool during work is calculated.

[0085] When calculating the position of the marker 430 from an image, there is a position error. In particular, when measuring the position of the marker 430 from an RGB image, the error in the position of the marker 430 is likely to be large due to insufficient resolution, image distortion due to the shooting direction, image distortion due to lens distortion, and the like. Regarding the problem of resolution, it is preferable to use a high-resolution camera, but the larger the number of pixels in the image, the longer the time required for overall image processing. It may be difficult to determine in real time during work whether the screw is fastened. On the other hand, if the resolution is reduced, the image processing can be accelerated, but the error in the position of the marker 430 increases. Furthermore, during work, it is rare for the marker 430 to be directly facing the imaging device 1. If the marker 430 is positioned at an angle to the imaging device 1, the marker 430 is photographed in a deformed state, and the error in the position of the marker 430 becomes large. Furthermore, during work, the MR device of the worker may be located in a position close to the tool. If the angle of view of the imaging device 1 is small, the tool may be out of the angle of view depending on the orientation of the imaging device 1. If the tool and the marker 430 are not captured in the image, the judgment using the marker 430 cannot be performed. For this reason, it is preferable that the imaging device 1 has a wide-angle lens and the angle of view of the imaging device 1 is large. On the other hand, when the angle of view is large, distortion becomes large, especially in the periphery of the image. If the marker 430 is located in the periphery of the image and distorted, the error in the position of the marker 430 becomes large.

[0086] If the error in the position of the marker 430 is large, the errors in the rotation, deformation, and translation of the tool calculated from the position also become large. As a result, there is a possibility that an erroneous determination result is output regarding whether or not a screw is fastened at the estimated location. By using a filter, even if the error in the position of the marker 430 is large, it is possible to reduce the deformation of the generated triangle in particular. As a result, it becomes possible to more accurately determine whether or not a screw is fastened at the estimated location.

[0087] (First Modification) In the flowchart shown in Fig. 12, after it is determined that the hand is in contact with the object, the process from step S3 onwards is executed. The process from step S3 onwards is executed only while the hand is in contact with the object. Alternatively, if it is determined in step S2 that the hand is in contact with the object, step S3 onwards may be executed only during a first period. If the hand does not come into contact with the object during the first period, the process is interrupted and returns to step S2.

[0088] The first period can be appropriately set and changed. Normally, the hands are always placed in a specific position during the fastening operation. In the example shown in Figs. 5 to 7, the hands grip the head 412 of the wrench 400, and the position of the hands does not change substantially. For this reason, the first period may be set to be short.

[0089] On the other hand, it may be difficult to keep the hands in a specific position when working in a special posture, working at a high place, etc. In such a case, an object may be displayed at a position where the hands pass during work, and the first period may be set to be long.

[0090] 22 to 24 are schematic diagrams for explaining a first modified example of the embodiment. 22, fastening points 221-225 of article 220 are located at a high position. Therefore, a worker uses a long wrench 400 to fasten screws to fastening points 221-225. During the fastening work, the worker cannot reach head 412 of wrench 400. Therefore, the worker grips bar 413 between head 412 and grip 411 instead of head 412 to fasten the screws.

[0091] In this case, the processing device 2 displays a virtual object 321 (an example of a first object) at a position where the hand gripping the bar 413 passes during work, as shown in FIG. 23. When the hand comes into contact with the object 321, the processing device 2 executes the processing from step S3 shown in FIG. 12 onwards. In this case, the hand comes into contact with the object 321 only at certain times during the work. For this reason, the first period is set long. For example, the first period is set to 10 seconds. Tightening of a general screw can be completed within 5 seconds. When the first period is set to 10 seconds, step S3 and subsequent steps are almost certainly executed as the screw is tightened.

[0092] When a digital tool is used, the timing at which a detection value is received from the digital tool may be set as the starting point of the count for the first period. The processing device 2 updates the starting point of the count for the first period every time a detection value is received from the digital tool. If the first period has elapsed without receiving a detection value from the digital tool, the processing device 2 interrupts the processing from step S3 onwards. In this case, the first period is also set to, for example, 10 seconds.

[0093] Alternatively, the process from step S3 onwards may be continued until the hand comes into contact with another object. The process from step S3 onwards may be continued until a fastening completion signal is generated. For example, in the case where a digital tool is used, the processing device 2 issues a fastening completion signal when it receives a detection value from the digital tool that exceeds a torque value required for fastening. Alternatively, the processing device 2 issues a fastening completion signal when it receives a detection value from the digital tool that indicates that the torque value required for fastening has been exceeded.

[0094] When the hand comes into contact with the object 321, the processing device 2 executes the processes from step S3 onward for a first period from the contact. If the hand comes into contact with the object 321 again before the first period has elapsed, the count of the first period is reset. That is, the processing device 2 executes the processes from step S3 onward until the first period has elapsed since the hand last came into contact with the object 321.

[0095] 24, a plurality of objects 321 to 323 may be displayed along the direction in which the hand is moved. When the hand comes into contact with any of the objects 321 to 323, the processing device 2 starts counting the first period.

[0096] When the position of head 412 is registered as the position of wrench 400 corresponding to marker 430, in the example shown in Figures 22 to 24, first distance d1 is longer than in the example shown in Figures 5 to 7. For this reason, the first threshold value compared with first distance d1 is also set to be larger than in the example shown in Figures 5 to 7. The first threshold value can be set based on the distance between a displayed object and a fastening point.

[0097] According to the first variant, even when the tool is used with an unusual holding manner, it is possible to estimate the location where the screw is fastened, and it is possible to determine with greater accuracy whether the screw is actually fastened at the estimated location.

[0098] (Second Modification) Fig. 25(a) is a schematic diagram showing an example of a first tool, and Fig. 25(b) is a schematic diagram showing an example of a second tool. The embodiment of the present invention can also be applied to a task in which multiple tools are used. For example, as shown in Fig. 25(a) and Fig. 25(b), a screw is fastened using a wrench 400 (an example of a first tool) and an extension bar 500 (an example of a second tool).

[0099] As described above, a plurality of markers 430 are attached to the wrench 400. Similarly, a plurality of markers 530 are attached to the extension bar 500. In the illustrated example, a jig 520 is attached to the extension bar 500. The jig 520 has the same shape as the jig 420 of the wrench 400. A plurality of markers 530 are affixed to each flat surface of the jig 520.

[0100] FIGS. 26(a), 26(b), 27(a), and 27(b) are schematic views for explaining a second modified example of the embodiment. When work is performed using the wrench 400 and the extension bar 500, for example, as shown in Fig. 26(a), the worker fits one end of the extension bar 500 onto the screw. The worker fits the head of the wrench 400 onto the other end of the extension bar 500. The worker holds the head of the wrench 400 with the right hand 252 and grasps the grip of the wrench 400 with the left hand 251. By rotating the wrench 400 in this state, the screw is fastened to the fastening point 204 via the extension bar 500.

[0101] For example, a worker wears the MR device 100 shown in Fig. 2 and performs work. The processing device 150 displays an object 331 (an example of a first object) at a position where the worker's hand should be placed, as shown in Fig. 26(b). When the extension bar 500 is used, the hand is moved away from the fastening point 204. Therefore, the object 331 is also displayed at a position away from the fastening point 204.

[0102] The processing device 150 further displays an object 332 (an example of a second object). The object 332 is located between the fastening point 204 and the object 331, and extends along a direction connecting the fastening point 204 and the object 331. The shape and size of the object 332 correspond to the shape and size of the extension bar 500. The object 332 indicates the position and posture of the extension bar 500 during work.

[0103] When the worker's hand comes into contact with the object 331, the processing device 150 calculates a first position P1 of the wrench 400 from the positions of the multiple markers 430. In addition, the processing device 150 calculates a second position P2 of the extension bar 500 from the positions of the multiple markers 530.

[0104] The processing device 150 calculates a first distance between the first position P1 and the object 331 touched by the hand. The processing device 150 compares the first distance with a preset first threshold. Furthermore, the processing device 150 calculates a second distance between the second position P2 and the object 332. The processing device 150 compares the second distance with a preset second threshold. The second threshold may be the same as the first threshold or may be different from the first threshold. The second threshold is preset by an administrator of the processing system 10. Alternatively, the second threshold may be automatically calculated from the interval between the fastening points, the size of the extension bar, or the like.

[0105] When the first distance is greater than the first threshold value or when the second distance is greater than the second threshold value, the processing device 150 determines that a screw is not fastened at the estimated location. When the first distance is equal to or less than the first threshold value and the second distance is equal to or less than the second threshold value, the processing device 150 determines that a screw is fastened at the estimated location.

[0106] 27(a), a screw is fastened at a location other than the estimated location while a hand not holding a tool is in contact with the object 331. In this case, the first distance d1 becomes larger than the first threshold value, and the second distance d2 becomes larger than the second threshold value. Therefore, the processing device 150 determines that a screw is not fastened at the estimated location.

[0107] As another example, as shown in FIG. 27(b), a screw is fastened at a location other than the estimated location while the hand holding the tool is in contact with the object 331. In this example, the extension bar 500 is used at an incorrect angle. In this case, the first distance d1 is smaller than the first threshold value, but the second distance d2 is larger than the second threshold value. Therefore, the processing device 150 determines that the screw is not fastened at the estimated location. As shown in FIG. 27(a) and FIG. 27(b), the processing device 150 may display a message 311 prompting the user to confirm the work.

[0108] FIG. 28 is a flowchart showing a processing method according to the second modified example of the embodiment. A processing method M2 according to a second modified example shown in Fig. 28 further includes steps S11 to S14 compared to the processing method M1 shown in Fig. 12. Moreover, the processing method M2 includes step S10 instead of step S1.

[0109] In step S10 after step S0, the processing device 2 displays a first object and a second object. The first object indicates a position where a hand holding a first tool should be placed. The second object indicates an appropriate position and posture of a second tool. Then, step S2 is executed. If it is determined in step S5 that the first distance is equal to or less than the first threshold, the processing device 2 calculates a second position of the second tool (step S11). The processing device 2 calculates a second distance between the second position and the second object (step S12). The processing device 2 determines whether the second distance is equal to or less than the second threshold (step S13). If the second distance is greater than the second threshold, the processing device 2 displays a message prompting the operator to confirm (step S14). Then, the second position is calculated again (step S11).

[0110] If the second distance is equal to or less than the second threshold, the processing device 2 accepts the detection value transmitted from the digital tool (step S7). The processing device 2 associates the detection value with data related to the fastening point and stores it in the storage device 5 (step S8). The processing device 2 determines whether the work is completed (step S9). If it is determined that the work is completed, the processing system 10 ends the processing method M2 shown in FIG.

[0111] In the processing method M2, the specific processing order is not limited to the example shown in Fig. 28 and can be changed as appropriate. For example, the first distance and the second distance may be calculated after the first position and the second position are calculated. The second position may be calculated, the second distance and the second threshold are compared, and then the first position may be calculated.

[0112] FIG. 29 is a flowchart showing the flow of data in the processing method according to the embodiment. The storage device 5 stores master data 51 and history data 52. The master data 51 includes operation master data 51a, tool master data 51b, base point master data 51c, and fastening point master data 51d. The master data 51 is prepared in advance before the screw tightening.

[0113] As shown in Fig. 29, the processing device 2 accepts the selection of an operation and a tool (step S40). The operation and the tool are selected by the worker. The operation to be performed and the tool to be used may be instructed by a higher-level system, and the processing device 2 may accept the selection by the instruction. Based on data obtained from the imaging device 1 or other sensors, the processing device 2 may determine the operation to be performed and the tool to be used. The processing device 2 selects the operation and the tool based on the determination result.

[0114] The work master data 51a stores a work ID, a work name, an item ID, and an item name. The processing device 2 can accept either the work ID, the work name, the item ID, or the item name as a work selection. The tool master data 51b stores a tool type, a tool ID, a marker ID, a marker position, a tool position, and marker attributes. The tool type indicates a classification of a tool based on structure, external shape, performance, or the like. The processing device 2 can accept either the tool type or the tool ID as a tool selection.

[0115] When the task and tool are selected, the processing device 2 executes one of the above-mentioned processing methods (step S41). In this processing method, when a base point is set, the base point master data 51c is referenced. The base point master data 51c stores a method for setting a base point for each task. The processing device 2 obtains the method for setting a base point for the selected task, and sets the base point based on an input by the worker.

[0116] The processing device 2 also refers to the fastening point master data 51d. In the fastening point master data 51d, the fastening point ID is stored, and for each fastening point ID, the fastening position, angle, extension type, torque value, object shape, display mode, and message content are stored. The fastening position indicates the position of each fastening point. The angle indicates the angle of the tool or extension bar when tightening a screw at each fastening point. The extension type indicates the classification of the extension bar according to the structure, external shape, or performance. The torque value indicates the magnitude of the torque required to tighten a screw at each fastening point. The object shape indicates the shape of an object displayed corresponding to each fastening point. The display mode indicates the color or pattern of each displayed object. The message content is the content displayed when it is determined that a screw is not tightened at the estimated point.

[0117] The processing device 2 displays each object based on the set base point and the referenced data. The position at which the object is displayed is calculated based on the base point, fastening position, angle, extension type, tool type, etc. For example, the length of the extension bar is specified by the extension type. A second object of the specified length is displayed at a set angle based on the fastening position. Also, a first object is displayed at a position separated by the length of the extension bar from the fastening position at a set angle.

[0118] When it is determined that a screw has been fastened at the estimated location and a detection value is received from the tool, the processing device 2 links the detection value to data relating to the fastening location and records the data in the history data 52. In the illustrated example, a torque value is recorded as the detection value. A fastening location ID is recorded as data relating to the fastening location. In addition, a job ID, a tool ID, and a tool model are linked to and recorded in this data.

[0119] FIG. 30 is a schematic diagram showing a hardware configuration. 30 is used as the processing device 2 or the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0120] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0121] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0122] The storage device 94 stores data necessary for executing the program and data obtained by executing the program. The storage device 94 includes a solid state drive (SSD) and the like. The storage device 94 may be used as the storage device 5 or the storage device 170.

[0123] An input interface (I / F) 95 can connect the computer 90 to the input device 3. The CPU 91 can read various data from the input device 3 via the input I / F 95.

[0124] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to the display device 4 via the output I / F 96 and cause the display device 4 to display information.

[0125] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0126] The data processing by the processing device 2 or the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0127] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0128] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0129] In the above description of the embodiment, an example in which the processing system 10 is realized as an MR device has been mainly described. The embodiment of the present invention is not limited to this example. The processing system 10 may be realized by a general-purpose personal computer. In this case, a keyboard, a microphone, a touch pad, or the like is used as the input device 3. A monitor is used as the display device 4.

[0130] In the above-described examples, the examples in which a screw is tightened at a fastening point have been mainly described. The embodiments of the present invention are applicable not only to cases in which a screw is tightened at a fastening point, but also to cases in which a screw at a fastening point is loosened. For example, a screw at a fastening point is loosened when performing maintenance, inspection, or repair of a product. According to the embodiments of the present invention, it is possible to more accurately determine whether or not a screw is actually being tightened at a fastening point corresponding to an object contacted by a hand. As a result, it is possible to create a more accurate work record.

[0131] Embodiments of the invention include the following features. (Feature 1) Acquire an image showing a worker's hand and a plurality of markers attached to a first tool for turning a screw; calculating a first position of the first tool from positions of the plurality of markers when the hand comes into contact with a first virtual object; A processing device that determines whether the screw is turned into a fastening point corresponding to the first object based on a first distance between the first object and the first position. (Feature 2) receiving a detection value detected by the first tool; The processing device according to feature 1, wherein when it is determined that the screw is being turned into the fastening point, the detection value is linked to data regarding the fastening point and stored. (Feature 3) calculating a second position of the second tool from positions of a plurality of markers attached to the second tool when the screw is turned by the first tool via a second tool; calculating a second distance between the second position and a virtual second object displayed between the fastening point and the first object; 3. The processing device according to feature 1 or 2, further comprising: a processor configured to determine whether the screw is turned relative to the fastening point based on the first distance and the second distance. (Feature 4) 4. The processing device according to any one of features 1 to 3, further comprising: a message displayed when it is determined that the screw has not been turned into the fastening portion. (Feature 5) When four or more of the markers are captured in the image, a first marker, a second marker, and a third marker are extracted from the four or more markers; 5. The processing device according to any one of features 1 to 4, further comprising: a processing device configured to measure the positions of the first to third markers and calculate the first position. (Feature 6) Calculating the amount of rotation and the amount of translation from the preregistered positions of each of the first to third markers to the measured positions of each of the first to third markers; The processing device described in feature 5 calculates the first position by rotating and translating the position of the first tool, which is pre-registered corresponding to the first to third markers, in accordance with the amount of rotation and the amount of translation. (Feature 7) calculating an affine transformation matrix indicating rotation, deformation, and translation from the preregistered positions of each of the first to third markers to the measured positions of each of the first to third markers; The processing device according to feature 5, wherein the first position is calculated by affine transforming a position of the first tool that is registered in advance using the affine transformation matrix. (Feature 8) Extracting a plurality of combinations of three markers from the plurality of markers; generating triangles for each of said combinations; extracting one or more of the triangles having a side along a first direction corresponding to an orientation of the first tool from the plurality of triangles; extracting one triangle from the one or more triangles, the triangle having the smallest angle between a vector connecting the observation point and the triangle and a normal vector of the triangle; The processing device according to any one of features 5 to 7, wherein three markers corresponding to one triangle are extracted from the plurality of markers as the first marker, the second marker, and the third marker. (Feature 9) Based on the plurality of markers, three points having a fixed relative positional relationship are set; 5. The processing device according to any one of features 1 to 4, wherein the first position is calculated using the positions of each of the three points. (Feature 10) A processing device according to any one of features 1 to 9, a display device for displaying the first object; an imaging device that images the hand and the plurality of markers; A processing system comprising: (Feature 11) displaying a virtual first object indicating a position where the worker's hand will be placed; Taking an image of a plurality of markers attached to a first tool for turning a screw and the hand; calculating a first distance between a first position of the first tool calculated from positions of the plurality of markers and the first object when the hand comes into contact with the first object; The mixed reality device displays a message if the first distance is greater than a first threshold. (Feature 12) displaying on a display device a first virtual object indicating a position where a worker's hand will be placed; causing a processor to calculate a first position of a first tool for turning a screw using a plurality of markers attached to the first tool; The processing method includes causing the processing device to determine whether the screw is turned into a fastening point corresponding to the first object based on a first distance between the first position and the first object. (Feature 13) A jig having a flat surface is attached to the first tool, the plurality of markers including a first marker, a second marker, and a third marker are provided on the plane; 13. The method of claim 12, wherein a distance between the first marker and the second marker is different from a distance between the second marker and the third marker. (Feature 14) 14. A program for causing a computer to execute the processing method according to feature 12 or 13. (Feature 15) A storage medium storing the program according to feature 14.

[0132] According to the embodiments described above, a processing device, a processing system, a mixed reality device, a processing method, a program, and a storage medium are provided that are capable of determining with higher accuracy whether or not a screw is actually fastened to a fastening point corresponding to an object touched by a hand. According to the embodiments, it is possible to more appropriately support work using screws.

[0133] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0134] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0135] 1: Photographing device, 2: Processing device, 3: Input device, 4: Display device, 5: Storage device, 10: Processing system, 51: Master data, 51a: Work master data, 51b: Tool master data, 51c: Base point master data, 51d: Fastening point master data, 52: History data, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: MR device, 101: Frame, 111, 112: Lens, 121, 122: Projection device, 131: Image camera, 132: Depth camera, 140: Sensor, 141: Microphone, 150: Processing device, 160: Battery, 170: Storage device, 200: Article, 201-208: fastening point, 210: marker, 215: screw, 220: object, 221-225: fastening point, 251: left hand, 252: right hand, 261, 262: object, 301-308: object, 311, 312: message, 321-323, 331, 332: object, 400: wrench, 411: grip, 412: head, 413: bar, 420: jig, 421-424: plane, 430: marker, 441-444: triangle, 451-453: area, 460, 461: filter, 460a-460c: vertex, 500: extension bar, 520: jig, 530: marker, C1, C2: Center point, IMG: Image, M1, M2: Processing method, d1: First distance, d2: Second distance, Da~Dc: Distance, p1, P1: First position, P2: Second position

Claims

1. Acquire an image showing a worker's hand and a plurality of markers attached to a first tool for turning a screw; calculating a first position of the first tool from positions of the plurality of markers when the hand comes into contact with a first virtual object; A processing device that determines whether the screw is turned into a fastening point corresponding to the first object based on a first distance between the first object and the first position.

2. receiving a detection value detected by the first tool; The processing device according to claim 1 , wherein, when it is determined that the screw is being turned into the fastening point, the detection value is associated with data relating to the fastening point and stored.

3. calculating a second position of the second tool from positions of a plurality of markers attached to the second tool when the screw is turned by the first tool via a second tool; calculating a second distance between the second position and a virtual second object displayed between the fastening point and the first object; The processing device according to claim 1 , further comprising: a determining unit configured to determine whether the screw is turned relative to the fastening point based on the first distance and the second distance.

4. The processing device according to claim 1 , further comprising: a display unit configured to display a message when it is determined that the screw is not turned into the fastening portion.

5. When four or more of the markers are captured in the image, a first marker, a second marker, and a third marker are extracted from the four or more markers; The processing device according to claim 1 , wherein the first position is calculated by measuring the positions of the first to third markers.

6. Calculating an amount of rotation and an amount of translation from the preregistered positions of each of the first to third markers to the measured positions of each of the first to third markers; 6. The processing device according to claim 5, wherein the first position is calculated by rotating and translating a position of the first tool that is pre-registered corresponding to the first to third markers in accordance with the amount of rotation and the amount of translation.

7. calculating an affine transformation matrix indicating rotation, deformation, and translation from the preregistered positions of each of the first to third markers to the measured positions of each of the first to third markers; The processing device according to claim 5 , wherein the first position is calculated by affine transforming a pre-registered position of the first tool using the affine transformation matrix.

8. Extracting a plurality of combinations of three markers from the plurality of markers; generating triangles for each of said combinations; extracting one or more of the triangles having a side along a first direction corresponding to an orientation of the first tool from the plurality of triangles; extracting one triangle from the one or more triangles, the triangle having the smallest angle between a vector connecting the observation point and the triangle and a normal vector of the triangle; The processing device according to claim 5 , wherein three of the markers corresponding to one of the triangles are extracted from the plurality of markers as the first marker, the second marker, and the third marker.

9. Based on the plurality of markers, three points having a fixed relative positional relationship are set; The processing device of claim 1 , wherein the first position is calculated using the positions of each of the three points.

10. A processing device according to any one of claims 1 to 9, a display device for displaying the first object; an imaging device that images the hand and the plurality of markers; A processing system comprising:

11. displaying a virtual first object indicating a position where the worker's hand will be placed; Taking an image of a plurality of markers attached to a first tool for turning a screw and the hand; calculating a first distance between a first position of the first tool calculated from positions of the plurality of markers and the first object when the hand comes into contact with the first object; A mixed reality device that displays a message if the first distance is greater than a first threshold.

12. displaying, on a display device, a virtual first object indicating a position where a worker's hand will be placed; causing a processor to calculate a first position of a first tool for turning a screw using a plurality of markers attached to the first tool; The processing method includes causing the processing device to determine whether the screw is turned into a fastening point corresponding to the first object based on a first distance between the first position and the first object.

13. A jig having a flat surface is attached to the first tool, the plurality of markers including a first marker, a second marker, and a third marker are provided on the plane; The method of claim 12 , wherein a distance between the first marker and the second marker is different from a distance between the second marker and the third marker.

14. A program for causing a computer to execute the processing method according to claim 12 or 13.

15. A storage medium storing the program according to claim 14.

Images