Processing system, mixed reality device, processing method, program and storage medium

Abstract

To provide a processing system, a mixed reality device, a processing method, a program and a storage medium configured to encourage a worker to perform work more appropriately.SOLUTION: A processing system is used for work to turn a screw at a fastening location with a tool. The processing system comprises a display device and a processing device. The display device is configured to display a virtual first object around a region where the tool can be positioned during the work. The processing device is configured to estimate a position of the tool. The processing device is configured to issue an alert when the tool is determined to be in contact with the first object based on the estimated position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] SUMMARY OF THE PRESENT EMBODIMENTS An embodiment of the present invention relates to a processing system, a mixed reality device, a processing method, a program, and a storage medium. [Background technology]

[0002] When manufacturing an article, a screw may be fastened. Or, when maintaining, inspecting, or repairing an article, a screw may be loosened. There is a demand for a technique that can prompt a worker to perform these operations using screws more appropriately. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 176645 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a processing system, a mixed reality device, a processing method, a program, and a storage medium that can prompt a worker to perform a more appropriate task. [Means for solving the problem]

[0005] A processing system according to an embodiment is used in an operation of turning a screw into a fastening point using a tool. The processing system includes a display device and a processing device. The display device displays a virtual first object around an area where the tool may be located during the operation. The processing device estimates a position of the tool. The processing device issues an alert when it is determined that the tool has come into contact with the first object based on the estimated position. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. [Diagram 3] FIG. 3 is a schematic diagram showing the state of real space. [Figure 4] FIG. 4 is a schematic diagram showing the state during work. [Diagram 5] 5(a) and 5(b) are schematic diagrams showing examples of displays by the processing system according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram illustrating the tool. [Figure 10] FIG. 10 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 11] FIG. 11 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 12] FIG. 12 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 13] FIG. 13 is a schematic diagram illustrating the tool. [Figure 14] FIG. 14 is a table showing changes in alerts. [Figure 15] 15(a) and 15(b) are schematic diagrams showing examples of displays by the processing system according to the embodiment. [Figure 16] FIG. 16 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 17] FIG. 17 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 18]FIG. 18 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 19] 19(a) and 19(b) are schematic diagrams showing examples of displays by the processing system according to the embodiment. [Figure 20] FIG. 20 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 21] 21(a) and 21(b) are schematic diagrams showing examples of displays by the processing system according to the embodiment. [Figure 22] FIG. 22 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 23] FIG. 23 is a schematic diagram showing the state of real space. [Figure 24] FIG. 24 is a schematic diagram showing the state of the real space. [Diagram 25] FIG. 25 is a schematic diagram showing the state during work. [Figure 26] FIG. 26 is a schematic diagram showing the state during work. [Figure 27] FIG. 27 is a schematic diagram showing a display example by the processing system according to the embodiment. [Figure 28] FIG. 28 is a schematic diagram illustrating the tool. [Figure 29] Fig. 29(a) is a schematic diagram showing an example of a tool, and Figs. 29(b) to 29(d) are schematic diagrams showing an example of a marker. [Diagram 30] 30(a) to 30(h) are schematic views for explaining the processing in the processing system according to the embodiment. [Diagram 31] FIG. 31 is a schematic diagram showing the movement of a tool. [Diagram 32] FIG. 32 shows simultaneous equations that represent the relationship between the position of each marker before movement and the position of each marker after movement. [Diagram 33] FIG. 33 is a determinant that represents the relationship between the position of the tool before movement and the position of the tool after movement. [Diagram 34] FIG. 34 is a flowchart illustrating a processing method according to an embodiment. [Diagram 35] FIG. 35 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. The invention according to the embodiment is applicable to a task of turning a screw using a tool. A processing system 1 includes a processing device 10, an imaging device 20, a display device 30, an input device 40, and a storage device 50, as shown in FIG.

[0009] The image capturing device 20 captures images of the state of work. For example, in the work, a fastener such as a screw is fastened to an article using a tool. Or, a screw fastened to the article is loosened using a tool. The article is a part, a unit, or a semi-finished product for producing a product. The tool is a wrench or a screwdriver. Here, an example in which the embodiment of the present invention is applied to a fastening work of tightening a screw will be mainly described.

[0010] When assembling an article, a worker holds a tool in his / her hand and tightens a screw. The image capturing device 20 captures images of the tool, the worker's hand, etc. For example, the image capturing device 20 includes a camera that captures an RGB image and a depth image.

[0011] The processing device 10 receives successive images (video) captured by the imaging device 20. The processing device 10 detects the left or right hand in the image. Hand tracking technology is used to detect the left or right hand. Hereinafter, when there is no particular distinction between the left hand and the right hand, the term "hand" will simply refer to at least either the left hand or the right hand.

[0012] The input device 40 is used by a worker to input information to the processing device 10. The input device 40 includes a microphone. The worker can input information to the processing device 10 by speaking into the input device 40. For example, a voice corresponding to a voice command is input to the input device 40. In addition to the input device 40, the worker can input information to the processing device 10 by hand gestures or the like.

[0013] The processing device 10 displays information for assisting the fastening work on the display device 30. The display device 30 displays information for the worker. For example, the processing device 10 displays hand detection results, virtual objects indicating areas where tools should not enter, work instructions, and the like during the fastening work.

[0014] The storage device 50 stores data necessary for processing by the processing device 10, data obtained by processing by the processing device 10, etc. For example, data related to work, data necessary for estimating the position of a tool described later, etc. are registered in the storage device 50.

[0015] The processing device 10 judges whether inappropriate work is being performed during the fastening work. Specifically, the display device 30 displays a virtual first object indicating an area where the tool must not enter during the fastening work. The first object is displayed around an area where the tool may be located during the fastening work. While the first object is being displayed, the processing device 10 estimates the position of the tool. The processing device 10 judges whether the tool is in contact with the first object based on the estimated tool position. If it is judged that the tool is in contact with the first object, the processing device 10 issues an alert.

[0016] This process can issue an alert if the tool is in an inappropriate position, if a screw is fastened to the wrong fastening location, etc. The alert can notify the worker that an inappropriate operation is being performed.

[0017] 1, the processing device 10 specifically includes functions as an acquisition unit 11, a detection unit 12, a control unit 13, an estimation unit 14, and an output unit 15. The acquisition unit 11 acquires an image acquired by an imaging device 20, data input by an input device 40, and the like in real time.

[0018] The detection unit 12 detects the user's hand appearing in the image. The detection unit 12 measures the three-dimensional position of each point of the detected hand. More specifically, the hand includes multiple joints such as a DIP joint, a PIP joint, a MP joint, and a CM joint. The position of any of these joints is used as the position of the hand. The position of the center of gravity of the multiple joints may be used as the position of the hand. Or, the position of the center of the entire hand may be used as the position of the hand.

[0019] The detection unit 12 repeats hand detection for successively acquired images and performs hand tracking. The detection unit 12 also detects hand gestures from time-series changes in the detected hand positions. For example, the detection unit 12 calculates the similarity between the change in hand position and the hand movement of each predefined hand gesture. When the similarity for any hand gesture exceeds a preset threshold, the detection unit 12 determines that the user's hand is performing the movement of that hand gesture.

[0020] When the acquisition unit 11 acquires voice data, the detection unit 12 detects a voice command from the voice. For example, the detection unit 12 executes voice recognition and converts the user's utterance into a character string. The detection unit 12 determines whether the utterance includes a character string of any of the predefined voice commands. When the utterance includes a character string of any of the voice commands, the detection unit 12 determines that the user is uttering that voice command.

[0021] The detection unit 12 performs detection of hands or commands, as well as detection of markers in an image, measurement of the three-dimensional position of the marker, and detection of contact with an object in a virtual space. The control unit 13 performs various processes and controls based on the information detected by the detection unit 12. The estimation unit 14 estimates the position of the tool during the fastening operation. The output unit 15 outputs a video signal indicating the detection result by the detection unit 12, data obtained by processing by the control unit 13, etc. to the display device 30. The display device 30 displays information based on the input video signal.

[0022] Hereinafter, the processing device 10 outputs a video signal to the display device 30, and the display device 30 displays information based on the video signal, will simply be referred to as "the processing device 10 (or the display device 30) displays information."

[0023] The details of the invention according to the embodiment will be described below with reference to a specific example. Here, an example in which the processing system 1 is implemented as an MR device will be described. In the MR device, a virtual space is displayed overlapping with a real space. A user can interact with an object displayed in the virtual space.

[0024] FIG. 2 is a schematic diagram illustrating a mixed reality device according to an embodiment. The processing system 1 shown in Fig. 1 is realized as, for example, a mixed reality (MR) device. The MR device 100 shown in Fig. 2 includes a frame 101, a lens 111, a lens 112, a projection device 121, a projection device 122, an image camera 131, a depth camera 132, a sensor 140, a microphone 141, a processing device 150, a battery 160, and a storage device 170.

[0025] The processing device 150 is an example of the processing device 10. The image camera 131 and the depth camera 132 are an example of the imaging device 20. The projection device 121 and the projection device 122 are an example of the display device 30. The microphone 141 is an example of the input device 40. The storage device 170 is an example of the storage device 50.

[0026] In the illustrated example, the MR device 100 is a twin-eye head-mounted display. Two lenses, 111 and 112, are fitted in a frame 101. A projection device 121 and a projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0027] The projection device 121 and the projection device 122 display the detection result of the body of the worker, a virtual object, etc. on the lens 111 and the lens 112. Only one of the projection device 121 and the projection device 122 may be provided, and information may be displayed on only one of the lens 111 and the lens 112.

[0028] The lenses 111 and 112 are optically transparent. The worker can visually recognize the real state through the lenses 111 and 112. The worker can also visually recognize information projected onto the lenses 111 and 112 by the projection devices 121 and 122. The information is displayed superimposed on the real space by projection by the projection devices 121 and 122.

[0029] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor and can detect three-axis angular velocity and three-axis acceleration. The microphone 141 accepts voice input.

[0030] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display by the projection device 121 and the projection device 122. The processing device 150 detects a movement of the field of view based on the detection result by the sensor 140. The processing device 150 changes the display by the projection device 121 and the projection device 122 in response to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, etc.

[0031] The battery 160 supplies the power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, etc. The storage device 170 may be provided outside the MR device 100 and may communicate with the processing device 150.

[0032] The MR device according to the embodiment is not limited to the illustrated example, and may be a single-lens head-mounted display. The MR device may be a glasses type as illustrated, or a helmet type.

[0033] FIG. 3 is a schematic diagram showing the state of real space. For example, an article 200 shown in FIG. 3 exists in a real space. The article 200 is a cylindrical member. The inside of the article 200 is hollow, and fastening points 201 to 206 exist at the bottom of the hollow. In this example, a screw fastening operation is performed on the article 200. In the fastening operation, a wrench and an extension bar are used to fasten the screws into each of the fastening points 201 to 206. The MR device is used to assist this fastening operation. For example, the appropriate posture of the tool, the fastening position, the position where the hands should be placed when fastening, and the like are shown by a virtual object.

[0034] A marker 210 is provided near the article 200 that is the target of the work. In the illustrated example, the marker 210 is an AR marker. As described below, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), or the like may be used as the marker 210. Alternatively, instead of a marker, a base point may be indicated by a hand gesture. The processing device 10 sets a three-dimensional coordinate system of the virtual space based on a plurality of points indicated by a hand gesture.

[0035] FIG. 4 is a schematic diagram showing the state during work. When a screw is fastened to the article 200, the worker places the screw at any of the fastening locations. The worker fits one end of the extension bar 251 onto the screw. The worker fits a wrench 252 onto the other end of the extension bar 251. As shown in FIG. 4 , the worker holds both ends of the wrench 252 with both hands. In this state, the worker turns the wrench 252 to rotate the extension bar 251, thereby tightening the screw at the fastening location.

[0036] When performing a fastening operation, it is preferable that the tool is used in a proper position. If the tool is used in an improper position, it may damage the article. There is also a possibility that the worker may be injured. In addition, depending on the fastening operation, there are cases where the order of fastening screws into multiple fastening points is specified, or where screws are fastened only into specific fastening points. In these cases, it is required to fasten the screws into the proper points.

[0037] 5(a), 5(b) and 6 to 8 are schematic diagrams showing examples of displays by the processing system according to the embodiment. The processing device 10 displays a virtual first object 310 shown in Fig. 5(a) in order to prompt an appropriate fastening operation. The first object 310 has a cylindrical shape following the shape of the inside of the article 200. As shown in Fig. 5(b), the first object 310 is displayed overlapping with the article 200 in the real space, and is located inside the article 200. The shape and size of the first object 310 are set according to the shape and size of the article 200.

[0038] In the illustrated example, the first object 310 is provided with a plurality of holes 311-316. The holes 311-316 are formed in areas where a tool can be positioned when screws are fastened to the fastening points 201-206, respectively. The positions and number of the holes are set according to the positions and number of the fastening points provided in the article 200. The first object 310 is positioned around the holes 311-316, which are areas where a tool can be positioned.

[0039] The shape, diameter, etc. of the holes 311-316 are set according to the tool to be used. In the illustrated example, the extension bar 251 is used. Therefore, the shape of the holes 311-316 is linear, the same as the shape of the extension bar 251. The holes 311-316 extend in the vertical direction. Also, the diameter of the holes 311-316 is the diameter of the extension bar 251 plus a margin.

[0040] During fastening work, the photographing device 20 photographs the article 200, the worker's left hand, and the worker's right hand. The processing device 10 (acquisition unit 11) acquires the photographed images. The processing device 10 (detection unit 12) detects the left hand and the right hand from the photographed images. The processing device 10 (control unit 13) causes the display device 30 to display the detection results of the hands. For example, as shown in FIG. 6, the processing device 10 (output unit 15) displays the detection results of the left hand 301 and the detection results of the right hand 302 superimposed on the hands in the real space. In the example shown in FIG. 6, a plurality of objects 301a and a plurality of objects 302a are displayed as the detection results of the left hand 301 and the right hand 302. The plurality of objects 301a indicate the plurality of joints of the left hand 301, respectively. The plurality of objects 302a indicate the plurality of joints of the right hand 302, respectively. Instead of the joints, objects indicating the surface shapes of the left hand 301 and the right hand 302, respectively, may be displayed.

[0041] When the extension bar 251 and the wrench 252 are used in an appropriate posture, the extension bar 251 passes through the inside of the hole as shown in Fig. 6. In other words, the extension bar 251 does not come into contact with the first object 310.

[0042] If the extension bar 251 or the wrench 252 is used in an inappropriate position, as shown in FIG. 7, the extension bar 251 comes out of the hole and comes into contact with the first object 310. The processing device 10 issues an alert when the tool comes into contact with the first object 310. In this example, a message 261 is displayed as the alert. Instead of the message, a warning color may be displayed. Instead of the display, sound, vibration, light, or the like may be issued as the alert. Alternatively, an alert that combines two or more selected from the display, sound, vibration, and light may be used.

[0043] When a screw is to be fastened only at a specific fastening point, a hole is provided only at a position corresponding to the fastening point. For example, it is determined that a screw is to be fastened first at the fastening point 206 among the fastening points 201 to 206. In this case, as shown in FIG. 8, only a hole 316 corresponding to the fastening point 206 is provided. A first object 310 is displayed in the area of ​​the holes 311 to 315. When a screw should be fastened at the fastening point 206, if the screw is fastened by mistake at another fastening point, the extension bar 251 comes into contact with the first object 310. As a result, the processing device 10 issues an alert.

[0044] Various methods can be used to estimate the position of the tool. For example, the positions of the fastening points 201 to 206 are registered in advance. During the fastening operation, the hand of the worker is detected by the processing device 10. When the extension bar 251 is used, one end of the extension bar 251 is gripped by the hand as shown in FIG. 4. In addition, in the fastening operation, the extension bar 251 is generally used so as to be parallel to the vertical or horizontal direction. The processing device 10 (estimation unit 14) estimates that the extension bar 251 is present in the area between the position of the hand and the fastening point aligned vertically or horizontally with the hand. The processing device 10 uses the position of that area as the position of the tool.

[0045] Alternatively, a sensor may be provided on the extension bar 251 or the wrench 252, and the position of the extension bar 251 may be estimated using a detection value of the sensor. The sensor may be an inclination sensor, an acceleration sensor, a gyro sensor, or the like. The detection value of the sensor may be combined with the detection result of the hand to estimate the position of the extension bar 251. For example, when the extension bar 251 is not parallel to the vertical or horizontal direction, the position of the extension bar 251 can be estimated with higher accuracy.

[0046] Alternatively, the processing device 10 may estimate the position of the extension bar 251 or the wrench 252 by image processing. For example, an image (template image) of the tool to be used is prepared in advance. The processing device 10 performs template matching and determines whether the tool in the template image is captured in an image obtained during work. The processing device 10 uses the position where it is determined that the tool in the template image is captured as the position of the tool.

[0047] FIG. 9 is a schematic diagram illustrating the tool. As shown in Fig. 9, a plurality of markers 251m for estimating the position of the extension bar 251 may be attached to the extension bar 251. The processing device 10 detects the plurality of markers 251m from the image. The processing device 10 measures the position of each marker 251m. The positional relationship between the plurality of markers 251m and the extension bar 251 is registered in advance. Based on this positional relationship, the processing device 10 calculates the position P1 of the extension bar 251 from the position of each marker 251m.

[0048] Advantages of the embodiment will be described. When turning a screw, it is required to perform the task appropriately. That is, it is required to use the tool in an appropriate posture, fasten the screw in an appropriate place, or loosen the screw in an appropriate place. According to the invention according to the embodiment, a virtual first object 310 is displayed during the task. The first object 310 is displayed around an area where the tool may be located during the task. In other words, the first object 310 indicates an area where the tool should not be located so that the tool is used in an appropriate posture or to turn a screw in a correct place.

[0049] Furthermore, while the first object 310 is being displayed, the position of the tool is estimated. If it is determined that the tool has come into contact with the first object 310, an alert is issued. The tool coming into contact with the first object 310 means that the tool is being used in an improper position, or that a screw is being turned in the wrong place. By issuing an alert when the tool comes into contact with the first object 310, it is possible to inform the worker that an inappropriate operation is being performed.

[0050] According to the embodiment, it is possible to encourage a worker to perform a more appropriate task. For example, it is possible to reduce the possibility that an item is damaged or a worker is injured when a tool is used in an inappropriate position. Or, it is possible to prevent a screw from being turned in an incorrect position.

[0051] More preferred examples of the invention according to the embodiments will be described below.

[0052] 10 to 12 are schematic diagrams showing examples of display by the processing system according to the embodiment. Different functions may be assigned to different parts of the first object 310. For example, as shown in Fig. 10, the first object 310 includes a first region 310a, a second region 310b, and a third region 310c. The second region 310b is located between the first region 310a and the fastening point. The third region 310c is located between the second region 310b and the fastening point. In other words, the second region 310b is located closer to the fastening point than the first region 310a, and is located farther from the fastening point than the third region 310c.

[0053] The processing device 10 makes the alert when the tool contacts the first region 310a, the alert when the tool contacts the second region 310b, and the alert when the tool contacts the third region 310c different from each other. For example, the alert when the tool contacts the second region 310b is stronger than the alert when the tool contacts the first region 310a and weaker than the alert when the tool contacts the third region 310c.

[0054] As an example, as shown in FIG. 11(a) to FIG. 11(c), a message 261 when the tool comes into contact with the second region 310b is displayed larger than a message 261 when the tool comes into contact with the first region 310a and smaller than a message 261 when the tool comes into contact with the third region 310c. As another example, when the tool comes into contact with the first region 310a, only the message 261 is displayed as shown in FIG. 12(a). When the tool comes into contact with the second region 310b, a warning color 262 is displayed in addition to the message 261 as shown in FIG. 12(b). When the tool comes into contact with the third region 310c, a stronger warning color 262 is displayed in addition to the message 261 as shown in FIG. 12(c). The warning color 262 may be displayed in a part of the field of view other than the message 261.

[0055] When the tool comes into contact with the first object 310 at a position far from the fastening point, the possibility of an inappropriate fastening operation being performed is low compared to when the tool comes into contact with the first object 310 at a position close to the fastening point. In addition, at a position far from the fastening point, the worker may move the tool to re-hold the tool, align the tool, or the like. If a strong alert is issued at a position far from the fastening point, it stresses the worker. By varying the alert when the tool comes into contact with the first object 310 depending on the distance from the fastening point, a more appropriate alert can be issued.

[0056] Note that instead of one first object 310, a first object having a function as the first region 310a, a second object having a function as the second region 310b, and a third object having a function as the third region 310c may be provided. In such a case, it can be considered that the first object 310 including the first region 310a to the third region 310c is provided. Furthermore, the number of regions set in the first object 310 is arbitrary. The number of regions set may be two or more than three.

[0057] FIG. 13 is a schematic diagram illustrating the tool. Instead of setting an area on the first object 310, the alert may change depending on the position of the tool that contacts the first object 310. For example, when the position of the extension bar 251 is estimated, the processing device 10 sets a first portion 251a, a second portion 251b, and a third portion 251c for the extension bar 251 as shown in Fig. 13. When the extension bar 251 is used, the second portion 251b is located between the fastening point and the first portion 251a. The third portion 251c is located between the fastening point and the second portion 251b.

[0058] The processing device 10 makes the alert when the first part 251a contacts the first object 310, the alert when the second part 251b contacts the first object 310, and the alert when the third part 251c contacts the first object 310 different from each other. For example, the alert when the second part 251b contacts the first object 310 is stronger than the alert when the third part 251c contacts the first object 310, and weaker than the alert when the first part 251a contacts the first object 310. That is, the stronger the alert is issued, the closer the part of the tool to the handle is to contact the first object 310. By making the alert different depending on the part of the tool that contacts the first object 310, a more appropriate alert can be issued. In addition, the number of parts set in the extension bar 251 is arbitrary. The number of parts set may be two or more than three.

[0059] FIG. 14 is a table showing changes in alerts. Setting of an area to the first object 310 and setting of a part to the tool may be combined. For example, as shown in FIG. 10, a first area 310a to a third area 310c are set to the first object 310, and as shown in FIG. 13, a first part 251a to a third part 251c are set to the extension bar 251. In this case, as shown in FIG. 14, the alert can be changed according to the combination of the part of the tool that is in contact and the area of ​​the first object 310. In the example shown in FIG. 14, the numerical value written in the cell indicates the strength of the alert. That is, the stronger the alert, the closer the part of the tool to the handle is in contact with the first object 310 and the closer the tool is to the fastening point in contact with the first object 310.

[0060] Figures 15(a), 15(b), 16 to 18, 19(a), 19(b), 20, 21(a), 21(b), and 22 are schematic diagrams showing examples of display by the processing system of the embodiment. As shown in Fig. 15(a), in addition to the first object 310, a virtual second object 320 may be displayed. In this example, the second object 320 is displayed at a position away from the fastening point. As shown in Fig. 15(b), one of the holes formed in the first object 310 is located between the fastening point and the second object 320. The second object 320 indicates an area where a worker's hand or tool should be positioned during the fastening work.

[0061] In the example shown in Fig. 15(a) and Fig. 15(b), the second object 320 is spherical and is displayed so as to be distinguishable from the first object 310. As shown in the figure, in addition to the second object 320, virtual objects 330 and 331 may be displayed. The object 330 is spherical and is displayed overlapping with the fastening point. The object 331 is linear and connects the second object 320 and the object 330. In this example, the second object 320 indicates an area where the tip or hand of the wrench 252 should be located. The object 330 indicates a fastening point of the screw. The object 331 indicates an area where the extension bar should be located.

[0062] The processing device 10 may detect that a predetermined object has come into contact with the second object 320. For example, the processing device 10 detects that a hand has come into contact with the second object 320. Specifically, the processing device 10 calculates the distance between the position of the hand and the second object 320. If the distance is less than a preset threshold, the processing device 10 determines that the hand has come into contact with the virtual object. As an example, in FIG. 15(b), the diameter of the second object 320 corresponds to the threshold. The sphere indicates the range in which it is determined that the hand has come into contact with the virtual object.

[0063] Alternatively, the processing device 10 may estimate the position of the wrench 252. The processing device 10 detects that the wrench 252 has come into contact with the second object 320 by using the distance between the wrench 252 and the second object 320. As with the estimation of the position of the extension bar 251, various methods can be used to estimate the position of the wrench 252. For example, as shown in FIG. 4, both ends of the wrench 252 are gripped by the hands during fastening work, so the processing device 10 estimates the position of the wrench 252 from the positions of both hands. As with the extension bar 251 shown in FIG. 9, a plurality of markers for estimating the position of the wrench 252 may be attached to the wrench 252.

[0064] By displaying the second object 320, the worker can easily understand where to position his / her hand or tool to perform the fastening work. In particular, when screws are fastened to an article with many fastening points or an article where the fastening points are difficult to see, by displaying the second object 320, the worker can perform the fastening work more smoothly.

[0065] During the fastening operation, a hand or a tool comes into contact with a second object 320, as shown in Fig. 16. One second object 320 is displayed corresponding to one fastening point. When the hand or the tool comes into contact with the second object 320, it can be estimated that a screw is being turned into the fastening point corresponding to that second object 320. Hereinafter, a point among one or more fastening points where a screw is estimated to be turned is referred to as an "estimated point."

[0066] If the location where the screw is being turned can be estimated, it becomes possible to automatically create a work record indicating the location where the screw was turned. When the worker completes the work of turning the screw, the processing device 10 records that the screw was turned into the estimated location.

[0067] Preferably, a digital tool is used in the work. The processing device 10 receives detection values ​​from the digital tool. The processing device 10 can use the detection values ​​to determine whether screw tightening to the estimated location has been completed. When it is determined that screw tightening has been completed, the processing device 10 inputs the work result into the work record. This method makes it possible to automatically create a more accurate work record.

[0068] For example, the digital tool is a digital torque wrench or a digital torque driver, and the detection value is a torque value detected by the digital tool. The digital torque wrench or digital torque driver detects the torque value and transmits it to the processing device 10. If the torque value exceeds a predetermined threshold, the processing device 10 determines that the screw tightening is completed. The digital tool may determine whether or not a torque exceeding a predetermined threshold has been detected. In this case, the digital tool may output the determination result as the detection value instead of the torque value. The digital tool may output both the determination result and the torque value. Alternatively, the digital tool may detect the rotation angle when the screw is turned. The processing device 10 may link the received detection value to data related to the estimated location.

[0069] When the tool comes into contact with the first object 310 after a predetermined object comes into contact with the second object 320, the processing device 10 releases the link to the data related to the estimated location. When the tool comes into contact with the first object 310, the tool may be used in an incorrect position, as shown in Fig. 17, and a screw may be fastened to a location other than the estimated location. If the detection value is linked to the data related to the estimated location even though the tool has come into contact with the first object 310, an incorrect work record may be created.

[0070] 18, the processing device 10 may not display the second object 320 when a tool comes into contact with the first object 310 after a predetermined object comes into contact with the second object 320. This makes it easier for the worker to know that an inappropriate task is being performed. Thereafter, when the contact of the tool with the first object 310 is no longer detected, the processing device 10 displays the second object 320 again.

[0071] Depending on the article, the order in which the screws are fastened may be determined. The order may be indicated by a virtual object. For example, as shown in FIG. 19(a), in one state, a second object 320 is displayed at only one of the fastening points 201-206. In another state, as shown in FIG. 19(b), a second object 320 is displayed at only another one of the fastening points 201-206.

[0072] 19(a) and 19(b), an object 340 (third object) showing an instruction regarding the fastening work may be further displayed. In the illustrated example, the torque value required for tightening the screw at the fastening point is displayed in the object 340.

[0073] 20, second objects 321-326 may be displayed corresponding to fastening points 201-206, respectively, and information indicating the fastening order may be displayed on the second objects 321-326. By indicating the fastening order by the objects, the worker can perform the fastening work more smoothly. Also, the occurrence of mistakes in the fastening order can be suppressed.

[0074] Also, a screw may be tightened multiple times at one location. For example, after the screws are tightened at the fastening locations 201 to 206, each screw may be retightened. In this case, the number of times the screw is tightened at each fastening location may be indicated by an object.

[0075] In the example shown in Fig. 21(a), second objects 321 to 326 are displayed at fastening points 201 to 206, respectively. A screw is fastened only to fastening point 201, and no screw is fastened to fastening points 202 to 206. In this case, the color of second object 321 is different from the colors of second objects 322 to 326. Instead of color, the size, pattern, shape, etc. of the objects may be different.

[0076] In the example shown in Fig. 21(b), screws are fastened at fastening points 201 to 206, and only fastening point 201 is retightened. At this time, the color of second object 321 is different from the colors of second objects 322 to 326. In addition, the color of second object 321 after the screws are retightened shown in Fig. 21(b) is different from the color of second object 321 before the screws are retightened shown in Fig. 21(a).

[0077] As shown in Figures 20, 21(a), and 21(b), the display modes of the second objects 321-326 differ depending on the order in which the multiple fastening points are fastened, or whether or not each of the multiple fastening points has been fastened, allowing the worker to perform the fastening work more smoothly.

[0078] 22, the second object 320 may be displayed inside the hole of the first object 310. This display allows the worker to easily visually recognize which area inside the first object 310 the tool should be passed through. In this case, the processing device 10 detects contact between the extension bar 251 and the second object 320. When it is determined that the extension bar 251 is in contact with the second object 320, it is presumed that a screw is fastened to a location corresponding to the second object 320.

[0079] The above-mentioned virtual objects are all displayed according to the position of the object in the real space. For example, the three-dimensional coordinate system of the virtual space when the object is prepared is set to be the same as the three-dimensional coordinate system of the virtual space when the fastening work is performed. Also, the positional relationship between the base point of the three-dimensional coordinate system and the work target when the object is prepared is set to be the same as the positional relationship between the base point of the three-dimensional coordinate system and the work target when the fastening work is performed.

[0080] In order to facilitate these settings, it is preferable that a marker 210 for setting a base point is prepared in real space as shown in Fig. 3. When an object is prepared, a three-dimensional coordinate system is set with the marker 210 as the base point. When a fastening operation is performed, the processing device 10 also detects the marker 210 from the image and sets a three-dimensional coordinate system with the marker 210 as the base point. The display position of each object is determined with the marker 210 as the reference. As long as the positional relationship between the marker 210 and the article being worked on does not change, the prepared object can be displayed in accordance with the article during the fastening operation.

[0081] FIG. 23 is a schematic diagram showing the state of real space. There are cases where the fastening points may move relative to the marker 210. For example, in the example shown in FIG. 23, the article 200 and the marker 210 are placed on a workbench 215. As long as the article 200 is not moved, the positional relationship between the article 200 and the marker 210 remains unchanged. However, the upper part 200a of the article 200 can rotate relative to the lower part 200b. The fastening points 201-206 are provided on the upper part 200a. Therefore, when the upper part 200a rotates relative to the lower part 200b, the positional relationship between the fastening points 201-206 and the marker 210 changes.

[0082] In this case, as shown in Fig. 23, a marker 211 whose positional relationship with the fastening point is invariable is further provided. The processing device 10 detects the marker 211 from the image, and sets a three-dimensional coordinate system for displaying objects with the marker 211 as the base point. The display positions of the first object 310, the second object 320, and the like are determined using the three-dimensional coordinate system of the marker 211. Therefore, when the marker 211 moves, the display positions of each object also move. Even if the upper part 200a rotates with respect to the lower part 200b, or the article 200 moves with respect to the marker 210, each object can be displayed at an appropriate position regardless of the rotation or movement.

[0083] When a fastening point exists in the lower part 200b, a virtual object may be displayed at the fastening point. In this case, the position of the object displayed on the lower part 200b is set using a three-dimensional coordinate system based on the marker 210. Therefore, regardless of a change in the position of the marker 211, the object can be displayed at an appropriate position on the lower part 200b.

[0084] Fig. 24 is a schematic diagram showing a state of real space, Fig. 25 and Fig. 26 are schematic diagrams showing a state during work, and Fig. 27 is a schematic diagram showing a display example by the processing system according to the embodiment. Fig. 24 is a schematic diagram illustrating another work target. A fastening operation may be performed on an article 220 shown in Fig. 24. The article 220 is cylindrical, and multiple fastening points 221 to 228 are present along the outer periphery of the article 220.

[0085] For example, a worker fastens a screw into fastening point 223. In this case, the worker places screw 230 in a screw hole of fastening point 223 as shown in FIG. 25. Next, the worker holds the grip of wrench 400 with his right hand. The worker fits the tip (head) of wrench 400 to which a socket is attached into screw 230. Thereafter, the worker rotates wrench 400 with right hand 302 while holding the head of wrench 400 with left hand 301 as shown in FIG. 26. This causes screw 230 to be fastened into fastening point 223.

[0086] During the work shown in Figs. 25 and 26, for example, as shown in Fig. 27, a first object 310 and a second object 320 are displayed. The first object 310 is displayed around an area where the wrench 400 may be located when the screw is fastened to the fastening point 223. The second object 320 is displayed in a position close to the fastening point 223. When the processing device 10 detects that the hand has come into contact with the second object 320, it estimates that the screw is being fastened to the fastening point 223. During the fastening work, the processing device 10 estimates the position of the wrench 400. When the wrench 400 comes into contact with the first object 310, the processing device 10 issues an alert. When the hand comes into contact with the first object 310, the processing device 10 does not issue an alert.

[0087] 24 to 27, a plurality of regions for differentiating alerts may be set for the first object 310, similar to the example shown in Fig. 10. A plurality of portions for differentiating alerts may be set for the wrench 400, similar to the example shown in Fig. 13.

[0088] As shown in Figures 15(a), 15(b), 22, and 27, the shape, position, size, etc. of the first object 310 and the second object 320 can be changed as appropriate depending on the tool used, the position of the fastening point, etc.

[0089] FIG. 28 is a schematic diagram illustrating the tool. A preferred example of the method for estimating the position of a tool will be described. For example, as shown in Fig. 28, a wrench 400 is attached with a plurality of markers 430. In the illustrated example, a jig 420 is attached to the wrench 400, and a plurality of markers 430 are attached to the jig 420. The jig 420 has a plurality of planes. Three markers 430 are attached to each plane.

[0090] For three markers 430 on one plane, the distance between one marker 430 and another marker 430 is different from the distance between the one marker 430 and yet another marker 430. In other words, the three markers 430 are attached so that when a virtual triangle connecting the three markers 430 is generated, the triangle is not an equilateral triangle.

[0091] Preferably, the multiple markers 430 are arranged so that one side of the triangle is parallel to the extension direction of the tool. In the example shown in FIG. 28, the wrench 400 extends in a first direction D1. The first direction D1 corresponds to a direction connecting the grip 411 of the wrench 400 and the head 412 of the wrench 400. A virtual triangle 441 based on the markers 430a to 430c has sides parallel to the first direction D1. In addition, here, two directions that are perpendicular to the first direction D1 and perpendicular to each other are defined as a second direction D2 and a third direction D3.

[0092] The processing device 10 detects multiple markers 430 from the image captured by the imaging device 20. The processing device 10 calculates the position of the wrench 400 from the positions of at least three markers 430. When a proper fastening operation is performed, the position of the part that overlaps or is close to the object is used as the position of the wrench 400. For example, the processing device 10 calculates the position of the head 412 from the positions of the at least three markers 430. The processing device 10 issues an alert when the head 412 comes into contact with the first object 310.

[0093] Fig. 29(a) is a schematic diagram showing an example of a tool, and Figs. 29(b) to 29(d) are schematic diagrams showing an example of a marker. The calculation method of the tool position will be described in more detail. The ID of the marker 430, the position of the marker 430, the tool position calculated by the marker 430, and the like are registered in advance before the fastening operation. Fig. 29(a) to Fig. 29(d) show the state of the jig 420 when viewed from different directions. The external shape of the jig 420 is a rectangular parallelepiped, and the jig 420 has four planes 421 to 424. Markers 430a to 430c are attached to the plane 421. Markers 430e to 430g are attached to the plane 422. Markers 430i to 430k are attached to the plane 423. Markers 430m to 430o are attached to the plane 424.

[0094] Markers 430a-430c are attached such that virtual triangle 441 obtained by connecting these markers is an isosceles triangle. Similarly, markers 430e-430g, markers 430i-430k, and markers 430m-430o are also attached such that virtual triangles 442-444 obtained by connecting a plurality of markers are isosceles triangles. Each isosceles triangle has sides parallel to the first direction D1. Also, each marker is provided at a position where triangles 441-444 are rotationally symmetrical with respect to the center of the first surface of wrench 400. The first surface is a surface perpendicular to the first direction D1 in which wrench 400 extends.

[0095] In advance preparation, the ID of each marker 430 and the position of each marker 430 in an arbitrary spatial coordinate system are registered. In addition, a part of the position of the wrench 400 is registered for each combination of markers 430a to 430c, markers 430e to 430g, markers 430i to 430k, and markers 430m to 430o. A different position may be registered for each combination of markers, or a common position may be registered for each combination. As an example, a position p0 shown in FIG. 29(a) is registered for each combination of markers. The registered position is the position where the operator holds the tool when the tool is used in a fastening operation.

[0096] In addition, a position-related attribute is further registered for each marker 430. The attribute indicates where the marker 430 exists in the areas 451 to 453. The areas 451 to 453 are aligned in a first direction D1. As will be described later, this attribute is used to improve the estimation accuracy of the position of the tool during work.

[0097] The ID of the marker 430, the position of the marker 430, the position of the tool corresponding to the marker, and the position attributes are stored in association with the ID of the tool. This completes the advance preparation for the marker. Hereinafter, the marker position and the tool position registered in the advance preparation are referred to as the advance marker position and the advance tool position, respectively.

[0098] The processing device 10 detects the markers 430 appearing in the image. When four or more markers 430 are detected, the processing device 10 extracts three markers 430. The processing device 10 calculates the position of the tool from the positions of the three extracted markers 430.

[0099] 30(a) to 30(h) are schematic views for explaining the processing in the processing system according to the embodiment. For example, the image IMG shown in Fig. 30(a) is obtained by the photographing device 20. Markers 430a to 430c and markers 430e to 430g are captured in the image IMG. The processing device 10 extracts combinations of three markers 430 from the markers 430a to 430c and the markers 430e to 430g. The processing device 10 generates a triangle connecting the three markers 430 for each combination. Figs. 30(b) to 30(h) show examples of the generated triangles.

[0100] Next, the processing device 10 refers to data of each detected marker 430. As described above, position attributes are registered for each marker 430. The processing device 10 extracts triangles in which the marker 430 exists in each of the regions 451 to 453 from the generated triangles. That is, the processing device 10 extracts triangles having sides along the first direction D1. As a result of the processing, the triangles shown in Figs. 30(b) to 30(e) are extracted, and the triangles shown in Figs. 30(f) to 30(h) are not extracted.

[0101] The processing device 10 calculates a first vector parallel to the normal of the extracted triangle. The processing device 10 also calculates a second vector connecting the observation point (the image capture device 20) and the center point of the triangle. The processing device 10 calculates the angle between the first vector and the second vector. The processing device 10 calculates the angle between the first vector and the second vector for each of the extracted multiple triangles.

[0102] The processing device 10 extracts one triangle that has the smallest angle from the extracted triangles. The processing device 10 determines the marker 430 corresponding to the extracted triangle as the marker 430 to be used in calculating the tool position. In the illustrated example, the markers 430a to 430c are determined as the markers to be used in calculating the tool position. Hereinafter, the three determined markers are also referred to as the first marker, the second marker, and the third marker.

[0103] The processing device 10 measures the position of each of the first to third markers. The processing device 10 also refers to the IDs of each of the first to third markers. The processing device 10 acquires the prior positions of the first to third markers and the prior position of the tool corresponding to the first to third markers. The processing device 10 calculates the position of the tool during work using the positions of the first to third markers obtained during work, the prior positions of the first to third markers, and the prior position of the tool.

[0104] FIG. 31 is a schematic diagram showing the movement of a tool. Even if a tool is used in work, the relative position of the tool to the positions of the first to third markers does not change. Here, as shown in FIG. 31, the prior position of the first marker is (x1, y1, z1). The prior position of the second marker is (x2, y2, z2). The prior position of the third marker is (x3, y3, z3). The prior position of the tool is (x0, y0, z0). In addition, the positions of the first to third markers measured during work are (x1', y1', z1'), (x2', y2', z2'), and (x3', y3', z3'), respectively. The position of the tool during work is (x0', y0', z0').

[0105] FIG. 32 shows simultaneous equations that represent the relationship between the position of each marker before movement and the position of each marker after movement. The relationship between the positions of the first to third markers beforehand and the positions of the first to third markers during work is expressed by the equation in FIG. 32. Each position is substituted into the relational equation shown in FIG. 32 to solve the simultaneous equations. At this time, coefficients b1 to b3 are set to zero. Coefficients b1 to b3 correspond to the amount of translation from the positions of the first to third markers beforehand to the positions of the first to third markers during work. Coefficients a11 to a33 are calculated by solving the simultaneous equations. Coefficients a11 to a33 indicate the amount of rotation and the amount of deformation from the positions of the first to third markers beforehand to the positions of the first to third markers during work.

[0106] Then, the difference between the previous position of the midpoint of the first to third markers and the position of the midpoint of the first to third markers during the work is calculated. The difference in the first direction D1, the difference in the second direction D2, and the difference in the third direction D3 are used as coefficients b1 to b3, respectively. Note that the spatial coordinate system used when registering the positions in advance may be different from the spatial coordinate system used during the work. In that case, the change in the base point of those spatial coordinate systems is also expressed as the rotation, deformation, and translation of each marker.

[0107] FIG. 33 is a determinant that represents the relationship between the position of the tool before movement and the position of the tool after movement. The processing device 10 uses the coefficients a11 to a33 and the coefficients b1 to b3 as variables of an affine transformation matrix. As shown in Fig. 33, the position (x0', y0', z0') of the tool during work is calculated by multiplying the previous position (x0, y0, z0) of the tool by the affine transformation matrix.

[0108] The rotation and translation may be calculated by a method other than the affine transformation matrix. For example, the processing device 10 calculates the difference between the center points of the first to third markers in advance and the center points of the first to third markers during work as the translation distance. The processing device 10 also calculates normal vectors of the first to third markers in advance. Here, the normal vector is called the previous normal vector. The processing device 10 calculates normal vectors of the first to third markers during work. Here, the normal vector is called the current normal vector. The processing device 10 calculates the direction of rotation and the rotation angle for matching the previous vector with the current vector. In this way, the translation and rotation of the first to third markers are calculated. The processing device 10 calculates the position of the tool during work by adding the calculated translation and rotation to the position of the tool registered in advance.

[0109] FIG. 34 is a flowchart illustrating a processing method according to an embodiment. In the processing method according to the embodiment, master data 51 is referenced and history data 52 is generated. The master data 51 and history data 52 are stored in a storage device 50. The master data 51 includes an operation master 51a, an object master 51b, and a tool master 51c. The master data 51 is prepared in advance before screw tightening.

[0110] First, the processing device 10 accepts the selection of a work step (step S1). For example, when manufacturing an article, multiple processes are performed. One process consists of one or more work steps. In each work step, fastening work may be performed. The work step is selected by the worker. The work to be performed may be instructed by a higher-level system, and the processing device 10 may accept the selection by the instruction. Based on data obtained from the imaging device 20 or another sensor, the processing device 10 may determine the work to be performed, and the processing device 10 may accept the selection by the determination. When a work step is selected, the processing device 10 refers to the work master 51a.

[0111] The task master 51a mainly stores data related to task steps, data related to fastening tasks, and data related to objects. As data related to task steps, the ID of each task step, the name of each task step, the ID of the target to be worked on in each task step, the name of the target, and the method of identifying the base point are registered. As data related to fastening tasks, the ID and position of each fastening point, the ID of the tool used, the type and angle of the tool, the number of times each fastening point is fastened, the torque value required for each fastening point, and the color of the mark are registered. The tool type indicates a classification of the tool according to structure, external shape, or performance. The angle indicates the angle of the tool when tightening the screw at each fastening point. The mark is a mark attached to the screw when fastening is completed. As data related to objects, the ID of the object displayed in the task step and the display mode of the object are registered.

[0112] In the object master 51b, the ID of each object, the ID of a sub-object (area) set in each object, the alert strength for each sub-object, and the 3D model (shape, size) of each sub-object are registered.

[0113] The ID of each tool, the part set for each tool, the alert for each part, and the 3D model (shape, size) of the tool are registered in the tool master 51c. When the position of the tool is estimated, the 3D model of the tool registered at the estimated position is virtually generated, and contact between the tool and the first object is determined.

[0114] In the task master 51a, data related to fastening tasks and data related to objects are associated with each task step. When a task step is selected, the processing device 10 acquires data for the selected task step, such as the ID and position of the fastening point, the object to be displayed, and the like.

[0115] Next, the processing device 10 specifies a base point in the work step (step S2). A three-dimensional coordinate system is set according to the specified base point. The processing device 10 displays each object including the first object and the second object in the set three-dimensional coordinate system (step S3). The positions at which the first object and the second object are displayed are determined based on the position of the fastening point, the type of tool, the angle of the tool, and the like. For example, an area in which the tool may be located during fastening work is calculated from the position of the fastening point, the type of tool, and the angle of the tool. The first object is displayed around that area. In addition, the second object is displayed at a position away from the fastening point by a length according to the type of tool at the set angle of the tool. When a position away from the end of the tool is gripped, the second object may be displayed at a position away by a predetermined percentage of the length of the tool at the set angle with respect to the fastening position.

[0116] The processing device 10 repeatedly judges whether a predetermined object has come into contact with the second object (step S4). When an object comes into contact with the second object, it is estimated that a screw has been fastened to a fastening point corresponding to the second object. When a digital tool is used, a detection value by the digital tool is linked to data on the estimated point.

[0117] When it is determined that an object has come into contact with the second object, the processing device 10 determines whether a tool has come into contact with the first object (step S5). When it is determined that a tool has come into contact with the first object, the processing device 10 issues an alert (step S6). At this time, the second object may be hidden. When the detection value is linked to the data of the estimated location, the link is released.

[0118] If it is determined that the tool is not in contact with the first object, the processing device 10 executes a return process (step S7). In the return process, if an alert has been issued, the processing device 10 stops the alert. If the second object is not displayed, the processing device 10 displays the second object again. If the association between the data of the estimated location and the detected value has been released, the data is re-associated.

[0119] The processing device 10 judges whether the fastening is completed (step S8). For example, when the torque value detected by the digital tool exceeds a preset value, the fastening is judged to be completed. Data indicating that the fastening is completed may be input by the worker. Until the fastening is completed, the judgment of whether the tool is in contact with the first object is repeated.

[0120] When it is determined that the fastening is completed, the processing device 10 links the detected value to data related to the estimated location, and records the data in the history data 52 (step S9). In the illustrated example, the fastening location ID is recorded as data related to the fastening location. In addition, the work step ID, tool ID, tool type, and object ID displayed in the work step are stored in association with the fastening location ID.

[0121] A mark indicating the completion of the work may be attached to the fastened screw. When the screw tightening is completed, the worker puts a mark on the screw or near it. The screw tightening tool may automatically put the mark on completion. The processing device 10 may detect the mark from the image. When the processing device 10 determines that the screw tightening is completed, it refers to the color of the mark used in the screw tightening. The processing device 10 determines whether the number of pixels of the mark color exceeds a preset threshold in the image obtained by the photographing device 20. If the number of pixels exceeds the threshold, the processing device 10 determines that a mark has been attached to the screw. In the illustrated example, the fastening location ID is further linked to a detection result indicating that a mark has been detected.

[0122] The processing device 10 judges whether the work step selected in step S1 is to be continued (step S10). If the work step is to be continued, the display of the object by step S3 continues. If the work step is completed, the processing device 10 judges whether all work is completed (step S11). If all work is not completed, step S1 is executed again and the next work step is selected.

[0123] In the above-described examples, the examples in which a screw is tightened at a fastening point have been mainly described. The embodiments of the present invention are applicable not only to the case in which a screw is tightened at a fastening point, but also to the case in which a screw at a fastening point is loosened. For example, a screw at a fastening point is loosened when performing maintenance, inspection, or repair of a product. According to the embodiments of the present invention, when loosening a screw, it is possible to notify a worker that a tool is being used in an inappropriate position or that a screw at an incorrect position has been loosened. This makes it possible to encourage the worker to perform a more appropriate task. For example, it is possible to reduce the possibility that an article is damaged or a worker is injured due to the use of a tool in an inappropriate position. Or, it is possible to suppress loosening of a screw at an incorrect fastening point.

[0124] FIG. 35 is a schematic diagram showing a hardware configuration. 35 is used as the processing device 10 or the processing device 150. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0125] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0126] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0127] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs. The storage device 94 includes a solid state drive (SSD) and the like. The storage device 94 may be used as the storage device 50 or the storage device 170.

[0128] An input interface (I / F) 95 can connect the computer 90 to the input device 40. The CPU 91 can read various data from the input device 40 via the input I / F 95.

[0129] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can transmit data to the display device 30 via the output I / F 96, and cause the display device 30 to display information.

[0130] The communication interface (I / F) 97 can connect the computer 90 to devices external to the computer 90. The communication I / F 97 connects, for example, a digital tool and the computer 90 via Bluetooth (registered trademark) communication.

[0131] The data processing by the processing device 10 or the processing device 150 may be executed by only one computer 90. A part of the data processing may be executed by a server or the like via the communication I / F 97.

[0132] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0133] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0134] The processing system 1 may be implemented by something other than an MR device. For example, the processing system 1 is implemented by using a general-purpose PC. In that case, a monitor is used as the display device 30. A keyboard, a microphone, a touch pad, a microphone, or the like is used as the input device 40. The imaging device 20 images a user at a position away from the user. The user inputs commands to the processing device 10 using the input device 40 while checking the display device 30.

[0135] Embodiments of the invention include the following features. (Feature 1) A processing system used in an operation of turning a screw into a fastening point using a tool, a display device that displays a virtual first object around an area where the tool may be positioned during the work; a processing device that estimates a position of the tool and issues an alert when it is determined that the tool has come into contact with the first object based on the estimated position; A processing system comprising: (Feature 2) Further comprising an imaging device for imaging the tool, The processing system according to feature 1, wherein when a marker present in real space is photographed by the photographing device, the processing device determines the display position of the first object based on the marker. (Feature 3) The marker is provided on an article including the fastening point, The processing system of feature 2, wherein the processing device changes the display position of the first object in response to movement of the item and the marker. (Feature 4) the display device displays a virtual second object at a position different from that of the first object; The processing system described in any one of features 1 to 3, wherein the processing device estimates that a screw is fastened to the fastening location corresponding to the second object when a specified object is in contact with the second object and the tool is not in contact with the first object. (Feature 5) The processing system of feature 4, wherein the processing device receives a detection value detected by the tool from the tool and associates the detection value with data regarding the corresponding fastening point. (Feature 6) When the tool comes into contact with the first object, the display device does not display the second object; The processing system according to feature 4 or 5, wherein the display device then displays the second object when the tool is no longer in contact with the first object. (Feature 7) the display device displays a plurality of the second objects respectively corresponding to the plurality of fastening points; The processing system according to any one of features 4 to 6, wherein the display manner of the second objects differs from one another depending on the order of fastening of the multiple fastening points, or whether or not each of the multiple fastening points has been fastened. (Feature 8) The first object includes a first region and a second region located between the first region and the fastening point, The processing system according to any one of features 1 to 7, wherein the processing device differentiates the alert when the tool comes into contact with the first area from the alert when the tool comes into contact with the second area. (Feature 9) The processing system according to any one of features 1 to 8, wherein the display device further displays a virtual third object indicating an instruction related to the work. (Feature 10) The processing device reads master data in which the position of the fastening point, the number of times the screw is fastened to the fastening point, the tools used in the work, and the objects to be displayed are registered, The processing system according to any one of features 1 to 9, wherein the first object is displayed based on the master data. (Feature 11) The processing system described in any one of features 1 to 10, wherein the processing device estimates the position of the tool from a marker attached to the tool, a sensor attached to the tool, or image processing of an image of the tool. (Feature 12) The process of turning the screw into the fastening point using a tool was filmed, Displaying a virtual first object around an area where the tool may be positioned during the work; A mixed reality device that issues an alert if the tool is in contact with the first object. (Feature 13) The computer Obtaining images of the process of turning a screw into a fastening point using a tool, causing a display device to display a virtual first object around an area in which the tool may be positioned during the work; estimating a position of the tool from the image, and issuing an alert when it is determined that the tool has come into contact with the first object based on the estimated position; Processing methods. (Feature 14) A program for causing the computer to execute the processing method according to feature 13. (Feature 15) A storage medium storing the program according to feature 14.

[0136] According to the embodiments described above, a processing system, a mixed reality device, a processing method, a program, and a storage medium are provided that can prompt a worker to perform a more appropriate task.

[0137] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0138] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0139] 1: Processing system, 10: Processing device, 11: Acquisition unit, 12: Detection unit, 13: Control unit, 14: Estimation unit, 15: Output unit, 20: Shooting device, 30: Display device, 40: Input device, 50: Storage device, 51: Master data, 51a: Work master, 51b: Object master, 51c: Tool master, 52: History data, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 96: Output interface, 97: Communication interface, 98: System bus, 100: Mixed reality device, 101: Frame, 111, 112: Lens, 121, 122: Projection device, 131: Image camera, 132: Depth camera, 140: Sensor, 141: Microphone, 150: Processing device, 160: battery, 170: storage device, 200: item, 200a: upper part, 200b: lower part, 201-206: fastening points, 210, 211: marker, 215: workbench, 220: item, 221-228: fastening points, 230: screw, 251: extension bar, 251a: first part, 251b: second part, 251c: third part, 251m: marker, 252: wrench, 261: message, 262: warning color, 301: left hand, 301a: object, 302: right hand, 302a: object, 310: first object, 310a: first area, 310b: second area, 310c: third area, 311 to 316: holes, 320 to 326: second object, 330, 331, 340: object, 400: wrench, 411: grip, 412: head, 420: jig, 421 to 424: plane, 430, 430a to 430o: marker, 441 to 444: triangle, 451 to 453: area, IMG: image

Claims

1. A processing system used in an operation of turning a screw into a fastening point using a tool, a display device that displays a virtual first object around an area where the tool may be positioned during the work; a processing device that estimates a position of the tool and issues an alert when it is determined that the tool has come into contact with the first object based on the estimated position; A processing system comprising:

2. Further comprising an imaging device for imaging the tool, The processing system according to claim 1 , wherein, when a marker present in real space is photographed by the photographing device, the processing device determines a display position of the first object based on the marker.

3. The marker is provided on an article including the fastening point, The processing system according to claim 2 , wherein the processing device changes the display position of the first object in response to movement of the article and the marker.

4. the display device displays a virtual second object at a position different from that of the first object; A processing system as described in any one of claims 1 to 3, wherein the processing device estimates that a screw is fastened to the fastening point corresponding to the second object when a specified object is in contact with the second object and the tool is not in contact with the first object.

5. The processing system according to claim 4 , wherein the processing device receives a detection value detected by the tool from the tool, and associates the detection value with data related to the corresponding fastening point.

6. When the tool comes into contact with the first object, the display device does not display the second object; The processing system according to claim 4 , wherein the display device subsequently displays the second object when the tool is no longer in contact with the first object.

7. the display device displays a plurality of the second objects respectively corresponding to the plurality of fastening points; The processing system according to claim 4 , wherein display modes of the second objects differ from one another depending on an order of fastening of the plurality of fastening points or whether or not each of the plurality of fastening points has been fastened.

8. the first object includes a first region and a second region located between the first region and the fastening point; The processing system of claim 1 , wherein the processing device differentiates the alert when the tool contacts the first region from the alert when the tool contacts the second region.

9. 4. The processing system according to claim 1, wherein the display device further displays a virtual third object indicating an instruction regarding the work.

10. The processing device reads master data in which the position of the fastening point, the number of times the screw is fastened to the fastening point, the tools used in the work, and the objects to be displayed are registered, The processing system according to any one of claims 1 to 3, wherein the first object is displayed based on the master data.

11. The processing system according to any one of claims 1 to 3, wherein the processing device estimates the position of the tool from a marker attached to the tool, a sensor attached to the tool, or image processing of an image of the tool.

12. The process of turning the screw into the fastening point using a tool was filmed, Displaying a virtual first object around an area where the tool may be positioned during the work; A mixed reality device that issues an alert if the tool is in contact with the first object.

13. The computer Obtaining images of the process of turning a screw into a fastening point using a tool, causing a display device to display a virtual first object around an area in which the tool may be positioned during the work; a position of the tool is estimated from the image, and an alert is issued when it is determined that the tool has come into contact with the first object based on the estimated position. Processing methods.

14. A program causing a computer to execute the processing method according to claim 13.

15. A storage medium storing the program according to claim 14.

Images