Method of mechanical fastening work based on augmented reality

Abstract

In the mechanical fastening work method of the present application, in an augmented reality space, a mechanical fastening portion and a mechanical fastening portion virtual body correspond one-to-one, a tool and a tool virtual body correspond one-to-one, an augmented reality constituting system that generates the augmented reality space has a camera that photographs a real space, a server that analyzes an image photographed by the camera and generates the augmented reality space, and a tool that acquires torque information observed when the mechanical fastening portion is tightened or loosened and receives and transmits the torque information to the server, and the mechanical fastening work method includes the steps of: when the tool detects torque information of a mechanical fastening portion that is tightened or loosened, determining a mechanical fastening portion virtual body that corresponds one-to-one to the mechanical fastening portion, and visualizing the determined mechanical fastening portion virtual body in a virtual space.

Description

TECHNICAL FIELD

[0001] The present application relates to a method of mechanical fastening work based on augmented reality. BACKGROUND

[0002] Augmented Reality is a technology of augmenting a space perceived by a person by overlapping a virtual space constructed by 3D-CAD data, CG (Computer Graphics) and the like technology by a server (computer) and the like in a real space perceived by a person, a camera and the like.

[0003] In the past, a work management system has been known, which ensures traceability by associating information on fastening of a bolt, a flange and the like, which is acquired by a tool and a measurer, and ID information of a tool and a measurer data, a worker and the like used for work, thereby realizing high-quality work management (for example, refer to Patent Literature 1).

[0004] Note that traceability (or also referred to as "trackability") means that a flow route of an article can be tracked from a production stage to a final consumption stage or a disposal stage.

[0005] PRIOR ART DOCUMENT

[0006] PATENT LITERATURE

[0007] Patent Literature 1: Japanese Patent No. 5065851 SUMMARY

[0008] PROBLEMS TO BE SOLVED BY THE INVENTION

[0009] The work management system described in Patent Literature 1 is a system in which, for each bolt fastening work data related to flange fastening, traceability of a fastening tool, a measurer, a bolt used and the like set and measured thereby can be ensured, and input error of a target value of a set fastening torque or a target value of a set fastening allowance, a recorded error of a measured fastening torque value, a measured fastening allowance caused by human error can be prevented.

[0010] However, the work management system described in Patent Literature 1 requires an RFIC tag in all of the bolts related to flange fastening. Moreover, in order to receive and transmit information of the RFIC tag provided in these bolts, a fastening tool having an antenna extending to a head portion holding these bolts is required, and thus the cost of use thereof can be large.

[0011] To achieve the above object, the present application provides a mechanical fastening work method which is less costly and of high quality, does not require an identification mark such as an RFIC tag to be attached to a fastening portion, does not require data to be transmitted and received between the fastening portion and a fastening tool, and promotes handover of work to another worker or the like by recording the content of mechanical fastening work performed by a certain worker using augmented reality to ensure traceability.

[0012] Means for solving the problem

[0013] To solve the above problem, one of the mechanical fastening work methods of the present application is a mechanical fastening work method performed by a first worker using an augmented reality space generated by superimposing a virtual space on a real space, wherein the real space includes a mechanical fastening portion and a tool fitted on the mechanical fastening portion, the virtual space includes a mechanical fastening portion virtual body simulating the mechanical fastening portion and a tool virtual body simulating the tool, in the augmented reality space, the mechanical fastening portion and the mechanical fastening portion virtual body correspond one-to-one, and the tool and the tool virtual body correspond one-to-one, an augmented reality system that generates the augmented reality space has a camera that photographs the real space, a server that is connected to the camera, analyzes an image photographed by the camera, and generates an augmented reality space, and the tool that acquires tightness information observed when the tightness of the mechanical fastening portion is observed, and transmits and receives the tightness information with the server, the mechanical fastening work method includes the steps of determining the mechanical fastening portion virtual body corresponding one-to-one to the mechanical fastening portion when the tool detects the tightness information of the tightness of the mechanical fastening portion, and visualizing the determined mechanical fastening portion virtual body in the virtual space.

[0014] Effects of the Invention

[0015] According to the present application, a mechanical fastening work method which is less costly and of high quality, does not require an identification mark such as an RFIC tag to be attached to a fastening portion, does not require data to be transmitted and received between the fastening portion and a fastening tool, and promotes handover of work to another worker or the like by recording the content of mechanical fastening work performed by a certain worker using augmented reality to ensure traceability can be provided.

[0016] The above problems, structures, and effects other than the above will become clear from the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a schematic view of a system configuration related to a mechanical fastening work method based on augmented reality according to an embodiment of the present application.

[0018] Figure 2is an assembly drawing showing an assembly work in an augmented reality-based mechanical fastening work method according to an embodiment of the present application.

[0019] Figure 3 is a drawing showing an example of a work instruction indicating a work of mounting a component to a base plate according to an embodiment of the present application.

[0020] Figure 4 is a schematic view showing a representative point representing a position of a bolt used in a mechanical fastening work according to an embodiment of the present application.

[0021] Figure 5 is a schematic view showing a relationship from a representative point of a tool to a representative point of a bolt when the bolt is fastened with the tool having a sleeve according to an embodiment of the present application.

[0022] Figure 6 is a schematic view showing a progress of a fastening torque observed when a bolt is fastened with a tool according to an embodiment of the present application.

[0023] Figure 7 is a drawing showing a flowchart (part 1) for explaining an augmented reality-based mechanical fastening work method according to an embodiment of the present application.

[0024] Figure 8 is a drawing showing a flowchart (part 2) for explaining an augmented reality-based mechanical fastening work method according to an embodiment of the present application.

[0025] Figure 9 is a drawing showing a flowchart (part 3) for explaining an augmented reality-based mechanical fastening work method according to an embodiment of the present application.

[0026] Figure 10 is a drawing showing a flowchart (part 4) for explaining an augmented reality-based mechanical fastening work method according to an embodiment of the present application.

[0027] Figure 11 is a drawing showing a flowchart (part 5) for explaining an augmented reality-based mechanical fastening work method according to an embodiment of the present application.

[0028] Figure 12 is a drawing showing a flowchart (part 6) for explaining an augmented reality-based mechanical fastening work method according to an embodiment of the present application.

[0029] Figure 13 is a schematic view showing a case where a first worker temporarily fastens a bolt to be inserted into a prescribed position according to an embodiment of the present application.

[0030] Figure 14 is a data sheet that records the bolt fastening work of the first worker, and is a data sheet before the bolt fastening work.

[0031] Figure 15 is a schematic view showing a case where a tool virtual body of a simulation tool is visualized in a virtual space, which is an embodiment of the present application.

[0032] Figure 16 is a schematic view showing a case where a bolt virtual body of a simulation bolt is visualized in a virtual space, which is an embodiment of the present application.

[0033] Figure 17 is a data sheet that records the bolt fastening work of the first worker, and is a data sheet after the bolt fastening work.

[0034] Figure 18 is an image displayed in an augmented reality space after the first worker fastens all the bolts, which is an embodiment of the present application, and is a schematic view of an image in which the fastened bolts and the fastening work completion information displayed in the vicinity of the fastened bolts are captured.

[0035] Figure 19 is a work instruction sheet (work manual) that instructs the fastening work to a second worker, which is made based on the fastening work of the first worker, which is an embodiment of the present application.

[0036] Figure 20 is an image of an augmented reality space displayed on a see-through screen of the second worker, which is an embodiment of the present application, and is a schematic view showing the position of a bolt to be fastened next by the second worker.

[0037] Figure 21 is a schematic view showing a case where the second worker temporarily fastens a bolt to be inserted into a prescribed position based on an image of an augmented reality space displayed on a see-through screen of the second worker, which is an embodiment of the present application. Figure 5

[0038] Figure 22 is a schematic view showing a step of determining a bolt that has been fastened with a tool in a virtual space when the second worker fastens the bolt with the tool, which is an embodiment of the present application.

[0039] Figure 23 is a data sheet that records the bolt fastening work of the second worker, and is a data sheet after the bolt fastening work.

[0040] Figure 24 ​is an image displayed in the augmented reality space after the second worker fastens all the bolts, and is a schematic view of an image capturing the fastened bolts and the fastening work completion information displayed in the vicinity of the fastened bolts.

[0041] Figure 25 is a block diagram of a computer system configured to function in the augmented reality-based mechanical fastening work method according to the embodiment of the present application. DETAILED DESCRIPTION

[0042] Hereinafter, an embodiment of the present application will be described with reference to the accompanying drawings. Note that the present application is not limited to this embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.

[0043] As described above, the present application relates to a method of recording a work content of a worker fastening or loosening a bolt or the like related to mechanical fastening, or giving a work instruction to enable another worker to work on a content worked on by a certain worker, using augmented reality, and easily confirming and saving a situation where the work instructioned work content is reliably implemented.

[0044] The real space in the present specification refers to a space in which an image obtained by a worker's own vision or an image captured by the camera 10 is placed. In addition, the virtual space in the present specification refers to a space in a server (computer) constructed by digital information such as 3D-CAD, CG, and the like.

[0045] In addition, the mechanical fastening in the present specification includes hexagonal bolt-based fastening, inner hexagonal bolt-based fastening, rivet-based fastening, and the like. The hexagonal bolt-based fastening is exemplified to describe a mode of implementing the present application.

[0046] <SYSTEM STRUCTURE>

[0047] First, with reference to Figure 1 The system structure related to the augmented reality-based mechanical fastening work method according to the embodiment of the present application will be described.

[0048] Figure 1 is a schematic view of the structure of the system 100 related to the augmented reality-based mechanical fastening work method. The system 100 is configured by the wearable device 7 worn on the body of a worker, the tool RTn having a communication function capable of receiving and transmitting torque information and the like, the network server 26 storing a design drawing, a work instruction, and the like, and the network 5 connected to the aforementioned various devices and the like.

[0049] [WEARABLE DEVICE 7]

[0050] The wearable device 7 is a device that the operator wears on the body. The wearable device 7 is called a head gear or the like in the case of being a device that the operator wears on the head, for example. The wearable device 7 is constituted of a camera 10 that takes an image in the visual angle of the operator, an augmented reality server 12 that performs image processing on the image or the like taken by the camera 10 and has a function of communicating with the network 5, and a see-through screen (see-through screen) 14 that forms the visual field of the operator.

[0051] The see-through screen 14 provided to the wearable device 7 has transparency so that the operator can secure the visual field. The camera 10 can take an image almost the same as the image visually captured by the operator through the see-through screen 14. The augmented reality server 12 can project the image of the augmented reality space obtained by superimposing the image of the virtual space created by the augmented reality server 12 on the image of the real space taken by the camera 10 to the see-through screen 14.

[0052] The augmented reality server 12 provided to the wearable device 7 is a computer having a storage area that stores application software for generating augmented reality, various data, a processing section that processes a request matter indicated by the operator via the wearable device 7, and a communication section that receives and transmits data with the network 5.

[0053] The augmented reality server 12 has a function of performing image analysis on the real space (entity) information taken by the camera 10, generating a tool virtual body ITm corresponding to the tool RTn and the sleeve RTSn described later, a bolt virtual body IBn corresponding to the bolt RBn, and the like, and causing them to appear in the virtual space, or causing the objects appearing in the virtual space to overlap the real space to create an augmented reality space, or judging the interference (collision) of the tool virtual body ITm and the bolt virtual body IBn in the virtual space.

[0054] The augmented reality server 12 acquires and stores the position information of each bolt, the direction information indicating the direction of the bolt from the tool as a starting point, and the like when the operator performs the bolt fastening work. For example, the augmented reality server 12 can acquire and keep the data of the fastening order used when each bolt is tightened, the fastening lower limit torque, the fastening target torque, the fastening upper limit torque, the fastening completion torque observed by the operator when actually fastening the bolt, the fastening date and time, and the like information that is taught to the operator.

[0055] Further, the augmented reality server 12 can have a function of saving the work content of the first operator, creating a work manual provided when the work content of the first operator is taught to the second operator, showing the work manual to the see-through screen 14 to indicate the work content to the second operator, and acquiring and keeping the fastening work result (fastening completion torque, work date and time, and the like information) of the second operator.

[0056] In addition, the augmented reality server 12 can also acquire, via the network 5, any information stored in the network server 26, and project the information to the see-through screen 14 to notify the worker of the information.

[0057] Note that the augmented reality server 12 can be integrated with the camera 10, or the memory area and processing section possessed by the augmented reality server 12 can be provided at a location remote from the wearable device 7 via the network 5.

[0058] [Tool RTn]

[0059] The tool RTn is a tool that tightens or loosens a mechanical fastening portion such as a bolt, a screw, or a rivet. Hereinafter, the action of tightening or loosening such a mechanical fastening portion will be referred to as "tightening".

[0060] Note that "n" of the tool RTn is a subscript indicating that a plurality of types are prepared in accordance with the purpose of each tool, the size of the mechanical fastening portion such as a bolt, and the like. In this specification, for the sake of explanation, a case in which the mechanical fastening portion is a bolt or a rivet is exemplified, but the mechanical fastening portion of the present application is not limited thereto, and can be any fastener such as a nail or a pin.

[0061] The tool RTn has a tool handle portion RTEn including a grip portion for a worker to hold, a tool head portion RTHn connected to the tool handle portion RTEn, and a communication display portion RTT that receives and transmits tightening information of the mechanical fastening portion tightened or loosened by the tool RTn to the network 5, and displays information related to the tightening to the worker.

[0062] In a case in which the mechanical fastening portion is a bolt or a screw, the tightening information is information of a tightening torque observed from when the tightening is started until when the tightening is completed when tightening the bolt (screw) RBn, and information of a residual torque measured when loosening the tightened bolt.

[0063] In addition, in a case in which the mechanical fastening portion is a rivet, the tightening information can also be information of a breakage of a shaft of a rivet held by a rivet machine that strikes the rivet.

[0064] In a case in which the mechanical fastening portion is a bolt, the tool head portion RTHn can be a torque wrench capable of wearing various sleeves RTSn corresponding to the size of the bolt and the type of the bolt to the head portion RTHn of the tool RTn, or can be various torque wrench shapes such as a wrench shape, a swing wrench shape, and the like. In addition, in a case in which the mechanical fastening portion is a rivet, the tool head portion RHTn can be a portion including a gripping case that holds the rivet.

[0065] The communication display unit RTT, which is integrated into the tool RTn, has the following functions: teaching the operator the target tightening torque value (Tr_Set), conveying to the operator that the bolt has been tightened to a value higher than the target torque value (Tr_Set) and lower than the upper tightening torque value (Tr_Max), and indicating that the tightening operation has been completed with the specified tightening torque value. This information can also be displayed on a transparent screen 14 or on the display unit of another wearable terminal, such as the operator's wrist.

[0066] In addition, when the mechanical fastening part is a rivet, the communication display unit RTT can be configured to be connected to the gripping housing that holds the rivet, and to receive and transmit the ON / OFF signal of the trigger provided by the handle equivalent to the tool handle RTEn.

[0067] [Network Server 26]

[0068] The network server 26 stores production work instructions showing the production content of manufactured objects (e.g., railway vehicles, motor vehicles, etc.), maintenance work instructions showing the maintenance work content, and so on. Furthermore, the network server 26 has the function of retrieving only the necessary data from the stored data and providing it to the augmented reality server 12 upon request from the augmented reality server 12.

[0069] In addition, the network server 26 has the following functions: it can capture images of the base plate and components (e.g., ...) from the camera 10. Figure 2 The images of the base plate RD1 and components RD2, tool RTn, sleeve RTSn, etc. shown are processed by edge processing and image analysis to extract feature points from the fastened object. The feature points are assigned a real space origin RO(0,0,0) or a virtual space origin IO(0,0,0) corresponding to the real space origin is generated in the virtual space corresponding to the real space origin.

[0070] It should be noted that the image parsing used here can be any method, such as the Canny algorithm, the second-order method, thresholding and continuous processing, or processing based on convolutional neural networks.

[0071] In addition, the network server 26 has the following functions: it can analyze the images captured by the camera 10 to obtain the main dimensions and other size information of the tool RTn and the sleeve RTSn, or obtain the position information of the tool RTn and the sleeve RTSn in real space (e.g., coordinates representing three-dimensional positions), or transfer the position and size information of the real space to the virtual space based on the virtual space origin IO, which corresponds one-to-one with the origin RO of the real space.

[0072] In addition, the network server 26 can also be configured to perform image analysis on a mark such as a QR code (registered trademark) marked on the tool RTn and the sleeve RTSn and the like, and read the information stored in the mark to acquire the size information and the position information described above.

[0073] Note that a part or most of the functions (processes) of the augmented reality server 12 described above can be replaced by the functions (processes) of the network server 26. Therefore, in the following description, the functions (processes) of the augmented reality server 12 and the functions (processes) of the network server 26 are not distinguished, and are simply described as the functions (processes) of the server as either the augmented reality server 12 or the network server 26 for convenience of explanation.

[0074] In addition, as described above, the mechanical fastening portion includes a bolt, a screw, a rivet, and the like, but in the following description, a rivet is exemplified as a representative mechanical fastening portion to describe the implementation mode.

[0075] <Job content>

[0076] Next, the assembly job in the augmented reality-based mechanical fastening job method of the embodiment of the present application will be described with reference to Figure 2

[0077] Figure 2 is an assembly view illustrating an example of the assembly job 200 in the augmented reality-based mechanical fastening job method of the embodiment of the present application. The assembly job 200 here is a job of fixing the component RD2 illustrated in FIG. 8 to the base disk RD1 by four bolts RB1 to RB4 (hereinafter, RBn (n = 1 to 4), or simply RBn) in a real space. The base disk RD1 has threaded holes RQ1 to RQ4 having threaded grooves in which the bolts RBn are screwed through the component RD2. In addition, the component RD2 has through-holes RP1 to RP4 for the bolts RBn to pass through in the vertical direction at the four corners thereof. Figure 2

[0078] Next, the job instruction sheet indicating the job of mounting a component on a base disk of the embodiment of the present application will be described with reference to Figure 3

[0079] Figure 3 is an example of a job instruction sheet 360 indicating the job of mounting a component on a base disk. As described above, the job instruction sheet 360 is a document for instructing the job of mounting a component on a base disk, and includes a drawing of the base disk and a drawing of the component to be mounted on the base disk. Figure 3 ​​​In the work instruction 360, for example, information such as a figure number, a drawing name, a component name, a material, a bolt size, a bolt neck length, a bolt material, a specified tightening torque value (a lower limit value, a target value, an upper limit value, etc.), a bolt number, a tool, and a sleeve type is managed. In a case where information such as a bolt position, an obstacle around and in front of the bolt, and the like is obtained in advance, a correspondence relationship between the bolt RBn and the sleeve RTSn for the bolt RBn can be recorded in the work instruction 360. Figure 3 The work instruction 360 is described.

[0080] Note that, in the work instruction 360, Figure 3 , information such as a bolt position, an obstacle around and in front of the bolt, and the like is obtained in advance, a correspondence relationship between the bolt RBn and the sleeve RTSn for the bolt RBn can be recorded in the work instruction 360. Figure 2 In the work instruction 360, a case where the bolt RB1, the bolt RB2, and the bolt RB4 are fastened by the sleeve RTS1 having a sleeve length (D21) of 50 mm, and the bolt RB3 is fastened by the sleeve RTS2 having a sleeve length (D22) of 80 mm is taken as an example.

[0081] <Definition of Representative Point Representing Bolt Position>

[0082] Next, with reference to Figure 4 , a representative point representing a position of a bolt for a mechanical fastening work will be described.

[0083] Figure 4 is a schematic view showing a representative point representing a position of a bolt for a mechanical fastening work. The position of the representative point representing a position of a bolt for a mechanical fastening work will be described taking a hexagonal bolt as an example. The hexagonal bolt is composed of a head portion a covered by a sleeve RTSn, a cylindrical portion b connected to the head portion a, and a threaded portion c connected to the cylindrical portion b. A bolt position (coordinates) ORBn representing a position of the bolt RBn is set to a point at which a surface connecting the head portion a and the cylindrical portion b intersects with an axis Rz of the bolt RBn.

[0084] The bolt position ORBn representing the position of the bolt RBn in the real space described above corresponds to the virtual bolt position OIBn representing the position of the virtual bolt IBn formed in the virtual space on a one-to-one basis.

[0085] After that, in a case where the bolt virtual body IBn is displayed so as to overlap with the bolt hole RPn of the component RD1 in a state where the bolt RBn is not inserted into the real space (augmented reality space, refer to Figure 14 ), the position OIBn of the bolt virtual body IBn is displayed on the surface of the component RD1, and an effect that the worker can easily grasp a state in which the bolt RBn is constructed is obtained.

[0086] Note that the definition method of the bolt position is described using hexagonal bolts as an example, but the same effect can be obtained as long as the representative position indicating the position of the bolt is set to the above position in the case of replacing the hexagonal bolts with socket head cap screws or screws.

[0087] <Explanation of positional relationship of tool, sleeve, and bolt in fastened state>

[0088] Next, with reference to Figure 5 , the relationship from the representative point of the tool to the representative point of the bolt when fastening the bolt with the tool provided with the sleeve is described for the embodiment of the present application.

[0089] Figure 5 is a schematic view showing the relationship from the representative point of the tool to the representative point of the bolt when fastening the bolt with the tool provided with the sleeve. More specifically, Figure 5 is a schematic view at the time of completion of the work of fastening the bolt RBn that fastens the member RD2 to the base disk RD1 with the tool RTn composed of the tool head RTHn and the tool shank RTEn at a prescribed fastening target torque (Tr Set).

[0090] When the representative position (coordinates) ORBn of the bolt RBn is arranged in a manner orthogonal to an x-y plane composed of a horizontal x axis and a y axis in the normal direction of the paper surface, the position (coordinates) ORTHn of the tool head RTHn overlaps the axis of the cylindrical sleeve RTSn in the z axis (in the paper surface, the axis in the vertical direction).

[0091] When the thickness (dimension in the z direction) of the tool head RTHn is set to D1n and the height dimension of the sleeve RTSn along the z axis is set to D2n, the tool dimension h1 (hereinafter, sometimes simply referred to as dimension h1) from the representative position (coordinates) ORTHn of the tool head RTHn to the position ORBn of the bolt RBn becomes the sum of the tool head thickness dimension D1n and the sleeve length dimension D2n.

[0092] Note that the side surface of the tool head RTHn and the side surface of the sleeve RTSn shown in Figure 5 are marked with marks for image analysis after being photographed by the camera 10.

[0093] <Explanation of case where the bolt fastening torque observed by the tool RTn is generated>

[0094] Next, with reference to Figure 6 , the progress of the fastening torque observed when fastening the bolt with the tool is described for the embodiment of the present application.

[0095] Figure 6is a graph showing the progress of the tightening torque value (hereinafter, sometimes simply referred to as torque) observed when tightening the bolt with the tool. Generally, for the bolt, in order not to cause the bolt breakage due to excessive tightening, looseness due to insufficient tightening, the tightening state of the bolt is preferably managed in accordance with the axial force generated in the bolt. However, it is not easy to directly observe the axial force of the bolt, and therefore, instead of the axial force of the bolt, the tightening state of the bolt is managed in accordance with the tightening torque which is proportional to the axial force of the bolt.

[0096] Figure 6 The horizontal axis thereof is an axis indicating time, and the vertical axis thereof is an axis indicating the tightening torque measured by the tool RTn when tightening the bolt RBn with the tool RTn. The display communication section RTT possessed by the tool RTn receives the tightening torque information (time duration data of the tightening torque, and the tightening torque, etc.) acquired when tightening the bolt RBn with the tool RTn, via the network (for example, the network 5 shown in Figure 1 Figure 1 The display communication section RTT possessed by the tool RTn receives the tightening torque information (time duration data of the tightening torque, and the tightening torque, etc.) acquired when tightening the bolt RBn with the tool RTn, via the network (for example, the network 5 shown in

[0097] When starting tightening the bolt RBn with the tool RTn, the tool RTn observes the tightening start torque Tr_B at the tightening start time t_B. Thereafter, the tool RTn is returned (return action of the ratchet) and the bolt RBn is tightened again with the tool RTn, and therefore, the inclined portion where the tightening torque is observed, the horizontal portion where the tightening torque is not observed at the time of the return action, and the inclined portion where the tightening torque is observed are alternately observed.

[0098] In particular, in the case where the inclination (for example, the rising rate of the torque value with respect to the angle swept by the tool handle RTEn) of the inclined portion at the time of observing the tightening start torque Tr_B exceeds a prescribed inclination, the network server (for example, the network server 26 shown in Figure 1 Figures 7-12 The above-described seizure (seizure) is detected in the step 150 and the step 390 of the flowchart described above. By including these steps, the seizure of the bolt can be detected early, and therefore, the tightening work with a prescribed quality can be achieved.

[0099] ​​The bolt RBn is provided with a fastening target torque Tr Set at which a prescribed axial force is obtained, a fastening lower limit torque Tr Min that is a lower limit that can be allowed on the insufficient side of the fastening torque with the fastening target torque Tr Set as a reference, and a fastening upper limit torque Tr Max that is an upper limit that can be allowed on the excessive side of the fastening torque. Generally, the fastening lower limit torque Tr Min and the fastening upper limit torque Tr Max can be decided taking into consideration the usage environment or the like.

[0100] <Method of recording work content of first worker and demonstrating produced work steps (manual) to second worker>

[0101] Next, referring to Figures 7 to 12 A method of mechanical fastening work based on augmented reality according to the present application will be described.

[0102] Figures 7 to 12 is a diagram showing Part 1 to Part 6 of a flowchart illustrating a method of mechanical fastening work based on augmented reality according to the present application. In addition, steps 10 (S10) to 220 (S220) are related to work of a first worker, and steps 230 (S230) to 460 (S460) are related to work of a second worker to whom work content of the first worker is demonstrated.

[0103] In the method of mechanical fastening work based on augmented reality according to the present embodiment, in the absence of information of a virtual space such as a 3D-CAD model, work content of a first worker can be recorded in an augmented reality space (ensuring traceability), a work step (manual) for a second worker is produced based on the work record of the first worker, and the produced work step is demonstrated to the second worker via the augmented reality space.

[0104] Specifically, the method of mechanical fastening work based on augmented reality according to the present application is a method in which an augmented reality server 12 (for example, Figure 1 or a network server 26) produces a work step (manual) using saved information of a first worker after saving the work content of the first worker, and demonstrates the produced work step to a second worker via a wearable device 7 worn by the second worker.

[0105] Hereinafter, referring to Figure 2 , an assembly work illustrated in the drawing will be taken as an example, and contents of each step constituting a method of mechanical fastening work based on augmented reality will be described with reference to the flowchart of Figures 7-12 .

[0106] <Fastening work of first worker>

[0107] Step 10 (S10): Start of mechanical fastening work (method) based on augmented reality.

[0108] Step 20 (S20): The first worker grasps the assembly work contents based on an assembly drawing (D1) and a work instruction (D2), and the like. Figure 2 Figure 3

[0109] Step 30 (S30): The first worker, based on the work contents grasped, after positioning the component RD2 on the base disk RD1, inserts the bolts RBn (n = 1 to 4) into the through holes RPn (n = 1 to 4) of the component RD2 and manually fastens (temporarily fastens) them to the threaded holes RQn (n = 1 to 4) of the base disk RD1 as shown in (D3). Figure 13

[0110] Note that the bolts RBn (n = 1 to 4) are in a state of being temporarily fastened, and therefore the heads a of the bolts RBn (n = 1 to 4) are in a state of floating from the upper surface of the component RD2.

[0111] Step 40 (S40): The first worker wears the wearable device 7, turns on the power, and starts the augmented space generation application software.

[0112] Step 50 (S50): The server creates a record table that records the work contents of the first worker. This record table is a data table before the first worker fastens the bolts (refer to (D4)). Figure 14

[0113] This data table has, for the bolts RBn (n = 1 to 4) that are the targets of the fastening work, a column that records the position coordinates (X, Y, Z) of the representative point (ORBn (n = 1 to 4)) of the bolt, a unit direction vector (in, jn, kn) (n = 1 to 4, hereinafter sometimes abbreviated as direction vector) that has a starting point at a point (tool head representative position ORTHn) on the tool RTn and represents the direction in which the bolt RBn (n = 1 to 4) exists when viewed from the tool RTn, a fastening torque lower limit value Tr_Min (Nm) of the bolt RBn (n = 1 to 4), a fastening target torque value Tr_Set (Nm), a fastening torque upper limit value Tr_Max (Nm), a torque value actually fastened (Tr_rslt), a sequence (Seq) in which the bolts RBn (n = 1 to 4) are fastened, a work date, and a work time. Figure 4

[0114] Note that, at the time point of S50, in this data table (D4), only the fastening torque lower limit value Tr_Min (Nm), the fastening target torque value Tr_Set (Nm), and the fastening torque upper limit value Tr_Max (Nm) of the bolts RBn (n = 1 to 4) are obtained from the work instruction (D2) and the like. Figure 14 Figure 3 Note that, at the time point of S50, in this data table (D4), only the fastening torque lower limit value Tr_Min (Nm), the fastening target torque value Tr_Set (Nm), and the fastening torque upper limit value Tr_Max (Nm) of the bolts RBn (n = 1 to 4) are obtained from the work instruction (D2) and the like.​​​​​​

[0115] Step 60 (S60): The server performs image processing on the work object (the base disk RD1 and the component RD2 temporarily fastened with the bolts RBn (n = 1 to 4) shown) captured by the camera 10, and after finding the characteristic point of the work object, configures the origin RO of the real space at the characteristic point. Figure 13 Step 60 (S60): The server performs image processing on the work object (the base disk RD1 and the component RD2 temporarily fastened with the bolts RBn (n = 1 to 4) shown) captured by the camera 10, and after finding the characteristic point of the work object, configures the origin RO of the real space at the characteristic point.

[0116] Step 70 (S70): The server configures the origin IO of the virtual space in the virtual space in correspondence with the origin RO of the real space.

[0117] Steps 80 (S80) to 200 (S200) are a cycle of work of the first worker fastening the four bolts RBn (n = 1 to 4). The first worker can decide the order of fastening the bolts RBn (n = 1 to 4) based on the size and features of the component RD2 and based on his or her experience, or can refer to a work instruction book (not shown) in which the order of fastening the bolts is recorded, or the like.

[0118] Step 90 (S90): The first worker fits the tool RTn equipped with the sleeve RTSn to the head portion a of the temporarily fastened bolt RB1.

[0119] In the case where the correspondence of the bolt RBn and the sleeve RTSn is able to be grasped in advance, and the correspondence information is recorded in the work instruction book shown, the procedure in which the first worker selects the correct sleeve RTSn and judges whether or not the bolt RBn is to be fastened can be added after S90. Figure 3

[0120] Step 100 (S100): The server grasps the tool head position ORTHn in the tool reference plane TSP from the image acquired by the camera 10.

[0121] Step 110 (S10): The server generates the unit normal vector NV of the tool reference plane TSP toward the space on the side of the tool reference plane TSP which is not visually observed by the camera 10, with the tool head position ORTHn included in the tool reference plane TSP as a starting point, and records the coordinates (in, jn, kn) (n = 1 to 4) of the unit normal vector NV.

[0122] The unit normal vector NV indicates the direction in which the bolt RBn (n = 1 to 4) exists on either one of the side of the one plane and the side of the other plane of the tool reference plane TSP. Therefore, hereinafter, the unit normal vector NV will be simply referred to as the direction vector NV.

[0123] Step 120 (S120): The server generates the tool virtual body ITm by performing image analysis on the tool head RTHn and the sleeve RTSn in the virtual space.

[0124] ​Step 130 (S130): The server causes the tool virtual body ITm to appear from the tool head position ORTHn of the origin of the direction vector NV along the direction vector NV.

[0125] Next, the server's actions (the steps of causing the tool virtual body ITm to appear in the virtual space) described in steps 90 to 120 will be explained in detail. Figure 15 Figure 15 is a schematic view showing a case where the tool virtual body ITm of the simulated tool appears in the virtual space.

[0126] Note that the ITm actually existing on the Zm axis overlaps the tool head RTHn and the sleeve RTSn, but for ease of explanation, the ITm is described separately from the tool head RTHn and the sleeve RTSn in Figure 15 and the following Figure 16 , Figure 17 .

[0127] First, the first worker visually observes the tool RTn having the sleeve RTSn fitted on the bolt RB1 through the see-through screen 14. At this time, the camera 10 also captures the image visually observed by the first worker.

[0128] Note that in order to make it easy for the camera 10 to grasp the features of the tool head RTHn, a cover or the like having features that are easy for the camera 10 to capture can be fitted on the tool head RTHn.

[0129] The server performs image processing on the tool head RTHn of the tool RTn captured by the camera 10, and generates the tool reference surface TSP in the virtual space. In addition, in correspondence therewith, the server records the tool head position coordinates ORTHn included in the tool reference surface TSP to the virtual space, and generates the tool head position OITHn in the virtual space.

[0130] The server generates a unit normal vector (direction vector) NV in the normal direction of the space toward the side of the tool reference surface TSP that is not visually observed by the camera 10, with the tool head position OITHn of the virtual space as the origin, and records the coordinates (in, jn, kn) of the direction vector NV (n = 1 to 4).

[0131] In addition, the server generates the tool virtual body ITm of a cylinder having an outer diameter RO and a length hi in the axial direction in the virtual space, based on the outer diameter OR of the cylindrical sleeve RTSn obtained through image processing, and the dimension hi of the sum of the tool head thickness dimension D1n and the sleeve length dimension D2n obtained through image processing.

[0132] ​Note that, in a case where the tool head thickness dimension D1n, the sleeve length dimension D2n, and the outer diameter OR of the sleeve RTHn are difficult to obtain through image analysis by the server, the tool head RTHn and the kind of the sleeve RTHn can also be identified through image analysis. Thereafter, the server can also acquire the tool head thickness dimension D1n, the sleeve length dimension D2n, and the outer diameter OR of the sleeve RTHn that are recorded in advance in the server and that are associated with the kind identified through image analysis, and calculate the dimension h1, thereby generating the tool virtual body ITm. The same applies in step 340 described later.

[0133] Next, the server superimposes the tool virtual body representative point OITm on the tool head position OITHn (= the tool head position ORTHn in the real space) of the virtual space, and causes the tool virtual body OITm to appear in the direction of the orientation vector NV.

[0134] Step 140 (S140): The first worker starts fastening (formal fastening) the bolt RB1 using the tool RTn with the sleeve RTHn, with the fastening target torque Tr_Set as the target.

[0135] Step 150 (S150): The server receives the fastening start torque Tr_B measured by the tool RTn, and records the timing at which the fastening start torque Tr_B is received as the fastening start timing t_B.

[0136] Step 160 (S160): The first worker completes fastening of the bolt BR1 by confirming that the fastening completion torque Tr_rslt observed by the tool RTn is close to the target torque Tr_Set (for example, within a predetermined range set in advance) and is between the fastening lower limit torque Tr_Min and the fastening upper limit torque Tr_Max.

[0137] Step 170 (S170): The server records the fastening completion torque Tr_rslt transmitted by the tool RTn, and records the timing at which the fastening completion torque Tr_rslt is observed as the fastening completion timing t_E.

[0138] Step 180 (S180): The server determines the cylindrical bolt virtual body IB1 that corresponds one-to-one to the bolt RBn loosened by the tool RTn, and generates and causes the bolt virtual body IB1 to appear in the virtual space in the direction of the orientation vector NV.

[0139] Step 190 (S190): The server records the coordinates (x1, y1, z1) of the representative position IOB1 of the virtual bolt body IB1 corresponding to the real bolt RB1, the coordinates (i1, j1, z1) of the direction vector NV, the operation date and time, and the tightening completion time of bolt RB1 (the moment when the tightening completion torque Tr_rslt is observed) in the first operation table. Figure 14 (The data table shown).

[0140] Here, the coordinates (x1, y1, z1) of the bolt virtual body IB1, and the direction vector (i1, j1, z1) representing the direction in which the bolt virtual body IB1 is located, starting from the tool head position ORTHn contained in the tool reference plane TSP, are the position information of the mechanical fastener.

[0141] At this time, the information of the sleeve RTSn used when tightening bolt RBn can also be recorded in the first work sheet. Usually, the sleeve RTSn selected when tightening bolt RBn is selected based on the position of bolt RBn, whether there is enough space in front of bolt RBn, and whether there are any obstacles.

[0142] Therefore, when the first operator is performing the operation for the first time, it may be difficult to organize the information related to the sleeve RTSn and record it in the first operation sheet in advance. Figure 14 ), Work Specifications ( Figure 3 Therefore, the sleeve information related to the sleeve RTSn selected by the first operator can be recorded in the first work sheet, the work content can be communicated to the second operator, and it can be confirmed whether an appropriate sleeve RTSn has been selected in the second operator's work based on images captured by the camera 10, such as in step 300 described later.

[0143] Step 200 (S200): Repeat steps S80 to S200 corresponding to the number of cycles (4 times) to reach the number of bolts RBn.

[0144] Next, refer to Figure 16 The actions of the server described in S180 to S190 (the generation and display of the bolt virtual body IBn, and the steps of recording the work performed by the first operator in the data table) are explained in detail. Figure 16 This is a schematic diagram illustrating an embodiment of the present invention, showing a virtual bolt body simulating a bolt in virtual space.

[0145] First, the server has control. Figure 2 Assembly diagram, or Figure 3 The work instructions, or the bolt size and bolt neck length obtained through image analysis, are used to form a cylindrical virtual bolt body IBn of the corresponding size.

[0146] Note that, in a case where it is difficult to grasp the bolt size, the bolt neck length, and the like by image analysis, the server can also grasp the bolt size, the bolt neck length, from the work instruction sheet (WIS) stored in the server, to form the bolt virtual body IBn. Figure 3 The full length of the bolt virtual body IBn is the sum of the size of the head a and the cylindrical portion b of the bolt RBn (n = 1 to 4) along the z-axis of the threaded portion c, and the outer diameter of the bolt virtual body IBn is the outer diameter of the head a when the head a of the bolt RBn is observed from the z-axis.

[0147] The server configures the bolt virtual body representative point OIBn at a position where the head a is lowered from the upper end portion of the bolt virtual body IBn by the size of the z-axis after the size of the z-axis of the bolt virtual body IBn is acquired.

[0148] In addition, the server visualizes the cylindrical bolt virtual body IBn in such a manner that the bolt virtual body representative point OIBn is located at a point of the sum of the tool head thickness size D1n and the sleeve length size D2n, that is, the size h1 in the direction of the direction vector NV from the tool virtual body representative point (ITm) when the tightening completion torque Tr_rslt is observed.

[0149] At this time, the cylindrical tool virtual body ITm and the cylindrical bolt virtual body IBn are visualized in the virtual space in such a manner that the axes of the tool virtual body ITm and the bolt virtual body IBn are configured on a common axis Zm including the direction vector NV, and the lower end portion of the tool virtual body ITm overlaps the upper end portion of the bolt virtual body IBn.

[0150] The bolt virtual body representative point OIBn is configured when the tightening completion torque Tr_rslt is observed, and thus the bolt virtual body representative point OIBn can be configured at a position where the sinking size due to the formal tightening of the bolt RBn that was temporarily tightened by manual tightening using the tool RTn is reflected, that is, the upper surface of the member RD2 (the surface on which the lower surface of the head a of the bolt abuts).

[0151] By using the method having the above-described features, when the bolt position is shown to the second worker in the augmented reality space using the bolt virtual body representative point OIBn, the position of the bolt RBn is located on the upper surface of the member RD2, and thus the second worker can easily and intuitively grasp the state at the time of bolt tightening, and an effect in which the work content can be more easily understood is obtained.

[0152] Next, the data table in which the bolt tightening work of the first worker is recorded will be described with reference to

[0153] Figure 17 to the embodiment of the present application.

[0154] ​Figure 17 is a data table including data on various data related to the work of the first worker recorded when the first worker completes fastening of all the bolts RBn (n = 1 to 4) (when the cycle of S80 to S200 is performed for 4 times).

[0155] Note that, Figure 17 the data table after the work of the first worker is a data table in which various data recorded by the first worker fastening the bolts RBn (n = 1 to 4) is written in the empty columns of the data table before the work of the first worker. Figure 14

[0156] More specifically, Figure 17 is a graph in which the coordinates (Xn, Yn, Zn) (n = 1 to 4) of the position coordinates of the virtual space corresponding to the position coordinates of the bolts RBn (n = 1 to 4) in the real space, the coordinates (in, jn, kn) (n = 1 to 4) of the direction vector NV with the representative point OITm of the tool virtual body ITm in the virtual space as the origin, the fastening completion torque Tr_rslt, the order of fastening (Seq), the work day, and the work time are written in the data table shown in Figure 14

[0157] Note that, in the data table shown in Figure 17 , the order of fastening (Seq), the work day, and the work time can be written as "time information" together. In addition, the work time is set to the time t_E at which the server receives the fastening completion torque Tr_rslt when the bolt RBn (n = 1 to 4) is fastened, but can be the bolt fastening start time t_B.

[0158] Step 210 (S210): The server grasps the order (Seq) in which the bolts RBn (n = 1 to 4) are fastened based on the work time of the bolts RBn (n = 1 to 4) and records it in the data table described above. In addition, in correspondence therewith, the server confirms that the fastening work of all the fastening target bolts RBn (n = 1 to 4) is completed again by writing data in all the empty columns of the data table shown in Figure 14

[0159] Furthermore, the server can easily calculate the man-hours of the first worker based on the time required from the start of fastening of the bolt RB1 to the completion of fastening of the bolt RB4.

[0160] ​​​Regarding the sequence (Seq), it can be the sequence of each record in the tightening completion time t_E of each bolt RBn (n=1~4) after the tightening is completed, or it can be set to the sequence formed by extracting the bolt tightening start time t_B or bolt tightening completion time t_E at the time point when all bolts are tightened and arranging them from the old time to the new time.

[0161] Step 220 (S220): As Figure 18 As shown, after the server tightens all bolts RBn (n=1~4), it captures and saves an image in the augmented reality space containing the objects of the tightening operation, displaying the work results (operator ID, work day, work time, tightening completion torque value Tr_rslt), work instructions, etc., near the bolts RBn (n=1~4). By saving this captured image, even if... Figure 17 Even if part of the data in the data sheet showing the bolt tightening operation of the first operator is damaged or lost, it is still possible to ensure evidence (traceability) related to the bolt tightening operation.

[0162] By employing the mechanical fastening method that includes the steps described above, a low-cost and high-quality mechanical fastening method can be provided. This method can ensure that the bolt fastening work to be performed by the operator is reliably carried out without any omissions, and can record the results of the work to ensure traceability.

[0163] Fastening work performed by a second operator.

[0164] The next step 230 and subsequent steps relate to the bolt tightening operation performed by a second operator who has been instructed on the work content based on the work instructions (manual) generated based on the work content of the first operator.

[0165] Step 230 (S230): The second operator puts on the wearable device 7, turns on the power, and starts the augmented space generation application software.

[0166] Step 240 (S240): The server, based on the first work schedule when the first operator completes the fastening operation ( Figure 17 The second work sheet for bolt tightening operations by the second operator was created by removing data on the tightening completion torque Tr_rslt, work day, and work time. Figure 19 (See the data table shown). At this time, the second work table contains the position information of the bolt RBn tightened by the first operator.

[0167] Step 250 (S250): The server processes the work object captured by camera 10. Figure 13The base plate RD1 and the component RD2 shown as being temporarily fastened by the bolts RBn (n = 1 to 4) are subjected to image processing, a characteristic point of the work object is found, and a real space origin RO is set. Thereafter, the server sets a virtual space origin IO in the virtual space which corresponds one-to-one to the real space origin RO.

[0168] Step 260 (S260): The server reads the second work sheet shown in FIG. 10 in the augmented reality space containing the work object of the second worker, and based on the coordinates (Xn, Yn, Zn) of the representative points OIBn of the bolt virtual bodies IBn (n = 1 to 4) and the coordinates (in, jn, kn) of the direction vectors NV from the representative point OITm of the tool virtual body ITm (n = 1 to 4), the server visualizes the bolt virtual bodies IBn (n = 1 to 4) in the augmented reality space on the see-through screen 14 of the second worker, with the virtual space origin IO set in S250 as a reference.

[0169] Next, the bolt fastening work by the second worker according to the embodiment of the present application will be described with reference to Figure 20

[0170] Figure 20 is an image of the augmented reality space shown on the see-through screen of the second worker, and is a schematic view showing the position of the bolt to be fastened next by the second worker.

[0171] The server creates a data table (shown in FIG. 10) which retains the bolt positions (Xn, Yn, Zn), the direction vectors NV (in, jn, kn) indicating the directions in which the bolts RBn are located from the tools RTn, and the fastening order, and eliminates only the fastening completion torque Tr_rslt, the work date, and the work time at which the work was performed by the first worker, based on the data table (shown in FIG. 9) which records the work history of the first worker who has completed the work. Figure 17 Figure 19 Thereafter, the server can display the bolt position, the number of bolts, and the order in which the bolts are to be fastened next by the second worker on the see-through screen 14 of the second worker, based on the second work sheet.

[0172] As a result, the second worker can accurately grasp the bolt position, the number of bolts, and the order in which the bolts are to be fastened next in a short time without browsing the assembly drawing (shown in FIG. 1) Figure 2 Figure 3

[0173] At this time, the representative position OIBn of the bolt virtual body IBn is defined as the center of the virtual body IBn, and the direction vector NV is defined as the direction from the representative point OITm of the tool virtual body ITm to the representative point OIBn of the bolt virtual body IBn. Figure 4 ​​​​The bolt shown represents the position ORBn, and thus the bolt virtual body IBn is displayed on the see-through screen 14 of the second worker in such a manner that the bolt virtual body IBn (n = 1 to 4) is positioned at the representative position OIBn on the surface of the member RD2, so that the second worker can easily grasp the state after the completion of the bolt fastening work.

[0174] By projecting the information of the second work table, which is created using the first work table (S230) in which the work history of the first worker is recorded (S220), on the see-through screen 14 (for example, the see-through screen 14 shown in FIG. 6), the second worker can be instructed of the fastening order of the bolts RBn. According to the instruction, the second worker can complete the fastening work in a state in which the member RD2 is maintained in parallel with respect to the base disk RD1, for example, and can easily fix the member RD2 to the member RD1 in a case where it is difficult to position the member RD2 in accordance with the shape, mounting attitude, or the like of the member RD2. Figure 17 Figure 1 The see-through screen 14 shown), so that the second worker can be instructed of the fastening order of the bolts RBn. According to the instruction, the second worker can complete the fastening work in a state in which the member RD2 is maintained in parallel with respect to the base disk RD1, for example, and can easily fix the member RD2 to the member RD1 in a case where it is difficult to position the member RD2 in accordance with the shape, mounting attitude, or the like of the member RD2.

[0175] As described above, the fastening order (Seq) of the bolts RBn (n = 1 to 4) can be set as the order recorded each time the fastening of each bolt is completed, or the fastening order of the bolts can be extracted at the time point at which the fastening of all the bolts is completed, the time t_B at which the fastening of the bolts is started, or the time t_E at which the fastening of the bolts is completed, and arranged in order from the old time to the new time.

[0176] Note that the mechanism for teaching the second worker of the work content (the positions of the fastened bolts, the order of fastening the bolts, and the like) of the first worker is not limited to the see-through screen 14 provided in the wearable device 7 worn by the second worker, and the work steps (manual) created based on the actions of the first worker can be taught to the second worker by being projected on a large screen or the like placed at the work site, for example.

[0177] Next, the steps from step 270 onward will be described. Figure 10

[0178] Step 270 (S270): After the member RD2 is positioned on the base disk RD1 based on the work content instructed via the see-through screen 14, the second worker inserts the bolts RBn (n = 1 to 4) in the through holes RPn (n = 1 to 4) of the member RD2 in a manner in which the bolt virtual bodies IBn (n = 1 to 4) shown in FIG. 6 overlap, and then manually fastens (temporarily fastens) the threaded holes RQn (n = 1 to 4) of the base disk RD1. Figure 20 Note that, as described above, the second worker can be instructed of the fastening order of the bolts RBn (n = 1 to 4) by projecting the information of the second work table, which is created using the first work table (S230) in which the work history of the first worker is recorded (S220), on the see-through screen 14 (for example, the see-through screen 14 shown in FIG. 6).

[0179] Figure 21 ​​​As shown, the bolt RBn (n = 1 to 4) is in a state of temporary fastening, and the head a of the bolt RBn (n = 1 to 4) becomes a state of floating from the upper surface of the member RD2.

[0180] Steps S280 to S430 are a cycle (repeat) of the work of the second worker fastening the four bolts RBn (n = 1 to 4).

[0181] Step S300: The server performs image analysis of the tool RTn fitted with the sleeve RTSn, and judges whether the second worker has selected an appropriate tool RTn and sleeve RTSn. If an appropriate tool RTn and sleeve RTSn have not been selected, the process returns to the upper level of S290, and an appropriate tool RTn is selected. If an appropriate tool RTn and sleeve RTSn have been selected, the process proceeds to the next step S300.

[0182] Step S310: The server grasps the tool head position OITHn in the virtual space corresponding to the tool head position ORTHn in the tool reference plane TSP, based on the image of the tool RTn and sleeve RTSn taken by the camera 10.

[0183] Step S320: The server generates a direction vector NV from the tool head position ORTHn to the normal direction of the space on the side of the tool reference plane TSP that is not visually observed by the camera 10, and records the coordinates (in, jn, kn) (n = 1 to 4) of the direction vector NV.

[0184] Step S330: The server generates a tool virtual body ITm by performing image analysis of the tool head RTHn and sleeve RTSn in the virtual space.

[0185] Step S340: The server causes the tool virtual body ITm to appear from the tool head position ORTHn, which is the starting point of the direction vector NV, along the direction vector NV.

[0186] Steps S340 to S430 are a cycle (repeat) of the work of the second worker fastening the four bolts RBn (n = 1 to 4).

[0187] Step S350: The second worker fits the tool RTn fitted with the sleeve RTSn to the head a of the bolt RB1 that has been temporarily fastened, and starts the fastening (formal fastening) of the bolt RB1, targeting the fastening target torque Tr Set.

[0188] Step 360 (S360): The server determines whether the tool virtual body ITm is in contact (interference) with the bolt virtual body IB1 corresponding to the bolt RB1 in the virtual space. If the server determines that the tool virtual body ITm is not in contact with the bolt virtual body IB1 in the virtual space, the process returns to the upper step of S360, and if the server determines that the tool virtual body ITm is in contact with the bolt virtual body IB1, the process proceeds to Step 370.

[0189] Step 370 (S370): The server determines whether the tool RTn has generated the tightening torque. If the server determines that the tool RTn has not generated the tightening torque, the process returns to the upper step of S370, and if the server determines that the tightening torque has been generated, the process proceeds to Step 380.

[0190] Step 380 (S380): In the condition that the tool virtual body ITm is in contact with the bolt virtual body IB1 (S360) and the tool RTn has generated the tightening torque (S370), the server determines that the bolt RB1 corresponding to the bolt virtual body IB1 is tightened.

[0191] Next, the process of S350 to S380 will be described in detail with reference to Figure 22 to S380. Figure 22 is a schematic diagram showing a step of determining a bolt that has been tightened by a tool in a virtual space when a second operator tightens the bolt with the tool.

[0192] First, the server analyzes an image of the tool head RTHn provided with the sleeve RTSn captured by the camera 10, and grasps the tool head thickness dimension D1n and the sleeve length dimension D2n, and then acquires the h1 dimension and the outer diameter OR. Thereafter, the server grasps the tool reference surface RTS of the tool RTn when the tool RTn is captured by the camera 10, and causes the tool virtual body ITm reflecting the h1 dimension and the outer diameter OR to appear in the virtual space from the virtual tool head representative point OITHn (= tool virtual body representative point OITm) included in the tool reference surface RTS.

[0193] When the tool RTn provided with the sleeve RTSn is fitted on the head a of the bolt RB1 in the real space, the tool virtual body ITm is positioned above the bolt virtual body IB1 in the virtual space, and becomes a state in which a part of the both is overlapped (in contact or interfered), and when the tightening completion torque Tr_rslt is observed in the tool RTn, the tool virtual body ITm becomes a state in which the tool virtual body ITm is reliably overlapped (in contact or interfered) with the bolt virtual body IB1.

[0194] Note that there are a plurality of methods of determining whether the tool virtual body ITm is in contact (interference, overlap) with the bolt virtual body IB1 in the virtual space as shown below.

[0195] (1) A contact (interference, overlap) judgment based on the contact of the tool virtual body ITm and the bolt virtual body IB1.

[0196] (2) A distance-based judgment of the case where the distance between the representative point OITm of the tool virtual body ITm and the representative point OIB1 of the bolt virtual body IB1 is less than a prescribed value (in this case, h1).

[0197] (3) A judgment based on distance and the orientation (direction) of the tool virtual body ITm and the bolt virtual body IB1, on the basis of the judgment of (2), plus a judgment of whether the direction of the axis of the tool virtual body ITm and the direction of the axis of the bolt virtual body IB1 are the same or different.

[0198] Moreover, under the condition that the tool virtual body ITm is in contact with the bolt virtual body IB1, the tightening torque (Tr_B to Tr_Max) is observed in the tool RTn, and in addition, the bolt RB1 in the real space that corresponds one-to-one to the bolt virtual body IB1 is confirmed. Figure 6

[0199] By this step, for example, in the case where the second worker notices an error immediately after the tool RTn is fitted to the bolt RBn (n = any one of 2 to 4) and replaces the tool RTn to be fitted to the bolt RB1, etc., there is a possibility that the bolt RBn to which the tool RTn was fitted previously is erroneously judged to be tightened. Therefore, by simultaneously adopting the torque generation condition (S370) in the contact judgment (S360), it is possible to suppress the case where the bolt RBn for which the torque is not generated is erroneously judged to have been tightened.

[0200] Step 390 (S390): The server receives the tightening start torque Tr_B measured by the tool RTn, and records the time at which the tightening start torque Tr_B was received as the tightening start time t_B.

[0201] Step 400 (S400): The second worker completes the tightening of the bolt BR1 by confirming that the tightening completion torque Tr_rslt observed by the tool RTn is close to the target torque Tr_Set (for example, within a prescribed range set in advance), and is between the tightening lower limit torque Tr_Min and the tightening upper limit torque Tr_Max.

[0202] Step 410 (S410): The server records the tightening completion torque Tr_rslt transmitted by the tool RTn, and records the time at which the tightening completion torque Tr_rslt was observed as the tightening completion time t_E.

[0203] ​Step 420 (S420): The server records the coordinates (x1, y1, z1) of the representative position IOB1 of the virtual bolt body IB1 corresponding to the bolt RB1 of the real space, the direction vector NV(i1, j1, z1), the date and time of the work, and the tightening completion time (the time when the tightening completion torque Tr_rslt is observed) of the bolt RB1 in the data table (for example, the data table shown in Figure 23 ).

[0204] Step 430 (S430): The steps S280 to S430 are repeated corresponding to the number of cycles (4 times) to reach the number of bolts RBn.

[0205] Step 440 (S440): The server confirms again that the tightening work of all the bolts RBn (n = 1 to 4) planned to be performed has been completed, based on the fact that the empty columns of the data table are all filled. Figure 23

[0206] Step 450 (S450): The server captures and saves an image in which the work results (worker ID, work date, work time, tightening completion torque value Tr_rslt), work instruction, and the like are displayed in the vicinity of the bolts RBn (n = 1 to 4) in the augmented reality space including the tightening work target object, after all the bolts RBn (n = 1 to 4) are tightened.

[0207] By saving the image captured here (for example, refer to Figure 24 ), even in the case of data damage or loss of a part of the data table in which the tightening work of the bolts by the second worker is recorded, as shown in Figure 23 , it is possible to ensure evidence (traceability) related to the tightening work of the bolts.

[0208] Step 460 (S460): The tightening work by the second worker is ended.

[0209] According to the embodiment described above, it is possible to provide a mechanical tightening work method with less cost and high quality, which promotes the work handover to other workers and the like by utilizing augmented reality and recording the content of the mechanical tightening work performed by a certain worker to ensure traceability.

[0210] In addition, in the embodiment described above, the augmented reality-based mechanical tightening method of the embodiment of the present application is described, but the embodiment of the present application is not limited thereto, and can be realized by a computer program, a device, and the like, for example.

[0211] ​For example, the functions of the embodiments of the present application can also be implemented by a program code of software. In this case, a storage medium having the program code of the functions of the present application stored therein can be supplied to a device, a system, and a computer device (CPU, MPU, etc.) mounted on the device, the system reads the program code and implements it. The functions of the above-described embodiments are implemented by the computer executing the program code read from the storage medium, and the program code for implementing the functions, the storage medium, and the computer device itself become constituent elements of the present application.

[0212] As the storage medium for recording the program code, for example, there are a floppy disk, a CD-ROM, a DVD-ROM, a hard disk, an optical disk, a magneto-optical disk, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, and the like.

[0213] Next, the computer system for implementing the embodiments of the present application will be described with reference to Figure 25 to the drawings. Figure 25 is a block diagram of a computer system configured to implement the functions in the above-described augmented reality-based mechanical fastening work method.

[0214] The main components of the computer system 300 include one or more processors 302, a memory 304, a terminal interface 312, a storage interface 314, an I / O (input / output) device interface 316, and a network interface 318. These components are connected to each other via a memory bus 306, an I / O bus 308, a bus interface unit 309, and an I / O bus interface unit 310.

[0215] The computer system 300 can also include one or more general-purpose programmable central processing devices (CPUs) 302A and 302B, which are collectively referred to as the processor 302. In some embodiments, the computer system 300 can have multiple processors, and in other embodiments, the computer system 300 can be a single CPU system. Each processor 302 executes commands stored in the memory 304 and can include an on-board cache.

[0216] In some embodiments, the memory 304 can include a random access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile) for storing data and programs. The memory 304 can store all or a part of the programs, modules, and data structures that implement the functions described in this specification. For example, the memory 304 can store the fastening work management application 350. In some embodiments, the fastening work management application 350 can include commands or descriptions that implement the functions described below when executed on the processor 302.

[0217] In some embodiments, the fastening job management application 350 can also be implemented by hardware via semiconductor devices, chips, logic gates, circuitry, circuit cards, and / or other physical hardware devices, in place of or in addition to the processor-based system. In some embodiments, the fastening job management application 350 can also contain data other than commands or instructions. In some embodiments, a camera, sensor, or other data input device (not shown) can also be provided in direct communication with the bus interface unit 309, the processor 302, or other hardware of the computer system 300.

[0218] The computer system 300 can also include a bus interface unit 309 that facilitates communication among the processor 302, the memory 304, the display system 324, and the I / O bus interface unit 310. The I / O bus interface unit 310 can also be linked with an I / O bus 308 for transferring data between various I / O units. The I / O bus interface unit 310 can also communicate with a plurality of I / O interface units 312, 314, 316, and 318, also known as I / O processors (IOPs) or I / O adapters (IOAs), via the I / O bus 308.

[0219] The display system 324 can also include a display controller, a display memory, or both. The display controller can provide video, audio, or both, data to the display device 326. In addition, the computer system 300 can also include one or more sensors or other devices configured to collect data and provide the data to the processor 302.

[0220] For example, the computer system 300 can include a biological sensor that collects heart rate data, stress level data, and the like, an environmental sensor that collects humidity data, temperature data, pressure data, and the like, a motion sensor that collects acceleration data, motion data, and the like, and the like. Other types of sensors can also be used. The display system 324 can be connected with a display device 326 that is a separate display screen, a television, a tablet, a portable device, or the like.

[0221] The I / O interface unit has a function of communicating with various memories or I / O devices. For example, the terminal interface unit 312 can install a user I / O device 320 such as a user output device such as a video display device, a speaker television, and the like, a user input device such as a keyboard, a mouse, a keyboard, a touchpad, a trackball, a button, a light pen, or other pointing devices, and the like. The user can operate the user input device using the user interface, and thereby input input data, instructions, and the like to the user I / O device 320 and the computer system 300, and accept output data from the computer system 300. The user interface can be displayed on the display device, for example, via the user I / O device 320, or played through the speaker, or printed via the printer.

[0222] The storage interface 314 can install one or a plurality of disk drives, a direct access storage device 322 (typically a disk drive storage device, but can be an array of disk drives or other storage devices configured to be considered as a single disk drive). In some embodiments, the storage device 322 can be installed as any secondary storage device. The contents of the memory 304 can also be stored in the storage device 322, and read from the storage device 322 as necessary. The I / O device interface 316 can also provide an interface to other I / O devices such as a printer, a facsimile machine, and the like. The network interface 318 can also provide a communication route to enable the computer system 300 to communicate with other devices. The communication route can be a network 330, for example.

[0223] In some embodiments, the computer system 300 can also be a multi-user host system, a single-user system, or a server computer or the like that does not have a direct user interface and receives requests from other computer systems (clients). In other embodiments, the computer system 300 can also be a desktop computer, a portable computer, a notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other appropriate electronic device.

[0224] According to the above-described mechanical fastening work method based on augmented reality, a mechanical fastening work method with less cost and high quality can be provided, which does not require an additional identification mark such as an RFIC tag to the fastening portion, reception and transmission of data between the fastening portion and the fastening tool, and the like, and promotes work handover to other workers and the like by utilizing augmented reality and recording the content of the mechanical fastening work performed by a certain worker to ensure traceability.

[0225] Explanation of Reference Signs:

[0226] R0…real space origin

[0227] IO…virtual space origin

[0228] RD1…base disk

[0229] RD2… component

[0230] RPn… through hole

[0231] RQn… threaded hole

[0232] RTn… tool (ratchet wrench)

[0233] RTEn… tool handle

[0234] RTHn… tool head

[0235] ORTHn… tool head position (coordinates)

[0236] D1n… head thickness dimension

[0237] RTT… communication display section

[0238] RTSn… sleeve

[0239] D2n… sleeve length dimension

[0240] OR… sleeve outer diameter

[0241] RBn… bolt

[0242] ORBn… bolt position (coordinates)

[0243] a… head

[0244] b… cylindrical portion

[0245] c… threaded portion

[0246] t_B… fastening start time

[0247] Tr_B… fastening start torque

[0248] Tr_Min… fastening lower limit torque

[0249] Tr_Set… fastening target torque

[0250] Tr_Max… fastening upper limit torque

[0251] Tr_rslt… fastening completion torque

[0252] t_rslt… fastening completion time

[0253] TSP… tool reference surface

[0254] ITm… tool virtual body (cylinder)

[0255] OITm… tool virtual body representative point (coordinates)

[0256] IBn… bolt virtual body (cylinder)

[0257] OIBn... Bolt virtual body representative point (coordinates)

[0258] NV... (unit) direction vector

[0259] 5... Network

[0260] 7... Wearable device (headset)

[0261] 10... Camera

[0262] 12... Augmented reality server

[0263] 14... See-through screen (HMD)

[0264] 26... Network server.

Claims

1. A mechanical fastening operation method based on augmented reality, characterized in that: The physical space includes a mechanical fastener and a tool fitted onto the mechanical fastener. The virtual space includes a virtual body simulating the mechanical fastener and a virtual body simulating the tool. In the augmented reality space, The mechanical fasteners correspond one-to-one with the virtual mechanical fastener bodies, and Each tool corresponds one-to-one with its virtual counterpart. The augmented reality system that generates the augmented reality space has: A camera that captures the real space described; The server is configured to be connected to the camera, analyze the images captured by the camera, and generate augmented reality space; as well as The tool acquires tension information observed when the mechanical fastener is loosened or tightened, and sends and receives this tension information to the server. The mechanical fastening operation method includes the following steps: When the tool detects the tightness information of the mechanical fastener. A virtual entity of the mechanical fastener is identified that corresponds one-to-one with the mechanical fastener, and the identified virtual entity of the mechanical fastener is displayed in the virtual space. The augmented reality-based mechanical fastening operation method includes the following steps: A first work sheet is created, which is generated based on the tightening operation of the first operator and includes the position information and tightness information of the mechanical fastener. The location information of the first worksheet is projected onto the augmented reality space to teach the second worker the work content of the first worker.

2. The augmented reality-based mechanical fastening method according to claim 1, characterized in that, The tool includes a handle held by the operator, a tool head connected to the handle, and a sleeve portion assembled to the tool head. The server performs image analysis on the tool head captured by the camera, generates a tool reference plane containing a tool representative point representing the position of the tool head, and defines a direction vector originating from the tool representative point in the direction from which the camera visually observes the tool from the tool reference plane. The server specifies the tool dimensions representing the tool head and the sleeve portion. The server generates and displays a virtual tool with the tool's dimensions in the virtual space, starting from the tool's representative point and along the direction vector.

3. The augmented reality-based mechanical fastening method according to claim 2, characterized in that, The virtual mechanical fastener simulates a bolt having a top, a shaft connected to the top, and a threaded portion connected to the shaft. A representative point of the virtual mechanical fastener is located at the intersection of the connection surface between the top and the shaft and the axis of the shaft. The server generates and displays the virtual mechanical fastener in the virtual space by configuring the virtual mechanical fastener representative point at a position along the direction vector and at a distance corresponding to the tool size from the tool representative point.

4. The augmented reality-based mechanical fastening method according to claim 3, characterized in that, The tightness information is the tightening completion torque value between the lower tightening torque value and the upper tightening torque value.

5. The augmented reality-based mechanical fastening method according to claim 2, characterized in that, The server issues a warning when it detects that the rate of increase of the observed tightening torque of the mechanical fastener exceeds a predetermined rate of increase of the tightening torque.

6. The augmented reality-based mechanical fastening method according to claim 4, characterized in that, The first worksheet has: The lower limit torque value for fastening; and The upper limit torque value of the fastening. The augmented reality-based mechanical fastening operation method includes the following steps: When the server detects the tightness information, it records in the first work table the coordinates of the virtual body representing the mechanical fastener, the coordinates of the direction vector, the torque value of the fastening completion, and the time information related to the tightness of the mechanical fastener, and verifies the number of mechanical fasteners.

7. The augmented reality-based mechanical fastening method according to claim 6, characterized in that, The augmented reality-based mechanical fastening operation method includes the following steps: The server acquires and saves an image in the augmented reality space displaying the torque value of the fastening completion and the time information near the mechanical fastener.

8. The augmented reality-based mechanical fastening method according to claim 6, characterized in that, The server creates a second worksheet by removing the tightening completion torque value and the time information from the first worksheet, and displays the virtual mechanical fastener in the augmented reality space based on the position information of the virtual mechanical fastener recorded in the first worksheet, thereby teaching the second operator the position of the mechanical fastener and the order of tightening and loosening.

9. The augmented reality-based mechanical fastening method according to claim 8, characterized in that, In the augmented reality space, when the second operator uses the tool to loosen or tighten the mechanical fastener, the server determines that the loosened or tightened mechanical fastener corresponds one-to-one with the mechanical fastener virtual body by simultaneously using interference judgment to judge the interference between the virtual tool body and the virtual mechanical fastener body, and the loosening or tightening information.

10. The augmented reality-based mechanical fastening method according to claim 9, characterized in that, The interference judgment is the overlap judgment between the virtual tool body and the virtual mechanical fastener body in the virtual space.

11. The augmented reality-based mechanical fastening method according to claim 9, characterized in that, The interference determination is based on the distance between the virtual tool representative point and the virtual mechanical fastener representative point in the virtual space.

12. The augmented reality-based mechanical fastening method according to claim 8, characterized in that, The server records the tightening completion torque value and the time information related to the tightening and loosening of the mechanical fastener in the second job table, and verifies the quantity.

13. The augmented reality-based mechanical fastening method according to claim 12, characterized in that, The server acquires and saves an image in the augmented reality space displaying the torque value of the fastening completion and the time information near the mechanical fastener.

Images