Tool system, tool, fastening place estimation method and program

Abstract

To provide a tool system capable of estimating a next fastening place to be fastened from a picked-up image with high accuracy.SOLUTION: A tool system is provided with a tool 2. The tool 2 has a drive part, an imaging part and a reference detection part. The drive part drives a tip tool 242. The reference detection part detects a reference direction. The tip tool 242 can fasten fastening components arranged at a plurality of fastening places X0 of a work object A1. The imaging part is arranged so as to include a set fastening place Xs with the tip tool 242 set within an imaging range W1. The tool system is provided with an estimation part. The estimation part estimates a next fastening place X5 to be fastened after the set fastening place Xs by using an angle θ1 formed by a first straight line M1 and a second straight line M2 set on a picked-up image G1. The first straight line M1 is set in the picked-up image G1 on the basis of the reference direction. The second straight line M2 is a straight line passing through position coordinates P1 of a fastening place X1 and a central coordinate P0 of the work object A1.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure generally relates to a tool system, a tool, a fastening point estimation method, and a program, and more particularly to a tool system including a portable tool, a tool, a fastening point estimation method, and a program. [Background technology]

[0002] The tool system described in Patent Document 1 includes a portable tool. The tool includes an imaging unit, an attitude detection unit, and a control unit. The imaging unit is arranged so that a socket (tip tool) attached to the tool is included in the imaging range. The imaging unit captures an image of a work target (work object) on which the socket is set when working with the tool. The attitude detection unit includes an acceleration sensor, a gyro sensor, etc., and detects the attitude of the tool. The control unit identifies the work target shown in the captured image generated by the imaging unit based on the attitude of the tool detected by the attitude detection unit. Furthermore, the control unit stops the operation of the tool if the identified work target does not match a preset work order. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-074812 Summary of the Invention [Problem to be solved by the invention]

[0004] The tool system of Patent Document 1 is used, for example, when the work target is a flange, in a case where the fastening parts arranged at a plurality of fastening points provided on the flange are fastened according to a preset work sequence. For such a case, a technology has been proposed that uses AI (artificial intelligence) learning technology to estimate the next fastening point to be fastened from a captured image showing the set fastening point where the tool socket is set and a part of the flange.

[0005] When using this type of AI learning, it is assumed that the positional relationship between the set fastening point and the flange in the image is the same between the image used in the learning stage and the image (captured image) used in the prediction stage.

[0006] For this reason, if the positional relationship shown in the image used in the prediction stage changes compared to the positional relationship shown in the image used in the learning stage due to, for example, the flange rotating at a certain angle and changing the arrangement of the set fastening points, it becomes impossible to accurately estimate the fastening point to be fastened next from the captured image. In this case, re-learning is required to improve the estimation accuracy, which is time-consuming.

[0007] The present disclosure has been made in consideration of the above-mentioned reasons, and aims to provide a tool system, a tool, a fastening point estimation method, and a program that can accurately estimate the next fastening point to be fastened from an captured image even if the positional relationship between the set fastening point where the tool tip is set and the work object changes in the captured image. [Means for solving the problem]

[0008] A tool system according to an aspect of the present disclosure is a tool system including a portable tool. The tool includes a drive unit, an imaging unit, and a reference detection unit. The drive unit drives a tool tip. The imaging unit generates an image. The reference detection unit detects a reference direction. The tool tip is capable of tightening fastening parts arranged at a plurality of fastening points of a work object. The imaging unit is arranged so as to include within an imaging range a set fastening point at which the tool tip is set among the plurality of fastening points. The tool system further includes an estimation unit. The estimation unit estimates a fastening point to be tightened next to the set fastening point by using an angle formed by a first straight line and a second straight line set in the captured image. The first straight line is set in the captured image based on the reference direction detected by the reference detection unit. The second straight line is a straight line passing through a position coordinate of the set fastening point included in the captured image and a center coordinate of the work object depicted in the captured image.

[0009] According to an aspect of the present disclosure, there is provided a tool for use in the tool system, the tool including the drive unit, the imaging unit, and the reference detection unit.

[0010] A fastening point estimation method according to one aspect of the present disclosure is a fastening point estimation method using a portable tool. The tool has a drive unit, an imaging unit, and a reference detection unit. The drive unit drives a tool tip. The imaging unit generates an imaged image. The reference detection unit detects a reference direction. The tool tip is capable of fastening fastening parts arranged at a plurality of fastening points of a work object. The imaging unit is arranged so as to include within an imaging range a set fastening point at which the tool tip is set among the plurality of fastening points. The fastening point estimation method includes an estimation step. In the estimation step, a fastening point to be fastened next to the set fastening point is estimated using an angle formed by a first straight line and a second straight line set in the captured image. The first straight line is set in the captured image based on the reference direction detected by the reference detection unit. The second straight line is a straight line passing through a position coordinate of the set fastening point included in the captured image and a center coordinate of the work object depicted in the captured image.

[0011] A program according to one aspect of the present disclosure is a program for causing one or more processors to execute the above-described fastening portion estimating method. Effect of the Invention

[0012] The tool system, tool, fastening point estimation method, and program disclosed herein have the advantage of being able to accurately estimate the next fastening point to be fastened from the captured image even if the positional relationship between the set fastening point where the tool tip is set and the work object changes in the captured image. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of a tool system according to an embodiment. [Diagram 2]FIG. 2 is an external perspective view of a tool provided in the tool system. [Diagram 3] FIG. 3 is a plan view showing an example of a work object and an example of a captured image. [Figure 4] FIG. 4 is an explanatory diagram for explaining how to determine the center coordinates of a work object from three points on the contour of the work object. [Diagram 5] FIG. 5 is a sequence diagram showing an operation of the tool system. [Figure 6] FIG. 6 is an explanatory diagram for explaining a method for determining the central coordinates of a work object in the second modification. [Figure 7] FIG. 7 is an explanatory diagram for explaining a method for determining the central coordinates of a work object in the third modification. [Figure 8] FIG. 8 is a comparative diagram comparing a captured image G2 in the case where a tip tool having a length Y1 is used with a captured image G3 in the case where a tip tool having a length Y2 is used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (Embodiment) (1) Overview An overview of a tool system 1 according to this embodiment will be described with reference to FIGS. 1 and 3. FIG.

[0015] As shown in FIG. 1, the tool system 1 according to the present embodiment is a tool system including a portable tool 2. The tool 2 includes a driving unit 24, an imaging unit 26, and a reference detection unit 28. The driving unit 24 drives a tip tool 242 (see FIG. 2). The imaging unit 26 generates an image G1 (see FIG. 3). The reference detection unit 28 detects a reference direction. The tip tool 242 can tighten fastening parts arranged at a plurality of fastening points X0 of the work target A1 (see FIG. 3). The imaging unit 26 is arranged so as to include within its imaging range a set fastening point Xs (Xs=X1 in FIG. 3) where the tip tool 242 is set among the plurality of fastening points X0. The tool system 1 includes an estimation unit 355. The estimation unit 355 estimates the fastening point X0 (fastening point X5 in FIG. 3) to be fastened next to the set fastening point Xs, using the angle θ1 formed by the first straight line M1 and the second straight line M2 set in the captured image G1. The first straight line M1 is set in the captured image G1 based on a reference direction (e.g., the direction of gravity) detected by the reference detection unit 28. The second straight line M2 is a straight line passing through the position coordinate P1 of the set fastening point Xs included in the captured image G1 and the center coordinate P0 of the work target A1 shown in the captured image G1.

[0016] According to this configuration, the fastening point X0 to be fastened next to the set fastening point Xs is estimated using the angle θ1 formed by the first straight line M1 and the second straight line M2 set in the captured image G1. Therefore, even if the relative positional relationship between the set fastening point Xs and the work object A1 changes in the captured image G1, the fastening point X0 to be fastened next to the set fastening point Xs can be accurately estimated from the captured image G1.

[0017] (2)Details Hereinafter, a detailed configuration of the tool system 1 according to this embodiment will be described with reference to Figs. 1 to 5. The tool system 1 according to this embodiment is used for tightening work of a plurality of tightening points X0 of a work object A1, for example, outdoors. The tightening points X0 are points tightened by tightening parts (for example, bolts or nuts) arranged at the tightening points X0. In this embodiment, it is assumed that the work object A1 is, for example, a pipe flange. It is also assumed that a bolt is arranged as a tightening part at each of the plurality of tightening points X0. That is, in this embodiment, it is assumed that a worker uses a tool 2 to tighten a plurality of bolts in order to attach a flange to a mating member.

[0018] (2.1) Tool system configuration As shown in FIG. 1, a tool system 1 of the present embodiment includes a tool 2 and a determination device 3.

[0019] (2.2) Tool configuration As shown in Fig. 1, the tool 2 of this embodiment is, for example, a portable impact wrench. The tool 2 includes a drive unit 24, a communication unit 25, an imaging unit 26, a processing unit 27, a reference detection unit 28, a display unit 211, and an operation unit 231 (input unit). As shown in Fig. 2, the tool 2 further includes a body 20 and a battery pack 201.

[0020] As shown in FIG. 2, the body 20 of the tool 2 has a trunk portion 21, a grip portion 22, and a mounting portion .

[0021] Body section 21 is formed in a tubular shape (cylindrical shape here). Grip section 22 protrudes in one direction (downward in FIG. 2 ) from a part of the circumferential surface of body section 21. Mounting section 23 is provided so that battery pack 201 can be removably mounted thereon. In other words, body section 21 and mounting section 23 are connected by grip section 22.

[0022] The body portion 21 accommodates a part (a part other than the output shaft portion 241) of the drive portion 24 (see FIG. 1) and the imaging portion 26. The output shaft portion 241 of the drive portion 24 protrudes from one end surface 21a of the body portion 21 in the axial direction.

[0023] The grip portion 22 is a portion that an operator grips when performing work. The grip portion 22 is provided with a trigger switch 221. The trigger switch 221 is a switch for controlling the on / off operation of the drive portion 24. The trigger switch 221 has an initial position and an on position, and the drive portion 24 operates when the operator presses the trigger switch 221 to the on position. In addition, the trigger switch 221 can adjust the rotation speed of the drive portion 24 according to the amount of pressing (amount of operation).

[0024] The mounting section 23 is formed in a flat rectangular parallelepiped shape. A battery pack 201 is removably mounted on one surface of the mounting section 23 opposite to the grip section 22.

[0025] The battery pack 201 is configured with, for example, a chargeable and dischargeable secondary battery (for example, a lithium ion battery, etc.) The battery pack 201 supplies power to the driving unit 24, the communication unit 25, the imaging unit 26, the processing unit 27, etc.

[0026] The mounting unit 23 is also provided with an operation unit 231. The operation unit 231 can be used to perform various settings and status confirmation related to the tool 2. That is, the worker can operate the operation unit 231 to check the remaining capacity of the battery pack 201, for example.

[0027] The display unit 211 displays, for example, the operating status of the tool 2. The display unit 211 is configured with, for example, an LED (Light Emitting Diode). The display unit 211 is provided at an end (i.e., a rear end) of the trunk portion 21 of the body 20 opposite to the output shaft portion 241 so that the operator can easily view the display unit 211 during work.

[0028] The drive unit 24 is configured to operate a tip tool 242 (e.g., a socket) with a motor in the drive unit 24 using power supplied from the battery pack 201. The drive unit 24 rotates the tip tool 242 using power from the motor to tighten a fastening part disposed at a tightening point X0 (see FIG. 3). The drive unit 24 has a motor, a reduction mechanism, an impact mechanism, an output shaft 241, and the tip tool 242.

[0029] The motor is a drive source for rotating the tool tip 242. The motor is driven by power from the battery pack 201.

[0030] The reduction mechanism reduces the rotation speed of the motor's rotating shaft and transmits the rotational force transmitted to the reduced rotation shaft to the output shaft 241. The reduction mechanism is, for example, a planetary gear mechanism, and converts the rotation speed and torque of the motor's rotating shaft into the rotation speed and torque required for the fastening operation. The "fastening operation" is an operation in which the tool 242 uses the driving force of the drive unit 24 to fasten a fastening part (for example, a bolt) arranged at one of the multiple fastening points X0.

[0031] The tip tool 242 is removably attached to the tip of the output shaft portion 241 and rotates together with the output shaft portion 241 to tighten a fastening part arranged at the fastening location X0 of the work object A1.

[0032] The output shaft portion 241 transmits the rotational force transmitted from the reduction mechanism to the tool tip 242. The output shaft portion 241 rotates about a rotational axis Ax1 along the protruding direction of the output shaft portion 241. That is, the drive portion 24 rotates the output shaft portion 241 about the rotational axis Ax1 using the rotational force of the motor. In other words, the torque output from the rotating shaft of the motor acts on the output shaft portion 241 via the reduction mechanism, causing the output shaft portion 241 to rotate.

[0033] A tip tool 242 for fastening a fastening part arranged at a fastening point X0 is detachably attached to the tip of the output shaft 241. The tip tool 242 is, for example, a socket for fastening a fastening part (for example, a bolt) arranged at the fastening point X0. The tip tool 242 rotates around the rotation axis Ax1 together with the output shaft 241. The size of the tip tool 242 attached to the output shaft 241 is appropriately selected by the operator according to the size of the fastening part. With this configuration, when the drive unit 24 operates, the output shaft 241 rotates and the tip tool 242 rotates together with the output shaft 241. At this time, if the tip tool 242 is fitted to the fastening part arranged at the fastening point X0, the fastening part rotates together with the tip tool 242, and the operation of fastening the fastening part (fastening operation) is realized. Therefore, the tool 2 can realize the operation of fastening the fastening part arranged at the fastening point X0 by the operation of the drive unit 24.

[0034] Moreover, a socket anvil can be attached to the output shaft portion 241 instead of a socket as the tip tool 242. The socket anvil is also detachably attached to the output shaft portion 241. In this case, a bit (for example, a driver bit or a drill bit) can be attached via the socket anvil.

[0035] The impact mechanism is a mechanism for applying an impact (shock) in the rotation direction of the output shaft portion 241 in response to the rotation of the motor. The impact mechanism is driven by the power of the motor. The impact mechanism includes a hammer rotatably supported by the rotation shaft of the motor, an anvil (striking portion) provided at the rear end of the output shaft portion 241, and the like. The hammer strikes the anvil in response to the rotation of the rotation shaft of the motor.

[0036] The impact mechanism applies an impact to the output shaft 241 in the rotational direction when the tightening torque (work value) of the tool tip 242 exceeds a predetermined level. This enables the tool 2 to apply a larger tightening torque to the tightening location X0 tightened by the tool tip 242. Note that it is not essential that the tool tip 242 and the impact mechanism are included in the components of the drive unit 24, and the components of the drive unit 24 do not necessarily need to include the tool tip 242 and the impact mechanism.

[0037] The imaging unit 26 generates data as a captured image. Here, the "captured image" is an image generated by the imaging unit 26, and includes a still image (still image) and a video (moving image). Furthermore, the "video" includes an image composed of a plurality of still images (frames) obtained by frame-by-frame shooting or the like. The captured image does not have to be the data itself output from the imaging unit 26. For example, the captured image may be appropriately compressed as necessary, converted into another data format, or processed by cutting out a part of the image captured by the imaging unit 26, adjusting the focus, adjusting the brightness, adjusting the contrast, or the like. In this embodiment, the captured image is, for example, a full-color video.

[0038] The imaging unit 26 is, for example, a camera having an imaging element and a lens. In this embodiment, as described above, the imaging unit 26 is housed in the body 20 (trunk portion 21) of the tool 2. The imaging unit 26 is mounted toward the tip side of the output shaft portion 241 so that the imaging range includes the set fastening position Xs where the tip tool 242 is set during work using the tool 2.

[0039] Specifically, the imaging unit 26 is disposed at the tip of the body 21 toward the tip side (the tip tool 242) of the output shaft 241 so that the tip tool 242 attached to the output shaft 241 falls within the imaging range (see FIG. 2). The optical axis of the imaging unit 26 is disposed along the rotation axis Ax1 of the output shaft 241. Here, the imaging unit 26 is disposed so that the optical axis is located within a predetermined distance from the rotation axis Ax1 of the output shaft 241 and is substantially parallel to the rotation axis Ax1. Note that it is not essential for the imaging unit 26 to generate an image so that the tip tool 242 attached to the output shaft 241 falls within the imaging range. It is sufficient for the imaging unit 26 to be able to generate an image for identifying the set fastening location Xs.

[0040] In the present disclosure, the "set fastening point" refers to the fastening point X0 in a state where the tool 2 (more specifically, the tip tool 242) is set among the multiple fastening points X0. "The state where the tool 2 is set at the fastening point X0" means a state where the tool 2 is prepared to perform work on the fastening point X0. In addition, "the state where the tool 2 is set at the fastening point X0" includes not only a state where the tool 2 is in contact with the fastening point X0, but also a state where the tool 2 is about to be in contact with the fastening point X0, that is, a state where the tool 2 is approaching the fastening point X0. In other words, when the tool 2 is set at the fastening point X0, the tool 2 may be in contact with the fastening point X0 or may be away from the fastening point X0.

[0041] In addition, "the imaging unit 26 is mounted so as to include the set fastening point Xs within the imaging range" does not mean limited to the case where the set fastening point Xs is directly captured within the imaging range, but also includes the case where the set fastening point Xs exists within the imaging range but is obscured by the tip tool 242 when the tip tool 242 is set.

[0042] Further, the "captured image for identifying the set fastening point Xs" is an image of the work object A1 (see FIG. 3) in a state where the tool 2 is set at the fastening point X0, generated by the imaging unit 26. The imaging range of the imaging unit 26 in this disclosure is assumed to include the set fastening point Xs where the tool 2 is set. Note that the above-mentioned captured image may be an image that allows the set fastening point Xs to be identified, and the set fastening point Xs does not have to be included in the imaging range of the above-mentioned captured image.

[0043] Since the imaging unit 26 is provided on the tool 2, the fastening point X0 can be imaged at a relatively close distance, which reduces the problem that the fastening point X0 is hidden by the tool or the worker and cannot be properly imaged.

[0044] The reference detection unit 28 detects a reference direction. The reference direction is used to identify the set tightening point Xs included in the captured image of the imaging unit 26, or to estimate the tightening point X0 to be tightened next to the set tightening point Xs included in the captured image. The reference detection unit 28 includes, for example, a gravity sensor, and detects the direction of gravity as the reference direction. That is, the reference detection unit 28 detects the direction of gravity using the gravity sensor, and outputs detection information including information (reference information) related to the detected direction of gravity (i.e., the reference direction) to the processing unit 27. By detecting the reference direction, it is possible to identify the inclination of the captured image with respect to the reference direction (i.e., to set the reference direction in the captured image).

[0045] The communication unit 25 includes a communication interface for wireless communication with the determination device 3. The communication unit 25 communicates with the determination device 3 by a wireless communication method conforming to standards such as Wi-Fi (registered trademark), Bluetooth (registered trademark), ZigBee (registered trademark), or low-power radio (specified low-power radio) that does not require a license.

[0046] The processing unit 27 mainly comprises, for example, a computer system having one or more processors and one or more memories. In the tool 2, the one or more processors execute a program recorded in the memory, thereby realizing the function of the processing unit 27. The program may be pre-recorded in the memory, may be provided via a telecommunication line such as the Internet, or may be provided by recording it on a non-transitory recording medium such as a memory card.

[0047] The processing unit 27 outputs control signals to each of the driving unit 24, the communication unit 25, the imaging unit 26, and the display unit 211, thereby controlling the driving unit 24, the communication unit 25, the imaging unit 26, and the display unit 211 individually.

[0048] The processing unit 27 transmits the captured image generated by the imaging unit 26, the detection information of the reference detection unit 28, and the tightening completion information, etc. to the determination device 3 via the communication unit 25. The tightening completion information includes information indicating that tightening of the tightening point X0 has been completed, and tightening torque information, etc. The processing unit 27 transmits the tightening completion information to the determination device 3 via the communication unit 25 every time the drive unit 24 performs a tightening operation.

[0049] Furthermore, the processing unit 27 receives a permission instruction or a restriction instruction for the tightening operation from the determination device 3 via the communication unit 25. When the processing unit 27 receives a permission instruction from the determination device 3, the processing unit 27 permits the tightening operation by the drive unit 24. That is, when the processing unit 27 receives a permission instruction from the determination device 3, the processing unit 27 operates the drive unit 24 in response to the operation of the trigger switch 221 by the operator. On the other hand, when the processing unit 27 receives a restriction instruction from the determination device 3, the processing unit 27 restricts the tightening operation by the drive unit 24. That is, when the processing unit 27 receives a restriction instruction from the determination device 3, the processing unit 27 does not operate the drive unit 24 even if the trigger switch 221 is pulled by the operator.

[0050] Furthermore, the processing unit 27 controls the display unit 211 in response to an instruction (permission instruction or restriction instruction) received from the determination device 3. For example, when the processing unit 27 receives a permission instruction from the determination device 3, it lights up the display unit 211 in green. Furthermore, when the processing unit 27 receives a restriction instruction from the determination device 3, it lights up the display unit 211 in red. By visually checking the lighting state of the display unit 211, the worker can recognize whether or not the fastening sequence of the fastening point X0 is correct. Note that the processing unit 27 may light up the display unit 211 when the trigger switch 221 is pulled in a state in which a restriction instruction has been received from the determination device 3.

[0051] (2.3) Judgment device The determination device 3 is embodied as a dedicated device, a personal computer, a server, a tablet terminal, a smartphone, or the like. In this embodiment, it is assumed that the determination device 3 is a tablet terminal. That is, the determination device 3 of this embodiment is configured to be portable by the worker.

[0052] As shown in FIG. 1, the determination device 3 includes a communication unit 31, a display unit 32, an operation unit 33 (input unit), a storage unit 34, and a processing unit 35.

[0053] The communication unit 31 includes a communication interface for wireless communication with the tool 2. The communication unit 31 communicates with the tool 2 by a wireless communication method conforming to standards such as Wi-Fi (registered trademark), Bluetooth (registered trademark), ZigBee (registered trademark), or low-power radio (specified low-power radio) that does not require a license.

[0054] The display unit 32 includes, for example, an LCD (Liquid Crystal Display). The display unit 32 displays an input screen. The input screen is a screen that prompts the worker to input basic information used to identify the set fastening point Xs included in the image captured by the imaging unit 26 and to estimate the fastening point X0 to be fastened after the first set fastening point Xs included in the image captured by the imaging unit 26. The input screen includes a display for prompting the worker to input, as the basic information, the identification information of the work object A1, the type of the work object A1, the type of the multiple fastening points X0, the number of the multiple fastening points X0, the arrangement of the multiple fastening points X0 (i.e., the positions of the multiple fastening points X0), the fastening rule, and the position coordinates of the set fastening points Xs.

[0055] Here, the "identification information of the work object A1" is information that can identify one work object A1 from various work objects A1. The "identification information of the work object A1" is, for example, a model number. The "type of the work object A1" is information that indicates whether the work object A1 is a round flange, a square flange, a wheel, or the like. The "type of the fastening point X0" is information that indicates whether the fastening point X0 is fastened by a bolt, a nut, a screw, or the like. The "type of the fastening point X0" also includes information on the size (or standard size) of the fastening point X0. The "arrangement of the multiple fastening points X0" includes an arrangement along a circular arc or an arrangement along a straight line. The "fastening rule" includes, for example, diagonal fastening, clockwise fastening, and counterclockwise fastening. The "position coordinates of the set fastening point Xs" are set values ​​in the individual coordinate system of the captured image, as described later. In this embodiment, the length of the tip tool 242 is fixed. In this case, the position of the set tightening point Xs in each captured image is the same in each captured image. Therefore, the setting value of the position coordinate of the set tightening point Xs can be input as a fixed value as basic information.

[0056] The operation unit 33 receives input of the basic information by receiving an operation from the worker. The operation unit 33 receives the operation from the worker and outputs a signal according to the operation from the worker. In this embodiment, it is assumed that the determination device 3 includes a touch panel display in which the display unit 32 and the operation unit 33 are integrated. In the case of a touch panel display, the determination device 3 determines that an object such as a button has been operated when the operation unit 33 detects an operation (tap, swipe, drag, etc.) of an object such as a button on each screen displayed on the display unit 32.

[0057] The storage unit 34 includes, for example, a semiconductor memory such as a Read Only Memory (ROM), a Random Access Memory (RAM), or an Electrically Erasable Programmable Read Only Memory (EEPROM). The storage unit 34 is not limited to a semiconductor memory, and may include a hard disk drive or the like. The storage unit 34 stores programs necessary for the operation of the determination device 3, information used in each process of the determination device 3, and the above basic information input via an operation on the operation unit 33.

[0058] The processing unit 35 mainly comprises, for example, a computer system having one or more processors and one or more memories. In the determination device 3, the one or more processors execute a program recorded in the memory, thereby realizing the function of the processing unit 35. The program may be pre-recorded in the memory, may be provided via a telecommunication line such as the Internet, or may be provided by recording it on a non-transitory recording medium such as a memory card.

[0059] As shown in FIG. 1, the processing unit 35 includes an acquisition unit 351, an operation reception unit 352, a correction unit 353, an identification unit 354, an estimation unit 355, and an order determination unit 356.

[0060] The acquisition unit 351 receives the above-mentioned detection information (that is, the detection information of the reference detection unit 28) from the tool 2 via the communication unit 31.

[0061] The operation reception unit 352 receives an operation by the operator on the operation unit 33. Note that the operation reception unit 352 may receive an operation by the operator on the operation unit 231 of the tool 2.

[0062] The correction unit 353 corrects distortion of the captured image generated by the imaging unit 26. The distortion of the captured image is, for example, distortion caused by the lens of the imaging unit 26. The imaging unit 26 uses, for example, a fisheye lens as the lens. When the lens is a fisheye lens, the captured image of the imaging unit 26 has a characteristic that the imaging range is wide but is significantly distorted into a barrel shape toward the end of the imaging range. In this embodiment, the correction unit 353 corrects distortion of the captured image caused by the fisheye lens, for example.

[0063] The identification unit 354 identifies the set fastening point Xs included in the captured image corrected by the correction unit 353 (i.e., the captured image generated by the imaging unit 26). That is, the identification unit 354 identifies which of the multiple fastening points X0 of the work target object A1 included in the captured image corresponds to the set fastening point Xs included in the captured image.

[0064] More specifically, the identification unit 354 identifies the position coordinates of the set fastening point Xs included in the captured image based on the above detection information (i.e., reference information regarding the reference direction) of the reference detection unit 28, and thereby identifies which of the multiple fastening points X0 of the work object A1 captured in the captured image the set fastening point Xs (Xs = X1 in Figure 3) is.

[0065] More specifically, the identification unit 354 identifies which of the multiple fastening points X0 the set fastening point Xs included in the captured image is, using the angle θ1 formed by the first line M1 and the second line M2 set in the captured image. The first line M1 is a line set in the captured image based on the reference direction (gravity direction) detected by the reference detection unit 28. The second line M2 is a line passing through the position coordinate P1 of the set fastening point Xs included in the captured image and the center coordinate P0 of the work target A1 shown in the captured image.

[0066] Details of the processing of the identification unit 354 will be described later. In the following description, the set fastening point Xs identified by the identification unit 354 may be referred to as a "specific fastening point Xs."

[0067] The estimation unit 355 estimates the order (fastening sequence) of fastening parts arranged at multiple fastening points X0 of the work target object A1 captured in the image captured by the imaging unit 26. Hereinafter, "fastening the fastening parts arranged at the fastening points X0" may be simply described as "fastening the fastening point X0".

[0068] More specifically, when the first (first) specific tightening point Xs is identified by the identification unit 354, the estimation unit 355 sets the tightening order of the identified first specific tightening point Xs to 1 (tightening order 1). Then, the estimation unit 355 estimates each tightening point X0 (more specifically, the position coordinates of each tightening point X0) that is tightened second or later, according to the detection information and the basic information.

[0069] More specifically, the estimation unit 355 estimates the tightening point X0 (i.e., the tightening point X0 of the tightening order No. 2) (the tightening point X5 in the example of FIG. 3) to be tightened next to the first specific tightening point Xs included in the captured image G1 by using the angle θ1 formed by the first straight line M1 and the second straight line M2 set in the captured image G1 (see FIG. 3). The first straight line M1 is a straight line set in the captured image G1 based on the reference direction (gravity direction) detected by the reference detection unit 28. Note that the second straight line M2 is a straight line passing through the position coordinate P1 of the first specific tightening point Xs (X1) included in the captured image G1 and the center coordinate P0 of the work target A1 shown in the captured image G1. When the tightening rule is diagonal tightening, the position coordinate P2 of the tightening point X5 of the tightening order No. 2 is the position coordinate of the tightening point X0 located in the diagonal direction of the position coordinate P1 of the specific tightening point Xs of the tightening order No. 1. Then, the estimation unit 355 estimates the position coordinates of each fastening point X0 in the fastening order from No. 3 onwards in accordance with the above fastening rule. The processing of the estimation unit 355 will be described later.

[0070] Each time the identification unit 354 identifies a specific fastening point Xs, the order determination unit 356 determines whether or not the fastening order of the identified specific fastening point Xs is correct based on the estimated order estimated by the estimation unit 355. More specifically, when the identification unit 354 first identifies a specific fastening point (fastening point number 1 in the fastening order) Xs, the order determination unit 356 does not determine whether or not the fastening order of the identified specific fastening point Xs is correct. In other words, the worker can freely select from which of the multiple fastening points X0 to start fastening.

[0071] Then, the order determination unit 356 compares the position coordinates of the specific tightening point Xs identified by the identification unit 354 as the Nth (N: an integer of 2 or more) and the position coordinates of the tightening point X0 of the tightening order Nth estimated by the estimation unit 355. If the position coordinates match as a result of the comparison, the order determination unit 356 determines that the tightening order of the specific tightening point Xs identified by the identification unit 354 as the Nth is correct. On the other hand, if the position coordinates do not match, the order determination unit 356 determines that the tightening order of the specific tightening point Xs identified by the identification unit 354 as the Nth is incorrect.

[0072] Note that the above "match" is not limited to a strict match. For example, if the specific tightening point Xs identified by the identification unit 354 at the Nth position and the tightening point X0 of the tightening sequence N estimated by the estimation unit 355 are the same, they are determined to be the same, and if they are not the same, an error may be allowed such that they are determined to be not the same. For example, in FIG. 3, if the specific tightening point Xs is X1, and the calculated angle θ1 is within the range of -(45° / 2)<θ1<+(45° / 2), the specific tightening point Xs is identified as X1. Note that 45° (=360° / (number of X0)) is the angle between adjacent X0.

[0073] When the order determination unit 356 determines that the fastening order of the specific fastening point Xs is correct, it transmits an instruction to permit the fastening operation to the tool 2 via the communication unit 31. As a result, when the order determination unit 356 determines that the fastening order of the specific fastening point Xs is correct, the fastening operation of the drive unit 24 of the tool 2 is permitted, and the specific fastening point Xs is fastened by the tool 2. On the other hand, when the order determination unit 356 determines that the fastening order of the specific fastening point Xs is incorrect, that is, the fastening order is incorrect, it transmits an instruction to restrict the fastening operation to the tool 2 via the communication unit 31. As a result, when the order determination unit 356 determines that the fastening order of the specific fastening point Xs is incorrect, the fastening operation of the tool 2 is restricted, and the specific fastening point Xs is not fastened by the tool 2.

[0074] (2.4) Details of the processing by the identification unit and the estimation unit The details of the processing of the identification unit 354 and the estimation unit 355 will be described with reference to FIG. 3. FIG. 3 illustrates an example in which the work object A1 is a round flange. FIG. 3 also illustrates an example in which the multiple fastening points X0 are eight fastening points X0, and hexagonal bolts are arranged as fastening parts at each of the fastening points X1 to X8. The eight fastening points X0 are arranged at equal intervals in an arc along the outer periphery of the work object A1, which is a round flange. That is, the distance between each position of the eight fastening points X0 and the center of the work object A1 is the same as each other. As shown in FIG. 3, the eight fastening points X0 include fastening point X1, fastening point X2, fastening point X3, fastening point X4, fastening point X5, fastening point X6, fastening point X7, and fastening point X8. The tightening point X1, the tightening point X2, the tightening point X3, the tightening point X4, the tightening point X5, the tightening point X6, the tightening point X7, and the tightening point X8 are arranged in this order clockwise. In the following description, when there is no need to distinguish between the multiple tightening points X0, they may simply be referred to as "tightening points X0."

[0075] Consider a case where an operator uses the tool 2 to first (first) tighten a fastening part arranged at a fastening point X1 among a plurality of fastening points X0. In this case, the imaging unit 26 of the tool 2 generates a captured image G1 of an imaging range W1 (see FIG. 3). The captured image G1 includes the set fastening point Xs where the tip tool 242 is set, and the work target A1 is also shown. The captured image G1 of the imaging unit 26 is an image corrected by the correction unit 353.

[0076] In this embodiment, the set fastening point Xs is located at the tip of the tip tool 242, and the length of the tip tool 242 is fixed. Therefore, the position of the set fastening point Xs (Xs=X1 in the example of FIG. 3) in the captured image G1 is the same in all captured images G1. Therefore, the position coordinates of the set fastening point Xs in the captured image G1 are preliminarily initialized as the same set value using the individual coordinate system set in the captured image G1. Note that the individual coordinate system of each captured image G1 is a coordinate system having an individual origin and individual coordinate axes. The individual coordinate system of each captured image G1 has an origin at the same position in the captured image G1, a vertical axis in the same direction, and a horizontal axis in the same direction. Therefore, the set values ​​of the set fastening point Xs in the individual coordinate system in each captured image G1 are the same value. Note that this set value can be input from the operation unit 33 or the operation unit 231 as basic information.

[0077] As shown in FIG. 3, the identification unit 354 identifies an arc-shaped contour R1 of the outer periphery of the work object A1 (round flange) captured in the captured image G1 by image recognition processing.

[0078] The identification unit 354 then determines the center coordinate P0 of the work target A1 based on the identified contour R1. More specifically, as shown in Fig. 4, the identification unit 354 arbitrarily selects three points Q1 to Q3 from among a plurality of points included in the contour R1 of the outer shape of the work target A1 in the captured image G1. The identification unit 354 then sets a perpendicular bisector L1 of a line segment connecting points Q1 and Q2, and a perpendicular bisector L2 of a line segment connecting points Q2 and Q3 in the captured image G1. The identification unit 354 then determines the position coordinates of the intersection of the perpendicular bisector L1 and the perpendicular bisector L2 as the center coordinate P0 of the work target A1.

[0079] As shown in FIG. 3, the determination unit 354 determines the reference direction in the captured image G1 based on the detection information of the reference detection unit 28 (information related to the reference direction (e.g., the direction of gravity)). Then, the determination unit 354 sets a first straight line M1 in the captured image G1 based on the determined reference direction. The first straight line M1 is a straight line that passes through the center coordinate P0 and is parallel to the reference direction in the captured image G1. Then, the determination unit 354 determines the set fastening location Xs using the center coordinate P0 and the first straight line M1. That is, the determination unit 354 obtains the position coordinate P1 of the set fastening location Xs using an angle in a predetermined rotation (e.g., clockwise) from the first straight line M1 with the center coordinate P0 as the center.

[0080] More specifically, the identification unit 353 obtains a second straight line M2 passing through the position coordinate P1 of the specific fastening point Xs (Xs=X1 in FIG. 3) and the center coordinate P0 of the work target A1. Then, the identification unit 354 obtains an angle θ1 formed by the first straight line M1 and the second straight line M2. More specifically, the identification unit 354 determines, in the captured image G1, a range above the center coordinate P0 on the first straight line M1 (vertical axis) as a positive range M1a, and a range below the center coordinate P0 on the first straight line M1 as a negative range M1b. Then, the identification unit 354 obtains an angle from the positive range M1a of the first straight line M1 to the second straight line M2 in a predetermined rotation (for example, clockwise) as the angle θ1 formed by the first straight line M1 and the second straight line M2. This angle θ1 becomes the angle (position coordinate) for identifying the set fastening point Xs.

[0081] The distance between the set fastening point Xs and the central coordinate P0 is the same regardless of which fastening point X0 (X1 to X8) the set fastening point Xs is. Therefore, as described above, if the angle θ1 of the set fastening point X0 is obtained, it is possible to practically specify the set fastening point Xs. In this way, the specification unit 354 converts the position coordinate (position coordinate in the individual coordinate system) P1 of the preset set fastening point Xs (Xs=X1 in the example of FIG. 3) into the position coordinate P1 in the coordinate system using the angle θ1. In this way, the specification unit 354 specifies the set fastening point Xs by obtaining the position coordinate P1 of the set fastening point Xs using the angle θ1. That is, the specification unit 354 specifies which fastening point X0 of the multiple fastening points X0 the set fastening point Xs is by obtaining the position coordinate P1 of the set fastening point Xs using the angle θ1. Hereinafter, the specified set fastening point Xs may be referred to as the "specified fastening point Xs".

[0082] In this way, the identification unit 354 identifies the set fastening point Xs included in the captured image G1. The identification unit 354 identifies the set fastening point Xs by performing the same process even when a fastening point X0 other than the fastening point X1 is included as the set fastening point Xs in the captured image G1 of the imaging unit 26. That is, the identification unit 354 identifies the second or subsequent set fastening points Xs by performing the same process as described above every time the worker uses a tool to perform the fastening work of the second or subsequent fastening points X0.

[0083] The estimation unit 355 obtains a second straight line M2 passing through the position coordinate P1 of the specific fastening point Xs (Xs=X1 in FIG. 3) and the center coordinate P0 of the work target A1. Then, the estimation unit 355 obtains an angle θ1 formed by the first straight line M1 and the second straight line M2. More specifically, the estimation unit 355 determines, in the captured image G1, a range above the center coordinate P0 on the first straight line M1 (vertical axis) as a positive range M1a, and a range below the center coordinate P0 on the first straight line M1 as a negative range M1b. Then, the estimation unit 355 obtains an angle from the positive range M1a of the first straight line M1 to the second straight line M2 in a predetermined rotation (for example, clockwise) as the angle θ1 formed by the first straight line M1 and the second straight line M2. The process for determining this angle θ1 is the same as the process for determining the angle θ1 in the determination unit 354, and therefore the results of the process for determining the angle θ1 in the determination unit 354 may be used.

[0084] Then, the estimation unit 355 estimates the position coordinate P2 of the fastening point X0 of the fastening order No. 2, which is fastened next to the specific fastening point Xs of the fastening order No. 1, according to a preset fastening rule (for example, diagonal fastening), using the information of the angle θ1. More specifically, when the fastening rule is diagonal fastening, the fastening point X0 of the fastening order No. 2 is the fastening point X5 located diagonally to the specific fastening point Xs of the fastening order No. 1. Therefore, the estimation unit 355 estimates the position coordinate of the fastening point X5 as the position coordinate P2 using the information of the angle θ1. More specifically, the estimation unit 355 estimates the position coordinate of the fastening point X5 located in the rotation direction rotated by the angle θ2 from the range M1a on the positive side of the first straight line M1 and at a distance T1 from the center coordinate P0 as the position coordinate P2 of the fastening point X5. Here, the angle θ2 is an angle obtained by adding 180 degrees to the angle θ1. Further, the distance T1 is the same as the distance T1 between the position coordinate P1 and the center coordinate P0 of the specific tightening point Xs.

[0085] Furthermore, the estimation unit 355 estimates the position coordinates P3 to P8 of each of the fastening points X0 in the fastening order No. 3 to No. 8. More specifically, when the fastening rule is diagonal fastening, the fastening points X0 in the fastening order No. 3 to No. 8 are, for example, in the order of fastening point X3, fastening point X7, fastening point X2, fastening point X6, fastening point X4, and fastening point X8. The angle difference around the center coordinate P0 between two adjacent fastening points X0 is obtained by dividing 360 degrees by the number of fastening points X0 (8).

[0086] In this case, the position coordinate P3 of the fastening point X0, which is number 3 in the fastening sequence, is the position coordinate of the fastening point X3, more specifically, the position coordinate of a position located in a rotational direction rotated by an angle obtained by adding twice the above-mentioned angle difference to the angle θ1 from the positive range M1a of the first straight line M1, and at a distance T1 from the center coordinate P0.

[0087] In addition, the position coordinate P4 of the fastening point X0, which is number 4 in the fastening order, is the position coordinate of the fastening point X7, which is diagonally opposite to the fastening point X3, which is number 3 in the fastening order, and more specifically, is the position coordinate obtained by rotating the position coordinate P3 of the fastening point X3, which is number 3 in the fastening order, 180 degrees around the center coordinate P0.

[0088] Further, the position coordinate P5 of the fastening point X0, which is number 5 in the fastening sequence, is the position coordinate of the fastening point X2, more specifically, is the position coordinate of a position located in a rotational direction rotated by an angle obtained by adding the above-mentioned angle difference to the angle θ1 from the positive range M1a of the first straight line M1, and is a distance T1 away from the center coordinate P0.

[0089] In addition, the position coordinate P6 of the fastening point X0, which is the sixth fastening point in the fastening order, is the position coordinate of the fastening point X6, which is diagonally opposite to the fastening point X2, which is the fifth fastening point in the fastening order, and more specifically, is the position coordinate obtained by rotating the position coordinate P5 of the fastening point X2, which is the fifth fastening point in the fastening order, 180 degrees around the center coordinate P0.

[0090] Further, the position coordinate P7 of the fastening point X0 of the fastening sequence number 7 is the position coordinate of the fastening point X4, more specifically, the position coordinate of a position located in a rotational direction rotated by an angle obtained by adding the angle θ1 to three times the above-mentioned angle difference from the positive range M1a of the first straight line M1, and at a distance T1 from the center coordinate P0.

[0091] In addition, the position coordinate P8 of the fastening point X0, which is the 8th fastening point in the fastening order, is the position coordinate of the fastening point X8, which is diagonally opposite to the fastening point X7, which is the 7th fastening point in the fastening order, and more specifically, is the position coordinate obtained by rotating the position coordinate P7 of the fastening point X4, which is the 7th fastening point in the fastening order, 180 degrees around the center coordinate P0.

[0092] (2.5) Tool System Operation Next, the operation of the tool system 1 will be described with reference to FIG.

[0093] The tool system 1 accepts input of basic information, for example, via the operation unit 33 (step S1). Specifically, the tool system 1 accepts an operation of the worker on an input screen that prompts the worker to input the basic information. Depending on the operation of the worker on the input screen, the identification information of the work object A1, the type of the work object A1, the type of the multiple fastening points X0, the number of the multiple fastening points X0, the arrangement of the multiple fastening points X0, or the fastening rule is set as the basic information.

[0094] Next, the tool system 1 acquires a captured image of the tightening point X1 as a set tightening point Xs by the imaging unit 26 with the tool 2 set at the first (first) tightening point X0 (for example, tightening point X1) (step S2). The tool system 1 also acquires reference information regarding a reference direction (for example, a direction of gravity) via the reference detection unit 28 (step S3). Next, the tool system 1 identifies the position coordinates P1 of the first set tightening point Xs shown in the captured image acquired in step S2 based on the basic information input in step S1 and the reference information acquired in step S3 (step S4). Next, the tool system 1 permits the tightening operation of the tool 2 without determining whether the tightening order of the identified first set tightening point (i.e., the first specified tightening point) Xs is correct or not. This allows the operator to tighten the tightening parts arranged at the first set tightening point Xs using the tool 2 (step S5). In other words, when tightening the fastening part placed at the first (first) fastening point Xs, the worker can select any fastening point X0 from among the multiple fastening points X0 and tighten the fastening part placed at the selected fastening point X0.

[0095] Next, the tool system 1 estimates the fastening order of the fastening point X0 to be fastened second or later (step S6). Specifically, the tool system 1 first estimates the position coordinate P2 of the fastening point X0 to be fastened second (fastening point No. 2 in the fastening order). More specifically, when the set fastening rule is diagonal fastening, the tool system 1 estimates the position coordinate of the fastening point X5 located diagonally from the position coordinate P1 of the first specified fastening point Xs (X1) identified in step S4 as the position coordinate P2 of the fastening point X0 in the fastening order No. 2 with reference to FIG. 3. Then, the tool system 1 estimates the position coordinate P3 of the fastening point X0 in the fastening order No. 3 as the position coordinate of the fastening point X3. In addition, the tool system 1 estimates the position coordinate P4 of the fastening point X0 in the fastening order No. 4 as the position coordinate of the fastening point X7 located diagonally from the fastening point X3 in the fastening order No. 3. Moreover, the tool system 1 estimates the position coordinate P5 of the fastening point X0 of the fastening order No. 5 as the position coordinate of the fastening point X2. Moreover, the tool system 1 estimates the position coordinate P6 of the fastening point X0 of the fastening order No. 6 as the position coordinate of the fastening point X6 located diagonally from the fastening point X2 of the fastening order No. 5. Moreover, the tool system 1 estimates the position coordinate P7 of the fastening point X0 of the fastening order No. 7 as the position coordinate of the fastening point X4. Moreover, the tool system 1 estimates the position coordinate P8 of the fastening point X0 of the fastening order No. 8 as the position coordinate of the fastening point X8 located diagonally from the fastening point X4 of the fastening order No. 7.

[0096] Next, the tool system 1 acquires an image of the second set fastening point Xs captured by the imaging unit 26 (step S7). That is, the worker sets the tool 2 at the next (second) set fastening point Xs, and the imaging unit 26 images the next (second) set fastening point Xs where the tool 2 is set. In addition, the tool system 1 acquires reference information regarding the reference direction via the reference detection unit 28 (step S8).

[0097] Next, similarly to step S4, the tool system 1 identifies the position coordinates of the second set fastening point Xs appearing in the captured image acquired in step S8 (step S9). Specifically, the tool system 1 identifies the position coordinates of the second set fastening point Xs appearing in the captured image acquired in step S8 based on the basic information input in step S1 and the reference information acquired in step S9.

[0098] Next, the tool system 1 judges whether the fastening order of the specified second set fastening point Xs (specified fastening point of the fastening order No. 2) is correct (step S10). Specifically, the tool system 1 judges whether the position coordinate of the second set fastening point Xs specified in step S9 matches the position coordinate P2 of the fastening point X5 of the fastening order No. 2 estimated in step S6. In the sequence of FIG. 5, it is assumed that the judgment result of step S10 matches. When the tool system 1 judges that the fastening order of the second set fastening point Xs is correct, it permits the fastening operation of the tool 2. As a result, the worker uses the tool 2 to fasten the fastening parts arranged at the second set fastening point Xs (step S11). Then, the tool system 1 repeats the processing of steps S7 to S11 according to the number of fastening points X0 of the work object A1.

[0099] (3) Effects The tool system according to the present embodiment is a tool system including a portable tool 2. The tool 2 includes a driving unit 24, an imaging unit 26, and a reference detection unit 28. The driving unit 24 drives the tip tool 242. The imaging unit 26 generates a captured image G1. The reference detection unit 28 detects a reference direction. The tip tool 242 can tighten the fastening parts arranged at the multiple fastening points X0 of the work target A1. The imaging unit 26 is arranged so as to include within the imaging range W1 a set fastening point Xs (Xs=X1 in FIG. 3) at which the tip tool 242 is set among the multiple fastening points X0. The tool system 1 includes an estimation unit 355. The estimation unit 355 estimates the fastening point X0 (fastening point X5 in FIG. 3) to be fastened next to the set fastening point Xs by using the angle θ1 formed by the first straight line M1 and the second straight line M2 set in the captured image G1. The first straight line M1 is set in the captured image G1 based on the reference direction detected by the reference detection unit 28. The second straight line M2 is a straight line passing through the position coordinate P1 of the set fastening point Xs included in the captured image G1 and the center coordinate P0 of the work target A1 shown in the captured image G1.

[0100] According to this configuration, the fastening point X0 to be fastened next to the set fastening point Xs is estimated using the angle θ1 formed by the first straight line M1 and the second straight line M2 set in the captured image G1. Therefore, even if the relative positional relationship between the set fastening point Xs and the work object A1 changes in the captured image G1, the fastening point X0 to be fastened next to the first set fastening point Xs can be accurately estimated from the captured image G1.

[0101] Furthermore, in the tool system 1 according to this embodiment, the estimation unit 355 detects the contour R1 of the work object A1 captured in the captured image G1, and estimates the center coordinates P0 of the work object A1 based on the detected contour R1. According to this configuration, the center coordinates P0 of the work object A1 can be more accurately estimated based on the contour R1 of the work object A1.

[0102] Moreover, the tool system 1 according to the present embodiment further includes a correction unit 353. The correction unit 353 corrects distortion of the captured image G1. The estimation unit 355 obtains the center coordinates P0 based on the captured image G1 corrected by the correction unit 353. With this configuration, the center coordinates P0 of the work target object A1 can be obtained more accurately.

[0103] Moreover, the tool system 1 according to this embodiment further includes an operation unit 33, 231 (input unit). The operation unit 33, 231 inputs number information regarding the number of multiple fastening points X0. The estimation unit 355 uses the number information to estimate the Nth fastening point X0 that will be fastened Nth (N: an integer of 3 or more) times. With this configuration, the Nth fastening point X0 that will be fastened Nth can be more accurately estimated.

[0104] Moreover, the tool system 1 according to the present embodiment further includes a storage unit 34. The storage unit 34 stores the number information input to the operation unit 33, 231 (input unit). With this configuration, it is not necessary to repeatedly input the number of the multiple fastening points X0.

[0105] Moreover, in the tool system 1 according to this embodiment, the work object A1 is a flange. The multiple fastening points X0 are multiple bolt fastening points. With this configuration, even if the work object A1 is a flange and the multiple fastening points X0 are bolt fastening points, even if the relative positional relationship between the set fastening points Xs in the captured image G1 and the work object A1 changes, the fastening point X0 to be fastened next to the first set fastening point Xs can be accurately estimated from the captured image G1. The bolt fastening point is the fastening point X0 that is fastened by a bolt.

[0106] Moreover, the tool 2 according to the present embodiment is a tool used in the tool system 1. The tool 2 includes a driving unit 24, an imaging unit 26, and a reference detection unit 28. According to this configuration, it is possible to provide the tool 2 used in the tool system 1 that exhibits the above-mentioned effects.

[0107] (4) Aspects other than the tool system The same functions as those of the tool system 1 according to the above embodiment may be embodied in a tool, a fastening point estimating method, a computer program (program), or a non-transitory recording medium on which a computer program is recorded.

[0108] The tool according to one aspect is a tool used in the tool system 1 according to the above embodiment, and includes a drive unit 24, an imaging unit 26, and a reference detection unit .

[0109] The tightening estimation method according to one embodiment is a tightening location estimation method using a portable tool 2. The tool 2 has a driving unit 24, an imaging unit 26, and a reference detection unit 28. The driving unit 24 drives a tip tool 242. The imaging unit 26 generates a captured image G1. The reference detection unit 28 detects a reference direction. The tip tool 242 is capable of tightening a tightening part arranged at a plurality of tightening locations X0 of the work target A1. The imaging unit 26 is arranged so as to include within an imaging range W1 a set tightening location Xs at which the tip tool 242 is set among the plurality of tightening locations X0. The tightening location estimation method has an estimation step. In the estimation step, the tightening location X0 to be tightened next to the set tightening location Xs is estimated using an angle θ1 formed by a first straight line M1 and a second straight line M2 set in the captured image G1. The first straight line M1 is set in the captured image G1 based on the reference direction detected by the reference detection unit 28. The second straight line M2 is a straight line passing through the position coordinate P1 of the set fastening location Xs included in the captured image G1 and the center coordinate P0 of the work target A1 depicted in the captured image G1.

[0110] A program according to one aspect is a program for causing a computer to execute the above-described fastening portion estimating method.

[0111] A non-transitory recording medium according to one embodiment is a recording medium for recording a program for causing a computer to execute the above estimation method.

[0112] (5) Variations Modifications of the above embodiment are listed below. The following modifications can be implemented in combination.

[0113] (5.1) Variation 1 In the above embodiment, the identification unit 354 determines the central coordinates P0 of the work target A1 based on the contour R1 of the outer periphery of the identified area of ​​the work target A1. However, the identification unit 354 may determine the central coordinates P0 of the work target A1 based on the contour R2 of the inner periphery of the identified area of ​​the work target A1 (see FIG. 3). In the first modification, the central coordinates P0 can be easily determined, as in the above embodiment.

[0114] (5.2) Variation 2 In the above embodiment, the identification unit 354 determines the center coordinate P0 of the work object A1 based on the contour R1 of the outer periphery of the identified region of the work object A1. However, the identification unit 354 determines the position coordinates N1 to N3 of the center positions of any three (predetermined number) fastening points X0 from the captured image of the imaging unit 26 by image recognition processing (see FIG. 6). In the example of FIG. 6, the position coordinate N2 is the position coordinate of the center position of the set fastening point Xs, and the position coordinates N1 and N3 are, for example, the position coordinates of the center positions of the fastening points X0 located on both sides of the set fastening point Xs. The identification unit 354 uses a preset setting value as the position coordinate N2 of the set fastening point Xs. In addition, when determining the position coordinates N1 and N3, the identification unit 354 determines the regions of any two fastening points X0 from the captured image G2 by image recognition processing, and determines the position coordinates of the center positions of each of the identified regions as the position coordinates N1 and N3, respectively. Then, the identification unit 354 determines, as the center coordinate P0, the position coordinate of the intersection between the perpendicular bisector U1 of the line segment connecting the position coordinates N1 and N2 and the perpendicular bisector U2 of the line segment connecting the position coordinates N2 and N3, as in the above embodiment. In the second modification, the center coordinate P0 can be easily determined, as in the above embodiment.

[0115] (5.3) Variation 3 In the second modification, it is assumed that the captured image includes three fastening points X0. When only two fastening points X0 are included in the captured image (see FIG. 7), the identification unit 354 obtains the position coordinates (e.g., N1, N2) of the center positions of each of the two fastening points X0 by the same process as in the second modification. Then, the identification unit 354 obtains the perpendicular bisector U1 of the line segment connecting the position coordinates N1, N2 of each center position. Then, the identification unit 354 obtains the position moved by the length V1 from the midpoint N4 of the position coordinates N1, N2 of the two center positions toward the center of the work object A1 as the center coordinate P0 of the work object A1. Note that the length V1 is the same length as the radius of the outer shape, the radius of the inner shape, or the average value of the radius of the outer shape and the radius of the inner shape of the work object A1. The information of this length V1 can be set from the operation unit 33 or the operation unit 231 as basic information. The center coordinate P0 thus obtained is an approximation of the actual center coordinate P0 because the length from the midpoint N4 to the actual center coordinate P0 is approximated by the length V1. In the third modification, the center coordinate P0 can be easily obtained in the same manner as in the above embodiment.

[0116] (5.4) Variation 4 In the above embodiment, one set value corresponding to the tip tool 242 of a specific length Y1 is initially set as the position coordinate P1 of the set fastening point Xs in the captured image G2 of the imaging unit 26 (see FIG. 6A). Then, based on this set value, the identification unit 354 identifies the set fastening point Xs, and the estimation unit 355 estimates the position coordinates of each fastening point X0 in the fastening sequence 2 and thereafter.

[0117] Therefore, when the actually used tip tool 242 has a length Y2 different from the specific length Y1 (see FIG. 8B), the position coordinate P1a of the set fastening point Xs (X1) on the captured image G2 in this case is different from the position coordinate P1 corresponding to the specific length Y1. More specifically, the position coordinate P1a is a position coordinate shifted upward by ΔP in the captured image G2 from the position coordinate P1. In this way, when the tip tool 242 having the length Y2 is used, the above-mentioned identification and estimation must be performed based on the position coordinate P1a corresponding to the length Y2. However, in the above embodiment, only the set value of the position coordinate P1 corresponding to the specific length Y1 is set. Therefore, even when the length of the actually used tip tool 242 is the length Y2, the above-mentioned identification and estimation are performed based on the set value of the position coordinate P1. That is, when the tip tool 242 having the length Y2 is used, the above-mentioned identification and estimation are performed based on the position coordinate P1 (the above-mentioned set value) of the set fastening point Xs corresponding to the specific length Y1. Therefore, the accuracy of the above-mentioned identification and estimation is reduced.

[0118] On the other hand, in the fourth modification, a plurality of set values ​​corresponding one-to-one to a plurality of tool tips 242 having different lengths are initially set as position coordinates of the set fastening point Xs on the captured image. In the fourth modification, it is possible to select one set value from the plurality of set values ​​by operating the operation unit 33 or the operation unit 231. That is, in the fourth modification, the operator selects a desired tool tip 242 from a plurality of tool tips 242 having different lengths, and attaches the selected tool tip 242 to the output shaft portion 241 of the tool 2. Then, the operator operates the operation unit 33 or the operation unit 231 to select an installation value corresponding to the tool tip 242 selected by the operator from the plurality of set values. In the fourth modification, the identification unit 354 identifies the set fastening point Xs based on the selected set value from the plurality of set values. Also, the estimation unit 355 performs the above estimation based on the set fastening point (specific fastening point) Xs identified by the identification unit 354.

[0119] According to the fourth modification, it is possible to select one setting value (a setting value corresponding to the tool tip actually used) from among a plurality of setting values, so that the accuracy of the above-mentioned specification and estimation can be improved.

[0120] (5.5) Variation 5 In the fourth modification, one setting value (a setting value corresponding to the actually used tip tool 242) is selected from a plurality of setting values ​​by inputting (i.e. manually) to the operation unit 33 or the operation unit 231. In contrast, in the fifth modification, the length of the actually used tip tool 242 is automatically identified, and based on the identification result, a setting value corresponding to the length of the actually used tip tool 242 is automatically selected from the above-mentioned plurality of setting values.

[0121] More specifically, in the fifth modification, the identifying unit 354 identifies the region of the tip tool 242 from the captured image of the imaging unit 26 by image recognition processing, thereby automatically identifying the length of the tip tool 242. A plurality of set values ​​corresponding one-to-one to a plurality of tip tools 242 having different lengths are pre-initialized in the storage unit 34. Then, the identifying unit 354 selects a set value corresponding to the identified length (i.e., a set value corresponding to the tip tool 242 actually used) from the above-mentioned plurality of set values. Then, the identifying unit 354 identifies the set fastening position Xs based on the selected set value from the above-mentioned plurality of set values. Then, the estimating unit 355 estimates the position coordinates of each fastening position X0 from the second fastening position onward based on the set fastening position (specific fastening position) Xs identified by the identifying unit 354.

[0122] According to variant example 5, a setting value corresponding to the tip tool 242 actually being used is automatically selected from among a plurality of setting values, thereby eliminating the need to select one setting value from among a plurality of setting values.

[0123] (5.6) Variation 6 In the above embodiment, the reference detection unit 28 detects the direction of gravity as the reference direction. In contrast, in the sixth modification, the reference detection unit 28 detects the geomagnetic direction (for example, the direction in which the N pole faces) as the reference direction. In the above embodiment, the direction of gravity is detected as the reference direction, so that when the imaging direction (optical axis) of the imaging unit 26 coincides with the direction of gravity, the direction of gravity is perpendicular to the screen of the captured image, and therefore the first straight line based on the reference direction (direction of gravity) cannot be set in the captured image. In contrast, in the sixth modification, the geomagnetic direction is detected as the reference direction, so that even if the imaging direction of the imaging unit 26 coincides with the direction of gravity, it is possible to set the first straight line based on the reference direction in the captured image. According to the sixth modification, even if the imaging direction of the imaging unit 26 coincides with the direction of gravity, the identification unit 354 can identify the set fastening point Xs, and the estimation unit 355 can estimate the position coordinates of each fastening point X0 in the second and subsequent fastening orders.

[0124] (5.7) Variation 7 In the above embodiment, the battery pack 201 is included in the components of the tool 2, but it does not have to be included in the components of the tool 2. In this case, the battery pack 201 may be separate from the tool 2 (tool body) and configured to be detachable from the tool 2 (tool body).

[0125] (5.8) Variation 8 In the above embodiment, the identification unit 354 identifies the set fastening location Xs by expressing the position coordinate P1 of the set fastening location Xs in a coordinate system (θ coordinate system) using an angle θ1. The θ coordinate system is an example of a common coordinate system. The θ coordinate system is a coordinate system using only the deflection angle θ of a polar coordinate system. The polar coordinate system is a coordinate system expressed by a pair (r, θ) of a distance r to the origin and a deflection angle θ from a starting line (a line that defines the beginning of the deflection angle θ). The common coordinate system is a coordinate system having an identified common origin and identified common coordinate axes in each captured image. For example, instead of the θ coordinate system, an orthogonal coordinate system may be used as the common coordinate system. The above orthogonal coordinate system has, for example, a center coordinate P0 as the origin, a first straight line M1 as the vertical axis, and a straight line that passes through the center coordinate P0 and is perpendicular to the first straight line M1 as the horizontal axis. Although the estimation unit 355 also uses the θ coordinate system as an example, the above-mentioned orthogonal coordinate system may be used as a traffic coordinate system instead of the θ coordinate system.

[0126] (5.9) Other Modifications The execution subject of the tool system 1 or the fastening location estimation method in the present disclosure includes a computer system. The computer system is mainly composed of a processor and a memory as hardware. The processor executes a program recorded in the memory of the computer system to realize the function of the execution subject of the tool system 1 or the fastening location estimation method in the present disclosure. The program may be pre-recorded in the memory of the computer system, may be provided through an electric communication line, or may be provided by being recorded in a non-transitory recording medium such as a memory card, an optical disk, or a hard disk drive that can be read by the computer system. The processor of the computer system is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or a large-scale integrated circuit (LSI). The integrated circuits such as IC or LSI referred to here are called by different names depending on the degree of integration, and include integrated circuits called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). Furthermore, a field-programmable gate array (FPGA) that is programmed after the manufacture of the LSI, or a logic device that can reconfigure the joint relationship inside the LSI or the circuit partition inside the LSI, can also be adopted as a processor. The electronic circuits may be integrated in one chip or distributed among multiple chips. The chips may be integrated in one device or distributed among multiple devices. The computer system referred to here includes a microcontroller having one or more processors and one or more memories. Thus, the microcontroller is also composed of one or more electronic circuits including a semiconductor integrated circuit or a large-scale integrated circuit.

[0127] Furthermore, it is not essential for the tool system 1 that a plurality of functions are integrated into one housing, and the components of the tool system 1 may be distributed across a plurality of housings. For example, the components of the determination device 3 of the tool system 1 may be distributed across a plurality of housings. Furthermore, at least some of the functions of the tool system 1, for example, some of the functions of the determination device 3, may be realized by cloud (cloud computing) or the like.

[0128] (6) Summary As is apparent from the above-described embodiment and modified examples, the present specification discloses the following aspects.

[0129] The tool system (1) according to the first embodiment is a tool system including a portable tool (2). The tool (2) includes a drive unit (24), an imaging unit (26), and a reference detection unit (28). The drive unit (24) drives a tool tip (242). The imaging unit (26) generates captured images (G1; G2, G3). The reference detection unit (28) detects a reference direction. The tool tip (242) can tighten fastening parts arranged at a plurality of fastening points (X0) of a work object (A1). The imaging unit (26) is arranged so that a set fastening point (Xs) where the tool tip (242) is set among the plurality of fastening points (X0) is included within an imaging range (W1). The tool system (1) further includes an estimation unit (355). The estimation unit (355) estimates the tightening point (X5) to be tightened next to the set tightening point (Xs) by using the angle (θ1) formed by the first straight line (M1) and the second straight line (M2) set in the captured image (G1). The first straight line (M1) is set in the captured image (G1) based on the reference direction detected by the reference detection unit (28). The second straight line (M2) is a straight line passing through the position coordinate (P1) of the set tightening point (Xs) included in the captured image (G1) and the center coordinate (P0) of the work object (A1) shown in the captured image (G1).

[0130] According to this configuration, the tightening point (X0) to be tightened next to the set tightening point (Xs) is estimated using the angle (θ1) formed by the first straight line (M1) and the second straight line (M2) set in the captured image (G1). Therefore, even if the relative positional relationship between the set tightening point (Xs) and the work object (A1) changes in the captured image (G1), the tightening point (X0) to be tightened next can be estimated with high accuracy from the captured images (G1; G2, G3).

[0131] In the tool system (1) of the second aspect, in the first aspect, the estimation unit (355) detects the contour (R1; R2) of the work object (A1) captured in the captured image (G1) and estimates the center coordinate (P0) of the work object (A1) based on the detected contour (R1; R2).

[0132] According to this configuration, the central coordinates (P0) of the work object (A1) can be estimated more accurately based on the contour (R1; R2) (eg, a circular arc) of the work object (A1).

[0133] In the tool system (1) according to the third aspect, in the first aspect, the estimation unit (355) determines the position coordinates (N1 to N3) of each of a predetermined number of fastening points (X0) included in the captured image (G1) among a plurality of fastening points (X0), and estimates the center coordinate (P0) of the work object (A1) based on the determined position coordinates (N1 to N3).

[0134] According to this configuration, the central coordinates (P0) of the work object (A1) can be estimated more accurately based on the position coordinates (N1 to N3) of each of the predetermined number of fastening points (X0) of the work object (A1).

[0135] The tool system (1) according to a fourth aspect is any one of the first to third aspects, and further includes a correction unit (353). The correction unit (353) corrects distortion of the captured image (G1). The estimation unit (355) calculates center coordinates (P0) based on the captured image (G1) corrected by the correction unit (353).

[0136] According to this configuration, the central coordinates (P0) of the work object (A1) can be obtained more accurately.

[0137] The tool system (1) according to a fifth aspect is any one of the first to fourth aspects, and further includes an input unit (33, 231). The input unit (33, 231) inputs number information regarding the number of the multiple fastening points (X0). The estimation unit (355) estimates the N-th fastening point (X0) that will be fastened Nth (N: an integer of 3 or more) using the number information.

[0138] According to this configuration, the Nth tightening point (X0) that is tightened the Nth time can be estimated more accurately.

[0139] The tool system (1) according to a sixth aspect is the fifth aspect, further including a storage unit (34). The storage unit (34) stores the quantity information input to the input unit (33, 231).

[0140] According to this configuration, it is not necessary to repeatedly input the number of fastening points (X0).

[0141] In the tool system (1) according to the seventh aspect, in any one of the first to sixth aspects, the work object (A1) is a flange, and the plurality of fastening points (X0) are a plurality of bolt fastening points.

[0142] According to this configuration, even if the work object (A1) is a flange and the multiple fastening points (X0) are bolt fastening points, even if the relative positional relationship between the set fastening points (Xs) in the captured image (G1) and the work object (A1) changes, the next fastening point (X0) to be fastened can be accurately estimated from the captured image (G1).

[0143] A tool (2) according to an eighth aspect is a tool used in the tool system (1) according to any one of the first to seventh aspects. The tool (2) includes a drive unit (24), an imaging unit (26), and a reference detection unit (28).

[0144] According to this configuration, it is possible to provide a tool (2) for use in the tool system (1) that achieves the above-mentioned effects.

[0145] The fastening point estimation method according to the ninth aspect is a fastening point estimation method using a portable tool (2). The tool (2) has a drive unit (24), an imaging unit (26), and a reference detection unit (28). The drive unit (24) drives a tip tool (242). The imaging unit (26) generates captured images (G1; G2, G3). The reference detection unit (28) detects a reference direction. The tip tool (242) is capable of fastening parts arranged at a plurality of fastening points (X0) of a work object (A1). The imaging unit (26) is arranged so that a set fastening point (Xs) where the tip tool (242) is set among the plurality of fastening points (X0) is included within an imaging range (W1). The fastening point estimation method has an estimation step. In the estimation step, the fastening point (X0) to be fastened next to the set fastening point (Xs) is estimated using the angle (θ1) formed by the first straight line (M1) and the second straight line (M2) set in the captured image (G1). The first straight line (M1) is set in the captured image (G1) based on the reference direction detected by the reference detection unit (28). The second straight line (M2) is a straight line passing through the position coordinate (P1) of the set fastening point (Xs) included in the captured image (G1) and the center coordinate (P0) of the work object (A1) shown in the captured image (G1).

[0146] According to this configuration, the tightening point (X0) to be tightened next to the set tightening point (Xs) is estimated using the angle (θ1) formed by the first straight line (M1) and the second straight line (M2) set in the captured image (G1). Therefore, even if the relative positional relationship between the set tightening point (Xs) and the work object (A1) changes in the captured image (G1), the tightening point (X0) to be tightened next can be estimated with high accuracy from the captured image (G1).

[0147] A program according to a tenth aspect is a program for causing one or more processors to execute the fastening point estimating method according to the ninth aspect.

[0148] According to this configuration, it is possible to provide a program capable of executing the fastening portion estimating method. [Explanation of symbols]

[0149] 1 Tool System 2 tools 24 Drive unit 26 Imaging unit 28 Reference detection section 33,231 Operation section (input section) 34 Storage section 242 Tip tools 353 Correction Department 355 Estimation Department A1 Work object G1~G3 Images M1 1st straight line M2 2nd straight line N1~N3 Position coordinates P1 position coordinates P0 center coordinate R1,R2 Contour W1 Image range X0 Fastening point Xs Set tightening point

Claims

1. A tool system including a portable tool having a drive unit that drives a tool tip, an imaging unit that generates an image, and a reference detection unit that detects a reference direction, The tool bit is capable of fastening parts arranged at a plurality of fastening points of a workpiece, The imaging unit is disposed so as to include within an imaging range a set fastening point at which the tool tip is set among the plurality of fastening points, An estimation unit that estimates a tightening point to be tightened next to the set tightening point by using an angle formed by a first straight line and a second straight line set in the captured image, the first straight line is set in the captured image based on the reference direction detected by the reference detection unit, The second straight line is a straight line passing through the position coordinates of the set fastening portion included in the captured image and the center coordinates of the work object shown in the captured image. Tool system.

2. The estimation unit detects an outline of the work object captured in the captured image, and estimates the center coordinates of the work object based on the detected outline. The tool system of claim 1 .

3. The estimation unit determines position coordinates of each of a predetermined number of fastening points included in the captured image among the plurality of fastening points, and estimates the center coordinates of the work object based on the determined position coordinates. The tool system of claim 1 .

4. A correction unit that corrects distortion of the captured image, The estimation unit determines the center coordinates based on the captured image corrected by the correction unit. A tool system according to any one of claims 1 to 3.

5. An input unit for inputting number information regarding the number of the plurality of fastening points, The estimation unit estimates an N-th tightening point that will be tightened Nth (N: an integer of 3 or more) using the number information, A tool system according to any one of claims 1 to 3.

6. The input unit may further include a storage unit configured to store the number information inputted to the input unit. The tool system of claim 5.

7. The workpiece is a flange, The plurality of fastening points are a plurality of bolt fastening points. A tool system according to any one of claims 1 to 3.

8. A tool used in the tool system according to any one of claims 1 to 3, The driving unit, the imaging unit, and the reference detection unit are provided. tool.

9. A fastening point estimation method using a portable tool having a drive unit that drives a tool tip, an imaging unit that generates an image, and a reference detection unit that detects a reference direction, The tool bit is capable of fastening parts arranged at a plurality of fastening points of a workpiece, The imaging unit is disposed so as to include within an imaging range a set fastening point at which the tool tip is set among the plurality of fastening points, An estimation step of estimating a tightening point to be tightened next to the set tightening point by using an angle formed by a first straight line and a second straight line set in the captured image, the first straight line is set in the captured image based on the reference direction detected by the reference detection unit, The second straight line is a straight line passing through the position coordinates of the set fastening portion included in the captured image and the center coordinates of the work object shown in the captured image. Method for estimating fastening location.

10. A program for causing one or more processors to execute the fastening point estimating method according to claim 9.

Images