Determination device, determination system, determination method, and determination program

Abstract

To suppress reduction in accuracy of determining bolt tightness.SOLUTION: A determination device (1) is provided, comprising a selection unit (106) for selecting an image to be used for tightness determination from among multiple determination target images of a tightness determination target bolt on the basis of how the bolt appears in the images, and a determination unit (107) configured to determine tightness of the bolt based on the image selected by the selection unit (106).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a determination device for determining the fastening state of a bolt. [Background technology]

[0002] There have been known technologies for automatically determining the fastening state of a bolt. For example, Patent Document 1 listed below discloses a system for detecting the fastening state of a high-strength bolt, which detects the marking angles of markings on a plate, a washer, a nut, and the high-strength bolt based on an image of the fastened state of the high-strength bolt, and determines the fastening state based on the detected marking angles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-009932 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional techniques have a problem in that the accuracy of the judgment can be reduced depending on the appearance of the marking in the captured image, etc. One aspect of the present invention aims to realize a judgment device etc. that can suppress a decrease in the accuracy of judging the fastening state of a bolt even if the judgment image includes an image that is not suitable for judgment. [Means for solving the problem]

[0005] In order to solve the above problem, a determination device according to one embodiment of the present invention includes a selection unit that selects a second image to be used for determining the fastening state from a plurality of first images that show a bolt whose fastening state is to be determined, based on how the bolt appears in the first image, and a determination unit that determines the fastening state of the bolt based on the second image selected by the selection unit.

[0006] In addition, in order to solve the above problem, a determination method according to one embodiment of the present invention is a determination method executed by one or more information processing devices, and includes a selection step of selecting a second image to be used for determining the fastening state from a plurality of first images showing a bolt whose fastening state is to be determined, based on how the bolt appears in the first image, and a determination step of determining the fastening state of the bolt based on the second image selected in the selection step. [Effects of the Invention]

[0007] According to one aspect of the present invention, even if the determination image includes an image that is not suitable for determination, it is possible to prevent a decrease in the accuracy of determining the fastening state of the bolt. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a configuration of a main part of a determination device according to an embodiment of the present invention; [Figure 2] FIG. 1 is a diagram showing an overview of a determination system including the determination device. [Figure 3] 5A and 5B are diagrams illustrating examples of engagement states determined by the determination device. [Figure 4] 10A and 10B are diagrams illustrating an example of generating a target image and a determination image. [Figure 5] FIG. 10 is a diagram showing an example of a detection result obtained by distinguishing between an area in a target image in which a bolt with a broken pintail is shown and an area in which a bolt with an unbroken pintail is shown. [Figure 6] FIG. 10 is a diagram showing an example in which an image for determination is selected based on the degree of blurring. [Figure 7] FIG. 10 is a diagram illustrating an example in which feature amounts extracted by a decision model are embedded in a feature space. [Figure 8] 10A and 10B are diagrams illustrating examples of display of determination results of the determination device. [Figure 9] 4 is a flowchart illustrating an example of a process executed by the determination device. DETAILED DESCRIPTION OF THE INVENTION

[0009] [System Configuration] A determination system 5 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an overview of the determination system 5. The determination system 5 is a system that automatically determines the fastening state of a bolt, and includes a determination device 1 and a head-mounted display 2 as shown in the figure.

[0010] The head mounted display 2 is a goggle- or eyeglass-like device worn by the user on the head, and as shown in the figure, includes a photographing device 21 and a display device 22. The photographing device 21 is positioned so that the photographing direction coincides with the orientation of the face of the user wearing the head mounted display 2, and photographs the front direction of the user's face. The display device 22 is positioned so as to be located in front of the eyes of the user wearing the head mounted display 2. Although not shown, the head mounted display 2 may also include a communication unit for communicating with the determination device 1.

[0011] In this embodiment, an example will be described in which the display device 22 is a transmissive display device. That is, a user wearing the head-mounted display 2 can view the image displayed by the display device 22, and can also view the situation on the other side of the display device 22 through the display device 22. Note that a non-transmissive display device can also be used as the display device 22. In this case, the display device 22 may be configured to display images (moving images) captured by the imaging device 21 in real time.

[0012] The determination device 1 determines the fastening state of a bolt within the field of view of a user wearing the head-mounted display 2, using an image captured by the imaging device 21. The determination device 1 then displays the determination result of the fastening state of the bolt on the display device 22. Note that while Fig. 2 shows an example in which the determination device 1 is a notebook personal computer, the determination device 1 may be any information processing device having predetermined functions such as determining the fastening state, and is not limited to a notebook personal computer.

[0013] 2 also shows image IMG1, which shows a number of bolts. When a bolt comes into the user's field of view, the image captured by the photographing device 21 shows the bolt as in image IMG 1. The determination device 1 analyzes such an image to determine the fastening state of the bolts shown in the image, and displays the determination result on the display device 22.

[0014] As described above, the determination system 5 includes the head-mounted display 2 equipped with the image capture device 21 and the display device 22, and the determination device 1. The determination device 1 uses images captured by the image capture device 21 to determine the fastening state of a bolt within the field of view of the user wearing the head-mounted display 2, and displays the determination result on the display device 22. This configuration allows the user performing bolt tightening work to easily recognize the fastening state of the bolt in front of them. This in turn makes it possible to achieve more efficient and accurate bolt tightening work.

[0015] The determination device 1 can also be used in combination with a normal imaging device that is not mounted on the head-mounted display 2. In this case, an image of the bolt taken by the normal imaging device can be input to the determination device 1 to determine the fastening state of the bolt, and the determination result can be displayed on the display unit of the determination device 1 or another display device.

[0016] [Regarding fastening status] Here, the bolt tightening procedure and the tightening state determined by the determination device 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing examples of the tightening state determined by the determination device 1. Fig. 3 shows a perspective view of bolt A1 and five tightening states of bolt A1 (tightening states 31 to 35).

[0017] The bolt A1 shown in Figure 3 is a cylindrical bolt with a thread cut on the outer periphery. The bolt A1 is typically a high-strength bolt, but the judgment device 1 can also be used to judge the fastening state of bolts other than high-strength bolts. First, a washer A3 is passed through the bolt A1, and then a nut A2 is fastened to the bolt A1. This is the primary fastening process.

[0018] After the primary tightening is complete, marking is performed on the bolt A1, etc. Then, after marking, secondary tightening is performed. In the secondary tightening, the nut A2 is tightened with a dedicated tool so that the tightening force is within a predetermined range.

[0019] The markings are used to confirm whether or not the secondary tightening has been performed appropriately, and marking A4 is attached to bolt A1, etc., as shown in Figure 3. Marking A4 is a linear marking drawn in a straight line from bolt A1 through nut A2 and washer A3 to the surface of the component to be tightened.

[0020] When the secondary tightening is performed normally, the bolt A1 and washer A3 do not move, and only the nut A2 rotates. Therefore, as shown in the tightening state 31 in Figure 3, the bolt A1, washer A3, and marking A4 on the member to be tightened remain aligned, while the marking A4 on the nut A2 is shifted in the direction of rotation of the nut A2.

[0021] On the other hand, if the secondary tightening is not performed normally, the appearance of the marking A4 will be different from that of the tightened state 31. For example, if the secondary tightening is not performed, i.e., if the bolt is not tightened properly, the marking A4 will remain in a straight line from the bolt A1 to the object to be tightened, as shown in tightened state 32 in Figure 3.

[0022] Furthermore, for example, if, when tightening nut A2, bolt A1 rotates along with nut A2, i.e., if axial rotation occurs, bolt A1 and marking A4 on washer A3 will not be aligned in a straight line, as shown in fastening state 33 in Figure 3, and marking A4 on bolt A1 will be shifted in the rotation direction of nut A2.

[0023] Furthermore, for example, if the washer A3 also rotates when the nut A2 is tightened, i.e., if corotation occurs, the marking A4 on the washer A3 will be shifted in the rotation direction of the nut A2 relative to the marking A4 on the component to be fastened, as shown in fastened state 34 in Figure 3. Whether axial rotation or corotation occurs, the appropriate fastening force will not be obtained.

[0024] 3, no markings are applied to the bolt A1, etc. If no markings are applied, it is difficult to determine whether the fastened state is normal or not. Therefore, it is preferable that the determination device 1 determine that the fastened state 35 without a mark is also abnormal.

[0025] After secondary tightening, the determination device 1 can determine whether the tightening state is normal or not based on the presence or absence of the marking A4 or the state of the marking A4. Furthermore, if the determination device 1 determines that the tightening state is not normal, it can determine the state of the bolt A1, etc., i.e., whether it is a missed tightening, axial rotation, co-rotation, or no mark. Note that the determination device 1 is only required to determine whether the tightening state of the bolt is normal or not, and it is not essential to classify abnormal tightening states as described above.

[0026] [Device configuration] The configuration of the determination device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing an example of the configuration of the main parts of the determination device 1. As shown in the figure, the determination device 1 includes a control unit 10 that controls each unit of the determination device 1 and a storage unit 11 that stores various data used by the determination device 1. The determination device 1 also includes a communication unit 12 that enables the determination device 1 to communicate with other devices, an input unit 13 that accepts various data input to the determination device 1, and an output unit 14 that enables the determination device 1 to output various data. The control unit 10 also includes an image acquisition unit 101, a target image generation unit 102, a region detection unit 103, a determination image generation unit (image generation unit) 104, a threshold setting unit 105, a selection unit 106, a determination unit 107, and a display control unit 108. The storage unit 11 stores a detection model 111 and a determination model 112.

[0027] The image acquisition unit 101 acquires an image that shows one or more bolts. For example, the image acquisition unit 101 may acquire an image captured by the image capture device 21 included in the head-mounted display 2 shown in FIG. 2. In this case, the image acquisition unit 101 may acquire the image captured by the image capture device 21 by communicating with the head-mounted display 2 via the communication unit 12. Furthermore, for example, the image acquisition unit 101 may acquire an image generated by capturing an image of a bolt with a normal image capture device (e.g., a camera or video camera).

[0028] The image acquired by the image acquisition unit 101 may be a still image showing a bolt. For example, if the photographing device 21 is configured to capture a still image, the image acquisition unit 101 may acquire the captured still image as is. If the photographing device 21 is configured to capture a moving image, the image acquisition unit 101 may acquire frame images constituting the moving image from the moving image.

[0029] The target image generating unit 102 generates a target image (a third image showing one or more bolts) that is to be used to determine the fastening state, from the image acquired by the image acquiring unit 101. A method for generating the target image will be described later with reference to FIG.

[0030] The area detection unit 103 detects areas in which each bolt appears from the target image generated by the target image generation unit 102. A detection model 111 is used to detect areas in which each bolt appears.

[0031] Detection model 111 is a trained model that has been machine-learned to detect areas in which one or more bolts appear from an image in which the bolts appear. Detection model 111 may be generated, for example, by machine learning using training data in which the positions and ranges of areas in which the individual bolts appear are associated as ground truth data for an image in which one or more bolts appear. As a specific example, a deep learning model such as YOLO (You Only Look Once) may be used as detection model 111.

[0032] It is preferable that the training data for the detection model 111 include various images corresponding to the expected shooting distance, shooting angle, etc. However, the annotation work of associating correct data with an image requires time and effort. For this reason, a single original image may be converted by image processing to generate multiple images, and each image may be pasted onto a background image, thereby minimizing the annotation work and generating diverse training data. Examples of the image processing include resizing, flipping horizontally, rotating, and changing the contrast.

[0033] The determination image generating unit 104 cuts out the area detected by the area detecting unit 103 (the area in which each bolt appears) from the target image generated by the target image generating unit 102, and generates a determination image (first image).

[0034] The threshold setting unit 105 sets a threshold used by the selection unit 106 to select an image. Specifically, for each determination image generated by the determination image generation unit 104, the threshold setting unit 105 sets a threshold used by the selection unit 106 to determine whether or not to select the determination image, depending on the size of the bolt depicted in the determination image or the shooting distance of the determination image. Note that the "bolt size" here does not refer to the actual size of the bolt, but the size of the bolt on the determination image. Because the determination image is a cutout of the area in which each bolt appears, the size of the determination image can also be considered the size of the bolt.

[0035] The selection unit 106 selects an image (second image) to be used for determining the fastening state from a plurality of determination images showing the bolt whose fastening state is to be determined, based on how the bolt appears in the determination image. As will be described in detail later, the selection unit 106 uses a threshold set by the threshold setting unit 105 when selecting an image.

[0036] Determination unit 107 determines the fastening state of the bolt based on the image selected by selection unit 106. Determination of the fastening state of the bolt uses determination model 112. Determination model 112 is a model generated by learning so that when feature amounts extracted from a group of images of bolts with different fastening states are embedded in a feature space, the distance between the feature amounts derived from a group of images with a common fastening state becomes small.

[0037] The display control unit 108 displays the determination result of the determination unit 107. The device on which the determination result is displayed is not particularly limited. For example, the display control unit 108 may display the determination result on the display device 22 included in the head-mounted display 2 shown in FIG. 2. In this case, the image acquisition unit 101 may communicate with the head-mounted display 2 via the communication unit 12 to cause the display device 22 to display the determination result. Furthermore, for example, if the output unit 14 is a display device, the display control unit 108 may cause the output unit 14 to display the determination result.

[0038] As described above, the judgment device 1 comprises a selection unit 106 that selects an image to be used for judging the fastening state from among a plurality of judgment images showing the bolt whose fastening state is to be judged, based on how the bolt appears in the judgment image, and a judgment unit 107 that judges the fastening state of the bolt based on the image selected by the selection unit 106. With this configuration, it is possible to prevent a decrease in the accuracy of judging the fastening state of the bolt, even if the judgment images include images that are not suitable for judgment.

[0039] As described above, the judgment device 1 may also include an area detection unit 103 that detects areas in which each bolt appears from a target image in which one or more bolts appear, and an image-for-judgment generation unit 104 that cuts out areas in which each bolt appears from the target image to generate an image for judgment. This makes it possible to automatically judge the fastening state of each bolt based on a target image in which one or more bolts appear. Note that the judgment device 1 can also judge the fastening state of a bolt using an image for judgment input via the input unit 13 or the like, and the area detection unit 103 and the image-for-judgment generation unit 104 may be omitted.

[0040] [Method for generating target images and judgment images] A method for generating a target image by the target image generating unit 102 and a method for generating a determination image by the determination image generating unit 104 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of generating a target image and a determination image.

[0041] 4 is an example of an image captured by the image capturing device 21 included in the head-mounted display 2. The target image generating unit 102 may use image IMG2 as the target image as is, but in that case, depending on the distance from the image capturing device 21 to the bolt, the size of the bolt captured in image IMG2 may become small, which may reduce the accuracy of detecting the area where the bolt is captured. Also, a large proportion of the area in image IMG2 does not capture the bolt.

[0042] Therefore, the target image generation unit 102 may cut out a part of the image IMG2 to use as the target image. In the example of FIG. 4, the target image generation unit 102 cuts out an area AR1 in the center of the image IMG2 to generate the target image IMG3. Since it is considered that a bolt is often present in the center of the field of view of a user (a person wearing the head-mounted display 2) performing a bolt tightening operation, a method of cutting out an area AR1 in the center of the image IMG2 to use as the target image is effective. The size of the area AR1 may be a predetermined size that can be input to the detection model 111, that is, the same size as the image of the training data used to train the detection model 111. Note that the method of determining the area to be cut out is arbitrary and is not limited to this example. Furthermore, the size of the area AR1 may be different from the image of the training data. In this case, the target image may be resized to a predetermined size when input to the detection model 111.

[0043] Next, area detection unit 103 detects areas in target image IMG3 where individual bolts are shown. For example, area detection unit 103 may detect areas in which individual bolts are shown based on output values ​​obtained by inputting target image IMG3 to detection model 111. Target image IMG3' shown in Figure 4 shows the detection results of area detection unit 103, and areas AR2 and AR2a, etc., indicated by dashed rectangles are areas detected by area detection unit 103 (areas in which individual bolts are shown).

[0044] Then, the determination image generation unit 104 generates a determination image IMG4 by cutting out the area AR2, etc. detected by the area detection unit 103, from the target image IMG3. Note that although only one determination image IMG4 is shown in Fig. 4, a determination image is generated for each of the areas detected by the area detection unit 103.

[0045] Here, in the target image IMG3' in Figure 4, the area AR2a was detected as an area in which the bolt was captured, but the area AR2a only captured part of the bolt, not the entire bolt. Therefore, if the area AR2a is cut out and used as the image for determination, the determination result of the fastening state based on that image for determination may not be valid. In particular, if an image for determination in which some or all of the marking is not captured is used, the determination result of the fastening state is likely to be invalid.

[0046] Therefore, the determination image generating unit 104 may generate the determination image by cutting out the remaining areas, excluding the areas located along the outer edge of the target image IMG3', from the areas detected by the area detection unit 103. For example, the determination image generating unit 104 may generate the determination image by cutting out the remaining six areas, including the area AR2, from the nine areas shown in the target image IMG3', excluding the bottom three areas including the area AR2a.

[0047] This reduces the possibility that an image in which part of the bolt is cut off at the edge will be used as the determination image. The area located along the outer edge of the target image IMG3' may be defined in advance. A method for determining whether an area detected by the area detection unit 103 is located within the area located along the outer edge of the target image IMG3' may also be defined in advance. For example, the determination may be made based on whether the representative coordinates (e.g., coordinates of the center position or coordinates of the upper left corner) of the area detected by the area detection unit 103 are included within the area located along the outer edge of the target image IMG3'. The determination may also be made based on the distance from the representative coordinates to the edge of the target image.

[0048] Furthermore, when the area detected by the area detection unit 103 is too large or too small, the shooting distance may be too close or too far. It is also possible that the detection result is incorrect. In any case, when the detected area is too large or too small, even if that area is cut out and used as a determination image, it is considered that it is difficult to obtain a valid determination result from that determination image.

[0049] Therefore, the determination image generating unit 104 may generate a determination image by cutting out an area whose size is within a predetermined normal range from the area detected by the area detecting unit 103. This makes it possible to generate a determination image suitable for determination.

[0050] For example, the determination image generating unit 104 may cut out an area whose width is within a predetermined normal range, or may cut out an area whose height is within a predetermined normal range, or may cut out an area whose product or sum of width and height is within a predetermined normal range. Note that the normal range may be defined by an upper limit and a lower limit, or may be defined by either the upper limit or the lower limit.

[0051] 4, the target image IMG3 is generated by cutting out the central area AR1 of the image IMG2, but the image IMG2 may be divided into a plurality of parts (for example, four parts) and each of the obtained images may be used as the target image. In this case, the determination results for each of the target images obtained by division may be used as the determination results for the original image IMG2.

[0052] By dividing image IMG2 to generate target images, it becomes possible to detect bolts appearing anywhere in image IMG2. Therefore, this configuration is effective when bolts are visible throughout image IMG2. Note that it is also possible to detect bolts appearing anywhere in image IMG2 by appropriately resizing image IMG2 and inputting it to detection model 111 without dividing it. However, in this case, the area in which each bolt appears becomes smaller, which may result in a decrease in detection accuracy. In this regard, when dividing image IMG2 to generate multiple target images, the area in which each bolt appears can be increased by resizing each target image when inputting it to detection model 111, thereby preventing the above-mentioned decrease in detection accuracy.

[0053] [Determining whether a pintail has broken] Among high-strength bolts, there are those called torsion-type high-strength bolts. Torsion-type high-strength bolts are designed so that the tip of the bolt shank (a part called the pintail) breaks during secondary tightening. In other words, with torsion-type high-strength bolts, it is possible to tell at a glance whether secondary tightening has been performed or not by checking whether the pintail has broken or not.

[0054] Therefore, the area detection unit 103 may detect areas where bolts with broken pintails are captured, distinguishing between areas where bolts with non-fractured pintails are captured. In this case, the area where bolts are captured can be detected from the target image, and the detection model 111 that has been machine-learned to classify the bolts captured in the detected areas as either bolts with broken pintails or bolts with non-fractured pintails can be used.

[0055] Fig. 5 is a diagram showing an example of the detection results obtained by distinguishing between areas in the target image IMG31 where a bolt with a broken pintail is captured and areas where a bolt with an unbroken pintail is captured. In Fig. 5, area AR21 is detected as an area where a bolt with a broken pintail is captured, and area AR22 is detected as an area where a bolt with an unbroken pintail is captured. As shown in the figure, the bolt captured in area AR22 has its pintail A11 remaining unbroken.

[0056] As described above, a bolt whose pintail A11 remains unbroken is a bolt that has not undergone secondary fastening, and it is pointless to determine the fastening state of the secondary fastening for such a bolt. For this reason, the determination image generation unit 104 generates a determination image by cutting out area AR21, which shows the bolt with a broken pintail, from areas AR21 and AR22 detected in the target image IMG31, and does not use area AR22 in generating the determination image.

[0057] In this way, the area detection unit 103 may detect an area where a bolt with a broken pintail is captured and an area where a bolt with an unbroken pintail is captured, and the determination image generation unit 104 may generate an determination image by cutting out the area where a bolt with a broken pintail is captured from the target image.

[0058] As described above, the pintail of a torsion-type high-strength bolt breaks when it is tightened, and whether the pintail remains broken or not makes it possible to tell at a glance whether the bolt has been tightened or not. Therefore, with the above configuration, images of bolts in which the pintail is not broken and which are clearly not tightened can be excluded from the images to be used for judgment of the tightening state by judgment unit 107, thereby realizing efficient judgment of the tightening state.

[0059] The area detection unit 103 may sequentially detect an area in which a bolt is captured and determine whether the pintail of the bolt captured in the detected area is broken or not. In this case, the area detection unit 103 may use a detection model that detects an area in which a bolt is captured and a classification model that classifies the bolts captured in the detected area into bolts with broken pintails and bolts with non-broken pintails.

[0060] 5, the character string "OK" is displayed in the area of ​​the bolt where the pintail is broken, and "NG" is displayed in the area of ​​the bolt where the pintail is not broken, and next to this character string is displayed a numerical value indicating the accuracy (likelihood) of the detection result. Such a numerical value can be output by the detection model 111. Since there is a possibility that there is a problem in an area where the accuracy is low, such as the bolt not being clearly visible, the determination image generation unit 104 may generate the determination image by cutting out an area where the accuracy is equal to or greater than a predetermined threshold, and may not use an area where the accuracy is less than the threshold in generating the determination image.

[0061] [Selection of judgment images based on the degree of blur] The selection unit 106 selects from the determination images generated as described above, the image to be used for determining the fastening state based on the appearance of the bolt in the determination image. For example, the selection unit 106 may make the above selection based on the degree of blurring of the determination image. This makes it possible to prevent a decrease in the accuracy of determining the fastening state of the bolt, even if the determination images include images that are blurred to an extent that they are unsuitable for determination by the determination unit 107.

[0062] Fig. 6 is a diagram showing an example of selecting a determination image based on the degree of blur. Fig. 6 shows determination image IMG41 and determination image IMG42. Of these, determination image IMG41 is a blurred and unclear image, while the other determination image IMG42 is a clear image without blur. Naturally, when the blurred and unclear determination image IMG41 is used, the accuracy of the determination by determination unit 107 is more likely to be lower than when the clear determination image IMG42 is used.

[0063] For this reason, the selection unit 106 may not select the determination image IMG41 as the object to be determined by the determination unit 107, but may select the determination image IMG42 as the object to be determined by the determination unit 107. The method for evaluating the degree of blur is not particularly limited. For example, the selection unit 106 may select an image to be used for determination by using the variance of pixel values ​​in the outline of the bolt shown in the determination image as an index value indicating the degree of blur of the determination image.

[0064] This is because the degree of blurring of the determination image is reflected in the variance of pixel values ​​in the outline of the bolt in the determination image. Specifically, the greater the degree of blurring of the determination image, the smaller the variance of pixel values ​​in the outline of the bolt in the determination image. Therefore, by using the variance of pixel values ​​in the outline of the bolt as an index value indicating the degree of blurring of the determination image, the selection unit 106 can select a determination image with a degree of blurring that will not interfere with determination.

[0065] When selecting an image for determination based on the variance of pixel values ​​in the outline of the bolt, the selection unit 106 first extracts the outline of the bolt from the image for determination. For example, if the image for determination is a color image, the selection unit 106 may convert the image for determination into a black and white image and then apply an edge detection filter such as a Laplacian filter to extract the outline of the bolt.

[0066] The selection unit 106 may then calculate the variance of pixel values ​​in the extracted contour portion, and select an image for determination for which the calculated value is equal to or greater than a threshold value as a target for determination by the determination unit 107. The threshold value may be a fixed value set in advance, or may be set by the threshold setting unit 105 as described below.

[0067] The threshold setting unit 105 may set the above threshold according to the size of the bolt appearing in the determination image. The threshold setting unit 105 may also set the above threshold according to the shooting distance of the determination image.

[0068] Generally, the longer the shooting distance, the more likely the image is to be blurred, so the evaluation image captured at a long shooting distance is likely to be blurred. Also, the longer the shooting distance, the smaller the bolt size in the image captured, so the evaluation image in which the bolt is small is also likely to be blurred.

[0069] Therefore, as described above, a configuration may be adopted in which the threshold value of the index value is set according to the size of the bolt shown in the determination image or the shooting distance of the determination image, and determination images whose index value is equal to or greater than the set threshold are selected as determination targets by the determination unit 107. This makes it possible to accurately exclude determination images that are likely to be blurred and select appropriate images as determination targets for the determination unit 107.

[0070] Note that since the determination image is a cutout of the region in which each bolt is detected by the region detection unit 103, the size of the bolts in the determination image corresponds to the width and height of the determination image. Therefore, the threshold setting unit 105 may set a threshold by regarding the size of the determination image as the size of the bolts in the determination image. For example, the threshold setting unit 105 may set a threshold corresponding to the width, height, or the product or sum of the width and height of the determination image. Furthermore, when using the shooting distance, the threshold setting unit 105 may acquire the shooting distance measured by, for example, a distance sensor, and set a threshold corresponding to this shooting distance.

[0071] The smaller the size of the bolt in the determination image (the greater the shooting distance), the smaller the threshold value that the threshold setting unit 105 sets. For example, optimal threshold values ​​experimentally determined for each range of bolt sizes (or shooting distances) may be stored in advance in the storage unit 11. In this case, the threshold setting unit 105 may refer to the storage unit 11 and identify the optimal threshold value according to the size of the bolt in the determination image (or shooting distance).

[0072] [Determining the fastening state using a determination model] As described above, the judgment model 112 is a model generated by learning so that when feature quantities extracted from a group of images of bolts with different fastening states are embedded in a feature space, the distance between the feature quantities derived from a group of images with a common fastening state becomes small.

[0073] For example, when the determination model 112 is to determine which of the fastened states 31 to 35 shown in Fig. 3 the fastened state of a bolt shown in a determination image corresponds to, each of the fastened states 31 to 35 is set as a class to be determined, and a group of images belonging to each class is set as training data. Then, using this training data, the determination model 112 can be generated by learning so that the distance between feature amounts of data in the same class is small and the distance between feature amounts of data in different classes is large.

[0074] The images used as training data are images of the same size as the judgment image and are images of a known fastening state. Considering that the shooting distance and shooting location of the imaging device 21 are not constant, it is preferable to include in the training data images obtained by converting the contrast of an original image of a known fastening state or an image obtained by blurring the original image.

[0075] By using a blurred image as training data, it is possible to generate a determination model 112 that can output a valid determination result even if the determination image is slightly blurred. Of course, if the determination image is too blurred, it is conceivable that the determination accuracy will decrease even with such a determination model 112. In particular, since the image capturing device 21 of the head-mounted display 2 moves in conjunction with the movement of the user's head, there is a high possibility that a significantly blurred image will be captured, and the above-mentioned decrease in determination accuracy cannot be ignored. For this reason, the determination device 1 employs a configuration in which the selection unit 106 selects an image to be used for determining the fastening state based on the degree of blurring of the determination image, making it less likely that the above-mentioned decrease in determination accuracy will occur.

[0076] FIG. 7 is a diagram showing an example in which feature amounts extracted by the determination model 112 are embedded in a feature space. The feature space shown in FIG. 7 is a two-dimensional feature space with x on the horizontal axis and y on the vertical axis. Point P1 plotted in this feature space indicates a feature amount extracted from a determination image used to determine the fastening state. Meanwhile, points P2 to P6 indicate the center points of the feature amounts extracted from the image groups of each class. Specifically, points P2 to P6 indicate the center points of the feature amounts extracted from the image groups belonging to the classes of missed tightening, shaft rotation, normal, no mark, and co-rotation, respectively (hereinafter referred to as the center points of the classes).

[0077] A short distance in this feature space indicates a high probability that the features are common. Therefore, the determination unit 107 may calculate the distance between the feature extracted from the determination image and the center point of each class. In FIG. 7, the lengths of the dashed line segments P1-P2, P1-P3, P1-P4, P1-P5, and P1-P6 shown in G in the same figure correspond to the distances to be calculated. The determination unit 107 may then identify the center point with the shortest calculated distance and determine that the bolt in the determination image belongs to the class corresponding to that center point. In this case, in the example of FIG. 7, since the distance between P1 and P4 is shortest, the determination unit 107 determines that the bolt in the determination image belongs to the class corresponding to point P4, i.e., that the fastening state of the bolt is normal.

[0078] Furthermore, for example, the determination unit 107 may determine the fastening state based on whether point P1 falls within the range of each class that is set based on points P2 to P6, which are the center points of each class. For example, in FIG. 7H, a circle is drawn with point P2 at its center. This circle indicates the range of the class corresponding to point P2, i.e., the class in which the fastening state is "missed tightening." Similarly, each circle with points P3 to P6 at its center indicates the range of the class in which the fastening state is "rotational," "normal," "unmarked," and "rotational together," respectively.

[0079] 7, point P1 is not included in the range of any class, so the determination unit 107 determines that the fastening state cannot be determined. For a bolt determined to be undeterminable, the display control unit 108 may cause the display device 22 to display an image prompting the user to take another photograph of the bolt (or to point the photographing device 21 at the bolt and not move the photographing device 21).

[0080] The range of each class can be set independently. That is, as in the example of H in Figure 7, the range of the class corresponding to point P2 can be set wider than the ranges of the other classes. The range of each class can be set, for example, to reduce the rate of erroneous determination for each class, or to reduce the probability of erroneously determining that a bolt whose fastening state is not normal is normal (for example, the range of the normal class can be narrowed and the ranges of the other classes adjacent to it can be widened).

[0081] As described above, the determination unit 107 may determine the fastening state of a bolt using a determination model 112 generated by learning so that when feature amounts extracted from a group of images of bolts with different fastening states are embedded in a feature space, the distance between the feature amounts derived from a group of images with a common fastening state becomes small. This makes it possible to perform the highly difficult process of determining the fastening state with high accuracy.

[0082] The determination model 112 may be designed to output an output value indicating the determined class (for example, the confidence level of each class), or may be designed to output a feature value. The confidence level is a numerical value between 0 and 1 that indicates the likelihood of the determination result.

[0083] The decision model 112 can be generated by, for example, deep metric learning. Deep metric learning is a learning method for features embedded in a feature space such that the distance between features of data of the same class is small and the distance between features of data of different classes is large. During learning, the distance between features may be expressed as Euclidean distance or an angle.

[0084] It is also possible to use a model such as a convolutional neural network as the determination model 112. In this case, too, by learning using teacher data similar to that in the case of deep metric learning, it is possible to generate the determination model 112 that outputs an output value indicating which of a plurality of fastening states the input determination image corresponds to.

[0085] [Display of judgment results] Fig. 8 is a diagram showing an example of the display of the determination result of the determination device 1. Image IMG5 shown in Fig. 8 shows a plurality of bolts, and the areas in which the individual bolts detected by the area detection unit 103 are shown are indicated by frame lines. In Fig. 8, the frame line indicated by R3 indicates that the fastening state of the bolts surrounded by the frame line is normal. On the other hand, in Fig. 8, the frame line indicated by R4 indicates that the fastening state of the bolts surrounded by the frame line is not normal (specifically, they are rotating together).

[0086] Image IMG5 also includes a message R1 indicating that an improperly fastened bolt has been detected, and a legend R2 of the fastening state determination result. Specifically, message R1 is a message stating "abnormality detected." The content of message R1 may be anything that allows the user to recognize that an improperly fastened bolt has been detected.

[0087] Legend R2 shows five fastening states: normal, axial rotation, corotation, mistightened, and no mark. The display color of the text indicating each fastening state may be the same color or a similar color as the frame of the bolt determined to be in that fastening state. For example, when the display control unit 108 displays the word "normal" in legend R2 in blue, it may also use blue for the frame lines R3, etc., surrounding bolts determined to be normal. On the other hand, when the display control unit 108 displays the word "corotation" in legend R2 in red, it may also use red for the frame line R4 surrounding bolts determined to be corotation. This allows the user to easily recognize the fastening state determination results for each bolt.

[0088] The display format of the judgment results may be any format that allows the user to recognize the judgment results for each bolt. For example, instead of displaying legend R2, the display control unit 108 may display the judgment results for each bolt in association with that bolt.

[0089] When the image IMG5 is displayed on the display device 22 of the head-mounted display 2, the display control unit 108 may display the image IMG5, for example, near the edge of the user's field of view (for example, the upper right corner). In this case, the user can visually recognize the bolts through the display device 22 and can confirm the judgment results for those bolts from the image IMG5. Note that the image IMG5 may be generated based on the target image generated by the target image generation unit 102.

[0090] The display control unit 108 may also display the determination result at the position of the bolt that is visible through the display device 22. In this case, the user wearing the head-mounted display 2 sees the determination result superimposed on the bolt in their field of view. In other words, the user sees an image like image IMG5.

[0091] [Processing flow] The flow of the processing (determination method) executed by the determination device 1 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the processing executed by the determination device 1. Note that the following describes an example in which an image of a bolt is captured by the imaging device 21 provided in the head-mounted display 2 shown in Fig. 2, and the fastening state determination result is displayed on the display device 22 provided in the head-mounted display 2.

[0092] In S11, the image acquisition unit 101 acquires an image to be used for the determination. Specifically, the image acquisition unit 101 acquires an image of the bolt captured by the photographing device 21 from the head-mounted display 2. Note that if the image acquired from the head-mounted display 2 is a moving image, the image acquisition unit 101 acquires frame images that make up the moving image from the moving image.

[0093] In S12, the target image generation unit 102 generates a target image from the image acquired in S11. For example, the target image generation unit 102 may cut out an area of ​​a predetermined size in the center of the image acquired in S11 and use it as the target image (see area AR1 in image IMG2 in FIG. 4).

[0094] In S13, the area detection unit 103 detects areas in which each bolt appears from the target image generated in S12. The detection model 111 is used to detect areas in which each bolt appears. As described with reference to FIG. 5, in S13, the area detection unit 103 may detect areas in which bolts with broken pintails appear separately from areas in which bolts with unbroken pintails appear.

[0095] In S14, the determination image generation unit 104 determines which areas of the areas detected in S13 to exclude from the generation of the determination image. For example, assume that areas showing bolts with broken pintails and areas showing bolts with unbroken pintails are detected separately in S13. In this case, the determination image generation unit 104 determines to exclude the areas showing bolts with unbroken pintails. In addition, the determination image generation unit 104 may determine to exclude either or both of the areas detected in S13 that are located within the area along the outer edge of the target image and the areas whose size is outside a predetermined normal range.

[0096] In S15, the determination image generating unit 104 generates a determination image by cutting out the area that was not subject to exclusion in S14 from the area detected in S13 from the target image generated in S12.

[0097] In S16, the threshold setting unit 105 sets a threshold for each determination image generated in S15 to determine whether the selection unit 106 will select the determination image. For example, the threshold setting unit 105 may set the threshold according to the size of the bolt appearing in the determination image or the shooting distance of the determination image. Note that in S16, the threshold setting unit 105 may set the threshold by regarding the size of the determination image as the size of the bolt appearing in the determination image, as described above.

[0098] In S17, the selection unit 106 calculates an index value indicating the degree of blurring of each of the determination images generated in S15. For example, the selection unit 106 may calculate the variance of pixel values ​​in the outline of a bolt appearing in the determination image as the index value indicating the degree of blurring of the determination image.

[0099] In S18, the selection unit 106 selects an image to be used for determining the fastening state from among the determination images generated in S15, based on how the bolt appears in the determination image. For example, the selection unit 106 selects a determination image for which the index value calculated in S17 is equal to or greater than the threshold value set in S16, as the image to be used for determining the fastening state.

[0100] In S19, the determination unit 107 determines the fastening state of the bolt based on the image selected in S18. Specifically, the determination unit 107 determines the fastening state of the bolt shown in the determination image selected in S18 based on an output value obtained by inputting the determination image selected in S18 to the determination model 112. This process is performed for each of the determination images selected in S18.

[0101] In S20, the display control unit 108 displays the determination result of S19 on the display device 22. As described with reference to Fig. 8, the display control unit 108 may generate an image in which information (e.g., a frame line) indicating the area detected in S13 and information indicating the determination result of S20 are superimposed on the target image generated in S12, and display the image on the display device 22. Furthermore, the display control unit 108 may display information indicating that the pintail is not fractured in an area in the image in which a bolt with an unfractured pintail is shown.

[0102] When the process of S20 is completed, the process returns to the process of S11, and the image acquisition unit 101 acquires a new image. The new image is an image captured chronologically later than the previously acquired image, or a frame image extracted from the same video that is chronologically later than the previously acquired frame image. By repeating this process, the fastening state of the bolt within the user's field of view is displayed on the display device 22 in approximately real time.

[0103] Note that before making the determination in S19, a process for emphasizing the marking portion of the bolt may be performed. For example, the marking portion of the bolt can be emphasized by performing a sharpening process on the determination image. This is expected to improve the accuracy of the determination in S19. The process for emphasizing the marking portion may be performed by the determination unit 107, or a new processing block (emphasis unit) that performs this process may be provided.

[0104] As described above, the determination method according to this embodiment includes a selection step (S18) of selecting an image to be used for determining the fastening state from among a plurality of determination images showing the bolt whose fastening state is to be determined, based on how the bolt appears in the determination image, and a determination step (S19) of determining the fastening state of the bolt based on the image selected in S18. This makes it possible to prevent a decrease in the accuracy of determining the fastening state of the bolt, even if the determination images include images that are not suitable for determination.

[0105] [Modification] In the above embodiment, an example has been described in which the degree of blurring of each determination image is used to determine whether or not to select that determination image as an image to be used to determine the fastening state. However, this selection may be made based on how the bolt appears in the image, and is not limited to the above example in which selection is made based on the degree of blurring.

[0106] (Selection of judgement images based on their position / size on the target image) For example, the selection unit 106 may select, as an image to be used for determining the fastening state, a determination image that is cut out from an area other than an area located along the outer edge of the target image, from among the determination images generated by the determination image generation unit 104. In this case, the determination image generation unit 104 may generate a determination image from each area detected by the area detection unit 103, regardless of the position of the area detected by the area detection unit 103.

[0107] Similarly, the selection unit 106 may select, as an image to be used for determining the fastening state, a determination image whose size is within a predetermined normal range from among the determination images generated by the determination image generation unit 104. In this case, the determination image generation unit 104 may generate a determination image from each area detected by the area detection unit 103, regardless of the size of the area detected by the area detection unit 103.

[0108] In this way, when the determination image (first image) is an image generated by cutting out an area in which each bolt appears from a target image (third image) in which one or more bolts appear, the selection unit 106 may select a determination image cut out from an area other than an area located along the outer edge of the target image as the image to be used for determining the fastening state. Also, when the determination image (first image) is an image generated by cutting out an area in which each bolt appears from a target image (third image) in which one or more bolts appear, the selection unit 106 may select a determination image whose size is within a predetermined normal range as the image to be used for determining the fastening state.

[0109] (About images to be selected) In the above embodiment, an example has been described in which a determination image used to determine the fastening state is selected based on the degree of blurring of the determination image, but the images to be selected are not limited to the determination image. In other words, the selection unit 106 may select an image based on the degree of blurring of an image acquired by the image acquisition unit 101 or a target image generated from the image.

[0110] Specifically, when the selection unit 106 selects an image captured by the image capture device 21, it calculates an index value indicating the degree of blurring of the image acquired by the image acquisition unit 101. Then, the selection unit 106 compares the calculated index value with a predetermined threshold to determine whether or not to select the image as an image to be used for determining the fastening state. In this case, a target image is generated from the image selected by the selection unit 106, and a determination image is generated from the target image, and the fastening state is determined using the determination image. Note that the threshold may be set by the threshold setting unit 105. In this case, the threshold setting unit 105 may set the threshold according to the size of each bolt appearing in the image captured by the image capture device 21 or the shooting distance of the image.

[0111] In this way, the selection unit 106 may select an image (second image) to be used for determining the fastening state based on how the bolt appears in the image from among a plurality of images (first images) that show the bolts whose fastening state is to be determined and that have been taken by the photographing device 21. Note that, as described above, the selection criteria are not limited to the degree of blurring, and may be, for example, the position on the image where each bolt appears or the size of the area where each bolt appears.

[0112] Furthermore, when the selection unit 106 selects a target image generated by the target image generation unit 102, it calculates an index value indicating the degree of blurring of the target image. Then, the selection unit 106 compares the calculated index value with a predetermined threshold to determine whether or not to select the target image as an image to be used for determining the fastening state. In this case, a determination image is generated from the target image selected by the selection unit 106, and the fastening state is determined using the determination image. Note that the threshold may be set by the threshold setting unit 105. In this case, the threshold setting unit 105 may set the threshold according to the size of each bolt appearing in the target image or the shooting distance of the target image.

[0113] In this way, the selection unit 106 may select a target image (second image) to be used for determining the fastening state based on how the bolts appear in the target image from among the multiple target images (first images) generated by the target image generation unit 102. Note that, as described above, the selection criteria are not limited to the degree of blurring, and may be, for example, the position at which each bolt appears on the target image or the size of the area in which each bolt appears.

[0114] (About the entity that performs the processing) Furthermore, the execution entities of each process described in the above-described embodiment are arbitrary and are not limited to the above-described examples. In other words, the functions of the determination device 1 can be substituted by a plurality of information processing devices (which can also be called processors) that can communicate with each other. For example, by distributing the blocks shown in FIG. 1 among a plurality of information processing devices, a system having the same functions as the determination device 1 can be constructed. Therefore, the execution entities of the determination method shown in FIG. 9 can also be a plurality of information processing devices.

[0115] [Software implementation example] The functions of the judgment device 1 can be realized by a program (judgment program) that causes a computer to function as the judgment device 1, and that causes a computer to function as each control block of the judgment device 1 (particularly each part included in the control unit 10).

[0116] In this case, the determination device 1 includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0117] The program may be stored non-transitoryly in one or more computer-readable storage media. The storage media may or may not be included in the determination device 1. In the latter case, the program may be supplied to the determination device 1 via any wired or wireless transmission medium.

[0118] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0119] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0120] 1 Judgment device 103 Area detection unit 104 Determination image generation unit (image generation unit) 105 Threshold setting unit 106 Selection section 107 Judgment section 112 Decision Model 2. Head-mounted display 21 Imaging equipment 22 Display device 5 Judgment System

Claims

1. a selection unit that selects a second image to be used for determining the fastening state from a plurality of first images that show a bolt whose fastening state is to be determined, based on how the bolt is shown in the first image; a determination unit that determines a fastening state of the bolt based on the second image selected by the selection unit; and an area detection unit that detects areas in which each of the bolts is captured from a third image in which one or more of the bolts are captured; an image generating unit that generates the first image by cutting out an area in which each of the bolts is captured from the third image, The area detection unit detects an area in which the bolt with a broken pintail is photographed and an area in which the bolt with an unbroken pintail is photographed, The image generation unit generates the first image by cutting out an area in which the bolt with the broken pintail is captured from the third image.

2. A selection unit that selects a second image to be used for determining the fastening state from a plurality of first images that show the bolt to be determined the fastening state based on how the bolt is shown in the first image; a determination unit that determines a fastening state of the bolt based on the second image selected by the selection unit, The judgment unit performs the judgment using a judgment model generated by learning using training data including images of known fastened states and images in which the fastened states are blurred, so that when feature amounts extracted from a group of images of the bolts in different fastened states are embedded in a feature space, the distance between the feature amounts derived from a group of images in which the fastened states are common becomes small.

3. The determination device according to claim 1 , wherein the selection unit selects the second image based on a degree of blurring of the first image.

4. 4. The determination device according to claim 3, wherein the selection unit selects the second image using a variance of pixel values ​​in an outline portion of the bolt shown in the first image as an index value indicating a degree of blurring of the first image.

5. a threshold setting unit that sets a threshold value of the index value according to a size of the bolt shown in the first image or a shooting distance of the first image, The determination device according to claim 4 , wherein the selection unit selects the first image for which the index value is equal to or greater than the threshold value as the second image.

6. The determination device according to claim 1 , wherein the image generation unit generates the first image by cutting out an area other than an area located along an outer edge of the third image from the area detected by the area detection unit.

7. The determination device according to claim 1 , wherein the image generation unit generates the first image by cutting out an area, the size of which is within a predetermined normal range, from the area detected by the area detection unit.

8. a head-mounted display including an imaging device and a display device; The determination device according to claim 1 or 2, The judgment device uses images captured by the imaging device to judge the fastening state of bolts within the field of view of a user wearing the head-mounted display, and displays the results of the judgment on the display device.

9. A determination method executed by one or more information processing devices, a selection step of selecting a second image to be used for determining the fastening state from a plurality of first images in which the bolt, the fastening state of which is to be determined, is captured based on how the bolt is captured in the first image; a determination step of determining a fastening state of the bolt based on the second image selected in the selection step; a region detection step of detecting regions in which each bolt appears from a third image in which one or more of the bolts appear; an image generating step of generating the first image by cutting out an area in which each of the bolts appears from the third image, In the area detection step, an area in which the bolt with a broken pintail is photographed is distinguished from an area in which the bolt with an unbroken pintail is photographed and detected, In the image generating step, the first image is generated by cutting out an area of ​​the bolt with its pintail broken from the third image.

10. A determination method executed by one or more information processing devices, comprising: a selection step of selecting a second image to be used for determining the fastening state from a plurality of first images in which the bolt, the fastening state of which is to be determined, is captured based on how the bolt is captured in the first image; a determination step of determining a fastening state of the bolt based on the second image selected in the selection step, In the judgment step, the judgment is made using a judgment model generated by learning using training data including images of known fastened states and images in which the fastened states are blurred, so that when feature amounts extracted from a group of images of the bolts in different fastened states are embedded in a feature space, the distance between the feature amounts derived from a group of images in which the fastened states are common becomes small.

11. 2. A determination program for causing a computer to function as the determination device according to claim 1, the determination program causing a computer to function as the selection unit, the determination unit, the area detection unit, and the image generation unit.

12. A judgment program for causing a computer to function as the judgment device described in claim 2, which causes a computer to function as the selection unit and the judgment unit.

Images