Method, device and electronic equipment for detecting bolt loosening state

Abstract

The application provides a bolt loosening state detection method, device and electronic equipment, and belongs to the technical field of image detection. A region of interest image is respectively input into two different processing branches. One branch uses instance segmentation to obtain a nut mask expressing a physical shape boundary. The other branch uses color space conversion and feature extraction to obtain a lockwire mark mask expressing color attributes. The nut mask and the lockwire mark mask are fused in spatial position, and effective mark boundary point coordinates are accurately calculated. A relative displacement calculated by the coordinates is normalized in scale by using a bounding box data, a unified alignment index is obtained, and the loosening state detection is realized. Without manual intervention and external sensors, the accuracy and robustness of bolt loosening state recognition are greatly improved. In this process, the loosening state detection can be realized by using an industrial camera to shoot the bolt, the implementation cost is reduced, and the detection efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a method, apparatus, and electronic device for detecting the state of loose bolts. Background Technology

[0002] Bolts are prone to loosening when exposed to vibration, temperature differences, aging, and mechanical stress. To facilitate observation of whether bolts have loosened, anti-loosening marks are often drawn, and personnel conduct regular inspections to confirm whether the bolts have become loose.

[0003] However, while manual inspection is reliable, it is costly and inefficient when there are many bolts, bolts are installed at high positions, or bolts are located in dangerous areas. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for detecting bolt looseness, which solves the problems of high cost and low efficiency in the existing technology for detecting and inspecting bolt tightness, and achieves efficient inspection of bolt tightness.

[0005] This invention provides a method for detecting the loosening state of bolts, comprising the following steps.

[0006] Obtain a region of interest image containing the entire bolt and nut image, wherein the region of interest image is obtained by performing overall bolt and nut detection and cropping on the first image; The region of interest image is segmented to obtain a nut mask; The region of interest image is subjected to color space conversion to obtain a second image represented in hue-saturation-brightness space, and anti-loosening line mark detection is performed based on the second image to obtain an anti-loosening line mark mask; Based on the nut mask and the anti-loosening line marking mask, determine the coordinates of the effective marking boundary point corresponding to each nut; Based on the coordinates of the effective marked boundary points and the bounding box data corresponding to the nut mask, the alignment index of the nut is determined; The looseness of the bolt is determined based on the alignment index of the nut.

[0007] According to a method for detecting bolt loosening state provided by the present invention, the step of determining the alignment index of the nut based on the coordinates of the effective marked boundary points and the bounding box data corresponding to the nut mask includes: Based on the coordinates of the effective marked boundary points, determine the first boundary point on the first nut and the second boundary point on the second nut; Calculate the Euclidean distance between the position coordinates of the first boundary point and the position coordinates of the second boundary point to obtain the nut spacing between the first nut and the second nut; Based on the bounding box data corresponding to the nut mask, determine the maximum feature size corresponding to the first nut or the second nut; The ratio of the nut spacing to the maximum feature size is used as the alignment index of the nut; The first nut and the second nut are two adjacent nuts.

[0008] According to a method for detecting bolt loosening provided by the present invention, the step of acquiring a region of interest image containing the entire image of the bolt and nut includes: The first image is input into a pre-trained detection network to perform overall bolt and nut detection, and the bounding box data containing the overall bolt and nut image is obtained from the output of the detection network. The first image is cropped based on the bounding box data to obtain a region of interest image that includes the entire image of the bolt and nut.

[0009] According to the present invention, a method for detecting bolt loosening status, wherein detecting anti-loosening line marks based on a second image to obtain an anti-loosening line mark mask includes: Obtain the first hue-saturation-brightness range, where the first hue-saturation-brightness range is a predefined hue-saturation-brightness range to which the anti-loosening line mark belongs; Based on the first hue-saturation-brightness range, the second image is segmented by thresholding to obtain the anti-loosening line mark mask.

[0010] According to a method for detecting bolt loosening conditions provided by the present invention, the method further includes: If the anti-loosening line mark is not detected, output the first detection conclusion; The first detection conclusion indicates that no loose bolts were detected.

[0011] According to the present invention, a method for detecting bolt loosening includes determining the coordinates of the effective mark boundary points corresponding to each nut based on the nut mask and the anti-loosening line mark mask, comprising: The outer contour boundary points of the nut mask are extracted to obtain the first candidate point set; The first candidate point set and the anti-loosening line mark mask are spatially intersected to obtain a candidate intersection point set. Each candidate intersection point in the candidate intersection point set is located within the first candidate point set and belongs to the anti-loosening line mark mask. The coordinates of candidate intersection points that satisfy the first condition are selected from the candidate intersection point set and used as the coordinates of the valid marked boundary points corresponding to the nut. Wherein, the first condition is that the vector angle between the first vector and the second vector is less than or equal to 60°, the first vector is the vector from the geometric center point of the nut mask to the candidate intersection point, and the second vector is the vector from the geometric center point of the nut mask to the geometric center point of the adjacent nut mask.

[0012] According to a method for detecting bolt loosening state provided by the present invention, the step of determining the bolt loosening state based on the alignment index of the nut includes: Based on the alignment index of the nut being less than or equal to a preset alignment index threshold, the loose state of the bolt is determined to be a tight state. Based on the fact that the alignment index of the nut is greater than the preset alignment index threshold, the bolt is determined to be in a loose state.

[0013] According to a method for detecting bolt loosening conditions provided by the present invention, the method further includes: If there is only one valid marker boundary point coordinate corresponding to a nut, output the second detection conclusion; The second detection conclusion indicates that only one side of the nut has valid marked boundary point coordinates.

[0014] The present invention also provides a device for detecting the loosening state of bolts, comprising the following modules: The acquisition module is used to acquire a region of interest image containing the overall image of the bolt and nut, wherein the region of interest image is obtained by performing overall bolt and nut detection and cropping on the first image; The segmentation module is used to perform instance segmentation on the region of interest image to obtain a nut mask; The conversion detection module is used to perform color space conversion on the image of the region of interest to obtain a second image represented in hue-saturation-brightness space, and to perform anti-loosening line mark detection based on the second image to obtain an anti-loosening line mark mask; The first determining module is used to determine the coordinates of the effective marking boundary points corresponding to each nut based on the nut mask and the anti-loosening line marking mask; The second determining module is used to determine the alignment index of the nut based on the coordinates of the effective marked boundary points and the bounding box data corresponding to the nut mask. The third determining module is used to determine the loosening state of the bolt based on the alignment index of the nut.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the bolt loosening detection method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the bolt loosening detection method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a method for detecting bolt loosening as described above.

[0018] This invention provides a method, apparatus, and electronic device for detecting bolt loosening. By cropping a first image containing global data, a region of interest (ROI) image is obtained, eliminating a large amount of invalid background data from the data processing input and reducing the algorithm's memory footprint. The ROI image is input into two different processing branches: one uses instance segmentation to obtain a nut mask representing the physical shape boundary, and the other uses color space conversion and feature extraction to obtain a mask representing the anti-loosening line markings. This multimodal feature decoupling data processing effectively avoids the coupling interference of illumination changes on the target boundary and color extraction. Furthermore, the geometric boundary data (nut mask) and color region data (anti-loosening line marking mask) are spatially fused to accurately calculate the coordinates of effective marking boundary points. The relative displacement calculated from the coordinates is then normalized using the bounding box data to obtain a unified alignment index. This transforms the purely pixel-dimensional visual features in the image into geometric judgment parameters that conform to physical engineering realities, significantly improving the accuracy and robustness of bolt loosening state identification without manual intervention or reliance on external sensors.

[0019] In this process, the loosening state of bolts can be detected by using an industrial camera to photograph them, which reduces the implementation cost and improves the detection efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts illustrating the bolt loosening detection method provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the process of determining the alignment index of a nut based on the coordinates of effectively marked boundary points and the bounding box data corresponding to the nut mask, as provided by this invention.

[0023] Figure 3 This is a schematic diagram of the process for obtaining a region of interest image containing an overall image of a bolt and nut, provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the process provided by the present invention for detecting anti-loosening line marks based on a second image to obtain an anti-loosening line mark mask.

[0025] Figure 5 This is a flowchart illustrating the process of determining the coordinates of the effective marking boundary points corresponding to the nut based on the nut mask and the anti-loosening line marking mask provided by the present invention.

[0026] Figure 6 This is the second flowchart of the bolt loosening detection method provided by the present invention.

[0027] Figure 7 This is a schematic diagram of the bolt ROI image provided by the present invention.

[0028] Figure 8 This is a schematic diagram of the nut mask provided by the present invention.

[0029] Figure 9 This is a schematic diagram of the bolt ROI-HSV image provided by the present invention.

[0030] Figure 10 This is a schematic diagram of the anti-loosening marking area mask provided by the present invention.

[0031] Figure 11 This is a schematic diagram of the detection in a real working scenario provided by the present invention.

[0032] Figure 12 This is a schematic block diagram of the bolt loosening detection device provided by the present invention.

[0033] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0034] Figure label: 1201: Acquisition module; 1202: Segmentation module; 1203: Conversion detection module; 1204: First determination module; 1205: Second determination module; 1206: Third determination module; 1310: Processor; 1320: Communication interface; 1330: Memory; 1340: Communication interface. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0036] The present invention provides a method for detecting the loose state of bolts. The subject executing the method can be a computer device with image data processing capabilities, such as a local server, cloud server, industrial inspection terminal, portable computer, or computing module mounted on a drone. This application embodiment does not specifically limit this.

[0037] The following is combined with Figures 1 to 13 The present invention describes a method, apparatus, and electronic device for detecting bolt loosening.

[0038] Figure 1 This is one of the flowcharts illustrating the bolt loosening detection method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain the region of interest image containing the overall image of the bolt and nut. The region of interest image is obtained by performing overall bolt and nut detection and cropping on the first image.

[0039] In some embodiments, during routine inspections of industrial equipment, images of the equipment surface are typically captured using industrial cameras, drones, or mobile terminals. These images serve as input data for this solution, namely the aforementioned first image. The first image is usually a global raw image containing various background environments and multiple mechanical parts. To eliminate interference from complex backgrounds and focus on the detection target, the computer device first performs a global bolt and nut detection on the first image. The purpose of the global detection is to locate the overall area containing the bolt head, nut, and any possible gaskets or other structures within the global image. After locating the specific position information, the computer device performs a cropping and extraction operation on the first image based on this position information, thereby generating multiple independent Region of Interest (ROI) images. This ROI image essentially reduces the complexity of the global first image to a local image slice focusing only on a single bolt and its adjacent area. This pre-extraction method can significantly reduce the computational complexity of subsequent pixel-level processing.

[0040] Step 102: Perform instance segmentation on the region of interest image to obtain the nut mask.

[0041] After acquiring the region of interest (ROI) image, since anti-loosening lines are typically drawn between the nut and bolt heads or between adjacent nuts, the computer equipment needs to further extract the specific physical component outlines from the image. The computer equipment performs instance segmentation on the ROI image. Instance segmentation not only distinguishes between foreground and background at the pixel level but also differentiates different individuals of the same category. Through instance segmentation, the computer equipment accurately extracts the nut pixel region from the ROI image, forming an independent nut mask. This nut mask is essentially a binary two-dimensional matrix, where the pixel positions belonging to the nut are assigned specific values, thus achieving a digital representation of the nut's physical boundaries and providing precise geometric constraints for subsequent boundary intersection calculations.

[0042] Step 103: Perform color space conversion on the region of interest image to obtain a second image represented in hue-saturation-luminance space, and perform anti-loosening line mark detection based on the second image to obtain an anti-loosening line mark mask.

[0043] In addition to extracting the shape of the physical parts, the computer equipment also needs to extract information about the anti-loosening line markings. Since anti-loosening lines are usually artificially applied with specific colored paint or markings, they are easily affected by environmental factors such as uneven lighting and shadows in common red, green, and blue primary color images. Therefore, the computer equipment converts the region of interest image to a hue-saturation-luminance space to generate a second image. In the hue-saturation-luminance space, the type of color is decoupled from brightness and color purity, allowing the computer equipment to more reliably identify anti-loosening lines of specific colors. Based on predefined color feature parameters, the computer equipment performs region extraction on the second image, separating pixel regions that match the color characteristics of the anti-loosening lines to generate an anti-loosening line marking mask. This mask is also a pixel-level binary matrix, representing the precise digital area covered by the anti-loosening line paint in the image.

[0044] Step 104: Based on the nut mask and the anti-loosening line mark mask, determine the coordinates of the effective mark boundary point corresponding to each nut.

[0045] After acquiring the nut mask representing the physical part's extent and the anti-loosening line mark mask representing the paint coverage area, the computer equipment needs to fuse these two independent features. The anti-loosening line mark can only serve as a reference point for determining looseness when it crosses the nut's edge. Therefore, the computer equipment determines the specific crossing position of the anti-loosening line mark on each nut edge by performing spatial intersection operations and logical filtering on the nut mask and the anti-loosening line mark mask. The pixel coordinates that are retained after filtering and accurately reflect the relative position of the anti-loosening line are the effective mark boundary point coordinates. This process transforms the originally scattered contour pixels and color pixels into point data with physical reference significance through spatial positional relationships.

[0046] Step 105: Determine the alignment index of the nut based on the coordinates of the valid marked boundary points and the bounding box data corresponding to the nut mask.

[0047] After determining the coordinates of the effective marker boundary points, the computer equipment needs to quantify the relative displacement of the markers on different parts. Since the bolts are captured at different distances and scales in different images, simply comparing the absolute distances between pixels cannot accurately reflect the true degree of looseness. Therefore, the computer equipment combines the bounding box data corresponding to the nut mask for evaluation. The bounding box data reflects the pixel scale of the nut in the current image. The computer equipment uses the coordinates of the effective marker boundary points on different nuts to calculate their relative offsets and incorporates the nut's bounding box data for scale normalization, thereby calculating a dimensionless value independent of the image capture distance—the nut alignment index. This index objectively reflects the degree of breakage and misalignment of the anti-loosening line markers.

[0048] Step 106: Determine the bolt loosening status based on the nut alignment index.

[0049] Finally, the computer equipment compares the calculated alignment index with the preset judgment logic. Since the alignment index has been standardized in the previous processing step, its value directly reflects whether the anti-loosening line is aligned at the physical level. Based on the value range of this index, the computer equipment can automatically output the detection conclusion of whether the bolt is in a normal tightening state or has become loose, completing the entire vision-based automated detection process.

[0050] In this embodiment, the region of interest (ROI) image is obtained by cropping the first image containing the global device, thus eliminating a large amount of invalid background data from the input of the data processing and reducing the memory usage of the algorithm. By inputting the ROI image into two different processing branches, one branch uses instance segmentation to obtain a nut mask representing the physical shape boundary, and the other branch uses color space conversion and feature extraction to obtain a loosening line mark mask representing the color attribute. This multimodal feature decoupling data processing method effectively avoids the coupling interference of illumination changes on the target boundary and color extraction. Furthermore, the geometric boundary data (nut mask) and color region data (loosening line mark mask) are spatially fused to accurately calculate the coordinates of the effective mark boundary points. The relative displacement calculated by the coordinates is scaled using the bounding box data to obtain a unified alignment index. This transforms the purely pixel-dimensional visual features in the image into geometric judgment parameters that conform to the actual physical engineering. Without human intervention and without relying on external sensors, the accuracy and robustness of bolt loosening state recognition are greatly improved.

[0051] In this process, the loosening state of bolts can be detected by using an industrial camera to photograph them, which reduces the implementation cost and improves the detection efficiency.

[0052] In some embodiments, such as Figure 2 As shown, based on the coordinates of the validly marked boundary points and the bounding box data corresponding to the nut mask, the alignment index of the nut is determined, including: Step 201: Based on the valid marked boundary point coordinates, determine the first boundary point on the first nut and the second boundary point on the second nut, wherein the first nut and the second nut are two adjacent nuts.

[0053] In a single bolt structure, there are often two mating nuts, or the bolt head and nut are adjacent to each other. For ease of description, these are collectively referred to as the adjacent first nut and second nut. After the computer equipment obtains the coordinates of the valid mark boundary points corresponding to each nut from the previous processing step, it needs to extract representative reference points for distance calculation for these two adjacent parts. Specifically, the computer equipment will perform coordinate calculations on the coordinates of multiple valid mark boundary points distributed on the surfaces of the first nut and the second nut, respectively. For example, a unique first boundary point (such as point P1) can be obtained by calculating the average position (i.e., the coordinate mean) of all valid mark boundary point coordinates on the first nut; similarly, a unique second boundary point (such as point P2) can be obtained by calculating the average position of all valid mark boundary point coordinates on the second nut. These two boundary points represent the core breakpoint positions of the anti-loosening line markings at the edges of the two adjacent nuts.

[0054] Step 202: Calculate the Euclidean distance between the position coordinates of the first boundary point and the position coordinates of the second boundary point to obtain the nut spacing between the first nut and the second nut.

[0055] After determining the specific two-dimensional coordinates of the first and second boundary points, the computer device uses a geometric distance calculation formula in the image pixel coordinate system to calculate the Euclidean distance between the position coordinates of the first and second boundary points. This Euclidean distance actually reflects the absolute pixel separation length of the two parts of the anti-loosening line mark in the two-dimensional image space after physical cutting (i.e., crossing the seam between two adjacent nuts). The computer device uses this calculated absolute pixel length as the nut spacing (e.g., spacing d) between the first and second nuts.

[0056] Step 203: Based on the bounding box data corresponding to the nut mask, determine the maximum feature size corresponding to the first nut or the second nut.

[0057] Because drones or industrial cameras may vary in focal length, shooting distance, and viewing angle when photographing different bolts, bolts of the same physical size may occupy different pixel sizes in the image. Directly using the calculated pixel-level nut spacing to determine looseness would lead to serious misjudgments. Therefore, the computer needs to extract a baseline scale. The computer reads the bounding box data corresponding to the mask of the first or second nut, which includes the width and height information of the enclosing rectangle. By comparing the width and height values, the computer selects the maximum value as the maximum feature size (e.g., dimension L) of the first or second nut in this image. This maximum feature size represents the pixel-level scale baseline of the current nut in the image.

[0058] Step 204: Use the ratio of the nut spacing to the maximum feature size as the nut alignment index.

[0059] After obtaining the absolute pixel displacement (nut spacing) and the image scale reference (maximum feature size), the computer device performs a division operation, dividing the nut spacing by the maximum feature size. Through this ratio calculation, the computer device converts the absolute distance in pixels into a dimensionless relative proportion. The computer device uses this ratio (e.g., α = d / L) as the final alignment index characterizing the degree of misalignment of the anti-loosening line markings on adjacent nuts.

[0060] In this embodiment, representative first and second boundary points are obtained by calculating the coordinates of effective marker boundary points on adjacent first and second nuts, and the Euclidean distance between them is calculated to obtain the nut spacing, thus accurately quantifying the disconnection displacement of the anti-loosening marker at the pixel level. By introducing the maximum feature size in the nut mask bounding box data and performing a ratio calculation between the nut spacing and the maximum feature size, a scale normalization process is achieved at the data algorithm level. This combined processing method of feature data successfully eliminates the interference of pixel scale differences caused by different shooting distances, resolutions, and scaling ratios in different images, converting the absolute pixel distance into a dimensionless alignment index independent of the image scale. This gives the subsequent loosening state determination logic extremely strong engineering generalization ability and robustness.

[0061] In some embodiments, such as Figure 3 As shown, the region of interest image containing the entire bolt and nut image is obtained, including: Step 301: Input the first image into the pre-trained detection network to perform overall bolt and nut detection, and obtain the bounding box data containing the overall bolt and nut image output by the detection network.

[0062] In some embodiments, the pre-trained detection network is a deep learning object detection model pre-trained on a large-scale industrial dataset containing various bolt types, different lighting conditions, and shooting angles. For example, it can employ network structures such as YOLO, Faster R-CNN, or EfficientDet. This detection network adaptively extracts features and performs pattern matching on the first image, identifies the fastener assembly consisting of the bolt head and nut, and outputs bounding box data containing the overall image of the bolt and nut. This bounding box data (often called a Bounding Box) is a set of coordinate values ​​used to accurately define the position and size of the outer rectangular outline of each bolt assembly in the pixel coordinate system of the two-dimensional image.

[0063] Step 302: Crop the first image based on the bounding box data to obtain a region of interest image that includes the overall image of the bolt and nut.

[0064] To focus subsequent analysis on the effective region, the computer device performs a local region cropping operation on the original first image based on the specific pixel coordinates and size range indicated by the bounding box data. For example, using each bounding box as a reference, the computer device extracts all pixels within the bounding box and can appropriately expand outwards by a certain proportion of pixels to include neighboring regions, thereby generating an independent region of interest image that includes the overall image of the bolt and nut. Through this cropping step, the computer device completes the construction of a local image slice for a single bolt target.

[0065] In this embodiment, the first global image is input into a pre-trained detection network with strong generalization capabilities to obtain bounding box data. Based on the bounding box data, pixel cropping is performed on the first image in spatial dimension. At the data processing level, dimensionality reduction and filtering from global complex industrial background pixels to high-value local target pixels are achieved. This processing mechanism enables the system to effectively remove interference data such as lighting and background that are irrelevant to bolt detection in the early stage, and to segment the high-resolution whole image into several smaller region of interest images with highly focused targets. This not only significantly reduces the memory usage and computational complexity of subsequent pixel-level processing algorithms such as instance segmentation, but also improves the computational efficiency and accuracy of the overall bolt loosening detection and analysis framework from the input end.

[0066] In some embodiments, such as Figure 4 As shown, based on the second image, the anti-loosening line mark detection is performed to obtain the anti-loosening line mark mask, including: Step 401: Obtain the first hue-saturation-brightness range. The first hue-saturation-brightness range is the hue-saturation-brightness range to which the predefined anti-loosening line mark belongs.

[0067] Because the anti-loosening markings on bolts in industrial settings are typically of specific colors (such as red, yellow, or green), computer equipment first needs to acquire predefined color threshold parameters, i.e., the first hue-saturation-value range, in order to accurately identify these markings in an image. This range is essentially a data set consisting of the upper and lower limits of the values ​​for the hue, saturation, and value channels. In practice, the computer equipment can acquire this range in various ways. For example, it can directly read a fixed value range preset for the current inspection task from a local configuration file; it can also automatically retrieve the corresponding marking color threshold from an industrial standard database on a cloud server based on the model of the equipment being inspected; or it can receive dynamically adjusted parameters input by engineers through an interactive interface. For instance, if the industrial equipment uses typical red anti-loosening paint, the first hue-saturation-value range acquired by the computer equipment can be defined as follows: hue values ​​within the union of [0°, 10°] and [170°, 180°], saturation values ​​greater than 0.6, and value greater than 0.4. This range precisely covers the essential color attributes of the red anti-loosening mark under different lighting conditions.

[0068] Step 402: Based on the first hue-saturation-brightness range, perform threshold segmentation on the second image to obtain the anti-loosening line mark mask.

[0069] After obtaining the first hue-saturation-brightness range as the color judgment standard, the computer device applies it to the second image obtained from the aforementioned conversion (i.e., the image in the hue-saturation-brightness space) to perform pixel-by-pixel data comparison and filtering. This process is called threshold segmentation. Specifically, the computer device traverses each pixel in the second image matrix, extracts the corresponding hue, saturation, and brightness channel values, and compares these values ​​with the upper and lower threshold values ​​corresponding to the first hue-saturation-brightness range. If the values ​​of all three channels of a pixel strictly fall within the range, the computer device determines that the pixel belongs to the anti-loosening line marking area and sets the pixel value at that position in the output matrix to a logical value representing the foreground (such as 1 or 255). Conversely, if any channel value exceeds the range, it is determined to be a background or interference area, and the pixel value at the corresponding position is set to a logical value representing the background (such as 0). Through this data filtering process based on multi-channel threshold conditions, computer equipment can map a richly colored continuous image into a binary matrix containing only the target and background. This allows for the automatic and accurate extraction of the coverage area of ​​the coating markings, ultimately generating a mask for the anti-loosening line markings. This mask data requires no manual annotation or complex template matching, directly providing a clear digital mask for subsequent geometric morphology analysis.

[0070] In this embodiment, a predefined first hue-saturation-brightness range is obtained, providing clear numerical constraints for image feature filtering. Subsequently, threshold segmentation is performed on the second image based on this range. At the data processing level, pixel-by-pixel numerical comparison is used to retain pixels that conform to specific color channel distribution characteristics and remove background noise pixels that do not conform to the color distribution. This threshold filtering method based on multi-channel decoupled data (hue, saturation, and brightness are judged independently) significantly reduces data deviations caused by lighting changes, metallic reflections, and shadows compared to direct color extraction in the RGB space. This feature processing not only reduces the computational overhead of the feature extraction algorithm but also gives the final generated anti-loosening line marking mask a very high signal-to-noise ratio and anti-interference capability, thus laying a highly reliable mask data foundation for subsequent accurate calculation of marking boundary points and positioning bolt misalignment distances.

[0071] In some embodiments, the method for detecting bolt loosening further includes: If no anti-loosening line markings are detected, output the first detection conclusion; The first test result indicates that no loose bolts were detected.

[0072] In actual industrial inspection scenarios, due to equipment being exposed to outdoor or harsh environments for extended periods, some bolts may not have been originally marked with anti-loosening lines, or the original anti-loosening lines may have completely fallen off due to aging, weathering, or external friction, or may be severely obscured by light or soiled, making them completely unrecognizable in the image. From a data processing perspective, when the computer performs threshold segmentation on the second image according to the set first hue-saturation-brightness range, if the generated anti-loosening line mark mask does not contain any valid pixels, or the number of valid pixels is lower than the set minimum threshold (i.e., the mask matrix is ​​approximately an empty set), the computer determines that no anti-loosening line mark has been detected.

[0073] In the case of such data gaps, if the system is forced to continue executing subsequent boundary point coordinate extraction and alignment index calculations, not only will the calculation formulas fail or generate invalid noise data due to the lack of key data objects, but it is also highly likely to output incorrect looseness or tightness judgment results. Therefore, the computer equipment will promptly interrupt the subsequent calculation process of the current region of interest image and directly output the first detection conclusion. This first detection conclusion is a special status label, whose specific semantics are used to indicate the missing status of the anti-loosening line marking. For example, the system can mark the bolt as marked as undetected on the final inspection report or monitoring terminal, thereby prompting maintenance personnel that the bolt cannot be automatically judged as loose by the current visual anti-loosening line logic and requires manual review or repainting of the anti-loosening line marking.

[0074] In this embodiment, by introducing a logical judgment branch based on the effective pixel data amount of the mask after generating the anti-loosening line marking mask, the subsequent data processing flow is blocked in time when the mask data matrix is ​​empty or invalid. This effectively avoids abnormal problems such as division by zero errors or calculation crashes caused by the algorithm substituting meaningless background noise into the coordinate and spatial distance calculation formula. At the same time, by using the output of a dedicated first detection conclusion, such data missing samples are strictly isolated from normal loose or tight samples. This ensures that the loosening state judgment conclusions output by the system are all based on valid feature data, thereby greatly improving the fault tolerance and reliability of the output results of the entire algorithm model in complex and uncontrollable real industrial inspection environments.

[0075] In some embodiments, such as Figure 5 As shown, based on the nut mask and the anti-loosening line mark mask, the coordinates of the effective mark boundary points corresponding to the nut are determined, including: Step 501: Extract the outer contour boundary points of the nut mask to obtain the first candidate point set.

[0076] Specifically, after obtaining the nut mask output through instance segmentation, edge detection algorithms (such as Canny edge detection algorithm, contour tracking extraction algorithm, etc.) are used to traverse and calculate the pixel matrix of the binarized mask. By identifying the abrupt changes between foreground and background pixels in the mask matrix, the computer device can accurately extract the pixel coordinates representing the physical outer edge of the nut. These extracted pixel coordinates located at the outermost edge of the nut are combined to form the first candidate point set. The data in this set delineates the closed outer contour of the nut in the image from a pixel perspective, providing a precise basic search range for subsequently finding the intersection of the anti-loosening line and the nut boundary.

[0077] Step 502: Perform spatial intersection processing on the first candidate point set and the anti-loosening line mark mask to obtain a candidate intersection point set. Each candidate intersection point in the candidate intersection point set is located within the first candidate point set and belongs to the anti-loosening line mark mask.

[0078] After obtaining the first set of candidate points representing the physical edge, the computer device needs to further determine the specific location of the anti-loosening line within the nut's outline. At this point, the computer device compares the pixel coordinate data in the first set of candidate points with the two-dimensional matrix of the anti-loosening line marker mask, which represents the coverage area of ​​the anti-loosening line's color. Specifically, the computer device examines each boundary pixel in the first set of candidate points, determining whether the pixel corresponding to its coordinate position is within the valid region (i.e., the pixel value is a specific value representing the anti-loosening line target) in the anti-loosening line marker mask matrix. Through this spatial intersection operation, the computer device retains the pixels that are both located on the outer contour of the nut and covered by the digital area of ​​the anti-loosening line's paint, forming a candidate intersection set. This candidate intersection set initially locates the spatial crossing point of the anti-loosening line when it passes through the edge of the nut.

[0079] Step 503: Select the coordinates of candidate intersection points that meet the first condition from the candidate intersection point set as the coordinates of the valid mark boundary points corresponding to the nut.

[0080] The first condition is that the angle between the first vector and the second vector is less than or equal to 60°. The first vector is the vector pointing from the geometric center point of the nut mask to the candidate intersection point, and the second vector is the vector pointing from the geometric center point of the nut mask to the geometric center point of the adjacent nut mask.

[0081] In real-world industrial environments, the drawing of anti-loosening line markers may be affected by paint drips, environmental stains, or redundant color extraction points in non-critical directions due to excessively long marking lines. These factors can introduce unreliable noise coordinates into the candidate intersection point set. To accurately pinpoint the critical intersection points for determining looseness from the initially acquired candidate intersection point set, a directional vector filtering mechanism is introduced. First, the computer device uses methods such as mask moment calculation to calculate the coordinates of the geometric center point of the current nut and the geometric center point of the adjacent nut mask based on the two-dimensional pixel distribution. Then, for each point in the candidate intersection point set, the computer device constructs a first vector pointing from the current nut's geometric center point to that candidate intersection point, and simultaneously constructs a second vector pointing from the current nut's geometric center point to the geometric center point of the adjacent nut. Next, the computer device calculates the vector angle between the first and second vectors in spatial direction by calculating their dot product. The computer device compares the calculated vector angle with a preset angle threshold, which is 60° as defined in the first condition. When the included angle of the vector is less than or equal to 60°, it means that the candidate intersection point is spatially oriented toward the adjacent nut and located in the inner region opposite to the two nuts. This conforms to the expected physical trend of the anti-loosening line establishing a connection between the two parts. The computer device then determines that the candidate intersection point meets the first condition and filters and retains its coordinates as the final valid marker boundary point coordinates used to determine the alignment.

[0082] In this embodiment, by extracting the contour boundary points of the nut mask and performing spatial intersection processing with the anti-loosening line marking mask, overlapping data in two-dimensional space is accurately extracted from a data processing perspective. This reduces the dimensionality of the massive mask matrix operation into a discrete set of candidate intersection point coordinates, effectively reducing the computational load of redundant pixels. Secondly, based on the spatial intersection, a direction vector calculation mechanism based on the geometric center is further introduced. The angle formed by the first and second vectors is used as a data filtering condition to perform secondary purification of the candidate intersection point set. This filtering process based on geometric vector direction eliminates redundant noise point coordinates generated in non-oriented areas due to environmental contamination or edge paint overflow. This ensures that the final output of effective marking boundary point coordinate data is highly focused on the actual connection area of ​​the anti-loosening line between adjacent parts, greatly improving the data purity and anti-interference capability during subsequent feature size and distance calculations.

[0083] In some embodiments, determining the bolt loosening state based on the nut alignment index includes: Based on the alignment index of the nut being less than or equal to the preset alignment index threshold, the loose state of the bolt is determined to be the tight state. If the alignment index of the nut is greater than the preset alignment index threshold, the bolt is determined to be in a loose state.

[0084] Specifically, after obtaining the alignment index of the nut after scale normalization, the computer equipment needs to convert this continuous feature value into a conclusion about the physical state of the specific industrial equipment. During this process, the computer equipment pre-stores or configures a preset alignment index threshold. This preset alignment index threshold is a dimensionless value used to measure the tolerance boundary of the anti-loosening line misalignment. In actual industrial data acquisition, due to the tilted shooting angle of drones or industrial cameras, mask edge segmentation errors caused by lighting, and slight elastic deformation during normal operation of mechanical equipment, even if the bolt has not actually rotated and loosened, there may be extremely small numerical deviations between the calculated effective marking boundary points. To shield these feature data disturbances within a reasonable range, the computer equipment uses this preset alignment index threshold (for example, it can be set to 0.1) as a judgment standard.

[0085] When the computer device finds that the alignment index of the nut is less than or equal to the preset alignment index threshold, it indicates that the relative offset of the boundary points of the anti-loosening line markings on the first and second nuts in the data calculation space is within the system's fault tolerance range. Based on this, the computer device determines that the anti-loosening line markings are still aligned in physical space, and thus outputs the detection conclusion that the bolt is in a tight state (i.e., a normal tight state). Conversely, when the computer device finds that the alignment index of the nut is greater than the preset alignment index threshold, it means that the magnitude of the relative displacement data between the extracted marking boundary feature points has exceeded the system's set safety boundary, indicating that the nut has likely undergone relative rotation and uncoupling in physical space. In this case, the computer device directly determines and outputs an alarm or record message indicating that the bolt is in a loose state.

[0086] In this embodiment, by introducing a preset alignment index threshold at the end of the data processing flow and performing a numerical comparison operation, the continuous and normalized distance ratio feature data obtained from the underlying processing is accurately mapped to discrete physical state classification conclusions (tight or loose state). This data processing mechanism based on fixed threshold decision effectively filters out the unavoidable minor data noise interference in the image acquisition and preceding instance segmentation steps, reducing the false alarm rate of the algorithm. On the other hand, it completely transforms the complex image feature judgment into a simple numerical logic comparison, giving the judgment result strong objectivity and consistency. Thus, it achieves a perfect replacement for the subjective and ambiguous judgment of human eyes in unmanned detection scenarios, ensuring the high stability and reliability of the detection process in practical engineering applications.

[0087] In some embodiments, the method for detecting bolt loosening further includes: If there is only one valid marker boundary point coordinate corresponding to a nut, output the second detection conclusion; The second detection conclusion indicates that only one side of the nut has valid marked boundary point coordinates.

[0088] In this embodiment, if it is found that there are only one valid marker boundary point coordinates corresponding to a nut in the currently processed dataset, it means that the feature information extracted from the current region of interest image is incomplete, and there is a lack of a second reference point coordinate for comparing relative displacement. At this time, the computer device no longer continues to calculate the nut spacing and alignment index, but triggers a preset specific state branch and directly outputs the second detection conclusion.

[0089] By introducing a logical verification mechanism for the number of valid marker boundary point coordinate sets in the data processing chain, when a special data pattern is identified where only one side of the valid coordinate data is extracted, the system proactively truncates the original Euclidean distance and alignment calculation branch and outputs a targeted second detection conclusion instead. This branching processing mechanism based on data integrity effectively avoids algorithm calculation errors or logical collapses caused by missing reference point data, significantly enhancing the robustness of the computer vision processing pipeline to noise in complex industrial scenarios. Simultaneously, the output of the second detection conclusion provides accurate data-driven qualitative analysis of abnormal situations such as marker detachment or non-standardization in actual engineering, enriching the state classification dimensions of the detection algorithm and thus providing more detailed and realistic equipment status data for automated inspection.

[0090] In some embodiments, such as Figure 6 As shown, the methods for detecting bolt loosening include: Input image.

[0091] Specifically, the input image is a surface image of the industrial equipment to be inspected (such as mechanical connection parts). The image format is RGB standard format, with a resolution of no less than 1080p, ensuring that bolts and anti-loosening marks are clearly visible. The image can be taken by an industrial camera, drone, or mobile terminal, and must meet the basic conditions of uniform lighting and no severe obstruction.

[0092] Bolt BBOX inspection frame.

[0093] Specifically, a pre-trained deep learning object detection model (such as YOLOv8, Faster R-CNN, or EfficientDet) is used to infer the input image and output the bounding box (BBOX) for each bolt target. This network is trained on datasets containing various bolt types, different lighting conditions, and angles, demonstrating strong generalization ability. Output: A set of bolt BBOX coordinates, denoted as {B1, B2, ..., B...} n}

[0094] Bolt ROI image.

[0095] Based on each obtained bolt box (BBOX), the corresponding region of interest (ROI) is cropped from the original image to form an independent bolt ROI image. For example... Figure 7 As shown, this operation focuses on a single bolt and its surrounding area, reducing the complexity of subsequent processing and improving accuracy. Output: n bolt ROI images, denoted as {I1, I2, ..., I...} n}

[0096] Bolt and nut mask.

[0097] Specifically, an instance segmentation network (such as Mask R-CNN, YOLACT, or SOLOv2) is used to perform pixel-level semantic segmentation on the original bolt ROI image (non-HSV image), outputting accurate bolt and nut masks, representing the pixel regions of the bolt head and nut, respectively. Figure 8 As shown, the network, trained on an industrial dataset with labeled bolt / nut boundaries, effectively distinguishes fastener components. Output: Each ROI corresponds to two binary mask images: bolt mask (M_bolt) and nut mask (M_nut).

[0098] Bolt ROI-HSV image.

[0099] Specifically, a color space conversion is performed on each bolt ROI image, converting the RGB image to an HSV (Hue, Saturation, Value) space image to obtain the bolt ROI-HSV image. For example... Figure 9 As shown, this transformation aims to enhance color features, facilitating subsequent color threshold detection of anti-loosening marks. Output: n bolt ROI-HSV images, denoted as {HSV1,HSV2,...,HSV...} n}

[0100] Mask for anti-loosening marking area.

[0101] Specifically, for the generated bolt ROI-HSV image, such as Figure 10 As shown, an algorithm for extracting loose marking regions based on dual thresholds of hue (H) and saturation (S) is designed: (1) Define the typical HSV range of the anti-loosening mark (e.g., red mark corresponds to H∈[0°,10°]∪[170°,180°], saturation S>0.6, brightness V>0.4).

[0102] (2) Perform threshold segmentation on each ROI-HSV image to generate a binary mask image, that is, the anti-loosening mark region mask (M_lock). This step uses color features to automatically locate the painted mark, without requiring manual annotation or template matching. Output: the anti-loosening mark mask M_lock corresponding to each ROI.

[0103] The boundary between the bolt, the nut and the anti-loosening mark.

[0104] Specifically, boundary intersection point extraction and direction screening are performed, that is: Perform the following operations on each nut mask (M_nut): (1) Extract the set of external contour boundary points thereof.

[0105] (2) Perform spatial intersection with the anti-loosening mark mask (M_lock) to obtain a set of candidate intersection points that are located at the edge of the nut and belong to the anti-loosening mark. Specifically, as Figure 10 shown, the valid mark boundary points are pixel points of 3.2px.

[0106] (3) If two nuts are detected, calculate the geometric centers of the two nut masks, and construct direction vectors pointing from the respective centers to the midpoint of the two centers.

[0107] (4) Perform direction screening on the candidate intersection points: only retain the points whose angle between the position vector and the direction vector is smaller than a preset angle threshold (such as 60°) as valid mark boundary points. Output: the coordinates of the valid mark boundary points corresponding to each nut (if present).

[0108] Bolt loosening state.

[0109] Specifically, for the ROI containing two valid nuts, the following determination logic is executed: (1) Calculate the average positions of the valid mark boundary points on the two nuts respectively to obtain point P1 and point P2.

[0110] (2) Calculate the Euclidean distance between the two points d=‖P1-P2‖.

[0111] (3) Based on the bounding boxes of the two nut masks, determine the maximum characteristic dimension L of the nut (take the maximum value of the width and the height).

[0112] (4) Calculate the alignment index α=d / L.

[0113] (5) If α<ALIGNMENT_THRESHOLD (that is, the preset alignment index threshold in the present invention, for example, 0.1), it is determined that the anti-loosening marks are aligned and the bolt is in a normal tightened state; otherwise, it is determined to be in a loosened state.

[0114] (6) If only one valid marking point of the nut is detected, or no anti-loosening mark is detected, it is marked as a single-sided mark or a mark not detected, respectively. Output: Four status labels for each bolt (loose / normal / single-sided mark / not detected).

[0115] like Figure 11 As shown, for the identity ID, ID:0, ID:1, ID:3, ID:4, ID:5, ID:6, ID:7, ID:10, and ID:11 were not detected, while ID:2 was detected as normal and ID:9 was detected as loose.

[0116] In this embodiment, by analyzing the relative positional relationship of the anti-loosening marks on the bolt and nut in a single frame image, the bolt loosening state can be automatically identified and determined without the need for timing information or external sensor assistance.

[0117] The bolt loosening detection device provided by the present invention will be described below. The bolt loosening detection device described below and the bolt loosening detection method described above can be referred to in correspondence.

[0118] In some embodiments, such as Figure 12 As shown, a device for detecting bolt loosening is proposed, comprising: The acquisition module 1201 is used to acquire a region of interest image containing the overall image of the bolt and nut. The region of interest image is an image obtained by performing overall bolt and nut detection and cropping on the first image. The segmentation module 1202 is used to perform instance segmentation on the region of interest image to obtain a nut mask; The conversion detection module 1203 is used to perform color space conversion on the region of interest image to obtain a second image represented in hue-saturation-brightness space, and to perform anti-loosening line mark detection based on the second image to obtain an anti-loosening line mark mask; The first determining module 1204 is used to determine the coordinates of the effective marking boundary points corresponding to each nut based on the nut mask and the anti-loosening line marking mask; The second determining module 1205 is used to determine the alignment index of the nut based on the coordinates of the effective marked boundary points and the bounding box data corresponding to the nut mask. The third determining module 1206 is used to determine the loosening state of the bolt based on the alignment index of the nut.

[0119] Figure 13 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 13As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330, and a communication bus 1340, wherein the processor 1310, the communication interface 1320, and the memory 1330 communicate with each other through the communication bus 1340. The processor 1310 can call logical instructions in the memory 1330 to execute a method for detecting the loose state of a bolt. The method includes: acquiring a region of interest image containing an overall image of the bolt and nut, wherein the region of interest image is obtained by performing overall bolt and nut detection and cropping on a first image; performing instance segmentation on the region of interest image to obtain a nut mask; performing color space conversion on the region of interest image to obtain a second image represented in hue-saturation-luminance space; and performing anti-loosening line mark detection based on the second image to obtain an anti-loosening line mark mask; determining the coordinates of the effective mark boundary points corresponding to each nut based on the nut mask and the anti-loosening line mark mask; determining the alignment index of the nut based on the coordinates of the effective mark boundary points and the bounding box data corresponding to the nut mask; and determining the bolt loosening state based on the nut alignment index.

[0120] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the bolt loosening state detection method provided by the above methods. The method includes: acquiring a region of interest image containing an overall image of the bolt and nut, wherein the region of interest image is an image obtained by performing overall bolt and nut detection and cropping on a first image; performing instance segmentation on the region of interest image to obtain a nut mask; performing color space conversion on the region of interest image to obtain a second image represented in hue-saturation-luminance space; and performing anti-loosening line mark detection based on the second image to obtain an anti-loosening line mark mask; determining the coordinates of the effective mark boundary points corresponding to each nut based on the nut mask and the anti-loosening line mark mask; determining the alignment index of the nut based on the coordinates of the effective mark boundary points and the bounding box data corresponding to the nut mask; and determining the bolt loosening state based on the nut alignment index.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for detecting bolt loosening state provided by the above methods. The method includes: acquiring a region of interest image containing an overall image of the bolt and nut, the region of interest image being an image obtained by performing overall bolt and nut detection and cropping on a first image; performing instance segmentation on the region of interest image to obtain a nut mask; performing color space conversion on the region of interest image to obtain a second image represented in hue-saturation-luminance space; and performing anti-loosening line mark detection based on the second image to obtain an anti-loosening line mark mask; determining the coordinates of effective mark boundary points corresponding to each nut based on the nut mask and the anti-loosening line mark mask; determining the alignment index of the nut based on the coordinates of the effective mark boundary points and the bounding box data corresponding to the nut mask; and determining the bolt loosening state based on the nut alignment index.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the loosening state of bolts, characterized in that, include: Obtain a region of interest image containing the entire bolt and nut image, wherein the region of interest image is obtained by performing overall bolt and nut detection and cropping on the first image; The region of interest image is segmented to obtain a nut mask; The region of interest image is subjected to color space conversion to obtain a second image represented in hue-saturation-brightness space, and anti-loosening line mark detection is performed based on the second image to obtain an anti-loosening line mark mask; Based on the nut mask and the anti-loosening line marking mask, determine the coordinates of the effective marking boundary point corresponding to each nut; Based on the coordinates of the effective marked boundary points and the bounding box data corresponding to the nut mask, the alignment index of the nut is determined; The looseness of the bolt is determined based on the alignment index of the nut.

2. The method for detecting bolt loosening according to claim 1, characterized in that, The step of determining the alignment index of the nut based on the coordinates of the effective marked boundary points and the bounding box data corresponding to the nut mask includes: Based on the coordinates of the effective marked boundary points, determine the first boundary point on the first nut and the second boundary point on the second nut; Calculate the Euclidean distance between the position coordinates of the first boundary point and the position coordinates of the second boundary point to obtain the nut spacing between the first nut and the second nut; Based on the bounding box data corresponding to the nut mask, determine the maximum feature size corresponding to the first nut or the second nut; The ratio of the nut spacing to the maximum feature size is used as the alignment index of the nut; The first nut and the second nut are two adjacent nuts.

3. The method for detecting bolt loosening according to claim 1, characterized in that, The step of obtaining a region of interest image containing the entire image of the bolt and nut includes: The first image is input into a pre-trained detection network to perform overall bolt and nut detection, and the bounding box data containing the overall bolt and nut image is obtained from the output of the detection network. The first image is cropped based on the bounding box data to obtain a region of interest image that includes the entire image of the bolt and nut.

4. The method for detecting bolt loosening according to claim 1, characterized in that, The step of detecting the anti-loosening line markers based on the second image to obtain an anti-loosening line marker mask includes: Obtain the first hue-saturation-brightness range, where the first hue-saturation-brightness range is a predefined hue-saturation-brightness range to which the anti-loosening line mark belongs; Based on the first hue-saturation-brightness range, the second image is segmented by thresholding to obtain the anti-loosening line mark mask.

5. The method for detecting bolt loosening according to claim 4, characterized in that, The method for detecting bolt loosening also includes: If the anti-loosening line mark is not detected, output the first detection conclusion; The first detection conclusion indicates that no loose bolts were detected.

6. The method for detecting bolt loosening according to claim 1, characterized in that, The determination of the coordinates of the effective marking boundary points corresponding to each nut based on the nut mask and the anti-loosening line marking mask includes: The outer contour boundary points of the nut mask are extracted to obtain the first candidate point set; The first candidate point set and the anti-loosening line mark mask are spatially intersected to obtain a candidate intersection point set. Each candidate intersection point in the candidate intersection point set is located within the first candidate point set and belongs to the anti-loosening line mark mask. The coordinates of candidate intersection points that satisfy the first condition are selected from the candidate intersection point set and used as the coordinates of the valid marked boundary points corresponding to the nut. Wherein, the first condition is that the vector angle between the first vector and the second vector is less than or equal to 60°, the first vector is the vector from the geometric center point of the nut mask to the candidate intersection point, and the second vector is the vector from the geometric center point of the nut mask to the geometric center point of the adjacent nut mask.

7. The method for detecting bolt loosening according to any one of claims 1 to 6, characterized in that, Determining the bolt looseness state based on the alignment index of the nut includes: Based on the alignment index of the nut being less than or equal to a preset alignment index threshold, the loose state of the bolt is determined to be a tight state. Based on the fact that the alignment index of the nut is greater than the preset alignment index threshold, the bolt is determined to be in a loose state.

8. The method for detecting bolt loosening according to any one of claims 1 to 6, characterized in that, The method for detecting bolt loosening also includes: If there is only one valid marker boundary point coordinate corresponding to a nut, output the second detection conclusion; The second detection conclusion indicates that only one side of the nut has valid marked boundary point coordinates.

9. A device for detecting the looseness of bolts, characterized in that, include: The acquisition module is used to acquire a region of interest image containing the overall image of the bolt and nut, wherein the region of interest image is obtained by performing overall bolt and nut detection and cropping on the first image; The segmentation module is used to perform instance segmentation on the region of interest image to obtain a nut mask; The conversion detection module is used to perform color space conversion on the image of the region of interest to obtain a second image represented in hue-saturation-brightness space, and to perform anti-loosening line mark detection based on the second image to obtain an anti-loosening line mark mask; The first determining module is used to determine the coordinates of the effective marking boundary points corresponding to each nut based on the nut mask and the anti-loosening line marking mask; The second determining module is used to determine the alignment index of the nut based on the coordinates of the effective marked boundary points and the bounding box data corresponding to the nut mask. The third determining module is used to determine the loosening state of the bolt based on the alignment index of the nut.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the bolt loosening detection method as described in any one of claims 1 to 8.

Images