A computer vision-based hydraulic equipment fastener assembly anomaly detection method

Abstract

This invention discloses a computer vision-based method for detecting assembly anomalies in hydraulic equipment fasteners, relating to the field of computer vision detection technology. The method includes: performing target detection on a global image of the assembly surface of the hydraulic equipment fasteners to initially locate the fastener region; generating a high-quality fused image for each region using multi-mode active illumination imaging; processing the fused image using a multi-task attribute parsing network, and outputting the fastener category, mask, and coordinates of key engineering features in parallel; obtaining multi-view local images of the fasteners to solve their 3D pose, and calculating the offset from the theoretical pose to achieve preliminary loosening detection; constructing an instance attribute relationship graph and matching it with the theoretical graph, and using this graph for collaborative reasoning with the loosening detection results to comprehensively output a final judgment of multiple anomalies. This invention significantly improves the robustness and accuracy of fastener assembly anomaly detection in harsh industrial environments through a two-level vision strategy, multi-view 3D verification, and graph matching collaborative decision-making.

Description

Technical Field

[0001] This application relates to the field of computer vision inspection technology, and in particular to a method for detecting assembly anomalies in hydraulic equipment fasteners based on computer vision. Background Technology

[0002] The structural integrity, sealing reliability, and long-term operational stability of hydraulic equipment largely depend on the assembly quality of its connecting fasteners (mainly bolts and nuts). Traditional assembly quality inspection relies primarily on visual inspection by operators and periodic tool sampling. This method suffers from low efficiency, poor consistency, difficulty in establishing quantitative records, and insensitivity to potential problems such as early loosening.

[0003] In recent years, computer vision-based detection technology has been introduced into industrial scenarios. Existing technical solutions mainly focus on the following aspects: fastener location and presence detection based on template matching or traditional features; general object detection frameworks based on deep learning directly applied to fastener recognition; for loosening detection, single visual features are often used for judgment, such as the alignment status of anti-loosening marking lines and changes in the position of nut edges and corners.

[0004] However, when the above-mentioned existing technology is directly applied to the inspection of fasteners in hydraulic equipment, the following problems exist:

[0005] Strong environmental interference: The oil film, water stains, and irregular reflections from metal processing surfaces commonly found on the surface of hydraulic components can severely pollute image quality, making traditional feature-based methods such as edges and corners prone to failure, and causing deep learning models to degrade in performance due to feature blurring.

[0006] The targets are complex: fasteners are small and occupy a low proportion of the overall image; they also come in various models and specifications with high similarity in appearance; furthermore, the installation space for fasteners is compact, resulting in partial occlusion. This requires the detection algorithm to have both sensitivity to small targets and strong discriminative power against similar targets.

[0007] Loosening criteria are singular and unreliable: Existing methods mostly rely on a single visual feature that is easily damaged or contaminated in a hydraulic environment to determine loosening; once this feature is covered by oil, wiped away by reflection, or is not clearly marked during installation, the detection logic fails, lacking a redundant verification mechanism.

[0008] Fragmented Detection Dimensions: Existing solutions typically treat existence detection, compliance detection, and integrity detection as independent tasks, using separate models or processes. This ignores the inherent connections between tasks (for example, a "suspected loose" fastener may also be a "misfit" or mis-installed model), fails to form a synergy in decision-making, and results in a lengthy overall process.

[0009] Therefore, there is an urgent need for a highly reliable method for detecting fastener assembly anomalies in hydraulic equipment that can adapt to the harsh visual environment of hydraulic equipment, comprehensively judge multiple assembly defects, and enhance each other through synergistic cooperation among various detection dimensions. Summary of the Invention

[0010] This application provides a computer vision-based method for detecting assembly anomalies in hydraulic equipment fasteners. It aims to address the technical problems of robustly locating and identifying fasteners in environments with strong interference such as oil stains and reflective surfaces, reliance on single features for loosening detection, and fragmented judgment of multiple types of assembly anomalies. The core of this method lies in decomposing the complex assembly anomaly detection task through global and local image acquisition, as well as fastener target localization and attribute parsing based on deep learning. Furthermore, it integrates multi-source information for cross-validation and collaborative reasoning in anomaly judgment, thereby improving the reliability of fastener assembly anomaly detection.

[0011] This application provides a computer vision-based method for detecting assembly anomalies in hydraulic equipment fasteners, including: Step S10: Obtain a global image of the assembly surface of the hydraulic equipment to be inspected and perform illumination invariance preprocessing to suppress oil stains and reflection interference. Use a target detection network to initially locate the region of interest of all fastener targets in the global image. Step S20: For the region of interest, control the servo camera module to move above it, acquire axial light image and lateral light image through multi-mode active illumination imaging, and perform gradient domain fusion on the two images to generate a quality-enhanced fused image. Step S30: Apply a multi-task attribute parsing network to the fused image obtained in step S20 for deep recognition, and output the precise category of the fastener, pixel-level segmentation mask and coordinates of predefined engineering feature key points in parallel. Step S40: For each fastener identified by deep layer, acquire local images taken from different perspectives, combine the key engineering features and camera parameters obtained in step S30, and solve the three-dimensional position offset and attitude angle offset of the fastener relative to the preset theoretical pose through spatial geometric calculation, and perform preliminary loosening detection on the fastener based on the three-dimensional position offset and attitude angle offset. In step S50, based on the identification results of step S30 and the detection results of step S40, an instance attribute relationship graph is constructed and matched with the theoretical attribute relationship graph based on the digital twin model. Based on the matching results and the loosening detection results of step S40, collaborative reasoning is performed and a final judgment on missing, incorrect, loose, and suspected abnormalities is output.

[0012] Furthermore, step S10 includes the following detailed steps: Step S11: Use a fixed global camera to photograph the assembly surface of the hydraulic equipment to be inspected from multiple angles to obtain a global image containing all fasteners to be inspected; Step S12: Perform illumination invariance preprocessing on the global image, including converting it to the HSV color space, performing adaptive Gamma correction on the brightness channel to compress the highlight area and enhance the shadow area, while enhancing the saturation channel. Step S13: Input the preprocessed global image into the pre-trained target detection network. The network adopts a feature pyramid architecture and uses an attention mechanism to initially locate candidate regions of all fastener targets in the global image by focusing on the local texture and geometric structure regions of fasteners in the focused image.

[0013] Further, in step S20, for each candidate region located in step S10, the following steps are performed: Control a servo motion camera module equipped with a high-definition macro lens and a programmable ring LED light source to move it directly above the region of interest; control the light source through programming to acquire two high-definition images of the same region in axial light mode and lateral light mode respectively, to obtain axial light image and lateral light image; Axial light images effectively avoid the strong shadows that side light may produce, preserving features such as bolt head markings and washer colors; side light images, on the other hand, make the thread engagement state, minor scratches on the bolt head edge, and the accumulation of oil stains in the texture extremely clear.

[0014] The obtained axial light image and lateral light image are fused, specifically including: calculating the gradient map of the lateral light image to obtain its rich edge texture information; in the gradient domain, fusing the gradient information of the lateral light image with the pixel intensity information of the axial light image according to a preset weighting rule to generate a guided gradient field; finally, by solving the Poisson equation, the final fused image is reconstructed from the guided gradient field.

[0015] Further, step S30 inputs the fused image into a multi-task attribute parsing network, which employs a shared pre-trained feature encoder followed by three parallel decoders, including: Classification decoder: Outputs the precise category label of the fastener; Precision segmentation decoder: Outputs a pixel-level binary mask of the fastener, accurately outlining its contour; Keypoint Regression Decoder: Regresses and predicts the pixel coordinates of a set of predefined feature keypoints with engineering semantics; for bolts, keypoints include the center of the bolt head and each edge and corner point; for nuts, they are each edge and corner point.

[0016] Further, step S40 performs a preliminary loosening detection on each fastener that has been accurately identified, including: Step S41: For each deeply identified fastener, acquire local images of it taken in at least three different known camera poses, and extract the two-dimensional pixel coordinates of the fastener in each image from the same set of engineering feature key points determined in step S30. Step S42: Using multiple sets of the two-dimensional pixel coordinates, the corresponding known camera internal parameters, and the precise external pose parameters of the camera at each shooting angle as input, the PnP algorithm is used to solve the three-dimensional position coordinates and three-dimensional rotational attitude of the fastener in the unified world coordinate system. Step S43: From the preset digital twin model, retrieve the theoretical three-dimensional position coordinates and theoretical three-dimensional rotational attitude corresponding to the fastener; Step S44: Calculate the Euclidean distance between the solved three-dimensional position coordinates and the theoretical three-dimensional position coordinates, and use it as the position offset. The angle difference between the solved three-dimensional rotational attitude and the theoretical three-dimensional rotational attitude is calculated and used as the attitude angle offset. ; Step S45, the position offset and the attitude angle offset If any component exceeds the corresponding preset threshold, the fastener is determined to be loose.

[0017] Furthermore, step S50 implements multi-anomaly collaborative decision-making based on graph matching and rule reasoning, including: Based on the digital twin model, a theoretical attribute relationship graph is constructed with theoretical fasteners as nodes and their spatial adjacency relationships as edges. The node attributes include type, specifications, and theoretical pose. At the same time, based on the deep recognition results of step S30, actual fastener instances are obtained, and an instance attribute relationship graph is constructed with their three-dimensional spatial positions as nodes. The node attributes include the identified category, specifications, calculated current pose, and three-dimensional position and attitude angle offset. Graph matching is performed between the instance graph and the theoretical graph. By solving an optimization problem, the optimal node correspondence between the two graphs is found, which combines the similarity of node attributes with the similarity of graph structure. Based on the matching results, the following decision logic is executed: If a node in the theoretical diagram has no matching node in the instance diagram, it is determined to be a missing fastener. If the match is successful, but the key attributes (such as type and specification) between the matched nodes are inconsistent, it is determined that the fastener is misinstalled. If the match is successful and the attributes are consistent, but the pose offset of the corresponding instance ( or If the threshold is exceeded, it is determined that the fastener is loose; If the match is successful, the attributes are consistent, and the pose is normal, but the recognition confidence of the instance is lower than the preset threshold, it is marked as "suspected anomaly, re-examination recommended".

[0018] The output includes a comprehensive inspection report containing all anomaly types, locations, and detailed descriptions (such as the specific model of the misinstalled part and the offset of any loose parts).

[0019] This application discloses the following technical effects: This application provides a computer vision-based method and system for detecting assembly anomalies in hydraulic equipment fasteners. Through two-stage focusing—preliminary global image localization and precise local image recognition—combined with targeted light invariance preprocessing and multi-light fusion processing, it overcomes the interference of oil stains and reflections on the image quality of hydraulic equipment assembly surfaces, providing a precise data foundation for subsequent assembly anomaly detection and analysis. This method employs a multi-view 3D verification approach, which performs preliminary loosening detection by solving the actual 3D spatial pose changes of the fastener. This overcomes the reliance of traditional loosening detection methods on easily damaged and contaminated 2D markers such as anti-loosening lines, and provides more direct and robust detection results. Furthermore, this method introduces a graph matching architecture, which unifies the detection of multiple anomalies such as missing parts, incorrect parts, loose parts, and suspected anomalies into a global optimization process. It uses the spatial context relationship between fasteners for reasoning, so that the decision-making process is no longer an isolated judgment, thereby significantly reducing the false positive and false negative rates, and can identify ambiguous states such as suspected anomalies. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a computer vision-based method for detecting assembly anomalies in hydraulic equipment fasteners, as provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Example 1: This application provides a computer vision-based method for detecting assembly anomalies in hydraulic equipment fasteners, such as... Figure 1 As shown, the method includes: Step S10: Obtain a global image of the assembly surface of the hydraulic equipment to be inspected and perform illumination invariance preprocessing to suppress oil stains and reflection interference. Use a target detection network to initially locate the region of interest of all fastener targets in the global image.

[0023] In this embodiment, a 5-megapixel global area array industrial camera is used and fixedly installed above the assembly line to ensure that its field of view can completely cover the maximum assembly plane of the hydraulic equipment to be inspected (such as hydraulic valve blocks and pump housings); a polarizing filter is added in front of the camera lens to reduce direct specular reflection from the metal surface; the ambient lighting uses a uniformly diffused LED surface light source to reduce shadows and point highlights.

[0024] The acquired global image of the assembly surface of the hydraulic equipment under inspection is converted from the RGB channel to the HSV color space, separating the three components: hue (H), saturation (S), and lightness (V). This conversion decouples color information from lightness information for targeted processing, including V channel enhancement and S channel enhancement. Hydraulic equipment surfaces often exhibit both highlights (reflections) and shadows (oil buildup or deep holes). Ordinary global adjustments often miss some areas. Therefore, for the V channel image of the global image, an adaptive Gamma correction method is used to compress highlight areas and stretch shadow details to improve overall contrast, including: The V channel image is divided into multiple 8x8 sub-blocks. For each sub-block, calculate its pixel mean and variance; If the average value of the sub-block is higher than the set threshold of 200, it means that the sub-block belongs to the highlight area. Then, a larger Gamma value (1.8 in this embodiment) is used for correction to compress the dynamic range of brightness. If the mean value is lower than another threshold of 50, it indicates that the sub-block belongs to the dark area. In this case, a smaller Gamma value (0.5 in this embodiment) is used for correction to improve the brightness.

[0025] Oil stains on the surface of hydraulic equipment often result in dull colors and reduced saturation in images. Appropriately enhancing the S-channel helps to distinguish fasteners covered by oil stains from the background in the color dimension. Therefore, in this embodiment, a linear stretching function is applied to the S-channel image to map its pixel value range from [Min, Max] to [0, 255]. Here, Min and Max represent the minimum and maximum pixel values ​​in the image, respectively, which are determined by statistically analyzing the saturation distribution of non-extreme bright and dark areas in the image to avoid excessive amplification of noise.

[0026] The processed H, S, and V channel images are re-merged and converted back to RGB space to obtain the preprocessed global image.

[0027] This embodiment uses YOLOv8n as the target detection network. Since the fasteners (bolt heads, nuts) on the hydraulic equipment are small in size in the global image, all less than 0.1% of the image area, targeted improvements are made to this network: In YOLOv8n, a detection head is added after the shallow feature map (with higher spatial resolution) output by the feature pyramid, specifically for detecting tiny fastener targets; By embedding a coordinate attention module in the feature extraction stage of the network backbone, the network can focus more on the local texture and geometric structure areas in the image that may contain fasteners, thus suppressing interference from complex backgrounds.

[0028] The training data for the target detection network includes global images of different types of hydraulic equipment, different lighting conditions, different degrees of oil pollution, different fastener types (bolts, nuts, washers) and states (normal, partially occluded), and is finely labeled. Considering the environmental characteristics of hydraulic equipment, data augmentation methods are introduced during the training process, including: randomly overlaying semi-transparent spots with colors and textures to simulate oil and water stains on the image; and randomly adding overexposed pixel blocks to metal surface areas to simulate highlights. The training objective of the object detection network is set to high recall. On the validation set, the recall metric takes precedence over precision to ensure that no real fasteners are missed as much as possible. After training, a pre-trained object detection network is obtained, which is used to receive the pre-processed global image and output a series of bounding boxes, each containing the predicted fastener category and confidence level. For all predicted bounding boxes, non-maximum suppression is used for filtering, and all predicted bounding boxes with a confidence level higher than a preset threshold are retained. The coordinates of these bounding boxes are appropriately expanded to fully include the fastener and a small amount of surrounding background, forming the final list of regions of interest.

[0029] Step S20: For the region of interest, control the servo camera module to move above it, acquire axial light image and lateral light image through multi-mode active illumination imaging, and perform gradient domain fusion on the two images to generate a quality-enhanced fused image.

[0030] In this embodiment, the servo camera module consists of a high-precision three-axis linear motor motion platform, a 20-megapixel CMOS industrial camera mounted at the end of the platform, a telecentric macro lens (working distance 100mm, field of view approximately 20mm×15mm) for close-up details, and a coaxially mounted programmable ring LED light source. The programmable ring LED light source comprises four independently controlled LED rings. The inner ring LEDs emit light parallel to the camera's optical axis, generating axial light; the outer ring LEDs emit light at a 30° angle to the optical axis, generating lateral light.

[0031] First, based on the pixel coordinates of the region of interest in the global image output in step S10, and combined with the pre-calibrated hand-eye transformation matrix between the global camera and the servo motion platform, the three-dimensional spatial coordinates of the center of the region of interest in the motion platform coordinate system are calculated. The control platform moves the camera and the light source to align the camera's optical axis with these coordinates, and fine-tunes the Z-axis to ensure that the target area is clearly imaged within the depth of field of the macro lens; Turn on the inner ring LED (axial light) of the ring light source and turn off the outer ring LED. At this time, the light mainly shines from the front, which can better show the color, gray scale distribution and planar features of the subject's surface, and capture the axial light image. Turn off the inner ring LED and turn on the outer ring LED (side light) of the ring light source. At this time, the light sweeps across the subject's surface at a low angle, and the texture, threads and scratches of the bolt head will produce obvious shadows, thereby enhancing the edge and texture contrast, and capturing the side light image.

[0032] Secondly, after acquiring the axial light image and the lateral light image, these two images are fused in the gradient domain, including the following detailed steps: The axial light image and lateral light image are preprocessed to ensure their light invariance to oil stains and reflective interference on the surface of the hydraulic equipment. The gradient map of the preprocessed lateral light image is calculated, and gradient information is extracted from it to characterize the surface texture and edge details of the lateral light image. Pixel intensity information of the axial light image is obtained simultaneously. In the gradient domain, the gradient information of the lateral light image and the pixel intensity information of the axial light image are weighted and fused according to a preset fusion weight rule to construct a guiding gradient field: The fusion weighting rule is configured to make the fused guided gradient field approximate the gradient of the side-lit image in textured regions and tend to preserve the pixel intensity information of the axial-lit image in flat regions.

[0033] Based on the guided gradient field, the final fused image is reconstructed by solving the Poisson equation: The goal is to find a fusion diagram. To make its gradient field as close as possible to the constructed guided gradient field. This is transformed into solving the following Poisson equation:

[0034] in, It is the Laplace operator. It is a divergence operator; the discrete cosine transform method is used to solve the above equations, specifying boundary conditions; the solution obtained is... This is the final fused image, which retains the overall illumination uniformity and smooth color transition of the axial light image, while injecting strong edge and texture details from the lateral light image.

[0035] Step S30: Apply a multi-task attribute parsing network to the fused image obtained in step S20 for deep recognition, and output the precise category of the fastener, pixel-level segmentation mask, and coordinates of predefined engineering feature key points in parallel.

[0036] In this embodiment, the multi-task attribute parsing network includes a shared pre-trained feature encoder and three parallel pre-trained decoders: A ResNet-50 pre-trained on ImageNet is used as a shared feature encoder. The fused image obtained in step S30 is input, and multi-layer feature maps are extracted through the feature encoder. In order to preserve the small details of the fastener target, not only the features of the final layer are used, but also the features from the middle and high levels of the feature encoder are fused through the feature pyramid network structure to form a multi-scale feature map containing rich semantic and spatial details. Three parallel dedicated decoders are connected on top of the multi-scale feature maps to perform three tasks simultaneously: The classification decoder consists of a global average pooling layer, two fully connected layers, and a Softmax output layer. The decoder outputs a probability vector, which represents the confidence level of the fastener belonging to each specific category in the predefined category library (such as "M10 internal hex bolt - grade 8.8" and "M12 flange nut - A2-70"). Instance segmentation decoder: It adopts a U-Net structure and gradually restores the spatial resolution through a series of upsampling and skip connections. Finally, it outputs a binary mask of the same size as the input image, which accurately marks the position of the fastener pixels. Keypoint Regression Decoder: This decoder first extracts features through a convolutional module, and then outputs a heatmap tensor. The number of channels in this tensor is equal to the number of predefined engineering feature keypoints K. For the k-th keypoint, a Gaussian response centered on the keypoint is generated on the corresponding heatmap channel near the real location of the keypoint. By finding the peak position of the heatmap response of each channel, the two-dimensional pixel coordinates of the keypoint are decoded.

[0037] For different types of fasteners, a set of key engineering features with stable visual characteristics and clear engineering significance are predefined, for example: For hex bolts and nuts: Define 6 corner points and the center point of the bolt head or nut face, for a total of 7 points. These points can uniquely determine their planar orientation and center position.

[0038] For hex socket head cap screws: In addition to the edge and corner points, the center of the hex socket hole and several internal corner points are additionally defined to assist in identification and loosening judgment.

[0039] Step S40: For each fastener identified by deep layer, acquire local images taken from different perspectives. Combine the key engineering features and camera parameters obtained in step S30, calculate the three-dimensional position offset and attitude angle offset of the fastener relative to the preset theoretical pose through spatial geometric calculation. Based on the three-dimensional position offset and attitude angle offset, perform preliminary loosening detection on the fastener.

[0040] In this embodiment, for each fastener identified in step S30, the servo motion camera module is controlled to capture high-resolution local images of the fastener in at least three different and non-collinear spatial poses that have been calibrated: Pose 1 is where the camera's optical axis is perpendicular to the fastener's end face, obtaining a front view. Pose 2 and 3 are where the camera takes a slanted view from a ±30° lateral angle, respectively. Each pose capture triggers the multi-mode active illumination and image fusion process in step S20, generating a fused local image for each viewpoint.

[0041] For each viewpoint's local image, the keypoint regression decoder in the pre-trained multi-task attribute parsing network in step S30 is called to extract the two-dimensional pixel coordinates of the same set of predefined engineering feature keypoints in the viewpoint image; and the camera intrinsic parameter matrix, which is pre-calibrated using a checkerboard pattern, and the camera extrinsic parameter matrix relative to the world coordinate system under each shooting viewpoint are used as camera parameters.

[0042] Based on key engineering features and camera parameters, the three-dimensional position offset and attitude angle offset of the fastener relative to the preset theoretical pose are solved through spatial geometric calculations, including the following detailed steps: In the CAD model of the fastener, the three-dimensional coordinates of the key points of the engineering features corresponding to step S30 are defined; for example, for a standard hexagonal head bolt, the three-dimensional coordinates of its six corner points can be accurately calculated based on its nominal diameter and head height. The algorithm takes the 2D pixel coordinates of the same set of key points observed from multiple perspectives, their corresponding 3D model coordinates, and the camera's intrinsic and extrinsic parameters as inputs, and uses a robust PnP solution algorithm for calculation. This algorithm directly outputs the 3D rotation matrix of the fastener in the current world coordinate system by minimizing the reprojection error. Translation vector ; From the digital twin model of the hydraulic equipment, the theoretical pose of the fastener at the designed position, including the theoretical rotation matrix, is read. and theoretical translation vector Calculate the Euclidean distance of the translation vector to obtain the three-dimensional position offset. :

[0043] in, This describes the process of solving for the Euclidean distance. The theoretical rotation matrix and Multiplying the transposes yields , and then from Extracting rotation angle As attitude angle offset :

[0044] in, Represents the inverse cosine function. The trace of a matrix is... The sum of all elements on the main diagonal.

[0045] Set two thresholds: maximum allowed position offset threshold and maximum allowable attitude angle offset threshold These two thresholds are determined based on the fastener specifications, the functional importance of the connector, and process requirements; when the position offset... and attitude angle offset If any component exceeds the corresponding preset threshold and If the fastener is found to be loose, it is determined that the fastener is properly tightened; otherwise, it is determined that the fastener is properly tightened.

[0046] In step S50, based on the identification results of step S30 and the detection results of step S40, an instance attribute relationship graph is constructed and matched with the theoretical attribute relationship graph based on the digital twin model. Based on the matching results and the loosening detection results of step S40, collaborative reasoning is performed and a final judgment on missing, incorrect, loose, and suspected abnormalities is output.

[0047] In this embodiment, based on the digital twin model, a theoretical attribute relationship graph is constructed with theoretical fasteners as nodes and their spatial adjacency relationships as edges. The node attributes include fastener type, specifications, and theoretical pose. The digital twin model contains a structured database of all fastener design information. An edge connecting two theoretical fasteners indicates that the two fasteners are spatially adjacent (e.g., bolt rings belonging to the same flange face). The edges can have weights to indicate the tightness of the connection (e.g., the reciprocal of the bolt spacing).

[0048] Meanwhile, based on the deep recognition results of step S30 and the pose information obtained in step S40, an actual fastener instance is obtained. Using its three-dimensional spatial position as a node, an instance attribute relationship graph is constructed. The node attributes include the identified category, specifications, calculated current pose, and three-dimensional position and attitude angle offset.

[0049] Instance attribute relationship graph Relationship diagram with theoretical properties The goal of matching is to M nodes and Find an optimal set of correspondences among N nodes. This is a combinatorial optimization problem, typically solved by constructing and minimizing a matching cost function that integrates differences in node attributes and differences in structural consistency. The above problem is formalized as a quadratic assignment problem and solved efficiently using an extended version of the Hungarian algorithm. The solution assigns a best-matching actual node (or marks it as "no match") to each theoretical node, while each actual node matches at most one theoretical node.

[0050] Based on the matching results above, the following sequential decision logic will be executed: Missing item detection: Traversal All nodes in the set, if a certain theoretical node If a part is marked as "no match" in the solution results, it is determined that there is a missing part at that position, and the theoretical specifications and location of the missing part are recorded. Mismatch detection: Iterate through all successfully matched node pairs and compare their positions in the table. and If the type and specification attributes are inconsistent, the actual fastener is determined to be misinstalled, and the location of the misinstallation, theoretical specifications and actual specifications are recorded.

[0051] Looseness detection: For node pairs with correctly matched attributes, check their position in... Pose offset in , ),if or (The threshold is from step S40), then the fastener is determined to be "loose", and the location and specific offset of the loosening are recorded.

[0052] Suspected anomaly determination: For node pairs with correct attribute matching and normal pose, check their recognition confidence. If the confidence is less than the preset threshold, it is determined as "suspected anomaly". This usually indicates that the fastener may be severely contaminated, partially damaged or belongs to an untrained model, and manual re-inspection is required.

[0053] Conflict Resolution and Collaboration: The decision-making process is collaborative. For example, if a physical node simultaneously meets the conditions of "misinstallation" and "looseness," the system will prioritize the more serious "misinstallation" condition because the incorrect part is the fundamental problem. Simultaneously, the system will generate a comprehensive report indicating that "an incorrect Y-type bolt was installed at position X, and this bolt is also loose," providing more comprehensive maintenance guidance.

[0054] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A computer vision-based hydraulic equipment fastener assembly anomaly detection method, characterized by, The method includes: Step S10: Obtain a global image of the assembly surface of the hydraulic equipment to be inspected and perform illumination invariance preprocessing to suppress oil stains and reflection interference. Use a target detection network to initially locate the region of interest of all fastener targets in the global image. Step S20: For the region of interest, control the servo camera module to move above it, acquire axial light image and lateral light image through multi-mode active illumination imaging, and perform gradient domain fusion on the two images to generate a quality-enhanced fused image. Step S30: Apply a multi-task attribute parsing network to the fused image obtained in step S20 for deep recognition, and output the precise category of the fastener, pixel-level segmentation mask and coordinates of predefined engineering feature key points in parallel. Step S40: For each fastener identified by deep layer, acquire local images taken from different perspectives, combine the key engineering features and camera parameters obtained in step S30, and solve the three-dimensional position offset and attitude angle offset of the fastener relative to the preset theoretical pose through spatial geometric calculation, and perform preliminary loosening detection on the fastener based on the three-dimensional position offset and attitude angle offset. In step S50, based on the identification results of step S30 and the detection results of step S40, an instance attribute relationship graph is constructed and matched with the theoretical attribute relationship graph based on the digital twin model. Based on the matching results and the loosening detection results of step S40, collaborative reasoning is performed and a final judgment on missing, incorrect, loose, and suspected abnormalities is output.

2. The computer vision-based hydraulic equipment fastener assembly anomaly detection method of claim 1, wherein, Step S10 includes the following detailed steps: Step S11: Use a fixed global camera to photograph the assembly surface of the hydraulic equipment to be inspected from multiple angles to obtain a global image containing all fasteners to be inspected; Step S12: Perform illumination invariance preprocessing on the global image, including converting it to the HSV color space, performing adaptive Gamma correction on the brightness channel to compress the highlight area and enhance the shadow area, while enhancing the saturation channel. Step S13: Input the preprocessed global image into the pre-trained target detection network. The network adopts a feature pyramid architecture and uses an attention mechanism to initially locate the regions of interest of all fastener targets in the global image by focusing on the local texture and geometric structure regions of fasteners in the focused image.

3. The method for detecting assembly anomalies in hydraulic equipment fasteners based on computer vision as described in claim 1, characterized in that, In step S20, the gradient domain fusion includes the following detailed steps: The axial light image and lateral light image are preprocessed to ensure their light invariance to oil stains and reflective interference on the surface of the hydraulic equipment. Calculate the gradient map of the preprocessed lateral light image, extract gradient information from it to characterize the surface texture and edge details of the lateral light image, and simultaneously acquire the pixel intensity information of the axial light image; In the gradient domain, the gradient information of the lateral light image and the pixel intensity information of the axial light image are weighted and fused according to a preset fusion weighting rule to construct a guiding gradient field. Based on the aforementioned guiding gradient field, the final fused image is reconstructed by solving the Poisson equation; The fusion weighting rule is configured to make the fused guided gradient field approximate the gradient of the lateral light image in texture-rich regions, and tend to preserve the pixel intensity information of the axial light image in flat regions.

4. The method for detecting assembly anomalies in hydraulic equipment fasteners based on computer vision as described in claim 1, characterized in that, In step S30, the multi-task attribute parsing network includes a shared pre-trained feature encoder and three parallel decoders. The three decoders are: a classification decoder for predicting the precise category of fasteners, an instance segmentation decoder for outputting the pixel-level contours of fasteners, and a keypoint regression decoder for regressing the coordinates of predefined geometric feature keypoints.

5. The method for detecting assembly anomalies in hydraulic equipment fasteners based on computer vision as described in claim 1, characterized in that, Step S40 includes the following steps: Step S41: For each deeply identified fastener, acquire local images of it taken in at least three different known camera poses, and extract the two-dimensional pixel coordinates of the fastener in each image from the same set of engineering feature key points determined in step S30. Step S42: Using multiple sets of the two-dimensional pixel coordinates, the corresponding known camera internal parameters, and the precise external pose parameters of the camera at each shooting angle as input, the PnP algorithm is used to solve the three-dimensional position coordinates and three-dimensional rotational attitude of the fastener in the unified world coordinate system. Step S43: From the preset digital twin model, retrieve the theoretical three-dimensional position coordinates and theoretical three-dimensional rotational attitude corresponding to the fastener; Step S44: Calculate the Euclidean distance between the solved three-dimensional position coordinates and the theoretical three-dimensional position coordinates, and use it as the position offset. The angle difference between the solved three-dimensional rotational attitude and the theoretical three-dimensional rotational attitude is calculated and used as the attitude angle offset. ; Step S45, the position offset and the attitude angle offset If any component exceeds the corresponding preset threshold, the fastener is determined to be loose.

6. The method for detecting assembly anomalies in hydraulic equipment fasteners based on computer vision as described in claim 1, characterized in that, In step S50, based on the digital twin model, a theoretical attribute relationship graph is constructed with theoretical fasteners as nodes and their spatial adjacency relationships as edges. The node attributes include fastener type, specifications, and theoretical pose. Meanwhile, based on the deep recognition results of step S30 and the detection results of step S40, actual fastener instances are obtained. Using their three-dimensional spatial positions as nodes, an instance attribute relationship graph is constructed. The node attributes include the identified category, specifications, calculated current pose, and three-dimensional position and attitude angle offset.

7. The method for detecting assembly anomalies in hydraulic equipment fasteners based on computer vision as described in claim 1, characterized in that, In step S50, the rules for collaborative reasoning include: If a fastener node in the theoretical attribute relationship diagram has no matching node in the instance attribute relationship diagram, it is determined that the fastener is missing. If the attributes of a fastener node in the theoretical attribute relationship diagram are inconsistent with those of the matching node in the instance attribute relationship diagram, it is determined that the fastener is incorrectly installed. If a fastener node in the theoretical attribute relationship diagram has the same attributes as a matching node in the instance attribute relationship diagram, but the three-dimensional position offset or attitude angle offset of the fastener node exceeds the threshold, it is determined that the fastener is loose. If a fastener node in the theoretical attribute relationship diagram has the same attributes as a matching node in the instance attribute relationship diagram, and the three-dimensional position offset and attitude angle offset of the fastener node both exceed the threshold, but its identification confidence is lower than the preset confidence threshold, it is marked as a suspected fastener assembly anomaly.

Images