A locking wire visual inspection method based on semantic segmentation

Abstract

The application discloses a locking wire visual inspection method based on semantic segmentation, and comprises the following contents: S1, acquiring a pipe joint locking wire training set; S2, pipe locking wire existence fusion judgment based on semantic segmentation; and S3, locking wire locking direction discrimination based on a segmented image. The method solves the problems that in the prior art, artificial inspection of aviation pipe joint locking wires is prone to missing inspection and wrong inspection, and the accuracy of the locking wire discrimination algorithm is not high at the present stage.

Description

Technical Field

[0001] The present invention belongs to the field of image target detection, and in particular relates to a locking wire visual inspection method based on semantic segmentation. Background Art

[0002] The assembly process of aviation and aerospace engines involves a large number of pipe assemblies. Currently, most pipe assemblies use a combination of pipe joints and locking wires to tighten and seal the pipe joints. At the same time, it is also necessary to ensure that the locking direction of the locking wire is the tightening direction of the pipe joint nut, that is, clockwise. The structure is shown in the figure. Figure 1 As shown in the figure, due to the large number of such pipe locking screws in aviation and aerospace engines, it is inevitable that locking screws will be missed or installed in the wrong direction during manual assembly, which may affect flight safety. Therefore, it is necessary to conduct a visual inspection of the engine after the sub-assembly and final assembly to ensure that the pipe locking screws are not installed incorrectly or missed.

[0003] Currently, visual inspection of locking threads in pipe joints used in aviation and aerospace engines is primarily performed manually by process technicians. Due to the large number of inspections required, the entire engine surface, and the small size and complex manufacturing process of locking threads, manual inspections are prone to missed and incorrect inspections. Furthermore, this method is relatively inefficient, making it difficult to adapt to the large-scale production and increasingly complex structures of aviation and aerospace engines. Summary of the Invention

[0004] The purpose of the present invention is to provide a visual inspection method for locking wires based on semantic segmentation to solve the problems in the prior art that manual inspection of locking wires of aviation pipe joints is prone to missed detection and false detection, and the accuracy of the current locking wire identification algorithm is not high.

[0005] The present invention is implemented by the following technical solutions:

[0006] A locking wire visual inspection method based on semantic segmentation, comprising the following contents:

[0007] S1. Obtain the pipe joint locking wire training set:

[0008] Design and manufacture physical templates for pipe joint locking threads to simulate real-world environments; manually create positive and negative samples of locking threads on the templates; arrange image acquisition devices and capture positive and negative sample images of pipe joint locking threads; collect and ultimately form a sample set of pipe joint locking threads under multiple lighting conditions; create mask label files required for semantic segmentation network training, generate mask images that correspond one-to-one with the sample set, and combine these with the pipe joint locking thread sample set to form a pipe joint locking thread training set;

[0009] S2. Fusion determination of pipeline locking wire existence based on semantic segmentation:

[0010] The pipe joint locking wire training set obtained in step S1 is input into a deep learning segmentation network to train locking wire features; the framework structure of the deep learning training network model is optimized, and feature learning is performed on the locking wire images in the pipe joint locking wire training set, ultimately generating model weights that incorporate locking wire features; locking wire segmentation prediction is performed in a real environment and a preliminary determination is made as to whether a locking wire is present at the pipe joint; the segmented image is further processed and determined to obtain a final conclusion as to whether a locking wire is present at the pipe joint;

[0011] S3. Determination of the locking direction of the locking wire based on the segmented image:

[0012] The individual images of the locking wire in the original image are obtained by cutting out the original image; the characteristic points of the locking wire's twisted shape are extracted from the main locking wire segment; the direction of the locking wire is fitted and the final conclusion is drawn: the locking direction is correct or incorrect; based on the empty hole position n1 and the direction of the locking wire's characteristic points, it is determined whether the circumferential range of the locking wire winding meets the assembly process requirements.

[0013] Furthermore, in step S2, the specific method for optimizing the framework structure of the deep learning training network model is as follows: the backbone of the segmentation network model is the backbone of the VGG network model, and the pipe joint locking wire training model is divided into two parts, namely the backbone feature extraction network and the enhanced feature extraction network; wherein, the backbone feature extraction part is responsible for obtaining 5 preliminary effective feature layers, and the enhanced feature extraction network is responsible for upsampling the 5 preliminary effective feature layers and performing feature fusion, and finally obtaining an effective feature layer that integrates all locking wire features.

[0014] Furthermore, in step S2, the locking wire segmentation prediction process is specifically as follows: the image of the real pipe joint area to be tested is input into the locking wire feature model weight obtained after training and combined with the image segmentation prediction algorithm, and finally an output image of the pipe joint area with the same height and width as the input image is output. All other pixel information in the output image is masked, and only the image area of ​​the locking wire and the hole position is retained and a unified pixel point is generated in this area. At this time, it is preliminarily determined that a locking wire exists at the pipe joint.

[0015] Furthermore, in step S2, the segmented image is further processed and judged as follows: the pixel area occupied by the largest circumscribed rectangle of the locking wire is defined as S 矩 The total area of ​​all pixels in the locking wire area is S 实 , then the actual proportion of locking wire pixels A locking thread pixel ratio threshold Y is set according to different situations of locking threads of different pipe joints. When the actual pixel ratio of the locking thread P ≥ Y, it is determined that the locking thread exists. When the actual pixel ratio of the locking thread P < Y, it is determined that the locking thread does not exist.

[0016] Further, in step S3, the cutting process of the individual locking wire image is: after obtaining the segmented image, record the position information of all pixel points of the locking wire, since the segmented image and the pipe joint original image have the same height and width, the recorded coordinate information is used to cut the individual locking wire image in the original image.

[0017] Further, in step S3, the process of extracting the locking wire spiral feature point is: in the individual locking wire image, the main body segment of the locking wire around the pipe joint is found, and the locking wire monomer is subjected to Gaussian filtering, highlighting and binary image processing operations, and the spiral feature point of the locking wire in the remaining main body locking wire segment is extracted.

[0018] Further, in step S3, the method for fitting and judging the correctness of the locking wire direction is: according to the relationship between the fitting point and the image, the direction of the fitting curve is used to replace the actual winding direction of the locking wire; according to the fitting result of the feature point, the final conclusion of correct locking direction or incorrect locking direction can be obtained by using the slope judgment method.

[0019] Further, in step S3, the locking wire winding direction range determination method is: the hole position and the locking wire feature point direction are used to determine whether the locking wire winding direction range meets the assembly process requirements; that is:

[0020] If the locking direction is correct and the hole position is located at the end of the fitting curve in the opposite direction, 180°≤locking wire winding direction range≤360°, and the winding direction range is correct.

[0021] If the hole position is not detected and the absolute value of the locking wire slope is too small, it is determined that the winding direction range is incorrect.

[0022] Compared with the prior art, the present application has the following beneficial technical results:

[0023] 1) The pipe joint locking wire training set production process of the present application is exquisite, considering the problem of limited visual angle in actual engineering application of aviation and aerospace engines, the present application proposes a set of multi-model pipe joint locking wire mechanism entity sample, monocular light source locking wire image acquisition device and a training set production method for improving the inclusiveness of the pipe joint locking wire training sample set, which can realize the recognition of the locking wire under extreme visual angle, multiple models and different lighting conditions.

[0024] 2) The present application adopts a semantic segmentation method based on deep learning, which can effectively solve the interference of other targets in the locking wire region on the determination result in the judgment, and improve the detection accuracy of the pipe joint locking wire.

[0025] 3) This invention employs a method for determining the tightening direction of locking wires based on segmented images. By extracting the locking wire's characteristic points from the segmented image instead of the actual locking wire itself, this method effectively mitigates the impact of image quality on direction determination. Furthermore, it utilizes the remaining holes in pipe joints as an auxiliary method for determination, making the algorithm more sensitive to assembly processes. Overall, this method significantly improves the reliability of locking wire direction determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention relates to an example of an aviation and aerospace engine pipe joint and its locking wire;

[0027] Figure 2 This is a structural schematic diagram of a locking wire image acquisition device with a monocular light source in a specific embodiment of the present invention;

[0028] Figure 3 This is a design diagram of a three-dimensional sample model of locking wires for various types of engine pipe joints in a specific embodiment of the present invention;

[0029] Figure 4-1 Schematic diagram of the back view of the annotation method used in the training set of pipe joint locking wires in a specific embodiment of the present invention;

[0030] Figure 4-2 A schematic diagram from a forward perspective of the labeling method used in the pipe joint locking wire training set in a specific embodiment of the present invention;

[0031] Figure 5-1 This is an output diagram of the locking wire segmentation algorithm in the correct direction type pipe joint in the specific embodiment of the present invention;

[0032] Figure 5-2 This is an output diagram of the locking wire segmentation algorithm in the wrong direction type pipe joint in a specific embodiment of the present invention;

[0033] Figure 6-1 This is a diagram showing characteristic points of locking wires with correct directions and their fitting results in a specific embodiment of the present invention;

[0034] Figure 6-2 This is a diagram showing characteristic points of the locking wire of the wrong direction and its fitting results in a specific embodiment of the present invention;

[0035] Figure 7 This is a flow chart of a method for visual inspection of locking wires based on semantic segmentation of the present invention. DETAILED DESCRIPTION

[0036] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. It should be understood that the examples described are only preferred embodiments of the present invention and are not exhaustive. Based on the embodiments in the embodiments, other embodiments obtained by those skilled in the art without making any creative efforts are all within the scope of protection of the present invention.

[0037] The present invention provides a locking wire visual inspection method based on semantic segmentation, which specifically includes the following steps:

[0038] S1. The method for obtaining the pipe joint locking wire training set is as follows Figure 7 As shown, specifically:

[0039] S110. Design and produce physical samples of pipe joint locking wires to simulate the real environment: inspect the prototypes of aviation and aerospace engine piping systems, and draw three-dimensional sample models of engine pipe joint locking wires of various types, including but not limited to straight pipe joints, right-angle pipe joints, elbow pipe joints, etc. Figure 3 As shown, a physical template can be made that includes locking wire mechanisms of multiple models of pipe joints, which is used to shoot a data set for semantic segmentation of locking wires based on deep learning methods.

[0040] S120. Manually create positive and negative samples of locking threads on the sample: Manually create positive and negative samples of locking threads required for deep learning training on the sample pipe joint entity formed in S110. Positive samples are samples where the pipe joint locking thread exists and is locked in the correct direction, i.e., the locking direction is clockwise when viewed along the axial direction of the pipe joint. Negative samples are samples where the pipe joint locking thread does not exist or where the pipe joint locking thread exists but is locked in the wrong direction, i.e., the locking direction is counterclockwise when viewed along the axial direction of the pipe joint.

[0041] S130. Deploy an image acquisition device and collect positive and negative sample images of the pipe joint locking thread: Deploy a monocular light source locking thread image acquisition device to collect positive and negative sample images of multiple models of aviation and aerospace pipe joint locking threads from different viewing angles. Collect images of the pipe joint locking threads from at least five viewing angles, including the front, left and right sides, and top and bottom sides of the pipe joint. This ensures that even if one view angle of the pipe joint is obstructed, the presence of the locking thread can be determined from other viewing angles.

[0042] Among them, such as Figure 2As shown, the image acquisition device for locking wires using a single-lens light source includes: an industrial camera lens module with a reasonable focal length and moderate depth of field, and two rows of oblique side strip light sources, which are strip light sources I and II. The specific camera parameters are determined by the linear distance between the camera's photosensitive chip and the engine's outer envelope. To filter out the effects of other colors on the surface of the pipe joint on the algorithm's later recognition of the locking wire target, the image captured by the industrial camera is a 24-bit grayscale image. The strip light source can continuously provide the camera with the straight parallel light required for image acquisition. The illumination intensity of each light source can be individually adjusted between 0 and 255 levels. During acquisition, it is placed between the camera and the pipe joint to be inspected, as shown in the following figure. Figure 2 As shown in the arrangement, the strip light source should maintain a certain angle α with the camera and the straight line where the pipe joint to be inspected is located. The optimal bevel angle range is 30°≤α≤60° to ensure that the light can generate sufficient diffuse light on the surface of the locking wire to enter the camera photosensitive chip.

[0043] S140. Collect and ultimately form a sample set of pipe joint locking threads under multiple lighting conditions: adjust the light intensity of the bar light sources on both sides of the monocular light source locking thread image acquisition device to improve the inclusiveness of the pipe joint locking thread training sample set.

[0044] The specific method is as follows: (1) when collecting the first set of pipe joint images, the light intensity of the bar light source I and the bar light source II are simultaneously illuminated from 0, 10, 20 to 250 points in 26 levels to the locking wire entity; (2) when collecting the second set of pipe joint images, the bar light source I is illuminated from 0, 10, 20 to 250 points in 26 levels to the locking wire entity, and the light intensity of the bar light source II is kept at 50; (3) when collecting the third set of pipe joint images, the bar light source II is illuminated from 0, 10, 20 to 250 points in 26 levels to the locking wire entity, and the light intensity of the bar light source I is kept at 50, and finally a positive and negative sample set of the pipe joint locking wire is formed;

[0045] S150. Create the mask label file required for semantic segmentation network training, generate a mask image that corresponds one-to-one to the sample set, and form a pipe joint locking wire training set together with the pipe joint locking wire sample set: Create the mask label file required for semantic segmentation network training, generate a locking wire positive sample set and a locking wire direction error negative sample set for S140, and use the marking tool to mark the target area to be segmented on its image.

[0046] The specific method is: Figure 4-1 As shown in the figure, the starting point of the back-view locking wire is defined as t1. Under correct assembly conditions, the locking wire starts from the hole t1 on the right end of the pipe joint, winds along the locking direction of the pipe joint, and finally penetrates and fixes at the other end of the pipe interface. In this example, Figure 4-2 In the black area of ​​the left hole, please note that the solid line is the visible part of the locking wire from this perspective, and the dotted line is the invisible part of the locking wire from this perspective. Figure 4-2 As shown in the figure, the vacant hole position from the forward perspective is defined as n1. T1 and n1 are symmetrically arranged 180° along the axial direction of the pipe joint. Therefore, after the locking wire is assembled, one position is t1, and the remaining position is n1. Using the universal marking tool on the pipe joint image, define the locking wire wrapping area as "lockwire" and the vacant hole position as "not_through," and then generate mask label files with different colors for the two positions.

[0047] The present invention meticulously creates a training set for pipe joint locking threads. Considering the limited viewing angles required for photographing aviation and aerospace engines in practical engineering applications, the present invention proposes a set of physical prototypes of multiple pipe joint locking thread mechanisms, a monocular light source locking thread image acquisition device, and a method for creating a training set that improves the inclusiveness of the training set. This allows for locking thread recognition under extreme viewing angles, multiple models, and varying lighting conditions. The image acquisition device captures images of the pipe joint locking threads from at least five viewing angles, including the front, left and right sides, and top and bottom sides of the pipe joint. This ensures that even if one view is obstructed, the presence of the locking thread can be determined from other viewing angles.

[0048] S2. A fusion determination method for the existence of pipeline locking wires based on semantic segmentation, such as Figure 7 As shown, specifically:

[0049] S210: Input the pipe joint locking wire training set obtained in step S1 into the deep learning segmentation network to train the locking wire features:

[0050] The pipeline joint training set images are input and the locking thread features are trained using an Encoder-Decoder deep learning segmentation network. Step S150 obtains multiple multi-view semantic segmentation training images of pipeline joint locking threads and their mask label files. These images are used as input to the locking thread segmentation network. The segmentation network model's backbone feature extraction network and enhanced feature extraction network are used to obtain the feature points of the pipeline joint locking threads. Finally, a prediction network is used to determine the feature points.

[0051] S220. Optimize the framework structure of the deep learning training network model, perform feature learning on the locking wire images in the pipe joint locking wire training set, and finally generate a model weight that integrates the locking wire features.

[0052] S230: Perform locking wire segmentation prediction under a real environment and preliminarily determine whether a locking wire exists at a pipe joint.

[0053] S240: further process and determine the segmented image to obtain a final conclusion on whether a locking wire exists at the pipe joint.

[0054] The present invention provides a fusion determination method for the presence of a pipeline locking wire based on semantic segmentation, including locking wire target contour extraction based on semantic segmentation and locking wire presence determination based on a pixel area discrimination method. For an image of a pipe joint area captured by an image acquisition device, the locking wire target contour extraction step based on semantic segmentation identifies the locking wire position and extracts its contour, and generates a unified pixel point in the image area where the locking wire is located. The locking wire presence determination step based on a pixel area discrimination method further performs locking wire determination on the image after contour extraction, and finally generates a locking wire presence determination result for the area: whether the locking wire exists or not.

[0055] To address the low accuracy of identifying locking threads for aviation and aerospace pipelines, the present invention proposes a pipeline locking thread existence fusion determination module based on semantic segmentation. By optimizing the existing segmentation network model, the mean intersection over union (MIOU) of the locking thread segmentation algorithm is made ≥ 98%. Preferably, the accuracy of the locking thread identification algorithm is further improved by calculating the pixel ratio of the segmented image and fusing it with the segmentation determination result, achieving a 100% detection rate for missing locking threads in pipe joints. The mean intersection over union (MIOU) is calculated as follows:

[0056]

[0057] Among them, k represents the number of segmentation targets, i represents the true value, j represents the predicted value, and P ij Indicates that i is predicted to be j, P ji Indicates that j is predicted to be i, P ii It means predicting i as i.

[0058] S3, a method for distinguishing the locking direction of the locking wire based on the segmented image, such as Figure 7 As shown, specifically:

[0059] S310: After finally determining that the locking wire exists, the segmented image of the locking wire generated in step S230 is input and cropped to obtain a real image of the locking wire in the original image of the pipe joint.

[0060] S320, finding and cutting the locking wire that goes around the pipe joint toward the main body, and extracting the twisted feature points of the locking wire.

[0061] S330: Using the extracted locking wire feature points to replace the locking wire body itself, and using its direction to fit the locking wire direction.

[0062] S340: Based on the hole position n1 and the direction of the locking wire characteristic point obtained in step S230, determine whether the circumferential range of the locking wire meets the assembly process requirements.

[0063] The purpose of the locking wire locking direction determination method based on segmented images of the present invention is to determine whether the locking wire locking direction meets the locking wire assembly process requirements of the aircraft engine pipe joint after obtaining the locking wire contour extraction image of the pipe joint image.

[0064] Existing locking wire direction determination algorithms fail to address issues such as decreased accuracy due to image quality and insufficient integration with assembly processes. Therefore, this paper proposes a locking wire direction determination module based on segmented images. By extracting locking wire feature points from the segmented images instead of the locking wire itself, this effectively mitigates the impact of image quality on direction determination. It also utilizes residual holes in pipe joints to aid in determination, allowing the algorithm to better understand assembly processes. Overall, this significantly improves the reliability of locking wire direction determination.

[0065] In some embodiments, the specific method for optimizing the framework structure of the deep learning training network model in S220 is: the existing available segmentation network backbone feature part is mainly composed of convolution and maximum pooling stacking, and the segmentation network model backbone used in step S210 replaces the classic network model backbone with the VGG network model backbone. After the replacement, the pipe joint locking wire training model is divided into two parts: the backbone feature extraction network and the enhanced feature extraction network. The backbone feature extraction part is responsible for obtaining 5 preliminary effective feature layers, and the enhanced feature extraction network is responsible for upsampling these 5 preliminary effective feature layers and performing feature fusion, and finally obtaining an effective feature layer that integrates all locking wire features.

[0066] In some embodiments, the locking wire segmentation prediction process in S230 is specifically as follows: the image of the real pipe joint area to be tested is input into the locking wire feature model weight obtained after training and combined with the image segmentation prediction algorithm, and finally an output image of the pipe joint area with the same height and width as the input image is output. Take the test images of the locking wire direction correct class and the locking wire direction incorrect class as examples, as shown in FIG. Figure 5-1 and Figure 5-2 As shown, all other pixel information in the output image is masked, only the image area of ​​the locking wire and the hole position is retained and a unified pixel point is generated in this area. At this time, it is preliminarily determined that the locking wire exists at the pipe joint.

[0067] In some embodiments, the further processing and determination process of the segmented image in S240 is specifically as follows: Since there are certain noise points in the deep learning segmentation network that affect the existence determination result, the pixel area occupied by the maximum circumscribed rectangle of the locking wire in Figure 5 is defined as S 矩 The total area of ​​all pixels in the locking wire area is S 实 , then the actual proportion of locking wire pixels A locking thread pixel ratio threshold Y is set according to different situations of locking threads of different pipe joints. When the actual pixel ratio of the locking thread P ≥ Y, it is determined that the locking thread exists. When the actual pixel ratio of the locking thread P < Y, it is determined that the locking thread does not exist.

[0068] In some embodiments, in S310, the cropping process of the locking wire individual image is as follows: after obtaining the segmented image, the position information of all pixel areas of the locking wire is recorded. Since the segmented image has the same height and width as the original image of the pipe joint, the recorded coordinate information is used to crop the original image to obtain the individual image of the locking wire in the original image.

[0069] In some embodiments, in S320, the process of extracting the twisted feature points of the locking wire is specifically as follows: finding the locking wire section that goes around the pipe joint to the main body in the locking wire individual image, and at the same time, subjecting the locking wire monomer to Gaussian filtering, highlighting, and binarization image processing operations, and extracting the twisted feature points of the locking wire in the remaining main locking wire segment.

[0070] In some embodiments, in S330, the method for fitting and determining the correctness of the locking wire direction is specifically as follows: the image conditions of the locking wire body itself are complex, and the method of feature point extraction can simplify its context and facilitate fitting the overall locking wire direction. Figure 5-1 For example, the final fitting effect of the correct locking wire direction is as follows Figure 6-1 As shown, Figure 5-2 The final fitting effect of the locking wire direction error class is as follows Figure 6-2 As shown in the figure. Based on the relationship between the fitting points and the image, the direction of the fitting curve is used to replace the actual winding direction of the locking wire. Based on the fitting results of the feature points, the slope discrimination method can be used to finally draw a final conclusion: the locking direction is correct or incorrect.

[0071] In some embodiments, in S340, the method for determining the circumferential range of the locking wire winding direction is specifically as follows: using the empty hole position n1 and the direction of the locking wire characteristic point to determine whether the circumferential range of the locking wire winding direction meets the assembly process requirements; if the locking direction is correct and the n1 position is located in the opposite direction of the end of the fitting curve, then 180°≤the circumferential range of the locking wire winding direction≤360°, and it is determined that the winding direction range is correct; if the empty hole position n1 is not detected and the absolute value of the locking wire slope is too small, it is determined that the winding direction range is incorrect.

[0072] The production process of the pipe joint locking wire training set of the present invention is sophisticated. Taking into account the problem of limited actual shooting angle of aviation and aerospace engines in actual engineering applications, the present invention proposes a set of physical templates of multiple models of pipe joint locking wire mechanisms, a monocular light source locking wire image acquisition device and a training set production method that improves the inclusiveness of the pipe joint locking wire training sample set, which can realize locking wire recognition under extreme angles, multiple models and different lighting conditions.

[0073] The present invention adopts a semantic segmentation method based on deep learning to segment the main body of the target object. By calculating the pixel ratio of the segmented image and fusing it with the segmentation judgment result, it can effectively solve the interference of other targets in the locking wire area on the judgment result during the judgment, and improve the detection accuracy of the locking wire of the pipe joint.

[0074] This invention uses a segmented image-based method for determining the tightening direction of locking wires. By extracting the locking wire's characteristic points from the segmented image instead of the actual locking wire itself, this method effectively avoids the influence of image quality on direction determination. It also utilizes the remaining holes in pipe joints as an auxiliary method for determination, making the algorithm more sensitive to assembly processes. Overall, this method significantly improves the reliability of locking wire direction determination.

[0075] The above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A visual inspection method for locking wire based on semantic segmentation, characterized in that: Includes the following: S1. Obtain the pipe joint locking wire training set: Design and manufacture a physical sample of the pipe joint locking thread to simulate the real environment; manually make positive and negative samples of the locking thread on the sample; arrange an image acquisition device and capture images of the positive and negative samples of the pipe joint locking thread; Collect and eventually form a sample set of pipe joint locking wires under multiple lighting conditions; Create the mask label file required for semantic segmentation network training, generate mask images that correspond one-to-one with the sample set, and form the pipe joint locking wire training set together with the pipe joint locking wire sample set; S2. Fusion determination of pipeline locking wire existence based on semantic segmentation: Inputting the pipe joint locking wire training set obtained in step S1 into the deep learning segmentation network to train the locking wire features; optimizing the framework structure of the deep learning training network model, performing feature learning on the locking wire images in the pipe joint locking wire training set, and finally generating a model weight that integrates the locking wire features; Perform locking wire segmentation prediction in a real-world environment and preliminarily determine whether a locking wire exists at a pipe joint. Further process and determine the presence of a locking wire at a pipe joint using the segmented image to arrive at a final conclusion. Among them, the pipeline locking wire existence fusion judgment module based on semantic segmentation makes the locking wire segmentation algorithm's mean intersection over union (MIOU) ≥ 98%. By calculating the pixel ratio of the segmented image and fusing it with the segmentation judgment result, the accuracy of the locking wire recognition algorithm is improved. The mean intersection over union (MIOU) is calculated as follows: Among them, k represents the number of segmentation targets, i represents the true value, j represents the predicted value, and P ij Indicates that i is predicted to be j, P ji Indicates that j is predicted to be i, P ii It means predicting i as i; The further processing and judgment process of the segmented image is as follows: the pixel area occupied by the largest circumscribed rectangle of the locking wire is defined as S 矩 The total area of ​​all pixels in the locking wire area is S 实 , then the actual proportion of locking wire pixels A locking thread pixel ratio threshold Y is set according to the different situations of the locking threads of different pipe joints. When the actual pixel ratio of the locking thread P ≥ Y, it is determined that the locking thread exists. When the actual pixel ratio of the locking thread P < Y, it is determined that the locking thread does not exist. S3. Determination of the locking direction of the locking wire based on the segmented image: The original image is cropped to obtain an individual image of the locking wire; characteristic points of the locking wire's twisted shape are extracted from the main locking wire segment; the locking wire's direction is fitted and a final conclusion is drawn: the locking direction is correct or incorrect; based on the hole position n1 and the direction of the locking wire's characteristic points, whether the locking wire's winding circumference meets the assembly process requirements is determined; In the step S3, The method for fitting and determining the correctness of the locking wire direction is as follows: based on the relationship between the fitting points and the image, the direction of the fitting curve is used to replace the actual winding direction of the locking wire. Based on the fitting results of the feature points, the slope judgment method can finally be used to draw a final conclusion on whether the locking direction is correct or incorrect. The method for determining the circumferential range of the locking wire winding direction is as follows: using the position of the hole and the direction of the characteristic points of the locking wire, determine whether the circumferential range of the locking wire winding direction meets the assembly process requirements; that is: If the locking direction is correct and the hole position is in the opposite direction of the end of the fitting curve, then the winding range of the locking wire is 180°≤360°, and the winding range is correct. If the empty hole position is not detected and the absolute value of the locking wire slope is too small, it is determined that the winding range is wrong.

2. The method for visual inspection of locking wires based on semantic segmentation according to claim 1, characterized in that: In step S2, the specific method for optimizing the framework structure of the deep learning training network model is as follows: the backbone of the segmentation network model is the backbone of the VGG network model, and the pipe joint locking wire training model is divided into two parts, namely, the backbone feature extraction network and the enhanced feature extraction network; wherein, the backbone feature extraction part is responsible for obtaining 5 preliminary effective feature layers, and the enhanced feature extraction network is responsible for upsampling the 5 preliminary effective feature layers and performing feature fusion, and finally obtaining an effective feature layer that integrates all locking wire features.

3. The method for visual inspection of locking wires based on semantic segmentation according to claim 2, characterized in that: In step S2, the locking wire segmentation prediction process is specifically as follows: the image of the real pipe joint area to be tested is input into the locking wire feature model weight obtained after training, and the image segmentation prediction algorithm is used to finally output an output image of the pipe joint area with the same height and width as the input image. All other pixel information in the output image is masked, and only the image area of ​​the locking wire and the empty hole position is retained and a unified pixel point is generated in this area. At this time, it is preliminarily determined that a locking wire exists at the pipe joint.

4. The method for visual inspection of locking wires based on semantic segmentation according to claim 1, characterized in that: In step S3, the process of cutting the individual image of the locking wire is as follows: after obtaining the segmented image, the position information of all pixel areas of the locking wire is recorded. Since the segmented image has the same height and width as the original image of the pipe joint, the recorded coordinate information is used to cut the individual image of the locking wire in the original image.

5. The method for visual inspection of locking wires based on semantic segmentation according to claim 4, characterized in that: In step S3, the process of extracting the characteristic points of the twisted shape of the locking wire is specifically as follows: finding the locking wire section that goes around the pipe joint to the main body in the locking wire individual image, and at the same time, subjecting the locking wire monomer to Gaussian filtering, highlighting, and binarization image processing operations, and extracting the characteristic points of the twisted shape of the locking wire in the remaining main locking wire segment.

Images