A probe state detection method and system based on semantic segmentation and line fitting

Abstract

The application discloses a kind of based on semantic segmentation and straight line fitting's probe state detection method and system, belong to target detection technical field, including obtaining the probe image of shooting, classification model is classified to probe image and judges;Then the normal probe image is preprocessed, and using segmentation model is carried out the prediction of probe surface area, obtains the primary profile of probe;Then the primary profile is dilated and corroded and area threshold processing, obtains the fine profile of probe;Two column probes are carried out midline fitting, and profile is divided into two rows, and the centroid of two rows profile is respectively carried out straight line fitting, obtains two straight lines;According to the distance of each centroid of probe to the straight line of belonging, the concave-convex state of probe is judged, and the missing state of probe is judged according to the interval of probe profile projection.The application is used for probe state detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to a probe state detection method and system based on semantic segmentation and straight line fitting. BACKGROUND

[0002] With the pace of urban construction in China, the construction and development of the basic facilities of the transportation industry have made great progress, and the construction and maintenance of the railway track in China is an important part of the infrastructure, so how to ensure the safety and stability of train operation has become an important research direction. As a key component of train safe operation, the train wheel needs to be regularly safety detected.

[0003] In the prior art, the mainstream detection method in the railway track line relies on an ultrasonic flaw detection probe array device, which combines a large number of ultrasonic probes on the track plane to perform wheel flaw detection. However, the probe array may be damaged by the wheel and foreign matter during detection, and changes from a normal state to a concave or convex shape. In order to maintain the safe operation of the rail system, the state of the probe body needs to be checked. Since the probe array device has a large number of probe bodies and is located on the rail, manual screening detection has high operation risk, so the manual screening detection method is difficult to implement. Using image processing and deep learning technology, the image captured by the camera can be analyzed and detected, which can better solve the detection problem of the probe flaw detection device.

[0004] The existing two-stage target detection deep learning method based on faster-RCNN is used to detect the tilt of the ultrasonic probe device probe, and a new deformation proposal box is used to better represent the probe position with different rotation angles. Since the tilt state of the probe is very similar to the normal state, the target detection-based method cannot well predict the correct state of the probe, and the detection-based method cannot finely locate the probe position.

[0005] Or use the semantic segmentation method based on the U-net network to detect the defects of the part. In most segmentation method application scenarios, the defect part and the normal part are obviously different in color and texture. If such a method is used to distinguish the probe state with similar or identical local features, an undesirable model prediction result will be obtained. Therefore, such a method cannot be directly used to identify the detection of the concave-convex state of the probe. SUMMARY

[0006] In view of the problems in the prior art, the application provides a probe state detection method and system based on semantic segmentation and straight line fitting, which aims to use image processing and deep learning technology to detect whether each probe component of the probe flaw detection device has abnormal states such as concave, convex and missing, replace the manual screening method, and achieve high-precision and high-recall detection rate.

[0007] The technical scheme adopted by the present application to achieve the above-mentioned purpose is to provide a probe state detection method based on semantic segmentation and straight line fitting, comprising:

[0008] S1: obtaining a photographed probe image, a classification model classifies and judges the probe image, if the probe image is normal, the probe state is detected, if the probe image is abnormal, the camera is controlled to delay and re-shoot the probe image;

[0009] S2: pre-processing the normal probe image, and using a segmentation model to predict the probe surface area to obtain a primary contour map of the probe, including the primary contour and the centroid point of the probe;

[0010] S3: then dilating and eroding the primary contour map and performing area threshold processing to obtain a refined contour map of the probe, including the contour and the centroid point of the probe;

[0011] S4: fitting the midline of the two rows of probes, then dividing the contour into two rows based on the orientation of the centroid point relative to the midline, and performing straight line fitting on the centroid points of the two rows of contours respectively to obtain two straight lines;

[0012] S5: judging the concave-convex state of the probe according to the distance of each centroid point of the probe to the straight line to which it belongs, and judging the missing state of the probe according to the distance of the probe contour projection.

[0013] Preferably, the training process of the classification model in S1 is as follows:

[0014] S1.1: obtaining a photographed probe image, scaling the probe image to 224*224, and manually classifying it into four categories, specifically: normal image, blurred image, water spraying image and foreign matter image;

[0015] S1.2: reading the probe image in batches, and performing data enhancement on the probe image, the data enhancement including: horizontal flip, random cropping, adjusting brightness and Gamma;

[0016] S1.3: constructing a MobileNetV3 network, the basic architecture adopts MobileNetV3-small, the classifier of the last layer of the MobileNetV3 network is enhanced into a double-layer fully connected layer, and the middle layer adopts Hardswish activation unit and Dropout technology;

[0017] S1.4: inputting the data-enhanced probe image into the constructed MobileNetV3 network in batches, and then using Adam optimizer and cross-entropy loss function to optimize the model parameters of the MobileNetV3 network.

[0018] Preferably, the training process of the segmentation model in S2 of the present application is specifically as follows:

[0019] S2.1: Scale the input probe image to 480*480 and manually classify and label the probe surface;

[0020] S2.2: Then read the probe image in batches, and perform data enhancement on the probe image, wherein the data enhancement includes Gaussian blur, limited contrast adaptive histogram, equalization horizontal flip, random cropping, random brightness adjustment and Gamma;

[0021] S2.3: Construct a U-net segmentation network, wherein the skeleton network adopts Resnet18, and the loss function adopts a DiceLoss loss function;

[0022] S2.4: Input the probe image after data enhancement in S2.2 into the U-net segmentation network, and then use an Adam optimizer and a DiceLoss loss function to optimize the model parameters of the network.

[0023] Preferably, S3 of the present application is specifically as follows:

[0024] S3.1: Perform erosion and expansion on the obtained primary contour map, filter out the primary contour whose area is less than a minimum area threshold, and then extract each primary contour through a connected domain to store the information and centroid point of the primary contour;

[0025] S3.2: Determine whether the area of the stored contour is greater than a preset maximum area threshold, filter out the primary contour whose area is greater than the maximum area threshold, and also filter out the primary contour whose centroid point is outside the ROI region;

[0026] S3.3: Convert the contour shape into a minimum circumscribed rectangle to form a refined contour map, and obtain the contour of the probe, which is used to prevent the model from predicting an irregular surface to affect the subsequent state prediction.

[0027] Preferably, S4 of the present application is specifically as follows:

[0028] S4.1: Obtain the center line of all centroid points through a quadratic straight line fitting algorithm, wherein the definition of the quadratic straight line fitting algorithm is as follows: a distance threshold of an abnormal point is preset, the least square method is used to fit a straight line to obtain the center line for all centroid points, when the distance between some centroid points and the center line is greater than the distance threshold of the abnormal point, these abnormal contours are filtered out, and then the straight line fitting is performed again using the filtered centroid points until the distance between all centroid points and the center line is less than or equal to the distance threshold of the abnormal point, and the final center line is obtained;

[0029] S4.2: a preset probe number difference threshold value, the contours are divided into two rows according to the centroid points and the positions relative to the center line, if the difference between the number of probes allocated to the two rows is greater than the preset probe number difference threshold value, an alarm is given, and the secondary straight line fitting algorithm as described in step S4.1 is performed on the contours of the two rows respectively to fit the upper and lower straight lines.

[0030] Preferably, the concave-convex state of the probe in S5 of the present application is specifically:

[0031] The average area of the probe contour is calculated, the probe surface is taken as a square, the average side length of the probe surface is calculated according to the average area, the abnormal state threshold value is preset as 1 / 4 of the average side length of the probe surface, the distance from the centroid point of each contour to the straight line to which it belongs is calculated, if the distance is greater than the abnormal state threshold value, the probe is in a concave-convex state, and whether the probe is convex or concave is determined according to the position of the centroid point relative to the straight line, if the distance is not greater than the abnormal state threshold value, the probe is regarded as a normal probe.

[0032] Preferably, the missing state of the probe in S5 of the present application is specifically:

[0033] The contours of all probes are projected onto the straight lines to which they belong, the distance between the projected line segments of each contour on the straight line is calculated, and the average value thereof is calculated, if the distance between the projected line segments of adjacent probes is greater than 1.3 times the average value, the region between the two probes is predicted to be a missing probe, and whether the missing probe exists on both sides is determined by judging whether the distance from the probes on both sides to the ROI boundary is greater than 1.3 times the average value.

[0034] The present application also proposes a probe state detection system based on semantic segmentation and straight line fitting, comprising:

[0035] An image detection module: acquires a photographed probe image, classifies and judges the probe image through a classification model, if the probe image is normal, the probe state is detected, if the probe image is abnormal, the camera is controlled to delay and re-shoot the probe image;

[0036] An image preprocessing module: pre-processes the normal probe image, and uses a segmentation model to predict the probe surface region to obtain a primary contour map of the probe, including the primary contour of the probe and the centroid point.

[0037] An image fine module: then dilates and erodes the primary contour map and performs area threshold processing to obtain a fine contour map of the probe, including the contour of the probe and the centroid point.

[0038] A straight line fitting module: performs center line fitting on two rows of probes, then divides the contours into two rows according to the positions of the centroid points relative to the center line, and performs straight line fitting on the centroid points of the contours of the two rows respectively to obtain the upper and lower straight lines.

[0039] The probe state judging module judges the concave-convex state of the probe according to the distance from each mass center point of the probe to the straight line to which the mass center point belongs, and judges the missing state of the probe according to the spacing between the line segments formed by the projection of the probe contour onto the straight line.

[0040] Preferably, the training process of the classification model in the image detection module is specifically as follows:

[0041] Step 1.1: Obtain the photographed probe image, scale the probe image to 224*224, and artificially classify the probe image into four categories, specifically: normal image, blurred image, water spraying image and foreign matter image;

[0042] Step 1.2: Read the probe images in batches, and perform data enhancement on the probe images, wherein the data enhancement includes: horizontal flip, random cropping, adjusting brightness and Gamma;

[0043] Step 1.3: Construct a MobileNetV3 network, the basic architecture adopts MobileNetV3-small, the classifier of the last layer of the MobileNetV3 network is enhanced into a double-layer full connection layer, and the middle layer adopts a Hardswish activation unit and a Dropout technology;

[0044] Step 1.4: input the data-enhanced probe images into the constructed MobileNetV3 network in batches, and then use the Adam optimizer and the cross-entropy loss function to optimize the model parameters of the MobileNetV3 network.

[0045] Preferably, the training process of the segmentation model in the image preprocessing module is specifically as follows:

[0046] Step 2.1: scale the input probe image to 480*480, and artificially classify and label the probe surface;

[0047] Step 2.2: then read the probe images in batches, and perform data enhancement on the probe images, wherein the data enhancement includes: Gaussian blur, limited contrast adaptive histogram, equalization, horizontal flip, random cropping, adjusting brightness and Gamma;

[0048] Step 2.3: construct a U-net segmentation network, the skeleton network adopts Resnet18, and the loss function adopts a DiceLoss loss function;

[0049] Step 2.4: input the data-enhanced probe images in S2.2 into the U-net segmentation network in batches, and then use the Adam optimizer and the DiceLoss loss function to optimize the model parameters of the network.

[0050] Compared with the prior art, the technical scheme of the present application has the following advantages / benefits:

[0051] 1. The present application greatly improves the representation accuracy of the probe contour by using the U-net segmentation-based method to locate and represent the upper surface of the probe, making the detection result of the probe more accurate and accurate.

[0052] 2. The present application uses a quadratic line fitting algorithm to fit the centroid points between the probes, which can efficiently and accurately determine whether the probes are on the same horizontal line, and uses the contour projection method to accurately determine whether there is a missing between the probes, thereby improving the accuracy of the probe state prediction.

[0053] 3. The present application is aimed at the position distribution of the probe, and by fitting two straight lines above and below, it can better predict the probe state in the application scenario.

[0054] 4. The present application sets multiple hard parameter thresholds, such as minimum and maximum area thresholds, ROI regions, and abnormal point distance thresholds, which effectively improve the detection effect of some abnormal conditions, and can be adjusted to adapt to different scenes, such as different probe models. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical scheme of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, it should be understood that the following drawings only show some embodiments of the present application, therefore should not be regarded as a limitation on the scope, for those skilled in the art, without creative labor, other related drawings can also be obtained from these drawings.

[0056] Figure 1 is a flowchart of a probe state detection method based on semantic segmentation and line fitting according to the present application.

[0057] Figure 2 is a contour diagram of the probe according to embodiment 1 of the present application.

[0058] Figure 3 is a diagram showing two straight lines fitted according to embodiment 1 of the present application.

[0059] Figure 4 is a diagram showing the line segment formed by projecting the probe contour to the straight line according to embodiment 1 of the present application.

[0060] Figure 5 is a diagram showing the prediction result according to embodiment 1 of the present application. DETAILED DESCRIPTION

[0061] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. Therefore, the detailed description of the embodiments of the present application provided below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application.

[0062] Embodiment 1

[0063] The embodiment 1 proposes a probe state detection method based on semantic segmentation and line fitting, which comprises:

[0064] S1: acquiring the photographed probe image, the classification model classifies and judges the probe image, if the probe image is normal, the probe state detection is performed, if the probe image is abnormal, the camera is controlled to delay and re-photograph the probe image; the training process of the classification model in S1 is specifically:

[0065] S1.1: acquiring the photographed probe image, scaling the probe image into 224*224, and artificially classifying into four categories, specifically: normal image, blurred image, water spraying image and foreign matter image;

[0066] S1.2: reading the probe image in batches, and performing data enhancement on the probe image, the data enhancement comprising: horizontal flip, random cropping, adjusting brightness and Gamma;

[0067] S1.3: constructing a MobileNetV3 network, the basic architecture adopts MobileNetV3-small, enhancing the classifier of the last layer of the MobileNetV3 network into a double-layer fully connected layer, and using Hardswish activation unit and Dropout technology in the middle layer;

[0068] S1.4: inputting the data-enhanced probe image into the constructed MobileNetV3 network in batches, and then using Adam optimizer and cross-entropy loss function to optimize the model parameters of the MobileNetV3 network.

[0069] S2: pre-processing the normal probe image, and using the segmentation model to predict the probe surface area to obtain the primary contour map of the probe, including the primary contour and the centroid point of the probe; as shown in Figure 2

[0070] The training process of the segmentation model in S2 is specifically:

[0071] ​S2.1: scale the input probe image to 480*480 and manually classify and label the probe surface;

[0072] S2.2: Then read the probe image in batches and perform data augmentation on the probe image, including Gaussian blur, limited contrast adaptive histogram, equalization horizontal flip, random cropping, random brightness adjustment and Gamma;

[0073] S2.3: Construct a U-net segmentation network, the skeleton network uses Resnet18, and the loss function uses DiceLoss loss function;

[0074] S2.4: input the probe image after S2.2 data augmentation into the U-net segmentation network, and then use the Adam optimizer and the DiceLoss loss function to optimize the model parameters of the network.

[0075] S3: Then dilate and erode the primary contour map and perform area thresholding to obtain a refined contour map of the probe, including the contour and centroid point of the probe; S3 is specifically:

[0076] S3.1: erode and dilate the obtained primary contour map, filter out the primary contour whose area is less than the minimum area threshold, and then extract each primary contour by connected domain and store the information and centroid point of the primary contour;

[0077] S3.2: determine whether the stored contour area is greater than the preset maximum area threshold, filter out the primary contour whose area is greater than the maximum area threshold, and also filter out the primary contour whose centroid point is outside the ROI region;

[0078] S3.3: convert the contour shape to the minimum circumscribed rectangle to form a refined contour map and obtain the contour of the probe, which is used to prevent the model from predicting an irregular surface that affects the subsequent state prediction.

[0079] S4: fit the middle line of the two rows of probes, then divide the contour into two rows based on the orientation of the centroid point relative to the middle line, and perform linear fitting on the centroid points of the two rows of contours respectively to obtain two straight lines; as shown in Figure 3 .

[0080] S4 is specifically:

[0081] S4.1: Obtain the center line of all centroid points by a quadratic linear fitting algorithm, the definition of the quadratic linear fitting algorithm is that: preset an abnormal point distance threshold, use the least square method to fit a straight line to obtain the center line for all centroid points, when some centroid points to the center line distance is greater than the abnormal point distance threshold, filter these abnormal contours, then re-fit a straight line using the filtered centroid points, until all centroid points to the center line distance is less than or equal to the abnormal point distance threshold, the final center line is obtained;

[0082] S4.2: preset a probe number difference threshold, according to the centroid points and the orientation relative to the center line, the contour is divided into two rows, if the difference of the number of probes allocated to the two rows is greater than the set probe number difference threshold, an alarm is given, and then the quadratic linear fitting algorithm described in step S4.1 is performed on the two rows of contours respectively to fit two straight lines. At this time, in the quadratic linear fitting algorithm, all centroid points are the centroid points of the column of probes instead of the centroid points of the two columns of probes because two straight lines are to be fitted.

[0083] S5: According to the distance of each centroid point of the probe to the straight line to which it belongs, the concave-convex state of the probe is judged, and the missing state of the probe is judged according to the distance between the projections of the probe contours. As shown in Figure 5 , in the prediction result, the black probe contour is a normal probe contour, and the gray one is an abnormal probe. Here, the gray color is other colors that are obviously different from black in actual application.

[0084] The specific method of judging the concave-convex state of the probe in S5 is as follows:

[0085] The average area of the probe contour is calculated, the probe surface is regarded as a square, the average side length of the probe surface is calculated according to the average area, the abnormal state threshold is preset as 1 / 4 of the average side length of the probe surface, the distance of each contour centroid point to the straight line to which it belongs is calculated, if the distance is greater than the abnormal state threshold, the probe is in a concave-convex state, and according to the orientation of the centroid point relative to the straight line, it is judged whether the probe is convex or concave, if the distance is not greater than the abnormal state threshold, the probe is regarded as a normal probe.

[0086] The specific method of judging the missing state of the probe in S5 is as follows:

[0087] As shown in Figure 4 , project all probe contours onto the straight line to which they belong, calculate the distance between the projection line segments of each contour on the straight line, and calculate the average value, if the distance between the projection line segments of adjacent probes is greater than 1.3 times the average value, the area between the two probes is predicted to be a missing probe; then judge whether the distance of the two side probes to the ROI boundary is greater than 1.3 times the average value to judge whether there is a missing probe on both sides.

[0088] The embodiment 1 also proposes a probe state detection system based on semantic segmentation and line fitting, comprising:

[0089] An image detection module: acquires the photographed probe image, classifies and judges the probe image through a classification model, if the probe image is normal, detects the probe state, if the probe image is abnormal, controls the camera to delay and re-shoot the probe image; the training process of the classification model is specifically:

[0090] Step 1.1: acquire the photographed probe image, scale the probe image to 224*224, and artificially classify it into four categories, specifically: normal image, blurred image, water spraying image and foreign matter image;

[0091] Step 1.2: read the probe image in batches, and perform data enhancement on the probe image, the data enhancement includes: horizontal flip, random cropping, adjusting brightness and Gamma;

[0092] Step 1.3: construct a MobileNetV3 network, the basic architecture adopts MobileNetV3-smal, enhance the classifier of the last layer of the MobileNetV3 network into a double-layer fully connected layer, and use Hardswish activation unit and Dropout technology in the middle layer;

[0093] Step 1.4: input the data-enhanced probe image into the constructed MobileNetV3 network in batches, and then use Adam optimizer and cross-entropy loss function to optimize the model parameters of the MobileNetV3 network.

[0094] An image preprocessing module: pre-processes the normal probe image, and uses a segmentation model to predict the probe surface area to obtain a primary contour map of the probe, including the primary contour of the probe and the centroid point; the training process of the segmentation model is specifically:

[0095] Step 2.1: scale the input probe image to 480*480, and manually classify and label the probe surface;

[0096] Step 2.2: then read the probe image in batches, and perform data enhancement on the probe image, the data enhancement includes: Gaussian blur, limited contrast adaptive histogram, equalization horizontal flip, random cropping, random adjusting brightness and Gamma;

[0097] Step 2.3: construct a U-net segmentation network, the skeleton network adopts Resnet18, and the loss function adopts DiceLoss loss function;

[0098] Step 2.4: The probe image after data enhancement in S2.2 is input into the U-net segmentation network in batches, and then the model parameters of the network are optimized by using the Adam optimizer and the DiceLoss loss function.

[0099] Image refinement module: Then, the primary contour map is subjected to expansion and corrosion and area threshold processing to obtain a refined contour map of the probe, including the contour and the centroid point of the probe;

[0100] Straight line fitting module: The midline of the two columns of probes is fitted, then the contour is divided into two rows based on the orientation of the centroid point relative to the midline, and the centroid points of the two rows of contours are subjected to straight line fitting respectively to obtain two straight lines;

[0101] Probe state judgment module: According to the distance of each centroid point of the probe to the straight line to which it belongs, the concave-convex state of the probe is judged, and according to the spacing between the line segments formed by the projection of the probe contour onto the straight line to which it belongs, the missing state of the probe is judged.

[0102] The above is only the preferred embodiment of the present application, it should be pointed out that the above preferred embodiment should not be regarded as limiting the present application, the protection scope of the present application should be limited by the scope defined by the claims. For ordinary skilled in the art, without departing from the spirit and scope of the present application, a number of improvements and refinements can also be made, which should be regarded as the protection scope of the present application.

Claims

1. A probe state detection method based on semantic segmentation and line fitting, characterized in that, include: S1: Acquire the captured probe image. The classification model classifies and judges the probe image. If the probe image is normal, the probe status is detected. If the probe image is abnormal, the camera is controlled to delay and re-capture the probe image. S2: Preprocess the normal probe image and use the segmentation model to predict the probe surface area to obtain the primary contour map of the probe, including the primary contour and centroid of the probe. S3: Then, the primary contour map is subjected to dilatational erosion and area thresholding to obtain a refined contour map of the probe, including the probe's contour and centroid. S4: Fit the centerline of the two columns of probes, then divide the contour into upper and lower rows based on the orientation of the centroid point relative to the centerline, and fit the centroid points of the two rows of contours with straight lines to obtain the upper and lower straight lines. S5: Determine the concavity / convexity of the probe based on the distance from each centroid point of the probe to its corresponding straight line, and determine the missing state of the probe based on the spacing of the probe contour projection.

2. The probe state detection method based on semantic segmentation and line fitting according to claim 1, characterized in that, The specific training process for the classification model in S1 is as follows: S1.1: Acquire the captured probe image, scale the probe image to 224*224, and manually classify it into 4 categories: normal image, blurred image, water spray image, and foreign object image; S1.2: Read probe images in batches and perform data enhancement on the probe images. The data enhancement includes: horizontal flipping, random cropping, adjusting brightness and Gamma. S1.3: Construct the MobileNetV3 network. The basic architecture adopts MobileNetV3-small. The classifier of the last layer of the MobileNetV3 network is enhanced into a two-layer fully connected layer, and the intermediate layer adopts Hardswish activation unit and Dropout technology. S1.4: Input the augmented probe images into the constructed MobileNetV3 network in batches, and then optimize the model parameters of the MobileNetV3 network using the Adam optimizer and cross-entropy loss function.

3. The probe state detection method based on semantic segmentation and line fitting according to claim 1, characterized in that, The specific training process for the segmentation model in S2 is as follows: S2.1: Scale the input probe image to 480*480 and manually classify and label the probe surface; S2.2: Then, the probe images are read in batches and data enhancement is performed on the probe images. The data enhancement includes: Gaussian blur, limiting contrast adaptive histogram, equalizing horizontal flip, random cropping, and arbitrarily adjusting brightness and Gamma. S2.3: Construct the U-net segmentation network, using Resnet18 as the backbone network and DiceLoss loss function; S2.4: Input the probe images after data augmentation in S2.2 into the U-net segmentation network in batches, and then optimize the network model parameters using the Adam optimizer and the DiceLoss loss function.

4. The probe state detection method based on semantic segmentation and line fitting according to claim 1, characterized in that, S3 specifically refers to: S3.1: Erosion and dilation are performed on the obtained primary contour map to filter out primary contours whose contour area is less than the minimum area threshold. Then, each primary contour is extracted by connecting components, and the information and centroid of these primary contours are stored. S3.2: Determine whether the stored contour area is greater than the preset maximum area threshold, filter out the primary contours that are greater than the maximum area threshold, and also filter out the primary contours whose centroids are outside the ROI region. S3.3: Convert the contour shape into the minimum bounding rectangle to form a refined contour map, thus obtaining the probe's contour. This is used to prevent the model from predicting irregular surfaces that could affect subsequent state predictions.

5. The probe state detection method based on semantic segmentation and line fitting according to claim 1, characterized in that, S4 specifically refers to: S4.1: The midline of all centroids is obtained by a quadratic line fitting algorithm. The quadratic line fitting algorithm is defined as follows: a threshold for outlier distance is preset, and the midline is obtained by performing line fitting using the least squares method on all centroids. When the distance from some centroids to the midline is greater than the threshold for outlier distance, these outlier contours are filtered out. Then, the filtered centroids are used to perform line fitting again until the distance from all centroids to the midline is less than or equal to the threshold for outlier distance, and the final midline is obtained. S4.2: Set a threshold for the difference in the number of probes. Divide the contour into upper and lower rows according to the centroid and the orientation relative to the center line. If the difference in the number of probes allocated to the two rows is greater than the set threshold for the difference in the number of probes, an alarm will be triggered. Then, perform the quadratic straight line fitting algorithm as described in step S4.1 on the two rows of contours to fit the upper and lower straight lines.

6. The probe state detection method based on semantic segmentation and line fitting according to claim 1, characterized in that, The specific steps for determining the probe's convexity / concave state in S5 are as follows: Calculate the average area of ​​the probe contour. Treat the probe surface as a square and calculate the average side length of the probe surface based on the average area. Set the abnormal state threshold to 1 / 4 of the average side length of the probe surface. Calculate the distance from the centroid of each contour to its corresponding straight line. If the distance is greater than the abnormal state threshold, the probe is in a concave-convex state. The probe is judged to be convex or concave based on the orientation of the centroid relative to its corresponding straight line. If the distance is not greater than the abnormal state threshold, the probe is considered a normal probe.

7. The probe state detection method and system based on semantic segmentation and line fitting according to claim 1, characterized in that, The specific steps for determining the missing probe status in S5 are as follows: Project the contours of all probes onto their respective straight lines, calculate the distance between the projection segments of each contour on the straight line, and calculate their average. If the distance between the projection segments of adjacent probes is greater than 1.3 times the average, then the area between these two probes is predicted as a probe missing area. Then, determine whether the distance from the probes on both sides to the ROI boundary is greater than 1.3 times the average to determine whether there are probe missing areas on both sides.

8. A probe state detection system based on semantic segmentation and line fitting, characterized in that, include: Image detection module: Acquires the captured probe image, classifies and judges the probe image through a classification model. If the probe image is normal, the probe status is detected. If the probe image is abnormal, the camera is controlled to delay and re-capture the probe image. Image preprocessing module: preprocesses normal probe images and uses a segmentation model to predict the probe surface area to obtain the primary contour map of the probe, including the primary contour and centroid of the probe. Image refinement module: Then, the primary contour map is subjected to dilation erosion and area thresholding to obtain a refined contour map of the probe, including the probe's contour and centroid. Line Fitting Module: Fits the centerline of the two columns of probes, then divides the contour into upper and lower rows based on the orientation of the centroid point relative to the centerline, and fits the centroid points of the two rows of contours with lines to obtain the upper and lower lines. Probe status judgment module: Based on the distance from each centroid point of the probe to its corresponding straight line, the module determines the concavity or convexity of the probe, and based on the spacing between the line segments formed by the projection of the probe contour onto its corresponding straight line, the module determines the missing state of the probe.

9. A probe state detection system based on semantic segmentation and line fitting according to claim 8, characterized in that, The training process for the classification model in the image detection module is as follows: Step 1.1: Acquire the captured probe image, scale the probe image to 224*224, and manually classify it into 4 categories: normal image, blurred image, water spray image, and foreign object image; Step 1.2: Read probe images in batches and perform data enhancement on the probe images. The data enhancement includes: horizontal flipping, random cropping, brightness adjustment, and Gamma adjustment. Step 1.3: Construct the MobileNetV3 network. The basic architecture adopts MobileNetV3-small. The classifier of the last layer of the MobileNetV3 network is enhanced into a two-layer fully connected layer, and the intermediate layer adopts Hardswish activation unit and Dropout technology. Step 1.4: Input the augmented probe images into the constructed MobileNetV3 network in batches, and then optimize the model parameters of the MobileNetV3 network using the Adam optimizer and cross-entropy loss function.

10. A probe state detection system based on semantic segmentation and line fitting according to claim 8, characterized in that, The specific training process for the segmentation model in the image preprocessing module is as follows: Step 2.1: Scale the input probe image to 480*480 and manually classify and label the probe surface; Step 2.2: Then, read the probe images in batches and perform data enhancement on the probe images. The data enhancement includes: Gaussian blur, limiting contrast adaptive histogram, equalizing horizontal flip, random cropping, and arbitrarily adjusting brightness and Gamma. Step 2.3: Construct the U-net segmentation network, using ResNet18 as the backbone network and DiceLoss loss function; Step 2.4: Input the probe images after S2.2 data augmentation into the U-net segmentation network in batches, and then optimize the network model parameters using the Adam optimizer and DiceLoss loss function.

Images