A complex surface damage area identification and segmentation method

Abstract

The application belongs to the technical field of surface damage positioning method, and particularly relates to a complex surface damage area recognition and segmentation method. The method comprises the following steps: acquiring images containing complex surface damages of different degrees, and performing text area positioning and image repair cooperation based on PP-OCRv4 and Lama model cooperative training; removing small isolated noise points and connected small parts by using morphological opening operation, and using edge detection technology to capture damage edges and retain valuable damage information; and adopting a fixed threshold method to set a specific gray value range to perform binaryzation processing on the images, and determine damage areas and normal areas. The application provides an innovative and accurate deep learning-based method for evaluating complex surface damages, realizes full-process automatic evaluation from image acquisition to damage proportion calculation and result recording by comprehensively using advanced deep learning models, and aims to provide a scientific, efficient and accurate evaluation means for maintenance and guarantee of complex surfaces.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of surface damage positioning methods, and particularly relates to a complex surface damage area identification and segmentation method. BACKGROUND

[0002] In the modern industrial field, complex surface damage evaluation is a crucial task. Complex surface damage has various forms, including but not limited to wear, corrosion, cracks, and deformation. These damages not only affect the appearance of products, but also reduce the mechanical properties of products during long-term use, and may even cause serious safety accidents. Traditional surface damage detection methods mainly include visual detection, magnetic powder detection, penetration detection, and ultrasonic detection. Visual detection relies on the experience and skills of detection personnel, is highly subjective, and is difficult to detect small or hidden damages. Magnetic powder detection and penetration detection have high operation requirements for detection personnel, and can only detect surface open defects. Ultrasonic detection can detect internal defects, but has limited detection effect on workpieces with complex shapes and surface conditions, and the detection process is complex, requiring professional equipment and technical personnel. SUMMARY

[0003] The purpose of the application is to provide a complex surface damage area identification and segmentation method, which provides an effective way for automatic and accurate segmentation and evaluation of complex surface damage.

[0004] To achieve the above purpose, the application adopts the following technical solutions.

[0005] A complex surface damage area identification and segmentation method includes the following steps:

[0006] Step 1. Data collection, specifically referring to obtaining images containing different degrees of complex surface damage, wherein complex surface damage refers to at least one damage of wear, corrosion, cracks, and deformation on the surface;

[0007] Step 2. Text area positioning and image repair coordination based on PP-OCRv4 and Lama model collaborative training; specifically including B1-B3:

[0008] B1, input the preprocessed image into the PP-OCRv4 model to detect the text area position information in the image, and transfer to B2;

[0009] B2, according to the text position information provided by PP-OCRv4, determine the text area that needs to be repaired, and use the Lama model to repair the detected text area, and transfer to B3;

[0010] B3, check the repaired area using the PP-OCRv4 model to see if there are still residual characters, if there are residual characters, repeat steps B1 and B2 until the character interference in the image is eliminated;

[0011] Step 3. Image information processing, specifically referring to using morphological opening operation to remove small isolated noise points and connected small parts, using edge detection technology to capture damage edges and retain valuable damage information;

[0012] Step 4. Damage area segmentation

[0013] Using the fixed threshold method, set a specific gray value range to binarize the image, and determine the damage area and normal area.

[0014] Further improvement or preferred embodiment of the foregoing complex surface damage area recognition and segmentation method, said step 1 further comprises: cropping or padding the picture size to a fixed size, and dividing the entire data set into a training set and a test set according to a ratio of 8:2, wherein 10% of the training set data is used as a validation set.

[0015] Further improvement or preferred embodiment of the foregoing complex surface damage area recognition and segmentation method, said PP-OCRv4 model is a two-stage OCR module, including a text detection module and a text recognition module, wherein the text detection module is based on DB algorithm, and the text recognition module is based on CRNN network, and a text direction classifier is added between the text detection module and the text recognition module to cope with different direction text recognition.

[0016] Further improvement or preferred embodiment of the foregoing complex surface damage area recognition and segmentation method, in step 2, the Lama model directly calls the Lama model standard data set and uses places2, places challenge model for training, while adopting the way of adding custom data set, obtaining real data by actual shooting, and carrying out secondary training on the model;

[0017] At the same time, the perceptual loss function is introduced, and the perceptual loss function respectively inputs the real image and the generated image into the pre-trained VGG19 network to evaluate the model training result; the perceptual loss function can be expressed as:

[0018]

[0019] Wherein, x is the input image, y is the target image, Fi(x) and Fi(y) respectively represent their feature representations in the i-th layer of the VGG19 network, and N is the number of feature layers.

[0020] Further improvement or preferred embodiment of the foregoing complex surface damage area recognition and segmentation method, the step 2 further comprises: using the accuracy P and recall rate R and average precision AP as evaluation index to evaluate the performance of PP OCRv4 recognition model for single class target, and adjusting and optimizing the parameters of PP OCRv4 recognition model according to the evaluation result, wherein

[0021]

[0022] TP represents the number of correct target detection, FP represents the number of incorrect target detection, and FN represents the number of undetected targets; the area of the figure surrounded by the accuracy-recall rate curve is the average precision AP of the class target detection;

[0023] For multi-class target, the average precision mean mAP is used for evaluation, wherein

[0024] N represents the number of target classes in the data set.

[0025] Further improvement or preferred embodiment of the foregoing complex surface damage area recognition and segmentation method, the step 2 further comprises: continuously monitoring the loss function value and the change of evaluation index of the model on the training set and the validation set, drawing the value and index curve, judging whether the model exists overfitting or underfitting, and adjusting the training parameters or using the regularization method for optimization processing according to the result;

[0026] And using the mean square error MSE and the frame per second FPS to evaluate the performance of the Lama model, wherein

[0027] n represents the total number of pixels in the image, I original (x i ,y i ) represents the pixel value of the original image at coordinate (x i ,y i ), I repaired (x i ,y i ) represents the pixel value of the repaired image at coordinate (x i ,y i ); N' represents the number of processed image frames, and T represents the total running time.

[0028] Further improvement or preferred embodiment of the foregoing complex surface damage area identification and segmentation method, the step 4 further comprises, in order to make the calculated damage area closer to the result of machine scanning, by comparing the error analysis of damage area of experimental results and calculation data results, constantly adjust the gray value threshold, determine the preferred threshold range which makes the calculated damage area most similar to the actual damage area.

[0029] Further improvement or preferred embodiment of the foregoing complex surface damage area identification and segmentation method, in the step 4, the gray value threshold of fixed threshold method is set to 70-80.

[0030] Further improvement or preferred embodiment of the foregoing complex surface damage area identification and segmentation method, in the step 4, the gray value threshold of fixed threshold method is set to 75. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the system block diagram of PP-OCRv4 model;

[0032] Figure 2 is the loss function change curve in the training process;

[0033] Figure 3 is the recognition result of test set picture;

[0034] Figure 4 is the loss function change curve of LAMA model in the training process;

[0035] Figure 5 is the processing result of test set picture;

[0036] Figure 6 is the binary output comparison chart of different methods. DETAILED DESCRIPTION

[0037] The application will be described in detail below in combination with specific embodiments.

[0038] Complex surface detection plays a key role in modern industrial detection, and its damage condition is directly related to the performance and safety of industrial metals. The present application provides an innovative and accurate deep learning-based method for evaluating the damage of complex surfaces, which realizes the full-process automatic evaluation from image acquisition to damage proportion calculation and result recording by comprehensively using advanced deep learning models, aiming to provide a scientific, efficient and accurate evaluation means for the maintenance and protection of complex surfaces.

[0039] The main step flow of the complex surface damage area identification and segmentation method of the present application specifically comprises:

[0040] Step 1. Data collection:

[0041] Obtaining images containing different degrees of complex surface damage, that is, a surface including at least one of wear, corrosion, cracks, and deformation;

[0042] In actual implementation, images containing complex surface damage should be obtained as much as possible to ensure that the actual industrial scene images cover various damage types (such as wear, corrosion, cracks, etc.), different degrees of damage, and various character marks, so that more accurate and more consistent with the actual situation of the industrial scene recognition processing results can be obtained.

[0043] In a specific implementation, the picture size should also be cropped or padded to a fixed size, and the entire data set should be divided into a training set and a test set in a ratio of 8:2, wherein 10% of the training set data is used as a validation set.

[0044] Step 2. Text region positioning and image repair cooperation based on PP-OCRv4 and Lama model cooperative training; specifically including B1-B3:

[0045] B1, inputting the preprocessed image into the PP-OCRv4 model to detect the text region position information in the image, and transferring to B2;

[0046] B2, determining the text region that needs to be repaired according to the text position information provided by the PP-OCRv4, and repairing the detected text region by using the Lama model, and transferring to B3;

[0047] B3, using the PP-OCRv4 model to check the repaired region to see if there is residual text, and if there is residual text, repeating steps B1 and B2 until the desired text removal effect is achieved to ensure that the text interference in the image is completely eliminated;

[0048] The PP-OCRv4 model in the present application is a two-stage OCR model, including a text detection module and a text recognition module, wherein the text detection module is established based on a DB algorithm, the text recognition module is established based on a CRNN network, and a text direction classifier is added between the text detection module and the text recognition module to cope with text recognition in different directions. The system block diagram of the PP-OCRv4 model is shown in Figure 1 Based on the foregoing model, higher accuracy and robustness can be achieved when processing text and characters.

[0049] As shown in Table 1, the character recognition accuracy of the PP_OCRv4 recognition model of the present application on the test number results of the data set in the present application reaches 75.45%, which meets the application requirements.

[0050] Table 1 Comparison of results of the PP-OCRv4 model of the present application with existing models

[0051] Model Model size Hmean (%) Cpu+mk1 dnn speed (ms) PP OCRv3 3.4 76.22 69 PP OCRv4 4.8 79.87 67

[0052] Based on the processing manner of the present application, the non-text area can be effectively avoided from being repaired, the integrity and accuracy of the restored image are ensured, and the information mismatch and error accumulation problems in the working of the traditional recognition model are avoided.

[0053] To ensure the effectiveness of the Lama model, the official dataset of the Lama model is directly called and the places2 and placeschallenge models are used for training, and in the present application, a self-defined dataset is added, real data is obtained by actual shooting, and the model is trained again.

[0054] To ensure that the Lama model can effectively learn and generate accurate and natural repair results, and improve the structural similarity between the real image and the repaired image, a perception loss function is introduced, the real image and the generated image are input into the pre-trained VGG19 network, the Euclidean distance of the activation values of each intermediate layer is calculated, and the perception loss function can be expressed as:

[0055] Wherein, x is an input image, y is a target image, Fi(x) and Fi(y) represent their feature representations in the i-th layer of the VGG19 network, and N is the number of feature layers.

[0056] In particular, in the present application, the detection accuracy is used as an evaluation index to evaluate the performance of the PP_OCRv4 recognition model for single-class targets, and the parameters of the PP_OCRv4 recognition model are adjusted and optimized according to the evaluation results, wherein

[0057]

[0058] TP represents the number of correct target detection, FP represents the number of incorrect target detection, FN represents the number of undetected targets, and the area of the graph surrounded by the accuracy and recall curve is the average detection accuracy AP of the target class.

[0059] For multi-class targets, the average precision mean mAP is used for evaluation, wherein

[0060] N represents the number of target classes in the dataset.

[0061] In the model training process, the loss function values and evaluation index changes of the model on the training set and the validation set are monitored, the corresponding numerical value and index curve are drawn, whether the model has overfitting or underfitting is judged, and the training parameters are adjusted or the regularization method is optimized according to the results.

[0062] In this application, mean square error (MSE) and frames per second (FPS) are used to evaluate the performance of the Lama model, where:

[0063]

[0064] n represents the total number of pixels in the image, I original (x i ,y i ) represents the pixel value of the original image at coordinates (x i ,y i ), I repaired (x i ,y i ) represents the pixel value of the repaired image at coordinates (x i ,y i ); N' represents the number of processed image frames, and T represents the total running time.

[0065] The selected software and hardware facilities in this experiment are shown in Table 2:

[0066] Table 2 Experimental software and hardware facilities

[0067]

[0068] All input image sizes are cropped or padded to 1002x852, and the entire dataset is divided into training and test sets in a ratio of 8:2, with 10% of the training set data as a validation set. After 100 rounds of training, the loss function change curve during training is shown in Figure 2 , where the horizontal axis represents the number of training rounds, and the vertical axis represents the loss value of training.

[0069] As can be seen from the figure, the minimum training loss value is at the 81st round, so the 81st round training result is taken as the final training weight. After completing the training, the test set detection results are shown in Table 3.

[0070] Table 3 PP-OCRv4 detection results based on iterative optimization

[0071]

[0072] As can be seen from Table 3, the average precision (AP) of PP-OCRv4 for the three types of targets is 95.83%, 96.31%, and 92.28%, respectively, and the average mean average precision (mAP) is 94.81%. The detection precision of special symbols is the lowest, mainly because the number of special symbol images in the dataset is relatively small.

[0073] Figure 3For the test set picture detection results, the left picture is the original picture, and the right picture is the white area of the recognized text area. As can be seen, the model can well complete the detection of the text target.

[0074] After 100 rounds of training, the loss function curve of the Lama model during training is as shown in Figure 4 The horizontal coordinate represents the training round, and the vertical coordinate represents the training loss value. As can be seen from the figure, the loss value of the model decreases rapidly to 0.06 in a short time, which meets the requirement of rapid convergence. In addition, the training loss gradually decreases and finally reaches a stable state, and good results can also be achieved on the validation set and the test set. The stability of the model meets the demand. In order to improve the smoothness of operation, multi-thread programming is adopted in the present application.

[0075] Table 4 is the experimental results on the self-defined data set. The improved algorithm is better than the original algorithm in various evaluation indexes. The MSE is reduced by 0.32%, and the FPS is increased by 24.

[0076] Table 4 is the Lama detection result based on iterative optimization

[0077] Method MSE (%) FPS Original Lama 2.44 30 Lama of the present application 2.12 54

[0078] Figure 5 For the test set picture detection results, the left picture is the original picture, and the right picture is the final image output after removing the text area and repairing the image. As can be seen from the above, the model can well complete the redrawing work of the specified block in the complex surface image.

[0079] Step 3. Image information processing

[0080] The morphological opening operation is used to remove small isolated noise points and connected small parts. The edge detection technology can capture the damage edge and retain valuable damage information.

[0081] By the above-mentioned way, irrelevant scratches, isolated noise points and other interference information in the image are removed, while defects that have an impact on damage calculation are retained, the image quality is improved, and the accuracy of the damage area is ensured.

[0082] As shown in Figure 6 , it is a comparison chart of binary output of different methods.

[0083] Step 4. Damage area cutting

[0084] The fixed threshold method is adopted to set a specific gray value range to perform binaryzation processing on the image, so as to determine the damage area and the normal area.

[0085] In order to make the calculated damage area closer to the result of machine scanning, through multiple experiments and data analysis, by continuously adjusting the threshold and comparing with the damage area obtained by machine scanning, it is determined that when the preferred threshold setting is 70-80, the calculated damage area is most similar to the conclusion given by the machine;

[0086] Through a large number of experimental verification, the fixed threshold method based on the foregoing threshold can accurately process complex surface damage features, effectively segment the damage area, and avoid problems such as over-segmentation and misjudgment in complex surface images.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not a limitation on the protection scope of the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the present application.

Claims

1. A complex surface damage area identification segmentation method, characterized in that, Comprising the following steps: Step 1. Data collection, specifically referring to obtaining images containing different degrees of complex surface damage, complex surface damage refers to a surface including at least one of wear, corrosion, crack, deformation; Step 2. Text region positioning and image repair cooperation based on PP-OCRv4 and Lama model cooperative training; Specifically including B1~B3: B1, input the preprocessed image into the PP-OCRv4 model to detect the text region position information in the image, and transfer to B2; The PP-OCRv4 model is a two-stage OCR module, including a text detection module and a text recognition module, wherein the text detection module is based on DB algorithm, and the text recognition module is based on CRNN network, and a text direction classifier is added between the text detection module and the text recognition module to cope with text recognition in different directions; B2, according to the text position information provided by PP-OCRv4, determine the text area that needs to be repaired, and use Lama model to repair the detected text area, and transfer to B3; The Lama model directly calls a Lama model standard data set and is trained using a places2, places challenge model, while adding a custom data set, obtaining real data through actual shooting, and performing secondary training on the model; meanwhile, a perception loss function is introduced, the perception loss function respectively inputs a real image and a generated image into a pre-trained VGG19 network to evaluate the model training result; the perception loss function can be expressed as: ; Wherein, x is the input image, y is the target image, Fi(x) and Fi(y) represent their feature representations in the i-th layer of VGG19 network respectively, and N is the number of feature layers; B3, use the PP-OCRv4 model to check the repaired area, and check whether there is residual text, if there is residual text, repeat steps B1 and B2 until the text interference in the image is eliminated; Step 3. Image information processing, specifically referring to using morphological opening operation to remove small isolated noise points and connected small parts, and using edge detection technology to capture damage edges and retain valuable damage information; Step 4. Damage area cutting The fixed threshold method is adopted, a certain gray value range is set to perform binaryzation processing on the image, and the damage area and the normal area are determined.

2. The complex surface damage region identification and segmentation method according to claim 1, characterized in that, The step 1 further comprises: cropping or padding the picture size to a fixed size, and dividing the entire data set into a training set and a test set according to a ratio of 8:2, wherein 10% of the training set data is used as a validation set.

3. The complex surface damage region identification and segmentation method according to claim 1, wherein, The step 2 further comprises: taking the accuracy and the recall rate and the average precision as evaluation indexes to evaluate the performance of the PP_OCRv4 recognition model on the single-class target, and adjusting and optimizing the parameters of the PP_OCRv4 recognition model according to the evaluation result. ， ； TP represents the number of correct target detection, FP represents the number of incorrect target detection, and FN represents the number of undetected targets; The area of the figure surrounded by the accuracy and recall rate curve is the average precision AP of the target detection; For multi-class targets, the average precision mean is used The evaluation is performed, wherein ; represents the number of classes of targets in the data set.

4. The complex surface damage region identification and segmentation method according to claim 3, characterized in that, The step 2 further comprises continuously monitoring the loss function value and the change of the evaluation index of the model on the training set and the validation set, drawing the value and index curve, judging whether the model has overfitting or underfitting, and adjusting the training parameters or using the regularization method for optimization processing according to the result; And use mean square error and the number of frames processed per second To evaluate the performance of the Lama model, among which ; ; n represents the total number of pixels in the image. Indicates the original image in coordinates Pixel value at that location, Indicates the coordinates of the repaired image. Pixel value at; The number of image frames processed is represented by T, and the total running time is represented by T.

5. The complex surface damage region identification and segmentation method according to claim 1, wherein, The step 4 further comprises, in order to make the calculated damage area closer to the result of machine scanning, through error analysis on the damage area of the experimental result and the calculated data result, the gray value threshold is adjusted constantly, and the optimal threshold range that makes the calculated damage area most similar to the actual damage area is determined.

6. The complex surface damage region identification and segmentation method according to claim 5, wherein, In the step 4, the gray value threshold of the fixed threshold method is set to 70~80.

7. The complex surface damage region identification and segmentation method according to claim 5, wherein, In the step 4, the gray value threshold of the fixed threshold method is set to 75.

Images