A multi-defect fusion method for shear speckle interference video images

Abstract

The present invention relates to a method for fusing multiple defects in a shear speckle interference video image, which includes: building a shear speckle interference non-destructive testing system, collecting phase images containing defects, establishing a defect recognition model and a defect skeleton extraction model; dynamically collecting internal defect phase images of a measured object by using the shear speckle interference non-destructive testing system, automatically recognizing defects in each frame of the phase map video by using the defect recognition model, so as to obtain defect sub-images in each frame of image; inputting the defect sub-images into the skeleton line extraction model to obtain the skeleton line images of the defect sub-images; comprehensively evaluating the quality of the defect sub-images according to the number and mean square deviation of the defect sub-image skeleton lines, selecting high-quality defect sub-images at each defect position in the dynamic video image, and replacing the images at the corresponding positions in the real image, so as to finally fuse into a high-quality defect evaluation image.

Description

Technical Field

[0001] The present invention relates to the technical field of optoelectronic detection, and particularly relates to a method for fusing multiple defects in a shear speckle interference video image. Background Art

[0002] Shear speckle interferometry (ESSPI) has been widely used in non-destructive testing of bonded structures and composite materials due to its advantages such as high measurement accuracy, stability, and wide measurement range.

[0003] When a shear speckle interferometer is used to measure internal defects of a material, an external action is required to load the object. The main loading methods include thermal loading, acoustic loading, vibration loading, and negative air pressure loading, etc. By uniformly loading the object, the object deforms, and small non-uniform deformations occur at the defect locations, which are then reflected on the surface of the object. Subsequently, the non-uniform deformations are captured by the shear speckle interference system to obtain defect detection.

[0004] In actual measurement, the external loading intensity usually increases gradually. Therefore, due to the differences in the quality of internal defects of the object, when the loading intensity is small, defects with poor quality are detected, while defects with better quality are not detected; as the loading intensity increases, defects with poor quality may have overly dense fringes due to excessive deformation, resulting in inability to be detected, while defects with better quality may be detected at this moment. Thus, it can be seen that at a certain moment during dynamic detection, some defects are detected well, while some defects cannot be detected. However, during the entire detection process, all defects can only be observed through the video image, which is not conducive to the intuitive expression of defects. Therefore, in order to better obtain the detection results of all defects, it is necessary to fuse all defects in the video image into a high-quality multi-defect image.

[0005] In view of the above, the invention of a method for fusing multiple defects in a shear speckle interference video image is particularly important in shear speckle interference defect detection. Summary of the Invention

[0006] The present invention provides a method for fusing multiple defects in a shear speckle interference video image, which can fuse the video images of defect detection into a high-quality multi-defect image, thus making the observation of defects more intuitive.

[0007] The above technical object of the present invention is achieved through the following technical solutions:

[0008] A method for fusing multiple defects in a shear speckle interference video image includes:

[0009] Construct a shear speckle interference non-destructive testing system, collect phase images containing defects, and establish a defect recognition model and a defect skeleton extraction model;

[0010] The phase images of the internal defects of the object to be measured are dynamically collected by using a shear speckle interference nondestructive testing system, and a defect recognition model is used to automatically identify the defects in each frame of the phase map video, so as to obtain the defect sub-images in each frame of the image;

[0011] The defect sub-images are input into the skeleton line extraction model to obtain the skeleton line images of the defect sub-images;

[0012] The quality of the defect sub-images is comprehensively evaluated according to the number and mean square deviation of the skeleton lines of the defect sub-images, and the high-quality defect sub-images at each defect position in the dynamic video image are selected and used to replace the images at the corresponding positions in the real image, so as to finally fuse into a high-quality defect evaluation image.

[0013] Furthermore, establishing a defect recognition model includes:

[0014] Manually intercept the defect sub-images in the phase map, and use the defect sub-images as inputs, and use deep learning networks such as R-CNN, Fast R-CNN, Faster R-CNN-, Detectron, YOLO, SSD, etc. to construct a defect recognition model.

[0015] Furthermore, establishing a defect skeleton extraction model includes:

[0016] The intercepted defect sub-images are processed by binarization, stripe thinning, and branch elimination image processing steps to obtain the skeleton line images of the defect sub-images, and the skeleton line images and the defect sub-images are used as input images, and trained by using a U-Net convolutional neural network to obtain a skeleton line extraction model.

[0017] Furthermore, using the skeleton line image and the defect sub-image as input images, and training by using a U-Net convolutional neural network, and the training process includes forward propagation and backward propagation;

[0018] Among them, in the forward propagation, the ESSPI defect stripe image is used as the input image and input into the U-Net network. After passing through L convolutional layers, it is finally converted into a feature vector with two channels; adding the defect stripe image I is represented as a set of {I i , i = 1, …, Z}, where I i represents the gray value corresponding to the pixel point i, and Z represents the number of pixels; then the feature vector can be expressed as:

[0019]

[0020] Among them, is a convolution operator, is the feature map output by the (l - 1)-th convolutional layer; ReLU is a rectified linear unit;

[0021] The output feature vector I of the defective stripe image I fm can be expressed as Then use the softmax function to determine whether a pixel belongs to a skeleton pixel I i or a non-skeleton pixel; The Softmax function is defined as:

[0022]

[0023] where p c (I i ) is the probability value that pixel I i belongs to class c, c = 1 or 2; Using the softmax function, the output result is a series of probability values;

[0024] Use the cross-entropy loss function to quantify the difference between the probability value obtained by forward propagation and the true value. The cross-entropy is expressed as follows:

[0025]

[0026] where y i is a 1×2 vector representing the true class of pixel I i ; y i = 0 or 1, and the judgment depends on the corresponding skeleton pixel S i ;

[0027] In backpropagation, by minimizing the cross-entropy loss function, the optimal parameters are finally obtained:

[0028] W l , B l ) = argmin(loss)

[0029] When the network parameters are determined, the training process ends.

[0030] Furthermore, use the shear speckle interferometry non-destructive testing system to dynamically collect the internal defect phase image of the object to be measured, including collecting a real object image as the displayed image before detection, and then applying a uniform and continuous load to the object to make its defects appear as surface deformations of the object. At the same time, the system dynamically captures its internal defect phase map, so as to obtain a video image of the defect phase map.

[0031] Furthermore, the process of obtaining the skeleton line image of the defect sub-image includes inputting the defect sub-image into the trained defect skeleton extraction model. Each pixel in the input defect image will be determined by the trained network whether it belongs to a skeleton pixel, so as to finally automatically extract the stripe skeleton of the defect image.

[0032] Further, comprehensively evaluating the quality of the defective sub-image based on the number of the defective sub-image skeleton lines and the mean square error includes obtaining a parameter for judging the quality of the defective sub-image, that is:

[0033] μ = n / σ

[0034] wherein, n is the number of stripe skeletons, and σ is the normalized mean square error of the defective sub-image;

[0035] Select the best defective sub-image at the same position in the defective video image according to the maximum parameter μ value, and replace the sub-image at the corresponding position of the real image with this sub-image; perform the same operation for each defect, and finally fuse them into a complete image containing all defects.

[0036] The beneficial effects of the present invention are as follows:

[0037] 1. Compared with observing the defect position and size by using video images in the existing shear speckle interferometry non-destructive testing, the present invention proposes to fuse video images with inconsistent defect quality into a complete high-quality defect map. The fused defect map is displayed on the surface of the real image, which is beneficial to finding out the specific position of the defect on the real object, and the fused defect map contains all defects in the detection, which is convenient for the detection personnel to draw a reliable conclusion.

[0038] 2. The present invention combines the deep learning automatic recognition technology and the stripe map skeleton line automatic extraction technology, and proposes an evaluation system for defective sub-images, which can be conveniently used to evaluate the quality of defective sub-images. Description of the Drawings

[0039] Figure 1 is the flow chart of multi-defect fusion of shear speckle interferometry video images;

[0040] Figure 2 is the flow chart of automatic extraction of defect skeletons;

[0041] Figure 3 is the structural schematic diagram of the shear speckle interferometry non-destructive testing system built according to the preferred embodiment of the present invention;

[0042] Figure 4 is the multi-defect fusion process and result of shear speckle interferometry video images according to the preferred embodiment of the present invention. Detailed Embodiment

[0043] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0044] In the description of the present invention, it should be noted that unless otherwise clearly specified and defined, the terms "installation", "connection", "coupling" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0045] To make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0046] A method for fusing multiple defects in a shear speckle interference video image, comprising:

[0047] Construct a shear speckle interference non-destructive testing system, collect phase images containing defects, and establish a defect recognition model and a defect skeleton extraction model;

[0048] Dynamically collect the phase images of the internal defects of the object to be measured by using the shear speckle interference non-destructive testing system, and automatically identify the defects in each frame of the phase map video by using the defect recognition model, so as to obtain the defect sub-images in each frame of the image;

[0049] Input the defect sub-images into the skeleton line extraction model to obtain the skeleton line images of the defect sub-images;

[0050] Comprehensively evaluate the quality of the defect sub-images according to the number and mean square deviation of the skeleton lines of the defect sub-images, select the high-quality defect sub-images at each defect position in the dynamic video image, and replace the images at the corresponding positions in the real image, so as to finally fuse into a high-quality defect evaluation image.

[0051] Optionally, establishing the defect recognition model includes:

[0052] Manually intercept the defective sub-images in the phase diagram, and use the defective sub-images as inputs. Utilize deep learning networks such as R-CNN, Fast R-CNN, Faster R-CNN, Detectron, YOLO, SSD, etc. to construct a defect recognition model.

[0053] Optionally, establishing a defect skeleton extraction model includes:

[0054] Perform binarization, stripe thinning, and branch elimination image processing steps on the intercepted defective sub-images to obtain the skeleton line image of the defective sub-images. Use the skeleton line image and the defective sub-images as input images, and utilize the U-Net convolutional neural network for training to obtain a skeleton line extraction model.

[0055] Optionally, use the skeleton line image and the defective sub-images as input images, and utilize the U-Net convolutional neural network for training. The training process includes forward propagation and backpropagation;

[0056] Among them, in forward propagation, the ESSPI defect stripe image is used as the input image and input into the U-Net network. After passing through L convolutional layers, it is finally converted into a feature vector with two channels; adding the defect stripe image I is represented as a set {I i , i = 1, …, Z}, where I i represents the gray value corresponding to pixel point i, and Z represents the number of pixels; then the feature vector can be expressed as:

[0057]

[0058] Among them, is a convolution operator, is the feature map output by the (l - 1)-th convolutional layer; ReLU is a rectified linear unit;

[0059] The output feature vector I fm of the defect stripe image I can be expressed as Then use the softmax function to determine whether a pixel belongs to a skeleton pixel I i or a non-skeleton pixel; The Softmax function is defined as:

[0060]

[0061] Among them, p c (I i ) is the probability value that pixel I i belongs to class c, c = 1 or 2; Using the softmax function, the output result is a series of probability values;

[0062] The cross - entropy loss function is used to quantify the difference between the probability values obtained from forward propagation and the true values. The cross - entropy is expressed as follows:

[0063]

[0064] where, y i is a 1×2 vector representing the true class of pixel I i ; y i = 0 or 1, and the judgment depends on the corresponding skeleton pixel S i ;

[0065] In backpropagation, by minimizing the cross - entropy loss function, the optimal parameters are finally obtained:

[0066] W l , B l ) = argmin(loss)

[0067] When the network parameters are determined, the training process ends.

[0068] Optionally, a shear - speckle interferometry non - destructive testing system is used to dynamically collect the phase images of internal defects of the object to be measured, including collecting a real object image as the displayed image before detection, and then uniformly and continuously loading the object so that its defects are manifested as surface deformations of the object. At the same time, the system dynamically captures the internal defect phase map, thereby obtaining a video image of the defect phase map.

[0069] Optionally, the process of obtaining the skeleton line image of the defect sub - image includes inputting the defect sub - image into a trained defect skeleton extraction model. Each pixel in the input defect image will be determined by the trained network whether it belongs to the skeleton pixel, and thus the stripe skeleton of the defect image is finally automatically extracted.

[0070] Optionally, comprehensively evaluating the quality of the defect sub - image according to the number of skeleton lines and the mean square error of the defect sub - image includes obtaining a parameter for judging the quality of the defect sub - image, that is:

[0071] μ = n / σ

[0072] where, n is the number of stripe skeletons, and σ is the normalized mean square error of the defect sub - image;

[0073] Select the best defect sub - image at the same position in the defect video image according to the maximum value of the parameter μ, and replace the sub - image at the corresponding position of the real image with this sub - image; perform the same operation for each defect, and finally fuse them into a complete image containing all defects.

[0074] Specifically, in this embodiment, a shear - speckle interferometry non - destructive testing system is built with thermal loading (similar to acoustic loading, vibration loading, and negative air pressure loading) as an example, asFigure 3 As shown, the system includes a laser, a Michelson interferometer optical path, a speckle camera, an excitation controller, and thermal excitation. According to the multi-defect fusion method provided by the present invention, Figure 1 FIG. is a flowchart of a preferred multi-defect fusion method for shear speckle interferometry video images, and the specific operations are as follows:

[0075] Before detection, a real object image is collected as the image to be displayed, and then the object is uniformly and continuously loaded so that its defects are manifested as surface deformations of the object. At the same time, the system dynamically captures the internal defect phase map, thereby obtaining a video image of the defect phase map.

[0076] Using the defect recognition model established by deep learning, defects are automatically recognized in the defect video image and the image positions of each defect are marked, thereby obtaining the image positions of each defect in each frame of the defect video image. The defect position results are as Figure 4 shown by the rectangular boxes in (b).

[0077] The trained U-Net network module is used to automatically extract the defect fringe skeleton lines in each frame, thereby automatically obtaining all the fringe skeleton lines in this defect image.

[0078] The quality of the defect sub-images is comprehensively evaluated using the obtained skeleton lines and the mean square error of the sub-images. The quality of the defect sub-images is judged by the following formula, that is:

[0079] μ = n / σ

[0080] where n is the number of fringe skeleton lines and σ is the normalized mean square error of the defect sub-image.

[0081] Thus, the best defect sub-image at the same position in the defect video image is selected according to the maximum parameter μ value. The selected result is the defect shown by the dashed box in Figure 4 (b). This sub-image is used to replace the sub-image at the corresponding position of the real image in Figure 4 (a). The same operation is performed for each defect. Finally, the defect video image is fused into a complete image containing all defects. The fused defect image is as shown in Figure 4 (c). The proposed method fuses video images with inconsistent defect qualities into a high-quality defect image. And displaying the fused defect map on the surface of the real image is beneficial to finding out the specific positions of the defects in the real object, and the fused defect map contains all the defects in the detection, which is convenient for the detection personnel to draw reliable conclusions.

[0082] It should be understood that the above specific embodiments of the present invention are only for illustrative explanation or interpretation of the principles of the present invention, and do not constitute a limitation on the present invention. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present invention shall be included within the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all variations and modifications that fall within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A multi-defect fusion method for shear speckle interference video images, characterized in that, Including: Construct a shear speckle interferometry non-destructive testing system, collect phase images containing defects, establish a defect recognition model and a defect skeleton extraction model; Use the shear speckle interferometry non-destructive testing system to dynamically collect the phase images of internal defects of the object to be measured. Use the defect recognition model to automatically identify the defects in each frame of the phase map video, so as to obtain the defect sub-images in each frame of the image; Input the defect sub-images into the skeleton line extraction model to obtain the skeleton line images of the defect sub-images; Comprehensively evaluate the quality of the defect sub-images according to the number and mean square deviation of the defect sub-image skeleton lines. Select the high-quality defect sub-images at each defect position in the dynamic video image, and replace the images at the corresponding positions in the real image, so as to finally fuse into a high-quality defect evaluation image; The establishment of the defect skeleton extraction model includes: Perform binarization, stripe thinning, and branch elimination image processing steps on the intercepted defect sub-images to obtain the skeleton line images of the defect sub-images. Use the U-Net convolutional neural network to train the skeleton line images and defect sub-images as input images to obtain the skeleton line extraction model; The use of the shear speckle interferometry non-destructive testing system to dynamically collect the phase images of internal defects of the object to be measured includes collecting a real object image as the displayed image before detection, and then uniformly and continuously loading the object to make its defects appear as surface deformations of the object. At the same time, the system dynamically captures the internal defect phase map, so as to obtain the video image of the defect phase map; The comprehensive evaluation of the quality of the defect sub-images according to the number and mean square deviation of the defect sub-image skeleton lines includes obtaining a parameter for judging the quality of the defect sub-images, that is: μ = N / σ Where n is the number of stripe skeletons, and σ is the normalized mean square deviation of the defect sub-image; Select the best defect sub-image at the same position in the defect video image according to the maximum parameter μ value, and replace the sub-image at the corresponding position in the real image with this sub-image; perform the same operation for each defect, and finally fuse into a complete image containing all defects.

2. The multi-defect fusion method for shear speckle interference video images according to claim 1, characterized in that: The establishment of the defect recognition model includes: Manually intercept the defect sub-images in the phase map, and use the defect sub-images as input. Use R-CNN, Fast R-CNN, Faster R-CNN-, Detectron, YOLO, SSD deep learning networks to construct a defect recognition model.

3. The multi-defect fusion method for shear speckle interference video images according to claim 1, characterized in that: Use the skeleton line image and the defect sub-image as input images, and use the U-Net convolutional neural network for training. The training process includes forward propagation and backward propagation; Among them, in the forward propagation, the ESSPI defect fringe image is input into the U-Net network as the input image. After passing through L convolutional layers, it is finally converted into a feature vector with two channels; adding the defect fringe image I is represented as a set of {I i , i = 1, …, Z}, where I i represents the gray value corresponding to pixel point i, and Z represents the number of pixels; then the feature vector can be expressed as: Among them, is a convolution operator, is the feature map output by the (l - 1)-th convolutional layer; ReLU is the rectified linear unit; Output feature vector I of the defect stripe image I fm can be expressed as Then, the softmax function is used to determine whether a pixel belongs to a skeleton pixel I i or a non-skeleton pixel; The Softmax function is defined as: where p c (I i ) is the probability value of pixel I i belonging to class c, where c = 1 or 2; using the softmax function, the output result is a series of probability values; Use the cross-entropy loss function to quantify the difference between the probability value obtained by forward propagation and the true value. The cross-entropy is expressed as follows: Among them, y i is a 1×2 vector, representing the true value category of pixel I i ; Y i = 0 or 1, and the judgment depends on the corresponding skeleton pixel S i ; In the backward propagation, by minimizing the cross-entropy loss function, the optimal parameters are finally obtained: W l ,B l ) = argmin (loss) When the network parameters are determined, the training process ends.

4. The multi-defect fusion method for shear speckle interference video images according to claim 1, characterized in that: The process of obtaining the skeleton line image of the defect sub-image includes inputting the defect sub-image into the trained defect skeleton extraction model. Each pixel in the input defect image will be determined by the trained network whether it belongs to the skeleton pixel, so as to finally automatically extract the stripe skeleton of the defect image.

Images