Automatic segmentation method of salient regions in industrial X-ray images

Abstract

The present invention discloses a method for automatic segmentation of salient areas of industrial X-ray images. In order to solve the problems of being unable to obtain large-scale sample training, low image contrast, unclear foreground and background distinction in industrial scenes, the present invention realizes the segmentation and extraction of salient areas of irregular images with subject-background confusion under the premise of small samples and low contrast by image enhancement that preserves image details and improving the interactive manual segmentation algorithm into an unsupervised learning self-annotation segmentation method. It has good robust effects for scenes such as automated defect detection on industrial assembly lines and detection of salient target areas during driving of unmanned vehicles. Moreover, the results of the present application are not only effective for images in the industrial field, but also have good generalization effects for the extraction of interest areas of images under low illumination.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a method for automatically segmenting a significant area of ​​an industrial X-ray image. Background Art

[0002] Industrial X-ray defect detection has problems such as low contrast and similar colors between the subject and the background. Currently, it is difficult to accurately extract the subject area based on threshold segmentation or region growing algorithms. Although deep learning methods such as UNet have good results, they require a large amount of sample data annotation.

[0003] Images taken with X-rays or low illumination have low contrast, and the foreground and background are not clearly distinguished. For such images, global image processing methods (such as overall image brightening or histogram equalization) will cause the loss of image details, which is not conducive to the subsequent segmentation and extraction of salient areas. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method for automatically segmenting salient areas of industrial X-ray images.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] The method for automatic segmentation of salient regions of industrial X-ray images comprises the following steps:

[0007] (S1) Input image and preprocessing;

[0008] (S2) Image contrast and brightness enhancement with detail preservation: This application adopts a self-designed method for local image enhancement processing to locally enhance the image, perform hierarchical processing according to the image grayscale, and map the original grayscale value of the image;

[0009] (S3) Adaptive generation of candidate regions: Before extracting salient regions, the candidate regions are initially screened, including the following steps:

[0010] (S31) Adaptive binary segmentation, (S32) Seed candidate region extraction, (S33) Image region segmentation calculation, (S34) Obtain candidate segmentation region, (S35) Foreground / background region labeling, (S36) Output labeling Mask image result;

[0011] (S4) using an automated salient region extraction method for segmentation estimation and iterative learning of model parameters;

[0012] (S5) Result output.

[0013] Furthermore, the image described in step (S1) is an RGB image and a grayscale image, and the preprocessing is to remove noise and perform Gaussian blur on the input image.

[0014] Further, in the step (S2):

[0015] The custom grayscale mapping formula is:

[0016] G(x′, y′)=B(G(x, y)), G(x, y)∈[0, 255] (1)

[0017] Where G(x, y) represents the grayscale value of the grayscale image at the coordinate (x, y);

[0018] Then the corresponding mapping formula B is:

[0019]

[0020] Where k represents the kth interlude calculation, and the control point is from P j Click to P i point, P is the gray value of the pixel, there are i-j+1 in total;

[0021] P and N in formula (2) are calculated as follows:

[0022]

[0023] j represents the starting control point, i represents the iteration parameter, which is equivalent to the control point being the starting point from P j To P j+k , the parameters i, k in the basis function N(G(x, y)) of formula (4) are synchronized with formula (3), and C represents the selection of r points from k+1.

[0024] Furthermore, in the seed candidate region extraction described in step (S32):

[0025] Assume that the binary image is A, the foreground point set is P, and the background point is Q, then the corresponding mathematical expression is:

[0026] D(p)=min(d(p, q)+f(q)), stp∈P, q∈Q, P∪Q=A (5)

[0027] Where d(p, q) is the distance between the foreground point and the background point, p is the foreground pixel of the image, q is the background pixel of the image, and f is the constraint function;

[0028] After obtaining the grid image D of the image distance transform, the image D is normalized. The normalized value interval is [0, 255]. The method is to find the maximum and minimum values ​​in the image D and scale it to 255 times. The obtained normalized image N is shown in formula (6):

[0029]

[0030] Where min(D) is the minimum grayscale value of image D(x, y), and max(D) is the maximum grayscale value of image D(x, y);

[0031] The image N is binarized again, and then morphological operations are performed to remove burrs, and the seed candidate region is extracted.

[0032] Furthermore, in the image region segmentation calculation described in step (S33): the candidate region extraction result of step (S32) is used as the seed region input, and the numbering value starts from 1; for the foreground area that does not enter the candidate area in the binary image, it is marked as an unknown area and numbered 0, thereby obtaining the region segmentation result, that is:

[0033] W(S rgb )=W(R unknown , R seed1 , R seed2 , …, R seedn ) (7)

[0034] S rgb is the input RGB source image, where R unknown The unknown area is marked and filled with 0. seed i is the i-th seed region, filled with i values, i∈{1, 2,…, n}.

[0035] Further, in the foreground / background region labeling described in step (S35): in step (S33), the image region segmentation calculation output result is used as a candidate segmentation region, and the final target region is obtained by further merging and dilating;

[0036] Here for W(S rgb ) area annotation, the annotation image M output satisfies formula (8):

[0037]

[0038] Where S rgb (x, y) is the (x, y) coordinate position in the source image, R bg is the background area, R oi is the region of interest, which comes from the image region segmentation calculation output result in step (S33), R pbg It is the suspected background area.

[0039] Furthermore, in the foreground / background area labeling described in step (S35), the automatic labeling process is as follows:

[0040] 1) Initialize the labeled image Mask, which is the same size as the original image;

[0041] 2) Set the initial value of the annotated image to suspected background mark 2;

[0042] 3) For the location of the region of interest obtained by segmentation, set it to 1 in the labeled image Mask;

[0043] 4) For the interference items that may be generated in the region of interest, the background flag is set to 0, that is, the region cannot be classified as the foreground.

[0044] Furthermore, the method for automatically extracting salient regions in step (S4) comprises the following steps:

[0045] (S41) using the labeled image Mask in (S36) as a definition, defining the background, foreground and suspected background areas in the source image, and substituting them into the Gaussian mixture model (GMM) of the grab cut;

[0046] (S42) using the GMM model to model the background and foreground, and marking undefined pixels as possible foreground or background;

[0047] (S43) According to the Mask mark in formula (7), the data has been divided into n categories. Here, each category is set to be normally distributed. Next, the GMM model is used to fit and the probability of the data belonging to a certain Gaussian model is iteratively calculated to minimize its estimated energy, thereby completing the aggregation classification of the data;

[0048] (S44) Estimation energy iterative optimization method: Through the MaxFlow-MinCut model of the Grab cut graph, the image is mapped into a weighted undirected graph G=<V,E> , each node N∈V in the graph corresponds to each pixel in the image, each edge∈E connects a pair of adjacent pixels, and the edge weight represents the non-negative similarity between adjacent pixels in terms of grayscale, color or texture; a segmentation S of an image is a cut of the graph, and each segmented region R∈S corresponds to a subgraph in the graph; the optimal principle of segmentation is to maximize the similarity within the divided subgraphs, while minimizing the similarity between subgraphs.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] This application analyzes low-contrast images, addresses the problem of not being able to obtain large-scale sample training in industrial scenarios, and achieves significant area segmentation and extraction of irregular, subject-background confused images under the premise of small samples and low contrast by preserving image details through image enhancement and improving the interactive manual segmentation algorithm to an unsupervised learning self-annotation segmentation method. It has good robust effects for scenes such as automated defect detection on industrial assembly lines and significant target area detection during unmanned vehicle driving. Moreover, the results of this application are not only effective for images in the industrial field, but also have good generalization effects for the extraction of interest areas in images under low illumination. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The image is obtained by using the automatic segmentation method of the significant area of ​​industrial X-ray images of the present application, wherein Figure 1 a is the X-ray image, Figure 1 b is the original chip image (partial), Figure 1 c is the image enhancement map, Figure 1 d is the image after adaptive binarization, Figure 1 e is the image obtained by extracting the seed candidate region. Figure 1 f is the image after the region of interest is automatically annotated. Figure 1 g is the automatic extraction map of the region of interest;

[0052] Figure 2 This is a flow chart of the automatic segmentation method for salient areas of industrial X-ray images;

[0053] Figure 3 Input image and preprocessing flow chart;

[0054] Figure 4 Adaptively generate a flow chart for the candidate regions;

[0055] Figure 5 More exemplary results obtained by using the method of the present invention are shown in FIG. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0057] The method for automatic segmentation of salient regions of industrial X-ray images comprises the following steps:

[0058] (S1) Input image and preprocessing;

[0059] (S2) Image contrast and brightness enhancement with detail preservation: This application adopts a self-designed method for local image enhancement processing to locally enhance the image, perform hierarchical processing according to the image grayscale, and map the original grayscale value of the image;

[0060] (S3) Adaptive generation of candidate regions: Before extracting salient regions, complete the initial screening of candidate regions, such as Figure 4 As shown, the following steps are included:

[0061] (S31) Adaptive binary segmentation, (S32) Seed candidate region extraction, (S33) Image region segmentation calculation, (S34) Obtain candidate segmentation region, (S35) Foreground / background region labeling, (S36) Output labeling Mask image result;

[0062] (S4) using an automated salient region extraction method for segmentation estimation and iterative learning of model parameters;

[0063] (S5) Result output.

[0064] Furthermore, the image described in step (S1) is an RGB image and a grayscale image, and the preprocessing is to remove noise and Gaussian blur the input image. The process is as follows: Figure 3 As shown. For the convenience of post-processing, you first need to input an RGB image and generate a grayscale image based on the RGB image. If there is only a grayscale image, it needs to be converted to an RGB image, that is, both the RGB image and the grayscale image need to exist. Next, perform basic image preprocessing operations such as noise removal and Gaussian blur on the input image. Digital image processing methods such as median filtering, Gaussian filtering, and bilateral filtering can be used. For images that are too large or too small, scaling operations can also be performed to ensure a balance between the algorithm effect and the processing speed.

[0065] Furthermore, in the step (S2), considering that the contrast of X-ray or low-light images is low and the foreground and background are not clearly distinguished, for such images, global image processing methods (such as overall image brightening or histogram equalization, etc.) will cause the loss of image details, which is not conducive to the subsequent segmentation and extraction of significant area images. Therefore, a self-designed image local enhancement processing method is used here to locally enhance the image, perform hierarchical processing according to the image grayscale, and map the original grayscale value of the image;

[0066] The custom grayscale mapping formula is:

[0067] G(x′, y′)=B(G(x, y)), G(x, y)∈[0,255] (1)

[0068] Where G(x, y) represents the grayscale value of the grayscale image at the coordinate (x, y);

[0069] Then the corresponding mapping formula B is:

[0070]

[0071] Where k represents the k-th interpolation curve, and the control points are from P j Click to P i There are i-j+1 points in total;

[0072] P and N in formula (2) are calculated as follows:

[0073]

[0074]

[0075] j represents the starting control point, i represents the iteration parameter, which is equivalent to the control point being the starting point from P j To P j+k , the parameters i and k in the basis function N(G(x, y)) of formula (4) are synchronized with those in formula (3). The image effect after local image enhancement is as follows Figure 1 (c) as shown.

[0076] Further, the adaptive binary segmentation described in step (S31): After completing the image enhancement of step (S2), the image needs to be binarized. The general binarization methods here are within the scope of protection of this application and do not affect the integrity of the method, including but not limited to OSTU threshold segmentation, adaptive threshold segmentation and other binarization methods. The segmented image is expressed in binary, where 255 represents the foreground (white) and 0 represents the background (black). Similarly, other binary expressions of numerical values ​​can also be used. The binary segmentation result is as follows: Figure 1 (d) as shown.

[0077] Furthermore, in the seed candidate region extraction described in step (S32): the binarized image region is the foreground region, and for the key region of the image, the image distance transformation method is used to extract the geometric center region of the image region, thereby realizing the extraction of the minimum seed region;

[0078] Assume that the binary image is A, the foreground point set is P, and the background point is Q, then the corresponding mathematical expression is:

[0079] D(p)=min(d(p, q)+f(q)), st p∈P, q∈Q, P∪Q=A (5)

[0080] Where d(p, q) is the distance between the foreground point and the background point, which may include but is not limited to Euclidean distance, Manhattan distance, etc., and f is a constraint function;

[0081] After obtaining the grid image D of the image distance transform, the image D is normalized. The normalized value interval is [0, 255]. The method is to find the maximum and minimum values ​​in the image D and scale it to 255 times. The obtained normalized image N is shown in formula (6):

[0082]

[0083] The image N is binarized again, and then morphological operations are performed to remove burrs. The seed candidate region is extracted. The result is as follows: Figure 1 (e) as shown.

[0084] Furthermore, in the image region segmentation calculation described in step (S33): the candidate region extraction result of step (S32) is used as the seed region input, and the numbering value starts from 1; for the foreground area that does not enter the candidate area in the binary image, it is marked as an unknown area and numbered 0, thereby obtaining the region segmentation result, that is:

[0085] W(S rgb )=W(R unknown , R seed1 , R seed2 , …, R seedn ) (7)

[0086] S rgb is the input RGB source image, where R unknown The unknown area is marked and filled with 0. seedi is the i-th seed region, filled with i values, i∈{1, 2,…, n}.

[0087] Further, in the foreground / background region annotation described in step (S35): in step (S33), the salient target region cannot be obtained only by image region segmentation calculation, but the output result of the image region segmentation calculation is included in the target solution; therefore, the output result of the image region segmentation calculation is used as the candidate segmentation region, and the final target region is obtained by further merging and dilating;

[0088] Here for W(S rgb ) area annotation, the annotation image M output satisfies formula (8):

[0089]

[0090] Where S rgb (x, y) is the (x, y) coordinate position in the source image, R bg is the background area, R oi is the region of interest, which comes from the image region segmentation calculation output result in step (S33), R pbg It is the suspected background area.

[0091] Furthermore, in the foreground / background area labeling described in step (S35), the automatic labeling process is as follows:

[0092] 1) Initialize the labeled image Mask, which is the same size as the original image;

[0093] 2) Set the initial value of the annotated image to suspected background mark 2;

[0094] 3) For the location of the region of interest obtained by segmentation, set it to 1 in the labeled image Mask;

[0095] 4) For the interference items that may affect the region of interest, the background flag is set to 0, that is, the region cannot be classified as the foreground. The final labeled segmented image is as follows: Figure 1 (f) as shown.

[0096] Furthermore, the annotated image obtained by (S3) can be iteratively segmented based on the algorithmic idea of ​​Grab Cut. Grab Cut is an image segmentation algorithm based on Graph Cut, which requires the user to input a rectangular bounding box as the segmentation target position to separate or segment the target from the background. However, in practical applications, the rectangular bounding box easily selects the background of irregular objects, and it is difficult to achieve regional segmentation in scenes where the front and back backgrounds are easily confused. At the same time, the rectangular bounding box requires manual interaction to produce, which makes the algorithm unsuitable for applications such as industrial automation assembly lines or unmanned driving. Therefore, this application adopts an automatic salient region extraction method for segmentation estimation and iterative learning of model parameters to achieve automation of salient region extraction. The specific process is as follows:

[0097] (S41) using the labeled image Mask in (S36) as a definition, defining the background, foreground and suspected background areas in the source image, and substituting them into the Gaussian mixture model (GMM) of the grab cut;

[0098] (S42) using the GMM model to model the background and foreground, and marking undefined pixels as possible foreground or background;

[0099] (S43) According to the Mask mark in formula (7), the data has been divided into n categories. Here, each category is set to be normally distributed. Next, the GMM model is used for fitting, and the probability of the data belonging to a certain Gaussian model is iteratively calculated to minimize its estimated energy, thereby completing the aggregation classification of the data;

[0100] (S44) Estimation energy iterative optimization method: Through the MaxFlow-MinCut model of the Grab cut graph, the image is mapped into a weighted undirected graph G=<V,E> , each node N∈V in the graph corresponds to each pixel in the image, each edge∈E connects a pair of adjacent pixels, and the edge weight represents the non-negative similarity between adjacent pixels in terms of grayscale, color or texture; a segmentation S of an image is a cut of the graph, and each segmented region R∈S corresponds to a subgraph in the graph; the optimal principle of segmentation is to maximize the similarity within the divided subgraphs, while minimizing the similarity between subgraphs. The output result is as follows Figure 1 (g) as shown.

[0101] Figure 1 (g) is the result output diagram of the above embodiment. More results are shown in Figure 5 As shown, it can be seen that the algorithm has strong robustness and has good robustness to irregular geometric figures, foreground and background confusion, small samples and so on.

[0102] This application aims to solve the problems of inability to obtain large-scale sample training, low image contrast, unclear foreground and background distinction in industrial scenes. By preserving image details for image enhancement and improving the interactive manual segmentation algorithm into an unsupervised learning self-annotation segmentation method, the significant area segmentation and extraction of irregular images with subject-background confusion under the premise of small samples and low contrast is achieved. It has good robust effects for scenes such as automated defect detection on industrial assembly lines and detection of significant target areas during the driving of unmanned vehicles. Moreover, the results of this application are not only effective for images in the industrial field, but also have good generalization effects for the extraction of interest areas in images under low illumination.

[0103] It should be pointed out that what is shown and described above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any changes, substitutions and modifications made by ordinary technicians in the field based on the embodiments of the present invention without making any creative work should fall within the scope of protection of the present invention.

Claims

1. Automatic segmentation method of salient regions in industrial X-ray images, It is characterized in that The following steps are involved: (S1) Input image and preprocessing; (S2) Image contrast and brightness enhancement with detail preservation; (S3) Adaptive generation of candidate regions: Before extracting salient regions, the initial screening of candidate regions is completed, including the following steps: (S31) Adaptive binary segmentation, (S32) Seed candidate region extraction, (S33) Image region segmentation calculation, (S34) Obtain candidate segmentation region, (S35) Foreground / background region annotation, (S36) Output annotation Mask image result; (S4) performing iterative segmentation based on the Grab Cut algorithm using the labeled Mask image result obtained in (S3); (S5) Result output; In the step (S2) described: The custom grayscale mapping formula is: (1) in Represents a grayscale image at coordinates The gray value at ; The corresponding mapping formula is for: (2) in express The interpolation calculation is performed, and the control points are Click to Points, a total of indivual; P and N in formula (2) are calculated as follows: (3) (4) It represents the starting control point. represents the iteration parameter, the basis function of formula (4) Parameters in Keep in sync with formula (3); In the image region segmentation calculation described in step (S33): the seed candidate region extraction result of step (S32) is used as the seed region input, and the numbering value starts from 1; for the foreground area in the binary image that does not enter the seed candidate area, it is marked as an unknown area and numbered 0, thereby obtaining the region segmentation result, that is: (7) is the input RGB source image, where The unknown area is marked and filled with 0. For the seed regions, Value filling, ; In the foreground / background region labeling described in step (S35), the automatic labeling process is as follows: 1) Initialize the labeled image Mask, which is the same size as the original image; 2) Set the initial value of the annotated image to suspected background mark 2; 3) For the position of the region of interest obtained by segmentation, set it to 1 in the labeled image Mask. The region of interest obtained by segmentation comes from the region segmentation result of step S33; 4) For any interference items that may affect the region of interest, the background flag is set to 0, that is, the region cannot be classified as the foreground.

Images