A multi-scale chip defect detection method based on sequence anomaly detection

Abstract

The present invention discloses a multi-scale chip defect detection method based on sequence anomaly detection, which improves the problem that the accurate size measurement and appearance detection of miniaturized chips in the prior art still need to improve efficiency. The invention contains the following steps: obtaining multiple chip images and converting them into two groups of one-dimensional time series; performing multi-scale analysis on the sequence data to obtain subsequences at multiple different scales; calculating the root mean square value of the difference between the sequence with defects and the sequence without defects at different scales, and obtaining the optimal detection scale after comparison; dividing the subsequence into T sequence segments of length N and obtaining a set of different states, converting the sequence to be tested into a state sequence; detecting the abnormal part in the time series to be tested by the time series anomaly detection algorithm; and obtaining the abnormal state position in the state sequence according to the anomaly detection result. This technology improves the speed and accuracy of detection and the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of chip defect detection, and in particular to a multi-scale chip defect detection method based on sequence anomaly detection. Background Art

[0002] With the rapid development of electronic information technology, the demand for high-quality and highly reliable chips in the civil, military and aviation fields is more urgent. The purpose of chip defect detection is to control the quality of chip shipments, eliminate defective products, and prevent defective products from entering the market. Therefore, it is particularly important to continue screening the chips before they leave the factory. In the process of chip screening, tests such as chip electrical properties are a necessary prerequisite for chip quality assurance, but the detection of chip appearance defects, such as chip marking, surface integrity, chip pins, chip size, etc., also has a far-reaching impact on ensuring the reliability of the chip. At present, third-party chip screening manufacturers mainly use traditional manual visual inspection methods in the field of chip appearance defect detection, which requires a lot of manpower and material resources, has high detection costs, low detection accuracy, and high missed detection and false detection rates. In view of the shortcomings of manual detection, the method of machine vision detection is gradually promoted. At the same time, with the development of artificial intelligence, the integration of machine learning methods with machine vision promotes more intelligent detection methods.

[0003] However, existing chip defect detection technologies based on machine learning and machine vision generally adopt supervised learning methods, which require a large amount of manual labeling to set labels for training. Moreover, in general, the contrast of defect images is low, and simply detecting by edge detection, gradient value or grayscale value will often lead to missed detection. In addition, miniaturized chips often require high-precision imaging equipment when achieving accurate size measurement and appearance inspection. The cost of these devices is several times higher than that of general imaging equipment. Existing chip defect detection technologies have the problems of high imaging cost, slow detection speed, low detection accuracy and weak generalization ability.

[0004] Therefore, this application proposes a multi-scale chip defect detection method based on sequence anomaly detection to solve the above problems. Summary of the invention

[0005] The present invention improves the problem that the efficiency of accurate size measurement and appearance inspection of miniaturized chips in the prior art still needs to be improved, and provides a multi-scale chip defect detection method based on sequence anomaly detection with low imaging cost, fast detection speed, high detection accuracy and strong generalization ability.

[0006] The technical solution of the present invention is to provide a multi-scale chip defect detection method based on sequence anomaly detection having the following steps:

[0007] Step 1, obtaining images of a chip without defects, images of a chip with defects, and images of a chip to be detected for defects, and converting the image data after graying into two sets of one-dimensional time series respectively;

[0008] Step 2, perform multi-scale analysis on the sequence data to obtain subsequences at multiple different scales;

[0009] Step 3, determine the best defect detection scale, calculate the root mean square value of the difference between the sequence with defects and the sequence without defects at different scales, obtain the best detection scale after comparison, and select the sequence at the best scale as the subsequence to be tested;

[0010] Step 4, identifying the sequence state, dividing the subsequence into T sequence segments of length N, classifying each sequence segment according to its features, obtaining a set of different states, and then converting the sequence to be tested into a state sequence according to the correspondence between the sequence segments and the states;

[0011] Step 5, detecting the abnormal part in the time series to be tested by using a time series anomaly detection algorithm;

[0012] Step 6, determine the location of the defect, obtain the abnormal state position in the state sequence based on the abnormal detection result, and further determine the location of the defect.

[0013] Preferably, the step 3 comprises the following steps,

[0014] Step 3.1, determine the optimal defect detection scale of the horizontal axis, calculate the similarity between the sequence with defects and the sequence without defects at different scales of the horizontal axis sequence, compare the similarity, consider the scale with the smallest similarity as the optimal detection scale of the horizontal axis, and select the sequence at this scale as the sequence to be detected;

[0015] Step 3.2, determine the optimal scale for defect detection on the ordinate, calculate the similarity between the sequence with defects and the sequence without defects at different scales of the ordinate sequence, compare the similarities, consider the scale with the smallest similarity as the optimal detection scale for the abscissa, and select the sequence at this scale as the sequence to be detected.

[0016] Preferably, the step 4 comprises the following steps,

[0017] Step 4.1, the subsequences of the horizontal and vertical coordinate sequences of the defect-free image at different scales are divided into multiple sequence segments; then, according to the characteristics of each sequence segment, such as but not limited to the mean, variance, maximum value and minimum value, the sequence segments of the characteristic phase are defined as the same state, and a set of different states is obtained;

[0018] Step 4.2: Segment the horizontal and vertical coordinate sequences of the defect detection image and identify their states, and then convert the subsequences at different scales into state sequences according to the order of each sequence segment.

[0019] Preferably, the specific process of establishing the Markov anomaly detection model in step 5 to realize anomaly detection of the sequence is as follows:

[0020] Step 5.1, construct the Markov state transfer matrix: The Markov state transfer matrix construction is divided into the following steps: First, calculate the initial state transfer probability M i It represents the number of sequence segments in state i in the subsequence at different scales of the horizontal and vertical coordinate sequence of the defect-free image, N represents the total number of states contained in a subsequence, and then calculates the state transition probability in Indicates that from state S τ Transfer to state S γ The number of times, and then establish a state transition probability matrix Finally, repeat the above steps to establish the Markov state transfer matrix at different scales of the horizontal and vertical coordinates.

[0021] Step 5.2: Calculate the cumulative state transition probability of the sequence to be detected and determine whether it is abnormal. The following steps are included: First, a sliding window of a certain length is given; then, according to the corresponding state transition matrix, the state s that appears at the earliest time in the window is calculated. t-l+1 As time passes, each state is transferred to the last state s t-1 The cumulative probability Finally, when P(s t |s t-l+1 …s t-1 ) is lower than a certain threshold, it is considered that the current last state s t-1 abnormal.

[0022] Preferably, in step 6, in an extreme case, at the optimal scale, multiple abnormal states appear in the horizontal coordinate state sequence or the vertical coordinate state sequence, that is, there are multiple positions in the image that may have defects. If it is necessary to further narrow the scope of the defects, the abnormality detection is repeated for the sequences at all scales; the position (x1, y1) detected as abnormal the most times will be determined as the position of the defect, and then all positions on the horizontal coordinate x=x1 and the vertical coordinate y=y1 will be excluded; the position (x2, y2) detected as abnormal the most times in the remaining positions will be determined as the position of the defect, and then all positions on the horizontal coordinate x=x2 and the vertical coordinate y=y2 will be excluded; after repeated multiple times, the coordinates (x1, y1) of all defect positions will be obtained. i ,y j )

[0023] Compared with the prior art, the multi-scale chip defect detection method based on sequence anomaly detection of the present invention has the following advantages: low-cost industrial cameras inevitably introduce a large amount of noise in imaging projects, and these noises will drown out small-sized defects on the chip, such as thin and deep scratches. Step three continues the multi-scale analysis of the sequence, which can fully highlight the characteristics of certain aspects of the problem through transformation, can perform localized analysis of time (space) frequency, and gradually perform multi-scale refinement of the signal (function) through telescoping and translation operations, ultimately achieving time subdivision at high frequencies and frequency subdivision at low frequencies, thereby focusing on any details of the signal. It can effectively detect the performance requirements of the model for imaging equipment and improve the speed and accuracy of detection.

[0024] Step five converts the image recognition problem into a sequence anomaly detection problem. On the one hand, it reduces the model complexity of the image defect detection algorithm and improves the robustness and generalization ability of the model. Ordinary technicians can easily generalize it to other types of chip defect detection scenarios. On the other hand, it reduces the hardware computing power requirements of the detection model on the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is an image of a chip without defects in the present invention;

[0026] Figure 2 is a chip image to be defect-detected in the present invention;

[0027] Figure 3 is a schematic diagram of a horizontal coordinate sequence of an image without defects and its subsequences at various scales in the present invention;

[0028] Figure 4 is a schematic diagram of a vertical coordinate sequence of an image without defects and its subsequences at various scales in the present invention;

[0029] Figure 5 is a schematic diagram of the horizontal coordinate sequence of images with defects and images without defects and their local differences at various scales in the present invention;

[0030] Figure 6 It is a schematic diagram of the vertical coordinate sequence of images with defects and images without defects and their local differences at various scales in the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] The following is a further description of the multi-scale chip defect detection method based on sequence anomaly detection of the present invention in conjunction with the accompanying drawings and specific embodiments: The main theoretical steps of this embodiment are as follows:

[0033] Sequence construction

[0034] Use industrial cameras to obtain images of chips without defects and images of chips with defects (including but not limited to defects such as missing chip pins, folded chip pins, and chip surface wear). Use industrial cameras to obtain images of chips to be inspected for defects.

[0035] The image is uniformly adjusted to n×m pixel format, and the image is converted into a grayscale image. Read each pixel point R(x, y), G(x, y), B(x, y) in turn, calculate the grayscale value Grey(x, y) (converted to an integer), assign the grayscale value to the corresponding position of the new image, and complete the conversion after traversing all pixel points. The calculation formula is as follows:

[0036] Grey(x, y)=0.299×R(x, y)+0.587×G(x, y)+0.114×B(x, y) (1)

[0037] The grayed image raw data can be regarded as an n×m matrix, Grey(x, y) represents the grayscale at the coordinate (x, y). To facilitate subsequent work, the data is converted into two sets of one-dimensional sequences x(i) and y(j). x(i) represents the average grayscale value of all pixels with abscissa i (i=1,2,3,...,n); y(j) represents the average grayscale value of all pixels with ordinate j (j=1,2,3,...,m). The calculation formula is as follows:

[0038]

[0039] An image without defects can be represented as Grey n×m (x, y), the defective image can be represented as Grey′ n×m (x, y). Grey n×m (x, y) and Grey′ n×m (x, y) are converted to x(i), y(j) and x′(i) and y′(i) respectively.

[0040] The image to be inspected can be represented as Grey″ n×m (x, y), Grey″ n×m (x, y) is converted to x″(i) and y″(i).

[0041] Sequential Multiscale Analysis

[0042] The reason why the present invention needs to perform multi-scale analysis is that low-cost industrial cameras introduce a lot of noise in the imaging process, which will drown out small-size defects (such as thin and deep scratches) on the chip. As a multi-scale analysis method, wavelet analysis is mainly characterized by being able to fully highlight certain aspects of the problem through transformation, localized analysis of time (space) frequency, and gradually multi-scale refinement of the signal (function) through telescoping and translation operations, ultimately achieving time subdivision at high frequencies and frequency subdivision at low frequencies, thereby focusing on any details of the signal.

[0043] Preferably, a fast discrete wavelet transform algorithm proposed by Mallat is selected. The formula for wavelet decomposition of the signal by this algorithm is as follows:

[0044] A0[f(t)]=f(t) (4)

[0045] A j [f(t)]=∑ k H(2t-k)A j-1 [f(t)] (5)

[0046] D j [f(t)]=∑ k G(2t-k)A j-1 [f(t)] (6)

[0047] f(t) is the sequence to be decomposed, A j is the wavelet coefficient of the approximate part (low-frequency part) of f(t) at layer j, D j It is the wavelet coefficient (i.e. decomposition result) of the detail part (high frequency part) of f(t) at layer j. Where j is the number of layers (j=1, 2, 3, ..., J; J=log2N). H and G represent the wavelet analysis filters in the time domain.

[0048] After decomposing x(i) and x′(i) into d subsequences of different scales in the above way, we can reconstruct x1(i), x2(i)…x d (i) and x′1(i), x′2(i)…x′ d (i); Similarly, decompose y(j) and y′(i) into d subsequences of different scales to obtain y1(j), y2(j)…y d(j) and y′1(j), y′2(j)…y′ d (j); Similarly, decompose x″(i) and y″(i) into d subsequences of different scales to obtain x″1(i), x″2(i)…x″ d (i) and y″1(j), y″2(j)…y″ d (j).

[0049] Determination of the optimal defect detection scale

[0050] 3.1 Determine the optimal defect detection scale of the horizontal axis

[0051] Calculate the similarity between the sequence with defects and the sequence without defects at different scales of x(i), preferably calculate the root mean square value e of the difference between the two sequences x1 , e x2 …e xd .

[0052]

[0053] Compare x1 , e x2 …e xd The size of e x1 , e x2 …e xd Sort from large to small, considering the largest e μ The scale where (μ∈{x1, x2…xd}) (similarity is minimum) is the optimal detection scale on the horizontal axis.

[0054] 3.2 Determine the optimal scale for vertical axis defect detection

[0055] Similarly, the similarity between the sequence with defects and the sequence without defects at different scales of y(j) is calculated, preferably the root mean square value of the difference between the two sequences is calculated as e y1 , e y2 …e yd , and for e y1 , e y2 …e yd Sort from large to small, considering the largest e ε The scale where (ε∈{y1, y2…yd}) (similarity is minimum) is the optimal detection scale on the horizontal axis.

[0056] Select the horizontal and vertical coordinate subsequence of the sequence under the optimal scale as the sequence to be detected as anomaly.

[0057] Sequence state recognition

[0058] 4.1 Sequence segmentation and state recognition

[0059] The defect-free image horizontal coordinate sequence xμ (i) Divide it into T sequence segments of length N {X μ (N), X μ (2N)…X μ (TN)}, where T ≤ n / N < T + 1. That is, X μ (N) = {x μ (1), x μ (2)…x μ (N)}; …; X μ (TN) = {x μ ((T - 1)N + 1), x μ ((T - 1)N + 2)…x μ (TN)}. Then, according to the characteristics of each sequence segment (including but not limited to mean, variance, maximum value, and minimum value), define the sequence segments with the same characteristics as the same state s, and obtain the set S μ = {s1, s2, s3…s τ}(0 < τ < n).

[0060] Similarly, divide the ordinate sequence y ε (j) of the defect - free image into K sequence segments of length F {Y ε (F), Y ε (2F)…Y ε (KF)}, where K ≤ m / F < K + 1. That is, Y ε (F) = {y ε (1), y ε (2)…y ε (F)}; …; Y ε (KF) = {y ε ((K - 1)F + 1), y ε ((K - 1)F + 2)…y ε (KF)}. Divide the states to obtain the state set U yε = {u1, u2, u3…u ∈}(0 < ε < m).

[0061] 4.2 Construction of the state sequence for anomaly detection

[0062] Divide the abscissa sequence x″ μ (i) of the image to be defect - detected into T sequence segments of length N X″ μ (n) = {X″ μ (N), X″ μ (2N)…X″ μ (TN)}, and divide the ordinate sequence y″ ε (j) of the image to be defect - detected into K sequence segments of length F Y″ ε (k) = {Y″ε (F), Y″ ε (2F)…Y″ ε (KF)}.

[0063] According to the correspondence between the sequence segments and states obtained in step 4.1, the state set S″ is obtained xμ ={s1, s2, s3…s τ} and U″ yε = {u1, u2, u3…u ∈}, and replace x″ in the order of each sequence segment. μ (n) and Y″ ε (k) Transformed into state sequence S″ xμ (n) and S″ yε (k).

[0064] Anomaly Detection

[0065] Preferably, a detection method based on Markov state transition theory is used instead of directly comparing the difference between the sequence without defects and the sequence to be detected for abnormality to determine whether the chip has defects. This is because the direct comparison method has strict requirements on the placement of the chip, and a slight position error will lead to a significant increase in the false detection rate. Especially on a fast-moving conveyor belt, the chip will inevitably be slightly displaced by vibration and inertia, which will lead to low accuracy, and reducing the conveyor belt speed will lead to low detection efficiency.

[0066] In step 4, x1(i), x2(i)…x d (i) is divided into several states and the corresponding state set S is obtained xμ ={s1, s2, s3…s τ}; y1(j), y2(j)…y d (j) is divided into several states and the corresponding state set U is obtained yε = {u1, u2, u3…u ∈}.

[0067] 5.1 Markov state transfer matrix construction

[0068] Construct x1(i), x2(i)…x d (i) and y1(j), y2(j)…y d (j) The corresponding transition probability matrix P x1 , P x2 …P xd , and P y1 , P y2 …P yd The Markov state transfer matrix is ​​constructed as follows, and the parameter set of a Markov model is expressed as:

[0069] λ={S,Q,P} (8)

[0070] Where λ represents the Markov model, s=(S1,S2…S N ) represents all possible states in the sequence. Q = {q1, q2, q3, ..., q N} represents the initial state transition probability set:

[0071]

[0072] M i represents the number of states i in the sequence, and N represents the total number of states.

[0073] Establish a state transition probability matrix P:

[0074]

[0075] in, Indicates that in state S τ After one step, it transfers to state S γ If the state s at time t-1 t-1 =S3, s at time t t =S5, then the state transition probability at this time is

[0076] The calculation formula is:

[0077]

[0078] Indicates that from state S τ Transfer to state S γ The expansion form of the state transition probability matrix P is as follows:

[0079]

[0080] Repeat the above steps to obtain x1(i), x2(i)…x d (i) and y1(j), y2(j)…y d (j) The corresponding state transition probability matrix P x1 , P x2 …P xd , and P y1 , P y2 …P yd .

[0081] Get x1(i), x2(i)…x d (i) and y1(j), y2(j)…yd (j) corresponding Markov model λ x1 , λ x2 …λ xd and λ y1 , λ y2 …λ yd .

[0082] 5.2 Anomaly Detection Based on Markov Model

[0083] According to Step 3, the sequence at the optimal scale is selected as the sequence to be detected for anomalies. In Step 4, the sequence to be detected for anomalies has been transformed into a state sequence S″(n) = {s1, s2…s T}. Calculate the cumulative state transition probability P(s t |s t-l+1 ,..., s t-1 ), which represents the cumulative probability that the state s t-l+1 that appears at the earliest moment transfers to the last state s t-1 along each state in a sliding window of length l. If this probability is smaller, the possibility that the last state is abnormal is greater. The calculation formula is as follows:

[0084]

[0085] If the cumulative state transition probability P(s t |s t-l+1 …s t-1 ) is lower than a certain threshold, then the state s t-1 is considered an abnormal state.

[0086] Detect the anomalies in the state sequence S″ xμ (n) according to the above method; similarly, detect the anomalies in S″ yε (k).

[0087] If no "anomaly" is detected, it is considered that there are no defects in this batch of chips; if an "anomaly" is detected, then proceed to the next step to determine the location of the defect.

[0088] Determine the location of the defect

[0089] According to the detection result in Step 5, if s xμ in the state sequence S″ a (n) (0 < a ≤ T) is an abnormal state, then the abscissa position of the defect is x i = a × T.

[0090] Similarly, if u vε in the state sequence set S″ b (k) (0 < b ≤ F) is an abnormal state, then the ordinate position of the defect is yj =b×F.

[0091] In summary, the location of the detected chip defect is (a×T, b×F).

[0092] For extreme cases, the best scale detection S″ x and U″ y If there is an abnormal state in the image (that is, there is more than one chip defect in the image), there may be defects in multiple locations. If there is a need to further narrow the scope of the defect, repeat step 5 to perform abnormality detection on sequences at all scales.

[0093] The position (x1, y1) detected as abnormal the most times will be determined as the position of the defect, and then all positions on the horizontal coordinate x = x1 and the vertical coordinate y = y1 will be excluded; among the remaining positions, the position (x2, y2) detected as abnormal the most times will be determined as the position of the defect, and then all positions on the horizontal coordinate x = x2 and the vertical coordinate y = y2 will be excluded; after repeating several times, the coordinates of all defect positions (x i ,y j ).

[0094] The specific calculation process of this embodiment is given below to achieve a clear and complete description of the technical solution in this embodiment.

[0095] Sequence construction

[0096] Use industrial cameras to obtain images of defect-free chips and defective chips. Figure 1 As shown in the figure, the chip image with defects and the chip image to be detected are as follows: Figure 2 shown.

[0097] According to formula 1, the image is uniformly adjusted to a 1900×1400 pixel format and converted into a grayscale image. Read R(x, y), G(x, y), and B(x, y) of each pixel point in turn, calculate the grayscale value Grey(x, y) (converted to an integer), assign the grayscale value to the corresponding position of the new image, and complete the conversion after traversing all the pixels once.

[0098] The grayed image raw data can be regarded as a 1900×1400 matrix, Grey(x, y) represents the grayscale at the coordinate (x, y). To facilitate subsequent work, the data is converted into two sets of one-dimensional sequences x(i) and y(j). x(i) represents the average grayscale value of all pixels with ordinate i (i=1,2,3,...,n); y(j) represents the average grayscale value of all pixels with ordinate j (j=1,2,3,...,m). The calculation formula is as follows:

[0099]

[0100] An image without defects can be represented as Grey 1900×1400 (x, y), the defective image can be represented as Grey′ 1900×1400 (x, y). Grey 1900×1400 (x, y) and Grey′ 1900×1400 (x, y) are converted to x(i), y(j) and x′(i) and y′(i) respectively.

[0101] The image to be inspected can be represented as Grey″ 1900×1400 (x, y), Grey″ 1900×1400 (x, y) is converted to x″(i) and y″(i).

[0102] An image without defects can be represented as Grey 1900×1400 (x, y), the defective image can be represented as Grey′ 1900×1400 (x, y). Grey 1900×1400 (x, y) and Grey′ 1900×1400 (x, y) are converted to x(i), y(j) and x′(i) and y′(i) respectively.

[0103] Sequential Multiscale Analysis

[0104] Preferably, the “db4” wavelet is used to decompose and reconstruct x(i) and x′(i) to obtain 5 subsequences at different scales, namely x1(i), x2(i), x3(i), x4(i), x5(i) and x′1(i), x′2(i), x′3(i), x′4(i), x′5(i); similarly, y(j) and y′(i) are decomposed and reconstructed to obtain y1(j), y2(j), y3(j), y4(j), y5(j) and y′1(j), y′2(j), y′3(j), y′4(j), y′5(j).

[0105] The image to be inspected can be represented as Grey″ 1900×1400 (x, y), Grey″ 1900×1400 (x, y) is converted to x″(i) and y″(i). Then, as in step 2, x″(i) and y″(i) are decomposed and reconstructed to obtain x″1(i), x″2(i), x″3(i), x″4(i), x″5(i) and y″1(j), y″2(j), y″3(j), y″4(j), y″5(j).

[0106] The horizontal axis sequence of the defect-free image and its subsequences at each scale are as follows: Figure 3As shown; the vertical coordinate sequence of the image without defects and its subsequences at each scale are as follows Figure 4 shown.

[0107] The horizontal coordinate sequence of the image with and without defects and the local differences at each scale are as follows: Figure 5 As shown; the ordinate sequence of images with and without defects and their local differences at each scale are shown as Figure 6 shown.

[0108] Determination of the optimal defect detection scale

[0109] 3.1 Determine the optimal defect detection scale of the horizontal axis

[0110] Calculate the similarity between the sequence with defects and the sequence without defects at different scales of x(i), preferably calculate the root mean square value e of the difference between the two sequences x1 , e x2 , e x3 , e x4 and e x5 .

[0111]

[0112] Compare x1 , e x2 , e x3 , e x4 and e x5 The size of e x3 The third scale is the best detection scale on the horizontal axis.

[0113] 3.2 Determine the optimal scale for vertical axis defect detection

[0114] Similarly, the similarity between the sequence with defects and the sequence without defects at different scales of y(j) is calculated, preferably the root mean square value of the difference between the two sequences is calculated as e y1 , e y2 , e y3 , e y4 and e y5 .

[0115]

[0116] where e y1 The first scale is the largest (similarity is the smallest), so the first scale is the optimal detection scale on the horizontal axis.

[0117] Select the sequences x″3(i) and y″1(j) under the optimal scale as the sequences to be detected as anomaly.

[0118] Sequence state recognition

[0119] 4.1 Sequence segmentation and state recognition

[0120] The defect-free image horizontal coordinate sequence x μ (i) Divide into T = 190 sequence segments of length N = 10 {X(N), x(2N) ... X(TN)} = {X(10), X(20) ... X(1900)}. Among them, X(N) = X(10) = {x μ (1), x μ (2)…x μ (10)};…;X(TN)=x(1900)={x μ (1891), x μ (1892)…x μ (1900)}. Then, according to the characteristics of each sequence segment (mean, variance, maximum and minimum), the sequence segments with the same characteristics are defined as the same state s, and a set of 10 different states S is obtained. x ={s1, s2, s3…s 10}.

[0121] Similarly, the defect-free image ordinate sequence yε(j) is divided into K = 140 sequence segments {Y(F), Y(2F)…Y(KF)} = {Y(10), Y(20)…Y(1400)} of length F = 10. ε (1), y ε (2)…y ε (F)}; ...; Y(KF) = Y(1400) = {y5(1391), y5(1392) ... y5(1400)}. Then, according to the characteristics of each sequence segment (mean, variance, maximum and minimum), the sequence segments with the same characteristics are defined as the same state s, and a set of 10 different states U is obtained. y = {u1, u2, u3…u 10}

[0122] 4.2 Construction of state sequence to be detected

[0123] The horizontal coordinate sequence x″ of the image to be inspected μ (i) Divide into T = 190 sequence segments of length N = 10 {X″(10), X″(20)…X″(1900)}. ε (j) Split into K = 140 sequence segments {Y″ ε (10), Y″ ε (20)…Y″ ε (1400)}.

[0124] According to the correspondence between the sequence segments and the states obtained in step 4.1, the state set S″ is obtained xμ ={s1, s2, s3…s τ} and U″ yε = {u1, u2, u3…u ∈}, according to the order of each sequence segment, X″ μ (n) and Y″ ε (k) According to the conversion into state sequence S″ xμ (n) and S″ yε (k).

[0125] Anomaly Detection

[0126] In step 4, x1(i), x2(i)…x5(i) are split into a set S of states x1 , S x2 …S x5 ; y1(j), y2(j)…y5(j) are divided into a set U of several states y1 , U y2 …U y5 .

[0127] 5.1 Markov state transfer matrix construction

[0128] Construct the state transition probability matrix P corresponding to x1(i), x2(i)…x5(i) and y1(j), y2(j)…y5(j) respectively x1 , P x2 …P x5 , and P y1 , P y2 …P y5 .

[0129] The parameter set of a Markov model is expressed as:

[0130] λ={S,Q,P} (13)

[0131] Where λ represents the Markov model, S=(S1,S2…S N ) represents all possible states in the sequence. Q = {q1, q2, q3, ..., q N} represents the initial state transition probability set:

[0132]

[0133] M i represents the number of states i in the sequence, and N represents the total number of states.

[0134] Establish a state transition probability matrix P:

[0135]

[0136] in, Indicates that in state S τ After one step, it transfers to state S γ If the state s at time t-1 t-1 =S3, s at time t t =S5, then the state transition probability at this time is

[0137] The calculation formula is:

[0138]

[0139] Indicates that from state S τ Transfer to state S γ The expansion form of the state transition probability matrix P is as follows:

[0140]

[0141] According to the above steps, we can obtain the state transition probability matrix P corresponding to x1(i), x2(i)…x5(i) and y1(j), y2(j)…y5(j) respectively. x1 , P x2 …P x5 , and P y1 , P y2 …P y5 .

[0142] Get the Markov model λ corresponding to x1(i), x2(i)…x5(i) and y1(j), y2(j)…y5(j) x1 ,λ x2 …λ x5 and λ y1 ,λ y2 …λ y5 .

[0143] 5.2 Anomaly Detection Based on Markov Model

[0144] Convert the sequence to be detected abnormally into the state sequence S″(n)={s1, s2…s 190}, calculate the cumulative state transition probability P(s t |s t-9 , ..., s t-1 ), which represents the earliest state s in a sliding window of length l = 10 t-9 As time passes, each state is transferred to the last state s t-1The smaller the probability, the greater the possibility that the final state is abnormal. The calculation formula is as follows:

[0145]

[0146] If the cumulative state transition probability P(s t |s t-1+1 ...s t-1 ) is lower than a certain threshold k = 0.5 × 10 -9 , the status of this position is considered abnormal.

[0147] According to the above method, the state sequence S″ is detected x3 The anomaly in (n) is detected and it is found that anomalies exist at x=a1; x=a2; x=a3 and x=a4.

[0148] Similarly, detect S″ y1 The anomaly in (k) is detected and found to exist at y=b1; y=b2; y=b3 and y=b4.

[0149] Determine the location of the defect

[0150] According to the detection results in step 5, the state sequence S″ is obtained x3 (n) There are anomalies at x = a1 × 10; x = a2 × 10; x = a3 × 10 and x = a4 × 10. Similarly, the state sequence S″ y1 In (k), there are anomalies at y=b1×10; y=b2×10; y=b3×10 and y=b4×10. It can be concluded that there may be multiple defects in the batch of chips to be tested.

[0151] The horizontal coordinate position of the defect is x i =a i ×10(i=1,2,3,4,5); the ordinate position of the defect is y j =b j ×10(j=1,2,3,4,5). In summary, the possible locations of chip defects are (x i ,y j ), i.e. (x1, y1); (x1, y2); (x1, y3); (x1, y4); (x2, y1); (x2, y2); (x2, y3); (x2, y4); (x3, y1); (x3, y2); (x3, y3); (x3, y4); (x4, y1); (x4, y2); (x4, y3); (x4, y4), a total of 16 positions.

[0152] To further narrow the scope of the defect, repeat the anomaly detection on the sequences at all scales.

[0153] The position (x1, y1) is detected as abnormal 4 times, and it is considered that there is a defect at (x1, y1). Then after excluding all positions on the horizontal coordinate x=x1 and the vertical coordinate y=y1, the following remain: (x2, y2); (x2, y3); (x2, y4); (x3, y2); (x3, y3); (x3, y4); (x4, y2); (x4, y3); (x4, y4).

[0154] Among the remaining positions, (x3, y3) will be judged as abnormal three times, and it is considered that there is a defect at (x3, y3). Then, after excluding all positions on the horizontal coordinate x=x3 and the vertical coordinate y=y3, there are still (x2, y2); (x2, y4); (x4, y2); (x4, y4).

[0155] Among the remaining positions, (x2, y2) will be judged as abnormal twice, and it is considered that there is a defect at (x2, y2). Then, after excluding all positions on the horizontal coordinate x=x2 and the vertical coordinate y=y2, (X4, y4) remains.

[0156] Finally, the defect locations are (x1, y1), (x3, y3), (x2, y2) and (x4, y4), as shown in Figure 2 shown.

[0157] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-scale chip defect detection method based on sequence anomaly detection, characterized in that: Contains the following steps, Step 1: Obtain images of defect-free chips, defective chips, and chips to be inspected for defects, and convert the grayed image data into two sets of one-dimensional time series, adjust the images to n×m pixel format, and convert the images into grayscale images. Read R(x, y), G(x, y), and B(x, y) of each pixel point in turn, calculate the grayscale value Grey(x, y), convert it to an integer, and assign the grayscale value to the corresponding position of the new image. After traversing all the pixels once, the conversion is completed. The calculation formula is as follows: Grey(x, y)=0.299×R(x, y)+0.587×G(x, y)+0.114×B(x, y) The grayed image raw data is regarded as an n×m matrix, Grey(x, y) represents the grayscale at the coordinate (x, y), and the data is converted into two sets of one-dimensional sequences x(i) and y(j), where x(i) represents the average grayscale value of all pixels with ordinate i (i=1, 2, 3, ..., n); y(j) represents the average grayscale value of all pixels with ordinate j (j=1, 2, 3, ..., m), and the calculation formula is as follows: Images without defects are represented as Grey n×m (x, y), the defective image is represented by Grey′ n×m (x, y), Grey n×m (x, y) and Grey′ n×m (x, y) are converted to x(i), y(j) and x′(i) and y′(i) respectively, and the image to be detected is represented by Grey″ n×m (x, y), Grey″ n×m (x, y) is converted to x”(i) and y”(i); Step 2, perform multi-scale analysis on the sequence data to obtain subsequences at multiple different scales; Step 3, determine the best defect detection scale, calculate the root mean square value of the difference between the sequence with defects and the sequence without defects at different scales, obtain the best detection scale after comparison, and select the sequence at the best scale as the subsequence to be tested; Step 3.1 Determine the optimal defect detection scale of the horizontal axis, calculate the similarity between the sequence with defects and the sequence without defects at different scales of x(i), and calculate the root mean square value e of the difference between the two sequences x1 , e x2 …e xd , Compare x1 , e x2 …e xd The size of e x1 , e x2 …e xd Sort from large to small, considering the largest e μ The scale where (μ∈{x1, x2…xd}) is located is the optimal detection scale of the horizontal axis; Step 3.2 Determine the optimal scale for vertical axis defect detection. Similarly, calculate the similarity between the sequence with defects and the sequence without defects at different scales of y(j), and calculate the root mean square value of the difference between the two sequences e y1 , e y2 …e yd , and for e y1 , e y2 …e yd Sort from large to small, considering the largest e ε The scale where (ε∈{y1, y2…yd}) is located is the optimal detection scale of the horizontal axis, and the horizontal and vertical subsequences of the sequence under the optimal scale are selected as the sequence to be detected as anomaly; Step 4, identifying the sequence state, dividing the subsequence to be tested into T sequence segments of length N, classifying each sequence segment according to its features, obtaining a set of different states, and then converting the subsequence to be tested into a state sequence according to the correspondence between the sequence segments and the states; Step 4.1 Sequence segmentation and state recognition The defect-free image horizontal coordinate sequence x μ (i) Split into T sequence segments of length N {X μ (N), X μ (2N)…X μ (TN)}, where T≤n / N<T+1, that is, X μ (N) = {x μ (1), x μ (2)…x μ (N)};…;X μ (TN) = {x μ ((T-1)N+1), x μ ((T-1)N+2)…x μ (TN)}; then, according to the characteristics of each sequence segment, including but not limited to the mean, variance, maximum value and minimum value, the sequence segment of the characteristic phase is defined as the same state s, and the set of different states S is obtained μ ={s1, s2, s3…s τ }(0<τ<n); Similarly, the ordinate sequence y of the defect-free image ε (j) Divide into K sequence segments of length F {Y ε (F), Y ε (2F)…Y ε (KF)}, where K≤m / F<K+1; that is, Y ε (F) = {y ε (1), y ε (2)…y ε (F)};…;Y ε (KF) = {y ε ((K-1)F+1), y ε ((K-1)F+2)…y ε (KF)}, divide the state to get the state set U yε = {u1, u2, u3…u ∈ }(0<ε<m); 4.2 Construction of state sequence to be detected The horizontal coordinate sequence x″ of the image to be inspected μ (i) Divide into T sequence segments X″ of length N μ (n) = {X″ μ (N), X″ μ (2N)…X″ μ (TN)}, the ordinate sequence y″ of the image to be inspected ε (j) Divide into K sequence segments Y″ of length F ε (k) = {Y″ ε (F), Y″ ε (2F)…Y″ ε (KF)}; According to the correspondence between the sequence segments and states obtained in step 4.1, the state set S″ is obtained xμ ={s1, s2, s3…s τ } and U″ yε = {u1, u2, u3…u ∈ }, and replace X″ in the order of each sequence segment. μ (n) and Y″ ε (k) Transformed into state sequence S″ xμ (n) and S″ yε (k); Step 5: Detect the abnormal part of the subsequence to be tested through the time series anomaly detection algorithm; establish a Markov anomaly detection model to realize anomaly detection of the sequence. The specific process is as follows: Step 5.1, construct the Markov state transfer matrix: The Markov state transfer matrix construction is divided into the following steps: First, calculate the initial state transfer probability M i represents the subsequence of the horizontal and vertical coordinates of the defect-free image to be tested, that is, the number of sequence segments in the sequence at the optimal scale in state i, N represents the total number of states contained in a subsequence to be tested, and then calculates the state transition probability in Indicates that from state S τ Transfer to state S γ The number of times, and then establish a state transition probability matrix Finally, repeat the above steps to establish the Markov state transfer matrix at different scales of horizontal and vertical coordinates; Step 5.2: Calculate the cumulative state transition probability of the subsequence to be detected and determine whether it is abnormal. The following steps are included: First, a sliding window of a certain length is given; then, according to the corresponding state transition matrix, the state s that appears at the earliest time in the window is calculated. t-l+1 As time passes, each state is transferred to the last state s t-1 The cumulative probability Finally, when P(s t |s t-l+1 …s t-1 ) is lower than a certain threshold, it is considered that the current last state s t-1 abnormal; Step 6, determine the location of the defect, obtain the abnormal state position in the state sequence based on the abnormal detection result, and further determine the location of the defect.

2. The multi-scale chip defect detection method based on sequence anomaly detection according to claim 1, characterized in that: In step 6, in the extreme case, at the optimal scale, multiple abnormal states appear in the horizontal coordinate state sequence or the vertical coordinate state sequence, that is, there are multiple positions in the image that may have defects. If it is necessary to further narrow the scope of the defects, the abnormality detection is repeated for the sequences at all scales; the position (x1, y1) detected as abnormal the most times will be determined as the position of the defect, and then all positions on the horizontal coordinate x=x1 and the vertical coordinate y=y1 will be excluded; the position (x2, y2) detected as abnormal the most times in the remaining positions will be determined as the position of the defect, and then all positions on the horizontal coordinate x=x2 and the vertical coordinate y=y2 will be excluded; after repeated multiple times, the coordinates of all defect positions (x1, y1) will be obtained. i ,y j ).

Images