Chip end face image definition evaluation method and system based on semantic information

Abstract

The application provides a chip end face image definition evaluation method and system based on semantic information, comprising: obtaining a chip end face image to be evaluated; inputting the chip end face image to an image degradation feature extraction module, extracting the degradation features of the image and performing global average pooling processing, and obtaining a guide vector through MLP network mapping; constructing an adaptive prompt word pair, processing the image to be evaluated through an image encoder to obtain an image feature vector, and simultaneously processing the adaptive prompt word pair through a text encoder to obtain a prompt word feature vector; calculating the definition score of the image to be evaluated through a Softmax function and comparing it with a preset threshold to obtain a determination result. The application overcomes the problem that the picture definition evaluation scale is not the same under the condition that the texture quantity and density are different, and makes the evaluation score only depend on the overall definition of the picture, so as to avoid the problem that the definition of the defects at different focal depths of the chip end face image causes the score to be biased and leads to inaccuracy.

Description

Technical Field

[0001] This invention relates to the field of chip appearance inspection technology, and more specifically, to a method and system for evaluating the sharpness of chip end-face images based on semantic information. Background Technology

[0002] As a core component of modern electronic devices, the quality of chips directly affects the stability and reliability of the entire system. Therefore, efficient and accurate chip defect detection plays a crucial role in the manufacturing process. Among numerous inspection technologies, visual inspection, with its high precision, high efficiency, and non-contact characteristics, has become an important means of chip defect detection.

[0003] However, the core of a visual inspection system lies in acquiring high-quality images. Accurate focusing is a prerequisite for capturing minute defects on the chip surface, because any image blurring caused by defocusing will mask defect features, leading to serious missed or false detections. To achieve automated and precise focusing, a sharpness evaluation function plays an indispensable role. It can quantitatively obtain image sharpness information and is widely used in camera autofocus and subsequent defect detection.

[0004] Existing sharpness evaluation functions, such as the Brenner gradient function, Tenengrad gradient function, and Laplacian gradient function, primarily obtain image sharpness scores based on the gradient information of the image. While these functions can meet the needs of most common application scenarios, they reveal significant shortcomings when applied to the evaluation of chip end-face images. Chip end-face images often exhibit the following complex characteristics: ① The size, quantity, and density of defect textures vary across different chip end-face images. ② Different depths of focus exist within the same chip end-face image, leading to varying sharpness in different areas within the same image. Due to these characteristics, traditional sharpness evaluation functions face two major limitations in chip end-face inspection: ① Inconsistent evaluation scale: When evaluating images that are "rich in texture but blurry" versus "sparse in texture but sharp," the former may receive a higher sharpness score than the latter. This reveals the limitation of traditional functions in applying inconsistent evaluation scales to different images, forcing the system to rely on comparing the scores of multiple images to determine relative sharpness, rather than simply judging sharpness based on the score of a single image. ② Susceptible to interference from local high-frequency features: When jagged small defects and blocky large defects appear simultaneously in different areas of a chip end-face image, the traditional evaluation function is easily "biased" by these jagged small defects because the gradient information contained in the jagged defects is richer, thus weakening the impact of the main large defects on the sharpness score. This limitation leads to the inaccuracy of traditional sharpness evaluation functions in chip end-face image evaluation.

[0005] To address the shortcomings of existing defect detection methods and evaluation functions, this invention proposes a chip end-face image sharpness evaluation method based on image semantic information. This method's score depends solely on the overall absolute sharpness of the image. It maintains a consistent evaluation scale even when dealing with images of varying texture richness, thus overcoming the limitations of traditional methods and enabling the direct determination of image sharpness based on a single image. Summary of the Invention

[0006] In view of this, the present invention proposes a method and system for evaluating the sharpness of chip end face images based on semantic information. It utilizes CLIP to extract semantic information from images, overcomes the problem of inconsistent image sharpness evaluation scales when the number and density of textures are different, and ensures that the evaluation score depends only on the overall image sharpness, thereby avoiding the problem of inaccurate scoring caused by the sharpness bias of defects at different depths of focus in chip end face images.

[0007] To achieve the above objectives, this invention proposes a chip end-face image sharpness evaluation method based on semantic information, comprising: A method for evaluating the sharpness of chip end-face images based on semantic information, characterized by comprising: Obtain an image of the chip's end face to be evaluated; The chip end face image is input to the image degradation feature extraction module, the degradation features of the image are extracted and global average pooling is performed, and the guidance vector is obtained by mapping through the MLP network; An adaptive prompt word pair is constructed. The image to be evaluated is processed by an image encoder to obtain an image feature vector. At the same time, the adaptive prompt word pair is processed by a text encoder to obtain a prompt word feature vector. The sharpness score of the image to be evaluated is calculated using the Softmax function; The sharpness score is compared with a preset threshold. If the sharpness score is higher than the threshold, the chip end face image is determined to be sharp; otherwise, it is determined to be blurry.

[0008] Furthermore, the adaptive prompt word pair includes prompt words corresponding to both clear and ambiguous semantic states; The method for constructing the adaptive prompt word pairs includes: The guidance vector is concatenated with the initial prompt word vector corresponding to the clear semantic state and the initial prompt word vector corresponding to the ambiguous semantic state, respectively, to obtain a prompt word pair feature vector that is adapted to the current input image.

[0009] Furthermore, the image degradation feature extraction module is constructed based on the CLIP model image encoder, loads CLIP pre-trained weights and freezes the backbone network, and only optimizes the parameters of the image degradation feature extraction module.

[0010] Furthermore, the preset threshold is determined based on the k-sigma criterion, specifically including: The chip end face is captured in a sequence of out-of-focus images, where the lens captures an image every preset step, and each sequence contains multiple images from severe out-of-focus to optimal focus; Determine the image at the focus position in each sequence, and identify the image at the focus position and its adjacent images before and after it as the sharp frame; Calculate the mean and standard deviation of the sharpness scores corresponding to the sharp frame and its adjacent frames, and calculate the preset threshold based on the mean and standard deviation.

[0011] Furthermore, the acquisition step size of the out-of-focus sequence image is 0.001mm, each sequence contains 20 images, and the sharpness score is first labeled with the image at the in-focus position as the maximum step size K of the sequence, then decreased sequentially towards both ends of the sequence, and finally normalized to the range of 0-1.

[0012] Furthermore, the calculation of the cosine similarity between the image feature vector and the prompt word feature vector includes: Calculate the first cosine similarity between the image feature vector and the feature vector of the clear state prompt word, and the second cosine similarity between the image feature vector and the feature vector of the blurred state prompt word; The first and second cosine similarities are normalized using the Softmax function to obtain the sharpness score.

[0013] Furthermore, during the model training phase, the MSE loss function and SGD optimizer are used, along with a cosine annealing learning rate scheduler with hot restart, and the initial learning rate is set to 0.01.

[0014] On the other hand, to achieve the above objectives, this invention proposes a chip end-face image sharpness evaluation system based on semantic information, comprising: Image acquisition module, used to acquire images of the chip end face to be evaluated; The adaptive prompt word construction module includes an image degradation feature extraction module, a global average pooling layer, and an MLP network. It is used to extract image degradation features and map them into guidance vectors, which are then concatenated with the initial prompt words to construct adaptive prompt word pairs. The feature encoding module includes an image encoder and a text encoder, which are used to encode the features of the image to be evaluated and the adaptive prompt word pair, respectively, to obtain image feature vectors and prompt word feature vectors; The sharpness calculation module is used to calculate the cosine similarity between the image feature vector and the prompt word feature vector, and obtain the sharpness score through the Softmax function; The sharpness determination module is used to compare the sharpness score with a threshold determined based on the k-sigma criterion and output a result indicating whether the image is sharp or blurry.

[0015] Furthermore, the system also includes: The defocus sequence acquisition module is used to acquire defocus sequence images of the chip end face in 0.001mm increments; The threshold calculation module is used to calculate the judgment threshold based on the sharpness score distribution of the in-focus position and its adjacent images in the acquired out-of-focus sequence, using the k-sigma criterion. Compared with the prior art, the beneficial effects of the present invention are as follows: This invention uses the CLIP model to extract semantic information from images, constructs a chip end-face defocus sequence dataset for chip end-face image focusing, and proposes an adaptive cue word method for targeted model training, improving the model's evaluation accuracy. It utilizes image semantic information to determine the sharpness of chip end-face images, unifying the evaluation scale under varying texture quantity, density, and morphology in different chip end-face images. A k-sigma-based criterion for judging image sharpness is proposed, allowing for the determination of image sharpness using a single chip end-face image, minimizing unnecessary movement of mechanical structures and improving evaluation efficiency. It can be widely applied to scenarios involving chip end-face image sharpness evaluation. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the chip end-face image sharpness evaluation method based on semantic information according to the present invention; Figure 2 This is a schematic diagram of the image degradation feature extraction module in an embodiment of the present invention; Figure 3 This is an architecture diagram of a chip end-face image sharpness evaluation model based on image semantic information in an embodiment of the present invention; Figure 4 This is a flowchart of the image sharpness evaluation process in an embodiment of the present invention; Figure 5 This is a performance comparison of various sharpness evaluation methods in chip defect detection scenarios in the embodiments of the present invention. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Because defects on different chip facets vary in number, shape, and density, the texture features of the acquired images differ. Furthermore, the chip background area is extremely flat, leading to the following limitations in traditional sharpness evaluation functions: first, blurry but texture-rich images may actually score higher than sharp but less textured images; second, the overall sharpness score is easily affected by local high-gradient information regions. These limitations make it difficult for traditional methods to accurately assess the absolute sharpness of a single image. Traditional methods typically rely on the camera acquiring multiple images from different focus positions on the same facet and comparing them to obtain relative sharpness. Therefore, this evaluation method becomes ineffective when the number of available images is limited.

[0019] Therefore, this embodiment proposes a chip end-face image sharpness evaluation method based on image semantic information, and proposes an adaptive prompt word method to train the model, thereby reducing the model training cost. Figure 1 This is a schematic diagram of the image sharpness evaluation method. First, the image to be evaluated is acquired, and cue word pairs are set for the network. The image to be evaluated is then fed into the degradation feature extraction module and concatenated with the cue word pairs to construct an adaptive cue word pair. Next, the adaptive cue word pair and the image to be evaluated are fed into the evaluation network to obtain the image feature vector and the adaptive cue word pair feature vector. Finally, the cosine similarity between the image feature vector and the cue word pair feature vector is calculated, and the sharpness score is obtained through Softmax calculation. This evaluation method scores image sharpness based on semantics, avoiding the influence of gradient information at different depths of focus on the score. Furthermore, the evaluation scale is uniform, allowing for objective evaluation of sharpness even with only a single chip end-face image.

[0020] The specific implementation plan is as follows: (1) Model training This embodiment proposes a training method based on adaptive prompts for input images to train the model in a targeted manner.

[0021] The specific training strategy for the model is as follows: Dataset Construction. The lens acquires chip end face images every 0.001mm. For each chip end face, 20 images are acquired to form a defocus sequence. Each sequence covers images from severe defocus to optimal focus. The maximum number of steps from the focus position to the two ends of the corresponding sequence is retrieved in each sequence and denoted as K. The image at the focus position is labeled as K. The label value decreases by 1 from the focus position to the two ends of the sequence. Finally, the values ​​in each sequence are normalized to the range of 0-1 to obtain the sharpness score corresponding to each image, as shown in Equation (1): (1) In the formula, This represents the image's sharpness score. Indicates the number of the current sequence. The annotation values ​​of the images, This represents the minimum value of the image annotation in the current sequence.

[0022] This embodiment further proposes a training method for adaptive prompts based on input images. The input image is processed as follows: Figure 2 The feature extraction module obtains the degradation features of the image. After global average pooling of the image degradation features, it is mapped to 16 guidance vectors through the MLP network. These guidance vectors are concatenated with the initial prompt words processed by the text encoder to complete the construction of adaptive prompt words. In terms of training strategy, the model loads the pre-trained weights of CLIP and freezes the backbone network, optimizing only the image degradation feature extraction module. This mechanism gives the model the ability to dynamically construct adaptive prompt words based on the input image. The adaptive prompt word construction method is as shown in equation (2): (2) In the formula, and These are the tensors formed by concatenating the corresponding prompt words and the guidance vectors.

[0023] Model optimization training. The MSE loss function and SGD optimizer were used. A cosine annealing learning rate scheduler with hot restart was employed, with an initial learning rate of 0.01.

[0024] (2) Evaluation of image sharpness on chip end face The specific steps for evaluating the clarity of the model are as follows: Figure 3 As shown: The image of the chip's end face captured by the camera is processed by the model's image encoder to obtain the corresponding feature vector. After passing through the image degradation feature extraction module, an adaptive guidance vector can be obtained; similarly, the prompt word pairs will also obtain corresponding feature vectors after passing through the text encoder, which are then concatenated with the adaptive guidance vector to obtain... and To avoid the ambiguity of language affecting the clarity evaluation, the feature vector of the image will be used to calculate the cosine similarity with the feature vector of the pair of antonyms, as shown in equation (3): (3) In the formula, and The cosine similarity between the feature vectors corresponding to the images and the feature vectors corresponding to the prompt words in the feature space of CLIP.

[0025] and The image sharpness score is obtained after Softmax calculation, as shown in equation (4): (4) In the formula, The sharpness score predicted by the model.

[0026] The chip end-face image sharpness evaluation is implemented because the camera's movement step size is 0.001mm during chip end-face image acquisition. The difference between two adjacent images of the in-focus image is almost imperceptible to the naked eye and can be considered sufficiently sharp. Furthermore, the distribution of sharp frames and their adjacent frames in different sequences calculated based on equation (4) can approximate a normal distribution. Therefore, it can be based on... k - The sigma criterion sets a threshold for the score. Images with a sharpness score higher than this threshold are considered sharp. The classification method is shown in equation (5): (5) In the formula, k Take 2, , , These are the sharpness scores for the sharp frame and the frames immediately before and after it in the sequence, respectively. T The threshold value is used.

[0027] The specific implementation process of this embodiment is as follows: Figure 4 As shown: First, CLIP is trained in the scenario of chip end face image defocus sequence using the adaptive prompt word method, and the threshold is calculated according to Equation (5), where the calculated threshold is 0.7698.

[0028] Then, the image to be evaluated is fed into the model, and the cosine similarity between the image and the prompt word pair is obtained according to Equation (3), and the clarity score is obtained according to Equation (4).

[0029] Finally, the score is compared with a threshold. If it is higher than the threshold, the image is considered clear enough in this scenario; if it is lower than the threshold, the image is blurry.

[0030] The existing sharpness evaluation function and this method were used to score 73 sets of chip-side defocused image sequences. The linear correlation (PLCC, the closer to 1, the better) and ranking correlation (SROCC, the closer to 1, the better) between the scores of each method and the image annotation values ​​were tested. Figure 5 As shown in the figure. The results indicate that the proposed method has a uniform scale and strong generalization ability when evaluating different chip defocus sequences.

[0031] In summary, this embodiment addresses the problem of inconsistent evaluation scales in existing sharpness evaluation methods when the number, density, and morphology of textures in different chip end-face images vary, and the problem that traditional methods rely on multiple images to determine the sharpness of a single image due to these inconsistent evaluation scales. It proposes a sharpness evaluation method for chip end-face images based on image semantic information. The main technical innovations include the following: (1) A chip end face image sharpness evaluation method based on image semantic information is proposed, which avoids the limitation of inconsistent evaluation scales in this field of traditional methods.

[0032] (2) An adaptive prompt word training method was proposed, which enables the model to dynamically adjust the prompt words according to the input image, thereby enhancing the model's generalization ability when evaluating chip end face images.

[0033] (3) A proposal based on k The sigma criterion for judging image sharpness avoids the problem of existing evaluation methods that require collecting multiple images to determine image sharpness even when the initially acquired images are sufficiently clear. It also avoids unnecessary movement of mechanical structures, improving evaluation efficiency and extending the lifespan of mechanical structures.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for evaluating the sharpness of chip end-face images based on semantic information, characterized in that, include: Obtain an image of the chip's end face to be evaluated; The chip end face image is input to the image degradation feature extraction module, the degradation features of the image are extracted and global average pooling is performed, and the guidance vector is obtained by mapping through the MLP network; An adaptive prompt word pair is constructed. The image to be evaluated is processed by an image encoder to obtain an image feature vector. At the same time, the adaptive prompt word pair is processed by a text encoder to obtain a prompt word feature vector. The sharpness score of the image to be evaluated is calculated using the Softmax function; The sharpness score is compared with a preset threshold. If the sharpness score is higher than the threshold, the chip end face image is determined to be sharp; otherwise, it is determined to be blurry.

2. The method according to claim 1, characterized in that, The adaptive prompt word pair includes prompt words corresponding to two semantic states: clear and ambiguous. The method for constructing the adaptive prompt word pairs includes: The guidance vector is concatenated with the initial prompt word vector corresponding to the clear semantic state and the initial prompt word vector corresponding to the ambiguous semantic state, respectively, to obtain a prompt word pair feature vector that is adapted to the current input image.

3. The method according to claim 1, characterized in that, The image degradation feature extraction module is built on the CLIP model image encoder, loads CLIP pre-trained weights and freezes the backbone network, and only optimizes the parameters of the image degradation feature extraction module.

4. The method according to claim 1, characterized in that, The preset threshold is determined based on the k-sigma criterion, specifically including: The chip end face is captured in a sequence of out-of-focus images, where the lens captures an image every preset step, and each sequence contains multiple images from severe out-of-focus to optimal focus; Determine the image at the focus position in each sequence, and identify the image at the focus position and its adjacent images before and after it as the sharp frame; Calculate the mean and standard deviation of the sharpness scores corresponding to the sharp frame and its adjacent frames, and calculate the preset threshold based on the mean and standard deviation.

5. The method according to claim 4, characterized in that, The acquisition step size of the out-of-focus sequence images is 0.001mm, and each sequence contains 20 images. When labeling, the sharpness score is first labeled with the image at the in-focus position as the maximum step size K of the sequence, and then decreased sequentially towards both ends of the sequence, and finally normalized to the range of 0-1.

6. The method according to claim 1, characterized in that, The calculation of the cosine similarity between the image feature vector and the prompt word feature vector includes: Calculate the first cosine similarity between the image feature vector and the feature vector of the clear state prompt word, and the second cosine similarity between the image feature vector and the feature vector of the blurred state prompt word; The first and second cosine similarities are normalized using the Softmax function to obtain the sharpness score.

7. The method according to claim 1, characterized in that, During the model training phase, the MSE loss function and SGD optimizer were used, along with a cosine annealing learning rate scheduler with hot restart. The initial learning rate was set to 0.

01.

8. A chip end-face image sharpness evaluation system based on semantic information, characterized in that, include: Image acquisition module, used to acquire images of the chip end face to be evaluated; The adaptive prompt word construction module includes an image degradation feature extraction module, a global average pooling layer, and an MLP network. It is used to extract image degradation features and map them into guidance vectors, which are then concatenated with the initial prompt words to construct adaptive prompt word pairs. The feature encoding module includes an image encoder and a text encoder, which are used to encode the features of the image to be evaluated and the adaptive prompt word pair, respectively, to obtain image feature vectors and prompt word feature vectors; The sharpness calculation module is used to calculate the cosine similarity between the image feature vector and the prompt word feature vector, and obtain the sharpness score through the Softmax function; The sharpness determination module is used to compare the sharpness score with a threshold determined based on the k-sigma criterion and output a result indicating whether the image is sharp or blurry.

9. The system according to claim 8, characterized in that, Also includes: The defocus sequence acquisition module is used to acquire defocus sequence images of the chip end face in 0.001mm increments; The threshold calculation module is used to calculate the judgment threshold based on the sharpness score distribution of the in-focus position and its adjacent images in the acquired out-of-focus sequence, using the k-sigma criterion.

Images