Image stitching method, evaluation method and system

Abstract

The application relates to the field of semiconductor technology, in particular to an image splicing method, an evaluation method and a system. The application comprises the following steps: inputting an original image and a semantic feature map into a first image repairing network; calculating an image quality index of a repaired image by using a multi-scale structural similarity, fusing a channel hardware confidence and the image quality index, and obtaining a final splicing weight corresponding to the repaired image; taking the repaired image corresponding to the highest splicing weight as a reference image, registering the remaining repaired images to a reference image coordinate system corresponding to the reference image through affine transformation, taking the reference image as the basis, adopting a weighted fusion algorithm to perform initial splicing on the repaired images, and obtaining an initial splicing image; and inputting the initial splicing image and the final splicing weight into a second image repairing network for smoothing processing, and obtaining a final splicing image. The application realizes fast splicing and high-quality output of images.

Description

Technical Field

[0001] This application relates to the field of semiconductor technology, and in particular to image stitching methods, evaluation methods and systems. Background Technology

[0002] When performing CIS (CMOS Image Sensor) chip testing, it is necessary to acquire the original image of the CIS chip and perform image restoration and stitching processing on the original image. However, existing image stitching methods usually focus on deep learning networks to perform computational operations. Such computational operations require manual parameter tuning and lack an intelligent optimization control system, which reduces the efficiency of image stitching computation and fails to obtain high-quality stitched images. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose an image stitching method, evaluation method and system to solve the above-mentioned technical problems.

[0004] To achieve the above objectives, this application provides an image stitching method, comprising:

[0005] During the wafer testing phase, the link layer information of multiple wafer testing channels is acquired in real time using a field-programmable gate array (FPGA). Key parameters in the link layer information are obtained, and the key parameters are normalized to obtain the hardware confidence score of each wafer test channel. The hardware confidence score is used as the prior weight for image stitching. The original images transmitted in each of the wafer test channels are acquired, and each original image and its corresponding prior weight are input into a lightweight semantic segmentation network as input data. The lightweight semantic segmentation network is used to infer the original images and obtain semantic feature maps. The original image and the semantic feature map are input into the first image inpainting network, and the inpainted image of the original image is obtained. The image quality index of the repaired image is calculated using multi-scale structural similarity. The channel hardware confidence is fused with the image quality index to obtain the final stitching weight corresponding to the repaired image. The highest stitching weight in the final stitching weight is then calculated. Using the repaired image corresponding to the highest stitching weight as the reference image, the remaining repaired images are registered to the reference image coordinate system corresponding to the reference image through affine transformation. Based on the reference image, a weighted fusion algorithm is used to initially stitch the repaired images to obtain an initial stitched image. The initial stitched image and the final stitching weights are input into a second image inpainting network for smoothing to obtain the final stitched image. The final stitching weights are then dynamically adjusted based on the quality of the final stitched image.

[0006] Optionally, obtaining key parameters from the link layer information and normalizing these key parameters to obtain the hardware confidence level of each wafer test channel includes: Obtain key parameters from the link layer information, including cyclic redundancy check error rate, number of data packet retransmissions, and actual signal eye diagram margin.

[0007] Optionally, the normalization process for the key parameters to obtain the hardware confidence level of each wafer test channel includes: The key parameters are normalized by using polarity alignment and weighted fusion to obtain the hardware confidence level of each wafer test channel; A maximum expected error rate threshold is set. When the cyclic redundancy check error rate exceeds the maximum expected error rate threshold, the data in the wafer test channel is severely corrupted. Set a threshold for the maximum expected number of retransmissions. The threshold is the maximum allowed number of retries. If the transmission still fails after exceeding the threshold, the packet will be treated as lost. Set a standard reference signal eye diagram margin. When the actual signal eye diagram margin is not less than the standard reference signal eye diagram margin, the signal integrity is in an ideal state. Preset empirical weight coefficients are dynamically determined based on prior statistics and multiple regression analysis. These preset empirical weight coefficients satisfy the normalization condition, and the specific formula is as follows: ; in, For hardware confidence, , A higher value indicates a higher level of confidence in the hardware. , , To preset the empirical weighting coefficients, Cyclic Redundancy Check (CR) error rate, The threshold for the expected maximum error rate. This refers to the number of data packet retransmissions. The threshold for the expected maximum number of retransmissions. This represents the actual signal eye diagram margin. This is the eye diagram margin for the standard reference signal.

[0008] Optionally, the step of dynamically determining preset empirical weight coefficients based on prior statistics and multiple regression analysis, wherein the preset empirical weight coefficients satisfy normalization conditions, includes: During the calibration phase, the cyclic redundancy check error rate, the number of data packet retransmissions, the actual signal eye diagram margin, and the corresponding image quality indicators transmitted in each wafer test channel are obtained. With image restoration quality as the optimization objective, the correlation coefficients of the cyclic redundancy check error rate, the number of data packet retransmissions, and the actual signal eye diagram margin are fitted using the least squares method and normalized to obtain the optimal combination of empirical weight coefficients. The optimal combination of empirical weight coefficients is used as the preset empirical weight coefficients, and the prior statistics and multiple regression analysis are used to dynamically determine them.

[0009] Optionally, inputting the original image and the semantic feature map into the first image inpainting network includes: The original image and the semantic feature map are input into a first image restoration network, which is a first U-shaped convolutional neural network (U-Net). The restored image of the original image is obtained by calculating the loss function of the first U-shaped convolutional neural network. The step of inputting the initial stitched image and the final stitched weights into the second image inpainting network for smoothing includes: The initial stitched image and the final stitched weights are input into a second image restoration network for smoothing. The second image restoration network is a second U-shaped convolutional neural network. The restored image is smoothed by calculating the loss function of the second U-shaped convolutional neural network.

[0010] Optionally, obtaining the repaired image of the original image by calculating the loss function of the first U-shaped convolutional neural repair network includes: The repaired image of the original image is obtained by calculating the multi-scale structural similarity loss and norm loss of the first U-shaped convolutional neural repair network. The smoothing process of the repaired image by calculating the loss function of the second U-shaped convolutional neural repair network includes: The repaired image is smoothed by calculating the multi-scale structural similarity loss and norm loss of the second U-shaped convolutional neural repair network. The loss function for both the first U-shaped convolutional neural repair network and the second U-shaped convolutional neural repair network is: ; in, For the total loss, This represents the multi-scale structural similarity loss, focusing on structural edges; Indicates norm loss, focusing on pixel brightness; The balancing weight coefficient is determined experimentally and statistically in the early stages of model training using hyperparameter optimization methods.

[0011] Optionally, obtaining the repaired image of the original image by calculating the loss function of the first U-shaped convolutional neural network, and smoothing the repaired image by calculating the loss function of the second U-shaped convolutional neural network, includes: Adjust the adaptive parameters of the first U-shaped convolutional neural repair network and / or the second U-shaped convolutional neural repair network. When the repaired image has clear edges but a significant shift in overall brightness, increase the parameters of the first U-shaped convolutional neural repair network and / or the second U-shaped convolutional neural repair network. When the repaired image has accurate colors but blurry texture details, the parameters of the first U-shaped convolutional neural network and / or the second U-shaped convolutional neural network are lowered to maintain the parameter balance of the first U-shaped convolutional neural network and / or the second U-shaped convolutional neural network for mass production.

[0012] Optionally, the process involves obtaining the repaired image of the original image by calculating the loss function of the first U-shaped convolutional neural network, and smoothing the repaired image by calculating the loss function of the second U-shaped convolutional neural network, wherein... Gradient descent is used to update the encoder, bottleneck layer, and decoder in both the first and second U-shaped convolutional neural repair networks.

[0013] Based on the same inventive concept, this application provides an evaluation method for evaluating the impact of the image stitching method described above on the probe station throughput, including: During the wafer testing phase, the current processing frame rate and the target test frame rate are obtained; The current processing frame rate and the target test frame rate are normalized, and the test throughput is calculated: By weighted fusion of the test throughput and image quality indicators, a comprehensive evaluation index is obtained to assess the impact of the image stitching method on the probe station throughput. The comprehensive evaluation index is compared with the adaptive threshold. If the comprehensive evaluation index is lower than the adaptive threshold, adaptive adjustment is triggered, and a high-quality stitched image is output after adjustment.

[0014] Based on the same inventive concept, this application also provides an image stitching system, comprising: The link layer information acquisition module is used to acquire link layer information of multiple wafer test channels in real time using a field-programmable gate array during the wafer testing phase. The hardware confidence configuration module is used to obtain key parameters in the link layer information, normalize the key parameters to obtain the hardware confidence of each wafer test channel, and use the hardware confidence as the prior weight for image stitching. The semantic feature map setting module is used to acquire the original images transmitted in each of the wafer test channels, input each original image and its corresponding prior weight as input data into the lightweight semantic segmentation network, and use the lightweight semantic segmentation network to infer the original images and obtain semantic feature maps. The first image restoration module is used to input the original image and the semantic feature map into the first image restoration network, and restore the original image to obtain a restored image. The stitching weight configuration module is used to calculate the image quality index of the repaired image using multi-scale structural similarity, fuse the channel hardware confidence with the image quality index to obtain the final stitching weight corresponding to the repaired image, and calculate the highest stitching weight among the final stitching weights. The initial stitching image setting module is used to take the repaired image corresponding to the highest stitching weight as the reference image, register the remaining repaired images to the reference image coordinate system corresponding to the reference image through affine transformation, and then use the reference image as the basis to perform initial stitching of the repaired images using a weighted fusion algorithm to obtain the initial stitching image. The second image restoration module is used to input the initial stitched image and the final stitching weight into the second image restoration network for smoothing processing to obtain the final stitched image; The stitching weight configuration module is also used to dynamically adjust the final stitching weight based on the quality of the final stitched image.

[0015] As can be seen from the above, the image stitching method, evaluation method and system provided in this application fuse channel hardware confidence with image quality indicators to obtain the final stitching weight and the highest stitching weight corresponding to the repaired image, ensuring that image stitching always takes the most reliable channel as the main factor. Then, the overlapping areas of the images to be stitched are smoothed multiple times. By using a lightweight semantic segmentation network, the original image is repaired separately first, and the repaired image is smoothed afterward, which completely eliminates the seams in the image stitching, thereby improving the image stitching effect and realizing fast image stitching and high-quality output. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of an image stitching method provided in an embodiment of this application; Figure 2 A flowchart of an image stitching method provided in another embodiment of this application; Figure 3 A flowchart of an image evaluation method provided in another embodiment of this application; Figure 4 This is a block diagram of an image stitching system provided in an embodiment of this application; Figure 5 This is a flowchart of an image stitching system processing according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the component or object preceding the word covers the components or objects listed after the word and their equivalents, without excluding other components or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] It should be noted that the method in this embodiment can be executed by a single device, such as a controller or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the process. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method.

[0021] When performing CIS chip testing, it is necessary to acquire the original image of the CIS chip and perform image restoration and stitching processing on the original image. However, existing image stitching methods based on global and local features usually focus on deep learning networks, using deep learning to achieve feature matching and projection fusion, and adopting a unidirectional stitching method, lacking a mechanism for verifying and providing feedback on the integrity of the stitching results.

[0022] Meanwhile, existing image stitching methods lack an adaptive closed-loop structure and require manual parameter adjustment, which leads to severe discontinuities in the image stitching seams. Furthermore, during image stitching, the need to manually adjust parameters during the optimization model process results in an over-reliance on manual parameter tuning and a lack of intelligent optimization. A large amount of data is required for label training, which leads to poor computational efficiency and accuracy.

[0023] Therefore, this application provides an image stitching method to solve the above problems.

[0024] Reference Figure 1 This application provides an image stitching method, including the following steps: Step S100: During the wafer testing phase, the link layer information of multiple wafer testing channels is acquired in real time using a field-programmable gate array (FPGA). Step S200: Obtain key parameters from the link layer information, normalize the key parameters to obtain the hardware confidence level of each wafer test channel, and use the hardware confidence level as the prior weight for image stitching. Step S300: Obtain the original images transmitted in each wafer test channel, and input each original image and its corresponding prior weight as input data into the lightweight semantic segmentation network. Use the lightweight semantic segmentation network to reason about the original images and obtain semantic feature maps. Step S400: Input the original image and semantic feature map into the first image inpainting network, and obtain the inpainted image of the original image; Step S500: Calculate the image quality index of the repaired image using multi-scale structural similarity, fuse the channel hardware confidence with the image quality index to obtain the final stitching weight corresponding to the repaired image, and calculate the highest stitching weight in the final stitching weight. Step S600: Using the repaired image corresponding to the highest stitching weight as the reference image, the remaining repaired images are registered to the reference image coordinate system corresponding to the reference image through affine transformation. Based on the reference image, the repaired images are initially stitched using a weighted fusion algorithm to obtain the initial stitched image. Step S700: Input the initial stitched image and the final stitching weights into the second image inpainting network for smoothing to obtain the final stitched image, and dynamically adjust the final stitching weights according to the quality of the final stitched image.

[0025] In this embodiment, the integrity of image data in image stitching is improved by repairing the original image. Specifically, this embodiment obtains the final stitching weight and the highest stitching weight corresponding to the repaired image by fusing channel hardware confidence with image quality indicators, ensuring that image stitching always prioritizes the most reliable channel. Subsequently, the overlapping areas of the images to be stitched are smoothed multiple times using MobileNet V3 (Mobile Network Version 3, a lightweight semantic segmentation network). By first repairing the original image separately and then smoothing the subsequent repaired images, seams in the image stitching are completely eliminated, thereby improving the stitching effect and achieving fast image stitching and high-quality output.

[0026] Before wafer testing, a probe station is used to support the wafer under test so that the probes on the probe station can make parallel contact with multiple chips.

[0027] In step S400, after inputting the original image and semantic feature map into the first image inpainting network, the following steps are included: Step S410: In the image encoding stage, the first image inpainting network introduces a semantic attention gating mechanism and adopts a differentiated inpainting strategy for different semantic regions in the semantic feature map to obtain the inpainted image of the original image. Specifically, in this embodiment, at the initial input of the first image restoration network, the original image is used as the main input image for preliminary feature extraction. The semantic feature map of the lightweight semantic segmentation network is used as the gating signal in the semantic attention gating mechanism to modulate the semantic feature maps of each layer of the encoder of the first image restoration network. The semantic feature map extracted by the lightweight semantic segmentation network is used as the guiding signal. In each downsampling stage of the encoder of the first image restoration network, the semantic feature maps of the corresponding scale are concatenated in the channel dimension through 1x1 convolution. The concatenated fused features are input into the space and the channel attention module, i.e. the semantic attention gating mechanism, to calculate and generate an adaptive weight mask. The weight mask is then used to multiply the main semantic feature map of the first image restoration network, thereby realizing the spatial modulation of the original image. Here, the semantic feature map contains full semantic information.

[0028] Furthermore, the differentiated repair strategy in this embodiment refers to the fact that, during the wafer testing stage, the content of the original image transmitted in each wafer testing channel is usually composed of high-precision circuit array regions and relatively flat non-circuit regions. The lightweight semantic segmentation network will assign different mathematical processing logic to these two types of regions according to the guiding signal. For the circuit array regions with regularity in the original image, the repair strategy tends to protect the integrity of its structure and prevent excessive blurring. For large-area random missing regions caused by hardware signal attenuation or packet loss, the repair strategy will use the surrounding context information to focus on large-scale pixel interpolation and noise reduction smoothing.

[0029] Meanwhile, the restored image of the original image obtained in this embodiment is a high-fidelity original image after pixel-level restoration of each channel. Here, the high-fidelity original image eliminates the damage caused by the underlying hardware transmission and enhances the key semantic features in the image, preparing for obtaining the initial stitched image in step S600.

[0030] In step S500, the image quality index of the repaired image is calculated using MS-SSIM (Multi-Scale Structural Similarity Index), including the following steps: Step S501: Utilize the three dimensions of brightness, contrast, and structure in multi-scale structural similarity to compare and repair the similarity between the image and the reference image. Calculate the three component indices at each computational scale, including: (1) Brightness similarity It measures the similarity in average brightness between two images, and the calculation formula is: ; (2) Contrast similarity It measures the similarity in the standard deviation of pixel value distribution between two images, and the calculation formula is: ; (3) Structural similarity It measures the correlation between two graphs, and the calculation formula is: ; Among them, in the above formula This represents the mean. and Indicates standard deviation, Describing covariance, , , This represents a small constant set to avoid the denominator being zero.

[0031] Step S502: Combine multiple spatial scales to perform a weighted evaluation of similarity and calculate a comprehensive score as the image quality index for the restored image. To capture image quality loss at different spatial frequencies, the system performs M iterations of downsampling on the original image, constructs an image pyramid, and calculates... The calculation formula is: ; in, , , These represent the weighting coefficients for luminance, contrast, and structure components at different scales. Luminance similarity is only considered at the coarsest scale. The calculation is performed at the M scales, while contrast and structural similarity are accumulated across all M scales.

[0032] Step S503: Set the channel hardware confidence level Image quality metrics The images are fused together to obtain the final stitching weights corresponding to the restored images, at the coarsest scale. The highest splicing weight in the final splicing weight is calculated below. The calculation formula is: .

[0033] In step S600, during the initial stitching of the repaired image using a weighted fusion algorithm, the pixel values ​​of the overlapping areas in the initial stitching are linearly mixed according to the final stitching weights to achieve a smooth transition in the overlapping areas. This specifically includes the following steps: Step S601: During the initial stitching, the stitching overlap area of ​​the initial stitching consists of the first repaired image and the second repaired image; The final stitching weight of the first restored image corresponds to the first pixel value of the stitching overlap area; The final stitching weight of the second repaired image corresponds to the second pixel value of the stitching overlap area.

[0034] Step S602: Linearly blend the first pixel value and the second pixel value to obtain the initial stitched image.

[0035] Specifically, in this embodiment, when performing image stitching, there will be an overlapping area between the first repaired image and the second repaired image due to overlap and redundancy. Here, the overlapping area is not simply the first repaired image superimposed on the second repaired image. Therefore, it is necessary to mix the pixels of the repaired images into a whole pixel value according to the final stitching weight ratio in order to improve the image stitching quality.

[0036] Furthermore, in the initial stitched image obtained in step S600, although the initial stitched image achieves a smooth transition on a macroscopic scale, there may still be minor nonlinear faults in the overlapping area. Therefore, the initial stitched image and the final stitching weights are input into the second image inpainting network for pixel-level smoothing and refining.

[0037] In step S700, the final stitched image It is a scalar. Before inputting the initial stitched image and the final stitched weights into the second image inpainting network for smoothing, this embodiment uses the final stitched image... Perform spatial mapping to generate a weight map with the same size as the image, and then stitch the final image together. Fill the corresponding pixel area to form a weight distribution map with the same spatial resolution as the initial stitched image. Then, combine the initial stitched image (e.g., a 3-channel initial stitched image) and the weight distribution map (e.g., a 1-channel initial stitched image) to synthesize an input tensor (e.g., a 4-channel output tensor). This input tensor is the input value of the second image inpainting network.

[0038] Here, in this embodiment, the scalar weight score of each wafer test channel is pre-calculated and mapped along the spatial dimension. The result is filled into the field of view corresponding to the total image coordinate system to generate a global weight distribution map with the same size as the initial stitched image. Then, the weight distribution map and the initial stitched image are fused together and input into the second image restoration network.

[0039] Furthermore, this embodiment utilizes the regional quality differences reflected by the weight distribution map to guide the model to automatically identify and focus on automatically repairing the connection gaps between images, resulting in a final stitched image. Transform it into a spatial weight map with the same size as the image to make the final stitched image... It can be input into a second image restoration network to achieve pixel-level smoothing and retouching of the original image.

[0040] In some embodiments, refer to Figure 2 In step S200, key parameters from the link layer information are obtained, and the key parameters are normalized to obtain the hardware confidence level of each wafer test channel, including: Step S201: Obtain key parameters from the link layer information. Key parameters include cyclic redundancy check error rate, number of data packet retransmissions, and actual signal eye diagram margin.

[0041] In this embodiment, when performing image stitching operations, the error rate based on cyclic redundancy check is used. Number of data packet retransmissions and actual signal eye diagram margin Key parameters are evaluated to assess the information transmission quality at the link layer. Subsequently, the key parameters that meet the evaluation criteria are normalized to facilitate unified calculations for subsequent image stitching.

[0042] In some embodiments, in step S202, the key parameters are normalized to obtain the hardware confidence level of each wafer test channel, referring to... Figure 2 This includes the following steps: Step S2021: Using polarity alignment and weighted fusion, the key parameters are normalized to obtain the hardware confidence level of each wafer test channel; Step S2022: Set the expected maximum error rate threshold. When the cyclic redundancy check error rate exceeds the expected maximum error rate threshold, the data in the wafer test channel is severely damaged. Step S2023: Set the expected maximum retransmission threshold. The expected maximum retransmission threshold is the maximum allowed number of retries. If the transmission still fails after exceeding the expected maximum retransmission threshold, packet loss will be processed. Step S2024: Set the standard reference signal eye diagram margin. When the actual signal eye diagram margin is not less than the standard reference signal eye diagram margin, the signal integrity is in an ideal state. Step S2025: Dynamically determine the preset empirical weight coefficients based on prior statistics and multiple regression analysis. The preset empirical weight coefficients satisfy the normalization condition, and the specific formula is as follows: ; in, For hardware confidence, , A higher value indicates a higher level of hardware confidence. , , To preset the empirical weighting coefficients, Cyclic Redundancy Check (CR) error rate, The threshold for the expected maximum error rate. This refers to the number of data packet retransmissions. The threshold for the expected maximum number of retransmissions. This represents the actual signal eye diagram margin. This is the eye diagram margin for the standard reference signal.

[0043] Meanwhile, this embodiment adopts a polarity alignment and weighted fusion approach to calculate the hardware confidence level based on the weight of each indicator's impact on channel quality. The hardware confidence level is then used to accurately and objectively evaluate the transmission quality of each wafer test channel, thereby obtaining the optimal wafer test channel and high-quality link layer information.

[0044] Furthermore, in this embodiment, the cyclic redundancy check error rate is analyzed. Number of data packet retransmissions and actual signal eye diagram margin Evaluation criteria are set to achieve quality control of hardware confidence, so that higher hardware confidence is more in line with actual computing scenarios. By weighting the calculation of key parameters, the accuracy of hardware confidence assessment under complex working conditions is significantly enhanced.

[0045] In some embodiments, in step S2025, preset empirical weight coefficients are dynamically determined based on prior statistics and multiple regression analysis. The preset empirical weight coefficients satisfy the normalization condition, including the following steps: Step S20251: During the calibration phase, obtain the cyclic redundancy check error rate, data packet retransmission count, actual signal eye diagram margin, and corresponding image quality indicators transmitted in each wafer test channel. Step S20252: Taking image restoration quality as the optimization objective, the least squares method is used to fit the correlation coefficients of cyclic redundancy check error rate, data packet retransmission count and actual signal eye diagram margin and normalize them to obtain the optimal combination of empirical weight coefficients. Step S20253: The optimal empirical weight coefficient combination is used as the preset empirical weight coefficient, and then dynamically determined by prior statistics and multiple regression analysis.

[0046] In this embodiment, the relationship between wafer data and actual image quality is established. To further improve image restoration quality, the cyclic redundancy check error rate is fitted using the least squares method. Number of data packet retransmissions Actual signal eye diagram margin Based on this, the preset empirical weight coefficients are further dynamically determined through prior statistics and multiple regression to objectively reflect the actual impact of hardware status on the final image quality, thereby enhancing the accuracy of hardware confidence assessment.

[0047] In some embodiments, in step S400, the original image and semantic feature map are input into the first image inpainting network, including: The original image and semantic feature map are input into the first image restoration network, which is a first U-shaped convolutional neural network. The restored image of the original image is obtained by calculating the loss function of the first U-shaped convolutional neural network. In step S700, the initial stitched image and the final stitched weights are input into the second image inpainting network for smoothing, including: The initial stitched image and the final stitched weights are input into the second image restoration network for smoothing. The second image restoration network is a second U-shaped convolutional neural network. The restoration image is smoothed by calculating the loss function of the second U-shaped convolutional neural network.

[0048] In this embodiment, by using a first U-shaped convolutional neural repair network and a second U-shaped convolutional neural repair network in stages, the original image and the repaired image are processed in a targeted manner, which further improves the repair accuracy and smoothness of the original image, ensures the repair quality of the original image, achieves smooth image stitching, and significantly improves the repair effect of the original image.

[0049] In some embodiments, step S400 involves obtaining the repaired image of the original image by calculating the loss function of the first U-shaped convolutional neural repair network, including: The repaired image of the original image is obtained by calculating the multi-scale structural similarity loss and norm loss of the first U-shaped convolutional neural repair network. In step S700, the repaired image is smoothed by calculating the loss function of the second U-shaped convolutional neural repair network, including: The repaired image is smoothed by calculating the multi-scale structural similarity loss and norm loss of the second U-shaped convolutional neural repair network. The loss function for either the first or second U-shaped convolutional neural repair network is: ; in, For the total loss, This represents the multi-scale structural similarity loss, focusing on structural edges; Indicates norm loss, focusing on pixel brightness; The balancing weight coefficient is determined experimentally and statistically in the early stages of model training using hyperparameter optimization methods.

[0050] Optionally, the first image restoration network and the second image restoration network in this embodiment have different unfolding structures and parameter configurations.

[0051] In this embodiment, a joint loss function based on multi-scale structural similarity loss and norm loss is set to repair pixels lost during transmission. Here, multi-scale structural similarity loss can improve the image edge sharpness, while norm loss can ensure the high fidelity of the brightness and color of pixels in the image. By reasonably adjusting the weights of multi-scale structural similarity loss and norm loss, the sharpness of the stitched image can be significantly improved and repaired.

[0052] In some embodiments, step S400 involves inputting the original image and semantic feature map into a first image inpainting network, and inputting the initial stitched image and final stitching weights into a second image inpainting network for smoothing processing, including: Adjust the adaptive parameters of the first U-shaped convolutional neural network and / or the second U-shaped convolutional neural network. When the repaired image has clear edges but a significant shift in overall brightness, increase the parameters of the first U-shaped convolutional neural network and / or the second U-shaped convolutional neural network. When the repaired image has accurate colors but blurry texture details, decrease the parameters of the first U-shaped convolutional neural network and / or the second U-shaped convolutional neural network to maintain parameter balance and use it for mass production.

[0053] In this embodiment, the joint loss function is set inside the first U-shaped convolutional neural repair network and the second U-shaped convolutional neural repair network, that is, it is set in the offline preparation stage or the adaptive fine-tuning stage in the closed loop of the test system, and it is a structure that belongs to the first U-shaped convolutional neural repair network and the second U-shaped convolutional neural repair network.

[0054] In some embodiments, in step S400, the repaired image of the original image is obtained by calculating the loss function of the first U-shaped convolutional neural repair network, and the repaired image is smoothed by calculating the loss function of the second U-shaped convolutional neural repair network.

[0055] In the first and second U-shaped convolutional neural repair networks, gradient descent is used to update the encoder, bottleneck layer, and decoder.

[0056] Here, Table 1 in this embodiment shows the unfolded structure and parameter configuration of the first U-shaped convolutional neural repair network and the second U-shaped convolutional neural repair network, Table 2 shows the unfolded structure and parameter configuration of the first U-shaped convolutional neural repair network, and Table 3 shows the unfolded structure and parameter configuration of the second U-shaped convolutional neural repair network. Here, the parameter configuration of the first U-shaped convolutional neural repair network and the second U-shaped convolutional neural repair network uses channel multiplication and residuals, and the block structure uses the mature basic framework of the first U-shaped convolutional neural repair network and the second U-shaped convolutional neural repair network.

[0057] Table 1

[0058] Table 2

[0059] Table 3

[0060] Here, the number of input layer channels differs between the first image inpainting network and the second image inpainting network in this embodiment.

[0061] The first image inpainting network's input layer consists of pixel data channels from the original image and feature map channels extracted and aligned by a lightweight semantic segmentation network. The second image inpainting network's input layer consists of three channels of an RGB image and a channel weight map.

[0062] Furthermore, the attention mechanisms of the first and second image inpainting networks in this embodiment have different positions and functions.

[0063] In the first image inpainting network, during the encoder's downsampling stage, a mask generated from semantic features is used to multiply the feature map, aiming to distinguish between circuit areas and background areas. The second image inpainting network uses the input weight map to adaptively adjust the response intensity of the convolutional kernel at the physical seam location, performing local optimization for the seam.

[0064] Based on the same inventive concept, this application provides an evaluation method for assessing the impact of the above image stitching method on probe station throughput, referring to... Figure 3 This includes the following steps: Step S801: During the wafer testing phase, obtain the current processing frame rate and the target test frame rate; Step S802: Normalize the current processing frame rate and the target test frame rate, and calculate the test throughput: Step S803: Weighted fusion test throughput and image quality index to obtain a comprehensive evaluation index for evaluating the impact of image stitching method on probe station throughput; Step S804: Compare the comprehensive evaluation index with the adaptive threshold. If the comprehensive evaluation index is lower than the adaptive threshold, trigger adaptive adjustment and output a high-quality stitched image after adjustment.

[0065] In this embodiment, the actual impact of the image stitching algorithm on the wafer testing efficiency on the probe station is calculated to avoid reducing wafer testing efficiency due to probe station testing lag, thereby ensuring the high-quality application of the image stitching method in actual industrial production.

[0066] Furthermore, in this embodiment, the target frame rate for testing... The benchmark threshold is set based on the actual production capacity requirements of wafer inspection and the mechanical stepping cycle of the probe station. On this basis, the test system counts the current processing frame rate in real time. The test throughput was calculated using a ratio normalization method. The specific formula is as follows: ; Then, the throughput will be tested. Image quality metrics The two indicators are weighted and fused to obtain the comprehensive evaluation index S. The specific calculation formula is as follows: ; in, and Let represent the weighting coefficients of the image quality metric and the test throughput, respectively, and satisfy . It can be adjusted according to the focus of the current testing phase.

[0067] Furthermore, in this embodiment, the FAR (False Alarm Rate) generated by stitched images under different comprehensive evaluation indicators S in the historical sample dataset after being input into the defect detection algorithm is statistically analyzed. This FAR represents the probability that overlapping stitched parts are mistakenly identified as wafer defects. This allows for the fitting of a negative correlation mapping curve between the comprehensive evaluation indicator S and the false alarm rate. Based on the maximum allowable false alarm rate required by the current production line, the critical value of the corresponding comprehensive evaluation indicator S is derived from the mapping curve, which is the adaptive threshold θ for the current detection task.

[0068] Here, when the comprehensive evaluation index S is less than the adaptive threshold θ, the test system will perform closed-loop feedback adjustment; If the low S-value of the comprehensive evaluation index is caused by multi-scale structural similarity, the system will increase the number of computation steps of the smoothing network and increase the upper limit of the number of retransmissions of multiple wafer test channel interfaces. If it is based on the throughput test This causes the testing system to reduce the mechanical step rate of the probe station, causing the actual frame rate to rematch the baseline beat.

[0069] Based on the same inventive concept, this application also provides an image stitching system, referring to... Figure 4 It includes the following modules: The link layer information acquisition module is used to acquire link layer information of multiple wafer test channels in real time using a field-programmable gate array during the wafer testing phase. The hardware confidence configuration module is used to obtain key parameters in the link layer information, normalize the key parameters to obtain the hardware confidence of each wafer test channel, and use the hardware confidence as the prior weight for image stitching. The semantic feature map setting module is used to acquire the original images transmitted in each wafer test channel, and input each original image and its corresponding prior weight as input data into the lightweight semantic segmentation network. The lightweight semantic segmentation network is used to infer the original images and obtain the semantic feature map. The first image inpainting module is used to input the original image and semantic feature map into the first image inpainting network and inpaint the original image to obtain the inpainted image. The stitching weight configuration module is used to calculate the image quality index of the repaired image using multi-scale structural similarity, fuse the channel hardware confidence with the image quality index to obtain the final stitching weight corresponding to the repaired image, and calculate the highest stitching weight in the final stitching weight. The initial stitching image setting module is used to use the repaired image corresponding to the highest stitching weight as the reference image. After registering the remaining repaired images to the reference image coordinate system corresponding to the reference image through affine transformation, the repaired images are initially stitched together based on the reference image using a weighted fusion algorithm to obtain the initial stitched image. The second image restoration module is used to input the initial stitched image and the final stitching weights into the second image restoration network for smoothing to obtain the final stitched image; The stitching weight configuration module is also used to dynamically adjust the final stitching weight based on the quality of the final stitched image.

[0070] In this embodiment, by using hardware confidence as a priori weight for image stitching, the image stitching method is ensured to always prioritize high-quality wafer test channels, thereby improving the stability of obtaining link layer information for each wafer test channel.

[0071] Meanwhile, in this embodiment, the final stitching weights are dynamically adjusted based on the quality of the final stitched image, thus forming an adaptive closed-loop optimization control. This improves the accuracy of the image stitching algorithm and gradually reduces image restoration errors. By utilizing multi-scale structural similarity to calculate the image quality index of the restored image, and fusing the channel hardware confidence with the image quality index, the final stitching weights corresponding to the restored image are obtained. This weight is used in the image processing stage to achieve accurate localization and restoration of problems in the image by the semantic feature map, eliminate seams during image stitching, and improve the quality of the final output stitched image.

[0072] Furthermore, refer to Figure 5In this embodiment, the image stitching system takes the link layer information of multiple wafer test channels as input. After processing by a field-programmable gate array (FPGA), it extracts RAW data (original image data) through protocol unpacking. Simultaneously, it statistically analyzes the information of each wafer test channel and normalizes it to generate hardware confidence scores. Subsequently, a lightweight semantic segmentation network is used to generate semantic feature maps. Combined with a first image inpainting network (a first U-shaped convolutional neural network), image quality indicators and stitching weights are calculated to generate a preliminary stitched image. Then, it is processed by a second image inpainting network (a second U-shaped convolutional neural network). Finally, it is determined whether the comprehensive evaluation index S is not less than the adaptive threshold θ. If so, a high-quality stitched image is output; otherwise, it returns to generating semantic feature maps using the lightweight semantic segmentation network and re-executes the image inpainting operation.

[0073] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0074] The embodiments of the various products and devices in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar embodiments can be referred to interchangeably. For ease of description, the above devices are divided into various modules based on their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software or hardware components.

[0075] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the above embodiments of this application, which are not provided in detail for the sake of brevity.

[0076] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to the integrated circuit (IC) wafer and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0077] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description.

[0078] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. An image stitching method, characterized in that, include: During the wafer testing phase, the link layer information of multiple wafer testing channels is acquired in real time using a field-programmable gate array (FPGA). Key parameters in the link layer information are obtained, and the key parameters are normalized to obtain the hardware confidence score of each wafer test channel. The hardware confidence score is used as the prior weight for image stitching. The original images transmitted in each of the wafer test channels are acquired, and each original image and its corresponding prior weight are input into a lightweight semantic segmentation network as input data. The lightweight semantic segmentation network is used to infer the original images and obtain semantic feature maps. The original image and the semantic feature map are input into the first image inpainting network, and the inpainted image of the original image is obtained. The image quality index of the repaired image is calculated using multi-scale structural similarity. The channel hardware confidence is fused with the image quality index to obtain the final stitching weight corresponding to the repaired image. The highest stitching weight in the final stitching weight is then calculated. Using the repaired image corresponding to the highest stitching weight as the reference image, the remaining repaired images are registered to the reference image coordinate system corresponding to the reference image through affine transformation. Based on the reference image, a weighted fusion algorithm is used to initially stitch the repaired images to obtain an initial stitched image. The initial stitched image and the final stitching weights are input into a second image inpainting network for smoothing to obtain the final stitched image. The final stitching weights are then dynamically adjusted based on the quality of the final stitched image.

2. The image stitching method according to claim 1, characterized in that, The process of obtaining key parameters from the link layer information and normalizing these key parameters to obtain the hardware confidence level of each wafer test channel includes: Obtain key parameters from the link layer information, including cyclic redundancy check error rate, number of data packet retransmissions, and actual signal eye diagram margin.

3. The image stitching method according to claim 1, characterized in that, The normalization process for the key parameters to obtain the hardware confidence level of each wafer test channel includes: The key parameters are normalized by using polarity alignment and weighted fusion to obtain the hardware confidence level of each wafer test channel; A maximum expected error rate threshold is set. When the cyclic redundancy check error rate exceeds the maximum expected error rate threshold, the data in the wafer test channel is severely corrupted. Set a threshold for the maximum expected number of retransmissions. The threshold is the maximum allowed number of retries. If the transmission still fails after exceeding the threshold, the packet will be treated as lost. Set a standard reference signal eye diagram margin. When the actual signal eye diagram margin is not less than the standard reference signal eye diagram margin, the signal integrity is in an ideal state. Preset empirical weight coefficients are dynamically determined based on prior statistics and multiple regression analysis. These preset empirical weight coefficients satisfy the normalization condition, and the specific formula is as follows: ； in, For hardware confidence, , A higher value indicates a higher level of confidence in the hardware. , , To preset the empirical weighting coefficients, Cyclic Redundancy Check (CR) error rate, The threshold for the expected maximum error rate. This refers to the number of data packet retransmissions. The threshold for the expected maximum number of retransmissions. This represents the actual signal eye diagram margin. This is the eye diagram margin for the standard reference signal.

4. The image stitching method according to claim 3, characterized in that, The method of dynamically determining preset empirical weight coefficients based on prior statistics and multiple regression analysis, wherein the preset empirical weight coefficients satisfy normalization conditions, includes: During the calibration phase, the cyclic redundancy check error rate, the number of data packet retransmissions, the actual signal eye diagram margin, and the corresponding image quality indicators transmitted in each wafer test channel are obtained. With image restoration quality as the optimization objective, the correlation coefficients of the cyclic redundancy check error rate, the number of data packet retransmissions, and the actual signal eye diagram margin are fitted using the least squares method and normalized to obtain the optimal combination of empirical weight coefficients. The optimal combination of empirical weight coefficients is used as the preset empirical weight coefficients, and the prior statistics and multiple regression analysis are used to dynamically determine them.

5. The image stitching method according to claim 1, characterized in that, The step of inputting the original image and the semantic feature map into the first image inpainting network includes: The original image and the semantic feature map are input into a first image restoration network, which is a first U-shaped convolutional neural network. The restored image of the original image is obtained by calculating the loss function of the first U-shaped convolutional neural network. The step of inputting the initial stitched image and the final stitched weights into the second image inpainting network for smoothing includes: The initial stitched image and the final stitched weights are input into a second image restoration network for smoothing. The second image restoration network is a second U-shaped convolutional neural network. The restored image is smoothed by calculating the loss function of the second U-shaped convolutional neural network.

6. The image stitching method according to claim 5, characterized in that, The process of obtaining the repaired image from the original image by calculating the loss function of the first U-shaped convolutional neural repair network includes: The repaired image of the original image is obtained by calculating the multi-scale structural similarity loss and norm loss of the first U-shaped convolutional neural repair network. The smoothing process of the repaired image by calculating the loss function of the second U-shaped convolutional neural repair network includes: The repaired image is smoothed by calculating the multi-scale structural similarity loss and norm loss of the second U-shaped convolutional neural repair network. The loss function for both the first U-shaped convolutional neural repair network and the second U-shaped convolutional neural repair network is: ； in, For the total loss, This represents the multi-scale structural similarity loss, focusing on structural edges; Indicates norm loss, focusing on pixel brightness; The balancing weight coefficient is determined experimentally and statistically in the early stages of model training using hyperparameter optimization methods.

7. The image stitching method according to claim 5, characterized in that, The steps of obtaining the repaired image of the original image by calculating the loss function of the first U-shaped convolutional neural network and smoothing the repaired image by calculating the loss function of the second U-shaped convolutional neural network include: Adjust the adaptive parameters of the first U-shaped convolutional neural repair network and / or the second U-shaped convolutional neural repair network. When the repaired image has clear edges but a significant shift in overall brightness, increase the parameters of the first U-shaped convolutional neural repair network and / or the second U-shaped convolutional neural repair network. When the repaired image has accurate colors but blurry texture details, the parameters of the first U-shaped convolutional neural network and / or the second U-shaped convolutional neural network are lowered to maintain the parameter balance of the first U-shaped convolutional neural network and / or the second U-shaped convolutional neural network for mass production.

8. The image stitching method according to claim 5, characterized in that, The process involves obtaining the repaired image of the original image by calculating the loss function of the first U-shaped convolutional neural network, and smoothing the repaired image by calculating the loss function of the second U-shaped convolutional neural network. Gradient descent is used to update the encoder, bottleneck layer, and decoder in both the first and second U-shaped convolutional neural repair networks.

9. An evaluation method for evaluating the impact of the image stitching method as described in any one of claims 1-8 on probe station throughput, characterized in that, include: During the wafer testing phase, the current processing frame rate and the target test frame rate are obtained; The current processing frame rate and the target test frame rate are normalized, and the test throughput is calculated: By weighted fusion of the test throughput and image quality indicators, a comprehensive evaluation index is obtained to assess the impact of the image stitching method on the probe station throughput. The comprehensive evaluation index is compared with the adaptive threshold. If the comprehensive evaluation index is lower than the adaptive threshold, adaptive adjustment is triggered, and a high-quality stitched image is output after adjustment.

10. An image stitching system, characterized in that, include: The link layer information acquisition module is used to acquire link layer information of multiple wafer test channels in real time using a field-programmable gate array during the wafer testing phase. The hardware confidence configuration module is used to obtain key parameters in the link layer information, normalize the key parameters to obtain the hardware confidence of each wafer test channel, and use the hardware confidence as the prior weight for image stitching. The semantic feature map setting module is used to acquire the original images transmitted in each of the wafer test channels, input each original image and its corresponding prior weight as input data into the lightweight semantic segmentation network, and use the lightweight semantic segmentation network to infer the original images and obtain semantic feature maps. The first image restoration module is used to input the original image and the semantic feature map into the first image restoration network, and restore the original image to obtain a restored image. The stitching weight configuration module is used to calculate the image quality index of the repaired image using multi-scale structural similarity, fuse the channel hardware confidence with the image quality index to obtain the final stitching weight corresponding to the repaired image, and calculate the highest stitching weight among the final stitching weights. The initial stitching image setting module is used to take the repaired image corresponding to the highest stitching weight as the reference image, register the remaining repaired images to the reference image coordinate system corresponding to the reference image through affine transformation, and then use the reference image as the basis to perform initial stitching of the repaired images using a weighted fusion algorithm to obtain the initial stitching image. The second image restoration module is used to input the initial stitched image and the final stitching weight into the second image restoration network for smoothing processing to obtain the final stitched image; The stitching weight configuration module is also used to dynamically adjust the final stitching weight based on the quality of the final stitched image.

Images