Interference fringe image enhancement method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an interference fringe image enhancement method and device, equipment and a storage medium. The method comprises the following steps: acquiring a to-be-enhanced current interference fringe image; determining a current image feature corresponding to the current interference fringe image based on the current interference fringe image and a target visual encoder; determining a current text feature corresponding to the current interference fringe image based on the current interference fringe image and a target multi-modal language model; and determining an enhanced target interference fringe image based on the current image feature, the current text feature and the target large language model. Through the technical scheme of the embodiment of the invention, accurate noise removal and accurate retention of image texture details can be realized, so that the enhanced target interference fringe image is determined, and the image enhancement accuracy is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to an interference fringe image enhancement method, device, equipment and storage medium. Background Art

[0002] Wafers are key materials for high-end computing chip manufacturing, and the top and bottom surface morphology has a significant impact on the implementation of subsequent advanced chip process technology. Accurate measurement of wafer surface morphology is an indispensable part of the chip manufacturing process. It is not only the key to ensuring wafer yield, but also the prerequisite for ensuring and improving the reliability of subsequent chip processes. As a representative of optical non-contact measurement methods, Fizeau interferometry has the advantages of high sensitivity, holography, and non-contact, and is favored by semiconductor manufacturing R&D institutions.

[0003] The Fizeau interferometer is sensitive to the environment during the measurement process, which causes the interference fringe image of the wafer surface collected by the camera to be interfered by a variety of complex mixed noises. The fringe image contains three-dimensional morphological information related to the wafer surface to be measured. Therefore, the noise in the interference image has a serious impact on the accuracy of wafer surface reconstruction. The current methods for denoising interference fringe images still have problems such as low applicability, loss of texture details, and over-smoothing of the image. Therefore, there is an urgent need for an image enhancement method that can balance the accurate removal of noise and the preservation of image texture details. Summary of the invention

[0004] The embodiments of the present invention provide an interference fringe image enhancement method, device, equipment and storage medium to achieve accurate noise removal and accurate retention of image texture details, thereby determining an enhanced target interference fringe image and improving the accuracy of image enhancement.

[0005] In a first aspect, an embodiment of the present invention provides an interference fringe image enhancement method, comprising:

[0006] Acquire the current interference fringe image to be enhanced;

[0007] Based on the current interference fringe image and the target visual encoder, determining a current image feature corresponding to the current interference fringe image;

[0008] Based on the current interference fringe image and the target multimodal language model, determining a current text feature corresponding to the current interference fringe image;

[0009] An enhanced target interference fringe image is determined based on the current image feature, the current text feature and a target large language model.

[0010] Optionally, the method also includes: inputting the current interference fringe image into a target visual encoder for feature extraction to obtain reference image features corresponding to the current interference fringe image; and performing feature dimension alignment based on the reference image features and the target linear layer to determine current image features corresponding to the current interference fringe image.

[0011] Optionally, the method further includes: performing attention interaction based on the current image features and the current text features to determine image-text association features corresponding to the current interference fringe image; and determining an enhanced target interference fringe image based on the image-text association features and the target large language model.

[0012] Optionally, the method further includes: determining target image features for removing noise based on the image-text association features and the target large language model; and determining an enhanced target interference fringe image based on the target image features and the target visual encoder.

[0013] Optionally, the method also includes: performing phase extraction based on a target interference fringe image corresponding to a target wafer to determine phase information corresponding to the target interference fringe image; performing phase unfolding based on the phase information to determine unfolded phase information corresponding to the target interference fringe image; performing surface fitting based on discrete height information corresponding to the unfolded phase information to determine target surface information corresponding to the target wafer.

[0014] Optionally, the method also includes: acquiring training sample images; the training sample images include: noise-free images and noisy images corresponding to each wafer; based on the training sample images and the target visual encoder, determining current image features and standard image features corresponding to each wafer; based on the noisy images and the target multimodal language model, determining current text features corresponding to each wafer; training a preset large language model based on the current image features, the current text features and the standard image features until a preset convergence condition is met, thereby obtaining a trained target large language model.

[0015] In a second aspect, an embodiment of the present invention further provides an interference fringe image enhancement device, the device comprising:

[0016] A current interference fringe image acquisition module is used to acquire the current interference fringe image to be enhanced;

[0017] A current image feature determination module, used to determine a current image feature corresponding to the current interference fringe image based on the current interference fringe image and a target visual encoder;

[0018] A first current text feature determination module, used to determine a current text feature corresponding to the current interference fringe image based on the current interference fringe image and a target multimodal language model;

[0019] The target interference fringe image determination module is used to determine an enhanced target interference fringe image based on the current image features, the current text features and the target large language model.

[0020] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0021] one or more processors;

[0022] A memory for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the interference fringe image enhancement method provided by any embodiment of the present invention.

[0024] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interference fringe image enhancement method provided in any embodiment of the present invention.

[0025] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the interference fringe image enhancement method provided in any embodiment of the present invention.

[0026] The technical solution of the embodiment of the present invention is to obtain a current interference fringe image to be enhanced; determine the current image features corresponding to the current interference fringe image based on the current interference fringe image and a target visual encoder; determine the current text features corresponding to the current interference fringe image based on the current interference fringe image and a target multimodal language model; based on the current image features, the current text features and the target large language model, accurate noise removal and accurate retention of image texture details can be achieved, thereby determining an enhanced target interference fringe image and improving image enhancement accuracy.

[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a flow chart of an interference fringe image enhancement method provided in Embodiment 1 of the present invention;

[0030] Figure 2 is a flow chart of an interference fringe image enhancement method provided by Embodiment 2 of the present invention;

[0031] Figure 3 is a schematic structural diagram of an interference fringe image enhancement device provided in Embodiment 3 of the present invention;

[0032] Figure 4 It is a schematic diagram of the structure of an electronic device for implementing the interference fringe image enhancement method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment 1

[0036] Figure 1A flowchart of an interference fringe image enhancement method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a variety of complex mixed noises are removed from the interference fringe image on the surface of a wafer. The method can be performed by an interference fringe image enhancement device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0037] S110, obtaining a current interference fringe image to be enhanced.

[0038] Among them, the current interference fringe image may refer to an interference fringe image corresponding to the wafer containing at least one kind of noise. The current interference fringe image may be an interference fringe image from which noise needs to be removed. The current interference fringe image may be an image of interference between reference light and measurement light generated by an optical system such as a Fizeau interferometer using a monochromatic laser to shoot at a wafer. However, due to the influence of environmental factors such as vibration, illumination, temperature and humidity, the current interference fringe image contains noise. The current interference fringe image can be understood as a low-quality image.

[0039] Specifically, a portion of the monochromatic laser light emitted by the He-Ne laser of the Fizeau interferometer is reflected on the reference surface as reference light, and the other portion is reflected on the wafer surface as measurement light through the reference surface. The reference light and the measurement light interfere with each other, and the camera in the Fizeau interferometer captures the current interference fringe image. The current interference fringe image to be enhanced captured by the camera is obtained.

[0040] S120, determining a current image feature corresponding to the current interference fringe image based on the current interference fringe image and the target visual encoder.

[0041] Among them, the visual encoder may refer to a model for image understanding tasks, which can handle multiple image understanding tasks such as classification, localization, retrieval, subtitles, and question-answering through a single model. The visual encoder usually includes two parts: an encoder and a decoder, where the encoder is responsible for compressing the image data into a low-dimensional representation, and the decoder restores these low-dimensional representations to the original image data. For example, the visual encoder may be, but is not limited to, a masked autoencoder (MAE). MAE can encode an image into a series of visual features and reconstruct the original image through image tokens of the masked image. The target visual encoder may refer to a pre-trained visual encoder. The current image features may refer to image features with noise.

[0042] Specifically, the current interference fringe image is input into the target visual encoder, and the encoder is used to encode the current image features corresponding to the current interference fringe image.

[0043] On the basis of the above technical solution, "determining the current image features corresponding to the current interference fringe image based on the current interference fringe image and the target visual encoder" may include: inputting the current interference fringe image into the target visual encoder for feature extraction to obtain the reference image features corresponding to the current interference fringe image; aligning the feature dimensions based on the reference image features and the target linear layer to determine the current image features corresponding to the current interference fringe image.

[0044] Among them, the reference image feature may refer to an image feature whose feature dimension is not aligned. The target large language model cannot accurately understand the reference image feature. The current image feature may refer to an image feature whose feature dimension is aligned. The target large language model can accurately understand the current image feature. The linear layer is also a fully connected layer. The linear layer can be used to perform linear transformations, linearly combining the input data through a weight matrix and a bias to map it to the output data. Linear layer structure Linear connection is a form of full connection, but full connection is not necessarily a linear connection. The fully connected layer can use nonlinear activation functions, while the linear connection only performs a simple linear mapping. Training a target linear layer between the target visual encoder and the target large language model can enable the target large language model to have visual understanding capabilities, further improving the accuracy of image enhancement.

[0045] S130 : Determine current text features corresponding to the current interference fringe image based on the current interference fringe image and the target multimodal language model.

[0046] Among them, a multimodal large language model (MLLM) may refer to a model that combines the natural language processing capabilities of large language models (LLMs) with the understanding and generation capabilities of other modal data (such as vision, audio, etc.). For example, the multimodal language model may be, but is not limited to, DepictQA. The target multimodal language model may refer to a pre-trained multimodal language model. The preset multimodal language model to be trained may be a modeling method for a mixed noise interference fringe image, and generate text features for content understanding of the interference fringe image, detailed distortion description of the scale, mixed noise type, and noise degree of each type of noise. Image distortion scale generally refers to a concept for measuring image or video quality, which describes the degree of difference between the original image or video and the processed image or video. Modeling an image using a multimodal language model generally means combining image and text information for modeling. This model can process visual and text data simultaneously, so as to more comprehensively understand and explain the image content. For example, in an image annotation task, a multimodal model can combine the image and the corresponding text description information to improve the accuracy and effect of image annotation. Content understanding and scale-detailed distortion description can be used to preserve image texture details; mixed noise types and noise levels can be used to accurately remove noise; thus balancing accurate noise removal with image texture detail preservation to achieve accurate image enhancement.

[0047] Specifically, the current interference fringe image is input into the target multimodal language model for feature extraction, and the current text features corresponding to the current interference fringe image are determined.

[0048] Exemplarily, "determining current text features corresponding to the current interference fringe image based on the current interference fringe image and the target multimodal language model" may include: inputting the current interference fringe image into the target multimodal language model for feature extraction to obtain reference text features corresponding to the current interference fringe image; performing feature dimension alignment based on the reference text features and the target linear layer to determine the current text features corresponding to the current interference fringe image.

[0049] The reference text feature may refer to a text feature whose feature dimension is not aligned. The target large language model cannot accurately understand the reference text feature. The current text feature may refer to a text feature whose feature dimension is aligned. The target large language model can accurately understand the current text feature.

[0050] S140: Determine an enhanced target interference fringe image based on current image features, current text features and a target large language model.

[0051] The target interference fringe image may refer to an interference fringe image with noise removed. For example, the target interference fringe image may be a noise-free image. The target interference fringe image may be understood as a high-quality image.

[0052] Specifically, the current text features describing the current interference fringe image (low-quality image) output by the target multimodal language model and the current image features extracted by the target visual encoder are input into the target large language model, and the current image features are adjusted accordingly based on the description information of the current text features to achieve accurate removal of mixed noise and obtain an enhanced target interference fringe image.

[0053] The technical solution of the embodiment of the present invention obtains the current interference fringe image to be enhanced; determines the current image features corresponding to the current interference fringe image based on the current interference fringe image and the target visual encoder; determines the current text features corresponding to the current interference fringe image based on the current interference fringe image and the target multimodal language model; based on the current image features, the current text features and the target large language model, accurate noise removal and accurate retention of image texture details can be achieved, thereby determining an enhanced target interference fringe image and improving the accuracy of image enhancement.

[0054] Based on the above technical solution, the method also includes: performing phase extraction based on the target interference fringe image corresponding to the target wafer to determine the phase information corresponding to the target interference fringe image; performing phase unfolding based on the phase information to determine the unfolded phase information corresponding to the target interference fringe image; performing surface fitting based on discrete height information corresponding to the unfolded phase information to determine the target surface information corresponding to the target wafer.

[0055] The target wafer may refer to the wafer corresponding to the current interference fringe image. The phase information may refer to the phase difference. The target surface information may refer to the relevant information of the wafer surface generated by fitting. The target surface information may be used to evaluate the surface shape of the wafer surface.

[0056] Specifically, phase extraction is performed based on the target interference fringe image corresponding to the target wafer, and the phase difference between the reference light and the measurement light is calculated. Phase unwrapping is performed based on the phase information, so that the overlapping jump phase is unwrapped. The discrete height information corresponding to the unwrapped phase is linearly represented by Zernike polynomials, fitted into a smooth surface, and the target surface information corresponding to the target wafer is determined. And the surface shape of the wafer surface can be evaluated based on the relevant information of the surface.

[0057] It should be noted that the inventors have found that there are at least the following problems in the prior art: In the stage of image preprocessing to obtain high-quality images, the noise removal strength contained in the interference fringe image is not strong enough, which will lead to many irregular artifacts in the result of phase solution (phase information). Excessive denoising will lead to phase loss in the result of phase solution. Both of these situations deviate from the ideal denoising state. It can be seen that the quality of the interference fringe image has a serious impact on the final results of the subsequent phase unwrapping and surface fitting process, and the low-quality image reduces the accuracy of wafer surface reconstruction.

[0058] On the basis of the above technical solution, the method also includes: acquiring training sample images; the training sample images include: noise-free images and noisy images corresponding to each wafer; based on the training sample images and the target visual encoder, determining the current image features and standard image features corresponding to each wafer; based on the noisy image and the target multimodal language model, determining the current text features corresponding to each wafer; training a preset large language model based on the current image features, the current text features and the standard image features until the preset convergence conditions are met, thereby obtaining a trained target large language model.

[0059] The training sample image may be a clear and noise-free interference fringe image constructed by using Zernike polynomials and used as a noise-free image GT (high quality image, HQ Image), and then various forms of noise are added to the noise-free image GT, such as Gaussian noise, Poisson noise, salt and pepper noise, speckle noise, etc. The preset convergence condition may be, but is not limited to, the number of model iteration training times reaching a preset number of iterations or the accuracy of the model output reaching a preset threshold.

[0060] Specifically, noise-free images and noisy images corresponding to each wafer are obtained and used for model training. The noise-free image is input into the target visual encoder for encoding and compression to obtain standard image features. The noisy image is input into the target visual encoder for encoding and compression to obtain current image features. The noisy image is input into the target multimodal language model for feature extraction to determine the current text features corresponding to each wafer. The preset large language model is trained based on the current image features, current text features and standard image features until the preset convergence conditions are met to obtain the trained target large language model.

[0061] It should be noted that in order to effectively monitor the large language model to accurately infer the noise-free image features, a dual strategy is adopted in the loss function design. On the one hand, the visual features of the noise-free image are encoded to obtain the standard image features (after the target linear layer), and the L1 loss function is used to constrain the difference between the standard image features and the image features inferred by the large language model (without the target linear layer). On the other hand, the cross entropy loss function is used to measure the difference between the visual features of the noise-free image (without the target linear layer) and the feature statistical distribution of the large model inference output (after the target linear layer). The L1 loss function is favored for its robustness to outliers, while the cross entropy loss function focuses on evaluating the difference between the probability distribution predicted by the model and the true label. By combining these two loss functions, the goal is to balance the model's sensitivity to outliers and classification accuracy to achieve better image denoising effects and achieve accurate noise removal of interference fringe images.

[0062] Embodiment 2

[0063] Figure 2 This is a flow chart of an interference fringe image enhancement method provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining the enhanced target interference fringe image. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0064] S210: Acquire a current interference fringe image to be enhanced.

[0065] S220. Determine current image features corresponding to the current interference fringe image based on the current interference fringe image and the target visual encoder.

[0066] S230 : Determine current text features corresponding to the current interference fringe image based on the current interference fringe image and the target multimodal language model.

[0067] S240: Perform attention interaction based on current image features and current text features to determine image-text association features corresponding to the current interference fringe image.

[0068] Among them, the image-text association feature can refer to the association information within and between modalities. Potential semantic connections can be mined through the image-text association feature.

[0069] Specifically, the current image features and the current text features are concatenated to form multimodal features. Through attention mechanisms, such as bidirectional attention or multimodal attention, the model can focus on the image or text parts that are most relevant to the task (noise removal and image texture detail preservation). The use of context-aware attention mechanisms enables the model to selectively focus on important text fragments or image regions. Through the correlation information within and between modalities, potential semantic connections are mined.

[0070] It should be noted that the image features after attention interaction can also be fused with the text features to form enhanced image features, thereby using the target large language model to determine the enhanced target interference fringe image.

[0071] S250: Determine an enhanced target interference fringe image based on the image-text association features and the target large language model.

[0072] Specifically, the image-text association features are input into the target large language model, so that the target large language model can accurately remove the mixed noise contained in the current interference fringe image based on the current image features under the guidance of the current text features, and obtain an enhanced target interference fringe image.

[0073] Based on the above technical solution, "determining an enhanced target interference fringe image based on image-text association features and a target large language model" may include: determining target image features for removing noise based on image-text association features and a target large language model; determining an enhanced target interference fringe image based on target image features and a target visual encoder.

[0074] The target image feature may refer to an image feature that balances accurate noise removal and preservation of image texture details.

[0075] Specifically, the image-text association features are input into the target large language model, so that the target large language model can accurately remove the mixed noise contained in the current interference fringe image based on the current image features under the guidance of the current text features, and the regression output balances the accurate removal of noise and the target image features that retain image texture details. The target image features are input into the target visual encoder for decoding and restoration to obtain an enhanced target interference fringe image.

[0076] The technical solution of the embodiment of the present invention determines the image-text association features corresponding to the current interference fringe image by performing attention interaction based on the current image features and the current text features, thereby utilizing the generalization ability of the target large language model and its ability to understand image and text features. Under the guidance of the current text features, the mixed noise contained in the current interference fringe image is accurately removed based on the current image features, thereby obtaining an enhanced target interference fringe image, and further improving the accuracy of image enhancement.

[0077] The following is an embodiment of the interference fringe image enhancement device provided in an embodiment of the present invention. The device and the interference fringe image enhancement methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the interference fringe image enhancement device, reference can be made to the embodiment of the above-mentioned interference fringe image enhancement method.

[0078] Embodiment 3

[0079] Figure 3 This is a schematic diagram of the structure of an interference fringe image enhancement device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a current interference fringe image acquisition module 310 , a current image feature determination module 320 , a first current text feature determination module 330 and a target interference fringe image determination module 340 .

[0080] Among them, the current interference fringe image acquisition module 310 is used to acquire the current interference fringe image to be enhanced; the current image feature determination module 320 is used to determine the current image features corresponding to the current interference fringe image based on the current interference fringe image and the target visual encoder; the first current text feature determination module 330 is used to determine the current text features corresponding to the current interference fringe image based on the current interference fringe image and the target multimodal language model; the target interference fringe image determination module 340 is used to determine the enhanced target interference fringe image based on the current image features, the current text features and the target large language model.

[0081] The technical solution of the embodiment of the present invention obtains the current interference fringe image to be enhanced; determines the current image features corresponding to the current interference fringe image based on the current interference fringe image and the target visual encoder; determines the current text features corresponding to the current interference fringe image based on the current interference fringe image and the target multimodal language model; based on the current image features, the current text features and the target large language model, accurate noise removal and accurate retention of image texture details can be achieved, thereby determining an enhanced target interference fringe image and improving the accuracy of image enhancement.

[0082] Based on the above technical solution, the current image feature determination module 320 is specifically used to: input the current interference fringe image into the target visual encoder for feature extraction to obtain the reference image features corresponding to the current interference fringe image; perform feature dimension alignment based on the reference image features and the target linear layer to determine the current image features corresponding to the current interference fringe image.

[0083] Based on the above technical solution, the target interference fringe image determination module 340 may include:

[0084] The image-text association feature determination submodule is used to perform attention interaction based on the current image features and the current text features to determine the image-text association features corresponding to the current interference fringe image;

[0085] The target interference fringe image determination submodule is used to determine the enhanced target interference fringe image based on the image-text association features and the target large language model.

[0086] Based on the above technical solution, the target interference fringe image determination submodule is specifically used to: determine the noise-removed target image features based on the image-text association features and the target large language model; determine the enhanced target interference fringe image based on the target image features and the target visual encoder.

[0087] On the basis of the above technical solution, the device also includes:

[0088] A phase information determination module is used to perform phase extraction based on a target interference fringe image corresponding to the target wafer, and determine the phase information corresponding to the target interference fringe image;

[0089] An unwrapped phase information determination module is used to perform phase unwrapping based on the phase information and determine the unwrapped phase information corresponding to the target interference fringe image;

[0090] The target surface information determination module is used to perform surface fitting based on discrete height information corresponding to the unfolded phase information to determine the target surface information corresponding to the target wafer.

[0091] On the basis of the above technical solution, the device also includes:

[0092] The training sample image acquisition module is used to acquire the training sample images; the training sample images include: noise-free images and noisy images corresponding to each wafer;

[0093] A standard image feature determination module, used to determine the current image features and standard image features corresponding to each wafer based on the training sample image and the target visual encoder;

[0094] A second current text feature determination module, used to determine the current text features corresponding to each wafer based on the noisy image and the target multimodal language model;

[0095] The target large language model training module is used to train the preset large language model based on the current image features, the current text features and the standard image features until the preset convergence conditions are met to obtain a trained target large language model.

[0096] The interference fringe image enhancement device provided in the embodiment of the present invention can execute the interference fringe image enhancement method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the interference fringe image enhancement method.

[0097] It is worth noting that in the above-mentioned embodiment of interference fringe image enhancement, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0098] Embodiment 4

[0099] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0100] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0101] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the interference fringe image enhancement method.

[0103] In some embodiments, the interference fringe image enhancement method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the interference fringe image enhancement method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the interference fringe image enhancement method in any other appropriate manner (e.g., by means of firmware).

[0104] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0106] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0108] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0109] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0110] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the interference fringe image enhancement method provided in any embodiment of the present application.

[0111] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, and the programming language includes an object-oriented programming language, such as python, Matlab, Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The program product and the interference fringe image enhancement method disclosed in each embodiment of the present application belong to the same inventive concept, so it is not repeated here.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for enhancing interference fringe images, characterized in that: include: Acquire the current interference fringe image to be enhanced; Based on the current interference fringe image and the target visual encoder, determining a current image feature corresponding to the current interference fringe image; Based on the current interference fringe image and the target multimodal language model, determining a current text feature corresponding to the current interference fringe image; An enhanced target interference fringe image is determined based on the current image feature, the current text feature and a target large language model.

2. The method according to claim 1, characterized in that The determining, based on the current interference fringe image and the target visual encoder, a current image feature corresponding to the current interference fringe image comprises: Inputting the current interference fringe image into a target visual encoder for feature extraction to obtain reference image features corresponding to the current interference fringe image; Feature dimensions are aligned based on the reference image features and the target linear layer to determine current image features corresponding to the current interference fringe image.

3. The method according to claim 1, characterized in that: The step of determining an enhanced target interference fringe image based on the current image feature, the current text feature and the target large language model comprises: Performing attention interaction based on the current image feature and the current text feature to determine the image-text association feature corresponding to the current interference fringe image; Based on the image-text association features and the target large language model, an enhanced target interference fringe image is determined.

4. The method according to claim 3, characterized in that The step of determining an enhanced target interference fringe image based on the image-text association feature and the target large language model includes: Determine target image features for noise removal based on the image-text association features and the target large language model; Based on the target image features and the target visual encoder, an enhanced target interference fringe image is determined.

5. The method according to claim 1, characterized in that The method further comprises: Performing phase extraction based on a target interference fringe image corresponding to a target wafer to determine phase information corresponding to the target interference fringe image; Perform phase unwrapping based on the phase information to determine the unwrapped phase information corresponding to the target interference fringe image; Surface fitting is performed based on discrete height information corresponding to the unfolded phase information to determine target surface information corresponding to the target wafer.

6. The method according to claim 1, characterized in that The method further comprises: Acquire training sample images; the training sample images include: noise-free images and noisy images corresponding to each wafer; Based on the training sample images and the target visual encoder, determining current image features and standard image features corresponding to each wafer; Determining current text features corresponding to each of the wafers based on the noisy image and the target multimodal language model; The preset large language model is trained based on the current image feature, the current text feature and the standard image feature until a preset convergence condition is met, thereby obtaining a trained target large language model.

7. An interference fringe image enhancement device, characterized in that: The device comprises: A current interference fringe image acquisition module is used to acquire the current interference fringe image to be enhanced; A current image feature determination module, used to determine a current image feature corresponding to the current interference fringe image based on the current interference fringe image and a target visual encoder; A first current text feature determination module, used to determine a current text feature corresponding to the current interference fringe image based on the current interference fringe image and a target multimodal language model; The target interference fringe image determination module is used to determine an enhanced target interference fringe image based on the current image features, the current text features and the target large language model.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the interference fringe image enhancement method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the interference fringe image enhancement method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the interference fringe image enhancement method as described in any one of claims 1 to 6.

Images