Fringe projection image enhancement method and device based on multi-view stereo vision and super-resolution, electronic device and storage medium

Abstract

This disclosure relates to a method, apparatus, electronic device, and storage medium for enhancing fringe projection images based on multi-view stereo vision and super-resolution. The method includes: acquiring a front view image and a side view image of an object under test projecting a fringe pattern; reconstructing the side view image to obtain a reconstructed side view image, wherein the quality of the reconstructed side view image is superior to the original side view image; determining a quality index for the reconstructed side view image; if the quality index does not meet a quality condition, determining an optimized side view image based on the reconstructed side view image, using the optimized side view image as a new side view image, iteratively performing the reconstruction to the determination of the optimized side view image operation until the quality index of the reconstructed side view image meets the quality condition, thereby obtaining a target side view image. The target side view image is used in conjunction with the front view image to determine the morphological data of the object under test. This method can improve measurement accuracy without increasing hardware costs.

Description

Technical Field

[0001] This disclosure relates to the field of measurement technology, and in particular to a method, apparatus, electronic device, and storage medium for enhancing fringe projection images based on multi-view stereo vision and super-resolution. Background Art

[0002] Semiconductor manufacturing processes are becoming increasingly miniaturized and compact. Especially with the development of advanced packaging technologies, the objects being inspected are densely stacked in the height direction, making three-dimensional optical inspection an indispensable tool. Fringe projection profilometry, as a mature method in industrial online measurement, performs excellently in balancing speed and accuracy. However, because the projection and imaging must maintain a certain angle, this method is susceptible to shading and multiple reflection effects, resulting in lower modulation of the fringe image. This can lead to phase ambiguity and phase breaks, causing a decrease in local measurement accuracy.

[0003] While multi-camera solutions can mitigate these issues to some extent, post-processing methods such as phase or height interpolation and fusion do not fundamentally improve fringe quality, resulting in unsatisfactory regional accuracy. Furthermore, increasing the camera array size and number can improve measurement accuracy. However, this increases costs with the addition of hardware. Moreover, this approach leads to a dramatic increase in data acquisition and computational load, severely impacting measurement timeliness. Summary of the Invention

[0004] In view of this, this disclosure proposes a striped projection image enhancement scheme based on multi-view stereo vision and super-resolution.

[0005] According to one aspect of this disclosure, a method for enhancing fringe projection images based on multi-view stereo vision and super-resolution is provided. The method includes: acquiring a front view image and a side view image of an object to be tested, including a projected fringe pattern; reconstructing the side view image to obtain a reconstructed side view image, wherein the quality of the reconstructed side view image is superior to that of the original side view image; determining a quality index of the reconstructed side view image; if the quality index does not meet a quality condition, determining an optimized side view image based on the reconstructed side view image, using the optimized side view image as a new side view image, iteratively performing the reconstruction to the determination of the optimized side view image operation until the quality index of the reconstructed side view image meets the quality condition, thereby obtaining a target side view image, wherein the target side view image is used in conjunction with the front view image to determine the morphological data of the object to be tested.

[0006] In one possible implementation, reconstructing the side view image to obtain a reconstructed side view image includes: determining depth data based on the front view image and the side view image; determining a mapping relationship between the front view image and the side view image based on the depth data; and reconstructing the side view image based on the front view image using the mapping relationship as a guiding parameter to obtain the reconstructed side view image.

[0007] In one possible implementation, determining the quality index of the reconstructed side view image includes: dividing the reconstructed side view image into multiple first regions; and determining a first quality index for each of the first regions.

[0008] In one possible implementation, the quality condition includes a first quality condition, and determining an optimized side view image based on the reconstructed side view image when the quality index does not meet the quality condition includes: using the reconstructed side view image as the optimized side view image when any of the first quality indexes does not meet the first quality condition.

[0009] In one possible implementation, determining the quality index of the reconstructed side view image includes: downsampling the reconstructed side view image to obtain a low-resolution image, the low-resolution image having the same resolution as the side view image; dividing the low-resolution image into multiple second regions; and determining a second quality index for each of the second regions.

[0010] In one possible implementation, the quality condition includes a second quality condition. The step of determining an optimized side view image based on the reconstructed side view image when the quality indicator does not meet the quality condition includes: dividing the side view image into multiple third regions; determining a target second region where the second quality indicator does not meet the second quality condition; and replacing the target second region with a third region corresponding to the target second region to obtain the optimized side view image.

[0011] In one possible implementation, determining the quality index of the reconstructed side view image includes: determining sub-indicators, which include the following: stripe pattern period, foreground and background contrast of the stripe pattern, grayscale variation range of the stripe pattern, position of the pixel in the stripe pattern period, material of the object under test, brightness distribution of the stripe pattern, and quality of neighboring pixels; and determining the quality index based on the sub-indicators.

[0012] According to another aspect of this disclosure, a striped projection image enhancement device based on multi-view stereo vision and super-resolution is provided. The device includes:

[0013] The image acquisition unit is used to acquire a front view image and a side view image of the object under test, which includes a projected striped pattern.

[0014] A reconstructed side view image determination unit is used to reconstruct the side view image to obtain a reconstructed side view image, wherein the quality of the reconstructed side view image is better than that of the original side view image.

[0015] A quality index determination unit is used to determine the quality index of the reconstructed side view image;

[0016] The target side view image determination unit is used to determine an optimized side view image based on the reconstructed side view image when the quality index does not meet the quality conditions. The optimized side view image is used as a new side view image, and the operation of reconstructing to determine the optimized side view image is performed iteratively until the quality index of the reconstructed side view image meets the quality conditions, so as to obtain a target side view image. The target side view image is used to determine the shape data of the object under test in combination with the front view image.

[0017] In one possible implementation, the reconstructed side view image determination unit is further configured to:

[0018] Determine depth data based on the front view image and the side view image;

[0019] Based on the depth data, the mapping relationship between the front view image and the side view image is determined;

[0020] Using the mapping relationship as a guiding parameter, the side view image is reconstructed based on the front view image to obtain the reconstructed side view image.

[0021] In one possible implementation, the quality index determination unit is further configured to:

[0022] The reconstructed side view image is divided into multiple first regions;

[0023] Determine the first quality index for each of the first regions.

[0024] In one possible implementation, the quality condition includes a first quality condition, and the target side view image determination unit is further configured to:

[0025] If any of the first quality indicators fails to meet the first quality condition, the reconstructed side view image shall be used as the optimized side view image.

[0026] In one possible implementation, the quality index determination unit is further configured to:

[0027] The reconstructed side view image is downsampled to obtain a low-resolution image, which has the same resolution as the side view image.

[0028] The low-resolution image is divided into multiple second regions;

[0029] Determine the second quality index for each of the second regions.

[0030] In one possible implementation, the quality condition includes a second quality condition, and the target side view image determination unit includes:

[0031] The side view image is divided into multiple third regions;

[0032] Identify the target second region where the second quality indicator does not meet the second quality condition;

[0033] The second target region is replaced with a third region corresponding to the second target region to obtain the optimized side view image.

[0034] In one possible implementation, the quality index determination unit is further configured to:

[0035] The sub-indicators are determined, including the following: stripe pattern period, foreground and background contrast of the stripe pattern, grayscale variation range of the stripe pattern, position of the pixel in the stripe pattern period, material of the object under test, brightness distribution of the stripe pattern, and quality of neighboring pixels.

[0036] The quality indicators are determined based on the aforementioned sub-indicators.

[0037] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0038] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0039] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0040] In this embodiment, the side view image is iteratively reconstructed, its quality assessed, and an optimized image determined based on the reconstructed side view image until the quality index of the side view image meets the quality conditions. This improves the quality of the side view images acquired from each angle, thereby enhancing the accuracy of the measurement of the object under test. No increase in camera data volume or camera array size is required, and existing measurement equipment remains unchanged, resulting in no additional hardware costs. Furthermore, since the number of cameras and the array size are not increased, the amount of data acquisition and the number of images to be processed remain unchanged, thus not affecting the timeliness of the measurement. Therefore, the method of this disclosure can improve measurement accuracy without increasing hardware costs.

[0041] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0042] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0043] Figure 1 This is a schematic flowchart illustrating the method for enhancing fringe projection images based on multi-view stereo vision and super-resolution provided in this embodiment of the disclosure.

[0044] Figure 2 This is a schematic diagram of the structure of an image acquisition device provided in an embodiment of this disclosure.

[0045] Figure 3 This is a flowchart illustrating another method for enhancing striped projection images based on multi-view stereo vision and super-resolution, provided in an embodiment of this disclosure.

[0046] Figure 4 This is a schematic diagram of the structure of the striped projection image enhancement device based on multi-view stereo vision and super-resolution provided in the embodiments of this disclosure.

[0047] Figure 5 This is a schematic diagram of the structure of an electronic device for enhancing fringe projection images based on multi-view stereo vision and super-resolution, as provided in an embodiment of this disclosure. Detailed Implementation

[0048] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0049] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0050] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0051] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0052] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0053] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0054] Figure 1 This is a schematic flowchart illustrating the fringe projection image enhancement method based on multi-view stereo vision and super-resolution provided in an embodiment of this disclosure. Figure 1 As shown, the method includes:

[0055] S11, acquire a front view image and a side view image of the object to be tested, which include a projected striped pattern.

[0056] The front view image can be acquired by the camera directly above the object under test. The side view image can be acquired by the camera not directly above the object under test. The resolution of the front view image can be higher than that of the side view image. The distortion rate of the front view image can be lower than that of the side view image. The object under test can be a wafer bump.

[0057] The cameras used to acquire the front view image and the side view image can be different. For ease of description, the camera acquiring the front view image will be called the front-view camera, and the camera acquiring the side view image will be called the side-view camera. For example, multiple side-view cameras and one front-view camera can be set up, with the object to be measured positioned directly below the front-view camera. The front-view camera and the multiple side-view cameras can acquire images simultaneously to shorten image acquisition time.

[0058] Figure 2 This is a schematic diagram of the image acquisition device provided in an embodiment of the present disclosure. The device includes a front view image acquisition unit 10 and multiple side view image acquisition units 11. The front view image acquisition unit 10 includes a front view camera 100, a projection device 104, and a beam splitter 102. The beam splitter allows the optical paths of the front view camera 100 and the projection device 104 to be coaxial or nearly coaxial. The projection device 104 can project a striped pattern orthogonally onto the object under test. The projection device 104 can use a Ronchi grating or a sinusoidal grating combined with piezoelectric ceramics to achieve phase-shift projection. Alternatively, the projection device can use an optical engine to achieve phase-shift projection through pattern switching. The optical engine can include one or more of the following: a digital micromirror element optical engine, an off-axis optical projection system optical engine, an off-axis optical projection system, etc. Furthermore, a first telecentric lens 101 can be disposed between the front view camera 100 and the beam splitter 102. A second telecentric lens 102 can be disposed between the beam splitter 102 and the object under test. The front view image acquisition unit 10 is positioned directly above the object under test. The main camera 100 can be a large-format monochrome or color industrial camera. This large-format monochrome or color industrial camera can be an industrial camera with a resolution of at least 4 megapixels.

[0059] A single side-view image acquisition unit 11 includes: a side-view camera 112, a third telecentric lens 111, and a Sham angle adjustment device 110 disposed between the side-view camera 112 and the third telecentric lens 111 to maintain lateral depth of field. The resolution of the main-view camera can be higher than that of the side-view camera to reduce data transmission. In embodiments of this disclosure, this device can be used to acquire both a main-view image and a side-view image. The main-view image can serve as a reference image for super-resolution reconstruction of the side-view image.

[0060] S12, the side view image is reconstructed to obtain a reconstructed side view image, the quality of which is better than that of the side view image.

[0061] In this embodiment of the disclosure, a side view image can be reconstructed using super-resolution reconstruction to obtain a reconstructed side view image. Exemplarily, super-resolution reconstruction of the side view image can be performed based on a super-resolution interpolation method. The super-resolution interpolation method may include at least one of the following: median, bicubic interpolation, sparse coding, nonlocal averaging, etc. Exemplarily, super-resolution reconstruction of the side view image can be performed based on a neural network. In this embodiment of the disclosure, the method of super-resolution reconstruction is not limited.

[0062] The reconstructed side view image is of better quality than the original side view image. Quality can be reflected in one or more of the following aspects: resolution, sharpness, brightness, local contrast, noise level, etc. The above are merely examples, and the quality metrics for measuring image quality are not limited in this disclosure. These will be described in detail below.

[0063] S13, determine the quality index of the reconstructed side view image.

[0064] The quality metric can be an overall quality metric of the reconstructed image, and / or a local quality metric, or a quality metric for each pixel, etc. In one example, mathematical statistical methods can be used to evaluate the quality metric of the reconstructed side view image. In another example, a pre-trained evaluation model can be used to evaluate the quality metric of the reconstructed side view image. In this embodiment of the disclosure, the method for determining the quality metric is not limited.

[0065] S14, if the quality index does not meet the quality conditions, based on the reconstructed side view image, an optimized side view image is determined, and the optimized side view image is used as a new side view image. The operation of reconstructing to the determined optimized side view image is iteratively performed until the quality index of the reconstructed side view image meets the quality conditions, and a target side view image is obtained. The target side view image is used to determine the shape data of the object under test in combination with the front view image.

[0066] Quality conditions can be either that a quality indicator is greater than a quality threshold, or that a quality indicator is less than a quality threshold. When there is more than one quality indicator, the quality condition can be that all quality indicators are greater than the quality threshold, or that all quality indicators are less than the quality threshold. Alternatively, the quality condition can be that some quality indicators are greater than a first quality threshold, and another part of the quality indicators are less than the first quality threshold. Quality conditions can be designed based on actual quality indicators and actual needs. The quality threshold, first quality threshold, and second quality threshold involved in this disclosure can be thresholds set based on experience, or thresholds obtained based on training and inference using an artificial intelligence model.

[0067] If the quality indicators do not meet the quality conditions, an optimized side view image can be determined based on the reconstructed side view image. For example, the reconstructed side view image can be used as the optimized side view image. For example, a portion of the reconstructed side view image can be reconstructed to obtain the optimized side view image. The above are merely examples, and the methods for determining the optimized side view image in this disclosure are not limited. The following will describe different scenarios.

[0068] In this embodiment of the disclosure, the optimized side view image can be used as the new side view image, and S12-S14 can be executed iteratively until the quality index meets the quality condition, at which point the iteration stops. The side view image obtained in the last iteration can be used as the target side view image.

[0069] The quality of the side view image should be no less than that of the front view image. The front view image and the side view image represent the object under test from different angles. This can reduce the negative impact of shadow occlusion on the accuracy of the topographic data.

[0070] Based on a single front view image, the first height of each entity point of the object under test can be determined. Based on a single side view image, the second height of each entity point of the object under test can be determined. For example, the second heights and first heights of the same entity point can be firstly fused to obtain a first target height. Based on the first target heights of each entity point, the shape data of the object under test is determined. For example, the second heights of the same entity point can be secondly fused to obtain a side view height; the first heights can be thirdly fused to obtain a front view height. Then, the side view heights and front view heights of the same entity point are fused to obtain a second target height. Based on the second target heights of each entity point, the shape data of the object under test is determined.

[0071] In this embodiment, the side view image is iteratively reconstructed, its quality assessed, and an optimized image determined based on the reconstructed side view image until the quality index of the side view image meets the quality conditions. This improves the quality of the side view images acquired from each angle, thereby enhancing the accuracy of the measurement of the object under test. No increase in camera data volume or camera array size is required, and existing measurement equipment remains unchanged, resulting in no additional hardware costs. Furthermore, since the number of cameras and the array size are not increased, the amount of data acquisition and the number of images to be processed remain unchanged, thus not affecting the timeliness of the measurement. Therefore, the method of this disclosure can improve measurement accuracy without increasing hardware costs.

[0072] In one possible implementation, reconstructing the side view image to obtain a reconstructed side view image includes: determining depth data based on the front view image and the side view image; determining a mapping relationship between the front view image and the side view image based on the depth data; and reconstructing the side view image based on the front view image using the mapping relationship as a guiding parameter to obtain the reconstructed side view image.

[0073] In this embodiment, the front view image and multiple side view images obtained from a single capture can be processed, for example, by interpolation, to ensure that the front view image and side view images obtained from a single capture are of the same size. In this way, the front view image and side view images can constitute a multi-view stereo vision system. Based on this multi-view stereo vision system, corresponding feature points (such as object edges, textures, etc.) in the front view image and side view images can be identified. Utilizing the positional differences of these feature points in different images, and combining them with camera parameters, the depth value of each pixel (i.e., the distance from the object surface to the camera) can be calculated to obtain depth data. The depth data can be a depth map, which contains the depth value of each pixel on the depth map.

[0074] Based on depth data, the mapping relationship between the front view image and the side view image corresponding to that depth data can be determined. Specifically, for a pixel in the front view image, its position in three-dimensional space is determined based on the depth value. Then, a pixel corresponding to that position in three-dimensional space is searched in the side view image. By repeating the above process, a corresponding point in the side view image is found for each pixel in the front view image, thereby establishing a pixel-level mapping relationship between the front view image and the side view image.

[0075] In this embodiment of the disclosure, a pixel to be reconstructed can be selected on the side view image, and based on the mapping relationship, a main view pixel corresponding to the pixel to be reconstructed can be determined on the main view image. Based on the main view pixel, super-resolution reconstruction is performed on the pixel to be reconstructed to obtain the reconstructed pixel, thereby obtaining the reconstructed side view image.

[0076] In this embodiment, the mapping relationship can be used as a guiding parameter, thereby enabling the reconstruction of the side view image based on the front view image. This makes the reconstructed side view image more accurate in representing the object under test, reducing the probability of image distortion due to image reconstruction.

[0077] In addition, combined with the iterations mentioned above, the mapping relationship is redefined in each iteration. By improving the accuracy of the mapping relationship, the quality of the reconstructed side view image can be improved, making the target side view image more accurate.

[0078] In one possible implementation, determining the quality index of the reconstructed side view image includes: dividing the reconstructed side view image into multiple first regions; and determining a first quality index for each of the first regions.

[0079] In this embodiment of the disclosure, a first window can be set and slid across the reconstructed side view image. The horizontal sliding step size can be the horizontal side length of the first window, and the vertical sliding step size can be the vertical side length of the first window. By sliding the first window across the reconstructed side view image, multiple first regions can be obtained, and these multiple first regions can together form the reconstructed side view image.

[0080] The quality metric may include a first quality metric characterizing the quality of the first region. In embodiments of this disclosure, a quality metric for each pixel in the reconstructed side view image may be determined. For ease of description, the quality metric for a single pixel in the reconstructed side view image is referred to as a third quality metric.

[0081] For example, a first quality assessment model can be pre-trained. The reconstructed side view image is input into the first quality assessment model to obtain a third quality index corresponding to each pixel in the reconstructed side view image. These third quality indices can be represented by a first confidence map. The size and number of pixels of the first confidence map are the same as those of the reconstructed side view image.

[0082] Each first region can correspond to multiple third quality indicators. The median of the multiple third quality indicators corresponding to the first region, or the first average of the remaining third quality indicators after removing the maximum and minimum values ​​from the multiple third quality indicators, can be used as the first quality indicator for the first region.

[0083] In this embodiment, dividing the reconstructed side view image into multiple first regions and determining a first quality index for each region can improve the accuracy and precision of the quality evaluation of the reconstructed side view image. Furthermore, compared to directly using a third quality index, determining the first quality index on a first-region basis reduces the workload of subsequent steps in determining whether quality conditions are met, thus improving efficiency. Therefore, a balance can be achieved between pursuing accuracy, precision, and efficiency in the target side view image.

[0084] In one possible implementation, the quality condition includes a first quality condition, and determining an optimized side view image based on the reconstructed side view image when the quality index does not meet the quality condition includes: using the reconstructed side view image as the optimized side view image when any of the first quality indexes does not meet the first quality condition.

[0085] In this embodiment of the disclosure, the first quality condition can be a first quality index greater than a first quality threshold. The first quality threshold can be the minimum quality of the first region under the condition of meeting the scenario requirements.

[0086] As mentioned earlier, a quality evaluation can be performed on each first region to obtain a primary quality index characterizing the quality of the first region. The primary quality indices can be sorted from smallest to largest according to their values. Then, based on this sorting, each primary quality index is checked to see if it meets the primary quality condition. When the first primary quality index is found to be non-compliant, subsequent primary quality indices will also fail to meet the primary quality condition, allowing for direct execution of the next steps. This saves time and improves efficiency.

[0087] If any of the first quality indicators fails to meet the first quality condition, the reconstructed side view image can be used as the optimized side view image. This allows for the determination of a new side view image. The quality of the new side view image is improved, the number of iterations is reduced, and efficiency is increased.

[0088] In one possible implementation, determining the quality index of the reconstructed side view image includes: downsampling the reconstructed side view image to obtain a low-resolution image, the low-resolution image having the same resolution as the side view image; dividing the low-resolution image into multiple second regions; and determining a second quality index for each of the second regions.

[0089] In this embodiment of the disclosure, the resolution of the reconstructed side view image can be reduced by downsampling processing to obtain a low-resolution image. Furthermore, the low-resolution image and the side view image can have the same resolution.

[0090] In this embodiment, a second window can be set and slid across the low-resolution image. The horizontal sliding step size can be the horizontal side length of the second window, and the vertical sliding step size can be the vertical side length of the second window. By sliding the second window across the low-resolution image, multiple second regions can be obtained, and these multiple second regions can together form the low-resolution image.

[0091] The quality metric may include a second quality metric characterizing the quality of the second region. In embodiments of this disclosure, a quality metric for each pixel in the low-resolution image can be determined. For ease of description, the quality metric for a single pixel in the low-resolution image is referred to as the fourth quality metric.

[0092] For example, a second quality assessment model can be pre-trained. The low-resolution image is input into the second quality assessment model to obtain a fourth quality index corresponding to each pixel in the low-resolution image. These fourth quality indices can be represented by a second confidence map. The size and number of pixels of the second confidence map are the same as those of the low-resolution image.

[0093] Each second region can correspond to multiple fourth quality indicators. The median of the multiple fourth quality indicators corresponding to the second region, or the second average of the remaining fourth quality indicators after removing the maximum and minimum values ​​from the multiple fourth quality indicators, can be used as the second quality indicator for the second region.

[0094] In this embodiment, the resolution of the reconstructed side view image is first reduced to obtain a low-resolution image, thereby reducing the amount of data and processing. The low-resolution image is then divided into multiple second regions, and a second quality index is determined for each region. This allows for maintaining the accuracy and precision of the low-resolution image quality evaluation while reducing processing load. Thus, a balance can be achieved between pursuing accuracy, precision, and efficiency in the target side view image.

[0095] In one possible implementation, the quality condition includes a second quality condition. The step of determining an optimized side view image based on the reconstructed side view image when the quality indicator does not meet the quality condition includes: dividing the side view image into multiple third regions; determining a target second region where the second quality indicator does not meet the second quality condition; and replacing the target second region with a third region corresponding to the target second region to obtain the optimized side view image.

[0096] The second quality condition can be that the second quality index is greater than the second quality threshold. The second quality threshold can be the minimum quality of the second region while meeting the requirements of the scenario.

[0097] In this embodiment of the disclosure, a second window can be used to slide on the side view image to obtain multiple third regions, which together constitute the side view image. The second and third regions can correspond to each other.

[0098] As mentioned earlier, quality evaluation can be performed on each second region to obtain a second quality index characterizing the quality of the second region. The second quality indices can be sorted from smallest to largest according to their values. Following this sorting, the first quality index is then checked to see if it meets the first quality condition. When the first second quality index is found to be unsatisfactory, subsequent second quality indices will also fail to meet the second quality condition, thus allowing all target second regions to be identified simultaneously. This saves time and improves efficiency.

[0099] A third region corresponding to each target's second region can be identified. The second region of each target in the low-resolution image is then replaced with its corresponding third region to obtain an optimized side view image. Furthermore, a new side view image can be determined.

[0100] In this embodiment, only the target second region that does not meet the second quality condition is replaced. This improves quality while reducing the data volume of the optimized side view image. It also improves the efficiency of subsequent data transmission and processing.

[0101] In one possible implementation, determining the quality index of the reconstructed side view image includes: determining sub-indicators, which include the following: stripe pattern period, foreground and background contrast of the stripe pattern, grayscale variation range of the stripe pattern, position of the pixel in the stripe pattern period, material of the object under test, brightness distribution of the stripe pattern, and quality of neighboring pixels; and determining the quality index based on the sub-indicators.

[0102] When determining quality indicators (or the third or fourth quality indicator), multiple sub-indicators can be identified first. Methods that can be used to obtain multiple sub-indicators include: principal component analysis, autoencoder, independent component analysis, singular value decomposition, factor analysis, etc.

[0103] In this embodiment, the weights corresponding to each sub-indicator can be calibrated using a calibration module, or the weights corresponding to each sub-indicator can be inferred using deep learning. These weights are then used to perform a weighted summation of the sub-indicators to obtain the quality index.

[0104] Combining the first quality assessment model mentioned earlier, the weights corresponding to each sub-indicator can be built into the first quality assessment model. For a single pixel in the reconstructed side view image, the sub-indicators of that pixel are determined, and the weights are used to perform a weighted summation of each sub-indicator to obtain the third quality index of that pixel, thereby obtaining the first confidence map.

[0105] Combining the previously mentioned second quality assessment model, the weights corresponding to each sub-indicator can be built into the second quality assessment model. For a single pixel in a low-resolution image, the sub-indicators of that pixel are determined, and the weights are used to perform a weighted summation of the sub-indicators to obtain the fourth quality index of that pixel, thus obtaining the second confidence map.

[0106] In this embodiment, multiple sub-indicators can be determined for multiple dimensions, and a weighted average of these sub-indicators can be performed to determine the quality index. This makes the quality evaluation of side view images more comprehensive, objective, and reliable.

[0107] Figure 3 This is a flowchart illustrating another method for enhancing striped projection images based on multi-view stereo vision and super-resolution, provided in an embodiment of this disclosure.

[0108] The projection device projects stripes onto the object under test. A main-view camera and multiple side-view cameras take pictures of the object to obtain a main-view image and side-view images from different angles. The main-view camera can be positioned directly above the object, while the multiple side-view cameras are positioned at different locations on the object.

[0109] Establish mapping relationship

[0110] The front view image and the side view image can constitute a multi-view stereo vision system. Based on this multi-view stereo vision system, the pixel-level mapping relationship between the front view image and the side view image can be determined.

[0111] Super-resolution reconstruction

[0112] Using the mapping relationship as a guiding parameter, the side view image can be super-resolution reconstructed using the front view image as a reference, resulting in a reconstructed side view image.

[0113] Quality assessment

[0114] A pre-trained evaluation model can be used to assess the quality metric of the reconstructed side view image. This quality metric can be either a first quality metric or a second quality metric.

[0115] If the quality indicators meet the quality conditions, the side view image can be used as the target side view image. If the quality indicators do not meet the quality conditions, an optimized side view image can be determined based on the target image. The optimized side view image is then used as the new side view image. The steps of establishing mapping relationships, super-resolution reconstruction, and quality evaluation are iteratively performed until the quality indicators meet the quality conditions. The side view image obtained in the last iteration is then used as the target side view image.

[0116] In this way, multiple target side view images can be obtained. Based on the front view image and multiple target side view images, the shape data of the object under test can be determined.

[0117] The method disclosed herein can improve measurement accuracy without increasing hardware costs.

[0118] Figure 4 A schematic diagram of the structure of a striped projection image enhancement device based on multi-view stereo vision and super-resolution provided in an embodiment of this disclosure. The device 20 includes:

[0119] Image acquisition unit 21 is used to acquire a front view image and a side view image of the object under test, which includes a projected striped pattern.

[0120] The reconstructed side view image determination unit 22 is used to reconstruct the side view image to obtain a reconstructed side view image, wherein the quality of the reconstructed side view image is better than that of the side view image.

[0121] Quality index determination unit 23 is used to determine the quality index of the reconstructed side view image;

[0122] The target side view image determination unit 24 is used to determine an optimized side view image based on the reconstructed side view image when the quality index does not meet the quality conditions, and to use the optimized side view image as a new side view image, iteratively performing the operation from reconstruction to the determination of the optimized side view image until the quality index of the reconstructed side view image meets the quality conditions, thereby obtaining a target side view image. The target side view image is used to determine the morphological data of the object under test in combination with the front view image.

[0123] In one possible implementation, the reconstructed side view image determination unit 22 is further configured to:

[0124] Determine depth data based on the front view image and the side view image;

[0125] Based on the depth data, the mapping relationship between the front view image and the side view image is determined;

[0126] Using the mapping relationship as a guiding parameter, the side view image is reconstructed based on the front view image to obtain the reconstructed side view image.

[0127] In one possible implementation, the quality index determination unit 23 is further configured to:

[0128] The reconstructed side view image is divided into multiple first regions;

[0129] Determine the first quality index for each of the first regions.

[0130] In one possible implementation, the quality condition includes a first quality condition, and the target side view image determination unit 24 is further configured to:

[0131] If any of the first quality indicators fails to meet the first quality condition, the reconstructed side view image shall be used as the optimized side view image.

[0132] In one possible implementation, the quality index determination unit 23 is further configured to:

[0133] The reconstructed side view image is downsampled to obtain a low-resolution image, which has the same resolution as the side view image.

[0134] The low-resolution image is divided into multiple second regions;

[0135] Determine the second quality index for each of the second regions.

[0136] In one possible implementation, the quality condition includes a second quality condition, and the target side view image determination unit 24 includes:

[0137] The side view image is divided into multiple third regions;

[0138] Identify the target second region where the second quality indicator does not meet the second quality condition;

[0139] The second target region is replaced with a third region corresponding to the second target region to obtain the optimized side view image.

[0140] In one possible implementation, the quality index determination unit 23 is further configured to:

[0141] The sub-indicators are determined, including the following: stripe pattern period, foreground and background contrast of the stripe pattern, grayscale variation range of the stripe pattern, position of the pixel in the stripe pattern period, material of the object under test, brightness distribution of the stripe pattern, and quality of neighboring pixels.

[0142] The quality indicators are determined based on the aforementioned sub-indicators.

[0143] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0144] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0145] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0146] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0147] Figure 5 This is a schematic diagram of the structure of an electronic device for enhancing fringe projection images based on multi-view stereo vision and super-resolution, as provided in an embodiment of this disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. (Refer to...) Figure 5The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0148] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0149] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0150] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0151] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0152] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0153] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0154] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0155] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0157] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for enhancing fringe projection images based on multi-view stereo vision and super-resolution, characterized in that, include: Acquire a front view image and a side view image of the object under test, which include the projected striped pattern; The side view image is reconstructed to obtain a reconstructed side view image, the quality of which is better than that of the original side view image; Determine the quality index of the reconstructed side view image; If the quality index does not meet the quality conditions, an optimized side view image is determined based on the reconstructed side view image. The optimized side view image is used as a new side view image, and the operation of reconstructing to the determined optimized side view image is iteratively performed until the quality index of the reconstructed side view image meets the quality conditions, and a target side view image is obtained. The target side view image is used to determine the shape data of the object under test in combination with the front view image. The process of reconstructing the side view image to obtain a reconstructed side view image includes: Determine depth data based on the front view image and the side view image; Based on the depth data, the mapping relationship between the front view image and the side view image is determined; Using the mapping relationship as a guiding parameter, the side view image is reconstructed based on the front view image to obtain the reconstructed side view image; Determining the quality index of the reconstructed side view image includes: The sub-indicators are determined, including the following: stripe pattern period, foreground and background contrast of the stripe pattern, grayscale variation range of the stripe pattern, position of the pixel in the stripe pattern period, material of the object under test, brightness distribution of the stripe pattern, and quality of neighboring pixels. The quality indicators are determined based on the aforementioned sub-indicators.

2. The method according to claim 1, characterized in that, Determining the quality index of the reconstructed side view image includes: The reconstructed side view image is divided into multiple first regions; Determine the first quality index for each of the first regions.

3. The method according to claim 2, characterized in that, The quality conditions include a first quality condition. The step of determining an optimized side view image based on the reconstructed side view image when the quality index does not meet the quality conditions includes: If any of the first quality indicators fails to meet the first quality condition, the reconstructed side view image shall be used as the optimized side view image.

4. The method according to claim 1, characterized in that, Determining the quality index of the reconstructed side view image includes: The reconstructed side view image is downsampled to obtain a low-resolution image, which has the same resolution as the side view image. The low-resolution image is divided into multiple second regions; Determine the second quality index for each of the second regions.

5. The method according to claim 4, characterized in that, The quality conditions include a second quality condition. The step of determining an optimized side view image based on the reconstructed side view image when the quality indicators do not meet the quality conditions includes: The side view image is divided into multiple third regions; Identify the target second region where the second quality indicator does not meet the second quality condition; The second target region is replaced with a third region corresponding to the second target region to obtain the optimized side view image.

6. A striped projection image enhancement device based on multi-view stereo vision and super-resolution, characterized in that, include: The image acquisition unit is used to acquire a front view image and a side view image of the object under test, which includes a projected striped pattern. A reconstructed side view image determination unit is used to reconstruct the side view image to obtain a reconstructed side view image, wherein the quality of the reconstructed side view image is better than that of the original side view image. A quality index determination unit is used to determine the quality index of the reconstructed side view image; The target side view image determination unit is used to determine an optimized side view image based on the reconstructed side view image when the quality index does not meet the quality conditions. The optimized side view image is used as a new side view image, and the operation of reconstructing to determine the optimized side view image is performed iteratively until the quality index of the reconstructed side view image meets the quality conditions, thereby obtaining a target side view image. The target side view image is used to determine the morphological data of the object under test in combination with the front view image. The reconstructed side view image determination unit is further used for: Determine depth data based on the front view image and the side view image; Based on the depth data, the mapping relationship between the front view image and the side view image is determined; Using the mapping relationship as a guiding parameter, the side view image is reconstructed based on the front view image to obtain the reconstructed side view image; In one possible implementation, the quality index determination unit is further configured to: The reconstructed side view image is divided into multiple first regions; Determine the first quality indicator for each of the first regions; The quality index determination unit is further used for: The sub-indicators are determined, including the following: stripe pattern period, foreground and background contrast of the stripe pattern, grayscale variation range of the stripe pattern, position of the pixel in the stripe pattern period, material of the object under test, brightness distribution of the stripe pattern, and quality of neighboring pixels. The quality indicators are determined based on the aforementioned sub-indicators.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

8. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images