Depth map fusion method

Abstract

The invention provides a depth map fusion method, which comprises the following steps of: acquiring a plurality of depth maps of a plurality of visual angles for a target object, and determining a corresponding grey-scale map according to each depth map; for each grey-scale map, extracting a frequency characteristic of the grey-scale map, determining a frequency score of the grey-scale map according to the frequency characteristic, extracting a gradient characteristic of the grey-scale map, and determining a gradient score according to the gradient characteristic; determining a comprehensive score of the grey-scale map according to the frequency score and the gradient score, and optimizing a depth map corresponding to the grey-scale map according to the comprehensive score and a preset threshold range; and fusing the plurality of optimized depth images to obtain a fused image.

Description

Technical Field

[0001] The present invention relates to the field of automatic optical detection, and in particular to a depth map fusion method. Background Art

[0002] Existing automatic optical inspection (AOI) systems often encounter problems such as holes, noise, and loss of depth information in reflective areas when performing 3D inspection on complex surfaces (such as electronic components, reflective materials, and complex mechanical parts). This is because the reconstruction of the depth map from a single perspective has limitations in terms of occlusion and light incident angle, resulting in the inability to accurately obtain 3D information in certain areas. In addition, highly reflective surfaces can generate excessive optical noise, resulting in depth estimation errors.

[0003] Therefore, a depth map fusion method is needed. Summary of the invention

[0004] The purpose of the present invention is to provide a depth map fusion method, which obtains depth maps and grayscale images of multiple perspectives through multiple telecentric lens cameras, fuses the depth maps according to frequency analysis and gradient scoring, and solves the problems of depth map holes, noise and data loss generated when the prior art processes complex surfaces, highly reflective areas and shadow areas, thereby improving the three-dimensional reconstruction accuracy of the AOI system.

[0005] To achieve the above object, the present invention provides a depth map fusion method, comprising:

[0006] Acquire multiple depth maps of multiple viewing angles for the target object, and determine a corresponding grayscale image according to each depth map;

[0007] For each grayscale image, extract the frequency features of the grayscale image, determine the frequency score of the grayscale image according to the frequency features, and extract the gradient features of the grayscale image, determine the gradient score according to the gradient features; determine the comprehensive score of the grayscale image according to the frequency score and the gradient score, and optimize the depth map corresponding to the grayscale image according to the comprehensive score and a preset threshold range;

[0008] The optimized multiple depth maps are fused to obtain a fused map.

[0009] Preferably, extracting the frequency characteristics of the grayscale image and determining the frequency score of the grayscale image based on the frequency characteristics includes: performing a two-dimensional Fourier transform on the grayscale image to obtain a spectrum diagram of the grayscale image, and determining the frequency score based on the frequency distribution in the spectrum diagram.

[0010] Preferably, extracting the gradient features of the grayscale image and determining the gradient score based on the gradient features includes: obtaining the gradient strength of the grayscale image by using a Sobel operator, and determining the gradient score based on the gradient strength of the grayscale image.

[0011] Preferably, optimizing the depth map corresponding to the grayscale image includes: determining whether the comprehensive score is within the threshold range, and if it is not within the threshold range, adjusting the exposure value of the depth map, and determining whether to continue adjusting the exposure value of the depth map based on the comprehensive score of the grayscale image corresponding to the adjusted depth map.

[0012] Specifically, fusing multiple optimized depth maps to obtain a fused map includes: performing outlier depth value detection on each optimized depth map, determining invalid depth values ​​and valid depth values ​​in each depth map, and deleting invalid depth values ​​from each depth map; and performing weighted fusion on the valid depth values ​​in the multiple depth maps to obtain the fused map.

[0013] Specifically, outlier depth value detection is performed to determine invalid depth values ​​and valid depth values ​​in each depth map, including:

[0014] Determine the median deviation of the depth values ​​in the depth map, and determine the invalid depth values ​​and valid depth values ​​in each depth map according to the median deviation and a preset outlier threshold; the outlier detection is based on the median absolute deviation (MAD) method, including:

[0015] The median deviation is calculated by the following formula:

[0016]

[0017] Where MAD represents the median deviation, N represents the total number of depth values ​​in the depth map, and d i Represents the depth value of the image, and median represents the median of the depth value set.

[0018] Specifically, weighted fusion is performed on the effective depth values ​​in the multiple depth maps to obtain the fused map, including:

[0019] For each image position in the fusion image, if the grayscale values ​​of multiple depth values ​​corresponding to the image position in multiple depth maps are all higher than a preset maximum grayscale value, the depth value of the image position is the smallest value among the multiple depth values; if the grayscale values ​​of multiple depth values ​​corresponding to the image position in multiple depth maps are all lower than the preset minimum grayscale value, the depth value of the image position is the largest value among the multiple depth values.

[0020] Preferably, performing weighted fusion on the effective depth values ​​in the multiple depth maps to obtain the fused map includes: performing adaptive filtering on a fusion result of weighted fusion of the effective depth values ​​in the multiple depth maps to obtain the fused map.

[0021] Specifically, the target object has one or more of a high-reflection area, a shadow area, and an occluded area.

[0022] Specifically, the fusion graph is used for automatic optical inspection (AOI), and the automatic optical inspection includes at least one or more of electronic component inspection and mechanical parts inspection.

[0023] Compared with the prior art, the present invention has the following advantages: the present invention can improve the depth reconstruction accuracy of reflective areas, effectively deal with the problem of reflective surfaces through multiple exposure technology, avoid overexposure or underexposure under single exposure conditions, and eliminate unreliable depth estimation through a multi-view verification mechanism combined with perspective and normal consistency verification, thereby reducing noise and hole problems in the depth map, and can be better applied to various complex industrial inspection scenarios, especially in electronic components, precision machined parts and other high-demand quality control scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of basic steps of a depth map fusion method provided by an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of a process for obtaining a benchmark score provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of an automatic exposure setting process and an image acquisition process provided by an embodiment of the present invention;

[0027] Figure 4 A schematic flow chart of a multi-projection depth fusion method combining a depth map and a corresponding grayscale map provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0030] In the description of the embodiments of the present invention, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0031] Existing AOI systems often encounter problems such as holes, noise, and loss of depth information in reflective areas when performing 3D inspection on complex surfaces such as electronic components, reflective materials, and complex mechanical parts. This is because the reconstruction of a single-view depth map has limitations in terms of occlusion and light incident angle, resulting in the inability to accurately obtain 3D information in certain areas. In addition, highly reflective surfaces can generate excessive optical noise, resulting in depth estimation errors.

[0032] To solve these problems, existing technologies have introduced methods of automatic exposure, multi-view and structured light projection. Multiple exposures can capture images under different brightness conditions by changing the exposure time, which is suitable for processing reflective areas and shadow areas. Multi-view verification increases the reliability of depth estimation and reduces holes by using multiple perspectives. However, the existing technology still has problems with inconsistent depth information and high noise when processing multi-view fusion, especially on complex surfaces and occluded areas. Developing a depth map fusion and filtering algorithm that combines automatic exposure, multi-view verification and a four-projector-telecentric lens camera can improve the detection accuracy of complex surfaces, reflective areas and shadow areas.

[0033] In order to overcome the shortcomings of the existing technology, a depth map fusion method is proposed. Figure 1 A schematic diagram of the basic steps of a depth map fusion method provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0034] S101: Acquire multiple depth maps of multiple perspectives of a target object, and determine a corresponding grayscale image according to each depth map;

[0035] For example, a telecentric lens camera and four structured light projectors are used to form an acquisition system. The camera is equipped with a telecentric lens to ensure that the captured image is not affected by the change in the distance between the object and the camera, eliminating geometric distortion. The four projectors surround the object and project structured light patterns from different angles to ensure that complex surfaces and highly reflective areas can be fully scanned. A frame of image is captured using the initial exposure setting, and the characteristics of the phase-shifted fringes are extracted through image processing technology.

[0036] S102: For each grayscale image, extract the frequency features of the grayscale image, determine the frequency score of the grayscale image according to the frequency features, and extract the gradient features of the grayscale image, determine the gradient score according to the gradient features; determine the comprehensive score of the grayscale image according to the frequency score and the gradient score, and optimize the depth map corresponding to the grayscale image according to the comprehensive score and a preset threshold range;

[0037] In one embodiment, frequency features of the grayscale image are extracted, and a frequency score of the grayscale image is determined based on the frequency features, including: performing a two-dimensional Fourier transform on the grayscale image to obtain a frequency spectrum of the grayscale image, and determining the frequency score based on the frequency distribution in the frequency spectrum.

[0038] For example, for the grayscale images I(x,y) acquired one by one, we use two-dimensional Fourier transform (FFT) to convert them into the frequency domain. Perform frequency fraction f score To calculate, first perform Fourier transform: Given a grayscale image I(x,y), the frequency domain representation can be obtained through two-dimensional Fourier transform:

[0039]

[0040] Where e is a natural constant, M and N are the width and height of the image, u and v are the coordinates in the frequency domain, F(u,v) is the frequency spectrum, and I(x,y) is the pixel value of the grayscale image;

[0041] After the spectrum is obtained, the amplitude can be determined according to the spectrum. In a specific embodiment, the amplitude of the obtained spectrum can be calculated by the following formula:

[0042]

[0043] Among them, |F(u,v)| is the amplitude of the spectrum, Re and Im are the real and imaginary parts respectively, and F(u,v) is the spectrum;

[0044] Finally, the frequency score is defined. The frequency score can be defined by the statistical characteristics of the spectrum. The following indicators are usually used: main frequency, frequency intensity, standard deviation;

[0045] In a specific embodiment, the main frequency can be calculated by the following formula:

[0046] f max = argmax (u,v) |F(u,v)|

[0047] Among them, |F(u,v)| is the amplitude of the spectrum, f maxThe frequency corresponding to the maximum amplitude in the spectrum is calculated as the main frequency; in a specific embodiment, the frequency intensity can be calculated by the following formula:

[0048]

[0049] Among them, |F(u,v)| is the amplitude of the spectrum, E is the frequency intensity, and is the sum of the squares of the calculated spectrum energy;

[0050] In a specific embodiment, the standard deviation can be calculated by the following formula:

[0051]

[0052] Where N represents the total number of frequencies of all images, f i Represents each image frequency, μ f represents the frequency mean, σ f is the standard deviation; calculate the standard deviation of the frequency distribution to reflect the concentration of the frequency;

[0053] In a specific embodiment, the comprehensive frequency score can be calculated by the following formula:

[0054]

[0055] Among them, e is a natural constant, f max is the main frequency in the spectrum, E is the frequency intensity, σ f is the standard deviation of the frequency distribution, ∈ is a small value to prevent zero division errors, ω1, ω2, ω3, ω4 are weight parameters, and f score is the comprehensive frequency score.

[0056] In one embodiment, extracting the gradient features of the grayscale image and determining the gradient score according to the gradient features includes: obtaining the gradient strength of the grayscale image by using a Sobel operator, and determining the gradient score according to the gradient strength of the grayscale image.

[0057] For example, the image gradient score G score It is used to measure the degree of grayscale change of the image and can reflect the edge clarity of the stripes. The larger the gradient, the more significant the grayscale change of the image and the higher the stripe clarity. First, use the Sobel operator to calculate the horizontal and vertical derivatives of the image to obtain the gradient strength. The formula is as follows:

[0058]

[0059] Among them, I(x,y) is the pixel value of the grayscale image, G x (x, y) is the gradient in the horizontal direction, G y (x, y) is the gradient in the vertical direction, and its partial derivative is calculated;

[0060] In a specific embodiment, the gradient amplitude calculation formula is then performed:

[0061]

[0062] Among them, G x (x, y) is the gradient in the horizontal direction, G y (x, y) is the gradient in the vertical direction, G(x, y) is the gradient amplitude of the grayscale image;

[0063] In a specific embodiment, the average gradient can be calculated by the following formula:

[0064]

[0065] Where M and N are the width and height of the image, respectively; G(x, y) is the gradient amplitude of the grayscale image; the average gradient is used to measure the intensity of the overall grayscale change of the image;

[0066] After obtaining the average gradient, the gradient score can be determined based on the average gradient. In a specific embodiment, the gradient score can be calculated by the following formula:

[0067]

[0068] Among them, G score is the gradient score, e is a natural constant, ω5 is the set weight parameter, is the average gradient;

[0069] The final comprehensive score can be combined with the frequency score and the gradient score:

[0070] final score =f score +G score

[0071] Among them, f score is the comprehensive frequency score, G score Score the gradient.

[0072] In one embodiment, optimizing the depth map corresponding to the grayscale image includes: determining whether the comprehensive score is within the threshold range, and if it is not within the threshold range, adjusting the exposure value of the depth map, and determining whether to continue adjusting the exposure value of the depth map based on the comprehensive score of the grayscale image corresponding to the adjusted depth map.

[0073] Figure 2 A schematic diagram of a process for obtaining a benchmark score provided by an embodiment of the present invention is shown in FIG. Figure 2As shown, a flat, non-reflective plane is first selected as a reference plane, and four groups of projectors project a phase shift image in a time-sharing manner to obtain the frequency and gradient scores of the image, which are used as the reference scores. Figure 3 A schematic diagram of an automatic exposure setting process and an image acquisition process provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the initial exposure value is set, and four groups of projectors project a phase shift map in time-sharing, and the frequency and gradient scores of the images corresponding to the four groups of projectors are obtained respectively, and it is judged whether the image is within the baseline score threshold range of the image. If it is within the baseline score threshold range, four groups of phase shift patterns are collected according to the exposure values ​​corresponding to the four groups of projectors at this time, and reconstructed using the phase shift method. If the score is lower than the lower threshold, the exposure value is increased (gradually increased, such as increasing by 10%). If the score is higher than the upper threshold, the exposure value is reduced (gradually reduced, such as reducing by 10%). During the adaptation process, ensure that the exposure value is within the range supported by the camera.

[0074] In one embodiment, multiple optimized depth maps are fused to obtain a fused map, including: performing outlier depth value detection on each optimized depth map, determining invalid depth values ​​and valid depth values ​​in each depth map, and deleting invalid depth values ​​from each depth map; performing weighted fusion on the valid depth values ​​in the multiple depth maps to obtain the fused map. Specifically, outlier depth value detection is performed to determine invalid depth values ​​and valid depth values ​​in each depth map, including: determining the median deviation of the depth value in the depth map, and determining the invalid depth value and valid depth value in each depth map according to the median deviation and a preset outlier threshold; the outlier detection is based on the median absolute deviation (MAD) method, including: the median deviation is calculated by the following formula:

[0075]

[0076] Where MAD represents the median deviation, N represents the total number of depth values ​​in the depth map, and d i Represents the depth value of the image, median represents the median of the depth value set; set the outlier threshold threshold, and remove the depth values ​​whose absolute difference exceeds threshold×MAD. Specifically, weighted fusion is performed on the effective depth values ​​in the multiple depth maps to obtain the fused map, including: for each image position in the fused map, if the grayscale values ​​of the multiple depth values ​​corresponding to the image position in the multiple depth maps are all higher than the preset maximum grayscale value, the depth value of the image position is the smallest value among the multiple depth values; if the grayscale values ​​of the multiple depth values ​​corresponding to the image position in the multiple depth maps are all lower than the preset minimum grayscale value, the depth value of the image position is the largest value among the multiple depth values.

[0077] Figure 4A schematic flow chart of a multi-projection depth fusion method combining a depth map and a corresponding grayscale map provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, after multi-view checking the initial depth maps of different perspectives and eliminating inconsistent depth estimates, the method refers to the multi-view reconstruction method, and the depth map of each perspective is combined with multiple depth maps and corresponding grayscale images. The depth information is fused and optimized through the following steps: First, read the corresponding grayscale image and depth map of the projector respectively, use the median absolute deviation (MAD) method to determine whether the depth value is an outlier, and set the outlier threshold threshold, the default value of which is 3.0, for example. Calculate the median of the depth value set median, calculate the median absolute deviation formula:

[0078]

[0079] Where MAD is the median absolute deviation, N represents the total number of depth values, and d i Represents the depth value of the image, and median represents the median of the depth value set; if the absolute difference of a depth value is greater than the outlier threshold multiplied by the median absolute deviation (threshold×MAD), the value is considered an outlier. According to the range of grayscale values ​​(maximum grayscale value and minimum grayscale value), select valid depth values ​​for fusion. If all grayscale values ​​are greater than the maximum grayscale value, select the minimum value of the corresponding depth value. If all grayscale values ​​are less than the minimum grayscale value, select the maximum value of the corresponding depth value. Otherwise, calculate the average value of the valid depth values, remove the outliers to obtain the final depth map, where the maximum grayscale value and the minimum grayscale value are selected as 220 and 50 respectively; output fused depth map: return the fused depth map for subsequent 3D reconstruction and other applications

[0080] S103: Fuse the multiple optimized depth maps to obtain a fused map.

[0081] In different embodiments, the generation process of the final depth map may be different. In a specific embodiment, weighted fusion is performed on the effective depth values ​​in the multiple depth maps to obtain the fused map, including: adaptively filtering the fusion result of weighted fusion of the effective depth values ​​in the multiple depth maps to obtain the fused map.

[0082] In different embodiments, the specific structure of the target object may be different. In a specific embodiment, the target object has one or more of a high-reflective area, a shadow area, and an occluded area. Specifically, the fusion graph is used for automatic optical inspection (AOI), and the automatic optical inspection includes at least one or more of electronic component inspection and mechanical part inspection.

[0083] It is understood that the method steps in the embodiments of the present invention can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0084] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0085] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A depth map fusion method, comprising: Acquire multiple depth maps of multiple viewing angles for the target object, and determine a corresponding grayscale image according to each depth map; For each grayscale image, extract the frequency features of the grayscale image, determine the frequency score of the grayscale image according to the frequency features, and extract the gradient features of the grayscale image, determine the gradient score according to the gradient features; determine the comprehensive score of the grayscale image according to the frequency score and the gradient score, and optimize the depth map corresponding to the grayscale image according to the comprehensive score and a preset threshold range; The optimized multiple depth maps are fused to obtain a fused map.

2. The method according to claim 1, wherein: Extracting the frequency features of the grayscale image and determining the frequency score of the grayscale image according to the frequency features includes: performing a two-dimensional Fourier transform on the grayscale image to obtain a frequency spectrum of the grayscale image, and determining the frequency score according to the frequency distribution in the frequency spectrum.

3. The method according to claim 1, wherein extracting the gradient features of the grayscale image and determining the gradient score according to the gradient features comprises: The gradient strength of the grayscale image is obtained by using a Sobel operator, and the gradient score is determined according to the gradient strength of the grayscale image.

4. The method according to claim 1, wherein: Optimizing the depth map corresponding to the grayscale image includes: determining whether the comprehensive score is within the threshold range; if it is not within the threshold range, adjusting the exposure value of the depth map; and determining whether to continue adjusting the exposure value of the depth map based on the comprehensive score of the grayscale image corresponding to the adjusted depth map.

5. The method according to claim 1, wherein: The method comprises fusing the optimized multiple depth maps to obtain a fused map, comprising: performing outlier depth value detection on each optimized depth map, determining invalid depth values ​​and valid depth values ​​in each depth map, and deleting invalid depth values ​​from each depth map; and performing weighted fusion on the valid depth values ​​in the multiple depth maps to obtain the fused map.

6. The method according to claim 5, wherein: Perform outlier depth value detection to determine invalid and valid depth values ​​in each depth map, including: Determine the median deviation of the depth values ​​in the depth map, and determine the invalid depth values ​​and valid depth values ​​in each depth map according to the median deviation and a preset outlier threshold; the outlier detection is based on the median absolute deviation (MAD) method, including: The median deviation is calculated by the following formula: Where MAD represents the median deviation, N represents the total number of depth values ​​in the depth map, and d i Represents the depth value of the image, and median represents the median of the depth value set.

7. The method according to claim 5, wherein: Performing weighted fusion on the effective depth values ​​in the multiple depth maps to obtain the fused map, comprising: For each image position in the fusion image, if the grayscale values ​​of multiple depth values ​​corresponding to the image position in multiple depth maps are all higher than a preset maximum grayscale value, the depth value of the image position is the smallest value among the multiple depth values; if the grayscale values ​​of multiple depth values ​​corresponding to the image position in multiple depth maps are all lower than the preset minimum grayscale value, the depth value of the image position is the largest value among the multiple depth values.

8. The method according to claim 1, wherein: Performing weighted fusion on the effective depth values ​​in the multiple depth maps to obtain the fused map includes: performing adaptive filtering on a fusion result of weighted fusion of the effective depth values ​​in the multiple depth maps to obtain the fused map.

9. The method according to claim 1, wherein: The surface of the target object has one or more of a high-reflection area, a shadow area and a shielded area.

10. The method according to claim 9, wherein: The fusion graph is used for automatic optical inspection (AOI), and the automatic optical inspection includes at least one or more of electronic component inspection and mechanical parts inspection.

Images