Multi-modal three-dimensional measurement method, system and medium

Abstract

The invention provides a multi-mode three-dimensional measurement method and system, a medium and an imaging system. According to the three-dimensional measurement method provided by the invention, according to the mixed spectral image acquired by the beam splitter prism camera, based on the relationship between the channel image and the imaging light path, the reflection type of the mixed spectral image is adaptively judged by calculating the contrast difference value of the pixels and the light path intensity, and then differential fusion strategies are adopted for different reflection regions, so that three-dimensional measurement is realized. The dual-mode information of fringe projection and phase deflection is synchronously obtained through single exposure, and signal crosstalk is completely eradicated from an imaging source; a self-adaptive fusion mechanism is provided according to the physical characteristics of the channel image, and the precision, robustness and anti-interference capability of three-dimensional measurement on a complex surface are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-modal three-dimensional measurement method, system and medium. BACKGROUND

[0002] In recent years, optical measurement of three-dimensional topography of object surface has become an important research direction in application fields such as precision machining, industrial detection and reverse engineering. The reflection properties of the material of the object surface mainly include diffuse reflection and specular reflection.

[0003] At present, three-dimensional measurement of the object surface is mostly carried out separately for a certain property. The structured light stripe projection technology is used for measuring the object surface with diffuse reflection property, and the phase deflection technique is used for measuring the object surface with specular reflection property. However, there are many composite surfaces in the actual production process, and diffuse reflection and specular reflection may exist at the same time. Therefore, a single method cannot obtain good three-dimensional topography measurement results.

[0004] Chinese patent CN120141345A proposes a device for acquiring an image of an object surface to be measured, an image selection method and a defect detection method. The patent combines the phase deflection technique and the stripe projection profilometry in a set of defect detection systems, and realizes the rapid switching of two three-dimensional measurement defect detection techniques for products with complex material and different surface roughness, thereby improving the detection efficiency. The patent only relates to defect detection, and is different from the technical problems, technical means and technical effects of the present application.

[0005] Chinese patent CN117685905A proposes a multi-modal fusion three-dimensional imaging method, device, equipment and storage medium. The patent projects an encoding pattern to the surface of an object to be measured through a projection device and a display device, respectively acquires reflection data and determines three-dimensional coordinates, calculates the modulation coefficients of all three-dimensional coordinates, sets a threshold value, and fuses the two three-dimensional coordinates into one. The patent is different from the technical problems of the present application.

[0006] Chinese patent CN120953500A discloses a complex reflective object surface reconstruction method and system based on multi-system point cloud fusion. The patent projects a target object based on a projection grating phase method and a phase deflection technique path, respectively, acquires the structure light patterns of both through a camera, solves the phase of the structure light patterns acquired by the camera, combines joint calibration information, respectively solves the absolute phase and completes independent three-dimensional measurement, identifies the high-reflective area combining the full-white map, realizes point cloud-level adaptive replacement and fusion, and finally generates a three-dimensional topography model. The patent is different from the technical problems of the present application.

[0007] The prior art realizes the fusion of the phase-shifting technology and the fringe projection profilometry three-dimensional measurement result by respectively performing three-dimensional measurement on the phase-shifting technology and the fringe projection profilometry, fusing the two reconstruction results based on the calibration result or the point cloud matching, and obtaining the final reconstruction result. The three-dimensional measurement results obtained by using different imaging principles may have defects in splicing results in the fusion, and the fusion accuracy and quality of the final reconstruction image are low. Moreover, the prior art does not consider the color imaging problem in three-dimensional imaging.

[0008] Regarding the color imaging of phase-shifting, Chinese patent CN120576681A discloses a phase-shifting measurement method and device based on four-step phase-shifting, which optimizes the traditional four-step phase-shifting and Gray code combination mode by improving the projection mode and pattern design, realizes independent projection in the RGB channel, greatly reduces the number of projection images, reduces the number of images required for three-dimensional measurement to a smaller number of frames, and meets the industrial high-speed detection demand. The patent and the present scheme are different in technical means.

[0009] Regarding the color imaging of fringe projection, Chinese patent CN115839677A discloses a high dynamic range object surface three-dimensional topography measurement method and system, which uses a color camera to separate the blue and green channel corresponding fringe images from the deformed blue fringe image by using the different color channel response of the single color fringe projection and the color image color channel separation technology, and uses the deformed blue uniform image to assign values to the blue-green channel mask, and then synthesizes a high dynamic fringe image. Then, a phase solving method is applied to realize the measurement of the high dynamic range object surface. In the patent, the color camera is used for imaging, which cannot avoid the color crosstalk caused by the interpolation calculation of the color camera image, and cannot guarantee the high resolution of each channel image, thereby affecting the accuracy of the reconstruction result. SUMMARY

[0010] The present application provides a multi-modal three-dimensional measurement method, system and medium, which at least solves one of the above technical problems.

[0011] To achieve the above object, the present application provides the following technical scheme:

[0012] The present application provides a multi-modal three-dimensional measurement method, system and medium, which at least solves one of the above technical problems.

[0013] The present application provides a multi-modal three-dimensional measurement method, system and medium, which at least solves one of the above technical problems.

[0014] The present application provides a multi-modal three-dimensional measurement method, system and medium, which at least solves one of the above technical problems.

[0015] In each light path image, a ratio of a difference between a maximum value and a minimum value of pixel values in each sliding window to a difference between a maximum value and a minimum value of pixels of a current image is taken as a contrast of a center pixel point of the current window, wherein the sliding window at least contains a pixel range corresponding to one period of a sinusoidal fringe;

[0016] A ratio of a pixel value of each pixel point in each light path image to a sum of pixel values of the same pixel position in all channel images is calculated as a light path intensity of the pixel point.

[0017] Based on each light path image and a fusion coefficient, a final height map is calculated to realize three-dimensional measurement.

[0018] Further, the light path image comprises: if the phase deflection light path or the fringe projection light path corresponds to multiple channel images, a channel image with high light path intensity is selected as the current light path image according to the light path intensity of the same light path in different channel images.

[0019] Further, the light path image comprises: if the phase deflection light path or the fringe projection light path corresponds to multiple channel images, the multiple channel images of the same light path are fused to obtain a fusion image as the current light path image.

[0020] Further, the reflection type comprises:

[0021] An absolute difference of the contrast of the same pixel position in two light path images is calculated.

[0022] If the absolute difference is greater than a preset difference threshold, the contrast of the phase deflection light path is greater than the contrast of the fringe projection light path, and the current position is a specular reflection dominant area; if the absolute difference is greater than the difference threshold, the contrast of the phase deflection light path is less than the contrast of the fringe projection light path, and the current position is a diffuse reflection dominant area; if the absolute difference is not greater than the difference threshold, it indicates that the current position is a mixed reflection.

[0023] Further, the fusion coefficient comprises:

[0024] If the absolute difference of the contrast is greater than the difference threshold, the fusion coefficient of the light path image with high contrast is 1, and the fusion coefficient of the light path image with low contrast is 0; if the absolute difference |Δδ| is not greater than the difference threshold a, the fusion coefficient of the light path image with high contrast is a+|Δδ|, and the fusion coefficient of the light path image with low contrast is 1-a-|Δδ|; wherein the value range of the fusion coefficient is [0, 1].

[0025] Further, the reflection type comprises:

[0026] An absolute difference of the light path intensity of the same pixel position in two light path images is calculated.

[0027] If the absolute difference of the light path intensity falls within the preset numerical range, it is determined that the current pixel point is a mixed reflection area; if it does not fall within the numerical range, the light path intensity of the phase deflection light path is greater than the light path intensity of the fringe projection light path, the current position is a specular reflection dominant area; the light path intensity of the phase deflection light path is less than the light path intensity of the fringe projection light path, and the current position is a diffuse reflection dominant area.

[0028] Further, the fusion coefficient includes: if the absolute difference of the light path intensity falls within the numerical range, the light path intensity is taken as the fusion coefficient; if it does not fall within the numerical range, the fusion coefficient of the light path image with large light path intensity is 1, and the fusion coefficient of the light path image with small light path intensity is 0.

[0029] Based on the same inventive concept, the application also proposes a multi-modal three-dimensional measurement system, comprising:

[0030] The light splitting prism camera collects multi-channel images.

[0031] The phase deflection light path and the fringe projection light path correspond to at least one channel image respectively.

[0032] The diffuse reflection light source irradiates the fringe pattern on the surface of the measured object to form a phase deflection light path with the light splitting prism camera.

[0033] The projector projects the fringe pattern on the surface of the measured object to form a fringe projection light path with the light splitting prism camera.

[0034] The fringe patterns generated by the diffuse reflection light source and the projector are only different in spectral wavelength.

[0035] The calculation module splits the multi-channel images into channel images according to the spectral wavelength to screen out each light path image.

[0036] The contrast and / or light path intensity of each pixel point in each light path image are calculated to determine the reflection type of the current pixel point and obtain the fusion coefficient of the current pixel point.

[0037] In each light path image, the ratio of the difference between the maximum and minimum pixel values in each sliding window to the difference between the maximum and minimum pixel values of the current image is taken as the contrast of the central pixel point of the current window; wherein the sliding window at least contains a pixel range corresponding to one period of sinusoidal fringe.

[0038] The ratio of the pixel value of each pixel point in each light path image to the sum of the pixel values of the same pixel position in all channel images is taken as the light path intensity of the pixel point.

[0039] The reconstruction module calculates the final height map based on each light path image and the fusion coefficient to realize three-dimensional measurement.

[0040] Further, the fringe pattern comprises:

[0041] The diffuse reflection light source and the fringe pattern generated by the projector are two different wavelengths in the visible light band;

[0042] Alternatively, the projector generates a fringe pattern in the infrared light band, and the diffuse reflection light source generates a fringe pattern in the visible light band.

[0043] In another aspect, the application also provides a computer readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above method.

[0044] Based on the same inventive concept, the application also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus,

[0045] The memory is used to store a computer program;

[0046] The processor is used to execute the program stored on the memory to implement the above method.

[0047] The beneficial effects of the embodiments of the application are as follows:

[0048] The three-dimensional measurement system based on FPP and PMD provided by the application simultaneously acquires double-mode information through single exposure, eliminates time delay and dynamic error caused by time-sharing measurement, improves detection efficiency, and is suitable for high-speed detection application scenarios. In addition, the image collected by single exposure is a mixed spectral image of PMD and FPP, and there is no registration error caused by object micro-motion, which provides a high-quality data basis for subsequent data fusion and improves the overall measurement accuracy.

[0049] The application physically combines two originally independent imaging light paths into a single camera through the binding relationship between the spatial angle and the spectral channel in the light splitting prism; this makes the entire system structure extremely compact and stable, avoids the complex external parameter calibration and data fusion problems caused by multiple cameras, and reduces the system cost and use threshold.

[0050] The application uses the physical light splitting principle of the light splitting prism camera, and the light rays incident on the light splitting prism at different angles form different imaging light paths, so that the signals of different imaging light paths reach the corresponding image sensors to form images of the corresponding light paths, which reduces the risk of signal crosstalk from the imaging source, and ensures the accuracy of subsequent reconstruction.

[0051] In the multi-modal three-dimensional measurement method provided in the application, according to the physical light splitting principle of the light splitting prism camera, the imaging corresponding to the phase deflection light path and the fringe projection light path is obtained; and according to the pixel values of the same pixel position in the corresponding light path images, the contrast is calculated; and based on the contrast difference value, the reflection type of the current pixel point is determined.

[0052] In addition, the application also proposes a strategy of judging the reflection type of the pixel point by the light path intensity, and taking the light path intensity as the fusion calculation, which is simple in calculation and high in efficiency.

[0053] In the three-dimensional measurement process, the fusion coefficients corresponding to different reflection types calculated based on the contrast difference value of the channel image are used as adaptive weight factors for height map calculation. In the specular reflection (main) area, the fusion coefficient corresponding to the phase deflection light path image is increased to make the phase deflection imaging dominant in the reconstruction result; in the diffuse reflection (main) area, the fusion coefficient corresponding to the fringe projection light path image is increased to make the fringe projection imaging dominant in the reconstruction result. The adaptive fusion mechanism based on physical characteristics can effectively suppress the noise or distortion of a single mode in a non-proficient area, comprehensively utilize the advantages of the two modes, and significantly enhance the reliability, robustness and overall anti-interference ability of the three-dimensional measurement result. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a schematic diagram of the principle of the phase deflection light path;

[0055] Figure 2 is a schematic diagram of the principle of the fringe projection light path;

[0056] Figure 3 is a flowchart of the multi-modal three-dimensional measurement method provided in the application;

[0057] Figure 4 is a structural schematic diagram of the electronic device provided in the application. DETAILED DESCRIPTION

[0058] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the application.

[0059] The terms used herein are only used to describe specific embodiments, and are not intended to limit the disclosure. The terms "include", "contain" and the like used herein indicate the existence of the described features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0060] All terms used herein, including technical and scientific terms, have the meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein are to be interpreted as having a meaning that is consistent with the context of the specification, and should not be interpreted in an idealized or overly formal sense.

[0061] As shown in Figure 1 The measurement system of Phase Measuring Deflectometry (PMD) generally consists of a light source, a camera and an object to be measured.

[0062] Generally, the display screen acts as a diffuse light source to illuminate the structured light pattern to the surface of the object to be measured, and the camera is responsible for capturing the distorted structured light pattern modulated by the surface of the object to be measured.

[0063] Structured light is usually used to add a feature pattern to a featureless surface to mark different positions on the surface. Common structured light patterns include phase-shifted fringes, Gray codes, speckles, etc. Among them, phase-shifted fringes are the most commonly used structured light pattern in phase measuring deflectometry due to their high resolution and strong anti-noise performance.

[0064] Assuming the distance between the display screen, the surface of the object to be measured and the camera is known, the surface gradient information of the object to be measured is obtained by using the slope equation.

[0065] Since the optical path is reversible, the light path is described in reverse order. Any light ray emitted by a camera image pixel is reflected from the s point on the reference plane to the q point on the display screen. If the angle between the s point on the surface of the object to be measured and the reference plane (reference plane) is θ, then the angle between the light rays sq and so emitted by the camera is 2θ.

[0066] In Δsqo, let ∠osq = α, according to the sine theorem, there is

[0067] If the period of the fringe on the display screen is P, then |oq| represents the phase change of the fringe pattern

[0068] According to the above formula, we have From this, the gradient tanθ of the surface of the object to be measured can be obtained.

[0069] Generally, if α ≈ 90° and θ is very small, the calculation formula can be simplified as

[0070] The simplified formula shows that the gradient information of the object to be measured is converted to the modulation phase of the reflected fringe through modulation. After the reflected fringe pattern is processed by phase extraction and unwrapping techniques, the horizontal and vertical phase offsets Δx , Δ y , tan2θ x , tan2θ y , tan2θ

[0071] As shown in the figure, the Fringe Projection Profilometry (FPP) measurement system is generally composed of a projector, a camera and a measured object. Figure 2

[0072] The projector imaging has the characteristics of large field of view and strong adaptability, and is suitable for obtaining the overall shape of the object, especially for the highly reconstructed stability of the diffuse reflection area.

[0073] The projector sequentially projects structured light patterns onto the surface of the measured object according to the set order, and the pattern adopts a typical phase coding scheme. This process corresponds to the projection grating phase method, which uses the phase distortion of the structured light on the surface of the measured object to restore the three-dimensional profile, and is particularly suitable for diffuse reflection surfaces. The camera synchronously collects the deformed image sequence of the pattern on the object surface, thereby providing data for subsequent phase calculation and three-dimensional point cloud generation.

[0074] Generally, the phase coding pattern, including but not limited to step fringe pattern, Gray code fringe pattern, sinusoidal fringe pattern, binary fringe pattern and phase shift fringe pattern, can be used for pixel-by-pixel coding of structured light coding patterns. Generally, four-step or eight-step phase shift fringe pattern is selected as the structured light pattern.

[0075] The specific principle is that by projecting the phase shift fringe onto the surface of the measured object, due to the height change of the surface of the measured object, the fringe will be deformed. By collecting these deformed fringe images and processing these images to obtain the phase representing the height information of the object, the three-dimensional profile information of the object is finally calculated.

[0076] Based on the above measurement principle, the present application proposes a multi-modal three-dimensional measurement system, comprising:

[0077] A split prism camera collects multi-channel images.

[0078] The phase deflection light path and the fringe projection light path correspond to at least one channel image respectively.

[0079] A diffuse reflection light source irradiates the fringe pattern on the surface of the measured object, and forms a phase deflection light path with the split prism camera.

[0080] A projector projects the fringe pattern on the surface of the measured object, and forms a fringe projection light path with the split prism camera. ​

[0081] The diffuse reflection light source and the fringe pattern generated by the projector only differ in spectral wavelength.

[0082] In this embodiment, the diffuse reflection light source and the fringe pattern generated by the projector only differ in corresponding spectral wavelength information, and other features (such as the frequency and size of the fringe) remain the same.

[0083] The fringe pattern generated by the diffuse reflection light source and the projector is two different wavelengths in the visible light band.

[0084] Preferably, the diffuse reflection light source generates a fringe pattern of a first wavelength λ1, and the projector generates a fringe pattern of a second wavelength λ2.

[0085] Preferably, the diffuse reflection light source generates a fringe pattern of two different wavelengths (a first wavelength λ1 and a second wavelength λ2), and the projector generates a fringe pattern of a third wavelength λ3.

[0086] Preferably, the diffuse reflection light source generates a fringe pattern of a first wavelength λ1, and the projector generates a fringe pattern of two wavelengths (a second wavelength λ2 and a third wavelength λ3).

[0087] Preferably, the projector can also generate a fringe pattern in the infrared light band (a first wavelength λ1), and the diffuse reflection light source generates a fringe pattern in the visible light band.

[0088] Preferably, the fringe pattern in the visible light band can be one wavelength (a second wavelength λ2) or two wavelengths (a second wavelength λ2 and a third wavelength λ3).

[0089] Preferably, if the fringe pattern in the infrared band is used, a corresponding infrared sensor needs to be set to obtain the imaging of the fringe projection light path.

[0090] The present application generates fringe patterns of different spectral information by two different imaging light paths, and the split-beam effect of the split-beam prism camera is used to split the incident mixed spectral signals to obtain the imaging of the two light paths.

[0091] The split-beam effect of the split-beam prism camera is from the split-beam prism. The split-beam prism splits the at least two different colors of incident light reflected by the surface of the object to be measured into different beam signals according to the spectrum, so that each color of beam signal is transmitted to an independent image sensor, so that a completely registered synchronous image can be obtained by a single exposure.

[0092] According to the measurement system proposed in the present application, the camera can directly collect the fringe pattern (mixed spectral image) generated by the phase deflection and the fringe projection, realize the isolation of the information at the physical layer, avoid the motion blur or resolution loss caused by the traditional time division multiplexing or space division multiplexing, and truly realize the single-frame three-dimensional capture of the dynamic scene.

[0093] The multi-modal three-dimensional measurement system further comprises:

[0094] The computing module splits the multi-channel image into channel images according to the spectral wavelengths to determine each light path image.

[0095] The contrast and / or light path intensity of each pixel point in each light path image is calculated, the reflection type of the current pixel point is determined, and the fusion coefficient of the current pixel point is obtained.

[0096] In each light path image, the ratio of the difference between the maximum and minimum pixel values in each sliding window to the difference between the maximum and minimum pixel values of the current image is taken as the contrast of the center pixel point of the current window. The sliding window contains at least a pixel range corresponding to one period of the sinusoidal fringe.

[0097] The ratio of the pixel value of each pixel point in each light path image to the sum of the pixel values of the same pixel position in all channel images is calculated as the light path intensity of the pixel point.

[0098] The reconstruction module calculates a final height map based on each light path image and the fusion coefficient to realize three-dimensional measurement.

[0099] Based on the above system, the embodiment proposes a multi-modal three-dimensional measurement method, as shown in Figure 3 The method comprises the following steps:

[0100] First, the phase deflection light path and the fringe projection light path are respectively pose calibrated to obtain a three-dimensional space mapping relationship between the phase deflection light path and the fringe projection light path.

[0101] The fringe projection light path takes a standard white diffuse reflection plate as a calibration object. The pose calibration process specifically comprises:

[0102] The relative position and attitude between the color camera and the projector are solved through a traditional FPP calibration algorithm (such as a translation calibration method), and a mapping lookup table (LUT) of phase values and three-dimensional coordinates is established.

[0103] The phase deflection light path is calibrated with a plane mirror as a calibration object, and the color camera observes a clear mirror image of the diffuse reflection plate through the plane mirror. The pose calibration process specifically comprises:

[0104] By moving the plane mirror or changing its attitude, and combining the intrinsic parameters of the color camera (the camera intrinsic parameters are known), the virtual position of the diffuse reflection plate in the camera coordinate system can be calibrated, and a relationship model between the gradient change of the object surface and the phase (or the displacement of the feature point) in the image is established to describe the functional relationship between the phase difference and the surface slope.

[0105] According to pose calibration of the phase deflecting light path and the fringe projection light path, a three-dimensional space mapping relationship between the two light paths can be obtained.

[0106] After calibration is completed, based on the above system, a multi-channel image collected by the spectrometer prism camera is obtained, and each channel image is split according to a spectral wavelength to determine each light path image; wherein the phase deflecting light path and the fringe projection light path correspond to at least one channel image respectively.

[0107] Preferably, the corresponding channel images are split according to the spectral wavelength.

[0108] For example, if the two light paths correspond to one wavelength respectively, the phase deflecting light path generates a fringe pattern of the first wavelength, which is imaged on the image sensor of the first channel, and the first channel image is the phase deflecting light path image; the fringe projection light path generates a fringe pattern of the second wavelength, which is imaged on the image sensor of the second channel, and the second channel image is the fringe projection light path image.

[0109] Preferably, if a projector or a diffuse reflection light source generates two or more wavelengths of fringe patterns, the light path corresponds to multiple channel images; one channel image corresponds to one wavelength. At this time, one light path image should be determined from the multiple channel images corresponding to the light path, so as to facilitate subsequent reconstruction calculation.

[0110] Preferably, if the phase deflecting light path or the fringe projection light path corresponds to multiple channel images, according to the light path intensity corresponding to different channel images under the same light path, the channel image with high light path intensity is selected as the current light path image.

[0111] Preferably, if the phase deflecting light path or the fringe projection light path corresponds to multiple channel images, the multiple channel images of the same light path are fused to obtain a fused image as the current light path image.

[0112] In this embodiment, the specific fusion method is not limited, and can be determined according to actual sampling. For example, one of the multiple channel images of the same light path is specified as the current light path image; or according to the quality evaluation of each channel image, a corresponding weight is given, and a fused image is calculated as the current light path image; or several channel images are randomly selected, and a fused image is calculated in the form of mean value or weight as the current light path image.

[0113] The above fusion method is only an example, and other calculation methods or recombination of the above fusion method can also be used to calculate the fused image.

[0114] Preferably, the above fusion calculation or screening method (the calculation method of the light path intensity is described below) can be used to determine one light path image from the multiple channel images corresponding to the light path, so as to facilitate subsequent reconstruction calculation.

[0115] If the current light path corresponds to only one channel, the corresponding channel image is taken as the current light path image, and the above process of determining the light path image is not needed.

[0116] The contrast and / or light path intensity of each pixel in each light path image is calculated, the reflection type of the current pixel is determined, and the fusion coefficient of the current pixel is obtained.

[0117] According to the application scenarios of phase deflection and fringe projection, the reflection types of the surface of the object to be measured can be divided into specular reflection, mixed reflection (specular reflection + diffuse reflection), and diffuse reflection.

[0118] Preferably, if the reflection type is determined by the light path intensity of the current pixel and the fusion coefficient is determined, it specifically includes:

[0119] The ratio of the pixel value of each pixel in each light path image to the sum of the pixel values of the same pixel position in all channel images is taken as the light path intensity of the pixel.

[0120] The light path intensity (spec) of any pixel is used to reflect the spectral composition of the pixel. By calculating the proportion of the pixel value of the current position in any channel (pixel ratio), the reflection type of the current position is indicated.

[0121] Specifically, it includes calculating the absolute difference of the light path intensity of the same pixel position in the two light path images;

[0122] If the absolute difference of the light path intensity falls within a preset numerical range, it is determined that the current pixel is in a mixed reflection region;

[0123] If it does not fall within the numerical range, the light path intensity of the phase deflection light path is greater than that of the fringe projection light path, and the current position is in a specular reflection dominant region; and the light path intensity of the phase deflection light path is less than that of the fringe projection light path, and the current position is in a diffuse reflection dominant region.

[0124] The preset numerical range represents the degree of refinement for separating the diffuse reflection region and the specular reflection region, which is set in advance according to the resolution of the current image and the application scenario. The larger the numerical range, the greater the proportion of the mixed reflection region, and the more blurred the distinction between the diffuse reflection region and the specular reflection region; the smaller the numerical range, the smaller the proportion of the mixed reflection region, and the more refined the distinction between the diffuse reflection region and the specular reflection region.

[0125] If there are two wavelength fringe patterns, the absolute difference between the light path intensity of the same pixel position in the first (wavelength) channel image and the light path intensity of the second (wavelength) channel image falls within the numerical range, and it is determined that the current pixel is in a mixed reflection region; if it does not fall within the numerical range, the dominant reflection type is determined according to the size of the light path intensity of each channel image.

[0126] Take visible light as an example, the diffuse reflection light source generates a fringe pattern of the first wavelength λ1, and the projector generates a fringe pattern of the second wavelength λ2.

[0127] If it does not fall into the numerical range, under the same position, the light path intensity of the first (wavelength) channel (spec_λ1) is much greater than the light path intensity of the second (wavelength) channel (spec_λ2), indicating that the current position is a specular reflection dominant area (spec_λ1>>spec_λ2); if the light path intensity of the first (wavelength) channel (spec_λ1) is much lower than the light path intensity of the second (wavelength) channel (spec_λ2), indicating that the current position is a diffuse reflection dominant area (spec_λ1<<spec_λ2).

[0128] If there are three wavelengths of fringe patterns. If the phase deflection light path or the fringe projection light path corresponds to multiple channel images, according to the light path intensity corresponding to the same light path of different channel images, the channel image with high light path intensity is selected as the current light path image.

[0129] For example, the diffuse reflection light source generates fringe patterns of two different wavelengths (the first wavelength λ1 and the second wavelength λ2), and the projector generates a fringe pattern of the third wavelength λ3. If the diffuse reflection light source generates a fringe pattern of one wavelength, and the projector generates fringe patterns of two wavelengths, the calculation method can be exchanged.

[0130] Preferably, the wavelength channel with high light path intensity can be selected as the imaging signal of the phase deflection light path; that is, the light path intensity of the pixel point in the phase deflection light path is max (spec_λ1, spec_λ2); whether the absolute difference between max (spec_λ1, spec_λ2) and the light path intensity spec_λ3 of the fringe projection light path falls within the numerical range is used to determine the reflection type of the current pixel point.

[0131] Preferably, if the phase deflection light path or the fringe projection light path corresponds to multiple channel images, multiple channel images of the same light path are fused to obtain a fused image as the current light path image.

[0132] For example, the diffuse reflection light source generates fringe patterns of two different wavelengths (the first wavelength λ1 and the second wavelength λ2), and the projector generates a fringe pattern of the third wavelength λ3. If the diffuse reflection light source generates a fringe pattern of one wavelength, and the projector generates fringe patterns of two wavelengths, the calculation method can be exchanged.

[0133] That is, the image of the first (wavelength) channel and the image of the second (wavelength) channel are fused to calculate a fused image as phase deflection light path imaging, and the light path intensity spec_λ1-λ2 of each pixel point in the fused image is calculated; whether the absolute difference between spec_λ1-λ2 and the light path intensity spec_λ3 of the fringe projection light path falls within a numerical range is compared to determine the reflection type of the current pixel point.

[0134] If the absolute difference of the light path intensity falls within the numerical range, the light path intensity is used as the fusion coefficient; if it does not fall within the numerical range, the fusion coefficient of the light path image with the larger light path intensity is 1, and the fusion coefficient of the light path image with the smaller light path intensity is 0.

[0135] Preferably, if the current pixel point is a diffuse reflection or a specular reflection dominant area, the image corresponding to the fringe projection light path or the specular reflection light path is directly used as the reconstruction basis.

[0136] If the current pixel point is a mixed reflection, the light path intensity corresponding to the current channel is used as the fusion coefficient of the current light path.

[0137] Preferably, the contrast of the current pixel point can also be calculated to determine the reflection type and fusion coefficient of the pixel point. Specifically, it includes:

[0138] In each light path image, the ratio of the difference between the maximum and minimum pixel values in each sliding window to the difference between the maximum and minimum pixel values of the current image is used as the contrast of the center pixel point of the current window.

[0139] Preferably, at least one period of sinusoidal fringe is used as the pixel range corresponding to the sliding window.

[0140] Any channel image in this application is a sinusoidal structured light image, the period of the sinusoidal fringe is 2π, and there are several periods in one (fringe) image, and the maximum and minimum values (i.e. amplitude) of each period are slightly different. In order to further improve the accuracy of the calculation result, a sinusoidal fringe pattern with high fringe period frequency can be selected to increase the number of sliding windows.

[0141] The contrast δ (δ ∈ [0, 1]) is calculated point by point in each light path image, and the difference Δδ (Δδ ∈ [-1, 1]) between the contrasts of the same pixel point in different light path images is used to determine the reflection type of the current pixel point.

[0142] Specifically, it includes calculating the absolute difference of the contrasts of the same pixel position in two light path images;

[0143] If the absolute difference is greater than the preset difference threshold, and the contrast of the phase deflection light path is greater than the contrast of the fringe projection light path, the current position is a specular reflection dominant region; if the absolute difference is greater than the difference threshold, and the contrast of the phase deflection light path is less than the contrast of the fringe projection light path, the current position is a diffuse reflection dominant region; if the absolute difference is not greater than the difference threshold, it indicates that the current position is a mixed reflection.

[0144] Wherein, the preset difference threshold a is used to distinguish the specular reflection dominant region and the diffuse reflection dominant region, and the size of the difference threshold increases with the increase of the absolute difference of the two light path images.

[0145] Wherein, the absolute difference is the absolute difference of the pixel mean value of the two light path images, or the absolute difference of the pixel median value, or the absolute difference between any specified pixel value.

[0146] Wherein, the absolute difference ΔI ∈ [0, 1] (the pixel value of the two light path images has been normalized to [0, 1]).

[0147] If there are two kinds of fringe patterns of different wavelengths.

[0148] Taking visible light as an example, a diffuse reflection light source generates a fringe pattern of the first wavelength λ1, and a projector generates a fringe pattern of the second wavelength λ2. There is a first contrast δ1 and a second contrast δ2 at the same pixel point.

[0149] The first contrast is the ratio of the difference between the maximum and minimum pixel values in each window in the channel image corresponding to the phase deflection light path to the difference between the maximum and minimum pixel values in the current channel image.

[0150] The second contrast is the ratio of the difference between the maximum and minimum pixel values in each window in the channel image corresponding to the fringe projection light path to the difference between the maximum and minimum pixel values in the current channel image.

[0151] If the contrast difference is greater than the difference threshold (1>Δδ>a), it indicates that the first contrast is greater than the second contrast, that is, the light energy at the current position mainly enters the PMD imaging light path through specular reflection, so the current position is a specular reflection dominant region.

[0152] If the contrast difference is less than the negative of the difference threshold (-1≤Δδ<-a), it indicates that the first contrast is less than the second contrast, that is, the light energy at the current position mainly enters the FPP imaging light path through diffuse reflection, so the current position is a diffuse reflection dominant region.

[0153] If the contrast difference is not less than the negative of the difference threshold and not greater than the difference threshold (-a≤Δδ≤a), it indicates that the light energy at the current position enters the imaging light path through the joint action of diffuse reflection and specular reflection, so the current position is a mixed region.

[0154] If the absolute difference of the contrast is greater than the difference threshold value, the fusion coefficient of the light path image with high contrast is 1, and the fusion coefficient of the light path image with low contrast is 0; if the absolute difference |Δδ| is not greater than the difference threshold value a, the fusion coefficient of the light path image with high contrast is a+|Δδ|, and the fusion coefficient of the light path image with low contrast is 1-a-|Δδ|; wherein the value range of the fusion coefficient is [0, 1].

[0155] Preferably, if the current pixel point is a diffuse reflection or a specular reflection dominant area, the image corresponding to the fringe projection light path or the specular reflection light path is directly taken as the reconstruction basis.

[0156] If the current pixel point is a mixed reflection (-a≤Δδ≤a), the fusion coefficient of the current pixel point is determined according to the contrast difference and the difference threshold value. Specifically, it includes:

[0157] If the contrast difference is greater than 0 (0<Δδ≤a), it indicates that the current position is a mixed reflection dominated by specular reflection, and the fusion coefficient w1 of the channel image (the first wavelength channel image) corresponding to the phase deflection light path is Δδ+a, and the fusion coefficient w2 of the channel image (the second wavelength channel image) corresponding to the fringe projection light path is 1-a-Δδ;

[0158] If the contrast difference is not greater than 0 (-a≤Δδ≤0), it indicates that the current position is a mixed reflection dominated by diffuse reflection, and the fusion coefficient w1 of the channel image (the first wavelength channel image) corresponding to the fringe light path is 1-a-|Δδ|, and the fusion coefficient w2 of the channel image (the second wavelength channel image) corresponding to the fringe projection light path is a+|Δδ|.

[0159] Wherein, w1+w2=1, w1∈[0,1], w2∈[0,1]; in the mixed reflection calculation, if the fusion coefficient (w1, w2) is greater than 1, the fusion coefficient is truncated to 1; if the fusion coefficient is less than 0, the fusion coefficient is truncated to 0.

[0160] Wherein, the difference threshold value a is calculated according to the absolute difference of the channel images corresponding to the two light paths, which can use a linear function or a nonlinear function (such as Sigmoid type) to constrain the value range of a. The calculation result of the fusion coefficient is smooth, and then the final fusion result is smooth, avoiding the discontinuous problem such as pixel mutation.

[0161] Preferably, if the phase deflection light path or the fringe projection light path corresponds to multiple channel images, according to the light path intensity corresponding to different channel images under the same light path, the channel image with high light path intensity is selected as the current light path image.

[0162] If there are three wavelength fringe patterns, the diffuse reflection light source generates two different wavelength (first wavelength λ1, second wavelength λ2) fringe patterns, and the projector generates a third wavelength λ3 fringe pattern. If the diffuse reflection light source generates a wavelength fringe pattern, and the projector generates two wavelength fringe patterns, the calculation method can be exchanged.

[0163] According to the screening result of the light path intensity, the light path intensity of the pixel point in the phase deflection light path is max (spec_λ1, spec_λ2), and the channel image corresponding to max (spec_λ1, spec_λ2) is taken as the phase deflection light path image. The contrast of the channel image is calculated as the contrast of the phase deflection light path. According to the calculation process of the above two wavelength channels, the subsequent calculation and judgment are completed, and the fusion coefficient corresponding to each light path is obtained.

[0164] Preferably, if the phase deflection light path or the fringe projection light path corresponds to multiple channel images, the multiple channel images of the same light path are fused to take the fused image as the current light path image.

[0165] If the diffuse reflection light source generates two different wavelength (first wavelength λ1, second wavelength λ2) fringe patterns, and the projector generates a third wavelength λ3 fringe pattern. If the diffuse reflection light source generates a wavelength fringe pattern, and the projector generates two wavelength fringe patterns, the calculation method can be exchanged.

[0166] That is, the image of the first wavelength channel and the image of the second wavelength channel are used to calculate the fused image, which is used for phase deflection light path imaging, and the contrast δ_λ1-λ2 of each pixel point in the fused image is calculated. The contrast difference between δ_λ1-λ2 and the contrast δ_λ3 of the fringe projection light path is calculated to determine the reflection type of the current pixel point and obtain the fusion coefficient of each pixel point.

[0167] Based on each light path image and the fusion coefficient, the final height map is calculated to realize three-dimensional measurement.

[0168] Based on the above reflection type determination process, if any light path is the dominant reflection area, the phase calculation of the light path image is not performed, and the light path image of the dominant reflection is directly used for three-dimensional reconstruction.

[0169] If the current light path image is a diffuse reflection dominant area, the phase calculation is directly performed on the image corresponding to the fringe projection light path to obtain the corresponding height map as the basis for three-dimensional reconstruction. If the current light path image is a specular reflection dominant area, the phase calculation is directly performed on the image corresponding to the phase deflection light path to obtain the corresponding height map as the basis for three-dimensional reconstruction.

[0170] If it is a mixed reflection, the phase calculation is performed on the two light path images respectively to obtain the corresponding height map. As follows:

[0171] According to the pose calibration, the phase deflection light path image is obtained by phase unwrapping to obtain a gradient map; then a first height map Z_pmd is reconstructed from the gradient field by using a frequency domain integration method or the like.

[0172] According to the pose calibration, the second height map Z_fpp is obtained by phase unwrapping of the fringe projection light path image by using the FPP algorithm.

[0173] The fusion height map Z_end is calculated based on the fusion coefficients w corresponding to the light path images.

[0174] Z_end=w_fpp×Z_fpp+w_pmd×Z_pmd is taken as the final height map Z.

[0175] In the formula, w_fpp is the fusion coefficient of the fringe projection light path image, and w_pmd is the fusion coefficient of the phase deflection light path image.

[0176] Preferably, the Z_end obtained by solving and acquiring can be subjected to smoothing filtering or missing area interpolation according to image requirements to obtain the final height map Z.

[0177] Based on the final height map, a three-dimensional point cloud (converted into a point cloud or a mesh model) is generated to realize three-dimensional measurement.

[0178] Based on the same inventive concept, the application further provides a computer readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to realize the method described above.

[0179] The application also provides an electronic device, as shown in the accompanying drawings, which comprises a processor 301, a communication interface 302, a memory 303 and a communication bus 304. Figure 4 The processor 301, the communication interface 302 and the memory 303 complete mutual communication through the communication bus 304.

[0180] The memory 303 is used for storing a computer program.

[0181] The processor 301 is used for executing the program stored in the memory 303 to realize the method described above.

[0182] The three-dimensional measurement method provided by the application realizes three-dimensional measurement according to the mixed spectral imaging obtained by the split prism camera, based on the relationship between the channel image and the imaging light path, by calculating the contrast difference value of the pixel and the light path intensity, adaptively judging the reflection type, and then using a differentiated fusion strategy for different reflection regions.

[0183] The application synchronously acquires the fringe projection and phase deflection dual-mode information through single exposure, and eliminates signal crosstalk from the imaging source; the adaptive fusion mechanism proposed according to the physical characteristics of the channel image significantly improves the precision, robustness and anti-interference ability of three-dimensional measurement on complex surfaces.

[0184] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0185] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal three-dimensional measurement method, characterized in that, include: Acquire multi-channel images from a beam splitter camera, and split the images of each channel according to the spectral wavelength to filter out the image of each optical path; wherein, the phase-deflection optical path and the fringe projection optical path each correspond to at least one channel image. Calculate the contrast and / or light intensity of each pixel in each optical path image, determine the reflection type of the current pixel, and obtain the fusion coefficient of the current pixel; In each optical path image, the ratio of the difference between the maximum and minimum pixel values ​​in each sliding window to the difference between the maximum and minimum pixel values ​​in the current image is used as the contrast of the center pixel of the current window; wherein, the sliding window contains at least a pixel range corresponding to a sine fringe period; The ratio of the pixel value of each pixel in each optical path image to the sum of the pixel values ​​at the same pixel position in all channel images is used as the optical path intensity of that pixel. Based on the image of each optical path and the fusion coefficient, the final height map is calculated to achieve three-dimensional measurement.

2. The three-dimensional measurement method according to claim 1, characterized in that, The optical path image includes: if the phase-deflected optical path or the fringe projection optical path corresponds to multiple channel images, the channel image with the higher optical path intensity is selected as the current optical path image based on the optical path intensity corresponding to different channel images under the same optical path.

3. The three-dimensional measurement method according to claim 1, characterized in that, The optical path image includes: if the phase-deflected optical path or the fringe projection optical path corresponds to multiple channel images, the multiple channel images of the same optical path are fused, and the fused image is used as the current optical path image.

4. The three-dimensional measurement method according to any one of claims 1-3, characterized in that, Reflection types include: Calculate the absolute difference in contrast between the same pixel location in two optical path images; If the absolute difference is greater than the preset difference threshold, and the contrast of the phase-deflection optical path is greater than the contrast of the fringe projection optical path, then the current position is a region dominated by specular reflection; if the absolute difference is greater than the difference threshold, and the contrast of the phase-deflection optical path is less than the contrast of the fringe projection optical path, then the current position is a region dominated by diffuse reflection; if the absolute difference is not greater than the difference threshold, it indicates that the current position is a region dominated by mixed reflection.

5. The three-dimensional measurement method according to claim 4, characterized in that, Fusion coefficient, including: If the absolute difference in contrast is greater than the difference threshold, the fusion coefficient of the optical path image with high contrast is 1, and the fusion coefficient of the optical path image with low contrast is 0; if the absolute difference |Δδ| is not greater than the difference threshold a, the fusion coefficient of the optical path image with high contrast is a+|Δδ|, and the fusion coefficient of the optical path image with low contrast is 1-a-|Δδ|; where the value range of the fusion coefficient is [0,1].

6. The three-dimensional measurement method according to any one of claims 1-3, characterized in that, Reflection types include: Calculate the absolute difference in light path intensity at the same pixel location in two light path images; If the absolute difference in light path intensity falls within a preset range, the current pixel is determined to be a mixed reflection region; if it does not fall within the range, and the light path intensity of the phase-deflection light path is greater than that of the fringe projection light path, the current position is a specular reflection dominant region; if the light path intensity of the phase-deflection light path is less than that of the fringe projection light path, the current position is a diffuse reflection dominant region.

7. The three-dimensional measurement method according to claim 6, characterized in that, The fusion coefficient includes: if the absolute difference in optical path intensity falls within the numerical range, then the optical path intensity is used as the fusion coefficient; if it does not fall within the numerical range, the fusion coefficient of the optical path image with higher optical path intensity is 1, and the fusion coefficient of the optical path image with lower optical path intensity is 0.

8. A multimodal three-dimensional measurement system, characterized in that, include: A beam-splitting prism camera acquires multi-channel images; The phase-deflection optical path and the fringe projection optical path each correspond to at least one channel image; A diffuse reflection light source illuminates the stripe pattern onto the surface of the object under test, forming a phase-deflecting light path with the beam-splitter camera; The projector projects the striped pattern onto the surface of the object to be measured, forming a striped projection light path with the beam splitter camera; Among them, the diffuse reflection light source and the stripe pattern generated by the projector differ only in their spectral wavelengths; The calculation module splits the multi-channel image into individual channel images based on the spectral wavelength in order to filter out the image of each optical path; Calculate the contrast and / or light intensity of each pixel in each optical path image, determine the reflection type of the current pixel, and obtain the fusion coefficient of the current pixel; In each optical path image, the ratio of the difference between the maximum and minimum pixel values ​​in each sliding window to the difference between the maximum and minimum pixel values ​​in the current image is used as the contrast of the center pixel of the current window; wherein, the sliding window contains at least a pixel range corresponding to a sine fringe period; The ratio of the pixel value of each pixel in each optical path image to the sum of the pixel values ​​at the same pixel position in all channel images is used as the optical path intensity of that pixel. The reconstruction module calculates the final height map based on the image of each optical path and the fusion coefficient, thus realizing three-dimensional measurement.

9. The three-dimensional measurement system according to claim 8, characterized in that, Striped patterns, including: The stripe pattern generated by the diffuse light source and the projector represents two different wavelengths in the visible light band. Alternatively, the projector generates a striped pattern in the infrared band, while the diffuse light source generates a striped pattern in the visible light band.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the method as described in any one of claims 1-7.

Images