A multi-scale parallel single-pixel 3D imaging method based on FPP constraints

Abstract

This invention discloses a multi-scale parallel single-pixel 3D imaging method based on FPP constraints, belonging to the field of structured light 3D measurement technology. The method includes: projecting a series of N-step phase-shifted structured light patterns of specific frequencies and directions onto the surface of the object under test according to the selected scale; additionally projecting a set of single-frequency vertical stripes; acquiring the structured light pattern modulated on the surface of the object under test; calculating the light transmission coefficient of each pixel of the camera using a single-pixel reconstruction algorithm; and solving the phase using the single-frequency vertical stripe pattern based on the FPP method; scanning the phase solved by FPP on a column to form a constraint range on the epipolar line of the projector target surface to assist in locating the direct illumination component; and reconstructing the 3D topography information using the coordinate points corresponding to the camera and projector. This invention can provide adaptive depth constraints, breaking through the limitations of absolute measurement depth and inter-layer measurement depth, and can realize the reconstruction of multi-layer scene 3D information without depth constraints.

Description

Technical Field

[0001] The present invention relates to the technical field of structured light three-dimensional measurement, and in particular to a multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints. Background Art

[0002] Parallel single pixel imaging (PSI) is a recently emerged computational measurement technology that can perform three-dimensional reconstruction in complex illumination mixed scenes based on traditional structured light systems. By extending single pixel imaging (SI) to digital cameras and treating each camera pixel as a single pixel detector, PSI can obtain a complete 4D light transmission coefficient for separating complex illumination components and achieving accurate 3D measurement in complex illumination mixed scenes. A three-dimensional measurement method based on multi-scale parallel single pixel three-dimensional imaging is disclosed in CN202410321901.4. This method divides the scale reconstruction sub-area information and copies and splices the complete projector view image to reduce the imaging target surface at the same sampling rate to obtain more accurate imaging results, so that the parallel single pixel method is no longer limited to static scene measurement. First, the projector target surface is divided equally according to the scale factor, and several projection patterns of corresponding frequencies are generated and projected to the object to be measured, and the camera is controlled to shoot, the modulated image is obtained, and the direct illumination component of each pixel on the pixel array is solved and located. Secondly, a multi-scale parallel single-pixel imaging image reconstruction method is performed on all pixels on the camera image to obtain an image with projector resolution corresponding to each camera pixel.

[0003] However, multi-scale parallel single-pixel depth calibration needs to be performed in advance, and the depth has no correlation with the measured scene, which limits the absolute measurement depth of the measured scene. For multi-layer scene measurement, it limits the inter-layer depth of the measured scene. Summary of the invention

[0004] The purpose of the present invention is to overcome the problem of limited depth of measured scene in the prior art, and to provide a multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0006] A multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints, comprising the following steps:

[0007] S1: Project a series of N-step phase-shifted structured light patterns with specific frequencies and directions onto the surface of the object under test according to the selected scale. In addition, an additional set of single-frequency vertical stripes needs to be projected;

[0008] S2: Collect the structured light pattern modulated by the surface of the object under test, calculate the light transmission coefficient of each pixel of the camera using the single-pixel reconstruction algorithm, and use the single-frequency vertical stripe pattern to solve the phase based on the FPP method;

[0009] S3: Scan the phase of the FPP solution on a column to form a constraint range on the polar line of the projector target surface, assist in locating the direct illumination component, and reconstruct the three-dimensional morphology information using the corresponding coordinate points of the camera and projector.

[0010] By adopting the above technical solution, a set of single-frequency vertical stripes with N-step phase shift are additionally projected and the phase is solved. The inter-layer constraint effect is derived by using the vector superposition of the phase. This provides an adaptive depth constraint that changes with the depth of the scene. The constraint information comes from the scene itself, which solves the depth limitation problem of the prior art and can realize dynamic three-dimensional measurement of multi-layer scenes with unlimited depth.

[0011] As a preferred solution of the present invention, the structured light pattern in step S1 is two groups of stripes in different directions, and the selection frequency depends on the scale, which is expressed as:

[0012]

[0013] Among them, I n Represents the spectral coefficient of the structured light image, a is the background light intensity, b is the fringe modulation, u p 、v p is the pixel of the projector target surface, f u 、f v is the frequency of the structured light image in the horizontal and vertical directions; and an additional set of single-frequency vertical stripe patterns needs to be projected for directly solving the phase, which is expressed as:

[0014]

[0015] As a preferred solution of the present invention, the phase φ calculated based on FPP using single-frequency vertical stripes in step S2 can be represented by a phase angle of a vector B, that is:

[0016]

[0017] Among them, u c 、v c is the pixel of the camera target surface, B I , B R are the imaginary and real parts of vector B. Therefore, we can think of phase in the same way as we think of vector.

[0018] As a preferred solution of the present invention, in step S2, for a multi-layer scene, the fringe pattern received by the camera is a superposition of modulation patterns of multiple layers of objects, and a double layer is taken as an example here, that is:

[0019] I=I b +I f ;

[0020] Among them, I b ,I f are the spectral coefficients of the structured light images modulated by the rear object and the front object respectively. The superposition phase calculated based on FPP is expressed as:

[0021]

[0022] Among them, I c 1n Represents the spectral coefficients of a set of structured light images modulated by the back-layer objects, I c 2n represents the spectral coefficients of a set of structured light images modulated by the front layer objects, B b is the vector corresponding to the object in the back layer, B f is the vector corresponding to the front layer object, B bI , B bR The vector B b The imaginary and real parts of B fI , B fR The vector B f The imaginary and real parts of the two vectors. Therefore, the superimposed phase can be expressed by the phase angle after the two vectors are superimposed, which is called the vector superposition of the phase. Since the phase angle of the superimposed two vectors must be in the middle of the phase angles of the individual vectors, and the phase angle can be equivalent to describe the phase, the superimposed phase must be between the two layers of individual phases, which is called the interlayer constraint effect.

[0023] As a preferred solution of the present invention, step S3 includes: scanning the phase of the FPP solution on a column and corresponding to the projector target surface, statistically analyzing the depth distribution histogram, selecting the maximum and minimum depth values ​​as constraint endpoints, and the corresponding constraint line segments on the epipolar lines as the constraint depth of the camera pixel point. According to the inter-layer constraint effect, all two-layer direct illumination components on the column must be within the constrained range, so the constraint is an adaptive depth constraint, which is expressed as:

[0024] u p (P f )∈[u p (P2),u p (P1)],u p (P b )∈[u p (P2),u p (P1)];

[0025] u p (P2) = min[u p (v c )],up (P1) = max[u p (v c )];

[0026] Among them, u p is the horizontal coordinate of the projector target surface, P f is the front layer information positioning point, P b is the positioning point of the back layer information, P1 is the maximum value point of the constraint range, and P2 is the minimum value point of the constraint range.

[0027] On the other hand, an electronic device is disclosed, comprising at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described above.

[0028] Compared with the prior art, the beneficial effects of the present invention are: by additionally projecting a set of single-frequency vertical stripes with N-step phase shifts and solving the phase, the inter-layer constraint effect is derived using the vector superposition of the phase, providing an adaptive depth constraint that changes with the depth of the scene, and the constraint information comes from the scene itself, which solves the depth limitation problem of the prior art and can realize dynamic three-dimensional measurement of multi-layer scenes with unlimited depth. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints according to the present invention;

[0030] Figure 2 It is a schematic diagram of the phase vector superposition property of the present invention;

[0031] Figure 3 It is a schematic diagram of the interlayer constraint effect of the present invention;

[0032] Figure 4 It is a schematic diagram of the FPP constraint positioning method of the present invention;

[0033] Figure 5 A schematic diagram of the FPP constraint range selection according to the present invention;

[0034] Figure 6 A flow chart for selecting the FPP constraint range according to the present invention;

[0035] Figure 7 This is a diagram of the three-dimensional measurement results of a multi-layer scene with absolute measurement depth changes as described in the present invention.

[0036] Figure 8 This is a diagram of three-dimensional measurement results of a multi-layer scene with inter-layer depth variation as described in the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] Please refer to Figure 1 , which shows a flowchart of a multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints provided by an example of the present invention, the method comprising:

[0040] S1: Project a series of N-step phase-shifted structured light patterns with specific frequencies and directions onto the surface of the object under test according to the selected scale. In addition, an additional set of single-frequency vertical stripes needs to be projected;

[0041] S2: Collect the structured light pattern modulated by the surface of the object under test, calculate the light transmission coefficient of each pixel of the camera using the single-pixel reconstruction algorithm, and use the single-frequency vertical stripe pattern to solve the phase based on the FPP method;

[0042] S3: Scan the phase of the FPP solution on a column to form a constraint range on the polar line of the projector target surface, assist in locating the direct illumination component, and reconstruct the three-dimensional morphology information using the corresponding coordinate points of the camera and projector.

[0043] The embodiments of the present invention are further introduced and described below in conjunction with specific implementation methods.

[0044] First, the phase vector superposition property and interlayer constraint effect are explained. Figure 2 and Figure 3As shown in the figure, based on the formula for solving the phase using FPP, the numerator and denominator of the inverse tangent operation are respectively regarded as the imaginary and real parts of a vector. The phase angle of the vector can be regarded as the phase, that is, the phase can be viewed with the idea of ​​viewing the vector. In a multi-layer scene, the stripe pattern captured by the camera is the superposition of multiple stripe patterns. Here, a two-layer scene is taken as an example. If the two-layer phases are represented by two vectors respectively, the phase obtained by solving the aliased stripes using FPP is the phase angle of the composite vector of the sum of the two vectors, that is, the vector superposition of the phase. It can be determined from the vector superposition property of the vector that the phase angle of the composite vector must be in the middle of the phase angles of the two individual vectors, that is, the inter-layer constraint effect.

[0045] Next, the FPP constraint positioning method is explained. Figure 4 and Figure 5 As shown in the figure, the aliasing phase calculated on a column is mapped to the projector target surface, and the depth distribution histogram is counted. The maximum and minimum depth values ​​are selected as the constraint endpoints, and the corresponding constraint line segments on the epipolar lines are used as the constraint depth of the camera pixel point, corresponding to Figure 5 For the line segments in , according to the inter-layer constraint effect, all the two-layer direct illumination components on this column must be within the constrained range, so this constraint is an adaptive depth constraint.

[0046] According to a specific implementation, as described in step 1 above, Figure 7 and Figure 8 Taking a pixel point in a multi-layer scene as an example, the multi-scale parallel single-pixel 3D imaging method based on FPP constraints is described. This point is located on a multi-layer scene and receives energy reflected from both the semi-transparent surface of the front layer and the teddy bear of the back layer. The projected base pattern is selected with a size of 1024×1024 and filled with black to a projector size of 1920×1080. The scale factor is set to 4, and the spectrum sampling path is set to the vertical direction and the oblique 45° direction. The spectrum sampling coefficient in each direction is selected to be 5. According to the points that need to be sampled on the spectrum, the N-step phase-shifted base pattern of the corresponding frequency is generated, which is expressed as:

[0047]

[0048] Among them, I n Represents the spectral coefficient of the structured light image, a is the background light intensity, b is the fringe modulation, u p 、v p is the pixel of the projector target surface, f u 、f v is the frequency of the structured light image in the horizontal and vertical directions; and an additional set of single-frequency vertical stripe patterns needs to be projected for directly solving the phase, which is expressed as:

[0049]

[0050] According to a specific implementation, as described in step 2 above, the phase φ is calculated based on FPP using single-frequency vertical stripes, which can be represented by a phase angle of a vector B, that is:

[0051]

[0052] Among them, u c 、v c is the pixel of the camera target surface, B I , B R are the imaginary and real parts of vector B, respectively. Therefore, the phase can be viewed in the same way as a vector. For a multi-layer scene, the fringe pattern received by the camera is the superposition of the modulation patterns of the multi-layer objects. Here, we take a double layer as an example, namely:

[0053] I=I b +I f ,

[0054] Among them, I b ,I f are the spectral coefficients of the structured light images modulated by the rear object and the front object respectively. The superposition phase calculated based on FPP is expressed as:

[0055]

[0056] Among them, I c 1n Represents the spectral coefficients of a set of structured light images modulated by the back-layer objects, I c 2n represents the spectral coefficients of a set of structured light images modulated by the front layer objects, B b is the vector corresponding to the object in the back layer, B f is the vector corresponding to the front layer object, B bI , B bR The vector B b The imaginary and real parts of B fI , B fR The vector B f The imaginary and real parts of the two vectors. Therefore, the superimposed phase can be expressed by the phase angle after the two vectors are superimposed, which is called the vector superposition of the phase. Since the phase angle of the superimposed two vectors must be in the middle of the phase angles of the individual vectors, and the phase angle can be equivalent to describe the phase, the superimposed phase must be between the two layers of individual phases, which is called the interlayer constraint effect.

[0057] According to a specific implementation, as described in step 3 above, the phase of the FPP solution on a column is scanned and corresponds to the projector target surface, the depth distribution histogram is statistically analyzed, the maximum and minimum depth values ​​are selected as constraint endpoints, and the corresponding constraint line segments on the epipolar lines are used as the constraint depth of the camera pixel point. According to the inter-layer constraint effect described in claim 4, all two-layer direct illumination components on the column must be within the constrained range, so the constraint is an adaptive depth constraint, which is expressed as:

[0058] u p (P f )∈[u p (P2),u p (P1)],u p (P b )∈[u p (P2),u p (P1)]

[0059] u p (P2) = min[u p (v c )],u p (P1)=max[u p (v c )]

[0060] Among them, u p is the horizontal coordinate of the projector target surface, P f is the front layer information positioning point, P b is the positioning point of the back layer information, P1 is the maximum value point of the constraint range, and P2 is the minimum value point of the constraint range.

[0061] By changing the absolute measurement depth and the inter-layer depth distance, the three-dimensional morphology results are obtained. Figure 5 This result shows that the present invention can achieve three-dimensional imaging of multi-layer scenes without depth restriction.

[0062] It should be noted that in the embodiment of the present invention, the camera unit can be implemented by a camera, and the projection unit can be implemented by a projector. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0063] In the embodiments provided in the present application, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0064] Through the description of the above embodiments, it is clear to those skilled in the art that the method in the embodiment of the present application can be implemented by hardware, firmware, or a combination thereof. When software is used for implementation, the above functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any medium that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a computer can access. Taking this as an example but not limited to: a computer-readable medium can include RAM, ROM, an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read Only Memory, EEPROM), a compact disc (Compact Disc Read-Only Memory, CD-ROM) or other optical disc storage, a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition. Any connection can be appropriately a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, wireless, and microwave are included in the fixation of the medium. As used in the embodiments of the present application, disks and discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks usually copy data magnetically, while discs use lasers to optically copy data. The above combinations should also be included in the scope of protection of computer-readable media.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints, characterized in that: The following steps are involved: S1: Project a series of N-step phase-shifted structured light patterns of specific frequency and direction onto the surface of the object under test at the selected scale. In addition, an additional set of single-frequency vertical stripes is projected; S2: Collect the structured light pattern modulated on the surface of the object under test, calculate the light transmission coefficient of each pixel using the single-pixel reconstruction algorithm, and use the single-frequency vertical stripe pattern to solve the phase based on the FPP method; S3: Scan the FPP solved phase on a column, form a constraint range on the epipolar line of the projector target surface, assist in locating the direct illumination component, and reconstruct the 3D shape information using the corresponding coordinate points of the camera and projector; For multi-layer scenes, the fringe pattern received by the camera is the superposition of the modulation patterns of the multi-layer objects. Based on the vector superposition of the phase in FPP, there is an inter-layer constraint effect. When the double-layer object modulates the pattern: I=I b +I f ; Among them, I b ,I f are the spectral coefficients of the structured light images modulated by the rear object and the front object respectively; the superposition phase calculated based on the FPP solution is expressed as: Among them, φ′(u c , v c ) is the superposition phase, u c and v c is the pixel coordinate of the camera, represents the spectral coefficients of a structured light image modulated by a set of back-layer objects, represents the spectral coefficients of a set of structured light images modulated by the front layer objects, B b is the vector corresponding to the object in the back layer, B f is the vector corresponding to the front layer object, B bI , B bR The vector B b The imaginary and real parts of B fI , B fR The vector B f The imaginary and real parts of .

2. The multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints according to claim 1, characterized in that: The structured light pattern in step S1 is two groups of stripes in different directions, and the selection frequency depends on the scale, which is expressed as: Among them, I n Represents the spectral coefficient of the structured light image, a is the background light intensity, b is the fringe modulation, u p 、v p is the pixel of the projector target surface, f u 、f v is the frequency of the structured light image in the horizontal and vertical directions; and an additional set of single-frequency vertical stripe patterns needs to be projected for directly solving the phase, which is expressed as: Among them, I n Represents the spectral coefficient of the structured light image, a is the background light intensity, b is the fringe modulation, v p is the pixel of the projector target surface.

3. The multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints according to claim 1, characterized in that: The phase calculated based on the FPP solution using the single-frequency vertical stripes in step S2 can be represented by a phase angle of a vector.

4. The multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints according to claim 1, characterized in that: Step S3 includes: S1. Scan the phase of the FPP solution on a column and correspond it to the projector target surface; S2. Calculate the depth distribution histogram and select the maximum and minimum depth values ​​as constraint endpoints; S3. The corresponding constraint line segment on the epipolar line is used as the constraint depth of the camera pixel point.

5. The multi-scale parallel single-pixel three-dimensional imaging method based on FPP constraints according to claim 4, characterized in that: The inter-layer constraint effect mentioned above means that all two-layer direct illumination components on the column must be within the constrained range, so the constraint is an adaptive depth constraint, which is expressed as: in p (P f )∈[u p (P2),in p (P1)],in p (P b )∈[u p (P2), in p (P1)]; u p (P2)=min[u p (v c )],u p (P1)=max[u p (v c )]; Among them, u p is the horizontal coordinate of the projector target surface, P f is the front layer information positioning point, P b is the positioning point of the back layer information, P1 is the maximum value point of the constraint range, and P2 is the minimum value point of the constraint range.

6. An electronic device, characterized in that: The invention comprises at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 5.

Images