Structured light decoding method, system, device, medium and program product

Abstract

This application provides a structured light decoding method, system, device, medium, and program product. The method includes: acquiring a deformed fringe image modulated by the surface height of an object, acquired by an image acquisition unit; performing spatial gradient calculation on the deformed fringe image to obtain a gradient image of the deformed fringe image; and performing phase calculation based on the gradient image to obtain the phase information of the deformed fringe image. This method is used to improve the accuracy of structured light decoding.

Description

Technical Field

[0001] This application relates to the field of image data processing, and in particular to a structured light decoding method, system, device, medium, and program product. Background Technology

[0002] In structured light 3D measurement systems, phase-shifting structured light is a commonly used high-precision 3D reconstruction method. It can obtain the phase information corresponding to each pixel by projecting multiple frames of sinusoidal fringe patterns with fixed phase differences onto the target object and performing phase calculation on the image sequence acquired by the camera, thereby recovering the 3D shape of the object's surface.

[0003] However, in real-world applications, such as shopping malls or industrial settings, ambient lighting is typically generated by AC-powered light-emitting diode (LED) lamps or fluorescent lamps. These light sources exhibit periodic brightness fluctuations, known as flickering. When a structured light system acquires phase-shifted fringe images, if the acquisition frame rate is not synchronized with the ambient flicker frequency, overall brightness variations will occur between different frames. These brightness variations will be superimposed on the structured light fringe signal, thus affecting the stability of traditional phase-shift decoding algorithms.

[0004] Therefore, improving the accuracy of structured light decoding is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a structured light decoding method, system, device, medium, and program product to improve the accuracy of structured light decoding.

[0006] In a first aspect, embodiments of this application provide a structured light decoding method, including:

[0007] Acquire the deformed stripe image acquired by the image acquisition unit after being modulated by the height of the object's surface;

[0008] Spatial gradient calculation is performed on the deformed stripe image to obtain the gradient image of the deformed stripe image;

[0009] Phase calculation is performed based on the gradient image to obtain the phase information of the deformed stripe image.

[0010] In one possible implementation, performing spatial gradient calculation on the deformed stripe image to obtain the gradient image of the deformed stripe image includes:

[0011] Gradient operations are performed on the deformed stripe image along a preset spatial encoding direction to obtain the gradient image of the deformed stripe image.

[0012] In one possible implementation, performing spatial gradient calculation on the deformed stripe image to obtain the gradient image of the deformed stripe image includes:

[0013] The deformed stripe image is decomposed into multiple scales to obtain multiple deformed stripe image components of different scales.

[0014] Gradient operations are performed on the multiple deformed stripe image components of different scales in a preset spatial coding direction to obtain multiple gradient images of different scales.

[0015] The gradient images of the multiple different scales are fused to obtain the gradient image of the deformed stripe image.

[0016] In one possible implementation, the step of performing phase calculation based on the gradient image to obtain the phase information of the deformed stripe image includes:

[0017] Spatial gradient calculation is performed on the fringe modulation map to obtain the modulation gradient map;

[0018] Based on the modulation gradient map and the gradient image of the deformed fringe image, as well as the preset gradient domain phase shift model, phase calculation is performed to obtain the phase information of the deformed fringe image.

[0019] In one possible implementation, the method further includes:

[0020] The image acquisition unit acquires a fully bright image and a fully dark image after the surface height of the object is modulated.

[0021] The fringe tone map is determined based on the full-brightness map and the full-darkness map.

[0022] In one possible implementation, the step of performing phase calculation based on the gradient image to obtain the phase information of the deformed stripe image includes:

[0023] The spatial gradient of the stripe modulation plot is estimated to be 0;

[0024] Phase information of the deformed stripe image is obtained by performing phase calculation based on the gradient image of the deformed stripe image and the preset gradient domain phase shift model.

[0025] In one possible implementation, after performing phase calculation based on the gradient image to obtain the phase information of the deformed stripe image, the method further includes:

[0026] Based on the phase information of the deformed stripe image, three-dimensional reconstruction is performed to obtain three-dimensional point cloud data.

[0027] Secondly, embodiments of this application provide a structured light decoding system, comprising:

[0028] Projection unit, used to project structured light stripe patterns onto the surface of an object;

[0029] The image acquisition unit is used to acquire the deformed stripe image after being modulated by the height of the object's surface;

[0030] The computational processing unit is used to perform spatial gradient calculation on the deformed stripe image to obtain the gradient image of the deformed stripe image; and to perform phase calculation based on the gradient image to obtain the phase information of the deformed stripe image.

[0031] In one possible implementation, the projection unit is further configured to: project a full-brightness image and a full-darkness image onto the surface of the object;

[0032] The image acquisition unit is also used to: acquire a full-brightness image and a full-darkness image modulated by the surface height of the object;

[0033] The computational processing unit is further configured to: determine a stripe modulation map based on the full brightness map and full darkness map modulated by the surface height of the object; perform spatial gradient calculation on the stripe modulation map to obtain a modulation gradient map; and perform phase calculation based on the modulation gradient map, the gradient image of the deformed stripe image, and a preset gradient domain phase shift model to obtain the phase information of the deformed stripe image.

[0034] In one possible implementation, the projection unit and the image acquisition unit are the projection unit and image acquisition unit of a three-dimensional structured light camera.

[0035] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0036] The memory stores computer-executed instructions;

[0037] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any of the first aspects above.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0039] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0040] This application provides a structured light decoding method, system, device, medium, and program product. First, it acquires a deformed fringe image highly modulated by the object's surface. This image contains modulation information of the object's surface topography on the projected fringes, as well as brightness variation information caused by environmental flicker and other interference. Second, this application performs spatial gradient calculations on the deformed fringe image to obtain a gradient image. Spatial gradient calculations naturally eliminate the DC component of ambient light and overall frame brightness variations, thus effectively suppressing flicker interference. Compared to directly performing phase calculations based on the original deformed fringe image, this application successfully models and calculates the phase of the projected structured light in the gradient domain, reducing interference from environmental flicker and improving the accuracy of structured light decoding. Furthermore, this application does not require changes to the original projection encoding method or additional hardware. It is applicable not only to the four-step phase-shift algorithm but can also be extended to multi-step phase-shift or other phase-calculation-based structured light methods. Therefore, this method has good versatility and engineering application value. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 A schematic flowchart of a structured light decoding method provided in an embodiment of this application;

[0043] Figure 2 This application provides a schematic diagram of the structure of a structured light decoding system.

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device provided in this application.

[0045] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0047] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0048] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0049] In structured light 3D measurement systems, phase-shifting structured light is a commonly used high-precision 3D reconstruction method. This method projects multiple frames of sinusoidal fringe patterns with a fixed phase difference onto the target object and performs phase calculation on the image sequence acquired by the camera to obtain the phase information corresponding to each pixel, thereby recovering the 3D shape of the object's surface.

[0050] In real-world applications, such as shopping malls or industrial settings, ambient lighting is typically generated by AC-powered light-emitting diode (LED) lamps or fluorescent lamps. These light sources exhibit periodic brightness fluctuations, known as flickering. When a structured light system acquires phase-shifted fringe images, if the acquisition frame rate is not synchronized with the ambient flicker frequency, overall brightness variations will occur between different frames. These brightness variations will be superimposed on the structured light fringe signal, thus affecting the stability of traditional phase-shift decoding algorithms.

[0051] For example, in a conventional phase-shift structured light system, the projection optical engine projects a set of sinusoidal fringe patterns with different phase shifts onto the surface of an object. The camera... The pixel intensity captured in the frame can be expressed by formula (1):

[0052] (1)

[0053] in, Indicates the first Frame image in pixels The brightness value at that location; This represents the DC component formed by the average reflection of ambient light from an object. Indicates the stripe tone system; Let be the phase value to be determined; For the first Phase shift of frame projection fringes. Taking a four-step phase shift algorithm as an example, The phase values ​​are typically 0, π / 2, π, and 3π / 2, respectively. Based on the four images, the phase can be calculated using the classic four-step phase shift formula (2):

[0054] (2)

[0055] in, Indicates when the phase shifts At that time, the first Frame image in pixels The brightness value at that location; Indicates when the phase shifts At that time, the first Frame image in pixels The brightness value at that location; Indicates when the phase shifts At that time, the first Frame image in pixels The brightness value at that location; Indicates when the phase shifts At that time, the first Frame image in pixels The brightness value at that location.

[0056] This formula can accurately recover phase information under ideal conditions, but it relies on the assumption that only fringe phase changes exist between all images, without any additional brightness variations. In the presence of environmental flicker, the brightness of the images captured by the camera is superimposed with a global brightness term that varies over time. This global brightness term cannot be canceled out during the calculation, thus introducing errors into the phase calculation. When the flicker amplitude is large or the fringe contrast is low, this error can significantly affect the final depth calculation result, thus requiring a new decoding method to eliminate this effect.

[0057] Since flicker interference manifests as a global brightness shift (approximately constant) in the spatial domain, its spatial derivative is zero; while structured light stripe signals exhibit significant gradient changes in the spatial domain, this embodiment of the application migrates structured light phase-shift decoding from the intensity domain to the gradient domain. Before performing phase decoding, spatial derivative operations are performed on the phase-shifted stripe image, thereby performing phase calculations in the gradient domain. The spatial derivative operation is used to eliminate global brightness fluctuation interference from flickering ambient light, thus achieving robust suppression of flicker interference without changing the original encoding method and hardware configuration.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0059] It should be noted that the implementing entity of this application can be a structured light decoding system, which may include a projection unit, an image acquisition unit, and a computing processing unit.

[0060] Figure 1 This is a flowchart illustrating a structured light decoding method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0061] S101. Acquire the image of deformed stripes modulated by the height of the object surface acquired by the image acquisition unit.

[0062] Optionally, the image acquisition unit can capture the stripe pattern modulated by the object's height after it is projected onto the object's surface and record it as image data for subsequent processing. A deformed stripe image is generated when a structured light stripe pattern with a specific pattern (such as sinusoidal stripes) is projected onto an object's surface, and the reflected stripe pattern is deformed due to changes in the object's surface height. This stripe image, which differs from the original projected stripes in shape, phase, etc., after being modulated by the object's surface height, is called a deformed stripe image.

[0063] Optionally, the structured light decoding system and the image acquisition unit can be connected via an interface for data transmission. Through this interface, the structured light decoding system can acquire the deformed stripe image of the object surface height modulated by the image acquisition unit.

[0064] For example, a deformed stripe image can be represented by the following formula (3).

[0065] (3)

[0066] in, This indicates that in the presence of environmental flicker, the first Frame image in pixels The brightness value at that location; Indicates the first The overall brightness shift in a frame image caused by environmental flicker, i.e., the flicker component, is usually approximately constant in space, but varies with the frame number in time. The meanings of the other parameters are consistent with those in formula (1) above, and will not be repeated here.

[0067] S102. Perform spatial gradient calculation on the deformed stripe image to obtain the gradient image of the deformed stripe image.

[0068] Optionally, spatial gradient operation can be used to describe the rate of change of pixel brightness in space in an image. It can measure the severity of brightness changes in local areas of an image by calculating the brightness difference of pixels in the horizontal and vertical directions, and can highlight feature information such as edges and contours in the image.

[0069] Optionally, the gradient image can be the result image obtained by performing spatial gradient operation on the deformed stripe image, and each pixel value in the gradient image can represent the rate of change of pixel brightness at the corresponding position in the deformed stripe image.

[0070] For example, suppose For the first A frame-deformed stripe image, for example, by calculating the image derivative in the x-direction, can have its gradient image represented by the following formula (4).

[0071] (4)

[0072] In discrete images, approximate calculations can be performed using the central difference or the Sobel operator, as shown in equation (5):

[0073] (5)

[0074] Due to flicker components Since it is a constant in space, its spatial derivative can be expressed by the following formula (6):

[0075] (6)

[0076] Since the brightness changes caused by environmental flicker are usually constant or slowly changing terms in space, and their spatial derivatives are close to zero, flicker interference is naturally eliminated in gradient images.

[0077] In one embodiment, the structured light decoding system can perform gradient calculations on the deformed stripe image in a preset spatial coding direction to obtain the gradient image of the deformed stripe image.

[0078] Optionally, the preset spatial encoding direction can be a gradient operation direction pre-set in the structured light decoding system, perpendicular to the stripe direction in the deformed stripe image. For example, when the deformed stripe image is horizontally distributed, the undulations of the object's surface will cause the horizontal stripes to be compressed / stretched in the vertical direction. In this case, using gradient operation in the vertical direction can accurately capture the rate of intensity change in this direction, thereby extracting gradient information reflecting the change in the height of the object's surface. Conversely, if the deformed stripe image is vertically distributed, and the preset spatial encoding direction is horizontal, horizontal gradient operation is required.

[0079] Since the stripes of structured light projection have a specific direction, the embodiments of this application perform gradient operations on the deformed stripe image in a preset spatial encoding direction to obtain a gradient image, which can accurately capture the changes of the stripes in that direction, thereby improving the accuracy of the gradient image of the obtained deformed stripe image.

[0080] S103. Perform phase calculation based on the gradient image to obtain the phase information of the deformed stripe image.

[0081] Optionally, since the deformed stripe image contains the phase change corresponding to the height information of the object's surface, the phase calculation can be the process of calculating the phase value corresponding to each pixel from the gradient image of the deformed stripe image through a preset algorithm.

[0082] Optionally, phase information is closely related to the three-dimensional shape of the object's surface and is key data for recovering the three-dimensional structure of the object's surface. Phase information can exist in the form of the phase value of each pixel in the deformed fringe image, reflecting the phase shift of the fringe at that pixel relative to the original projected fringe. The height information of the object's surface at that point can be further derived from the phase information. For example, the phase information of the deformed fringe image can be obtained through… This indicates... It should be noted that... With the above The meanings they represent are consistent, both indicating the phase information of the deformed stripe image.

[0083] For example, based on the above formulas (3) and (4), formula (7) can be obtained.

[0084] (7)

[0085] It should be noted that With the above The meanings represented are consistent, both indicating the gradient image of the deformed stripe image. Further expansion of formula (7) yields formula (8).

[0086] (8)

[0087] In formula (8), let:

[0088]

[0089]

[0090]

[0091] The gradient domain phase shift model can then be obtained, as shown in equation (9):

[0092] (9)

[0093] Therefore, it can be seen that the embodiments of this application strictly eliminate the flicker term mathematically through spatial gradient operation, thereby ensuring that the subsequent phase shift decoding process is no longer affected by environmental flicker interference.

[0094] Furthermore, the above formula (9) can be expanded to obtain formula (10).

[0095] (10)

[0096] Formula (10) can be rearranged into formula (11).

[0097] (11)

[0098] P can be represented by formula (12), and Q can be represented by formula (13).

[0099] (12)

[0100] (13)

[0101] Taking a four-step phase shift as an example, Formulas (14)-(17) can be obtained from the above formulas (10)-(13):

[0102] (14)

[0103] (15)

[0104] (16)

[0105] (17)

[0106] By rearranging formulas (12) and (17), we can obtain the following formulas (18) and (20):

[0107] (18)

[0108] (19)

[0109] (20)

[0110] Based on the above formulas (19)-(20), the phase information of the deformed stripe image can be obtained. .

[0111] This embodiment first acquires a deformed fringe image modulated by the height of an object's surface. This image contains modulation information of the object's surface topography on the projected fringes, as well as brightness variation information caused by environmental flicker and other interference. Secondly, this embodiment performs spatial gradient calculation on the deformed fringe image to obtain a gradient image. Spatial gradient calculation naturally eliminates the DC component of ambient light and overall frame brightness variations, thus exhibiting good suppression of flicker interference. Compared to directly performing phase calculation based on the original deformed fringe image, this embodiment successfully models and calculates the phase of the projected structured light decoding in the gradient domain, reducing interference from environmental flicker and improving the accuracy of phase structured light decoding. Furthermore, this embodiment does not require changes to the original projection encoding method or additional hardware. It is applicable not only to the four-step phase-shift algorithm but can also be extended to multi-step phase-shift or other phase-calculation-based structured light methods. Therefore, this method has good versatility and engineering application value.

[0112] Based on the above embodiments, the embodiments of this application may further perform multi-scale decomposition on the deformed stripe image to obtain multiple deformed stripe image components of different scales.

[0113] Optionally, multi-scale decomposition can decompose an image into multiple different scales or resolutions. Different scales can capture features of different sizes in the image. Large scales can reflect the global structural information of the image, while small scales can better reflect the local detail information of the image.

[0114] Optionally, the deformed stripe image components can be image representations at various scales obtained after multi-scale decomposition, with each component containing feature information of the original deformed stripe image within different scale ranges.

[0115] Optionally, the structured light decoding system can use any one or more multi-scale decomposition methods such as Gaussian pyramid decomposition and wavelet transform. Taking Gaussian pyramid decomposition as an example, the structured light decoding system can first Gaussian blur the original deformed fringe image, and then downsample the blurred image (e.g., halve the image size) to obtain the first-level decomposed image; then repeat the above Gaussian blurring and downsampling operation on the first-level image to obtain the second-level decomposed image, and so on, thereby obtaining multiple deformed fringe image components at different scales. Wavelet transform, on the other hand, decomposes the image into approximate components and detail components at different scales by selecting appropriate wavelet basis functions.

[0116] Secondly, gradient operations are performed on the deformed stripe image components at multiple different scales along a preset spatial encoding direction to obtain multiple gradient images at different scales. This process is similar to that described above and will not be repeated here.

[0117] Finally, the structured light decoding system can fuse multiple gradient images at different scales to obtain a gradient image of the deformed stripe image. Optionally, the structured light decoding system can use any one or more fusion methods, such as weighted averaging or pyramid-based fusion, to merge multiple gradient images at different scales into a single comprehensive gradient image.

[0118] For example, the weighted average method assigns a weight coefficient to the gradient image at each scale, then multiplies the corresponding pixel values ​​of the gradient images at each scale by the weight and adds them together to obtain the fused gradient image. The pyramid-based fusion method first constructs a pyramid structure (such as a Laplacian pyramid) from the gradient images at each scale, then performs a fusion operation at each layer of the pyramid (such as selecting pixel values ​​with larger coefficients or performing a weighted average), and finally reconstructs the gradient image of the fused deformed stripe image through pyramid reconstruction.

[0119] This application embodiment first performs multi-scale decomposition on the deformed stripe image, then performs gradient operations on the components at different scales and fuses them to obtain a gradient image. Multi-scale decomposition can separate information of different frequencies in the image, and image components at different scales contain image features at different levels. Performing gradient operations on each component can capture the stripe variation features at different scales in greater detail. Fusing multi-scale gradient images combines the advantages of each scale, so that the final gradient image can both retain the overall structural information of the image and highlight detailed features, providing richer information support for subsequent accurate phase calculations and improving the quality and applicability of the gradient image.

[0120] The following section details how to perform phase calculation based on gradient images to obtain the phase information of deformed stripe images.

[0121] In one implementation, the structured light decoding system can first perform spatial gradient calculations on the fringe modulation map to obtain a modulation gradient map.

[0122] Optionally, the fringe modulation pattern can reflect the modulation intensity distribution of the fringe pattern reflected or scattered by the surface of the object under test. The fringe modulation pattern can be the aforementioned fringe modulation... Spatial distribution display.

[0123] Optionally, the structured light decoding system can extract the amplitude of the fundamental frequency component in the frequency domain as the modulation index using the Fourier transform method, or use color stripe coding technology to encode multiple phase shift information into the red, green, and blue (RGB) channels to achieve single projection calculation, thereby obtaining the stripe modulation index map.

[0124] Optionally, the structured light decoding system may first acquire the full brightness image and full darkness image after being modulated by the surface height of the object by the image acquisition unit, and determine the stripe modulation map based on the full brightness image and full darkness image.

[0125] Optionally, the full-brightness image can be an image captured by the image acquisition unit when the structured light decoding system projects a uniform bright field (i.e., a light field with maximum intensity and uniform distribution) onto the object surface. In this image, the light intensity of each pixel mainly reflects the sum of ambient light and the reflected light from the object surface under uniform strong light illumination, including information such as ambient light and the reflective properties of the object surface. For example, the full-brightness image can be obtained through... Characterize it.

[0126] Optionally, the total dark image can be an image captured by the image acquisition unit when the structured light decoding system does not project any light (or projects light with zero intensity). In this case, the light intensity in the image mainly comes from ambient light, reflecting the distribution of ambient light on the camera's imaging plane. For example, the total dark image can be obtained through... Characterize it.

[0127] Furthermore, the stripe tone graph can be represented by formula (21).

[0128] (twenty one)

[0129] Secondly, the spatial gradient of the modulation system is calculated according to formula (21). , The structured light decoding system can perform phase calculation based on the modulation gradient map, the gradient image of the deformed fringe image, and a preset gradient domain phase shift model to obtain the phase information of the deformed fringe image. That is, the phase information of the deformed fringe image can be obtained according to formula (21), N, and formulas (19)-(20) derived from formula (9). That is, formula (22).

[0130] (twenty two)

[0131] S can be represented by formula (23).

[0132] (twenty three)

[0133] This application embodiment obtains a full brightness image and a full darkness image and determines a fringe modulation map based on them. The full brightness image and the full darkness image reflect the image information under maximum illumination and ambient light conditions, respectively. The fringe modulation map calculated by the two can reduce the influence of interference factors such as ambient light on the fringe contrast, thereby providing more accurate basic data for subsequent phase calculation based on the fringe modulation map, which helps to improve the accuracy and reliability of the entire structured light decoding method.

[0134] This application embodiment performs spatial gradient calculation on the fringe modulation map to obtain a modulation gradient map, and combines the gradient image of the deformed fringe image and the gradient domain phase shift model to perform phase calculation. It makes full use of the constraint effect of modulation information on phase change, reduces the phase calculation error caused by noise, image blur and other factors, and makes the obtained phase information more realistically reflect the modulation of the object surface height on the fringe, significantly improving the accuracy of phase calculation.

[0135] In another implementation, the structured light decoding system can estimate the spatial gradient of the fringe modulation map as 0. Optionally, the confidence (amplitude) in the actual structured light reconstruction scene varies little locally, so the spatial gradient of the fringe modulation map can be estimated as 0, as shown in Equation (24).

[0136] (twenty four)

[0137] The structured light decoding system can perform phase calculation based on the gradient image of the deformed fringe image and a preset gradient domain phase shift model to obtain the phase information of the deformed fringe image. That is, the phase information of the deformed fringe image can be obtained according to formula (24) and formulas (19)-(20) derived from formula (9). That is, formula (25).

[0138] (25)

[0139] Since the confidence level (amplitude) in actual structured light reconstruction scenarios varies little locally, the embodiment of this application estimates the spatial gradient of the fringe modulation map as 0, and performs phase calculation based on this by combining the gradient image of the deformed fringe image and the gradient domain phase shift model. This simplifies the calculation process and does not seriously affect the accuracy of the phase calculation. It can still solve the phase information based on the main variation characteristics of the fringe, thus improving the calculation efficiency while ensuring a certain level of accuracy.

[0140] After performing phase calculation based on the gradient image to obtain the phase information of the deformed stripe image, the embodiments of this application can also perform three-dimensional reconstruction based on the phase information of the deformed stripe image to obtain three-dimensional point cloud data.

[0141] Optionally, 3D reconstruction can be the process of recovering the shape, position, and size of an object in three-dimensional space based on the phase information of the acquired deformed fringe image, using specific mathematical models and algorithms. 3D point cloud data can be one form of representation of the 3D reconstruction result. It can be a dataset consisting of a large number of discrete 3D point coordinates, each point containing its position information in 3D space (usually represented by X, Y, and Z coordinates). These points can accurately describe the geometry and features of the object's surface.

[0142] For example, since there is a corresponding relationship between phase changes and changes in the height of an object's surface, the structured light decoding system can convert phase information into height information of each point on the object's surface relative to a reference plane based on a pre-set geometric model containing parameters such as image acquisition units and using principles such as triangulation. This enables the conversion from phase information to the object's three-dimensional spatial height distribution, thus completing three-dimensional reconstruction.

[0143] For example, after obtaining the height information of each point on the surface of the object, the two-dimensional image coordinates of each pixel in the deformed stripe image are fused with the corresponding height value by combining the conversion relationship between pixel coordinates and spatial coordinates to obtain the X, Y, and Z coordinates of each point in three-dimensional space, thus obtaining three-dimensional point cloud data that can characterize the shape of the object surface.

[0144] This application embodiment performs three-dimensional reconstruction based on the phase information of the deformed stripe image to obtain three-dimensional point cloud data, and converts the phase information into the height information of each point on the object surface, realizing the transformation from two-dimensional image information to three-dimensional spatial information, and promoting the application of structured light measurement technology in actual production and life.

[0145] For example, the overall process of the anti-flicker structured light decoding method proposed in this application embodiment is as follows:

[0146] First, the projector projects multiple sinusoidal fringe patterns onto the surface of the target object according to a preset phase-shift sequence, along with two images: one fully bright and one fully dark. The number of phase-shift steps can be three, four, or more.

[0147] Secondly, the camera acquires a corresponding number of stripe images under conditions of synchronization with the projection optical engine, forming a set of time-series images.

[0148] Subsequently, the gradient is calculated in the spatial direction for each frame of the acquired image, for example, by using the central difference operator or the Sobel operator, to obtain a gradient image sequence.

[0149] After spatially differentiating the fringe model, a new gradient signal model can be obtained. Since the environmental flicker term disappears under the spatial derivative, the gradient signal is mainly determined by the spatial variation of the fringe signal. The observed image model can be expressed as formula (7), which can be further expanded to obtain formula (8), thus obtaining the gradient domain phase shift model, i.e., formula (9). Expanding and summarizing formula (9) yields formula (11). Taking a four-step phase shift as an example, formulas (18)-(20) can be obtained.

[0150] At this point, regarding phase There are two constraints, but in practice, it is necessary to solve for N and S. There are three unknowns, and there are two methods to solve them:

[0151] Method 1: Assumption

[0152] In real-world structured light reconstruction scenarios, the confidence level (amplitude) varies relatively little locally, therefore Substituting the two equations into the above formulas (19) and (20) will give us the phase, which is formula (25).

[0153] Method 2: Calculate the amplitude parameters in the model using the full-brightness and full-darkness images projected by the optical engine, i.e., formula (21), based on... And with formulas (19) and (20), the phase can be solved, i.e., formula (22). When N approaches 0, formula (22) degenerates into formula (25), which is consistent with the result of method one.

[0154] The above are the method embodiments provided in this application. The system provided in this application will be described below.

[0155] Figure 2 This application provides a schematic diagram of the structure of a structured light decoding system, as shown below. Figure 2 As shown, the structured light decoding system 400 provided in this embodiment includes: a projection unit 401, an image acquisition unit 402, and a calculation and processing unit 403.

[0156] Projection unit 401 is used to project structured light stripe patterns onto the surface of an object.

[0157] Optionally, the structured light stripe pattern can be a light pattern with a specific light intensity distribution, such as a regular pattern like sinusoidal stripes or binary stripes generated by the projection unit according to certain encoding rules. For example, the structured light stripe pattern can be periodic sinusoidal stripes that change sequentially along the time axis according to a fixed phase shift step.

[0158] Optionally, the projection unit 401 can project structured light stripe patterns using any one or more projection methods, such as a digital light processing projector or a laser projector. Optionally, the projection unit 401 can be a projection optical engine.

[0159] Image acquisition unit 402 is used to acquire images of deformed stripes modulated by the height of the object surface. Optionally, image acquisition unit 402 can be a camera used to synchronously acquire reflected images of the object surface, thereby obtaining the deformation of the stripe pattern on the target surface for each frame.

[0160] The computational processing unit 403 is used to perform spatial gradient calculations on the deformed stripe image to obtain a gradient image of the deformed stripe image. Phase calculation is then performed based on the gradient image to obtain the phase information of the deformed stripe image. For example, the computational processing unit 403 can be used to process acquired image sequences, including operations such as spatial gradient calculation, phase shift decoding, phase recovery, and depth calculation.

[0161] Optionally, the projection unit 401 is also used to project a fully bright image and a fully dark image onto the surface of the object.

[0162] The image acquisition unit 402 is also used to acquire the full bright image and full dark image after being modulated by the height of the object surface.

[0163] The computational processing unit 403 is also used to determine the fringe modulation map based on the full brightness map and full darkness map after modulation by the surface height of the object. Spatial gradient calculation is performed on the fringe modulation map to obtain the modulation gradient map. Based on the modulation gradient map and the gradient image of the deformed fringe image, as well as a preset gradient domain phase shift model, phase calculation is performed to obtain the phase information of the deformed fringe image.

[0164] Optionally, the projection unit 401 and the image acquisition unit 402 are the projection unit and image acquisition unit of a three-dimensional structured light camera.

[0165] Optionally, during the calibration stage of the structured light decoding system, the geometric mapping relationship between the camera coordinate system and the projection coordinate system can be established through the calibration of the image acquisition unit 402 (e.g., camera) and the projection unit 401, thereby realizing the conversion of phase information to three-dimensional spatial coordinates.

[0166] The structured light decoding system provided in this embodiment can execute the methods provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0167] Figure 3 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 3As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0168] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0169] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0170] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0171] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0172] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0174] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0175] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0176] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0177] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0180] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0182] Finally, it should be noted that other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from its scope.

Claims

1. A structured light decoding method, characterized by, The method includes: Acquire the deformed stripe image acquired by the image acquisition unit after being modulated by the height of the object's surface; Spatial gradient calculation is performed on the deformed stripe image to obtain the gradient image of the deformed stripe image; Phase calculation is performed based on the gradient image to obtain the phase information of the deformed stripe image.

2. The method of claim 1, wherein, The step of performing spatial gradient calculation on the deformed stripe image to obtain the gradient image of the deformed stripe image includes: Gradient operations are performed on the deformed stripe image along a preset spatial encoding direction to obtain the gradient image of the deformed stripe image.

3. The method of claim 1, wherein, The step of performing spatial gradient calculation on the deformed stripe image to obtain the gradient image of the deformed stripe image includes: The deformed stripe image is decomposed into multiple scales to obtain multiple deformed stripe image components of different scales. Gradient operations are performed on the multiple deformed stripe image components of different scales in a preset spatial coding direction to obtain multiple gradient images of different scales. The gradient images of the multiple different scales are fused to obtain the gradient image of the deformed stripe image.

4. The method of claim 1, wherein, The step of performing phase calculation based on the gradient image to obtain the phase information of the deformed stripe image includes: Spatial gradient calculation is performed on the fringe modulation map to obtain the modulation gradient map; Based on the modulation gradient map and the gradient image of the deformed fringe image, as well as the preset gradient domain phase shift model, phase calculation is performed to obtain the phase information of the deformed fringe image.

5. The method of claim 4, wherein, The method further includes: The image acquisition unit acquires a fully bright image and a fully dark image after the surface height of the object is modulated. The fringe tone map is determined based on the full-brightness map and the full-darkness map.

6. The method of claim 1, wherein, The step of performing phase calculation based on the gradient image to obtain the phase information of the deformed stripe image includes: The spatial gradient of the stripe modulation plot is estimated to be 0; Phase information of the deformed stripe image is obtained by performing phase calculation based on the gradient image of the deformed stripe image and the preset gradient domain phase shift model.

7. The method according to any one of claims 1 to 6, characterized in that, After performing phase calculation based on the gradient image to obtain the phase information of the deformed stripe image, the method further includes: Based on the phase information of the deformed stripe image, three-dimensional reconstruction is performed to obtain three-dimensional point cloud data.

8. A structured light decoding system, characterized in that, include: Projection unit, used to project structured light stripe patterns onto the surface of an object; The image acquisition unit is used to acquire the deformed stripe image after being modulated by the height of the object's surface; The computational processing unit is used to perform spatial gradient calculation on the deformed stripe image to obtain the gradient image of the deformed stripe image; and to perform phase calculation based on the gradient image to obtain the phase information of the deformed stripe image.

9. The system according to claim 8, characterized in that, The projection unit is also used to: project a fully bright image and a fully dark image onto the surface of the object; The image acquisition unit is also used to: acquire a full-brightness image and a full-darkness image modulated by the surface height of the object; The computational processing unit is further configured to: determine a stripe modulation map based on the full brightness map and full darkness map modulated by the surface height of the object; perform spatial gradient calculation on the stripe modulation map to obtain a modulation gradient map; and perform phase calculation based on the modulation gradient map, the gradient image of the deformed stripe image, and a preset gradient domain phase shift model to obtain the phase information of the deformed stripe image.

10. The system according to claim 8 or 9, characterized in that, The projection unit and the image acquisition unit are the projection unit and image acquisition unit of a three-dimensional structured light camera.

11. A structured light decoding device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images