Structured light reconstruction module

Abstract

The application provides a structured light reconstruction module, comprising: a structured light projector configured to project structured light to a target; a floodlight projector configured to project floodlight to the target; an infrared camera configured to acquire a background image, an infrared structured light image and an infrared image; a processor module configured to obtain a preset frame frequency value, sequentially acquire the background image, the infrared structured light image and the infrared image of the same target according to the frame frequency value, or sequentially acquire the infrared image, the infrared structured light image and the background image of the same target, generate a target structured light image according to the difference between the gray values of corresponding pixels of the background image and the infrared structured light image, and generate a depth image according to the target structured light image for depth reconstruction or three-dimensional reconstruction. In the application, the depth reconstruction result of the measured target under the condition that the background light is strong can be improved, the time interval between the acquisition of adjacent two frames of images can be shortened, the difficulty of attack on the depth camera is improved, and the safety of the depth camera is higher.

Description

Technical Field

[0001] This invention relates to structured light 3D reconstruction, and more specifically, to a structured light remodeling group. Background Technology

[0002] Mobile payment has become the mainstream payment method in China. As of now, the number of mobile payment users has reached 1.05 billion, and the annual transaction volume of mobile payment has reached 30 trillion yuan, making it a pillar of the financial system. The explosion of mobile payment was driven by the popularization of smartphones. With the maturation of 4G / 5G, artificial intelligence, big data, and biometric technologies, 2019 ushered in the first year of facial recognition payment, which represents Payment 4.0. Facial recognition payment terminals are gradually being deployed in offline commercial scenarios and are about to be used on a large scale. It is estimated that the market size will reach 185 trillion yuan in 2022.

[0003] As a core component of facial recognition payment terminals, the facial recognition camera module plays a crucial role. Currently, the more mature facial recognition camera modules employ a structured light solution.

[0004] The structured light three-mode method is based on the principle of optical triangulation. An optical projector projects structured light of a specific pattern onto the surface of an object, forming a three-dimensional image of light stripes modulated by the shape of the object's surface. This three-dimensional image is detected by a camera at another location, thus obtaining a two-dimensional distorted image of the light stripes. The degree of distortion of the light stripes depends on the relative position between the optical projector and the camera, and the shape (height) of the object's surface. Intuitively, the displacement (or offset) along the light stripes is proportional to the height of the object's surface; twisting indicates changes in the plane, and discontinuity shows physical gaps on the surface. When the relative position between the optical projector and the camera is constant, the three-dimensional shape of the object's surface can be reconstructed from the coordinates of the distorted two-dimensional light stripe image.

[0005] Depth camera modules broaden the dimensions of front-end perception, effectively addressing the challenges of resisting spoofing attacks and reducing accuracy in extreme situations encountered in 2D face recognition. Their effectiveness has been recognized by the market, with strong demand, and they can be applied to scenarios such as door locks, access control, and payment systems based on 3D face recognition. For face recognition and similar applications, not only depth images of the target are needed, but also grayscale images. Generally, grayscale images are of two types: those captured by the same camera, those captured by different cameras, or a combination of both. In depth reconstruction applications, the acquired texture image includes the texture pattern projected by the projector and the background light. The texture pattern projected by the projector represents the effective signal, while the background light represents noise interference. In some situations, such as when the lighting intensity is strong, the background light is also strong, resulting in significant interference and a low signal-to-noise ratio. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a structured light remodeling group.

[0007] The structured light remodeling assembly provided by the present invention includes the following modules:

[0008] Structured light projector, used to project structured light onto a target;

[0009] A floodlight projector is used to project a floodlight onto a target.

[0010] Infrared cameras are used to acquire background images, infrared structured light images, and infrared images.

[0011] The processor module is used to acquire a preset frame rate value, and sequentially acquire a background image, an infrared structured light image, and an infrared image of the same target, or sequentially acquire an infrared image, an infrared structured light image, and a background image of the same target, generate a target structured light image by subtracting the gray values ​​of corresponding pixels in the background image and the infrared structured light image, and generate a depth image by performing depth reconstruction or three-dimensional reconstruction based on the target structured light image.

[0012] Preferably, image acquisition based on the frame rate value includes the following steps:

[0013] Step S1: Obtain the preset frame rate threshold and the original frame rate value, wherein the original frame rate value is the frame rate value of the infrared camera used to acquire infrared structured light images and infrared images;

[0014] Step S2: Determine the multiple between the original frame rate value and the frame rate threshold. When the multiple is less than or equal to a preset multiple threshold, the preset frame rate value is determined to be the product of the multiple threshold and the frame rate threshold. When the multiple is greater than or equal to the preset multiple threshold, the preset frame rate value is determined to be the original frame rate value.

[0015] Step S3: Based on the frame rate value, continuously and sequentially acquire the background image, infrared structured light image, and infrared image of the same target within a frame rate threshold sampling period, or continuously and sequentially acquire the infrared image, infrared structured light image, and background image of the same target.

[0016] Preferably, step S3 includes the following steps:

[0017] Step S301: Control the structured light projector and the floodlight projector to project structured light and floodlight sequentially toward the target;

[0018] Step S302: When the multiplier value is less than or equal to the preset multiplier threshold, the background image, infrared structured light image, and infrared image of the same target are continuously and sequentially acquired within the frame frequency threshold sampling period according to the frame frequency value, or the infrared image, infrared structured light image, and background image of the same target are continuously and sequentially acquired.

[0019] Step S303: When the multiplier value is greater than the preset multiplier threshold, according to the frame rate value, in each of the multiple sampling periods determined according to the frame rate threshold, three consecutive frames are sequentially acquired of the background image, infrared structured light image, and infrared image of the same target, or the infrared image, infrared structured light image, and background image of the same target are sequentially acquired.

[0020] Preferably, generating the target structured light image includes the following steps:

[0021] Step M1: Determine the pixel value of each pixel in the background image and the infrared structured light image;

[0022] Step M2: Align the background image and the infrared structured light image at the pixel level;

[0023] Step M3: Subtract the gray value of the corresponding pixel in the background image from the gray value of each pixel in the infrared structured light image to generate the target structured light image.

[0024] Preferably, the generation of the depth image includes the following steps:

[0025] Step N1: Calculate the disparity image of the target structured light image by comparing it with the known calibration information;

[0026] Step N2: Determine the distance between the optical center of the infrared camera and each disparity value in the disparity map based on the principle of triangulation, and generate depth information for each pixel;

[0027] Step N3: Perform depth reconstruction or 3D reconstruction based on the depth information of each pixel to generate a depth image.

[0028] Preferably, the infrared structured light image is an image containing coded textures, including any of the following structured light images:

[0029] - Scattered dot image;

[0030] - Striped image;

[0031] - Encoded image;

[0032] - Raster image.

[0033] Preferably, the preset frame rate threshold is 15 FPS, and the preset multiplier threshold is 3.

[0034] Preferably, the structured light projector includes a light source, a light source driver, and a light modulator;

[0035] The light source driver is connected to the light source and is used to drive the light source to emit light;

[0036] The light modulator is used to modulate the light projected by the light source into discrete dot matrix light and then project it onto the target.

[0037] Preferably, the infrared camera includes an optical imaging lens and a photodetector array; the photodetector array includes multiple photodetectors arranged in an array.

[0038] The optical imaging lens is used to ensure that the direction vector of the dot matrix light entering the photodetector array through the optical imaging lens has a one-to-one correspondence with the photodetector.

[0039] The photodetector is used to receive dot matrix light reflected by the target object.

[0040] The structured light remodeling assembly provided by the present invention includes the following modules:

[0041] Structured light projector, used to project structured light onto a target;

[0042] A floodlight projector is used to project a floodlight onto a target.

[0043] Infrared cameras are used to acquire background images, infrared structured light images, and infrared images.

[0044] The processor module is used to acquire a preset frame rate value, continuously acquire background images, infrared structured light images, and infrared images based on the frame rate value, and perform depth reconstruction or three-dimensional reconstruction based on the infrared structured light images to generate depth images.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] In this invention, the background image, infrared structured light image, and infrared image of the same target are continuously and sequentially acquired according to a preset frame rate value, or the infrared image, infrared structured light image, and background image of the same target are continuously and sequentially acquired, thereby realizing the continuous acquisition of three frames of images. This shortens the acquisition time interval between two adjacent frames of images, increases the difficulty of attacking the depth camera, and makes the depth camera more secure.

[0047] In this invention, a target structured light image is generated by subtracting the gray values ​​of corresponding pixels in the background image and the infrared structured light image. Then, a depth image is generated by performing depth reconstruction or three-dimensional reconstruction based on the target structured light image, which reduces the interference of background light and makes the depth camera suitable for environments with strong light intensity. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0049] Figure 1 This is a schematic diagram illustrating the working principle of the structured light remodeling group in an embodiment of the present invention;

[0050] Figure 2 This is a flowchart illustrating the steps for determining the frame rate value in an embodiment of the present invention;

[0051] Figure 3 This is a flowchart illustrating the steps of acquiring images based on frame rate values ​​in an embodiment of the present invention.

[0052] Figure 4 This is a flowchart illustrating the steps for generating a target structured light image in an embodiment of the present invention;

[0053] Figure 5 This is a flowchart illustrating the steps of depth reconstruction to generate a depth image in an embodiment of the present invention.

[0054] Figure 6 This is a schematic diagram of a structured light projector module in an embodiment of the present invention; and

[0055] Figure 7 This is a schematic diagram of the infrared camera module in an embodiment of the present invention. Detailed Implementation

[0056] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0057] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0059] The structured light remodeling group provided by this invention aims to solve the problems existing in the prior art.

[0060] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0061] Figure 1 This is a schematic diagram illustrating the working principle of the structured light remodeling group in an embodiment of the present invention, as shown below. Figure 1 As shown, the structured light remodeling group provided by this invention includes the following modules:

[0062] Structured light projector, used to project structured light onto a target;

[0063] A floodlight projector is used to project a floodlight onto a target.

[0064] Infrared cameras are used to acquire background images, infrared structured light images, and infrared images.

[0065] The processor module is used to acquire a preset frame rate value, and sequentially acquire a background image, an infrared structured light image, and an infrared image of the same target, or sequentially acquire an infrared image, an infrared structured light image, and a background image of the same target, generate a target structured light image by subtracting the gray values ​​of corresponding pixels in the background image and the infrared structured light image, and generate a depth image by performing depth reconstruction or three-dimensional reconstruction based on the target structured light image.

[0066] When using the depth camera provided by this invention, a background image can be acquired first using an infrared camera, then an infrared structured light image can be acquired after projecting structured light onto the target using a structured light projector, and finally an infrared image can be acquired after projecting flood light onto the target using a flood illuminator. Alternatively, a background image can be acquired first using an infrared camera, then an infrared structured light image can be acquired after projecting structured light onto the target using a structured light projector, and finally an infrared image can be acquired after projecting flood light onto the target using a flood illuminator. The infrared camera used is a 940nm infrared camera. The flood illuminator uses an LED light source.

[0067] Figure 2 This is a flowchart illustrating the steps for determining the frame rate value in an embodiment of the present invention, as follows: Figure 2 As shown, determining the frame rate value and performing image acquisition based on the frame rate value includes the following steps:

[0068] Step S1: Obtain the preset frame rate threshold and the original frame rate value, wherein the original frame rate value is the frame rate value of the infrared camera used to acquire infrared structured light images and infrared images;

[0069] Step S2: Determine the multiple between the original frame rate value and the frame rate threshold. When the multiple is less than or equal to a preset multiple threshold, the preset frame rate value is determined to be the product of the multiple threshold and the frame rate threshold. When the multiple is greater than or equal to the preset multiple threshold, the preset frame rate value is determined to be the original frame rate value.

[0070] Step S3: Based on the frame rate value, continuously and sequentially acquire the background image, infrared structured light image, and infrared image of the same target within a frame rate threshold sampling period, or continuously and sequentially acquire the infrared image, infrared structured light image, and background image of the same target.

[0071] In this embodiment of the invention, the preset frame rate threshold is 15 FPS, and the preset multiplier threshold is 3. When the original frame rate value is 30 FPS, the multiplier value is 2. Since the multiplier value 2 is less than the preset multiplier threshold 3, the preset frame rate value is determined to be the preset multiplier threshold 3 multiplied by the preset frame rate threshold 15, and the frame rate value is determined to be 45. When the original frame rate value is 60 FPS, the multiplier value is 4. Since the multiplier value 4 is greater than the preset multiplier threshold 3, the preset frame rate value is determined to be the original frame rate value of 60 FPS.

[0072] Figure 3 This is a flowchart illustrating the steps of acquiring images based on frame rate values ​​in an embodiment of the present invention, as follows: Figure 3 As shown, step S103 includes the following steps:

[0073] Step S301: Project structured light and floodlight onto the target using the light projector of the depth camera;

[0074] Step S302: When the multiplier value is less than or equal to the preset multiplier threshold, the background image, infrared structured light image, and infrared image of the same target are continuously and sequentially acquired within the frame frequency threshold sampling period according to the frame frequency value, or the infrared image, infrared structured light image, and background image of the same target are continuously and sequentially acquired.

[0075] Step S303: When the multiplier value is greater than the preset multiplier threshold, according to the frame rate value, in each of the multiple sampling periods determined according to the frame rate threshold, three consecutive frames are sequentially acquired of the background image, infrared structured light image, and infrared image of the same target, or the infrared image, infrared structured light image, and background image of the same target are sequentially acquired.

[0076] In this embodiment of the invention, when the multiplier value is 2, and the multiplier value is less than or equal to the preset multiplier threshold, the background image, infrared structured light image, and infrared image of the same target are continuously and sequentially acquired within 15 sampling periods of the frame frequency threshold according to the frame frequency value 45, or the infrared image, infrared structured light image, and background image of the same target are continuously and sequentially acquired.

[0077] Similarly, for example, when the multiplier value is 4, and the multiplier value is greater than the preset multiplier threshold, then the background image, infrared structured light image, and infrared image of the same target are continuously and sequentially acquired every 4 frames within 15 sampling periods of the frame frequency threshold, or the infrared image, infrared structured light image, and background image of the same target are continuously and sequentially acquired.

[0078] In this embodiment of the invention, the infrared structured light image is an image containing coded textures, including but not limited to any of the following structured light images:

[0079] - Scattered dot image;

[0080] - Striped image;

[0081] - Encoded image;

[0082] - Raster image.

[0083] Figure 4 This is a flowchart illustrating the steps for generating a target structured light image in an embodiment of the present invention, as follows: Figure 4 As shown, the generation of the target structured light image includes the following steps:

[0084] Step M1: Determine the grayscale value of each pixel in the background image and the infrared structured light image;

[0085] Step M2: Align the background image and the infrared structured light image at the pixel level;

[0086] Step M3: Subtract the gray value of the corresponding pixel in the background image from the gray value of each pixel in the infrared structured light image to generate the target structured light image.

[0087] Figure 5 This is a flowchart illustrating the steps of depth reconstruction to generate a depth image in an embodiment of the present invention, as follows: Figure 5 As shown, the generation of the depth image includes the following steps:

[0088] Step N1: Calculate the disparity image of the target structured light image by comparing it with the known calibration information;

[0089] Step N2: Determine the distance between the optical center of the infrared camera and each disparity value in the disparity map based on the principle of triangulation, and generate depth information for each pixel;

[0090] Step N3: Perform depth reconstruction or 3D reconstruction based on the depth information of each pixel to generate a depth image.

[0091] Figure 6 This is a schematic diagram of a structured light projector module in an embodiment of the present invention, such as... Figure 6 As shown, the structured light projector includes a light source, a light source driver, and a light modulator;

[0092] The light source driver is connected to the light source and is used to drive the light source to emit light;

[0093] The light modulator is used to modulate the light projected by the light source into discrete dot matrix light and then project it onto the object under test.

[0094] In this embodiment of the invention, the optical modulator is a diffraction grating (DOE) or a spatial light modulator (SLM).

[0095] Figure 7 This is a schematic diagram of the infrared camera module in an embodiment of the present invention, as shown below. Figure 7 As shown, the infrared camera includes an optical imaging lens 1 and a photodetector array 3; the photodetector array 3 includes multiple photodetectors arranged in an array.

[0096] The optical imaging lens 1 is used to make the direction vector of the collimated beam entering the photodetector array through the optical imaging lens correspond one-to-one with the photodetector.

[0097] The photodetector is used to receive the collimated light beam reflected by the target object.

[0098] In this embodiment of the invention, to filter background noise, the optical imaging lens typically also includes a narrowband filter 2, ensuring that the photodetector array can only pass incident collimated beams of a preset wavelength. The preset wavelength can be the wavelength of the incident collimated beam, or it can be between 50 nanometers less than and greater than 50 nanometers of the incident collimated beam. The photodetector array can be arranged periodically or non-periodically. Depending on the required number of discrete collimated beams, the photodetector array can be a combination of multiple single-point photodetectors or a sensor chip integrating multiple photodetectors. To further optimize the sensitivity of the photodetectors, the illumination spot of a discrete collimated beam on the target can correspond to one or more photodetectors. When multiple photodetectors correspond to the same illumination spot, the signals from each detector can be connected through a circuit, thereby merging into a photodetector with a larger detection area.

[0099] In this embodiment of the invention, the photodetector can be a CMOS photodetector, a CCD photodetector, or a SPAD photodetector. The detector end is an infrared detector, which receives the dot matrix light reflected by the target.

[0100] In this embodiment of the invention, the background image, infrared structured light image, and infrared image of the same target are continuously and sequentially acquired according to a preset frame rate value, or the infrared image, infrared structured light image, and background image of the same target are continuously acquired, thereby achieving continuous acquisition of three frames of images. This shortens the acquisition time interval between two adjacent frames, increases the difficulty of attacking the depth camera, and makes the depth camera more secure. In this embodiment of the invention, the target structured light image is generated by subtracting the gray values ​​of corresponding pixels in the background image and the infrared structured light image. Then, depth reconstruction or 3D reconstruction is performed based on the target structured light image to generate a depth image, reducing the interference of background light and enabling the depth camera to be used in environments with strong light intensity.

[0101] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0102] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A structured light reconstruction module, comprising: The application comprises the following modules: a structured light projector for projecting structured light to a target; a floodlight projector for projecting floodlight to a target; an infrared camera for collecting background images, infrared structured light images and infrared images; a processor module for obtaining a preset frame frequency value, continuously and sequentially collecting background images, infrared structured light images and infrared images of the same target according to the preset frame frequency value, or continuously and sequentially collecting infrared images, infrared structured light images and background images of the same target, generating target structured light images by subtracting the gray values of corresponding pixels of the background images and the infrared structured light images, and generating depth images by depth reconstruction or three-dimensional reconstruction according to the target structured light images; when collecting images according to the preset frame frequency value, the following steps are included: Step S1: obtaining a preset frame frequency threshold value and an original frame frequency value, the original frame frequency value being the frame frequency value of the infrared camera for collecting infrared structured light images and infrared images; Step S2: determining a multiple value between the original frame frequency value and the frame frequency threshold value, when the multiple value is less than or equal to a preset multiple threshold value, determining the preset frame frequency value as the product of the multiple threshold value and the frame frequency threshold value, and when the multiple value is greater than or equal to the preset multiple threshold value, determining the preset frame frequency value as the original frame frequency value; Step S3: continuously and sequentially collecting background images, infrared structured light images and infrared images of the same target within a frame frequency threshold value number of sampling periods according to the preset frame frequency value, or continuously and sequentially collecting infrared images, infrared structured light images and background images of the same target; when generating depth images, the following steps are included: Step N1: calculating the target structured light images with known calibration information to obtain parallax images of the target structured light images; Step N2: determining the distance between the optical center of the infrared camera and each parallax value in the parallax image according to the principle of triangulation to generate depth information of each pixel; Step N3: generating depth images by depth reconstruction or three-dimensional reconstruction according to the depth information of each pixel.

2. The structured light reconstruction module of claim 1, wherein, Step S3 includes the following steps: Step S301: controlling the structured light projector and the floodlight projector to sequentially project structured light and floodlight to a target; Step S302: when the multiple value is less than or equal to the preset multiple threshold value, continuously and sequentially collecting background images, infrared structured light images and infrared images of the same target within a frame frequency threshold value number of sampling periods according to the preset frame frequency value, or continuously and sequentially collecting infrared images, infrared structured light images and background images of the same target; Step S303: when the multiple value is greater than the preset multiple threshold value, sequentially collecting any three consecutive frames of background images, infrared structured light images and infrared images of the same target in each sampling period within a plurality of sampling periods determined according to the frame frequency threshold value according to the preset frame frequency value, or sequentially collecting infrared images, infrared structured light images and background images of the same target.

3. The structured light reconstruction module of claim 1, wherein, when generating target structured light images, the following steps are included: Step M1: determining the pixel value of each pixel in the background images and the infrared structured light images; Step M2: pixel-level alignment of the background image and the infrared structured light image; Step M3: subtracting the grayscale value of the corresponding pixel of the background image from each pixel in the infrared structured light image to generate a target structured light image.

4. The structured light reconstruction module of claim 1, wherein, The infrared structured light image includes any one of the following structured light images: - a speckle image; - a stripe image; - a coded image; - a grating image.

5. The structured light reconstruction module of claim 1, wherein, The preset frame frequency threshold is 15 FPS, and the preset multiple threshold is 3.

6. The structured light reconstruction module of claim 1, wherein, The structured light projector includes a light source, a light source driver, and a light modulator; The light source driver is connected with the light source and is configured to drive the light source to emit light; The light modulator is configured to modulate the projected light of the light source into discrete dot array light and project the light back to the target.

7. The structured light reconstruction module of claim 6, wherein, The infrared camera includes an optical imaging lens and a light detector array; the light detector array includes a plurality of light detectors arranged in an array; The optical imaging lens is configured to make the direction vectors of the dot array light entering the light detector array through the optical imaging lens and the light detectors in a one-to-one correspondence; The light detector is configured to receive the dot array light reflected by the target object.

Images