A multi-frame depth reconstruction method, system, device and storage medium

Abstract

The application provides a multi-frame depth reconstruction method, system, device and storage medium, comprising the following steps: acquiring a preset frame frequency value, continuously and sequentially acquiring a first depth image, a background image and a second depth image of a same target according to the frame frequency value; generating a first effective image by subtracting the gray values of corresponding pixels of the first depth image and the background image; generating a second effective image by subtracting the gray values of corresponding pixels of the second depth image and the background image; and performing multi-frame depth reconstruction or three-dimensional reconstruction according to the first effective image and the second effective image to generate a depth image. The application controls the image acquisition sequence and frequency, and performs corresponding processing on the image, so that the multi-frame depth reconstruction result of the measured target under the condition that the background light is relatively strong can be improved, the time interval between the acquisition of adjacent two images can be shortened, and the difficulty of the attack on the depth camera is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to structured light three-dimensional reconstruction, in particular, to a multi-frame depth reconstruction method, system, device and storage medium. BACKGROUND

[0002] Mobile payment has become a mainstream payment method in China and plays an increasingly important role in more and more fields. As the core device of the face payment terminal, the face recognition camera module plays a very key role. The current mature face recognition camera module adopts a structured light scheme.

[0003] The structured light three scheme is based on the optical triangulation measurement principle. An optical projector projects a certain pattern of structured light onto the surface of an object, forming a three-dimensional image of the light strip modulated by the shape of the measured object surface on the surface. The three-dimensional image is detected by another camera at another position, thereby obtaining a two-dimensional distorted image of the light strip. The distortion degree of the light strip depends on the relative position between the optical projector and the camera and the surface profile (height) of the object. Intuitively, the displacement (or offset) shown along the light strip is proportional to the height of the object surface, the kink represents the change of the plane, and the discontinuity shows the physical gap of the surface. When the relative position between the optical projector and the camera is certain, the three-dimensional profile of the object surface can be reproduced from the distorted two-dimensional light strip image coordinates.

[0004] The depth camera module widens the dimension of the front-end perception, and can well solve the problems of anti-prosthesis attack and reduced recognition accuracy in extreme conditions encountered by 2D face recognition, and the effect has been recognized by the market, the demand is strong, and can be applied to door locks, access control and payment scenes based on 3D face recognition. When face recognition and other needs are required, not only the depth image of the target is needed, but also the gray image. Generally, the gray image is divided into two types, one is the gray image shot by the same camera or the gray image shot by different cameras or both. In the application of multi-frame depth reconstruction, the texture image collected contains the texture pattern projected by the projector and the background light, wherein the texture pattern projected by the projector is equivalent to the effective signal, and the background light is equivalent to the noise interference. In some cases, such as strong light intensity, the background light is relatively strong, the interference is relatively serious, and the signal-to-noise ratio is low. SUMMARY

[0005] Therefore, the present application controls the image acquisition sequence and frequency, and processes the image accordingly, which can improve the multi-frame depth reconstruction result of the measured target in the case of strong background light, shorten the time interval between adjacent two frames of images, improve the difficulty of attack on the depth camera, and make the depth camera more secure and the data more clear.

[0006] In a first aspect, the present application provides a multi-frame depth reconstruction method, characterized in that it comprises the following steps:

[0007] Step S1: obtaining a preset frame frequency value, and sequentially collecting a first depth image, a background image and a second depth image of a same target according to the frame frequency value;

[0008] Step S2: generating a first effective image by subtracting the gray values of corresponding pixels of the first depth image and the background image, and generating a second effective image by subtracting the gray values of corresponding pixels of the second depth image and the background image;

[0009] Step S3: performing multi-frame depth reconstruction or three-dimensional reconstruction according to the first effective image and the second effective image to generate a depth image.

[0010] Optionally, the multi-frame depth reconstruction method has the following steps.

[0011] Step S101: obtaining a preset frame frequency threshold and an original frame frequency value, wherein the original frame frequency value is a frame frequency value of a depth camera used for collecting the first depth image and the second depth image;

[0012] Step S102: determining a multiple value between the original frame frequency value and the frame frequency threshold, wherein when the multiple value is less than or equal to a preset multiple threshold, the preset frame frequency value is determined as a product of the multiple threshold and the frame frequency threshold, and when the multiple value is greater than or equal to the preset multiple threshold, the preset frame frequency value is determined as the original frame frequency value;

[0013] Step S103: sequentially collecting the first depth image, the background image and the second depth image of the same target according to the frame frequency value in a frame frequency threshold number of sampling periods.

[0014] Optionally, the multi-frame depth reconstruction method has the following steps.

[0015] Step S1031: projecting structured light and flood light to the target by a light projector of the depth camera.

[0016] Step S1032: when the multiple value is less than or equal to the preset multiple threshold, sequentially collecting an infrared structured light image, a background image and an infrared image of the same target or sequentially collecting an infrared image, a background image and an infrared structured light image of the same target according to the frame frequency value in the frame frequency threshold number of sampling periods.

[0017] Step S1033: When the multiple value is greater than a preset multiple threshold, sequentially capturing infrared structured light images, background images, infrared images of the same target in any three consecutive frames in each sampling period of a plurality of sampling periods determined according to a frame frequency threshold according to the frame frequency value, or sequentially capturing infrared images, background images, infrared structured light images of the same target.

[0018] Optionally, the multi-frame depth reconstruction method has the step S2 comprising the following steps.

[0019] Step S201: determining pixel values of each pixel in the first depth image, the second depth image and the background image.

[0020] Step S202: performing pixel-level alignment on the first depth image, the second depth image and the background image.

[0021] Step S203: subtracting a gray value of a corresponding pixel of the background image from each pixel in the first depth image to generate a first effective image, and subtracting the gray value of the corresponding pixel of the background image from each pixel in the second depth image to generate a second effective image.

[0022] Optionally, the multi-frame depth reconstruction method has the step S3 comprising the following steps.

[0023] Step S301: calculating the infrared structured light image based on known calibration information to obtain a parallax image of the infrared structured light image.

[0024] Step S302: determining a distance between an optical center of the depth camera and each parallax value in the parallax image based on a triangulation principle to generate depth information of each pixel.

[0025] Step S303: performing multi-frame depth reconstruction or three-dimensional reconstruction based on the depth information of each pixel to generate a depth image.

[0026] Optionally, the multi-frame depth reconstruction method has the step S3 comprising the following steps.

[0027] Step S31: synthesizing the first effective image and the second effective image to obtain a third effective image.

[0028] Step S32: performing multi-frame depth reconstruction or three-dimensional reconstruction based on the third effective image to generate a depth image.

[0029] Optionally, the multi-frame depth reconstruction method has the preset frame frequency threshold being 15 FPS and the preset multiple threshold being 3.

[0030] In a second aspect, the present application provides a multi-frame depth reconstruction system for implementing the multi-frame depth reconstruction method described above, characterized in that it comprises:

[0031] an image acquisition module for acquiring a preset frame frequency value, and sequentially acquiring a first depth image, a background image and a second depth image of the same target according to the frame frequency value;

[0032] an image enhancement module for generating a first effective image by subtracting the gray values of corresponding pixels of the first depth image and the background image, and generating a second effective image by subtracting the gray values of corresponding pixels of the second depth image and the background image;

[0033] a multi-frame depth reconstruction module for performing multi-frame depth reconstruction or three-dimensional reconstruction to generate a depth image according to the first effective image and the second effective image.

[0034] In a third aspect, the present application provides a multi-frame depth reconstruction device, characterized in that it comprises:

[0035] a processor;

[0036] a memory having executable instructions of the processor stored therein;

[0037] wherein the processor is configured to execute the steps of the multi-frame depth reconstruction method described in any of the above aspects by executing the executable instructions.

[0038] In a fourth aspect, the present application provides a computer readable storage medium for storing a program, characterized in that the program, when executed, implements the steps of the multi-frame depth reconstruction method described in any of the above aspects.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] In the present application, the first depth image, the background image and the second depth image of the same target are sequentially acquired according to a preset frame frequency value, thereby realizing the continuous acquisition of three frames of images, shortening the time interval between the acquisition of adjacent two frames of images, improving the difficulty of attack on the depth camera, and making the depth camera more secure;

[0041] In the present application, the first effective image and the second effective image are generated by subtracting the gray values of corresponding pixels of the first depth image and the background image, and the second depth image and the background image, and then the depth image is generated by performing multi-frame depth reconstruction or three-dimensional reconstruction according to the first effective image and the second effective image, thereby reducing the interference of background light and making the depth camera applicable to environments with strong light intensity. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only aim at the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on the provided drawings. Other features, objects and advantages of the present application will become more apparent from the following detailed description of the non-limiting embodiments with reference to the following drawings:

[0043] Figure 1 A step flow chart of a multi-frame depth reconstruction method in an embodiment of the present application;

[0044] Figure 2 A transmitting and receiving schematic diagram in an embodiment of the present application;

[0045] Figure 3 A step flow chart of determining a frame frequency value in an embodiment of the present application;

[0046] Figure 4 A step flow chart of collecting an image according to the frame frequency value in an embodiment of the present application;

[0047] Figure 5 A step flow chart of generating a target structured light image in an embodiment of the present application;

[0048] Figure 6 A step flow chart of generating a depth image by performing multi-frame depth reconstruction in an embodiment of the present application;

[0049] Figure 7 A combining flow chart of a first effective image and a second effective image in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of collecting an image by a depth camera in an embodiment of the present application;

[0051] Figure 9 A module schematic diagram of a multi-frame depth reconstruction system in an embodiment of the present application;

[0052] Figure 10 A structure schematic diagram of a multi-frame depth reconstruction device in an embodiment of the present application; and

[0053] Figure 11 A structure schematic diagram of a computer readable storage medium in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The application will be described in detail below with specific examples. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made. These are within the scope of the application.

[0055] The terms "first", "second", "third", "fourth" and the like in the description, claims, and drawings of the application, and those above, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that, for example, embodiments of the application described herein can operate in other sequences than the one illustrated or other than the one explicitly described herein. Moreover, the terms "comprise", "comprising", "have", "having", "include", "including", and "contains", "containing", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, or contains a list of steps or elements does not include only those steps or elements but can include other not expressly listed steps or elements.

[0056] The technical solutions of the application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples.

[0057] The application provides a multi-frame depth reconstruction method, which aims to solve the problems in the prior art.

[0058] The technical solutions of the application and how the technical solutions of the application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples. The embodiments of the application will be described below with reference to the drawings.

[0059] Figure 1 The flow chart of the steps of the multi-frame depth reconstruction method in the embodiments of the application. As shown in Figure 1 The multi-frame depth reconstruction method provided by the application comprises the following steps:

[0060] Step S1: obtaining a preset frame frequency value, and sequentially acquiring a first depth image, a background image and a second depth image of the same target according to the frame frequency value.

[0061] In this step, the first depth image and the second depth image can be the same type of depth image or different types of depth image, for example, the first depth image and the second depth image can be infrared images, or the first depth image is an infrared image and the second depth image is a structured light image. The first depth image and the second depth image are depth images obtained by using active laser irradiation on the target object. The background image is a depth image obtained without active laser irradiation, and the commonly used is an infrared image obtained by an infrared sensor.

[0062] Figure 2 is a schematic diagram of a transmitting and receiving embodiment in the present application. As can be seen from Figure 2 It can be seen that f1 and f4 are the first depth image, f2 is the background image, and f3 and f5 are the second depth image. And f1, f2 and f3 are a group of signals, and f4, f5 and f6 are another group of signals. This way can make the signal obtain a higher frequency, and through the way of sharing the background signal, the frame frequency is improved. It should be noted that f1 and f3 are both transmitting a group of signals, which is not limited to one signal, but can be two or more signals combined, that is, f1 and f3 are each composed of two or more signals, so that the present embodiment can have more application forms and play a role in a wider application range.

[0063] Step S2: generating a first effective image according to the gray value subtraction of the corresponding pixels of the first depth image and the background image; and generating a second effective image according to the gray value subtraction of the corresponding pixels of the second depth image and the background image.

[0064] In this step, the first depth image, the second depth image and the background image all use the same image size, that is, any pixel point in the first depth image and the second depth image has a unique corresponding pixel point on the background image. The first depth image and the second depth image share a background image, which can improve the data acquisition frequency and obtain image data faster.

[0065] Step S3: performing multi-frame depth reconstruction or three-dimensional reconstruction according to the first effective image and the second effective image to generate a depth image.

[0066] In this step, the first effective image and the second effective image are depth data obtained at different times, which can cooperate with each other to perform multi-frame depth reconstruction or three-dimensional reconstruction.

[0067] Figure 3 is a step flow chart for determining the frame frequency value in the embodiment of the present application. Compared with step S1 in the previous embodiment, the present embodiment further includes the following steps:

[0068] Step S101: Obtain a preset frame frequency threshold value and an original frame frequency value, the original frame frequency value being a frame frequency value of a depth camera used for collecting a first depth image and a second depth image.

[0069] Step S102: Determine a multiple value between the original frame frequency value and the frame frequency threshold value, when the multiple value is less than or equal to a preset multiple threshold value, determine the preset frame frequency value as a product of the multiple threshold value and the frame frequency threshold value, when the multiple value is greater than or equal to the preset multiple threshold value, determine the preset frame frequency value as the original frame frequency value.

[0070] Step S103: Collect the first depth image, the background image, and the second depth image of the same target in sequence continuously within a frame frequency threshold value according to the frame frequency value.

[0071] In the embodiment of the present application, the preset frame frequency threshold value is 15 FPS, the preset multiple threshold value is 3, when the original frame frequency value is 30 FPS, the multiple value is 2, since the multiple value 2 is less than the preset multiple threshold value 3, the preset frame frequency value is determined as the preset multiple threshold value 3 multiplied by the preset frame frequency threshold value 15, and the frame frequency value is determined as 45; when the original frame frequency value is 60 FPS, the multiple value is 4, since the multiple value 4 is greater than the preset multiple threshold value 3, the preset frame frequency value is determined as the original frame frequency value 60 FPS.

[0072] Figure 4 The step flow chart for collecting images according to the frame frequency value in the embodiment of the present application. Compared with step S103 in the above embodiment, the present embodiment further includes the following steps:

[0073] Step S1031: Project structured light and flood light to the target through the light projector of the depth camera.

[0074] In the present step, two different laser types, structured light and flood light, are respectively irradiated.

[0075] Step S1032: When the multiple value is less than or equal to the preset multiple threshold value, collect infrared structured light images, background images, infrared images of the same target in sequence continuously within a frame frequency threshold value according to the frame frequency value, or collect infrared images, background images, infrared structured light images of the same target in sequence continuously.

[0076] Step S1033: When the multiple value is greater than the preset multiple threshold value, collect infrared structured light images, background images, infrared images of the same target in sequence continuously in any three frames in each sampling period within a plurality of sampling periods determined according to the frame frequency threshold value according to the frame frequency value, or collect infrared images, background images, infrared structured light images of the same target in sequence continuously.

[0077] In the embodiment of the present application, when the multiple value is 2, and the multiple value is less than or equal to the preset multiple threshold value, then the background image, the infrared structured light image, the infrared image of the same target are sequentially acquired in sequence within the frame frequency threshold value 15 sampling periods according to the frame frequency value 45, or the infrared image, the infrared structured light image, the background image of the same target are sequentially acquired in sequence.

[0078] By analogy, for example, when the multiple value is 4, and the multiple value is greater than the preset multiple threshold value, then the background image, the infrared structured light image, the infrared image of the same target are sequentially acquired in sequence every 4 frames within the frame frequency threshold value 15 sampling periods according to the frame frequency value 60, or the infrared image, the infrared structured light image, the background image of the same target are sequentially acquired in sequence.

[0079] Figure 5 The step flow chart for generating the target structured light image in the embodiment of the present application. Compared with the step S2 in the above embodiment, the embodiment further includes the following steps:

[0080] Step S201: Determine the pixel value of each pixel in the first depth image, the second depth image and the background image.

[0081] Step S202: Align the first depth image, the second depth image and the background image at the pixel level.

[0082] Step S203: Subtract the gray value of the corresponding pixel of the background image from each pixel in the first depth image to generate a first effective image; subtract the gray value of the corresponding pixel of the background image from each pixel in the second depth image to generate a second effective image.

[0083] Figure 6 The step flow chart for generating the depth image by multi-frame depth reconstruction in the embodiment of the present application. Compared with the step S3 in the above embodiment, the embodiment further includes the following steps:

[0084] Step S301: Calculate the parallax image of the infrared structured light image with the known calibration information.

[0085] Step S302: Determine the distance between the optical center of the depth camera and each parallax value in the parallax image according to the principle of triangulation to generate the depth information of each pixel.

[0086] Step S303: Generate the depth image by multi-frame depth reconstruction or three-dimensional reconstruction according to the depth information of each pixel.

[0087] Figure 7The flow chart of combining the first valid image and the second valid image in the embodiment of the present application. Compared with step S3 in the above embodiment, the embodiment further comprises the following steps:

[0088] Step S31: synthesizing the first valid image and the second valid image to obtain a third valid image.

[0089] In this step, the first valid image and the second valid image are synthesized so that there is only one valid image, i.e. the third valid image, in the group of signals. Compared with the first valid image and the second valid image, the third valid image has a better signal-to-noise ratio and better recognition in an environment with a relatively strong ambient light intensity.

[0090] Step S32: generating a depth image according to the third valid image through multi-frame depth reconstruction or three-dimensional reconstruction.

[0091] In this step, the third valid image is very important for the reconstructed depth image, and therefore the quality of the third valid image needs to be considered, and different module current values, exposure times and gain (Gain) values are selected so that the pixel values of the obtained third valid image are within an unexposed range.

[0092] In some embodiments, the intensity value and the judgment threshold value pre-stored in the original system are used to calculate a judgment threshold value suitable for the third valid image. In the original system, the depth camera generally has the intensity value and the judgment threshold value of the camera, which are used to adjust within a certain range according to the ambient light intensity. In this step, the existing information in the system is used to effectively reconstruct in the case of a relatively large external noise environment, and the terminal no longer needs to be calculated or tested, thereby improving the efficiency of system arrangement and reducing the cost of arrangement.

[0093] In this embodiment, the two signal frames share the same background frame, the frame time difference is reduced, signal failure caused by motion blur is prevented, and the signal-to-noise ratio is increased by two times through the superposition of the three frames of signals.

[0094] Figure 8 The schematic diagram of the collected image of the depth camera in the embodiment of the present application is shown in FIG. 1. Figure 8 As shown in FIG. 1, when the depth camera provided by the present application is used, the background image can be first collected by the infrared camera, then the infrared structure light image is collected after the structure light is projected onto the target by the structure light projector, and finally the infrared image is collected after the floodlight is projected onto the target by the floodlight projector. The infrared camera adopts a 940nm infrared camera. The floodlight projector adopts an LED light source.

[0095] Figure 9 As a schematic diagram of the module of the multi-frame depth reconstruction system in the embodiment of the present application, for realizing the multi-frame depth reconstruction method, it is characterized by comprising:

[0096] The image acquisition module 101 is configured to acquire a preset frame frequency value, and sequentially acquire a first depth image, a background image and a second depth image of the same target according to the frame frequency value.

[0097] The image enhancement module 102 is configured to generate a first effective image by subtracting the gray value of the corresponding pixels of the first depth image and the background image, and generate a second effective image by subtracting the gray value of the corresponding pixels of the second depth image and the background image.

[0098] The multi-frame depth reconstruction module 103 is configured to perform multi-frame depth reconstruction or three-dimensional reconstruction to generate a depth image according to the first effective image and the second effective image.

[0099] The embodiment of the present application also provides a multi-frame depth reconstruction device, which comprises a processor and a memory having executable instructions of the processor stored therein. The processor is configured to perform the steps of the multi-frame depth reconstruction method by executing the executable instructions.

[0100] As described above, the embodiment can sequentially acquire the background image, the infrared structured light image and the infrared image of the same target according to the preset frame frequency value, or sequentially acquire the infrared image, the infrared structured light image and the background image of the same target, realize the continuous acquisition of three frames of images, shorten the time interval between the acquisition of adjacent two frames of images, improve the difficulty of the attack on the depth camera, and make the depth camera more secure.

[0101] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software, which can be collectively referred to as "circuitry", "module" or "platform" here.

[0102] Figure 10 is a structural schematic diagram of the multi-frame depth reconstruction device in the embodiment of the present application. The electronic device 600 according to this embodiment of the present application will be described below with reference to Figure 10 The electronic device 600 shown is merely an example, and should not impose any limitation on the functions and use range of the embodiment of the present application. Figure 10 The electronic device 600 shown is merely an example, and should not impose any limitation on the functions and use range of the embodiment of the present application.

[0103] As Figure 10As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 that connects the various platform components including the storage unit 620 and the processing unit 610, a display unit 640, etc.

[0104] The storage unit stores program code that can be executed by the processing unit 610 to cause the processing unit 610 to perform the steps described above in the various exemplary embodiments of the present application. For example, the processing unit 610 can perform the steps shown in FIG. 6. Figure 1

[0105] The storage unit 620 can include a readable medium in the form of volatile storage such as a random access memory (RAM) 6201 and / or cache memory 6202, and also can include a non-volatile storage such as a read-only memory (ROM) 6203.

[0106] The storage unit 620 can further include a program / utility 6204 having a set of program modules 6205 that include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which may

[0107] The bus 630 can represent one or more of several types of bus structures, including a storage bus or bus controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus architectures.

[0108] The electronic device 600 also can communicate with one or more external devices 700 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices that enable a user to interact with the electronic device 600; and / or one or more devices that enable the electronic device 600 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 650. Still yet, the electronic device 600 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, via a network adapter 660. The network adapter 660 can be any of a plurality of different types of adapters to enable the electronic device 600 to communicate with such networks and is typically implemented as either a peripheral component or an on-board component of the electronic device 600. It will be appreciated that, although not shown, the electronic device 600 can further include a power supply, which can include a small battery to provide power to the electronic device 600 when the electronic device 600 is not being powered by a general power source such as a wall outlet. Figure 8 ​Other hardware and / or software modules can be used in conjunction with electronic device 600, as desired, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0109] The embodiment of the present application also provides a computer readable storage medium for storing a program, which, when executed, implements the steps of a multi-frame depth reconstruction method. In some possible implementation manners, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the above method part of the present application with reference to various exemplary embodiments of the present application when the program product is run on the terminal device.

[0110] As shown above, the program of the computer readable storage medium of the embodiment, when executed, continuously and sequentially collects the background image, the infrared structured light image, or the infrared image of the same target according to the preset frame frequency value, or continuously and sequentially collects the infrared image, the infrared structured light image, and the background image of the same target, realizes the continuous collection of three frames of images, shortens the time interval of the collection between adjacent two frames of images, improves the difficulty of the attack on the depth camera, and makes the security of the depth camera higher.

[0111] Figure 11 is a structural schematic diagram of the computer readable storage medium in the embodiment of the present application. Referring to Figure 11 As shown in the figure, the program product 800 for implementing the above method according to the embodiment of the present application can be in the form of a portable compact disc read-only memory (CD-ROM) and includes program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device, or apparatus.

[0112] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0113] The computer readable storage medium can include a computer-readable medium in baseband or propagated as a carrier wave in a propagated signal, wherein the propagated signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. Computer readable storage medium can be any medium that can be read by a machine, including any medium that can store or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, and the like, or any suitable combination thereof.

[0114] Program code used by or in connection with the described embodiments can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0115] In the embodiment of the present application, the background image, the infrared structured light image, the infrared image of the same target are sequentially acquired according to the preset frame frequency value, or the infrared image, the infrared structured light image, the background image of the same target are sequentially acquired, so that the continuous acquisition of three frames of images is realized, the time interval of the acquisition between adjacent two frames of images is shortened, the difficulty of the attack on the depth camera is improved, and the safety of the depth camera is higher; in the embodiment of the present application, the target structured light image is generated by subtracting the gray values of the corresponding pixels of the background image and the infrared structured light image, then the depth image is generated by multi-frame depth reconstruction or three-dimensional reconstruction according to the target structured light image, the interference of the background light is reduced, and the depth camera can be applied to the environment with high light intensity.

[0116] The various embodiments described in this specification are intended to be illustrative only and in no way limit the scope of the application. Those skilled in the art will be able to devise many variations that, although not explicitly described herein, embody the principles of the application and are included within its spirit and scope. The description is thus to be considered as given for purposes of exemplification only and not limitation. The patentable scope of the present application is deemed to be limited only by the claims that are included below.

[0117] The foregoing detailed description of the application has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed, and obviously many modifications and variations are possible in light of the teaching above. The described embodiments were chosen in order to best illustrate the principles of the application and its practical application to thereby enable others skilled in the art to best utilize the application.

Claims

1. A multi-frame depth reconstruction method, characterized in that, Includes the following steps: Step S1: Obtain a preset frame rate value, and sequentially acquire a first depth image, a background image, and a second depth image of the same target according to the frame rate value; Step S2: Subtract the grayscale values ​​of corresponding pixels in the first depth image and the background image to generate a first valid image; subtract the grayscale values ​​of corresponding pixels in the second depth image and the background image to generate a second valid image; Step S3: Generate a depth image by performing multi-frame depth reconstruction or 3D reconstruction based on the first valid image and the second valid image; Step S1 includes the following steps: Step S101: Obtain a preset frame rate threshold and an original frame rate value, wherein the original frame rate value is the frame rate value of the depth camera used to acquire the first depth image and the second depth image; Step S102: Determine the multiple between the original frame rate value and the frame rate threshold. When the multiple is less than or equal to a preset multiple threshold, the preset frame rate value is determined to be the product of the multiple threshold and the frame rate threshold. When the multiple is greater than or equal to the preset multiple threshold, the preset frame rate value is determined to be the original frame rate value. Step S103: Based on the frame rate value, continuously and sequentially acquire the first depth image, background image, and second depth image of the same target within a frame rate threshold sampling period.

2. The multi-frame depth reconstruction method according to claim 1, characterized in that, Step S103 includes the following steps: Step S1031: Project structured light and floodlight onto the target using the light projector of the depth camera; Step S1032: When the multiplier value is less than or equal to the preset multiplier threshold, the infrared structured light image, background image, and infrared image of the same target are continuously and sequentially acquired within the frame frequency threshold sampling period according to the frame frequency value, or the infrared image, background image, and infrared structured light image of the same target are continuously and sequentially acquired. Step S1033: When the multiplier value is greater than the preset multiplier threshold, according to the frame rate value, in each of the multiple sampling periods determined according to the frame rate threshold, three consecutive frames are sequentially acquired of the same target, including infrared structured light image, background image, and infrared image, or the infrared image, background image, and infrared structured light image of the same target are sequentially acquired.

3. The multi-frame depth reconstruction method according to claim 1, characterized in that, Step S2 includes the following steps: Step S201: Determine the pixel value of each pixel in the first depth image, the second depth image, and the background image; Step S202: Align the first depth image, the second depth image, and the background image at the pixel level; Step S203: Subtract the grayscale value of the corresponding pixel in the background image from each pixel in the first depth image to generate a first valid image; subtract the grayscale value of the corresponding pixel in the background image from each pixel in the second depth image to generate a second valid image.

4. The multi-frame depth reconstruction method according to claim 2, characterized in that, Step S3 includes the following steps: Step S301: Calculate the parallax image of the infrared structured light image by comparing it with the known calibration information; Step S302: Determine the distance between the optical center of the depth camera and each disparity value in the disparity map according to the principle of triangulation, and generate depth information for each pixel; Step S303: Generate a depth image by performing multi-frame depth reconstruction or three-dimensional reconstruction based on the depth information of each pixel.

5. The multi-frame depth reconstruction method according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Combine the first valid image and the second valid image to obtain a third valid image; Step S32: Generate a depth image by performing multi-frame depth reconstruction or three-dimensional reconstruction based on the third valid image.

6. The multi-frame depth reconstruction method according to claim 1, characterized in that, The preset frame rate threshold is 15 FPS, and the preset multiplier threshold is 3.

7. A multi-frame depth reconstruction system for implementing the multi-frame depth reconstruction method according to any one of claims 1 to 6, characterized in that, include: The image acquisition module is used to obtain a preset frame rate value and to continuously and sequentially acquire a first depth image, a background image, and a second depth image of the same target according to the frame rate value. An image enhancement module is used to generate a first effective image by subtracting the gray values ​​of corresponding pixels in the first depth image and the background image; and to generate a second effective image by subtracting the gray values ​​of corresponding pixels in the second depth image and the background image. A multi-frame depth reconstruction module is used to generate a depth image by performing multi-frame depth reconstruction or three-dimensional reconstruction based on the first valid image and the second valid image. The image acquisition module includes the following steps during processing: Step S101: Obtain a preset frame rate threshold and an original frame rate value, wherein the original frame rate value is the frame rate value of the depth camera used to acquire the first depth image and the second depth image; Step S102: Determine the multiple between the original frame rate value and the frame rate threshold. When the multiple is less than or equal to a preset multiple threshold, the preset frame rate value is determined to be the product of the multiple threshold and the frame rate threshold. When the multiple is greater than or equal to the preset multiple threshold, the preset frame rate value is determined to be the original frame rate value. Step S103: Based on the frame rate value, continuously and sequentially acquire the first depth image, background image, and second depth image of the same target within a frame rate threshold sampling period.

8. A multi-frame depth reconstruction device, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the multi-frame depth reconstruction method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the steps of the multi-frame depth reconstruction method according to any one of claims 1 to 6.

Images