Modulated light field based depth imaging systems, methods, devices, and media

Abstract

A depth imaging system based on modulated light field, characterized in that it comprises the following modules: a light projection module for projecting discrete light beams to a target object and switching between a modulated light field mode and a uniform light field mode; a light source control module for light emission and control of the switching of the light field mode; a TOF sensing module for receiving the measurement light beams reflected by the target object; a multipath interference elimination module for obtaining image data under the modulated light field mode and the uniform light field mode, estimating the multipath interference component therefrom, removing the measurement error caused by the multipath interference, and outputting a high-precision depth image. The present application can determine the multipath interference component by the background area data in the first image data collected when the discrete light beams are projected to the target object, process the light spot area data to remove the multipath interference component, thereby eliminating the measurement error caused by the multipath interference, and realizing the output of a high-precision depth image.

Description

Technical Field

[0001] This invention relates to TOF depth cameras, and more specifically, to a depth imaging system, method, device, and medium based on modulated light fields. Background Technology

[0002] Time-of-flight (TOF) depth cameras acquire depth images of the measured space by emitting a floodlight beam of a specific wavelength, receiving the reflected beam from objects in the measured space using sensors, and measuring the time it takes for the beam to travel through space. TOF depth cameras can simultaneously acquire grayscale and depth images and are widely used in 3D depth vision-related technologies such as gesture recognition, face recognition, 3D modeling, motion-sensing games, machine vision, autofocus assistance, security, and autonomous driving.

[0003] Traditional Time-of-Flight (TOF) depth cameras assume that the received light beam is reflected only once in the target scene. However, in real-world scenes, there are always surfaces with specular or diffuse reflection that reflect the incident light in various directions. As a result, the TOF sensor may receive a superposition of single and multiple reflected beams, which interferes with the accuracy of the TOF depth camera in measuring distance. This effect is called multipath interference.

[0004] Existing technologies primarily utilize multi-frequency, multi-frame fusion to estimate multipath interference and reconstruct the original depth. However, limited by the frame rate and the number of frequencies, these methods suffer from high computational complexity, poor robustness, and unsatisfactory reconstruction accuracy, making practical applications challenging. Therefore, how to suppress multipath interference and improve depth measurement accuracy is a pressing issue that needs to be addressed for the practical application of TOF depth cameras. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this invention is to provide a depth imaging system, method, device and medium based on modulated light field.

[0006] The depth imaging system based on modulated light field provided by the present invention includes the following modules:

[0007] The light projection module is used to project floodlight or multiple discrete beams onto a target object;

[0008] The light source control module is used to control the floodlight and the multiple discrete beams to switch and project according to a preset time period;

[0009] The TOF sensing module is used to receive multiple discrete light beams reflected by the target object and the floodlight, respectively, and then generate first image data and second image data.

[0010] a multipath interference elimination module, configured to determine a multipath interference component according to the first image data, and to generate a target depth image by fusing the first image data after removing the multipath interference component and the second image data.

[0011] Preferably, the multipath interference elimination module comprises the following modules:

[0012] a data acquisition module, configured to acquire first image data, the first image data comprising a plurality of spot region data and background region data;

[0013] a multipath interference determination module, configured to determine a multipath interference basic quantity corresponding to each spot region data according to the background region data;

[0014] a data generation module, configured to process each spot region data based on the multipath interference basic quantity to remove a multipath interference component corresponding to each spot region data, and to generate target spot region data;

[0015] a depth image generation module, configured to generate a target depth image by fusing a first depth image generated according to the target spot region data and a second depth image generated according to the second image data.

[0016] Preferably, the first image data comprises a plurality of infrared images acquired by a TOF sensor;

[0017] each infrared image comprises a plurality of spot regions and a background region;

[0018] the background region is a multipath interference region adjacent to the spot regions;

[0019] the multipath interference basic quantity and the multipath interference component are represented by any one of the following physical quantities: amplitude, gray value, pixel value, illumination, luminous flux and radiant power.

[0020] Preferably, the diameter of each spot region is two pixels;

[0021] the distance between any two adjacent spot regions is four pixels.

[0022] Preferably, the data generation module comprises the following modules:

[0023] an interference region determination module, configured to divide the plurality of spot regions into a plurality of groups of spot regions, each group of spot regions corresponding to or being adjacent to at least one background region;

[0024] an interference component generation module, configured to acquire a multipath interference basic quantity of each background region;

[0025] The interference component removing module is configured to determine the multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to each group of light spot areas, and further remove the multipath interference component corresponding to each light spot area data to generate target light spot area data.

[0026] Preferably, the data generating module comprises the following modules:

[0027] The area association module is configured to determine at least one background area corresponding to or adjacent to each light spot area.

[0028] The interference obtaining module is configured to obtain the multipath interference basic quantity of each background area.

[0029] The interference component removing module is configured to determine the multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to each group of light spot areas, and further remove the multipath interference component corresponding to each light spot area data to generate target light spot area data.

[0030] Preferably, the depth image generating module comprises the following modules:

[0031] The image obtaining module is configured to obtain a first infrared image corresponding to the first depth image and a second infrared image corresponding to the second depth image.

[0032] The confidence generating module is configured to determine a first confidence based on the depth information of each pixel point in the first depth image according to the first infrared image, and determine a second confidence based on the depth information of each pixel point in the second depth image according to the second infrared image.

[0033] The fusion coefficient generating module is configured to extract the edge contour region depth information of the target object in the second depth image, determine a first fusion coefficient based on the edge contour region depth information, and determine a second fusion coefficient based on the depth information of the region where the target object is located.

[0034] The image fusion module is configured to fuse the pixel points screened out from the first depth image based on the first confidence, the first fusion coefficient and the second fusion coefficient, and the pixel points screened out from the second depth image based on the second confidence to generate a target depth image.

[0035] According to the depth imaging method based on the modulated light field provided by the application, the following steps are included:

[0036] Step S1: projecting a floodlight or a plurality of discrete light beams to a target object;

[0037] Step S2: controlling the floodlight and the plurality of discrete light beams to be switched and projected according to a preset time period;

[0038] Step S3: receiving the multiple discrete light beams reflected by the target object and the floodlight, and further generating first image data and second image data;

[0039] Step S4: determining a multipath interference component according to the first image data, and further generating a target depth image according to the first image data after removing the multipath interference component and the second image data.

[0040] According to the present application, a depth imaging device based on a modulated light field is provided, comprising:

[0041] a processor;

[0042] a memory, wherein executable instructions of the processor are stored;

[0043] wherein the processor is configured to execute the steps of the depth imaging method based on a modulated light field by executing the executable instructions.

[0044] According to the present application, a computer readable storage medium is provided for storing a program, which is executed to implement the steps of the depth imaging method based on a modulated light field.

[0045] Compared with the prior art, the present application has the following beneficial effects:

[0046] The present application can determine the multipath interference component by the background area data in the first image data collected when the discrete light beams are projected to the target object, and further process the light spot area data to remove the multipath interference component, so as to eliminate the measurement error caused by the multipath interference and realize the output of high-precision depth image. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings. Other features, objects and advantages of the present application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the following drawings:

[0048] Figure 1 a schematic diagram of the depth imaging system based on a modulated light field in the embodiment of the present application;

[0049] Figure 2 a schematic diagram of the depth imaging system based on a modulated light field in the embodiment of the present application;

[0050] Figure 3A module schematic diagram of the multipath interference elimination module in the embodiment of the present application;

[0051] Figure 4 A module schematic diagram of the data generation module in the embodiment of the present application;

[0052] Figure 5 A module schematic diagram of the data generation module in the variant of the present application;

[0053] Figure 6 A module schematic diagram of the depth image generation module in the embodiment of the present application;

[0054] Figure 7 (a) is a schematic diagram of an infrared image in the embodiment of the present application;

[0055] Figure 7 (b) is a schematic diagram of another infrared image in the embodiment of the present application;

[0056] Figure 8 A step flow chart of the depth imaging method based on modulated light field in the embodiment of the present application;

[0057] Figure 9 A structural schematic diagram of the depth imaging device based on modulated light field in the embodiment of the present application; and

[0058] Figure 10 A structural schematic diagram of the computer readable storage medium in the embodiment of the present application.

[0059] In the figure:

[0060] 1 is a light spot area;

[0061] 2 is a background area. DETAILED DESCRIPTION

[0062] The present application will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of variations and improvements can be made. These all belong to the protection scope of the present application.

[0063] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, and above-described drawings, if any, are used to distinguish between similar objects and are not necessarily used to describe a particular sequential or chronological order. It is to be understood that the use of the terms so-termed, if any, can be interchanged, where appropriate, to permit herein description of embodiments of the application, for example, to be carried out in an order other than that illustrated or described herein. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or apparatus that includes a list of steps or units not necessarily limited to those explicitly listed, but can include other steps or units not expressly listed or inherent to such processes, methods, products or apparatus.

[0064] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples.

[0065] The present application provides a depth imaging method based on modulated light field, aiming at solving the problems in the prior art.

[0066] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples. The embodiments of the present application will be described below with reference to the drawings.

[0067] Figure 1 For the module schematic diagram of the depth imaging system based on modulated light field in the embodiments of the present application, as shown in Figure 1 The present application provides a depth imaging system based on modulated light field 100, which comprises the following modules:

[0068] The light projection module 101 is used for projecting floodlight or multiple discrete light beams to a target object;

[0069] The light source control module 102 is used for controlling the floodlight and the multiple discrete light beams to be switched and projected according to a preset time period;

[0070] The TOF sensing module 103 is used for receiving the multiple discrete light beams and the floodlight reflected by the target object respectively, and further generating first image data and second image data;

[0071] The multipath interference elimination module 104 is used for determining a multipath interference component according to the first image data, and further generating a target depth image by fusing the first image data after removing the multipath interference component and the second image data.

[0072] Figure 2 This is a schematic diagram illustrating the usage state of the depth imaging system based on modulated light field in an embodiment of the present invention. When using the depth imaging system based on modulated light field provided by the present invention, a discrete light beam is projected onto the target object through the modulated light projection module, and the spatial distribution of the light field intensity and phase of the projected discrete light beam is modulated. The modulated light field mode and the uniform light field mode can be switched as needed. The light source control module drives the light source to emit light and controls the switching of the light field mode. The TOF sensing module receives the measurement beam reflected back from the target object and outputs the first image data to the multipath interference cancellation module. The multipath interference cancellation module is connected to the light source control module and the TOF sensing module. Based on the image data under the modulated light field mode and the uniform light field, it estimates the multipath interference components, removes the measurement error caused by multipath interference, and outputs a high-precision depth image.

[0073] In this embodiment of the invention, the modulated light field projection module can switch between speckle light and uniform light projection by using a diffuser made of laser and nanophotonic chip, or it can project speckle light and uniform light respectively by using a speckle projector and a uniform light projector.

[0074] Figure 3 This is a schematic diagram of the multipath interference cancellation module in an embodiment of the present invention, as shown below. Figure 3 As shown, the multipath interference cancellation module 104 includes the following modules:

[0075] Data acquisition module 1041 is used to acquire first image data, the first image data including multiple spot area data and background area data;

[0076] In this embodiment of the invention, the first image data includes multiple infrared images acquired by a TOF sensor;

[0077] Each of the infrared images includes multiple spot areas 1 and background areas 2, such as Figure 7 (a) Figure 7 As shown in (b);

[0078] The background area is a multipath interference region adjacent to the spot area. The infrared image also includes areas other than the spot area and the background area, which are signal-free areas.

[0079] The diameter of each light spot region is two pixels; the distance between any two adjacent light spot regions is four pixels.

[0080] More specifically, the number of infrared images is four infrared images acquired in four image acquisition cycles, so as to calculate the time difference between the light signals based on the amplitude changes between the four infrared images and generate a first depth image.

[0081] The multi-path interference determination module 1042 is configured to determine a multi-path interference basic quantity corresponding to each spot region data according to the background region data.

[0082] In the embodiment of the present application, the influence of each local region of the infrared image is the same or similar, and thus the multi-path interference basic quantity can be determined by the background region. The multi-path interference basic quantity is the multi-path interference quantity of a single pixel point of the background region, which can be represented by amplitude.

[0083] In the embodiment of the present application, the multi-path interference basic quantity is represented by amplitude, and is first photoelectrically converted for processing, and the unit is changed into voltage. The voltage is quantified by ADC to generate corresponding digital quantity, and the unit is LSB.

[0084] In the embodiment of the present application, the multi-path interference basic quantity can also be represented by any one of a gray value, a pixel value, an illumination, a luminous flux and a radiant power.

[0085] The data generation module 1043 is configured to process each spot region data based on the multi-path interference basic quantity to remove the multi-path interference component corresponding to each spot region data, and further generate target spot region data.

[0086] The depth image generation module 1044 is configured to fuse a first depth image generated according to the target spot region data and a second depth image generated according to the second image data to generate a target depth image.

[0087] Figure 4 The module schematic diagram of the data generation module in the embodiment of the present application is shown in FIG. 4. Figure 4 As shown in FIG. 4, the data generation module 1043 includes the following modules.

[0088] The interference region determination module 10431 is configured to divide a plurality of the spot regions into a plurality of groups of spot regions, and each group of the spot regions corresponds to or is adjacent to at least one background region.

[0089] The interference component generation module 10432 is configured to obtain the multi-path interference basic quantity of each background region.

[0090] The interference component removal module 10433 is configured to determine the multi-path interference component corresponding to each spot region according to the multi-path interference basic quantity of the background region corresponding to each group of the spot regions, and further remove the multi-path interference component corresponding to each spot region data to generate target spot region data.

[0091] In the embodiment of the present application, the multi-path interference component of each spot region can be calculated according to the number of pixel points of the spot region and the multi-path interference basic quantity. For example, when the number of pixel points of the spot region is 2, the multi-path interference component of the spot region is twice the multi-path interference basic quantity.

[0092] In the embodiment of the present application, the target spot region data can be generated according to the difference between the amplitude of the spot region and the multipath interference component, or the difference between the illumination of the spot region and the illumination of the multipath interference region, or the difference between the gray value of the spot region and the illumination of the multipath interference region.

[0093] Figure 5 The module schematic diagram of the data generation module in the variant of the present application is shown in Figure 5 The data generation module 1043 includes the following modules as shown in the figure.

[0094] The region association module 10434 is configured to determine at least one background region corresponding to or adjacent to each of the spot regions.

[0095] The interference acquisition module 10435 is configured to acquire the multipath interference basic quantity of each of the background regions.

[0096] The interference component removal module 10436 is configured to determine the multipath interference component corresponding to each of the spot regions according to the multipath interference basic quantity of the background region corresponding to each of the spot regions, and further remove the multipath interference component corresponding to each of the spot region data to generate the target spot region data.

[0097] In the variant of the present application, the working process of the region association module 10434 is that, first, each of the spot regions is determined, then each of the background regions is determined, and finally the spot regions and the background regions are associated.

[0098] In the variant of the present application, the multipath interference basic quantity is determined according to the background region close to each of the spot regions, so that the multipath interference component corresponding to each of the spot regions can be determined more accurately.

[0099] Figure 6 The module schematic diagram of the depth image generation module in the embodiment of the present application is shown in Figure 6 The depth image generation module 1044 includes the following modules as shown in the figure.

[0100] The image acquisition module 10441 is configured to acquire the first infrared image corresponding to the first depth image and the second infrared image corresponding to the second depth image.

[0101] The confidence generation module 10442 is configured to determine a first confidence based on the depth information of each pixel point in the first depth image for the first infrared image, and determine a second confidence based on the depth information of each pixel point in the second depth image for the second infrared image.

[0102] In the embodiment of the present application, a first confidence is determined for each pixel point depth information in the first depth image based on the amplitude of each region in the first infrared image. When the amplitude of a region in the first infrared image is high, a higher confidence is given to the pixel points in the same region in the first depth image. When the amplitude of a region in the first infrared image is low, a lower confidence is given to the pixel points in the same region in the first depth image. Similarly, for the second depth image, when the amplitude of a region in the second infrared image is high, a higher confidence is given to the pixel points in the same region in the second depth image. When the amplitude of a region in the second infrared image is low, a lower confidence is given to the pixel points in the same region in the second depth image.

[0103] More specifically, when the amplitude of a region in the first infrared image is 300, a confidence of 3 is given to the pixel points in the same region in the first depth image. When the amplitude of a region in the first infrared image is 100, a confidence of 1 is given to the pixel points in the same region in the first depth image.

[0104] The fusion coefficient generation module 10443 is configured to extract edge contour region depth information of the target object from the second depth image, determine a first fusion coefficient for the edge contour region depth information, and determine a second fusion coefficient for the depth information of the region where the target object is located.

[0105] In the embodiment of the present application, the first fusion coefficient is greater than the second fusion system, for example, the first fusion coefficient can be set to 0.8, so that more pixel points are selected for the edge contour region, and the second fusion coefficient is 0.2, so that smaller pixel points are selected for the region where the target object is located, so that the edge of the generated target depth image can be more fine.

[0106] The image fusion module 10444 is configured to fuse the pixel points selected from the second depth image based on the second confidence and the pixel points selected from the first depth image based on the first confidence, the first fusion coefficient and the second fusion coefficient to generate a target depth image.

[0107] More specifically, the first screening coefficient is generated according to the second confidence and the first fusion coefficient, and the second screening coefficient is generated according to the second confidence and the second fusion coefficient; the first screening coefficient is used to select a plurality of pixel points of the target object and the scene edge profile region in the second depth image, and the second screening coefficient is used to select a plurality of pixel points of the target object and the scene region in the second depth image; the third screening coefficient is generated according to the first confidence, and a plurality of pixel points are selected in the first depth image according to the third screening coefficient; and the target depth image is generated by fusing the plurality of pixel points selected in the first depth image and the plurality of pixel points selected in the second depth image.

[0108] In the embodiment of the present application, the first screening coefficient, the second screening coefficient and the third screening coefficient are all greater than 0 and less than 1; when the second confidence of a region of the second depth image is 3, the first fusion coefficient is 0.7, when the second confidence of a region of the second depth image is 1, the first fusion coefficient is 0.2, and the value range of the confidence is [0, 5], the first screening coefficient can take a value of 3 / 5*0.7=0.42; the second screening coefficient can take a value of 1 / 5*0.2=0.04. When the first confidence is 4, the third screening coefficient is 4 / 5=0.8.

[0109] Figure 8 For the step flow chart of the depth imaging method based on the modulated light field in the embodiment of the present application, as shown in FIG. 1, the present application provides a depth imaging method based on the modulated light field, which comprises the following steps: Figure 8

[0110] Step S1: projecting a floodlight or a plurality of discrete light beams to a target object;

[0111] Step S2: controlling the floodlight and the plurality of discrete light beams to be projected in a switching manner according to a pre-set time period;

[0112] Step S3: receiving the plurality of discrete light beams and the floodlight reflected by the target object, and further generating first image data and second image data;

[0113] Step S4: determining a multipath interference component according to the first image data, and further generating a target depth image by fusing the first image data after removing the multipath interference component and the second image data.

[0114] The embodiment of the present application further provides a depth imaging device based on the modulated light field, which comprises a processor and a memory having executable instructions of the processor stored therein. The processor is configured to execute the steps of the depth imaging method based on the modulated light field by executing the executable instructions.

[0115] ​As described above, the embodiment can determine the multipath interference basic quantity by the background region data in the first image data collected when the discrete light beams are projected to the target object, and further can process the spot region data to remove the multipath interference component, so as to eliminate the measurement error caused by the multipath interference, and realize the output of the high-precision depth image.

[0116] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be embodied in the form of entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining software and hardware aspects, which can be collectively referred to as "circuitry", "module" or "platform" herein.

[0117] Figure 9 is a structural schematic diagram of the depth imaging device based on the modulated light field in the embodiment of the present application. The electronic device 600 according to this embodiment of the present application will be described below with reference to Figure 9 Figure 9 The displayed electronic device 600 is only an example and should not bring any limitation to the function and use range of the embodiment of the present application.

[0118] As shown in Figure 9 , the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0119] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the above depth imaging method based on the modulated light field part of the present specification. For example, the processing unit 610 can perform the steps as shown in Figure 1

[0120] The storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 6201 and / or a cache memory unit 6202, and can further include a read-only memory (ROM) 6203.

[0121] The storage unit 620 can further include a program / utility 6204 having a set of program modules 6205, which include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof can include implementation of a network environment. ​​

[0122] Bus 630 can be one of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, a graphics bus, or a local bus using any of a variety of bus architectures.

[0123] Electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or a pointing device, for example, by I / O interface 650. Additionally, electronic device 600 can communicate with one or more devices that enable a user to interact with electronic device 600, for example, by I / O interface 650. Further, electronic device 600 can communicate with one or more devices that enable Figure 9 Other hardware and / or software modules that can be used in conjunction with electronic device 600 can also be employed in electronic device 600, as indicated by the dashed lined box 670. Such modules can include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0124] The embodiment of the present application also provides a computer readable storage medium for storing a program, the program being executed to implement the steps of the depth imaging method based on modulated light field. In some possible implementation manners, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing terminal equipment to execute the steps described in the above depth imaging method based on modulated light field part of the specification according to various exemplary embodiments of the present application when the program product is run on the terminal equipment.

[0125] As shown above, when the program of the computer readable storage medium of the embodiment is executed, the present application can determine the multi-path interference basic quantity through the background region data in the first image data collected when the discrete light beams are projected to the target object, and further can process the spot region data to remove the multi-path interference component, so that the measurement error caused by the multi-path interference can be eliminated, and the output of the depth image with high precision can be realized.

[0126] Figure 10 is a structural schematic diagram of the computer readable storage medium in the embodiment of the present application. Referring to Figure 10As shown, a program product 800 for implementing the above-described method according to an embodiment of the present application is described, which can take the form of a portable compact disc read-only memory (CD-ROM) and includes a program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in the present document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0127] The program product can take any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] The computer readable storage medium can include a data signal transported, propagated or transmitted, in baseband or as part of a carrier wave, in which readable program code is embodied. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. The readable storage medium can also be any readable medium that is capable of storing, transmitting or transferring a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including, but not limited to, wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0129] The program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0130] In the embodiments of the present application, the present application can determine the multipath interference basic quantity by the background area data in the first image data collected when the discrete light beams are projected to the target object, and then can process the spot area data to remove the multipath interference component, so as to eliminate the measurement error caused by the multipath interference, and realize the output of the high-precision depth image.

[0131] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the various embodiments can be referred to each other. The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0132] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application.

Claims

1. A depth imaging system based on modulated light fields, characterized in that, The method comprises the following modules: A light projection module is configured to project discrete light beams to a target object and switch between a modulated light field mode and a uniform light field mode by modulation; A light source control module is configured to emit light and control the switching of the light field mode; A TOF sensing module is configured to receive a measurement light beam reflected by the target object; A multipath interference elimination module is configured to obtain image data under the modulated light field mode and the uniform light field mode, wherein the image data under the modulated light field mode comprises a plurality of spot region data and background region data, determine a multipath interference basic quantity corresponding to each spot region data according to the background region data, remove measurement errors caused by multipath interference, and then generate a target depth image by fusing the data under the modulated light field mode after removing the multipath interference component and the image data under the uniform light field.

2. The modulated light field based depth imaging system of claim 1, wherein, The multipath interference elimination module comprises the following modules: A data acquisition module is configured to obtain first image data, wherein the first image data comprises a plurality of spot region data and background region data, and the first image data is generated by a reflection signal under the modulated light field mode; A multipath interference determination module is configured to determine a multipath interference basic quantity corresponding to each spot region data according to the background region data; A data generation module is configured to process each spot region data based on the multipath interference basic quantity to remove a multipath interference component corresponding to each spot region data, and then generate target spot region data; A depth image generation module is configured to generate a target depth image by fusing a first depth image generated according to the target spot region data and a second depth image generated according to second image data, wherein the second image data is generated by a reflection signal under the uniform light field mode.

3. The modulated light field based depth imaging system of claim 2, wherein, The first image data comprises a plurality of infrared images collected by a TOF sensor; Each infrared image comprises a plurality of spot regions and a background region; The background region is a multipath interference region adjacent to the spot region; The multipath interference basic quantity and the multipath interference component are represented by any one of the following physical quantities: amplitude, gray value, pixel value, illumination, luminous flux, and radiant power.

4. The modulated light field based depth imaging system of claim 3, wherein, The diameter of each spot region is two pixels; The distance between any two adjacent spot regions is four pixels.

5. The modulated light field based depth imaging system of claim 3, wherein, The data generation module comprises the following modules: An interference region determination module is configured to divide a plurality of spot regions into a plurality of groups of spot regions, and each group of spot regions corresponds to or is adjacent to at least one background region; An interference component generation module is configured to obtain a multipath interference basic quantity of each background region; An interference component removal module is configured to determine a multipath interference component corresponding to each spot region according to the multipath interference basic quantity of the corresponding background region of each group of spot regions, and then remove the multipath interference component corresponding to each spot region data to generate target spot region data.

6. The modulated light field based depth imaging system of claim 3, wherein, The data generation module comprises the following modules: A region association module is configured to determine at least one background region corresponding to or adjacent to each spot region; An interference acquisition module is configured to obtain a multipath interference basic quantity of each background region; The interference component removing module is configured to determine a multipath interference component corresponding to each light spot area according to a multipath interference basic value of a background area corresponding to the light spot area, and to remove the multipath interference component corresponding to each light spot area data to generate target light spot area data.

7. The modulated light field based depth imaging system of claim 2, wherein, The depth image generating module comprises the following modules: The image acquisition module is configured to acquire a first infrared image corresponding to the first depth image and a second infrared image corresponding to the second depth image. The confidence generating module is configured to determine a first confidence based on the first infrared image and the depth information of each pixel point in the first depth image, and to determine a second confidence based on the second infrared image and the depth information of each pixel point in the second depth image. The fusion coefficient generating module is configured to extract edge contour region depth information of the target object in the second depth image, to determine a first fusion coefficient based on the edge contour region depth information, and to determine a second fusion coefficient based on the depth information of the region where the target object is located. The image fusion module is configured to fuse the pixel points selected from the first depth image based on the first confidence, the first fusion coefficient and the second fusion coefficient, and the pixel points selected from the second depth image based on the second confidence to generate a target depth image.

8. A method of depth imaging based on modulated light field, characterized in that, The method comprises the following steps: Step S1: projecting a discrete light beam to a target object, and switching a modulated light field mode and a uniform light field mode by modulation; Step S2: controlling the switching of the light field mode; Step S3: receiving a measurement light beam reflected by the target object; Step S4: acquiring image data under the modulated light field mode and the uniform light field mode, the image data under the modulated light field mode comprising a plurality of light spot area data and background area data, determining a multipath interference basic value corresponding to each light spot area data according to the background area data, removing measurement errors caused by multipath interference, and then fusing the data under the modulated light field mode after removing the multipath interference components and the image data under the uniform light field to generate a target depth image.

9. A depth imaging device based on modulated light field, characterized in that, The method comprises: a processor; a memory having executable instructions of the processor stored therein; wherein the processor is configured to execute the steps of the method of claim 8 by executing the executable instructions.

10. A computer readable storage medium for storing a program, characterized in that, The program is executed to implement the steps of the method of claim 8. The program is executed to implement the steps of the method of claim 8.

Images