Filtering method for depth camera module

Abstract

A filtering method for a depth camera module is disclosed, comprising: obtaining a depth image of a current frame and a grayscale image corresponding to the depth image; obtaining a reference grayscale value, the reference grayscale value being the maximum grayscale value of the grayscale image of a reference frame separated by a preset number of frames from the current frame; normalizing the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value; and, based on a preset mapping relationship between the depth value and the normalized grayscale value, filtering out pixels in the depth image of the current frame whose depth values ​​are determined to be invalid to obtain a filtered depth image. This filtering method can effectively address the problem of stray light interference during depth measurement, thereby improving the precision and accuracy of depth measurement.

Description

Technical Field

[0001] The present application relates to the field of depth information camera modules, and more specifically to a filtering method for a depth camera module, a filtering device for a depth camera module, and an electronic device. Background Art

[0002] In recent years, 3D vision technology has been widely used in many fields, including biometrics, smart transportation, healthcare, and industrial production. 3D vision technology refers to the use of image acquisition devices to obtain three-dimensional images with depth information to achieve target recognition, tracking, positioning, and other functions.

[0003] According to the imaging principle, the depth information camera modules currently used in 3D vision technology mainly include passive binocular camera modules, speckle structured light camera modules and TOF camera modules. Taking the TOF camera module as an example, its three-dimensional imaging principle is: Figure 1 As shown, the TOF camera module emits a modulated signal (for example, modulated infrared light) to the target object to be detected, and receives an echo signal reflected by the target object. The distance (i.e., depth value) between the target object to be detected and the TOF camera module is calculated by the time difference between the transmitted modulated signal and the received echo signal (or by the phase difference between the modulated signal and the echo signal) to obtain a three-dimensional image of the target object to be detected.

[0004] As application scenarios become increasingly complex, users' technical requirements for depth measurement and corresponding depth information camera modules are becoming increasingly demanding. However, improving the depth measurement performance of depth information camera modules encounters many problems, one of which is the impact of stray light on depth measurement.

[0005] For example, in actual scenarios, stray light in the environment will interfere with the optical signal received by the depth camera module, causing the depth camera module to receive not only the signal after the modulated signal transmitted to the target object is reflected, but also the stray light signal in the environment, such as Figure 2 As shown, this will cause a deviation between the calculated depth value and the actual depth value, making it difficult to obtain accurate depth information and causing problems such as image fading and decreased image contrast.

[0006] Therefore, a solution for processing stray light interference for depth information camera modules is desired. Summary of the Invention

[0007] One advantage of the present application is that it provides a filtering method for a depth camera module, a filtering device and an electronic device for a depth camera module, which can effectively solve the problem of stray light interference during depth measurement to improve the depth measurement accuracy and precision.

[0008] Another advantage of the present application is that it provides a filtering method for a depth camera module, a filtering device and an electronic device for a depth camera module, wherein the filtering method has relatively low latency and relatively high filtering efficiency.

[0009] Another advantage of the present application is that it provides a filtering method for a depth camera module, a filtering device and an electronic device for a depth camera module, wherein the filtering method performs filtering processing based on the irregularity of the depth value and grayscale value of the stray light signal to filter out the pixel points corresponding to the stray light signal in the depth image.

[0010] In order to achieve at least one of the above advantages or other advantages and purposes, according to one aspect of the present application, a filtering method for a depth camera module is provided, which includes:

[0011] Obtaining a depth image of a current frame and a grayscale image corresponding to the depth image;

[0012] Acquire a reference grayscale value, where the reference grayscale value is a maximum grayscale value of a grayscale image of a reference frame that is a preset number of frames apart from the current frame;

[0013] Normalizing the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value; and

[0014] Based on a preset mapping relationship between depth values ​​and the normalized grayscale values, pixels whose depth values ​​are determined to be invalid in the depth image of the current frame are filtered out to obtain a filtered depth image.

[0015] According to the filtering method for the depth camera module of the present application, the reference frame and the current frame are adjacent frames.

[0016] According to the filtering method for a depth camera module of the present application, the grayscale value of each pixel in the grayscale image of the current frame is normalized based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value, including: normalizing the grayscale value of each pixel in the grayscale image of the current frame according to the following formula, wherein the formula is Yi=Xi / Grey_max, wherein Yi represents the normalized grayscale value, Xi represents the grayscale value of each pixel in the grayscale image of the current frame, and Grey_max represents the reference grayscale value.

[0017] According to the filtering method for a depth camera module of the present application, the preset mapping relationship between the depth value and the normalized grayscale value is a linear mapping relationship, and the linear mapping relationship includes: the depth value and the normalized grayscale value are in a negatively correlated linear relationship, when the depth value is in the range of 0 to a first preset value, the normalized grayscale value is in the range of a second preset value to 1, and when the depth value is the first preset value, the normalized grayscale value is the second preset value; wherein, based on the preset mapping relationship between the depth value and the normalized grayscale value, filtering out pixels whose depth values ​​in the depth image of the current frame are determined to be invalid to obtain a filtered depth image, including: in response to the depth value of a pixel point in the depth image being in the range of 0 to the first preset value and the normalized grayscale value corresponding to the pixel point being less than the second preset value, determining that the depth value of the pixel point is invalid; and, in response to the depth value of a pixel point in the depth image being greater than the first preset value and the normalized grayscale value corresponding to the pixel point being in the range of the second preset value to 1, determining that the depth value of the pixel point is invalid.

[0018] According to the filtering method for the depth camera module of the present application, the linear mapping relationship is a multi-level linear mapping relationship.

[0019] According to the filtering method for a depth camera module of the present application, the acquiring of the depth image of the current frame and the grayscale image corresponding to the depth image and the acquiring of the reference grayscale value are performed in parallel.

[0020] According to the filtering method for the depth camera module of the present application, obtaining a reference grayscale value includes: obtaining the reference grayscale value from an external storage unit to a parallel chip, wherein the parallel chip is capable of processing data in parallel.

[0021] According to another aspect of the present application, a filtering device for a depth camera module is provided, comprising:

[0022] A current frame data acquisition unit, configured to acquire a depth image of a current frame and a grayscale image corresponding to the depth image;

[0023] A reference grayscale value acquiring unit, configured to acquire a reference grayscale value, wherein the reference grayscale value is a maximum grayscale value of a grayscale image of a reference frame that is a preset number of frames apart from the current frame;

[0024] a normalization processing unit, configured to normalize the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized image has a normalized grayscale value; and

[0025] The filtering unit is configured to filter out pixels whose depth values ​​are determined to be invalid in the depth image of the current frame based on a preset mapping relationship between the depth values ​​and the normalized grayscale values ​​to obtain a filtered depth image.

[0026] According to the filtering device for the depth camera module of the present application, the reference frame and the current frame are adjacent frames.

[0027] According to the filtering device for the depth camera module of the present application, the normalization processing unit is further used to: normalize the grayscale value of each pixel in the grayscale image of the current frame according to the following formula, wherein the formula is Yi=Xi / Grey_max, wherein Yi represents the normalized grayscale value, Xi represents the grayscale value of each pixel in the grayscale image of the current frame, and Grey_max represents the reference grayscale value.

[0028] According to the filtering device for a depth camera module of the present application, the preset mapping relationship between the depth value and the normalized grayscale value is a linear mapping relationship, and the linear mapping relationship includes: the depth value and the normalized grayscale value are in a negatively correlated linear relationship, when the depth value is in the range of 0 to a first preset value, the normalized grayscale value is in the range of a second preset value to 1, and when the depth value is the first preset value, the normalized grayscale value is the second preset value; wherein the filtering unit is further used to: in response to the depth value of a pixel point in the depth image being in the range of 0 to the first preset value and the normalized grayscale value corresponding to the pixel point being less than the second preset value, determine that the depth value of the pixel point is invalid; and, in response to the depth value of a pixel point in the depth image being greater than the first preset value and the normalized grayscale value corresponding to the pixel point being in the range of the second preset value to 1, determine that the depth value of the pixel point is invalid.

[0029] According to another aspect of the present application, an electronic device is provided, comprising: a memory; and a processor, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the filtering method for the depth camera module as described above.

[0030] In the electronic device according to the present application, the processor is a parallel chip, wherein the parallel chip can process data in parallel.

[0031] Further objectives and advantages of the present application will be fully reflected through understanding of the following description and drawings.

[0032] These and other objects, features and advantages of the present application are fully reflected in the following detailed description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] These and / or other aspects and advantages of the present application will become more apparent and easier to understand from the following detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, in which:

[0034] Figure 1 The figure shows a schematic diagram of the principle of depth measurement of the TOF camera module according to an embodiment of the present application.

[0035] Figure 2 The figure shows a schematic diagram of a depth information camera module being interfered with by stray light during depth measurement according to an embodiment of the present application.

[0036] Figure 3 The figure shows a flow chart of an existing filtering method for a depth camera module.

[0037] Figure 4 The figure illustrates a flow chart of a filtering method for a depth camera module according to an embodiment of the present application.

[0038] Figure 5 The figure shows a schematic diagram of the transmission of the maximum grayscale value in the filtering method according to an embodiment of the present application.

[0039] Figure 6 A schematic diagram of a function image between a depth value and a normalized grayscale value in a filtering method for a depth camera module according to an embodiment of the present application is shown.

[0040] Figure 7 The figure illustrates a flowchart of a filtering method for a depth camera module according to an embodiment of the present application, in which, based on a preset mapping relationship between the depth value and the normalized grayscale value, pixels whose depth values ​​in the depth image of the current frame are judged to be invalid are filtered out to obtain a filtered depth image.

[0041] Figure 8 The figure shows a schematic diagram of a depth point cloud before filtering in the filtering method according to an embodiment of the present application.

[0042] Figure 9 The figure shows a schematic diagram of a depth point cloud after filtering in the filtering method according to an embodiment of the present application.

[0043] Figure 10 The figure illustrates a block diagram of the filtering device for the depth camera module according to an embodiment of the present application.

[0044] Figure 11 The figure shows a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0046] Application Overview

[0047] As mentioned above, as application scenarios become increasingly complex, users' technical requirements for depth measurement and corresponding depth information camera modules are becoming increasingly demanding. However, improving the depth measurement performance of depth information camera modules encounters many problems, one of which is the impact of stray light on depth measurement.

[0048] For example, in actual scenarios, stray light in the environment will interfere with the optical signal received by the depth camera module, causing the depth camera module to not only receive the signal after the modulated signal it transmits to the target object is reflected, but also receive the stray light signal in the environment. This will cause a deviation between the calculated depth value and the actual depth value, making it difficult to obtain accurate depth information, and causing problems such as image fading and decreased image contrast.

[0049] Therefore, a solution for processing stray light interference for depth information camera modules is desired.

[0050] Specifically, in order to reduce the influence of stray light on depth measurement, some existing solutions filter out the pixels in the depth image that are affected by stray light by filtering the depth image. Figure 3 As shown, it first stores the data of the current frame (for example, a depth image, a grayscale image) in a memory; then, after traversing the grayscale values ​​of each pixel in the grayscale image of the current frame, the maximum grayscale value of each pixel in the grayscale image of the current frame is obtained; then, the maximum grayscale value is read out from the memory, and the grayscale value of each pixel in the grayscale image of the current frame is normalized using the maximum grayscale value. Finally, the grayscale value of the processed current frame is used to filter out the pixels in the image affected by stray light. The principle that this filtering scheme can improve the precision and accuracy of the depth measurement of the depth camera module is that the depth value and grayscale value of the pixel interfered by stray light do not satisfy the general rule, that is, the greater the depth value, the smaller the grayscale value. Therefore, each pixel in the grayscale image can be grayscale normalized, and stray light interference judgment can be performed based on the correspondence between the normalized grayscale value and the depth value.

[0051] However, the above-mentioned existing filtering methods have problems such as time extension and low filtering efficiency, which makes it difficult to meet the requirements of depth camera modules in some application scenarios (for example, the logistics industry and industrial production). The reason is found to be: although its filtering principle is accurate, it does not fully utilize the correlation characteristics and similarities between data when configuring the data processing process, so that it consumes too much computing resources and time when reading, storing, and processing data. In particular, before normalizing the grayscale value of the current frame, it needs to store the data of the current frame and traverse the data of the current frame to obtain the maximum grayscale value of the grayscale image of the current frame. Its data processing process is very time-consuming.

[0052] In response to the above technical problems, the basic idea of ​​the present application is to make full use of the correlation and similarity between data to optimize the filtering process for the depth camera module. Specifically, the inventor of the present application found that when using the depth camera module for depth measurement, the scene it faces and the target object in the scene will not change significantly in a short period of time, that is, in the depth image and grayscale image collected by the depth camera module, there is a high similarity and correlation between the grayscale images separated by a preset time. Therefore, the maximum grayscale value in the grayscale image separated by a preset frame from the current frame can be used as the maximum grayscale value for normalizing the grayscale values ​​of each pixel in the grayscale image of the current frame. In this way, the data processing flow of the filtering scheme is optimized to reduce latency and improve filtering efficiency.

[0053] Based on this, the present application proposes a filtering method for a depth camera module, which includes: obtaining a depth image of a current frame and a grayscale image corresponding to the depth image; obtaining a reference grayscale value, wherein the reference grayscale value is the maximum grayscale value of the grayscale image of a reference frame with a preset number of frames interval from the current frame; normalizing the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value; and, based on a preset mapping relationship between the depth value and the normalized grayscale value, filtering out the pixel points in the depth image of the current frame whose depth values ​​are judged to be invalid to obtain a filtered depth image.

[0054] In addition, the present application also provides a filtering device for a depth camera module, which includes: a current frame data acquisition unit, used to acquire a depth image of the current frame and a grayscale image corresponding to the depth image; a reference grayscale value acquisition unit, used to acquire a reference grayscale value, wherein the reference grayscale value is the maximum grayscale value of the grayscale image of a reference frame with a preset number of frames interval from the current frame; a normalization processing unit, used to normalize the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized image has a normalized grayscale value; and a filtering unit, used to filter out pixel points whose depth values ​​in the depth image of the current frame are judged to be invalid based on a preset mapping relationship between depth values ​​and the normalized grayscale values ​​to obtain a filtered depth image.

[0055] The present application also provides an electronic device, which includes: a memory and a processor, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the filtering method for the depth camera module as described above.

[0056] It is worth mentioning that although the filtering method is applied to the TOF camera module as an example in the embodiments of the present application, ordinary technicians in this field should know that the filtering method can also be applied to other active depth information camera modules, such as speckle structured light camera modules.

[0057] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0058] Exemplary Filtering Methods

[0059] like Figure 4 As shown, a filtering method for a depth camera module according to an embodiment of the present application is explained, wherein the filtering method includes: S110, obtaining a depth image of a current frame and a grayscale image corresponding to the depth image; S120, obtaining a reference grayscale value, wherein the reference grayscale value is the maximum grayscale value of the grayscale image of a reference frame with a preset number of frames interval from the current frame; S130, normalizing the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value; and, S140, based on a preset mapping relationship between the depth value and the normalized grayscale value, filtering out the pixel points whose depth values ​​in the depth image of the current frame are judged to be invalid to obtain a filtered depth image.

[0060] In step S110, a depth image of the current frame and a grayscale image corresponding to the depth image are obtained. Specifically, a depth image of the current frame and a grayscale image corresponding to the depth image are obtained from a depth camera module (e.g., a TOF camera module). Here, each pixel in the depth image of the current frame corresponds to each pixel in the grayscale image, and each pixel in the depth image has a depth value, and each pixel in the grayscale image has a grayscale value. The depth value of each pixel in the depth image is used to characterize the distance between the target object and the depth camera module, and the grayscale value of each pixel in the grayscale image is used to characterize the brightness of the environment where the target object is located.

[0061] Under ideal conditions, that is, when the depth value and the grayscale value are not affected by stray light during the process of obtaining the depth value and the grayscale value, the depth value and the grayscale value follow the following rule: the larger the depth value (the longer the distance), the smaller the grayscale value (the smaller the brightness). However, in actual scenes, the stray light in the environment affects the calculation of the depth value and the grayscale value, and noise data is introduced in the process of calculating the depth value and the grayscale value, that is, the depth value and grayscale value affected by stray light. The depth value and grayscale value affected by stray light do not follow the rule that the larger the depth value, the smaller the grayscale value. This feature can be used to filter out the depth value affected by stray light (invalid depth value), and then filter out the pixel points whose depth values ​​in the depth image are determined to be invalid.

[0062] It is worth mentioning that, in the process of determining the validity of the depth value, in order to facilitate data processing, the depth value or the grayscale value may be normalized to convert the range of the depth value or the grayscale value to a preset range. For example, the depth data or the grayscale value may be normalized using the maximum depth value or the maximum grayscale value.

[0063] Furthermore, considering that the depth value is limited by the physical structure of the depth camera module, the maximum depth value of each frame is fixed, while the grayscale value varies with factors such as the ambient brightness and shooting angle in the application scene, and the maximum grayscale value of each frame changes dynamically. Therefore, the grayscale value can be normalized by the maximum grayscale value.

[0064] Furthermore, as previously mentioned, if the grayscale value of the current frame is normalized using the maximum grayscale value of the current frame's grayscale image, a long wait time is required after obtaining the depth image and grayscale image of the current frame to store and traverse the data of the current frame in order to obtain the maximum grayscale value of the current frame's grayscale image. This long wait time results in a longer delay in the normalization process, which in turn causes a longer delay in the filtering process and lower filtering efficiency.

[0065] Based on this, considering that the scene and the target object in the scene will not change significantly in a short period of time, among the depth images and grayscale images captured by the depth camera module, the grayscale images separated by a preset number of frames have a high similarity and correlation. The grayscale value of the current frame can be normalized using the maximum grayscale value of the grayscale image of the reference frame separated by a preset number of frames from the current frame, such as Figure 5 shown.

[0066] Accordingly, in step S120, a reference grayscale value is obtained, and the reference grayscale value is the maximum grayscale value of the grayscale image of the reference frame that is spaced a preset number of frames from the current frame. In one example of the present application, the reference frame and the current frame are adjacent frames, that is, the preset number of frames is 1. It should be understood that in other examples of the present application, the preset number of frames between the reference frame and the current frame can be other values, for example, the preset number of frames is 2, that is, the reference grayscale value is the maximum value of the grayscale image of the reference frame that is spaced 2 frames from the current frame, and this is not limited to the present application. It should be noted that the preset number of frames should not be too large. When the preset number of frames is too large, the data of the reference frame and the data of the current frame may differ greatly, which is not suitable for normalizing the grayscale value of the grayscale image of the current frame.

[0067] It is worth mentioning that when acquiring the depth image and the grayscale image of the current frame, the reference grayscale value is also acquired, that is, the steps S110 and S120 are performed in parallel. In other words, acquiring the depth image of the current frame and the grayscale image corresponding to the depth image and acquiring the reference grayscale value are performed in parallel. In this way, there is no need to wait for the data of the current frame to be traversed before normalizing the grayscale value, which can reduce latency and improve filtering efficiency.

[0068] Specifically, the reference grayscale value is acquired to the processor through an external storage unit. In this way, there is no need to re-read the data from the cache space, which can further shorten the time of the normalization processing, reduce the latency, and improve the filtering efficiency. Moreover, by using the external storage unit to transmit data, there is no need to occupy the internal storage space, which reduces the memory overhead. More specifically, the processor is a parallel chip, such as an FPGA chip, an ASIC chip, or an SOC chip, wherein the parallel chip can process data in parallel, that is, the parallel chip has the ability to process in parallel. That is, in step S120, acquiring the reference grayscale value includes: acquiring the reference grayscale value from the external storage unit to the parallel chip, wherein the parallel chip can process data in parallel.

[0069] Accordingly, after obtaining the reference grayscale value, the grayscale value of each pixel in the grayscale image of the current frame can be normalized based on the reference grayscale value. Specifically, in step S130, the grayscale value of each pixel in the grayscale image of the current frame is normalized based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value, including: normalizing the grayscale value of each pixel in the grayscale image of the current frame according to the following formula, wherein the formula is Yi=Xi / Grey_max, wherein Yi represents the normalized grayscale value, Xi represents the grayscale value of each pixel in the grayscale image of the current frame, and Grey_max represents the reference grayscale value.

[0070] It should be understood that other methods may be used to normalize the grayscale values ​​and obtain the normalized grayscale values. For example, the normalized grayscale value may be calculated by calculating the ratio of the grayscale value of the grayscale image to the square of the reference grayscale value, which is not limited to this application. Accordingly, the reference grayscale value may also be implemented as other values, such as the square of the maximum grayscale value of the grayscale image of the reference frame, which is also not limited to this application.

[0071] Furthermore, after normalizing the grayscale values ​​of the grayscale image of the current frame and obtaining the normalized grayscale values, the pixel points whose depth values ​​are judged to be invalid in the depth image of the current frame can be filtered out based on a preset mapping relationship between the depth values ​​and the normalized grayscale values ​​to obtain a filtered depth image.

[0072] Specifically, the preset mapping relationship between the depth value and the normalized grayscale value is a linear mapping relationship, wherein the depth value and the normalized grayscale value are in a negatively correlated linear relationship. In other words, the depth value and the grayscale value change in a manner such that the larger the depth value, the smaller the grayscale value.

[0073] Figure 6 A schematic diagram of a function image between the depth value and the normalized grayscale value is shown, as shown in Figure 6 As shown, when the depth value is in the range of 0 to a first preset value (for example, 0 to D3), the normalized grayscale value is in the range of a second preset value to 1 (for example, G3 to 1), and when the depth value is the first preset value, the normalized grayscale value is the second preset value.

[0074] Correspondingly, when the depth value of a pixel point in the depth image is within the range of 0 to the first preset value, and the normalized gray value corresponding to the pixel point is less than the second preset value, the depth value of the pixel point is determined to be invalid. When the depth value of a pixel point in the depth image is greater than the first preset value, and the normalized gray value corresponding to the pixel point is within the range of the second preset value to 1, the depth value of the pixel point is also determined to be invalid.

[0075] Specifically, taking the depth value (Di) of a pixel point in the depth image within the range of 0 to D3 as an example, correspondingly, the value (Gi) of the normalized gray value of the pixel point in the gray image corresponding to the depth image is within the range of 1 to G3. Otherwise, the depth value of the pixel point in the depth image is determined to be invalid, and the points in the depth image whose depth values are determined to be invalid are determined as invalid pixel points. That is to say, the validity of the depth value can be judged by the following formula, and the formula is:

[0076] If (D3 > Di ≥ 0)

[0077] If (1 > Gi ≥ G3), it is determined that the depth data of the pixel point is valid;

[0078] Else (Gi < G3), it is determined that the depth data of the pixel point is invalid.

[0079] Else (D3 < Di)

[0080] If (1 > Gi ≥ G3), it is determined that the depth data of the pixel point is invalid;

[0081] Else (Gi < G3), it is determined that the depth data of the pixel point is valid;

[0082] Correspondingly, Figure 7 The figure illustrates a flowchart for filtering out the pixel points in the depth image of the current frame whose depth values are determined to be invalid based on the preset mapping relationship between the depth value and the normalized gray value to obtain a filtered depth image. As Figure 7As shown, in step S140, based on the preset mapping relationship between the depth value and the normalized grayscale value, the process of filtering out the pixel points whose depth values ​​in the depth image of the current frame are determined to be invalid to obtain a filtered depth image includes: S210, in response to the depth value of the pixel point in the depth image being in the range of 0 to the first preset value and the normalized grayscale value corresponding to the pixel point being less than the second preset value, determining that the depth value of the pixel point is invalid; and, S220, in response to the depth value of the pixel point in the depth image being greater than the first preset value and the normalized grayscale value corresponding to the pixel point being in the range of the second preset value to 1, determining that the depth value of the pixel point is invalid.

[0083] It is worth mentioning that the linear mapping relationship of the current frame is a multi-level linear mapping relationship. Considering that in actual applications, the mapping relationship between the depth values ​​within different ranges and the normalized grayscale values ​​corresponding to the depth values ​​may vary. Accordingly, the mapping relationship between the depth values ​​within different ranges and the normalized grayscale values ​​corresponding to the depth values ​​can be preset according to actual application conditions.

[0084] Specifically, if Figure 6 As shown, in actual applications, according to the ranging range of the depth camera module (the range of the depth value), the range of the depth value of each pixel point of the depth image of the current frame can be divided into at least two segments, that is, the range of the depth value includes at least two preset ranges, and the depth value within each preset range and the normalized grayscale value corresponding to the depth value correspond to a first-level preset linear mapping relationship.

[0085] Further, the following is a specific example. Figure 6 As shown, when the depth value is in the range of 0 to D1, the normalized grayscale value is in the range of G1 to 1. Accordingly, the linear mapping relationship between the depth value and the normalized grayscale value is G=k1*D+b1, where G represents the normalized grayscale value, D represents the depth value, k1 represents the first relationship coefficient, and b1 represents the first constant. When the depth value is D1, the normalized grayscale value is G1.

[0086] Accordingly, when the depth value of a pixel in the depth image is within the range of 0 to D1, and the normalized grayscale value corresponding to the pixel is less than G1, the depth value of the pixel is determined to be invalid. When the depth value of a pixel in the depth image is greater than D1, and the normalized grayscale value corresponding to the pixel is within the range of G1 to 1, the depth value of the pixel is also determined to be invalid.

[0087] When the depth value is within the range of D1 to D2, the normalized grayscale value is within the range of G2 to G1. Accordingly, the linear mapping relationship between the depth value and the normalized grayscale value is G=k2*D+b2, where k2 represents a second relationship coefficient and b2 represents a second constant. When the depth value is D2, the normalized grayscale value is G2.

[0088] Correspondingly, when the depth value of a pixel in the depth image is within the range of 0 to D2, and the normalized grayscale value corresponding to the pixel is less than G2, the depth value of the pixel is determined to be invalid. When the depth value of a pixel in the depth image is greater than D2, and the normalized grayscale value corresponding to the pixel is within the range of G2 to 1, the depth value of the pixel is also determined to be invalid.

[0089] The range of the depth value includes at least two preset ranges, for example, three, four, or five. Correspondingly, the multi-level linear mapping relationship includes at least two levels of mapping relationships, for example, three, four, or five. It is worth mentioning that, in actual applications, as the depth value increases, the absolute value of the relationship coefficient of the preset linear mapping relationship between the depth value within the preset range and the normalized grayscale value corresponding to the depth value gradually decreases. In other words, as the depth value increases, the rate at which the normalized depth value changes with the depth value slows down.

[0090] Further, the following is a specific example. Figure 6 As shown, when the depth value is in the range of 0 to D1, the linear mapping relationship between the depth value and the normalized grayscale value is G=k1*D+b1, when the depth value is in the range of D1 to D2, the linear mapping relationship between the depth value and the normalized grayscale value is G=k2*D+b2, when the depth value is in the range of D2 to D3, the linear mapping relationship between the depth value and the normalized grayscale value is G=k3*D+b3, D3>D2>D1, then, |K1|>|K2|>|K3|.

[0091] Furthermore, the setting method of the preset range of the depth value range, the number of preset ranges and the linear mapping relationship of each level can be adjusted according to the actual application situation. The linear mapping relationship of different levels can be used to determine the depth values ​​in different ranges to more accurately determine the validity of the depth values. Furthermore, based on the determination results of the depth values ​​in different ranges, invalid pixels in the depth image of the current frame can be filtered out in a partitioned manner, that is, the depth values ​​are determined to be invalid pixels. Invalid pixels in the depth image can be filtered out relatively accurately by partition filtering, thereby improving the filtering accuracy.

[0092] In summary, the filtering method for the depth camera module is explained. The filtering method uses the maximum grayscale value of the grayscale image of the reference frame to normalize the grayscale value of the grayscale image of the current frame, which can shorten the data processing time, reduce the delay in the filtering process, and improve the filtering efficiency. The filtering method can efficiently filter out pixels in the image affected by stray light and reduce the impact of stray light on the image. Moreover, during the filtering process, invalid pixels in the depth image of the current frame are filtered out by partition filtering, which can improve the filtering accuracy.

[0093] Figure 8 The figure shows a schematic diagram of a depth point cloud before filtering in the filtering method according to an embodiment of the present application. Figure 9 The figure shows a schematic diagram of a depth point cloud after filtering in the filtering method according to an embodiment of the present application.

[0094] like Figure 8 As shown in the figure, due to the interference of stray light in the environment, there are noise points in the depth point cloud before filtering, which makes it difficult to accurately reflect the actual information of the target object (for example, the actual shape and actual size). Specifically, the stray light in the environment will interfere with the optical signal received by the depth camera module, causing a deviation between the calculated depth value and the actual depth value, making it difficult to obtain accurate depth information. The point cloud data of the depth point cloud is obtained by coordinate transformation of the depth image. Inaccurate depth information causes the obtained point cloud data to be inaccurate.

[0095] like Figure 9 As shown in the figure, after filtering the depth point cloud using the above filtering method, the noise in the depth point cloud is filtered out as much as possible, and the effect is obvious. It can be seen that the filtering method can effectively filter out pixels in the depth point cloud affected by stray light, and reduce the impact of stray light on the depth point cloud.

[0096] Exemplary filtering devices

[0097] According to another aspect of the present application, a filtering device for a depth camera module is also provided. Figure 10As shown, a filtering device for a depth camera module according to an embodiment of the present application is illustrated, wherein the filtering device includes: a current frame data acquisition unit 10, for acquiring a depth image of the current frame and a grayscale image corresponding to the depth image; a reference grayscale value acquisition unit 20, for acquiring a reference grayscale value, wherein the reference grayscale value is the maximum grayscale value of the grayscale image of a reference frame with a preset number of frames interval from the current frame; a normalization processing unit 30, for normalizing the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized image has a normalized grayscale value; and a filtering unit 40, for filtering out pixel points whose depth values ​​in the depth image of the current frame are judged to be invalid based on a preset mapping relationship between the depth value and the normalized grayscale value to obtain a filtered depth image.

[0098] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the above-mentioned filtering device have been described in detail in the referenced examples. Figures 1 to 9 The filtering method for the depth camera module is described in detail, and therefore, its repeated description will be omitted.

[0099] As described above, the filtering device and the depth camera module can be separate devices, and the filtering device can be connected to the depth camera module through a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0100] Alternatively, in another example, the filtering device according to the embodiment of the present application can also be implemented in a depth camera module such as a TOF camera module. In one example, the filtering device according to the embodiment of the present application can be integrated into the depth camera module as a software module and / or hardware module. For example, the filtering device can be a software module in the operating system of the depth camera module, or it can be an application developed for the depth camera module; of course, the filtering device can also be one of the many hardware modules of the depth camera module.

[0101] Exemplary electronic devices

[0102] According to another aspect of the present application, an electronic device 80 is also provided. Figure 11 The electronic device 80 includes a memory 81 and a processor 82. The memory 81 stores computer program instructions, which, when executed by the processor 82, enable the processor 82 to execute any one of the filtering methods for the depth camera module described above.

[0103] Specifically, in an embodiment of the present application, the processor 82 can be implemented as an FPGA chip. After the electronic device 80 is powered on, the TOF camera module 80 is initialized and configured through the FPGA chip. After the TOF camera module 80 is initialized, first, the depth value of the current frame and the grayscale value corresponding to the depth value are calculated and obtained through the FPGA chip, and then, based on the depth value and the grayscale value, the depth image of the current frame and the grayscale image corresponding to the depth image can be obtained. Then, after the FPGA chip obtains the reference grayscale value, it can normalize the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value. Finally, the FPGA chip can filter out the pixel points whose depth values ​​in the depth image of the current frame are determined to be invalid based on the preset mapping relationship between the depth value and the normalized grayscale value to obtain a filtered depth image.

[0104] It should be understood that the processor 82 may also be implemented as other chips with parallel computing capabilities, such as an ASIC chip or a SOC chip. This is not limited to this application. In other words, the processor is a parallel chip that can process data in parallel.

[0105] Here, those skilled in the art will appreciate that the specific functions and operations of the processor 82 have been described above with reference to Figures 1 to 9 The filtering method for the depth camera module is described in detail, and therefore, its repeated description will be omitted.

[0106] In the embodiment of the present application, the memory 81 includes a double data rate synchronous dynamic random access memory (DDRIII SDRAM). Optionally, the memory further includes a Flash memory.

[0107] In summary, the electronic device 80 is explained, and the electronic device 80 can use the maximum grayscale value of the grayscale image of the reference frame to normalize the grayscale value of the grayscale image of the current frame to shorten the data processing time, reduce the delay in the filtering process, and improve the filtering efficiency.

[0108] Those skilled in the art will appreciate that the embodiments of the present invention described above and shown in the accompanying drawings are intended to be illustrative only and are not intended to limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and illustrated in the embodiments. Any variations or modifications may be made to the embodiments of the present invention without departing from the principles described.

Claims

1. A filtering method for a depth camera module, characterized in that: include: Obtaining a depth image of a current frame and a grayscale image corresponding to the depth image; Acquire a reference grayscale value, where the reference grayscale value is a maximum grayscale value of a grayscale image of a reference frame that is a preset number of frames apart from the current frame; Normalizing the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value; and Based on a preset mapping relationship between the depth value and the normalized grayscale value, filtering out pixels whose depth values ​​are determined to be invalid in the depth image of the current frame to obtain a filtered depth image, comprising: In response to the depth value of a pixel in the depth image being within a range from 0 to a first preset value and the normalized grayscale value corresponding to the pixel being less than a second preset value, determining that the depth value of the pixel is invalid; and In response to the depth value of a pixel in the depth image being greater than the first preset value and the normalized grayscale value corresponding to the pixel being within a range from the second preset value to 1, it is determined that the depth value of the pixel is invalid.

2. The filtering method for a depth camera module according to claim 1, wherein: The reference frame and the current frame are adjacent frames.

3. The filtering method for a depth camera module according to claim 1, wherein: Normalizing the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value, includes: The grayscale value of each pixel in the grayscale image of the current frame is normalized using the following formula, where the formula is Yi=Xi / Grey_max, where Yi represents the normalized grayscale value, Xi represents the grayscale value of each pixel in the grayscale image of the current frame, and Grey_max represents the reference grayscale value.

4. The filtering method for a depth camera module according to claim 1, wherein: The preset mapping relationship between the depth value and the normalized grayscale value is a linear mapping relationship, and the linear mapping relationship includes: the depth value and the normalized grayscale value are in a negatively correlated linear relationship, when the depth value is in the range of 0 to a first preset value, the normalized grayscale value is in the range of a second preset value to 1, and when the depth value is the first preset value, the normalized grayscale value is the second preset value.

5. The filtering method for a depth camera module according to claim 4, wherein: The linear mapping relationship is a multi-level linear mapping relationship.

6. The filtering method for a depth camera module according to claim 1, wherein: The acquiring of the depth image of the current frame and the grayscale image corresponding to the depth image and the acquiring of the reference grayscale value are performed in parallel.

7. The filtering method for a depth camera module according to claim 6, wherein: Get the reference grayscale value, including: The reference grayscale value is acquired from an external storage unit to a parallel chip, wherein the parallel chip is capable of processing data in parallel.

8. A filtering device for a depth camera module, characterized in that: include: A current frame data acquisition unit, configured to acquire a depth image of a current frame and a grayscale image corresponding to the depth image; A reference grayscale value acquiring unit, configured to acquire a reference grayscale value, wherein the reference grayscale value is a maximum grayscale value of a grayscale image of a reference frame that is a preset number of frames apart from the current frame; a normalization processing unit, configured to normalize the grayscale value of each pixel in the grayscale image of the current frame based on the reference grayscale value to obtain a normalized grayscale image, wherein each pixel in the normalized grayscale image has a normalized grayscale value; and a filtering unit, configured to filter out pixels whose depth values ​​are determined to be invalid in the depth image of the current frame based on a preset mapping relationship between the depth value and the normalized grayscale value to obtain a filtered depth image, wherein the filtering unit determines that the depth value of a pixel in the depth image is invalid in response to the depth value of the pixel being within a range of 0 to a first preset value and the normalized grayscale value corresponding to the pixel being less than a second preset value; and The filtering unit determines that the depth value of the pixel in the depth image is invalid in response to the depth value of the pixel being greater than the first preset value and the normalized grayscale value corresponding to the pixel being within the range from the second preset value to 1.

9. The filtering device for a depth camera module according to claim 8, wherein: The reference frame and the current frame are adjacent frames.

10. The filtering device for a depth camera module according to claim 8, wherein: The normalization processing unit is further used to: normalize the grayscale value of each pixel in the grayscale image of the current frame according to the following formula, wherein the formula is Yi=Xi / Grey_max, wherein Yi represents the normalized grayscale value, Xi represents the grayscale value of each pixel in the grayscale image of the current frame, and Grey_max represents the reference grayscale value.

11. The filtering device for a depth camera module according to claim 8, wherein: The preset mapping relationship between the depth value and the normalized grayscale value is a linear mapping relationship, and the linear mapping relationship includes: the depth value and the normalized grayscale value are in a negatively correlated linear relationship, when the depth value is in the range of 0 to a first preset value, the normalized grayscale value is in the range of a second preset value to 1, and when the depth value is the first preset value, the normalized grayscale value is the second preset value.

12. An electronic device, characterized in that: include: Memory; and A processor, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the filtering method for the depth camera module as described in any one of claims 1 to 7.

13. The electronic device according to claim 12, wherein: The processor is a parallel chip, wherein the parallel chip can process data in parallel.

Images