Depth image generation method and system, computer equipment and computer program product

Abstract

The invention belongs to the technical field of depth image processing, and provides a depth image generation method and system, computer equipment and a computer program product, and the method comprises the steps: collecting a plurality of groups of original image data of a target scene based on a two-dimensional heterogeneous data collection strategy, carrying out the pixel-level fusion based on the plurality of groups of original image data, and obtaining a depth image; and obtaining a depth value of each target pixel, and constructing a depth image of the target scene based on the depth value of each target pixel. According to the method provided by the invention, the precision and reliability of the depth value of each pixel are ensured through the two-dimensional heterogeneous multiple groups of original image data and pixel-level fusion, and a complete and reliable high-precision depth image of a target scene is generated in a high dynamic range scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of depth image processing, and particularly relates to a depth image generation method and system, a computer device and a computer program product. BACKGROUND

[0002] Time-Of-Flight (TOF) technology acquires depth information by measuring the time difference of light signals from emission to reflection by an object and return to the sensor, and has the characteristics of high real-time performance and relatively low computational complexity. This makes the TOF technology show great application potential in the fields of autonomous driving, robot navigation, human-computer interaction, augmented reality / virtual reality (AR / VR), industrial automation and precise medical imaging.

[0003] Although the existing TOF technology has broad prospects, in actual application, the performance of the TOF depth imaging system is not ideal in a high dynamic range (HDR) scene, which seriously limits the application robustness and reliability of the TOF technology in complex and high dynamic scenes, and cannot meet the scene requirements for the integrity and high precision of depth information.

[0004] Therefore, there is an urgent need for a depth image imaging method to generate high-precision and high-reliability depth images of a target scene in a high dynamic range scene. SUMMARY

[0005] In view of this, the embodiments of the present application provide a depth image generation method and system, a computer device and a computer program product to generate complete and reliable high-precision depth images of a target scene in a high dynamic range scene.

[0006] The first aspect of the embodiments of the present application provides a depth image generation method, comprising:

[0007] acquiring a plurality of groups of original image data of a target scene based on a dual-dimensional heterogeneous data acquisition strategy;

[0008] performing pixel-level fusion based on the plurality of groups of original image data to obtain a depth value of each target pixel;

[0009] constructing a depth image of the target scene based on the depth value of each target pixel.

[0010] In an implementation form of the first aspect, the dual-dimension heterogeneous data acquisition strategy comprises a first acquisition strategy, a second acquisition strategy and a third acquisition strategy, the first acquisition strategy being a first modulation frequency combined with a first exposure time, the second acquisition strategy being a second modulation frequency combined with a second exposure time, the third acquisition strategy being a third modulation frequency combined with a third exposure time, the first modulation frequency being greater than the second modulation frequency, the second modulation frequency being greater than the third modulation frequency, the first exposure time being greater than the second exposure time, the second exposure time being greater than the third exposure time.

[0011] acquiring a plurality of groups of original image data of the target scene based on the dual-dimension heterogeneous data acquisition strategy, comprises:

[0012] acquiring first original image data of the target scene based on the first acquisition strategy;

[0013] acquiring second original image data of the target scene based on the second acquisition strategy;

[0014] acquiring third original image data of the target scene based on the third acquisition strategy.

[0015] In an implementation form of the first aspect, the pixel-level fusion based on the plurality of groups of original image data to obtain the depth value of each target pixel comprises:

[0016] For each target pixel, in a case that corresponding pixel data of the target pixel in the first original image data, the second original image data and the third original image data are all valid, calculating a first depth value and a first wrapped residual error of the target pixel based on corresponding pixel data of the target pixel in the first original image data and the second original image data, and calculating a second depth value and a second wrapped residual error of the target pixel based on corresponding pixel data of the target pixel in the second original image data and the third original image data;

[0017] if the first wrapped residual error is not greater than the second wrapped residual error, determining the first depth value as the depth value of the target pixel;

[0018] if the first wrapped residual error is greater than the second wrapped residual error, determining the second depth value as the depth value of the target pixel.

[0019] In an implementation form of the first aspect, the pixel-level fusion based on the plurality of groups of original image data to obtain the depth value of each target pixel further comprises:

[0020] For each target pixel, if the pixel data corresponding to the target pixel in the second raw image data is valid and the pixel data corresponding to the target pixel in the third raw image data is invalid, the depth value of the target pixel is calculated based on the pixel data corresponding to the target pixel in the second raw image data.

[0021] In an implementation form of the first aspect, the pixel-level fusion based on the plurality of sets of raw image data to obtain the depth value of each target pixel further includes:

[0022] For each target pixel, if the pixel data corresponding to the target pixel in the second raw image data is valid and the pixel data corresponding to the target pixel in the third raw image data is invalid, the depth value of the target pixel is calculated based on the pixel data corresponding to the target pixel in the second raw image data.

[0023] For each target pixel, if the pixel data corresponding to the target pixel in the second raw image data is invalid and the pixel data corresponding to the target pixel in the third raw image data is valid, the depth value of the target pixel is calculated based on the pixel data corresponding to the target pixel in the third raw image data.

[0024] In an implementation form of the first aspect, the pixel-level fusion based on the plurality of sets of raw image data to obtain the depth value of each target pixel further includes:

[0025] For each target pixel, if the pixel data corresponding to the target pixel in the first raw image data, the second raw image data and the third raw image data are all invalid, the depth value of the target pixel is set to a preset invalid value.

[0026] In an implementation form of the first aspect, the method further includes:

[0027] For each target pixel, whether the pixel corresponding to the target pixel in the raw image data has valid saturation is determined according to the amplitude of the four-phase raw data of the pixel corresponding to the target pixel in the raw image data and a preset exposure threshold, and whether the pixel corresponding to the target pixel in the raw image data has valid signal-to-noise ratio is determined according to the signal amplitude of the pixel corresponding to the target pixel in the raw image data and a preset amplitude threshold.

[0028] In a case where the pixel corresponding to the target pixel in the raw image data has valid saturation and valid signal-to-noise ratio, it is determined that the pixel data corresponding to the target pixel in the raw image data is valid.

[0029] The second aspect of the embodiment of the present application provides a depth image generation system, comprising:

[0030] a data acquisition module, configured to acquire a plurality of groups of original image data of a target scene based on a dual-dimension heterogeneous data acquisition strategy;

[0031] a pixel fusion module, configured to perform pixel-level fusion based on the plurality of groups of original image data to obtain a depth value of each target pixel;

[0032] a depth image construction module, configured to construct a depth image of the target scene based on the depth value of each target pixel.

[0033] The third aspect of the embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the depth image generation method according to the first aspect when executing the computer program.

[0034] The fourth aspect of the embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed to make the depth image generation method according to the first aspect be executed.

[0035] The first aspect of the embodiment of the present application has the beneficial effects that: a plurality of groups of original image data of a target scene are acquired based on a dual-dimension heterogeneous data acquisition strategy, pixel-level fusion is performed based on the plurality of groups of original image data to obtain a depth value of each target pixel, and then a depth image of the target scene is constructed based on the depth value of each target pixel. The precision and reliability of the depth value of each pixel are ensured through the dual-dimension heterogeneous plurality of groups of original image data and the pixel-level fusion, and a complete and reliable high-precision depth image of the target scene is generated in a high dynamic range scene.

[0036] It can be understood that the beneficial effects of the second aspect to the fourth aspect can be referred to the related description in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0038] Figure 1 is the implementation flow diagram of the depth image generation method provided by the embodiment of the present application;

[0039] Figure 2is a structural schematic diagram of a depth image generation system provided by an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of a computer device provided by an embodiment of the present application;

[0041] Figure 4 is a schematic diagram of a computer program product provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description of the application with unnecessary detail.

[0043] It should be understood that the term "includes" when used in the specification and the appended claims herein, specifies the presence of stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0044] It should also be understood that the term "and / or" when used in the specification and the appended claims herein, means any one or more of the associated listed items and includes all possible combinations of the associated listed items.

[0045] As used in the description of the application and the appended claims herein, the term "if' can be interpreted as meaning "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]", depending on the context.

[0046] In addition, in the description of the application and the appended claims herein, the terms "first", "second", "third", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0047] Reference within the specification of this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places within specified

[0048] The embodiment of the present application provides a depth image generation method, which is used for generating a complete and reliable high-precision depth image of a target scene in a high dynamic range scene. The depth image generation method provided by the present application collects a plurality of groups of original image data of the target scene based on a dual-dimensional heterogeneous data acquisition strategy, performs pixel-level fusion based on the plurality of groups of original image data, obtains a depth value of each target pixel, and then constructs a depth image of the target scene based on the depth value of each target pixel. The dual-dimensional heterogeneous plurality of groups of original image data and the pixel fusion ensure the precision and reliability of the depth value of each pixel, and the complete and reliable high-precision depth image of the target scene in the high dynamic range scene is generated.

[0049] The depth image generation method provided by the embodiment of the present application can be applied to a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a server, a cloud server, a desktop computer and other computer devices. The embodiment of the present application does not make any limitation on the specific type of the computer device.

[0050] As shown in Figure 1 The embodiment of the present application provides a depth image generation method, which includes:

[0051] In step S1, a plurality of groups of original image data of a target scene are collected based on a dual-dimensional heterogeneous data acquisition strategy.

[0052] In the application, "dual-dimensional" refers to two dimensions of modulation frequency and exposure time, and "heterogeneous" refers to differential configuration in the two dimensions, and each group of original image data corresponds to a "dual-dimensional" configuration.

[0053] In application, the multiple groups of original image data can be sequentially collected or collected in parallel, and the type of target scene can be configured according to the type of target scene. For example, in the scene of security monitoring, the motion of objects in the scene is less or the motion speed is low, and the sequential collection of multiple groups of original image data is adopted to reduce the configuration requirement of collection hardware and save cost. In the scene of automatic driving and auxiliary driving, the parallel collection of multiple groups of original image data is adopted to ensure the synchronization of the multiple groups of original data collected.

[0054] In application, the multiple groups of original image data collected by the same TOF camera are spatially aligned, for example, the multiple groups of original image data are sequentially collected by sequentially configuring different modulation frequencies and exposure times by a single TOF camera. The multiple groups of original image data collected by different TOF cameras are corrected to the same view angle by external parameter calibration (rotation and translation), for example, the multiple groups of original image data are collected in parallel by configuring different modulation frequencies and exposure times by multiple TOF cameras.

[0055] In application, the sequential collection of multiple groups of original image data is ensured by shortening the collection interval to ensure the alignment degree of multiple groups of original image data in time or by various software compensation. In the case of parallel collection of multiple groups of original image data, the multiple groups of original image data collected are aligned in time.

[0056] Step S2, pixel-level fusion is performed based on the multiple groups of original image data to obtain the depth value of each target pixel.

[0057] In application, the multiple groups of original image data are ensured to be time-aligned and spatially aligned before pixel-level fusion.

[0058] Step S2, the depth image of the target scene is constructed based on the depth value of each target pixel.

[0059] In application, the depth value of the target pixel is filled into the corresponding pixel position of the depth image according to the position coordinates of the target pixel to obtain a first depth image. It can be understood that the depth image and the multiple groups of original image data have the same number of pixels.

[0060] In one embodiment, the dual-dimensional heterogeneous data collection strategy includes a first collection strategy, a second collection strategy and a third collection strategy, the first collection strategy is a first modulation frequency combined with a first exposure time, the second collection strategy is a second modulation frequency combined with a second exposure time, and the third collection strategy is a third modulation frequency combined with a third exposure time, the first modulation frequency is greater than the second modulation frequency, the second modulation frequency is greater than the third modulation frequency, the first exposure time is greater than the second exposure time, and the second exposure time is greater than the third exposure time.

[0061] The step S1 comprises:

[0062] The step S11 comprises collecting first original image data of the target scene based on the first acquisition strategy.

[0063] The step S12 comprises collecting second original image data of the target scene based on the second acquisition strategy.

[0064] The step S13 comprises collecting third original image data of the target scene based on the third acquisition strategy.

[0065] In an application, the first modulation frequency, the second modulation frequency and the third modulation frequency satisfy an integer multiple (integer ratio) relationship logic, for example, the first modulation frequency is 120MHz, the unambiguous range is 1.25m, the second modulation frequency is 75MHz, the unambiguous range is 2m, the two frequencies are in an integer ratio of 8:5, and the greatest common divisor is 15MHz.

[0066] In an embodiment, the first acquisition strategy is 120MHz combined with a first exposure time, the second acquisition strategy is 75MHz combined with a second exposure time, and the third acquisition strategy is 15MHz combined with a third exposure time.

[0067] The step S2 comprises:

[0068] The step S21 comprises, for each target pixel, in a case where corresponding pixel data of the target pixel in the first original image data, the second original image data and the third original image data are all valid, calculating a first depth value and a first wrapped residual error of the target pixel based on corresponding pixel data of the target pixel in the first original image data and the second original image data, and calculating a second depth value and a second wrapped residual error of the target pixel based on corresponding pixel data of the target pixel in the second original image data and the third original image data.

[0069] In an application, due to the periodicity (0 to 2π) of phase measurement, when the actual distance exceeds the maximum distance corresponding to a single modulation period, the measured phase will start from 0 again. For example, an object at a distance of 1.2 periods will have exactly the same phase reading as an object at a distance of 0.2 periods, which is phase wrapping, which limits the maximum unambiguous range of single-frequency measurement.

[0070] The maximum unambiguous range (d u ) is the maximum measurable distance in single-frequency measurement without phase wrapping. The calculation formula is Where c is the speed of light and f is the modulation frequency.

[0071] The basic inputs for dual-frequency calculation include: two modulation frequencies f A and f B (Hz), the phase values ​​obtained from the two measurements are respectively and The signal amplitude A obtained from the two measurements A and A B And the speed of light c (m / s).

[0072] Derived parameters include the effective frequency f E f E =gcd(f A ,f B Maximum unambiguous range gcd() is the greatest common divisor function.

[0073] Dividing the two original frequencies by their respective effective frequencies yields two coprime integers M. A M B ,Right now:

[0074]

[0075] They represent how many measurement cycles each of the two original frequencies contains within the extended maximum range.

[0076] Calculate the normalized phase p A ,p B :

[0077]

[0078] In application, the core of the algorithm is to determine the number of package cycles, with the goal of finding the unknown integer number of package cycles, n. A and n B Ideally, the actual distance d satisfies:

[0079] d = d uA (n A +p A )=d uB (n B +p B );

[0080] Calculate the phase difference term e:

[0081] e = pAMB - pBMAe = p A M B -p B M A ;

[0082] Establish key equations:

[0083] M A n B -M B n A = round(e);

[0084] Calculate modular multiplicative inverse k0:

[0085] mod(k0MA,MB) = 1 mod(k0M A ,M B ) = 1;

[0086] Solve n B :

[0087] n B = mod(k0·round(e),M B ) ;

[0088] Calculate final distance by weighted fusion. After determining the number of wrapping periods n B , the information of two measurements can be combined to calculate an accurate final distance.

[0089] Determine weight w:

[0090]

[0091] Calculate intermediate variable:

[0092] X = M A n B + M A p B + w(e - round(e));

[0093] Calculate final distance d:

[0094]

[0095] In applications, in certain measurement scenarios, such as corners of walls or edges of objects, multiple reflections of light signals (i.e., multipath effect) can cause inconsistent shifts in the phase of different frequency signals. This phenomenon can introduce residual errors when performing multi-frequency data fusion and phase unwrapping. The magnitude of the residual error can be used as a measure of the accuracy of the multi-frequency fusion result: the smaller the error, the higher the confidence in the calculated depth value.

[0096] In applications, how to accurately solve the number of wrapping periods under noise interference depends on whether the round() function can recover the correct integer.

[0097] Ideally, the actual distance d satisfies:

[0098] d = d A (n A +p′ A )=d B (n B +p′ B );

[0099] Where p′ A ,p′ B This is the true normalized phase without noise.

[0100] Will

[0101]

[0102] Substituting and simplifying, we get:

[0103] M B (n A +p′ A ) = M A (n B +p′ B );

[0104] Rearranging the terms, we get:

[0105] M A n B -M B n A =M B p′ A -M A p′ B ;

[0106] Since the left side is a linear combination of integers, the right side M B p′ A -M A p′ B It must be an integer. In actual measurements, however, the result is a noisy p. A ,p B Therefore, the calculation is as follows:

[0107] e = M B p A -M A p B ;

[0108] It fluctuates around the true integer value. As long as the noise is not too great, causing the value of e to deviate from the true integer by no more than ±0.5, then round(e) can accurately recover the correct integer, thus ensuring the correctness of subsequent calculations.

[0109] If the noise is too large, so that e is offset more than ±0.5, round() will output an incorrect integer. For example, when the true value is 5, but the calculation result is 4.4 or 5.6, round(e) will output 4 or 6, resulting in the solution (n A ,n B ) being completely wrong. The final distance result will have a huge jump error, with an error size equivalent to several times d A or d B .

[0110] The larger the standard deviation of e , the greater the probability of error:

[0111]

[0112] Step S22, if the first wrapping residual error is not greater than the second wrapping residual error, determining the first depth value as the depth value of the target pixel.

[0113] Step S23, if the first wrapping residual error is greater than the second wrapping residual error, determining the second depth value as the depth value of the target pixel.

[0114] Step S22, if the first wrapping residual error is not greater than the second wrapping residual error, determining the first depth value as the depth value of the target pixel.

[0115] Step S23, if the first wrapping residual error is greater than the second wrapping residual error, determining the second depth value as the depth value of the target pixel.

[0116] In application, through the above method, under the condition that the original image data quality is ideal, the precision advantage of high frequency signal can be fully utilized, and the uncertainty introduced by factors such as multipath effect can be intelligently avoided, to obtain a more accurate, reliable and comprehensive depth image.

[0117] In one embodiment, the step S2 of performing pixel-level fusion based on the plurality of sets of original image data to obtain the depth value of each target pixel further comprises:

[0118] Step S24, for each target pixel, if the pixel data corresponding to the target pixel in the second original image data and the pixel data corresponding to the target pixel in the third original image data are both valid, under the condition that the pixel data corresponding to the target pixel in the first original image data is invalid, calculating the depth value of the target pixel based on the pixel data corresponding to the target pixel in the second original image data and the third original image data.

[0119] In one embodiment, the step S2 of performing pixel-level fusion based on the multiple groups of raw image data to obtain the depth value of each target pixel further comprises:

[0120] The step S25 comprises: for each target pixel, if the pixel data corresponding to the target pixel in the second raw image data is valid and the pixel data corresponding to the target pixel in the third raw image data is invalid, calculating the depth value of the target pixel based on the pixel data corresponding to the target pixel in the second raw image data.

[0121] The step S26 comprises: for each target pixel, if the pixel data corresponding to the target pixel in the second raw image data is invalid and the pixel data corresponding to the target pixel in the third raw image data is valid, calculating the depth value of the target pixel based on the pixel data corresponding to the target pixel in the third raw image data.

[0122] In application, the process of calculating the depth value corresponding to a pixel based on any raw image data is as follows:

[0123] The phase corresponding to the raw image data The frequency multiplied by the unambiguous range The distance is obtained:

[0124] In one embodiment, the step S2 of performing pixel-level fusion based on the multiple groups of raw image data to obtain the depth value of each target pixel further comprises:

[0125] The step S27 comprises: for each target pixel, if the pixel data corresponding to the target pixel in the first raw image data, the second raw image data and the third raw image data are all invalid, setting the depth value of the target pixel as a preset invalid value.

[0126] In application, the step S3 further comprises: based on the first depth image, inserting a valid value into the pixel with the preset invalid value through neighborhood difference to obtain a second depth image. The specific interpolation algorithm can be realized through weighted average or weighted median of the trusted neighborhood. It can be understood that the accuracy and integrity of the depth image are improved through interpolation.

[0127] In one embodiment, the method further comprises:

[0128] Step S4, for each target pixel, determining whether the pixel corresponding to the target pixel in the original image data has valid saturation according to the amplitude of the four-phase raw data of the pixel corresponding to the target pixel in the original image data and a preset exposure threshold, and determining whether the pixel corresponding to the target pixel in the original image data has valid signal-to-noise ratio according to the signal amplitude of the pixel corresponding to the target pixel in the original image data and a preset amplitude threshold.

[0129] In application, for any original image data, according to the four-phase raw data of the pixel corresponding to the target pixel in the original image data: I0, I 90 , I 180 , I 270 , determine whether there is any phase raw data greater than a preset exposure threshold, if there is any phase raw data greater than the preset exposure threshold, it is determined that the data of the target pixel in the original image data is invalid, and the phase information of the oversaturated pixel has been completely distorted. The preset exposure threshold is generally the sensor exposure threshold (clipping), such as 2040, or slightly lower than the sensor exposure threshold, which can be taken, such as 2000.

[0130] In application, there is dark current and other noise in the sensor during imaging, and the signal is too weak to seriously affect the signal-to-noise ratio. For any original image data, the signal amplitude of the target pixel is calculated according to the four-phase raw data of the pixel corresponding to the target pixel in the original image data: If the signal amplitude A is lower than the preset threshold, it is considered that the signal-to-noise ratio of the pixel point is too low, and the phase information is unreliable.

[0131] In application, the above-mentioned valid saturation and valid signal-to-noise ratio are determined for different original image data to determine whether the pixel data corresponding to the target pixel in each original image data is valid.

[0132] Step S5, in the case that the pixel corresponding to the target pixel in the original image data has valid saturation and valid signal-to-noise ratio, it is determined that the pixel data corresponding to the target pixel in the original image data is valid.

[0133] In application, if the target pixel does not have valid saturation, it is not necessary to continue to calculate whether the target pixel has valid signal-to-noise ratio, and it is directly determined that the pixel data corresponding to the target pixel in the original image data is invalid, so as to avoid unnecessary amplitude calculation and improve data processing efficiency.

[0134] It should be understood that the size of the serial number of each step in the above-mentioned embodiment does not mean the order of execution, the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0135] The embodiment of the present application also provides a depth image generation system for executing the steps in the above depth image generation method embodiment. The depth image generation system can be a virtual appliance in a computer device, run by a processor of the computer device, or can be the computer device itself.

[0136] As shown in Figure 2 The embodiment of the present application provides a depth image generation system 20, which includes:

[0137] A data acquisition module 201 is configured to acquire a plurality of groups of original image data of a target scene based on a dual-dimensional heterogeneous data acquisition strategy;

[0138] A pixel fusion module 202 is configured to perform pixel-level fusion based on the plurality of groups of original image data to obtain a depth value of each target pixel;

[0139] A depth image construction module 203 is configured to construct a depth image of the target scene based on the depth value of each target pixel.

[0140] In one embodiment, the dual-dimensional heterogeneous data acquisition strategy includes a first acquisition strategy, a second acquisition strategy and a third acquisition strategy, the first acquisition strategy is a first modulation frequency combined with a first exposure time, the second acquisition strategy is a second modulation frequency combined with a second exposure time, the third acquisition strategy is a third modulation frequency combined with a third exposure time, the first modulation frequency is greater than the second modulation frequency, the second modulation frequency is greater than the third modulation frequency, the first exposure time is greater than the second exposure time, and the second exposure time is greater than the third exposure time;

[0141] The data acquisition module 201 is configured to:

[0142] acquire first original image data of a target scene based on the first acquisition strategy;

[0143] acquire second original image data of a target scene based on the second acquisition strategy;

[0144] acquire third original image data of a target scene based on the third acquisition strategy.

[0145] In one embodiment, the pixel fusion module 202 is configured to:

[0146] For each target pixel, if the corresponding pixel data of the target pixel in the first raw image data, the second raw image data and the third raw image data are all valid, calculating a first depth value and a first wrapping residual error of the target pixel based on the corresponding pixel data of the target pixel in the first raw image data and the second raw image data, and calculating a second depth value and a second wrapping residual error of the target pixel based on the corresponding pixel data of the target pixel in the second raw image data and the third raw image data;

[0147] If the first wrapping residual error is not greater than the second wrapping residual error, determining the first depth value as the depth value of the target pixel.

[0148] If the first wrapping residual error is greater than the second wrapping residual error, determining the second depth value as the depth value of the target pixel.

[0149] In one embodiment, the pixel fusion module 202 is further configured to:

[0150] For each target pixel, if the corresponding pixel data of the target pixel in the first raw image data is invalid, if the corresponding pixel data of the target pixel in the second raw image data and the corresponding pixel data of the target pixel in the third raw image data are both valid, calculating a depth value of the target pixel based on the corresponding pixel data of the target pixel in the second raw image data and the third raw image data.

[0151] In one embodiment, the pixel fusion module 202 is further configured to:

[0152] For each target pixel, if the corresponding pixel data of the target pixel in the second raw image data is valid and the corresponding pixel data of the target pixel in the third raw image data is invalid, calculating a depth value of the target pixel based on the corresponding pixel data of the target pixel in the second raw image data.

[0153] For each target pixel, if the corresponding pixel data of the target pixel in the second raw image data is invalid and the corresponding pixel data of the target pixel in the third raw image data is valid, calculating a depth value of the target pixel based on the corresponding pixel data of the target pixel in the third raw image data.

[0154] In one embodiment, the pixel fusion module 202 is further configured to:

[0155] For each target pixel, if the pixel data corresponding to the target pixel in the first original image data, the second original image data, and the third original image data are all invalid, then the depth value of the target pixel is set to a preset invalid value.

[0156] In one embodiment, the depth image generation system 20 further includes a validity determination module 204, used for:

[0157] For each target pixel, based on the amplitude of the four-phase raw data of the pixel corresponding to the target pixel in the original image data and a preset exposure threshold, it is determined whether the pixel corresponding to the target pixel in the original image data has effective saturation, and based on the signal amplitude of the pixel corresponding to the target pixel in the original image data and a preset amplitude threshold, it is determined whether the pixel corresponding to the target pixel in the original image data has effective signal-to-noise ratio.

[0158] If the pixel corresponding to the target pixel in the original image data has effective saturation and effective signal-to-noise ratio, the pixel data corresponding to the target pixel in the original image data is determined to be valid.

[0159] In applications, the modules in a depth image generation system can be software program modules, or they can be implemented through different logic circuits integrated in a processor, or they can be implemented through multiple distributed processors.

[0160] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 3 As shown, the computer device 3 of this embodiment includes: at least one processor 30 ( Figure 3 (Only one is shown) a processor, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, which, when executing the computer program 32, implements the steps in any of the above-described depth image generation method embodiments.

[0161] The computer device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0162] The processor 30 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0163] The memory 31 can be an internal storage unit of the computer device 3 in some embodiments, for example, a hard disk or a memory of the computer device 3. The memory 31 can also be an external storage device of the computer device 3 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 31 can include both the internal storage unit and the external storage device of the computer device 3. The memory 31 is used to store an operating system, an application program, a boot loader, data and other programs, for example, program codes of the computer program, etc. The memory 31 can also be used to temporarily store data that has been output or is to be output.

[0164] It should be noted that the information interaction, execution process, etc. between the above apparatuses / units, since based on the same concept as the method embodiments of the present application, the specific functions and the brought technical effects can be referred to the method embodiments part, and will not be described here in detail.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0166] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps in each method embodiment.

[0167] As shown in Figure 4 The embodiment of the present application provides a computer program product 3, which includes a computer program 32, and the computer program 32 is executed to make the depth image generation method as described above be executed.

[0168] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the embodiment of the present application realizes all or part of the processes in the above-mentioned method, which can be completed by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. The computer program is executed by a processor to realize the steps in each method embodiment. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the device / computer equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0169] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0170] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0171] In the embodiments provided in the present application, it should be understood that the disclosed computer device and method can be implemented in other ways. For example, the computer device embodiments described above are only schematic. The division of the modules or units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0172] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0173] The above described embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of generating a depth image, the method comprising: The method comprises: collecting multiple groups of original image data of a target scene based on a dual-dimensional heterogeneous data collection strategy; performing pixel-level fusion based on the multiple groups of original image data to obtain a depth value of each target pixel; constructing a depth image of the target scene based on the depth value of each target pixel.

2. The depth image generating method of claim 1, wherein, The dual-dimensional heterogeneous data collection strategy comprises a first collection strategy, a second collection strategy and a third collection strategy, the first collection strategy is a first modulation frequency combined with a first exposure time, the second collection strategy is a second modulation frequency combined with a second exposure time, and the third collection strategy is a third modulation frequency combined with a third exposure time, the first modulation frequency is greater than the second modulation frequency, the second modulation frequency is greater than the third modulation frequency, the first exposure time is greater than the second exposure time, and the second exposure time is greater than the third exposure time. The collecting multiple groups of original image data of a target scene based on a dual-dimensional heterogeneous data collection strategy comprises: collecting first original image data of a target scene based on the first collection strategy; collecting second original image data of a target scene based on the second collection strategy; collecting third original image data of a target scene based on the third collection strategy.

3. The depth image generation method of claim 2, wherein, The performing pixel-level fusion based on the multiple groups of original image data to obtain a depth value of each target pixel comprises: for each target pixel, if the corresponding pixel data of the target pixel in the first original image data, the second original image data and the third original image data are all valid, calculating a first depth value and a first wrapping residual error of the target pixel based on the corresponding pixel data of the target pixel in the first original image data and the second original image data, and calculating a second depth value and a second wrapping residual error of the target pixel based on the corresponding pixel data of the target pixel in the second original image data and the third original image data; if the first wrapping residual error is not greater than the second wrapping residual error, determining the first depth value as the depth value of the target pixel; if the first wrapping residual error is greater than the second wrapping residual error, determining the second depth value as the depth value of the target pixel.

4. The depth image generating method of claim 3, wherein, The performing pixel-level fusion based on the multiple groups of original image data to obtain a depth value of each target pixel further comprises: for each target pixel, if the corresponding pixel data of the target pixel in the first original image data is invalid, if the corresponding pixel data of the target pixel in the second original image data and the corresponding pixel data of the target pixel in the third original image data are both valid, calculating the depth value of the target pixel based on the corresponding pixel data of the target pixel in the second original image data and the third original image data.

5. The depth image generation method of claim 4, wherein, The performing pixel-level fusion based on the multiple groups of original image data to obtain a depth value of each target pixel further comprises: if the pixel data corresponding to the target pixel in the second raw image data is valid and the pixel data corresponding to the target pixel in the third raw image data is invalid, calculating the depth value of the target pixel based on the pixel data corresponding to the target pixel in the second raw image data; if the pixel data corresponding to the target pixel in the second raw image data is invalid and the pixel data corresponding to the target pixel in the third raw image data is valid, calculating the depth value of the target pixel based on the pixel data corresponding to the target pixel in the third raw image data.

6. The depth image generation method of claim 5, wherein, The pixel-level fusion based on the multiple sets of raw image data to obtain the depth value of each target pixel further comprises: if the pixel data corresponding to the target pixel in the first raw image data, the second raw image data and the third raw image data are all invalid, setting the depth value of the target pixel as a preset invalid value.

7. The depth image generation method of any one of claims 1 to 6, wherein, The method further comprises: for each target pixel, determining whether the pixel corresponding to the target pixel in the raw image data has valid saturation according to the amplitude of the four-phase raw data of the pixel corresponding to the target pixel in the raw image data and a preset exposure threshold, and determining whether the pixel corresponding to the target pixel in the raw image data has valid signal-to-noise ratio according to the signal amplitude of the pixel corresponding to the target pixel in the raw image data and a preset amplitude threshold; in the case that the pixel corresponding to the target pixel in the raw image data has valid saturation and valid signal-to-noise ratio, determining that the pixel data corresponding to the target pixel in the raw image data is valid.

8. A depth image generation system characterized by, comprise: a data acquisition module, configured to acquire multiple sets of raw image data of a target scene based on a dual-dimensional heterogeneous data acquisition strategy; a pixel fusion module, configured to perform pixel-level fusion based on the multiple sets of raw image data to obtain a depth value of each target pixel; a depth image construction module, configured to construct a depth image of the target scene based on the depth value of each target pixel.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the depth image generation method according to any one of claims 1 to 7.

10. A computer program product, characterised in that, comprise a computer program, which, when executed, causes the depth image generation method according to any one of claims 1 to 7 to be performed.

Images