Processing method of depth fusion error and ifof camera

Abstract

The application provides a processing method of depth fusion error, applied to an iToF camera, comprising: determining a plurality of connected regions in a to-be-processed depth image; the to-be-processed depth image is obtained by image acquisition on a collection object based on a target frequency combination, the target frequency combination comprising two different frequencies; determining physical features of a boundary region in the plurality of connected regions and physical features of an adjacent region corresponding to the boundary region; the boundary region and the adjacent region corresponding to the boundary region belong to different connected regions; the physical features comprise at least one of an amplitude, a depth value and a pixel point number; based on the physical features of the boundary region in each connected region and the physical features of the adjacent region corresponding to the boundary region, a to-be-corrected fusion error region is determined in the plurality of connected regions; the depth value of the fusion error region is corrected to obtain a corrected fusion error region. The application can solve the problem of depth calculation error caused by phase unwrapping.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a method for processing depth fusion errors and an iToF camera. Background Technology

[0002] Indirect Time-of-Flight (iToF) is a technology that uses two light waves with different modulation frequencies to measure the phase difference between the modulated and reflected light signals. The low-frequency phase measurement determines the number of cycles an object is in, while the high-frequency measurement provides a finer-grained phase, allowing for the calculation of the final distance. iToF is widely used in smartphone 3D sensing, robot navigation, and autonomous driving due to its low cost, high integration, and high resolution.

[0003] For dual-frequency iToF cameras, phase unwrapping is required to fuse high- and low-frequency data. The low-frequency data is used to calculate the number of extended periods, while the high-frequency data is used to calculate the precise distance, thus recovering the true distance (i.e., depth value). However, due to low-frequency measurement errors and other factors, the final fused distance calculation may be incorrect. Therefore, a method to effectively solve the distance calculation errors caused by phase unwrapping is urgently needed. Summary of the Invention

[0004] This application provides a method for processing depth fusion errors and an iToF camera, which can solve the problem of depth calculation errors caused by phase unfolding.

[0005] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a method for processing depth fusion errors, applied to an iToF camera. The method includes: determining multiple connected regions in a depth image to be processed; the depth image to be processed is obtained by image acquisition of an object based on a target frequency combination, the target frequency combination including two different frequencies; determining the physical features of the boundary regions and the physical features of the adjacent regions corresponding to the boundary regions in the multiple connected regions; the boundary regions and their corresponding adjacent regions belong to different connected regions; the physical features include at least one of amplitude, depth value, and number of pixels; determining a fusion region to be corrected in the multiple connected regions based on the physical features of the boundary regions and the physical features of the adjacent regions corresponding to the boundary regions in each connected region; and correcting the depth value of the fusion region to obtain the corrected fusion region.

[0006] In this embodiment, the connected regions of the depth map acquired by the iToF camera can be segmented, and the fault-fusion regions can be determined based on the physical characteristics of adjacent connected regions (such as the amplitude of pixels, the depth value of pixels, and the number of pixels). This allows for the correction of fault-fusion regions with frequency fusion errors in the event of a high-frequency spread period calculation error, thereby solving the problem of distance and depth errors caused by high-frequency spread period calculation errors.

[0007] In some implementations, the target frequency combination includes a first target frequency and a second target frequency. The target frequency combination is determined as follows: by simulating and calculating the frequency errors of the first frequency and / or the second frequency in multiple frequency combinations, the fusion range corresponding to each frequency combination and the minimum frequency error when deep fusion error occurs are obtained; the first frequency is higher than the second frequency; the first frequency includes the first target frequency, and the second frequency includes the second target frequency; a first frequency combination with a fusion range greater than a preset maximum measurement distance is selected from multiple frequency combinations; a second frequency combination with a first frequency higher than a preset frequency is selected from the first frequency combination; and a target frequency combination with a minimum frequency error of the maximum value is selected from the second frequency combination.

[0008] In this embodiment of the application, by simulating the error boundaries of different frequency combinations, the frequency selection method can be determined, thereby enabling the dual-frequency iToF camera to meet the application scenario requirements while reducing the probability of deep fault-finding problems.

[0009] In some implementations, the depth image to be processed includes multiple pixels; determining multiple connected regions in the depth image to be processed includes: determining multiple connected regions in the depth image to be processed based on the depth values ​​of each pixel in the depth image to be processed; the difference in fused depth values ​​of adjacent pixels in each connected region is less than a first preset difference.

[0010] In this embodiment, the depth map can be segmented into regions based on the difference in depth values ​​between pixels, thereby improving the accuracy of connected region segmentation and facilitating subsequent error fusion judgment based on the boundary regions of connected regions.

[0011] In some implementations, determining multiple connected regions in the depth image to be processed based on the depth values ​​of each pixel in the depth image to be processed includes: determining the initial label of each pixel in the depth image to be processed, and traversing multiple pixels in a preset order; the initial labels of each pixel in the depth image to be processed are the same; based on the depth value difference between the current pixel and its corresponding traversed neighboring points, determining target neighboring points belonging to the same connected region as the current pixel among the traversed neighboring points corresponding to the current pixel; the multiple pixels include the current pixel; updating the initial label of the current pixel based on the number of target neighboring points to obtain the updated label of the current pixel; determining multiple connected regions based on the updated labels of each pixel; and multiple pixels in each connected region having the same updated label.

[0012] In this embodiment, by traversing the pixels in the fusion depth map and based on the difference in fusion depth values ​​between the current pixel and its corresponding traversed neighboring points, accurate division of connected regions can be achieved, thereby improving the accuracy of subsequent fault-breaking jump point judgment.

[0013] In some implementations, the initial label of the current pixel is updated based on the number of target neighboring points to obtain the updated label of the current pixel, including: if there are no target neighboring points, determining the updated label of the current pixel as the first label; the first label of the current pixel is different from the updated labels of the target neighboring points; if there is only one target neighboring point, determining the updated label of the current pixel as the updated label of the target neighboring point; if there are multiple target neighboring points, determining the updated label of the current pixel as the smallest label among the updated labels of the multiple target neighboring points.

[0014] In this embodiment, the label of the current pixel can be updated based on the number of target neighboring points whose fusion depth difference with the current pixel is less than a threshold, thereby accurately dividing the connected regions based on the label values ​​of each pixel.

[0015] In some implementations, based on the physical characteristics of the boundary regions and the physical characteristics of the adjacent regions corresponding to the boundary regions, the fault-fusion region to be corrected is determined in multiple connected regions. This includes: determining the fault-fusion jump point caused by the frequency spreading period calculation error in each boundary region based on the amplitude of the neighboring points corresponding to the boundary points in each boundary region, and the depth value difference between the boundary points and their corresponding neighboring points in each boundary region; the adjacent regions corresponding to the boundary regions include the neighboring points corresponding to the boundary points; the depth value difference includes the depth value difference of dual-frequency fusion and / or the depth value difference of single-frequency fusion; determining the fault-fusion region in multiple connected regions based on the number of fault-fusion jump points in each boundary region; and the number of pixels includes the number of fault-fusion jump points.

[0016] In this embodiment, the difference between the amplitude of the neighboring point, the difference between the single-frequency depth value between the neighboring point and the current boundary point, and the difference between the fusion depth value between the neighboring point and the current boundary point can be used to determine whether the current boundary point is a fault-fusion jump point from multiple perspectives, thereby improving the accuracy of judging the fault-fusion jump point.

[0017] In some implementations, determining a fault-finding region in multiple connected regions based on the number of fault-finding jump points in each boundary region includes: determining candidate fault-finding regions in multiple connected regions based on the number of fault-finding jump points in each connected region; determining the fault-finding probability of each candidate fault-finding region exhibiting a fault-finding phenomenon; sequentially traversing multiple candidate fault-finding regions according to the magnitude of the fault-finding probability of each candidate fault-finding region; and determining whether the current candidate region is a fault-finding region based on the number of fault-finding jump points in the current candidate region.

[0018] In this embodiment of the application, it is possible to determine whether a candidate fault-free region is a fault-free region by traversing the candidate fault-free region and based on the number of fault-free jump points in the candidate fault-free region. That is, it is possible to accurately determine which side of the region located on both sides of the fault-free jump boundary is the fault-free region.

[0019] In some implementations, the depth value of the fault-tolerant region is corrected to obtain a corrected fault-tolerant region, including: determining an expansion period correction amount based on the depth value difference between the fault-tolerant transition point in the fault-tolerant region and the neighboring points corresponding to the fault-tolerant transition point; and correcting the depth values ​​of multiple pixels in the fault-tolerant region based on the expansion period correction amount to obtain the corrected fault-tolerant region.

[0020] In this embodiment, the depth correction amount of the extended period can be accurately determined based on the depth value difference between the fault-breaking jump point and the neighboring points corresponding to the fault-breaking jump point, so as to correct the depth of the fault-breaking region and ensure the correction effect of the fusion depth.

[0021] In some implementations, the method further includes: obtaining a region to be filtered out from a plurality of connected regions; the surrounding region of the region to be filtered out is a non-error-correction region or an invalid region; when the size of the region to be filtered out is less than a preset threshold and the difference in depth values ​​between the boundary points in the region to be filtered out and the neighboring points corresponding to the boundary points is greater than a preset difference, the depth values ​​of the plurality of pixels in the region to be filtered out are filtered out.

[0022] In this embodiment, the fusion region in the depth map acquired by the dual-frequency iToF camera can be effectively judged and filtered out, thereby more effectively solving the fusion phenomenon caused by the phase unfolding problem of the dual-frequency iToF camera, and making the corrected fusion region more accurately reflect the depth of the object in the actual environment.

[0023] Secondly, embodiments of this application provide an iToF camera, which includes a processor for implementing the depth fusion error processing method of the first aspect.

[0024] Thirdly, embodiments of this application provide a deep fusion error processing apparatus, which includes: a first determining module, a second determining module, a third determining module, and a correction module; wherein, the first determining module is configured to determine multiple connected regions in a depth image to be processed; the depth image to be processed is obtained by image acquisition of an object based on a target frequency combination, the target frequency combination including two different frequencies; the second determining module is configured to determine the physical features of the boundary regions and the physical features of the adjacent regions corresponding to the boundary regions in the multiple connected regions; the boundary regions and the adjacent regions corresponding to the boundary regions belong to different connected regions; the physical features include at least one of amplitude, depth value, and number of pixels; the third determining module is configured to determine the fusion error region to be corrected in the multiple connected regions based on the physical features of the boundary regions and the physical features of the adjacent regions corresponding to the boundary regions in each connected region; the correction module is configured to correct the depth value of the fusion error region to obtain the corrected fusion error region.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the method for processing deep fusion errors.

[0026] Fifthly, embodiments of this application provide a program product that includes a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement some or all of the steps in a method for processing deep fusion errors.

[0027] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0028] Figure 1 A schematic diagram illustrating an error in calculating the frequency fusion extension period, provided in an embodiment of this application; Figure 2 A schematic diagram of a frequency fusion error provided in an embodiment of this application; Figure 3 A method for processing deep fusion errors provided in this application embodiment; Figure 4 A schematic diagram of a fault-tolerant region provided in an embodiment of this application; Figure 5 A schematic diagram of the interface of an iToF dual-frequency ranging calculator software provided in an embodiment of this application; Figure 6 A schematic diagram of a fault-tolerant simulation provided in an embodiment of this application; Figure 7 A schematic diagram illustrating another fault-tolerant simulation provided in an embodiment of this application; Figure 8 This application provides a schematic diagram of a traversed neighborhood point as an embodiment of the present application. Figure 9 A schematic diagram of a connected region partitioning result provided in an embodiment of this application. Figure 1 ; Figure 10 A schematic diagram of a connected region partitioning result provided in an embodiment of this application. Figure 2 ; Figure 11 A schematic diagram of a fault-tolerant transition point provided in an embodiment of this application; Figure 12 A schematic diagram illustrating the error correction effect provided in an embodiment of this application; Figure 13 A flowchart illustrating the implementation of a method for determining abnormal boundary transitions in a region, as provided in this application embodiment; Figure 14 A flowchart illustrating the implementation of a fault-tolerant region determination method provided in this application embodiment; Figure 15 A schematic diagram of the composition structure of a deep fusion error processing device provided in an embodiment of this application; Figure 16 This is a schematic diagram of the structure of an iToF camera provided in an embodiment of this application.

[0029] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0033] In related technologies, for dual-frequency iToF cameras, the low-frequency period has a large range and low accuracy, while the high-frequency period has a small range and high accuracy. Therefore, it is necessary to fuse the image depth obtained from the high and low frequencies separately to recover the true distance. That is, the number of extended periods of the high frequency is determined based on the low-frequency phase, and the high-frequency phase is restored (i.e., phase unwrapping) based on the number of extended periods of the high frequency. Thus, the accurate distance is calculated based on the high-frequency period after phase unwrapping.

[0034] Since, under ideal conditions, the distances measured at high and low frequencies are equal, the number of extended cycles corresponding to the high frequencies can be calculated. Finally, the unwrapped distance is obtained. The method for determining the unwrapped distance can be referred to as formula (1): (1); in, Indicates the distance after unfolding. For high frequency, Low frequency and These represent the phases corresponding to high and low frequencies, respectively. and These represent the number of extended cycles corresponding to high frequency and low frequency, respectively.

[0035] The main steps of dual-frequency fusion are as follows: 1. Calculate the maximum number of extended cycles for low frequencies based on the preset maximum ranging distance. ; 2. Traverse the possible number of low-frequency extended cycles. ( ), and calculate the number of high-frequency spread cycles. Determine the number of high-frequency spread cycles. The method can be referred to as shown in formula (2): (2); 3. Based on high-frequency extended cycle number and low-frequency extended cycle number Calculate fusion error ,make Then we can obtain formula (3): (3); Among them, fusion error This indicates that the high and low frequencies are respectively represented by the corresponding extended cycle numbers. and When the distance difference is obtained, the high-frequency cycle number multiple is obtained (it is a floating-point number and may contain a decimal part).

[0036] 4. Calculate the number of all possible high-frequency extended cycles. In the middle, the fusion error is reduced. Get the minimum number of low-frequency extended cycles .

[0037] 5. The number of low-frequency extended cycles can be calculated based on formula (2). Corresponding high-frequency extended cycle number The distance after phase expansion at high frequencies is calculated as the final fusion distance. The method for determining the fusion distance can be found in formula (4): (4); in, Indicates the fusion distance.

[0038] However, when the above algorithm has significant errors in the distances obtained based on high and low frequencies respectively, it may lead to an incorrect number of expansion cycles when determining the minimum fusion error. (Reference) Figure 1 The diagram shown illustrates an error in the calculation of the frequency fusion extension period. Figure 1 As shown, This represents the low-frequency phase calculated by the camera. This represents the high-frequency phase calculated by the camera; each grid represents 2. The phase of the high-frequency range is one expansion period. Understandably, because the high-frequency range has higher accuracy, the final range measurement result uses the phase corresponding to the high-frequency range, while the low-frequency range is used to determine the number of expansion periods of the high-frequency range. Therefore, the measurement result will only appear at integer multiples of the expansion period in the high-frequency range. Normally, the distance detected by the camera should be at point A, because the fusion error of the two frequencies is minimized at this point. However, due to measurement errors in the low-frequency range (high-frequency range also has errors, but these are generally smaller than those in the low-frequency range; here we assume only the low-frequency error is considered), the calculated low-frequency phase is... At this point, the fusion error corresponding to position B is the smallest, which leads to an erroneous result where the measured position differs from the correct position by approximately one low-frequency range. (Reference) Figure 2 The diagram shown illustrates a frequency fusion error, as follows: Figure 2 As shown, Figure 2 The image on the left is the depth map, and the image on the right is the corresponding amplitude map. The arrow on the left-hand depth map indicates the area where the distance calculation is incorrect, indicating a frequency fusion error.

[0039] Traditional methods for solving the phase unfolding problem of dual-frequency iToF cameras fail to explore how to reduce the probability of phase unfolding errors from the perspective of frequency combination optimization. Instead, they only calculate the spread period from image data analysis, or require the fulfillment of preconditions that may not be met in reality, such as zero-curl constraint, or use the proportional relationship between radiated power and the square of the distance and combine it with amplitude maps for judgment, while ignoring the impact of different surface reflectivities of different objects on the amplitude.

[0040] Based on the above-mentioned technical problems, this application provides a method for processing deep fusion errors and an iToF camera, which can solve the problem of distance calculation errors caused by phase unfolding.

[0041] Figure 3 A method for processing deep fusion errors provided in this application embodiment can be executed by a processor (such as an ARM processor) in an iToF camera (such as a dual-frequency iToF camera). Figure 3 As shown, the method includes S301 to S304: S301, Identify multiple connected regions in the depth image to be processed.

[0042] In some implementations, when a dual-frequency iToF camera acquires images, it can send high-frequency and low-frequency light signals to the target object in the environment through its light-emitting module, and receive the light signals returned from the environment to obtain the phase of the high-frequency light signal and the phase of the low-frequency light signal. Based on the phase of the low-frequency light signal, the phase of the high-frequency light signal is expanded to determine the depth image to be processed. The depth image to be processed represents the dual-frequency fused depth value (hereinafter referred to as the fused depth value) corresponding to each pixel in the image.

[0043] For example, the light-emitting module may store a target frequency combination, which includes two different frequencies: a high-frequency frequency (hereinafter referred to as the first target frequency) and a low-frequency frequency (hereinafter referred to as the second target frequency). When the dual-frequency iToF camera performs image acquisition, it can use the light signal of the target frequency combination stored in the light-emitting module to perform image acquisition in order to determine the fusion depth value of each pixel in the depth image to be processed.

[0044] For example, after determining the depth image to be processed, the depth image to be processed can be segmented into regions based on the differences between the fused depth values ​​of each pixel in the depth image to be processed, so as to determine multiple connected regions in the depth image to be processed.

[0045] S302, determine the physical characteristics of the boundary regions and the physical characteristics of the adjacent regions corresponding to the boundary regions in multiple connected regions.

[0046] refer to Figure 4 The diagram shown is a schematic of a fault-tolerant region, such as Figure 4 As shown, after determining multiple connected regions in the fusion depth map, there may be connected regions (hereinafter referred to as fusion error regions) that experience fusion errors due to phase expansion calculation errors (i.e., errors in the calculation of the extended period of high-frequency frequencies). Figure 4 The region in blue. Therefore, the physical characteristics of adjacent connected regions can be determined, and based on the physical characteristics of adjacent connected regions, it can be determined whether there are fault-melting regions to be corrected in multiple connected regions.

[0047] For example, the physical features of adjacent connected regions can be the physical features of the boundary regions among multiple connected regions, and the physical features of the adjacent regions corresponding to each boundary region. Here, the boundary region and its corresponding adjacent region belong to different connected regions; the adjacent regions corresponding to the boundary region can include 4-neighbor points or 8-neighbor points corresponding to each pixel point (hereinafter referred to as boundary point) in the boundary region. The physical features can include at least one of the pixel magnitude, pixel depth value, and pixel count.

[0048] S303, Based on the physical characteristics of the boundary regions and the physical characteristics of the adjacent regions corresponding to the boundary regions in each connected region, determine the fault-melting regions to be corrected in multiple connected regions.

[0049] In some implementations, after determining multiple connected regions in the fusion depth map, the number of neighboring points (hereinafter referred to as first neighboring points) that meet preset conditions can be determined based on the physical characteristics of the boundary regions and their adjacent regions in each connected region, such as the amplitude, fusion depth value, and single-frequency depth value of the neighboring points that do not belong to the current connected region corresponding to the boundary points in each boundary region. The preset conditions may include at least one of the following: the amplitude of a neighboring point is greater than a preset amplitude; the difference in single-frequency depth values ​​between a neighboring point and the current boundary point is less than a preset difference (such as a second preset difference); the difference in fusion depth values ​​between a neighboring point and the current boundary point is greater than a preset difference (such as a third preset difference); and the third preset difference is close to the high-frequency range corresponding to the target frequency combination. The single-frequency depth value includes high-frequency depth values ​​and / or low-frequency depth values; the single-frequency depth value is the depth value corresponding to a single frequency before deep fusion.

[0050] If the number of first neighboring points corresponding to a boundary point in a boundary region is greater than a first preset number, the boundary point can be identified as a fault-melting jump point in the connected region corresponding to the boundary region; and if the number of fault-melting jump points in the connected region is greater than a second preset number, the connected region is identified as a fault-melting region to be corrected.

[0051] S304 corrects the depth value of the fault-tolerant region to obtain the corrected fault-tolerant region.

[0052] In some implementations, after determining the fault-melting region among multiple connected regions, the depth value of each pixel in the fault-melting region can be corrected to minimize the phase unwrapping error corresponding to the corrected pixel.

[0053] In this embodiment, the connected regions of the depth map acquired by the iToF camera can be segmented, and the fault-fusion regions can be determined based on the physical characteristics of adjacent connected regions (such as the amplitude of pixels, the depth value of pixels, and the number of pixels). This allows for the correction of fault-fusion regions with frequency fusion errors in the event of a high-frequency spread period calculation error, thereby solving the problem of distance calculation errors caused by high-frequency spread period calculation errors.

[0054] In some implementations, the target frequency combination includes a first target frequency and a second target frequency. The target frequency combination can be determined as follows: by simulating and calculating the frequency errors of the first frequency and / or the second frequency in multiple frequency combinations, the fusion range corresponding to each frequency combination and the minimum frequency error when deep fusion error occurs are obtained; the first frequency is higher than the second frequency; the first frequency includes the first target frequency, and the second frequency includes the second target frequency; from multiple frequency combinations, a first frequency combination with a fusion range greater than a preset maximum measurement distance is selected; from the first frequency combination, a second frequency combination with a first frequency higher than a preset frequency is selected; from the second frequency combination, a target frequency combination with a minimum frequency error equal to the maximum value is selected.

[0055] For example, for a dual-frequency iToF camera, the high-frequency and low-frequency frequencies need to be selected based on the range and accuracy requirements. Higher frequencies can achieve higher measurement accuracy, but the corresponding range is smaller and more prone to error fusion problems.

[0056] In some implementations, embodiments of this application provide iToF dual-frequency ranging calculator software to simulate the relationship between high and low frequencies and the occurrence of error fusion. (Reference) Figure 5 The diagram shown is an interface illustration of an iToF dual-frequency ranging calculator software. Figure 5 As shown, this software can simulate fault-fusing scenarios corresponding to different high and low frequency errors, such as... Figure 5 As shown, assuming the low-frequency is 17.679MHz, the corresponding single-cycle range is 8484mm, and the high-frequency is 99.000MHz, the corresponding single-cycle range is 1515mm. When the actual distance is 5000mm, the distance difference between the low-frequency at a spread period of 0 (N=0) and the high-frequency at a spread period of 3 (N=3) is minimal (ideally, the difference is 0). Therefore, the correct number of high-frequency spread periods is N=3, and thus the correct distance of 5000mm can be calculated.

[0057] refer to Figure 6 The diagram shown is a schematic of a fault-tolerant simulation, as follows: Figure 6 As shown, when the low frequency has an error of -1.8% (i.e., the phase shifts to the left by -1.8%) (as shown by the yellow dashed line in the figure), the distance difference between the low frequency at the extension period of 2 (N=2) and the high frequency at the extension period of 14 (N=14) is the smallest. At this time, the number of high frequency extension periods is N=14, and the calculated error distance is 21665mm.

[0058] refer to Figure 7 Another schematic diagram of fault-tolerant simulation is shown, such as Figure 7As shown, when the low frequency has an error of +1.8% (i.e., the phase shifts to the right by -1.8%) (as shown by the yellow dashed line in the figure), the distance difference between the low frequency at the extension period of 3 (N=2) and the high frequency at the extension period of 20 (N=20) is the smallest. At this time, the number of high frequency extension periods is N=20, and the calculated incorrect distance is 30755mm.

[0059] The above illustrates the impact of individual frequency errors on phase expansion calculations for a given high-low frequency combination. Therefore, this can be used as a basis to simulate the minimum single-frequency error when error fusion occurs in different frequency combinations, allowing us to summarize the patterns and derive the basic principles for frequency selection.

[0060] For example, the minimum frequency error corresponding to different high and low frequency combinations can be simulated using iToF dual-frequency ranging calculator software to determine at what minimum frequency error (including both positive and negative directions) corresponding to the first or second frequency will cause the error fusion phenomenon to occur, and the minimum frequency error at which the error fusion phenomenon occurs is recorded as the error boundary corresponding to each frequency combination (such as the absolute value of the percentage of the minimum frequency error).

[0061] For example, we can assume that only low-frequency errors exist, and simulate the minimum low-frequency error (including both positive and negative directions) required for error fusion to occur in different high- and low-frequency combinations. The minimum low-frequency error at which error fusion occurs can be recorded as the error boundary for each frequency combination. In the example above, the error boundary for the low-frequency combination of a high frequency of 99.000MHz and a low frequency of 17.679MHz is 1.8%. The following table, based on an iToF (Continuous Wave Modulation iToF, CW-iToF) camera and compiled by iToF dual-frequency ranging calculator software, shows the error boundaries for multiple frequency combinations: Table 1

[0062] Among them, the low-frequency range refers to the distance corresponding to a low-frequency cycle calculated based on the low-frequency frequency, and the fusion range refers to the distance corresponding to the maximum extended cycle without ambiguity, i.e., the unambiguous range.

[0063] In some examples, the low-frequency range can be determined by referring to formula (5): (5); in, For low-frequency range, c is the speed of light. Indicates high frequency. Indicates low frequency.

[0064] In some examples, the fusion range (i.e., the unambiguous distance) can be determined by referring to formula (6): (6); in, To integrate the measurement range, Used to calculate the greatest common divisor of frequencies.

[0065] In some implementations, the following pattern emerges from Table 1: for frequency combinations with the same fusion range / low-frequency range, the larger the multiple of the high / low frequency, the smaller the fusion boundary, and the easier it is for fusion errors to occur. Therefore, among the aforementioned frequency combinations, a frequency combination with a larger error boundary can be identified, thereby reducing the probability of dual-frequency fusion errors when image acquisition is performed based on this frequency combination.

[0066] For example, based on the maximum measurement distance Dmax required for the application scenario, a first frequency combination with a fusion range greater than the maximum measurement distance Dmax can be determined from the aforementioned multiple frequency combinations. Then, since the final ranging accuracy is determined by the high-frequency frequency, a second frequency combination with a high-frequency frequency greater than a preset frequency can be determined from the first frequency combination based on the accuracy requirements of the application scenario. Next, since a larger error boundary makes error fusion less likely, a target frequency combination with the largest error boundary can be determined from the second frequency combination. If multiple frequency combinations with the largest error boundary are determined from the second frequency combination, any one of these frequency combinations can be determined as the target frequency combination.

[0067] In other words, the fusion range corresponding to the target frequency combination is greater than the farthest measurement distance, the first target frequency (i.e., the high-frequency frequency) is greater than the preset frequency, and the error boundary corresponding to the target frequency combination is the frequency combination with the largest error boundary among multiple frequency combinations. After determining the target frequency combination, the target frequency combination can be written into the light emission module of the iToF camera.

[0068] In this embodiment of the application, by simulating the error boundaries of different frequency combinations, the frequency selection method can be determined, thereby enabling the dual-frequency iToF camera to meet the application scenario requirements while reducing the probability of deep fault-finding problems.

[0069] In some implementations, S301 includes: determining multiple connected regions in the depth image to be processed based on the depth values ​​of each pixel in the depth image to be processed.

[0070] For example, the depth values ​​of each pixel in the depth image to be processed can be obtained, and multiple connected regions in the depth image to be processed can be determined based on the differences in depth values ​​between the pixels. For instance, adjacent pixels whose depth value difference is less than a first preset difference can be determined as pixels belonging to the same connected region. That is, if the depth value difference between adjacent pixels is less than the first preset difference, it can be determined that the adjacent pixels are connected, i.e., the depth value difference between adjacent pixels in each connected region is less than the first preset difference.

[0071] In this embodiment, the depth map can be segmented into regions based on the depth value differences between pixels, thereby improving the accuracy of connected region segmentation and facilitating subsequent error fusion judgment based on the boundary regions of connected regions.

[0072] In some implementations, S302 includes: determining the initial label of each pixel in the depth image to be processed, and traversing multiple pixels in a preset order; based on the depth value difference between the current pixel and its corresponding traversed neighboring points, determining target neighboring points that belong to the same connected region as the current pixel among the traversed neighboring points; updating the initial label of the current pixel based on the number of target neighboring points to obtain the updated label of the current pixel; and determining multiple connected regions based on the updated labels of each pixel.

[0073] For example, multiple connected regions in the depth map to be processed can be determined by traversing multiple pixels. Before traversing multiple pixels, the initial label of each pixel can be determined, and multiple label sets can be established to store different connected regions. The initial labels of all pixels are the same, such as 0; the initial labels of each pixel can be stored in the form of a label matrix, the size of which is the same as the size of the fused depth map.

[0074] Next, multiple pixels can be traversed for the first time according to a preset order. This preset order can be from top to bottom and from left to right. In the captured image, there may be objects that are too far away or have very low reflectivity, making it impossible to calculate their depth values. Therefore, if the fused depth value of the currently traversed pixel (hereinafter referred to as the current pixel) is invalid (e.g., 0 or an outlier), or if the amplitude is less than the preset amplitude (depth values ​​of points with excessively small amplitudes are unreliable), the current pixel can be skipped, and the traversal can continue to the next pixel.

[0075] If the fusion depth value of the current pixel is valid and its magnitude is greater than or equal to a preset magnitude, the traversed neighboring points among the 8 neighboring points of the current pixel can be obtained. (Reference) Figure 8 The diagram shown illustrates a scenario where neighboring points have been traversed. Figure 8As shown, with the preset order being top-to-bottom and left-to-right, the traversed neighboring points can include the four neighboring points of the current pixel: left, upper-left, upper, and upper-right. For example, the traversed neighboring points corresponding to pixel E can include pixels A, B, C, and D. Understandably, if the current pixel does not have a corresponding traversed neighboring point, the traversal continues to the next pixel.

[0076] Next, among the traversed neighboring points corresponding to the current pixel, target neighboring points whose fusion depth difference with the current pixel is less than a first preset difference can be determined. That is, for each traversed neighboring point corresponding to the current pixel, if |d1 If d2| < the first preset difference T, then the current pixel can be considered connected to the target neighboring pixel, meaning they belong to the same connected region. Here, d1 represents the fusion depth value of the current pixel, and d2 represents the fusion depth values ​​of the traversed neighboring pixels corresponding to the current pixel.

[0077] For example, the initial label of the current pixel can be updated based on the number of target neighbor points corresponding to the current pixel, resulting in an updated label for the current pixel. The updated label of the current pixel indicates whether the current pixel and its corresponding traversed neighbor points belong to the same connected region. If the number of target neighbor points is 0, it indicates that the current pixel and its corresponding traversed neighbor points do not belong to the same connected region; if the number of target neighbor points is greater than 0, it indicates that among its corresponding traversed neighbor points, there is a neighbor point that belongs to the same connected region as the current pixel. For example, if the number of target neighbor points is 0, the initial label of the current pixel can be updated to a label value different from the updated labels of its corresponding traversed neighbor points (hereinafter referred to as the first label). If the number of target neighbor points is greater than 0, the initial label of the current pixel can be updated to the updated labels of its corresponding traversed neighbor points.

[0078] For example, if the number of target neighbor points corresponding to the current pixel is greater than 0, the updated labels of the current pixel and the corresponding target neighbor points can be stored in the same label set to indicate that the current pixel and the corresponding target neighbor points belong to the same connected region.

[0079] For example, after the first traversal of all pixels, multiple connected regions can be determined based on the updated labels of each pixel, where pixels within each connected region have the same updated label. For instance, different label sets can be identified as different connected regions. (See reference) Figure 9 and Figure 10 The diagram shown illustrates a connected component partitioning result, as follows: Figure 9 As shown, based on the above embodiments, Figure 9 After dividing the fusion depth map into connected regions, the following can be obtained: Figure 10 The connected region partitioning result shown is the 8-neighbor connected region partitioning result that satisfies the condition that "the fusion depth difference between adjacent pixels is less than the first preset difference".

[0080] In this embodiment, by traversing the pixels in the fusion depth map and based on the difference in fusion depth values ​​between the current pixel and its corresponding traversed neighboring points, accurate division of connected regions can be achieved, thereby improving the accuracy of subsequent fault-breaking jump point judgment.

[0081] In some implementations, the initial label of the current pixel is updated based on the number of target neighboring points to obtain the updated label of the current pixel, including: if there are no target neighboring points, determining the updated label of the current pixel as the first label; if there is only one target neighboring point, determining the updated label of the current pixel as the updated label of the target neighboring point; if there are multiple target neighboring points, determining the updated label of the current pixel as the smallest label among the updated labels of the multiple target neighboring points.

[0082] For example, when updating the initial label of the current pixel based on the number of target neighbors, if the number of target neighbors is 0 (i.e., there are no target neighbors), a new label can be assigned to the current pixel, such as updating the initial label of the current pixel to a first label different from the updated labels of its corresponding traversed neighbors. If there is only one target neighbor, the current pixel can inherit the label of that target neighbor, that is, update the initial label of the current pixel to the updated label of that target neighbor. If there are multiple target neighbors, the updated label of the current pixel can be determined to be the smallest label among the updated labels of the multiple target neighbors.

[0083] After completing the first traversal of all pixels, a second traversal can be performed on multiple pixels in each label set to determine the minimum label of each pixel in the label set. Based on this minimum label, the updated labels of all pixels in the corresponding label set are then unified. In other words, the label values ​​of all pixels in a label set can be unified to the minimum label value in that label set.

[0084] In this embodiment, the label of the current pixel can be updated based on the number of target neighboring points whose fusion depth difference with the current pixel is less than a threshold, thereby accurately dividing the connected regions based on the label values ​​of each pixel.

[0085] In some implementations, S303 includes: determining a fault-fusing jump point caused by a frequency spreading period calculation error in each boundary region based on the amplitude of the neighboring points corresponding to the boundary points in each boundary region and the depth value difference between the boundary points and their corresponding neighboring points in each boundary region; the adjacent regions corresponding to the boundary regions include the neighboring points corresponding to the boundary points; the depth value difference includes the dual-frequency fusion depth value difference and / or the single-frequency depth value difference; determining a fault-fusing region in multiple connected regions based on the number of fault-fusing jump points in each boundary region; and the number of pixels includes the number of fault-fusing jump points.

[0086] For example, after identifying multiple connected regions in the fusion depth map, the boundary points within each connected region can be traversed to determine whether each boundary point is a fusion error jump point, i.e., whether the boundary point is a point that jumps due to an error in the expansion period calculation. Here, a boundary point refers to the outermost pixel within a connected region. (See reference...) Figure 11 The diagram shown illustrates a fault-tolerant transition point, as follows: Figure 11 As shown in the figure, for the area indicated by the arrow, since there is a significant difference between its single-frequency depth values ​​and those of its upper neighboring points in the single-frequency (high-frequency and low-frequency) depth map, this difference does not belong to a jump caused by an error in the expansion period calculation, but rather to a normal depth difference between the foreground and background. Therefore, the boundary points above it are not considered fault-melting boundary points. However, since the difference between the single-frequency depth values ​​of its lower neighboring points and those of this area is relatively small, it is necessary to further determine whether the lower neighboring points are fault-melting boundary points.

[0087] For example, when traversing multiple boundary points in the current connected region, the fusion depth value and single-frequency depth value of the neighboring points that do not belong to the current connected region corresponding to the current boundary point can be obtained. Then, it can be determined whether there are any neighboring points (hereinafter referred to as the first neighboring point) that meet preset conditions. The preset conditions may include at least one of the following: the amplitude of the neighboring point is greater than a preset amplitude; the difference in single-frequency depth values ​​between the neighboring point and the current boundary point is less than a second preset difference; the difference in fusion depth values ​​between the neighboring point and the current boundary point is greater than a third preset difference, and the third preset difference is close to the high-frequency range corresponding to the target frequency combination. The single-frequency depth value includes a high-frequency depth value and / or a low-frequency depth value; the single-frequency depth value is the depth value corresponding to a single frequency before deep fusion.

[0088] In some examples, for a single-frequency depth map, its spread period is always 0, and the depth difference corresponds to the phase difference, with phase 0 and phase 2 being different. For the same depth value, the method to determine the difference of single-frequency depth values ​​of pixels in a single-frequency depth map can be referred to as formula (7): (7); in, This represents the difference in single-frequency depth values ​​between pixels in a single-frequency depth map, where D represents the range corresponding to a single frequency. and These represent the single-frequency depth values ​​corresponding to two adjacent pixels.

[0089] For example, to avoid the influence of noise errors, the current boundary point can be determined as a fault-melting transition point of the current connected region if the number of its first neighboring points is greater than a first preset number; and if the number of its first neighboring points is less than or equal to the first preset number, the current boundary point can be determined as a non-fault-melting transition point of the current connected region. After traversing all boundary points of the current connected region, the next connected region can be traversed until all connected regions have been traversed to determine the fault-melting transition points in each connected region.

[0090] In this embodiment, the difference between the amplitude of the neighboring point, the difference between the single-frequency depth value between the neighboring point and the current boundary point, and the difference between the fusion depth value between the neighboring point and the current boundary point can be used to determine whether the current boundary point is a fault-fusion jump point from multiple perspectives, thereby improving the accuracy of judging the fault-fusion jump point.

[0091] In some implementations, determining a fault-finding region in multiple connected regions based on the number of fault-finding jump points in each boundary region includes: determining candidate fault-finding regions in multiple connected regions based on the number of fault-finding jump points in each connected region; determining the fault-finding probability of each candidate fault-finding region exhibiting a fault-finding phenomenon; sequentially traversing multiple candidate fault-finding regions according to the magnitude of the fault-finding probability of each candidate fault-finding region; and determining whether the current candidate region is a fault-finding region based on the number of fault-finding jump points in the current candidate region.

[0092] For example, after determining the fault-free transition points of each connected region, it can be further determined whether each connected region is a fault-free region that needs to be corrected based on the number of fault-free transition points of each connected region.

[0093] For example, if the number of fault-fusing transition points in the current connected region is greater than a second preset number, the current connected region can be identified as a candidate fault-fusing region. Then, the fault-fusing probability of each candidate fault-fusing region occurring can be determined. In some examples, the fault-fusing probability of each candidate fault-fusing region occurring can be determined based on the total number of pixels in each candidate fault-fusing region. Understandably, for a properly calibrated iToF camera, the occurrence of frequency fusion errors can be considered a low-probability event; that is, the depth values ​​of most pixels in the depth image are normal, and small connected regions are more likely to be fault-fusing regions. Therefore, candidate fault-fusing regions can be sorted from smallest to largest according to the number of pixels, and each candidate fault-fusing region can be traversed sequentially to determine whether each candidate fault-fusing region is a fault-fusing region.

[0094] During the traversal of candidate fault-tolerant regions, the number of fault-tolerant jump points in the current candidate region can be used to determine whether the current candidate region is a fault-tolerant region. In some examples, the current candidate region can be determined as a fault-tolerant region if the number of fault-tolerant jump points in the current candidate region is greater than a preset number (such as a third preset number); and as a non-fault-tolerant region if the number of fault-tolerant jump points in the current candidate region is less than or equal to the third preset number. Alternatively, the proportion of fault-tolerant jump points in the current candidate region among the boundary points of the current candidate region can be calculated, and the current candidate region can be determined as a fault-tolerant region if the proportion of fault-tolerant jump points in the current candidate region is greater than a preset ratio; and as a non-fault-tolerant region if the proportion of fault-tolerant jump points in the current candidate region is less than or equal to the preset ratio. The proportion of fault-tolerant jump points can be determined based on the ratio of the number of fault-tolerant jump points in the candidate fault-tolerant region to the number of boundary points of the candidate fault-tolerant region. It should be noted that the above method of determining whether the current candidate region is a fault-tolerant region based on the number of fault-tolerant jump points in the current candidate region is only an example, and this embodiment of the application does not limit it.

[0095] In this embodiment of the application, it is possible to determine whether a candidate fault-free region is a fault-free region by traversing the candidate fault-free region and based on the number of fault-free jump points in the candidate fault-free region. That is, it is possible to accurately determine which side of the region located on both sides of the fault-free jump boundary is the fault-free region.

[0096] In some implementations, S304 includes: determining an extended period correction amount based on the depth value difference between the fault-breaking jump point in the fault-breaking region and the neighboring points corresponding to the fault-breaking jump point; and correcting the depth values ​​of multiple pixels in the fault-breaking region based on the extended period correction amount to obtain the corrected fault-breaking region.

[0097] For example, after determining the fault-matching region, the expansion period of each pixel in the fault-matching region can be corrected to minimize the depth jump of the corrected boundary points. In some examples, the average fusion depth difference between multiple fault-matching jump points in the fault-matching region and their corresponding neighboring points can be calculated, and the expansion period difference (i.e., the expansion period correction amount) can be calculated based on the average fusion depth difference. The fusion depth value of each pixel in the fault-matching region can then be corrected based on the expansion period correction amount to obtain the corrected fault-matching region.

[0098] For example, during the traversal of candidate fault-tolerant regions, the current candidate region identified as a fault-tolerant region can be corrected. After obtaining the corrected fault-tolerant region, the next candidate fault-tolerant region is traversed again. Based on the number of fault-tolerant transition points in the next candidate fault-tolerant region, it is determined whether the next candidate fault-tolerant region is indeed a fault-tolerant region. At this time, if the next candidate fault-tolerant region is adjacent to the corrected fault-tolerant region, the number of fault-tolerant transition points in the next candidate fault-tolerant region will change. Therefore, after each determination of whether the current candidate region is a fault-tolerant region, the number of fault-tolerant transition points in the next candidate fault-tolerant region can be recalculated to ensure that the determination of the next candidate fault-tolerant region is based on the corrected depth value.

[0099] After traversing all candidate error-correcting regions and correcting them, the corrected depth map can be obtained. (Reference) Figure 12 The diagram shown illustrates an error correction effect, as follows: Figure 12 As shown, after correcting the fault-tolerant regions (as shown in blue) in the depth map to be processed, the corrected depth map can be obtained.

[0100] In this embodiment, the depth correction amount of the extended period can be accurately determined based on the depth value difference between the fault-breaking jump point and the neighboring points corresponding to the fault-breaking jump point, so as to correct the depth of the fault-breaking region and ensure the correction effect of the fusion depth.

[0101] In some implementations, the method further includes: obtaining a region to be filtered out from a plurality of connected regions; the surrounding region of the region to be filtered out is a non-error-correction region or an invalid region; when the size of the region to be filtered out is less than a preset threshold and the difference in depth values ​​between the boundary points in the region to be filtered out and the neighboring points corresponding to the boundary points is greater than a preset difference, the depth values ​​of the plurality of pixels in the region to be filtered out are filtered out.

[0102] For example, in addition to correcting the fault-melting regions in the candidate fault-melting regions, filtering can also be performed on the regions to be filtered out (or isolated regions) in multiple connected regions. The regions to be filtered out refer to connected regions surrounded by normal regions (i.e., non-fault-melting regions) or invalid regions. Invalid regions include multiple invalid points. Invalid points are points whose depth values ​​are deemed unreliable due to insufficient energy of reflected light received by pixels on the image sensor in the iToF camera. Since the regions to be filtered out are surrounded by normal or invalid regions, they are fault-melting regions that were not identified as candidate fault-melting regions in the above embodiments. Therefore, it can be further determined whether filtering of the regions to be filtered out is necessary.

[0103] If the size of the region to be filtered is smaller than a preset threshold, and the difference (or average difference) in the fusion depth values ​​between the boundary points and their corresponding neighboring points in the region to be filtered is greater than a preset difference, then the depth values ​​of multiple pixels in the region to be filtered can be filtered out. In some examples, the filtering process may involve setting the depth value of a pixel to 0.

[0104] In this embodiment, the fusion region in the depth map acquired by the dual-frequency iToF camera can be effectively judged and filtered out, thereby more effectively solving the fusion phenomenon caused by the phase unfolding problem of the dual-frequency iToF camera, and making the corrected fusion region more accurately reflect the depth of the object in the actual environment.

[0105] Based on the above embodiments, this application also provides a solution to the phase unfolding problem of dual-frequency iToF cameras. To address the problem of incorrect high-frequency spread period calculation caused by phase unfolding issues, which in turn leads to incorrect distance calculation, this application provides solutions from two aspects: frequency selection and error correction.

[0106] 1. Frequency Selection For dual-frequency TOF cameras, the high-frequency and low-frequency settings need to be selected based on the range and accuracy requirements. Higher frequencies can achieve higher measurement accuracy, but the corresponding range is smaller and error integration problems are more likely to occur.

[0107] This application's embodiments designed and developed TOF dual-frequency ranging calculator software to study the relationship between high and low frequencies and the occurrence of fault fusion phenomena. The software interface can be found in [reference needed]. Figure 5 As shown, this software can simulate the error-fusing scenarios that occur under different high and low frequency errors, such as... Figure 5As shown, assuming the low frequency is 17.679MHz, the corresponding single-cycle range is 8484mm; and the high frequency is 99.000MHz, the corresponding single-cycle range is 1515mm. When the actual distance is 5000mm, the distance difference between the low frequency at the extension period of 0 and the high frequency at the extension period of 3 is minimal (ideally, the difference is 0), thus obtaining the correct high-frequency extension period number of 3, and therefore calculating the correct distance of 5000mm.

[0108] like Figure 6 As shown, when the low frequency has an error of -1.8% (as shown by the yellow dashed line in the figure), the distance difference between the low frequency at the extension period of 2 and the high frequency at the extension period of 14 is the smallest. At this time, the extension period of the high frequency is 14, and the calculated error distance is 21665mm.

[0109] like Figure 7 As shown, when the low frequency has an error of +1.8% (as shown by the yellow dashed line in the figure), the distance difference between the low frequency at the extension period of 3 and the high frequency at the extension period of 20 is the smallest. At this time, the extension period of the high frequency is 20, and the calculated error distance is 30755mm.

[0110] The above shows the impact of a single frequency error on phase expansion calculation for a certain high-low frequency combination. Based on this, we study the minimum value of single frequency error when the error fusion phenomenon occurs in different frequency combinations, and summarize the rules to derive the basic principles of frequency selection.

[0111] Let's assume that errors only exist at low frequencies. We can study at what minimum low-frequency error (including both positive and negative directions) will occur when the error fusion phenomenon occurs in different combinations of high and low frequencies. The absolute value of the minimum low-frequency error percentage at which the error fusion phenomenon occurs is recorded as the error boundary. For example, in the above example, the error boundary is 1.8% in the combination of 99.000MHz and 17.679MHz. The following uses an iToF continuous wave modulation (CW-iToF) camera as an example. The error boundary under different frequency combinations can be referred to Table 1. The method for determining the low-frequency range can be referred to Formula (5). The method for determining the fusion range (i.e., unambiguous distance) can be referred to Formula (6).

[0112] This leads to the following pattern: for frequency combinations with the same fusion range / low-frequency range, the larger the multiple of the high-frequency / low-frequency range, the smaller the fusion boundary, and the easier it is for fusion errors to occur.

[0113] For example, frequency selection can be performed by referring to the following steps: (1) Based on the maximum distance Dmax that needs to be measured in the application scenario, select the frequency combination with a fusion range greater than Dmax from multiple frequency combinations (i.e., the first frequency combination); in addition, it can be confirmed that the light power emitted by the selected TOF camera is sufficient to meet the requirement of the maximum distance to be measured (the light power is determined by the power of the camera hardware emission unit, which is a prerequisite for meeting the ranging requirements). (2) Since the final ranging accuracy is determined by the high frequency, the frequency combination that meets the conditions (such as being greater than the preset frequency) can be selected from the first frequency combination selected in step (1) according to the accuracy requirements of the application scenario (i.e., the second frequency combination). (3) Since the larger the error boundary, the less likely it is to cause error fusion, the target frequency combination with the largest error boundary can be selected from the second frequency combination selected in step (2).

[0114] Image acquisition based on the target frequency combination determined by the above frequency selection can reduce the probability of error fusion when the iToF camera performs dual-frequency fusion, which is a prerequisite for solving the phase unfolding problem.

[0115] 2. Error handling For example, the following steps can be used to handle the fault-finding problem: (1) Perform region connectivity segmentation on the depth map based on the depth differences of neighboring points; (2) Traverse all connected regions and find the fault-breaking transition boundary; (3) Determine the fusion region based on the relevant statistical characteristics of adjacent regions; (4) For regions identified as faulty, corrections are made based on the boundary jump values ​​of adjacent regions; (5) For isolated regions, select whether to perform error filtering based on the region size and the mean fusion error.

[0116] 2.1. Depth Map Region Connectivity Segmentation The depth map can be segmented using a quadratic traversal 8-neighborhood connectivity algorithm. The segmentation steps are as follows: (1) Initialize a label matrix of the same size as the depth map to 0, and create a label set to store different regions.

[0117] (2) Perform the first traversal in the order from top to bottom and from left to right.

[0118] (3) For each pixel, skip if the depth value is invalid (such as 0 or an outlier) or the amplitude is less than the threshold (the depth of a pixel with too small an amplitude is considered unreliable).

[0119] (4) Check the 4 neighbors that have been visited in its 8-neighborhood (left, top left, top, top right).

[0120] (5) For each neighbor, if |d1 If d2|<threshold T (i.e. the first preset difference), then the current pixel is considered to be connected to its neighbor.

[0121] (6) If there are no neighbors that meet the conditions, assign a new label to the current pixel.

[0122] (7) If there is only one neighbor that meets the conditions, then inherit the label of that neighbor.

[0123] (8) If there are multiple neighbors with different labels that satisfy the conditions, assign the smallest label to them and record the equivalence relationship between these labels in the label set.

[0124] (9) After completing the first traversal, perform the second traversal.

[0125] (10) In the second traversal, the temporary label of each pixel is found through the label set to achieve label merging and unification.

[0126] (11) The final result is the neighborhood connectivity region partitioning that satisfies the condition "the depth difference between adjacent pixels is less than the threshold". The effect of region connectivity can be seen by referring to Figure 10 As shown.

[0127] 2.2. Judgment of Regional Anomaly Jump Boundaries For example, next, all connected regions can be traversed, and the boundary points in each connected region can be found to calculate the depth difference between the boundary points and the 8 neighboring points that do not belong to this connected region, thereby determining whether the jump is caused by an error in the expansion period calculation. (See reference) Figure 13 The flowchart shown is an implementation flowchart of a method for judging the boundary of anomaly transitions in a region. Figure 13 As shown, the method includes S1201-S1209: S1201, traverse the boundary points of the connected region.

[0128] S1202, calculate the depth difference between the current boundary point of the current connected region and the 8 neighboring points that do not belong to the current connected region.

[0129] S1203, count the number of neighborhood points that meet the preset conditions among the neighborhood points corresponding to the current boundary point.

[0130] The preset conditions may include: the amplitude of a neighboring point is greater than a threshold (i.e., preset amplitude); the depth difference on a single high-frequency and low-frequency depth map is less than a threshold (i.e., second preset difference); and the depth difference on the depth map after dual-frequency fusion is greater than a threshold, and the difference is close to an integer multiple of the high-frequency range.

[0131] S1204, determine that the number of neighboring points (i.e., the first neighboring points) that meet the conditions in the neighborhood is greater than the threshold (i.e., the first preset number).

[0132] S1205, determine the current boundary point as a non-fault-crossing transition point.

[0133] S1206, determine the current boundary point as the fault-fusing jump point.

[0134] S1207, determine that the number of fault-breaking jump points in the boundary of the current connected region is greater than the threshold (i.e., the second preset number).

[0135] S1208, the current connected region is determined as a non-fault-evolving region.

[0136] S1209, the current connected region is identified as a candidate fault-correction region.

[0137] It is understood that the implementation methods of S1201-S1209 can be referred to the description of the above embodiments, and will not be repeated here.

[0138] For example, if the above three preset conditions are met simultaneously, the current boundary point can be determined as a fault-correcting jump point; and connected regions in the boundary with more than a threshold number of fault-correcting jump points can be judged as candidate fault-correcting regions, and then further judged whether the candidate fault-correcting regions are indeed fault-correcting regions. Figure 11 As shown in the figure, the upper boundary of the area indicated by the arrow is not a fault-fusing boundary because the upper neighboring points have obvious differences in a single high- and low-frequency depth map, which is a normal depth difference between the foreground and the background. However, the lower neighboring points can be determined as fault-fusing jump points under the three conditions mentioned above for judging fault-fusing jump points.

[0139] It should also be noted that for a depth map of a single frequency, the spread period is always 0, and the depth difference corresponds to the phase difference, with phase 0 and phase 2 being different. For the same depth, the depth difference on a single-frequency depth map can be calculated using formula (7).

[0140] 2.3. Error Detection and Correction The candidate fault-finding regions identified in step 2.2 indicate the presence of deep fault-finding transition boundaries, but it remains unclear which side of these boundaries constitutes a fault-finding region. Therefore, further determination is needed to confirm whether the candidate fault-finding region is indeed a fault-finding region.

[0141] For a properly calibrated iToF camera, the occurrence of frequency fusion errors in the depth image can be considered a low-probability event. This means that the depth values ​​of most pixels in the depth image are normal, and small connected regions are more likely to be fusion error regions. Therefore, candidate fusion error regions can be sorted by the number of pixels, and then iterated through from smallest to largest to determine whether a region is a fusion error region. (Reference) Figure 14 The flowchart shown is an implementation flowchart of a method for determining fault-tolerant regions. Figure 14 As shown, the method includes S1301-S1307: S1301, sort the candidate fault-correction regions from largest to smallest.

[0142] S1302, traverse the candidate fault-tolerant regions in ascending order.

[0143] S1303, calculates the percentage of fault-finding jump points in the region boundary.

[0144] S1304, the proportion of fault-fusing jump points in the region boundary is greater than the threshold (such as the preset ratio).

[0145] S1305, the current candidate fault-correcting region is determined to be a fault-correcting region.

[0146] S1306, corrects the depth of the current fault-correction region.

[0147] S1307, the current candidate fault-correcting region is determined to be a non-fault-correcting region.

[0148] For example, if the currently traversed candidate fault-correcting region is determined to be a fault-correcting region, the expansion period of the fault-correcting region is corrected to minimize the boundary depth jump. After correcting the expansion period and updating the region depth value of the current fault-correcting region, the next candidate fault-correcting region is judged, ensuring that the depth of the boundary neighborhood points is the corrected value when calculating the proportion of fault-correcting jump points in the next candidate fault-correcting region. The effect of fault-correction can be seen in [reference needed]. Figure 12 As shown.

[0149] It is understood that the implementation methods of S1301-S1307 can be referred to the description of the above embodiments, and will not be repeated here.

[0150] 2.4. Error filtering For isolated regions without adjacent regions (not candidate fusion regions, but possibly regions surrounded by normal regions or filtered invalid points), if the region size is less than the threshold and the mean fusion error is greater than the threshold, the points in that region can be filtered out for fusion.

[0151] 3. The technical problem solved by the embodiments of this application: (1) The embodiments of this application solve the frequency selection problem of dual-frequency iToF camera, enabling a better balance between range, ranging accuracy and error fusion probability; (2) The embodiments of this application have designed and implemented TOF dual-frequency ranging calculator software, which provides a visualization tool for selecting the dual-frequency of the iToF camera; (3) The embodiments of this application can effectively judge the distance fusing error caused by the camera phase unfolding period calculation error, and perform fusing error correction or filter out fusing error pixels; (4) Avoid the problem of using the magnitude of the amplitude to determine whether the expansion period is calculated incorrectly in the traditional phase expansion problem-solving method.

[0152] 5. Technical effects and advantages of the embodiments of this application: (1) It can solve the phase unfolding problem of dual-frequency iToF camera from the perspective of system, including frequency selection, error judgment, error correction and error filtering; (2) By studying the error boundaries of different frequency combinations, the steps and principles of frequency selection are pointed out, so that the dual-frequency iToF camera is less likely to have the problem of fusion error while meeting the application requirements. (3) Effectively identify and correct or filter out the faulty regions in the depth map acquired by the dual-frequency iToF camera.

[0153] Based on the foregoing embodiments, this application provides a deep fusion error processing device, which includes the included units and the modules included in each unit. It can be implemented by the processor in the iToF camera; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0154] Figure 15 This is a schematic diagram of the composition structure of a deep fusion error processing device provided in an embodiment of this application, as shown below. Figure 15 As shown, the depth fusion error processing device 1400 can be configured in an iToF camera. The depth fusion error processing device 1400 includes: a first determining module 1410, a second determining module 1420, a third determining module 1430, and a correction module 1440, wherein: The first determining module 1410 is configured to determine multiple connected regions in the depth image to be processed.

[0155] The depth image to be processed is obtained by acquiring images of the target object based on a target frequency combination, which includes two different frequencies.

[0156] The second determining module 1420 is configured to determine the physical characteristics of the boundary regions and the physical characteristics of the adjacent regions corresponding to the boundary regions in a plurality of connected regions.

[0157] The boundary region and its adjacent region are different connected regions; the physical features include at least one of amplitude, depth and number of pixels.

[0158] The third determining module 1430 is configured to determine the fault-melting region to be corrected in multiple connected regions based on the physical characteristics of the boundary regions and the physical characteristics of the adjacent regions corresponding to the boundary regions in each connected region.

[0159] The correction module 1440 is configured to correct the depth value of the fault-tolerant region to obtain the corrected fault-tolerant region.

[0160] In some implementations, the target frequency combination includes a first target frequency and a second target frequency. The target frequency combination is determined as follows: by simulating and calculating the frequency errors of the first frequency and / or the second frequency in multiple frequency combinations, the fusion range corresponding to each frequency combination and the minimum frequency error when deep fusion error occurs are obtained; the first frequency is higher than the second frequency; the first frequency includes the first target frequency, and the second frequency includes the second target frequency; a first frequency combination with a fusion range greater than a preset maximum measurement distance is selected from multiple frequency combinations; a second frequency combination with a first frequency higher than a preset frequency is selected from the first frequency combination; and a target frequency combination with a minimum frequency error of the maximum value is selected from the second frequency combination.

[0161] In some implementations, the depth image to be processed of the image to be processed includes multiple pixels; the first determining module 1410 is specifically configured to: determine multiple connected regions in the depth image to be processed of the image to be processed based on the depth values ​​of each pixel in the depth image to be processed of the image to be processed; the difference in the fused depth values ​​of adjacent pixels in each connected region is less than a first preset difference.

[0162] In some embodiments, the first determining module 1410 is specifically configured to: determine the initial label of each pixel in the depth image to be processed, and traverse multiple pixels in a preset order; the initial labels of each pixel in the depth image to be processed are the same; based on the depth value difference between the current pixel and the traversed neighboring points corresponding to the current pixel, determine target neighboring points belonging to the same connected region as the current pixel among the traversed neighboring points corresponding to the current pixel; the multiple pixels include the current pixel; update the initial label of the current pixel based on the number of target neighboring points to obtain the updated label of the current pixel; determine multiple connected regions based on the updated labels of each pixel; multiple pixels in each connected region have the same updated label.

[0163] In some implementations, the first determining module 1410 is specifically configured to: determine the updated label of the current pixel as the first label when there is no target neighbor point; the first label of the current pixel is different from the updated label of the target neighbor point; determine the updated label of the current pixel as the updated label of the target neighbor point when there is only one target neighbor point; and determine the updated label of the current pixel as the smallest label among the updated labels of the multiple target neighbor points when there are multiple target neighbor points.

[0164] In some implementations, the third determining module 1430 is specifically configured to: determine the fault-fusing jump point caused by the frequency spreading period calculation error in each boundary region based on the amplitude of the neighboring points corresponding to the boundary points in each boundary region, and the depth value difference between the boundary points and their corresponding neighboring points in each boundary region; the adjacent regions corresponding to the boundary regions include the neighboring points corresponding to the boundary points; the depth value difference includes the dual-frequency fusion depth value difference and / or the single-frequency depth value difference; determine the fault-fusing region in multiple connected regions based on the number of fault-fusing jump points in each boundary region; the number of pixels includes the number of fault-fusing jump points.

[0165] In some implementations, the third determining module 1430 is specifically configured to: determine candidate fault-breaking regions in multiple connected regions based on the number of fault-breaking jump points in each connected region; determine the fault-breaking probability of fault-breaking phenomena occurring in each candidate fault-breaking region; sequentially traverse multiple candidate fault-breaking regions according to the magnitude of the fault-breaking probability of each candidate fault-breaking region; and determine whether the current candidate region is a fault-breaking region based on the number of fault-breaking jump points in the current candidate region.

[0166] In some implementations, the correction module 1440 is specifically configured to: determine the extended period correction amount based on the depth value difference between the fault-breaking jump point in the fault-breaking region and the neighboring points corresponding to the fault-breaking jump point; and correct the depth values ​​of multiple pixels in the fault-breaking region based on the extended period correction amount to obtain the corrected fault-breaking region.

[0167] like Figure 15 As shown, the device also includes an acquisition module 1450 and a filtering module 1460.

[0168] In some implementations, the acquisition module 1450 is configured to: acquire a region to be filtered out from a plurality of connected regions; the surrounding region of the region to be filtered out is a non-error-correcting region or an invalid region; the filtering module 1460 is configured to: filter out the depth values ​​of a plurality of pixels in the region to be filtered out when the size of the region to be filtered out is less than a preset threshold and the difference in depth values ​​between the boundary points in the region to be filtered out and the neighboring points corresponding to the boundary points is greater than a preset difference.

[0169] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0170] It should be noted that, in the embodiments of this application, if the above-mentioned deep fusion error processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0171] Figure 16 This is a schematic diagram of the structure of an iToF camera provided in an embodiment of this application, as shown below. Figure 16 As shown, the iToF camera 1500 includes a processor 1510 and a light-emitting module 1520. The processor 1510 is used to implement some or all of the steps in the above method; the light-emitting module 1520 is used to store target frequency combinations, which include two different frequencies. The iToF camera 1500 is used to acquire images using the light signal of the target frequency combination stored in the light-emitting module 1520 during image acquisition.

[0172] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor in an iToF camera, the computer program implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0173] This application provides a computer program including computer-readable code, which, when executed in the processor of an iToF camera, allows the processor to perform some or all of the steps in the above-described method.

[0174] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a processor in an iToF camera, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0175] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0176] This application provides a computer storage medium that stores one or more programs that can be executed by a processor to implement the steps of the deep fusion error processing method described in the above embodiments.

[0177] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0178] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0179] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for processing deep fusion errors, characterized in that, When applied to iToF cameras, the method for handling depth fusion errors includes: Multiple connected regions in the depth image to be processed are identified; the depth image to be processed is obtained by image acquisition of the acquisition object based on a target frequency combination, the target frequency combination including two different frequencies; Determine the physical characteristics of the boundary regions and the physical characteristics of the adjacent regions corresponding to the boundary regions in the plurality of connected regions; the boundary regions and the adjacent regions corresponding to the boundary regions belong to different connected regions; the physical characteristics include at least one of amplitude, depth value and number of pixels; Based on the amplitude of the neighboring points corresponding to the boundary points in each of the boundary regions, and the depth value difference between the boundary points in each of the boundary regions and the neighboring points corresponding to the boundary points, the fault-fusing jump point caused by the frequency spread period calculation error is determined in each of the boundary regions; the adjacent region corresponding to the boundary region includes the neighboring points corresponding to the boundary points; the depth value difference includes the dual-frequency fusion depth value difference and / or the single-frequency depth value difference. Based on the number of fault-melting jump points in each of the aforementioned boundary regions, a fault-melting region to be corrected is determined in multiple of the aforementioned connected regions; the number of pixels includes the number of the fault-melting jump points; The depth value of the fault-tolerant region is corrected to obtain the corrected fault-tolerant region.

2. The method for processing deep fusion errors according to claim 1, characterized in that, The target frequency combination includes a first target frequency and a second target frequency, and the target frequency combination is determined in the following manner: By simulating and calculating the frequency error of the first frequency and / or the second frequency in multiple frequency combinations, the fusion range corresponding to each frequency combination and the minimum frequency error when deep fusion error occurs are obtained; the first frequency is higher than the second frequency; the first frequency includes the first target frequency, and the second frequency includes the second target frequency. From the multiple frequency combinations, a first frequency combination whose fusion range is greater than the preset maximum measurement distance is selected; Select a second frequency combination from the first frequency combination whose first frequency is higher than a preset frequency; The target frequency combination with the minimum frequency error being the maximum value is selected from the second frequency combination.

3. The method for processing deep fusion errors according to claim 1, characterized in that, The depth image to be processed includes multiple pixels; determining multiple connected regions in the depth image to be processed includes: Based on the depth values ​​of each pixel in the depth image to be processed, multiple connected regions in the depth image to be processed are determined; the difference in depth values ​​between adjacent pixels in each connected region is less than a first preset difference.

4. The method for processing deep fusion errors according to claim 3, characterized in that, Based on the depth values ​​of each pixel in the depth image to be processed, multiple connected regions in the depth image to be processed are determined, including: The initial label of each pixel in the depth image to be processed is determined, and the multiple pixels are traversed in a preset order; the initial labels of all pixels in the depth image to be processed are the same. Based on the depth difference between the current pixel and its corresponding traversed neighboring points, a target neighboring point belonging to the same connected region as the current pixel is determined among the traversed neighboring points corresponding to the current pixel; the plurality of pixels include the current pixel; The initial label of the current pixel is updated based on the number of the target neighboring points to obtain the updated label of the current pixel; Multiple connected regions are determined based on the update labels of each pixel; multiple pixels in each connected region have the same update label.

5. The method for processing deep fusion errors according to claim 4, characterized in that, The initial label of the current pixel is updated based on the number of the target neighboring points to obtain the updated label of the current pixel, including: If the target neighboring point does not exist, the updated label of the current pixel is determined to be the first label; the first label of the current pixel is different from the updated label of the target neighboring point. When the number of target neighbor points is one, the update label of the current pixel is determined to be the update label of the target neighbor point; When there are multiple target neighborhood points, the updated label of the current pixel is determined to be the smallest label among the updated labels of the multiple target neighborhood points.

6. The method for processing deep fusion errors according to claim 1, characterized in that, The step of determining the fault-tolerant region in multiple connected regions based on the number of fault-tolerant jump points in each of the boundary regions includes: Based on the number of fault-fusing jump points in each of the connected regions, candidate fault-fusing regions are determined among the multiple connected regions; Determine the probability of fault-finding occurring in each of the candidate fault-finding regions; Based on the fault-finding probability of each candidate fault-finding region, the multiple candidate fault-finding regions are traversed sequentially. Based on the number of fault-melting transition points in the current candidate region, determine whether the current candidate region is the fault-melting region.

7. The method for processing deep fusion errors according to claim 1, characterized in that, The depth value of the fault-tolerant region is corrected to obtain the corrected fault-tolerant region, including: The expansion period correction amount is determined based on the depth difference between the fault-tolerance jump point in the fault-tolerance region and the neighboring points corresponding to the fault-tolerance jump point. The depth values ​​of multiple pixels in the fault-melting region are corrected based on the extended period correction amount to obtain the corrected fault-melting region.

8. The method for processing deep fusion errors according to any one of claims 1-7, characterized in that, The method for handling deep fusion errors also includes: Obtain the region to be filtered out from the multiple connected regions; the surrounding region of the region to be filtered out is a non-error-correcting region or an invalid region; If the size of the region to be filtered is less than a preset threshold, and the difference in depth values ​​between the boundary point in the region to be filtered and the neighboring point corresponding to the boundary point is greater than a preset difference, the depth values ​​of multiple pixels in the region to be filtered are filtered out.

9. An iToF camera, characterized in that, The iToF camera includes a processor for implementing the depth fusion error processing method according to any one of claims 1-8.

Images