Flight time data processing device and data processing method based on intelligent processor

Abstract

The embodiment of the present application discloses a time-of-flight data processing device and a data processing method based on an intelligent processor. The data processing method includes the following steps: a vector calculation unit in an intelligent processor performs a conditional logic processing and an image difference calculation according to a sensing data generated by a time-of-flight ranging device to generate depth information; and a multiplication and accumulation unit in the intelligent processor performs a filtering process according to the depth information to generate output data.

Description

Technical Field

[0001] The present application relates to a time-of-flight ranging system, and more particularly to a data processing device and a data processing method for processing time-of-flight data in the system using an intelligent processor. Background Art

[0002] The raw data provided by the time-of-flight ranging sensor needs to be converted into effective distance or depth information through certain calculations. Therefore, the calculation time will affect the subsequent data processing and application of the overall system. In the prior art, most of them use general-purpose processors to perform the conversion of depth information, or use multi-threaded processing architectures to accelerate the conversion of depth information. However, in order to withstand the high amount of calculations, the above-mentioned technologies all require the use of processors or processing architectures with strong computing power, which increases the overall power consumption and hardware costs. Summary of the invention

[0003] The embodiments of the present application provide a time-of-flight data processing device and a data processing method using an intelligence processing unit to assist in processing time-of-flight data, so as to improve the deficiencies of the prior art.

[0004] The embodiment of the present application provides a time-of-flight data processing device, comprising: an intelligent processor, which performs the following steps on a sensing data generated by a time-of-flight ranging device: performs a conditional logic processing and an image difference calculation according to the sensing data to generate a depth information; and performs a filtering process according to the depth information to generate an output data. The intelligent processor includes a vector calculation unit and a multiplication-accumulation operation unit, the conditional logic processing is performed by the vector calculation unit, and the filtering process is performed by the multiplication-accumulation operation unit.

[0005] The present application provides a data processing method, comprising the following steps: performing a conditional logic process and an image difference calculation according to sensing data generated by a time-of-flight ranging device by a vector calculation unit in an intelligent processor to generate depth information; and performing a filtering process according to the depth information by a multiplication-accumulation operation unit in the intelligent processor to generate output data.

[0006] The features, implementation and effects of the present application are described in detail below with reference to the drawings as preferred embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 A schematic diagram of a distance measurement system provided according to an embodiment of the present application;

[0009] Figure 2 According to the embodiment of the present application, Figure 1 A calculation flow chart of the flight time algorithm executed by the intelligent processor in cooperation with the central processor;

[0010] Figure 3 According to the embodiment of the present application, Figure 1 memory to execute Figure 2 A schematic diagram of multiple data of phase conversion;

[0011] Figure 4A According to the embodiment of the present application, Figure 1 memory to execute Figure 2 A schematic diagram of multiple data of phase conversion;

[0012] Figure 4B According to the embodiment of the present application, Figure 1 memory to execute Figure 2 Schematic diagram of multiple data of depth conversion;

[0013] Figure 5 According to the embodiment of the present application, Figure 2 memory to execute Figure 2 Schematic diagram of multiple data of nonlinear functions in phase conversion and depth conversion;

[0014] Figure 6 According to the embodiment of the present application, Figure 2 memory to execute Figure 2 A schematic diagram of multiple data processed by filtering; and

[0015] Figure 7 The present invention provides a flowchart of a data processing method according to an embodiment of the present application.

[0016]

Explanation of symbols

[0017] 100: Distance measurement system

[0018] 110: Time of flight (TOF) ranging device

[0019] 112: Light source module

[0020] 114: Sensing module

[0021] 120: Flight time data processing device

[0022] 122: Memory

[0023] 124:CPU

[0024] 126: Intelligent Processor

[0025] 126A: Power and clock management module

[0026] 126B: Direct Memory Access (DMA) Controller

[0027] 126C: Vector computing unit

[0028] 126D: Table Lookup Unit

[0029] 126E: Instruction decoder

[0030] 126F: Multiplication and Accumulation (MAC) computing unit

[0031] 126G: Memory

[0032] 126H: bus

[0033] 700: Data processing method

[0034] D1~D4,D7~D8:Data

[0035] D5: Broadcast data

[0036] D6: At least one lookup table

[0037] D9: Convolution kernel data

[0038] DM: Shielded Data

[0039] DREF: Reference Data

[0040] P1: Transmission path

[0041] S210, S220, S230, S240: Steps

[0042] S710, S720, S730, S740, S750: Steps

[0043] SD: Sense Data DETAILED DESCRIPTION

[0044] All words used herein have their usual meanings. The definitions of the above words in commonly used dictionaries, including any use examples of the words discussed herein in the content of this application are only examples and should not limit the scope and meaning of this application. Similarly, this application is not limited to the various embodiments shown in this specification.

[0045] As used herein, "coupled" or "connected" may refer to two or more components making physical or electrical contact directly or indirectly, or two or more components operating or acting on each other. As used herein, the term "circuit" may refer to a device that is composed of at least one transistor and / or at least one active and passive component connected in a certain manner to process signals.

[0046] Figure 1 Schematic diagram of a distance measurement system 100 provided according to an embodiment of the present application. In some embodiments, the distance measurement system 100 may use a time-of-flight (TOF) algorithm to obtain distance information of an object, thereby calculating depth information and / or contour information of the object.

[0047] The distance measurement system 100 includes a TOF distance measurement device 110 and a time-of-flight data processing device 120. The TOF distance measurement device 110 includes a light source module 112 and a sensing module 114. The light source module 112 can send a light signal to an object according to the control of the sensing module 114, and the sensing module 114 can sense the light signal reflected from the object, and perform subsequent signal processing, analog-to-digital conversion and other operations on the signal to generate sensing data SD, and transmit the sensing data SD to the time-of-flight data processing device 120 for subsequent calculation.

[0048] In some embodiments, the sensing module 114 may include a clock generator (not shown) and a temperature sensing circuit. The clock generator may generate the timing required by each circuit, and the sensing module 114 may dynamically adjust the period of the optical signal according to the clock signal generated by the clock generator and the operating temperature of the current environment sensed by the temperature sensing circuit. The above configuration of the TOF ranging device 110 is for example only, and the present application is not limited thereto.

[0049] The flight time data processing device 120 includes a memory 122, a central processing unit 124, and an intelligence processing unit (IPU) 126. In some embodiments, the memory 122 may be, but is not limited to, a dynamic random access memory. The memory 122 is shared by the central processing unit 124 and the intelligence processing unit 126. The intelligence processing unit 126 is a processor that can execute a machine learning model and / or a neural network model. In some embodiments, the intelligence processing unit 126 can perform a convolution operation on the image data to analyze the feature information in the image data. In some embodiments, the intelligence processing unit 126 includes a power and clock management module 126A, a direct memory access (DMA) controller 126B, a vector calculation unit 126C, a table lookup unit 126D, an instruction decoder 126E, a multiply accumulate (MAC) calculation unit 126F, a memory 126G, and a bus (BUS) 126H, wherein the above-mentioned multiple modules and / or units can be coupled via the bus 126H to communicate with each other. The power and clock management module 126A may provide and set power and timing for other modules and / or units.

[0050] When the DMA controller 126B receives the instruction from the CPU 124, the DMA controller 126B may read at least a portion of the sensing data SD from the memory 122 and store it in the memory 126G. After the instruction decoder 126E decodes the instruction to be executed, at least one of the vector calculation unit 126C, the table lookup unit 126D and / or the MAC calculation unit 126F may read the corresponding data from the memory 126G to perform calculations. After all calculations are completed, the DMA controller 126B will store the calculation results back to the memory 122 and store them as output data. In some embodiments, the output data may include, but is not limited to, the depth information of the object to be detected. In some embodiments, the memory 126G may be, but is not limited to, a static random access memory.

[0051] In some related technologies, a general-purpose processor is used to calculate the raw data generated by the TOF ranging sensor. Since the raw data needs to be converted into depth information through certain operations, a processor with higher computing power is required in these technologies to load the above-mentioned computing process, resulting in higher overall power consumption and increased system costs. Compared with the above-mentioned technology, in some embodiments of the present application, most of the operations in processing TOF sensing data (such as sensing data SD) are assigned to the intelligent processor 126 for execution. In this way, the computing characteristics of the intelligent processor 126 can be fully utilized to process appropriate operations, and work in conjunction with the central processor 124 to process other remaining computing parts, so as to more efficiently obtain the distance and / or depth information of the object to be measured.

[0052] To illustrate the related operations performed by the intelligent processor 126, the basic concepts and operation flow of the TOF algorithm will be briefly described below. First, as mentioned above, the TOF algorithm is to measure the distance between the object to be measured and the light source module 112 by sending a light signal to the object to be measured through the light source module 112. Since the speed of light is constant, the above distance can be expressed as the following formula (1), where d is the distance between the object to be measured and the light source module 112, C0 is the speed of light, and ΔT is the time from the light source module 112 sending the light signal to the sensing module 114 receiving the reflected light signal.

[0053]

[0054] However, the above formula (1) assumes that the light source module 112 and the sensing module 114 are located at the same position. In practical applications, there will be a position deviation between the light source module 112 and the sensing module 114. This deviation can be corrected by further calibration. In addition, in some TOF algorithms, if the light source module 112 periodically emits a light signal, the phase shift (equivalent to the time offset) between the light signal emitted by the light source module 112 and the light signal received by the sensing module 114 can be analyzed by analyzing the image information in the sensing data SD, and the phase shift can be converted into the aforementioned distance d. For example, the above conversion relationship can be expressed as the following formula (2), where f m is the frequency of the optical signal emitted by the light source module 112, and This is the aforementioned phase shift.

[0055]

[0056] Figure 2 According to the embodiment of the present application, Figure 1126 cooperates with the central processor 124 to perform a TOF algorithm. In some embodiments, the TOF algorithm performed by the intelligent processor 126 in cooperation with the central processor 124 may include phase conversion (step S210), depth conversion (step S220), affine transformation (step S230) and filtering (step S240). For ease of understanding, the following description focuses on how to use the intelligent processor 126 to implement these steps. In some embodiments, the mathematical model, function and / or matrix used in the existing TOF algorithm can be applied to the above-mentioned multiple steps, so the detailed setting method of the mathematical model, function and / or matrix is ​​not repeated here.

[0057] In step S210, conditional logic processing is performed according to the sensing data (for example, the sensing data SD) to generate first intermediate data, and phase shift data (including phase shift information), and performs conditional logic processing based on the phase-shifted data to generate second intermediate data.

[0058] For example, to avoid calculation errors caused by overexposure or underexposure information, the smart processor 126 may first perform conditional logic processing on the sensing data SD to correct the data value in the sensing data SD to a first predetermined range, and generate first intermediate data (i.e., corrected sensing data SD). In some embodiments, the above-mentioned conditional logic processing may be performed by Figure 1 The vector calculation unit 126C is executed, and the steps described here will be referred to later. Figure 3 If the optical signal can be mathematically expressed as g ref (t) = cos(2πf m t), and for a given phase shift τ i , the correlation signal g corr It can be expressed as g corr (t) = g ref (t+τ i In this concept, a plurality of related image data A can be obtained by analyzing the first intermediate data. i , which can be expressed as the following formula (3), where s in (t) is equivalent to the image information (ie, sensing data SD) received by the sensing module 114 .

[0059]

[0060] Therefore, in the process of detecting the reflected light, the sensing module 114 can control the light source module 112 to use light sources at different positions to send light signals (or control the light source module 112 to use the same light source to send light signals at different time points) to obtain a plurality of related image data A. i For example, the phase shift τ iIt can be 0.5i×π, where i is 0, 1, 2, 3. In this way, four image data A0-A3 can be obtained, and the image difference between two of the image data A0-A3 can be further obtained (for example, A3-A1 and A0-A2 in the following formula (4)). Under this condition, the inverse tangent (arctan2) function can be used to process the image differences to obtain the aforementioned phase shift data (which includes the aforementioned phase shift information), where the operation of the inverse tangent function can be expressed as the following formula (4).

[0061]

[0062] In some embodiments, the image difference between the two images is calculated using the pixel values ​​at the same position. Figure 1 The image difference is obtained by performing an element-wise operation by the vector calculation unit 126C. The description here will be referred to later. Figure 4A In some embodiments, the inverse tangent function may be calculated using Figure 1 The table lookup unit 126D is executed, and the description here will be referred to later. Figure 5 For example, the table lookup unit 126D may perform a table lookup based on the values ​​of the two image differences (e.g., A3-A1 and A0-A2) to obtain the corresponding value of the distance d. After obtaining the phase shift data, in order to eliminate inappropriate phase shift values ​​to ensure the validity of the data, the phase shift data may be subjected to conditional logic processing to correct the data value in the phase shift data to a predetermined range and generate second intermediate data (i.e., corrected phase shift data). The operation description here will be referred to later. Figure 3 illustrate.

[0063] In step S220, depth information is generated according to the second intermediate data. For example, according to the above formula (2), it can be known that the phase shift Obtain the corresponding distance d. In some embodiments, Figure 1 The memory 126 may store at least one lookup table corresponding to equation (4) and equation (2) (e.g. Figure 5 In this way, the table lookup unit 126D can use at least one lookup table to perform the operations of equation (4) and equation (2) according to the image difference and the second intermediate data to generate intermediate data (hereinafter referred to as the third intermediate data). Then, the vector calculation unit 126C can perform element-by-element operations on the third intermediate data according to the correction matrix to generate depth information. The operation description here will be referred to later. Figure 4B and Figure 5 illustrate.

[0064] In step S230, transformation data is generated according to the transformation matrix and the depth information. The operation of step S230 is an affine transformation, in which the amount of depth information data will be adjusted according to the transformation matrix (for example, selection, translation, scaling) to change the resolution of TOF sensing. In some embodiments, the affine transformation is to amplify the depth information to improve the resolution of TOF sensing. In some embodiments, the affine transformation is to multiply the depth information with the transformation matrix to complete the transformation such as translation and scaling of the image, in which the transformation matrix can be Figure 1 After the CPU 124 calculates the transformation matrix, the CPU 124 returns the transformation matrix to the smart processor 126. In this way, the MAC calculation unit 126F can multiply the depth information with the transformation matrix to generate transformation data (i.e., the depth information after being magnified or reduced).

[0065] In some embodiments, the affine transformation method is to convert the data of each coordinate X, Y to x, y according to the following equation.

[0066]

[0067] The transformation matrix [[a 11 ,a 12 ],[a 21 ,a 22 ]] and [b1, b2], the intelligent processor 126 can use the MAC calculation unit 126F to complete the above calculation according to the above formula.

[0068] In some embodiments, there is a direct transmission path between the smart processor 126 and the central processor 124 (for example, Figure 1 Thus, when performing the related operations of affine transformation, the intelligent processor 126 and the central processor 124 can communicate directly via the transmission path P1, without having to communicate through other buses (which are usually connected to other circuits in the system). For example, when executing step S230, the central processor 124 can receive the request of the intelligent processor 126 via the transmission path P1 during one interruption period, calculate the transformation matrix during the next few interruptions, and return the transformation matrix to the intelligent processor 126 via the transmission path P1 for subsequent operations of step S230. In some embodiments, the intelligent processor 126 can also be connected to the memory 122 via the transmission path P1 to read the required data (for example, including, but not limited to, the sensing data SD).

[0069] In step S240, filtering is performed based on the transformed data to generate output data. The filtering process can further reduce the impact of unnecessary noise. In some embodiments, the output data carries information about the distance, depth and / or contour (i.e., the edge of the object) of the object to be measured. In some embodiments, the filtering process can be performed by Figure 1 The MAC calculation unit 126F is implemented. The description here will be referred to later. Figure 6 In some embodiments, the DMA controller 126B may store the output data in the memory 122 so that the CPU 124 can execute subsequent sensing applications according to the output data.

[0070] Figure 3 According to the embodiment of the present application, Figure 1 126G of memory is used to execute Figure 2 Schematic diagram of multiple data of phase conversion. In some embodiments, the memory 126G can store data D1, data DREF and mask data DM. In some embodiments, the vector calculation unit 126C can use the data D1, data DREF and mask data DM to perform Figure 2 Conditional logic processing in step S210 (i.e. phase conversion) in .

[0071] In some embodiments, the mask data DM can be used to store the comparison result between the data D1 and the data DREF. The comparison condition (equivalent to conditional logic processing) used for the aforementioned comparison result may include, but is not limited to, greater than, greater than or equal to, less than, less than or equal to, equal to, not equal to, etc. As described above, the vector calculation unit 126C can perform conditional logic processing on the sensing data SD to correct the data value in the sensing data SD to a first predetermined range (for example, between the first value and the second value described later) and generate first intermediate data. For example, it is assumed that the data processed by the time-of-flight data processing device 120 is 16 bits, and the sensing data SD is 12 bits. The DMA controller 126B can add bit 1 to the first four bits of the sensing data SD to correct the sensing data SD to 16 bits of data, and store the corrected sensing data SD as data D1. If the data value of the corresponding pixel in the data D1 (hereinafter referred to as pixel value) is less than the first value (for example, 0), it is determined that the pixel is underexposed. Alternatively, if the pixel value is greater than the second value (for example, 1), it is determined that the pixel is overexposed. Therefore, the DMA controller 126B may store the reference data DREF corresponding to the data D1 to the memory 126G, wherein the pixel values ​​in the reference data DREF are all first values ​​(e.g., 0). In this way, the vector calculation unit 126C may obtain the data D1 and the reference data DREF from the memory 126G, and compare the data D1 with the reference data DREF. For example, if the corresponding pixel value of the data D1 is less than the corresponding pixel value in the reference data DREF, it means that the pixel value in the data D1 (or the sensed data SD) is less than 0. Under this condition, the vector calculation unit 126C may mark the corresponding pixel value in the mask data DM as bit 1. Alternatively, if the corresponding pixel value in the data D1 is not less than the corresponding pixel value in the reference data DREF, it means that the pixel value in the data D1 (or the sensed data SD) is not less than 0. Under this condition, the vector calculation unit 126C may mark the corresponding pixel value in the mask data DM as bit 0. In other words, the mask data DM may indicate the comparison result between the data D1 and the reference data DREF.

[0072] Then, the vector calculation unit 126C may update the data D1 according to the mask data DM. For example, if the corresponding pixel value in the mask data DM is bit 1, the vector calculation unit 126C may update the corresponding pixel value in the data D1 to bit 0. Alternatively, if the corresponding pixel value in the mask data DM is bit 0, the vector calculation unit 126C may keep the corresponding pixel value in the data D1 as the original value. In this way, the underexposed pixel value in the data D1 (i.e., the sense data SD) may be corrected to bit 0.

[0073] Similarly, the vector calculation unit 126C may update all pixel values ​​of the reference data DREF to a second value (e.g., 1), and compare the updated data D1 with the updated reference data DREF again. For example, if the corresponding pixel value in the data D1 is greater than the corresponding pixel value in the reference data DREF, it means that the pixel value in the data D1 is greater than 1. Under this condition, the vector calculation unit 126C may mark the corresponding pixel value of the masking data DREF as bit 1. Alternatively, if the corresponding pixel value in the data D1 is not greater than the corresponding pixel value in the reference data DREF, it means that the pixel value in the data D1 is not greater than 1. Under this condition, the vector calculation unit 126C may mark the pixel of the masking data DM as bit 0. If the corresponding pixel value in the masking data DM is bit 1, the vector calculation unit 126C may update the corresponding pixel value in the data D1 to bit 1. Alternatively, if the corresponding pixel value in the masking data DM is bit 0, the vector calculation unit 126C may keep the corresponding pixel value in the data D1 as the original value. In this way, the overexposed pixel value in the data D1 (ie, the sensing data SD) can be corrected to bit 1. After completing the above operation, the vector calculation unit 126C can store the data D1 in the memory 126G as the aforementioned first intermediate data.

[0074] Then, as described above, after obtaining the phase-shifted data, the phase-shifted data may be subjected to conditional logic processing to correct the data value in the phase-shifted data to within a second predetermined range (e.g., between -π and π), and generate second intermediate data. As described above, the phase-shifted data is calculated by an inverse tangent function, and the numerical range of the inverse tangent function is from -π to π. Therefore, the vector calculation unit 126C may store the phase-shifted data as data D1, and update the data values ​​in the reference data DREF to the numerical values ​​corresponding to -π. Through the above-mentioned similar operations, the data values ​​lower than -π in the data D1 (i.e., the phase-shifted data) may be corrected. Then, the vector calculation unit 126C may update the data values ​​in the reference data DREF to the numerical values ​​corresponding to π. Through the same operation, the data values ​​exceeding π in the data D1 (i.e., the phase-shifted data) may be corrected. After completing the above-mentioned operations, the vector calculation unit 126C may store the data D1 in the memory 126G as the above-mentioned second intermediate data. Through the above-mentioned operations, the data validity in the phase-shifted data may be ensured to improve the accuracy of subsequent calculations.

[0075] Figure 4A According to the embodiment of the present application, Figure 1 memory to execute Figure 2 In some embodiments, the memory 126G can store data D2 and data D3, which can be used to perform Figure 2The element-by-element operation used in operation S210 of . As described above, before executing the inverse tangent function of equation (4), the difference between the two image data (e.g., A3-A1 and A0-A2) can be calculated by element-by-element operation. Since the image data A0-A3 have the same data format and dimensional format (i.e., have the same number of data values ​​(or element values)), the DMA controller 126B can store the first intermediate data (i.e., the sensing data SD corrected to the first predetermined range) obtained at different times into the memory 126G as data D2 and data D3, and the vector calculation unit 126C can perform element-by-element operation (in this example, subtraction) based on the data D2 and the data D3 to obtain the aforementioned image difference.

[0076] For example, assuming that the image data A0-A3 correspond to the first intermediate data of the first time, the second time, the third time and the fourth time in sequence, the DMA controller 126B may store the first intermediate data corresponding to the first time (equivalent to the image data A0) as data D2, and store the first intermediate data corresponding to the third time (equivalent to the image data A2) as data D3. In this way, the vector calculation unit 126C may perform element-by-element operations according to the data D2 and the data D3 to subtract the corresponding pixel value in the data D3 from the corresponding pixel value in the data D2 to calculate the image difference A0-A2. Through similar operations, the vector calculation unit 126C may calculate the image difference A3-A1.

[0077] Figure 4B According to the embodiment of the present application, Figure 1 memory to execute Figure 2 In some embodiments, the memory 126G can store data D4 and broadcast data D5, which can be used to perform Figure 2 The element-by-element operation used in step S220.

[0078] As described above, before executing the inverse tangent function of formula (4), the vector calculation unit 126C can perform an element-by-element operation on the third intermediate data according to the correction matrix to generate depth information. In this example, the element-by-element operation is a dot multiplication operation (or inner product operation). Since the depth information and the correction matrix have different dimensional forms (i.e., the number of data values ​​of the two is inconsistent), the vector calculation unit 126C can confirm that the number of data values ​​of the two is different according to the data D4 and the broadcast data D5, and use the broadcast operation to complete the operation. For example, the DMA controller 126B can store the intermediate data in the memory 126G as data D4 and store the correction matrix in the memory 126G as broadcast data D5. The vector calculation unit 126C can confirm that the two data have different dimensional forms according to the data D4 and the broadcast data D5, for example, the dimension of the data D4 is (1,224,224,3) and the dimension of the broadcast data D5 is (1,1,1,3). In this case, the vector calculation unit 126C may perform a dot multiplication operation on the three data values ​​in the data D4 and the one data value in the broadcast data D5, and perform 224×224 dot multiplication operations to generate depth information. In other words, if the dimensions of the two data to be element-by-element operated are different, the vector calculation unit 126C may use a method of repeating multiple dot multiplication operations to complete the overall operation on the data values ​​of the two.

[0079] In some related technologies, if a general-purpose processor is used to perform a dot multiplication operation on data with different dimensions, the data values ​​in the low-dimensional data will be copied multiple times so that the number of data values ​​in the low-dimensional data can be expanded to the same number of data values ​​in the high-dimensional data. In this way, the general-purpose processor can sequentially fetch the data values ​​from the memory to perform the dot multiplication operation one by one. In the above technology, a larger memory space and / or bandwidth will be required, and the general-purpose processor will spend more time accessing the memory to obtain the data values ​​when performing the operation. Compared with the above technology, the intelligent processor 126 can utilize the characteristics of the broadcast operation (i.e., repeating the operation multiple times) to perform the above operation without expanding the data value, so that the element-by-element operation here can be performed more efficiently.

[0080] In various embodiments, Figure 4A and Figure 4B The element-by-element operations described above can be used to implement operations such as addition, subtraction, multiplication, and division, but the present application is not limited thereto. Based on a similar configuration, the vector calculation unit 126C can perform operations such as trace, inverse, or transpose. According to different TOF algorithms, the intelligent processor 126 can also selectively use the above operations to perform data processing.

[0081] Figure 5 According to the embodiment of the present application, Figure 2126G of memory is used to execute Figure 2 Schematic diagram of multiple data of nonlinear functions in phase conversion and depth conversion in FIG. In some embodiments, the memory 126G can store a lookup table D6 and data D7, which are used to perform Figure 2 The partial operations used in operations S210 and S220 (eg, equation (2) and equation (4)) are shown in FIG. 1 . In some embodiments, the lookup table D6 may include a lookup table corresponding to equation (2) and a lookup table corresponding to equation (4).

[0082] In some embodiments, the memory 126G can store the calculation results of equation (4) and equation (2). For example, the inverse tangent function of equation (4) can be decomposed into one or more operations using an interpolation function, and the calculation results of the one or more operations can be stored as part of the data of the lookup table D6. Similarly, the calculation result of equation (2) can be stored as another part of the data of the lookup table D6. In some embodiments, the lookup table D6 can include a lookup table corresponding to equation (2) and a lookup table corresponding to equation (4), respectively. The lookup table unit 126D can query the lookup table corresponding to equation (4) in the lookup table D6 according to the aforementioned multiple image differences to obtain the phase shift data (for example, the phase shift in equation (4)). The vector calculation unit 126C may generate depth information by performing an element-by-element operation on the third intermediate data according to the correction matrix.

[0083] Through the above configuration, the intelligent processor 126 can execute nonlinear functions and achieve a certain degree of computational accuracy. Using a lookup table to decompose nonlinear functions can simplify the overall computation process. In this way, compared to using a general-purpose processor to directly perform nonlinear computations, the computational efficiency can be significantly improved.

[0084] Figure 6 According to the embodiment of the present application, Figure 2 126G of memory is used to execute Figure 2 In some embodiments, the memory 126G can store data D8, convolution kernel data D9 and bias data D10, which can be used to perform Figure 2 Operation S240.

[0085] For example, in step S230, the MAC calculation unit 126F may store the transformed data in the memory 126G as data D8. Then, the MAC calculation unit 126F may read the data D8, the convolution kernel data D9 and the offset data D10 from the memory 126G to perform a convolution operation (equivalent to a filtering process), and output the operation result as output data DO. In some embodiments, the convolution kernel data D9 is used to define a filter mask in image filtering, and the offset data D10 is used to define a moving distance of the filter mask on the data D8. The above filtering process conforms to the characteristics of the multiplication and addition operation originally set by the MAC calculation unit 126F. Therefore, using the MAC calculation unit 126F can speed up the processing speed of the filtering process.

[0086] In some embodiments, the aforementioned multiple operations may be combined with single instruction multiple data (SIMD) technology to further improve execution efficiency. In some embodiments, a general programming language (such as, but not limited to, Python) may be used to establish a software interface so that the user can provide data to the smart processor 126 through this interface to develop a model design of the TOF algorithm.

[0087] Figure 7 Flow chart of a data processing method 700 provided according to an embodiment of the present application. In some embodiments, the data processing method 700 may be, but is not limited to, Figure 1 The time-of-flight data processing device 120 executes.

[0088] In step S710, a vector calculation unit in the intelligent processor performs conditional logic processing according to the sensing data generated by the time-of-flight ranging device to generate first intermediate data. In step S720, a phase shift data is generated according to the first intermediate data. In step S730, depth information is generated according to the phase shift data. In step S740, transformation data is generated according to the transformation matrix and the depth information. In step S750, a multiplication and accumulation unit in the intelligent processor performs filtering processing according to the transformation data to generate output data.

[0089] In the embodiments of the present application, "plurality" refers to "two or more".

[0090] The description of the above-mentioned multiple operations can refer to the above-mentioned embodiments, so it will not be repeated. The multiple steps of the above-mentioned data processing method 700 are only examples, and are not limited to being executed in the order in this example. Without violating the operation mode and scope of each embodiment of the present application, the various operations under the data processing method 700 can be appropriately increased, replaced, omitted or executed in a different order (for example, they can be executed simultaneously or partially executed simultaneously).

[0091] In summary, the time-of-flight data processing device and the data processing method thereof in some embodiments of the present application can utilize the computing characteristics of the intelligent processor to process the time-of-flight data to accelerate the computing efficiency of processing the time-of-flight data.

[0092] The above is a detailed introduction to the flight time data processing device and data processing method based on the intelligent processor provided in the embodiment of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application; at the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A time-of-flight data processing device, characterized in that: include: An intelligent processor performs the following steps on a sensing data generated by a time-of-flight ranging device: Performing a conditional logic process and an image difference calculation according to the sensing data to generate depth information; as well as Perform a filtering process according to the depth information to generate output data; The intelligent processor includes a vector calculation unit and a multiplication-accumulation operation unit, the conditional logic processing is performed by the vector calculation unit, and the filtering processing is performed by the multiplication-accumulation operation unit; the intelligent processor includes a table lookup unit, which queries a lookup table according to a plurality of image differences generated by the image difference calculation to generate a phase shift data; The vector calculation unit is further used for correcting the phase shift data to correct the data value of the phase shift data to a second predetermined range, and the table lookup unit queries another lookup table according to the corrected phase shift data to generate the depth information.

2. The time-of-flight data processing device according to claim 1, characterized in that: The vector calculation unit performs the conditional logic processing according to the sensing data to correct a data value of the sensing data within a first predetermined range.

3. The time-of-flight data processing device according to claim 1, characterized in that: The vector calculation unit performs a dot multiplication operation on a correction matrix according to a query result of querying the other lookup table to generate the depth information.

4. The time-of-flight data processing device according to claim 1, characterized in that: The lookup table includes a calculation result of an inverse tangent function.

5. The time-of-flight data processing device according to claim 1, characterized in that: The device also includes: A central processing unit, for calculating a transformation matrix; The multiplication-accumulation operation unit further performs an affine transformation according to the transformation matrix and the depth information to adjust a resolution corresponding to the depth information.

6. The time-of-flight data processing device according to claim 5, characterized in that: The intelligent processor communicates directly with the central processor via a connection path to receive the transformation matrix.

7. A data processing method, characterized in that: include: A vector calculation unit in an intelligent processor performs a conditional logic process and an image difference calculation according to a sensing data generated by a time-of-flight ranging device to generate a depth information; as well as Performing a filtering process according to the depth information through a multiplication and accumulation unit in the intelligent processor to generate output data; A lookup table is searched by a lookup unit in the intelligent processor according to a plurality of image differences generated by the image difference calculation to generate a phase shift data; The vector calculation unit is further used for correcting the phase shift data to correct the data value of the phase shift data to a second predetermined range, and the table lookup unit queries another lookup table according to the corrected phase shift data to generate the depth information.

8. The data processing method according to claim 7, characterized in that: The conditional logic processing is performed by the vector calculation unit according to the sensing data to correct the data value of the sensing data to a first predetermined range.

9. The data processing method according to claim 7, characterized in that: The vector calculation unit performs a dot multiplication operation on a correction matrix according to a query result of querying the other lookup table to generate the depth information.

10. The data processing method according to claim 7, characterized in that: The lookup table includes a calculation result of an inverse tangent function.

11. The data processing method according to claim 7, characterized in that: The method further comprises: Calculate a transformation matrix by a central processing unit; The multiplication-accumulation operation unit further performs an affine transformation according to the transformation matrix and the depth information to adjust a resolution corresponding to the depth information.

Images