A data processing method and system on chip based on structured light
By processing the Grey code and phase shift decoding module of binocular images in parallel in the on-chip system, combined with the dynamic data bit width adjustment of the depth calculation module, the problem of inefficiency in structured light data processing is solved, and three-dimensional with high efficiency and low latency is realized Point cloud construction.
Patent Information
- Application Number
- CN202210143036.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-02-16
AI Technical Summary
In the existing structured optical data processing methods, the system processing speed is slow and the delay is large, which is mainly due to the pipeline processing of data per pixel point, which leads to inefficiency.
Using the system-on-chip Grey code decoding module, phase shift decoding module, phase correction module and depth calculation module, the binocular image is processed in parallel through the on-chip bus, dynamically adjust the data bit width, and use field programmable logic gate arrays and microprocessors. Optimize computing resources and realize parallel optimization of three-dimensional point cloud data.
It significantly improves system processing efficiency, reduces latency, and realizes high-precision and high-density three-dimensional point cloud construction, suitable for high-end industrial manufacturing and other fields.
Smart Images

Figure CN114596364B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of structured light technology, and in particular to a data processing method and a system on chip based on structured light. Background Art
[0002] Binocular stereo vision is a major branch of computer vision. It uses two cameras at different positions to shoot the same scene. The cameras use structured light to form a three-dimensional light pattern on the object being measured. The degree of light distortion projected on the object being measured depends on the depth of the object's surface. After the captured image is collected, the three-dimensional coordinate value of the point in space, i.e., the depth information, is obtained by calculating the parallax of the point in the two images (i.e., binocular images).
[0003] In the prior art, pipeline processing is usually performed according to the data of each pixel point, and the output result is that the three-dimensional data of each pixel point is output sequentially according to the system clock. This method will lead to slow system processing speed and large delay. Summary of the invention
[0004] In view of the above problems, a data processing method based on structured light and a system on chip are proposed to overcome the above problems or at least partially solve the above problems, including:
[0005] A data processing method based on structured light is applied to a system on chip, wherein the system on chip is deployed with a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module, wherein the Gray code decoding module, the phase shift decoding module, and the depth calculation module are implemented based on a field programmable logic gate array in the system on chip, and the system on chip is configured with a memory unit, and the memory unit is connected to a module implemented based on the field programmable logic gate array through an on-chip bus, and the method comprises:
[0006] A Gray code decoding module reads and processes the binocular image from the memory unit in parallel according to a first parallelism corresponding to the bus width of the on-chip bus to obtain a Gray code result;
[0007] The phase shift decoding module reads and processes the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus and in combination with the Gray code result, obtains the first absolute phase value, and stores the first absolute phase value in the memory unit;
[0008] A phase correction module performs phase correction on the first absolute phase value obtained from the memory unit to obtain a second absolute phase value, and writes the second absolute phase value into the memory unit;
[0009] The depth calculation module determines a third degree of parallelism according to currently available resources in the field programmable logic gate array, and determines in parallel the three-dimensional point cloud data corresponding to the pixel points in the binocular image according to the third degree of parallelism in combination with the second absolute phase value.
[0010] Optionally, it also includes:
[0011] In the process of the Gray code decoding module and the phase shift decoding module reading the binocular image from the memory unit, applying the first data bit width;
[0012] In the process of calculating the first absolute phase value by the phase shift decoding module, dynamically adjusting the applied data bit width to a second data bit width, where the second data bit width is greater than the first data bit width;
[0013] In the process of determining the three-dimensional point cloud data by the depth calculation module, the applied data bit width is dynamically adjusted to a third data bit width, and the third data bit width is greater than the first data bit width.
[0014] Optionally, according to the third degree of parallelism, in combination with the second absolute phase value, three-dimensional point cloud data corresponding to the pixel points in the binocular image are determined in parallel, including:
[0015] Determining a plurality of first pixel points in a first view of the binocular image according to a third degree of parallelism;
[0016] For each first pixel point, in combination with the second absolute phase value, pixel matching is performed in the second view of the binocular image to obtain a second pixel point that is successfully matched;
[0017] According to the successfully matched pixel pairs, the depth information corresponding to the pixel points in the binocular image is determined, and the three-dimensional point cloud data is determined based on the depth information.
[0018] Optionally, for each first pixel point, in combination with the second absolute phase value, pixel matching is performed in the second view to obtain a successfully matched second pixel point, including:
[0019] In the second view, the third pixel point of the most recent successful pixel matching is determined, and the pixel point range for this pixel matching is determined based on the third pixel point;
[0020] For each first pixel point, pixel matching is performed in a pixel point range of the second view in combination with the second absolute phase value to obtain a successfully matched second pixel point.
[0021] Optionally, taking the third pixel point as a reference, determining a pixel point range for pixel point matching this time includes:
[0022] According to the third degree of parallelism, the size of the range of pixel matching to be performed this time is determined;
[0023] According to the size of the range, the pixel range for pixel matching this time is determined with the third pixel as a reference.
[0024] Optionally, the phase correction module is implemented based on a microprocessor in a system on chip, and the microprocessor is connected to the memory unit.
[0025] Optionally, the first degree of parallelism and the second degree of parallelism are maximum degrees of parallelism corresponding to a bus bit width of the on-chip bus, and the maximum degree of parallelism is negatively correlated with the first data bit width.
[0026] A system on chip, wherein the system on chip is deployed with a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module, wherein the Gray code decoding module, the phase shift decoding module, and the depth calculation module are implemented based on a field programmable logic gate array in the system on chip, and the system on chip is configured with a memory unit, and the memory unit is connected to a module implemented based on the field programmable logic gate array through an on-chip bus;
[0027] A Gray code decoding module, used for reading and processing the binocular image from the memory unit in parallel according to a first parallelism corresponding to the bus width of the on-chip bus, to obtain a Gray code result;
[0028] A phase shift decoding module is used to read and process the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus and in combination with the Gray code result, to obtain a first absolute phase value, and store the first absolute phase value in the memory unit;
[0029] A phase correction module, used for performing phase correction on the first absolute phase value obtained from the memory unit to obtain a second absolute phase value, and writing the second absolute phase value into the memory unit;
[0030] The depth calculation module is used to determine the third degree of parallelism according to the current available resources in the field programmable logic gate array, and to determine the three-dimensional point cloud data corresponding to the pixel points in the binocular image in parallel according to the third degree of parallelism and in combination with the second absolute phase value.
[0031] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the data processing method based on structured light as described above is implemented.
[0032] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method based on structured light as described above is implemented.
[0033] The embodiments of the present invention have the following advantages:
[0034] In an embodiment of the present invention, the Gray code decoding module reads and processes the binocular image from the memory unit in parallel according to the first parallelism corresponding to the bus width of the on-chip bus, and obtains the Gray code result. The phase shift decoding module reads and processes the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus, combined with the Gray code result, obtains the first absolute phase value, and stores the first absolute phase value in the memory unit. The phase correction module performs phase correction on the first absolute phase value obtained from the memory unit to obtain the second absolute phase value, and writes the second absolute phase value into the memory unit. The depth calculation module determines the third parallelism according to the current available resources in the field programmable logic gate array, and determines the three-dimensional point cloud data corresponding to the pixel points in the binocular image in parallel according to the third parallelism and combined with the second absolute phase value, thereby realizing parallel optimization of obtaining three-dimensional point cloud data according to the binocular image, improving the efficiency of system processing, and reducing delay. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0036] Figure 1a is a schematic diagram of a point cloud construction system provided by an embodiment of the present invention;
[0037] Figure 1b is a flowchart of a data processing method based on structured light provided by an embodiment of the present invention;
[0038] Figure 1c is a schematic diagram of a system on a chip provided by an embodiment of the present invention;
[0039] Figure 1d is a schematic diagram of another system on chip provided by an embodiment of the present invention;
[0040] Figure 1e is a schematic diagram of a module operation provided by an embodiment of the present invention;
[0041] Figure 2a is a flowchart of another data processing method based on structured light provided by an embodiment of the present invention;
[0042] Figure 2b This is a schematic diagram of pixel matching provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] In the field of structured light technology, the point cloud construction method based on the combination of binocular and surface structured light coding technology can construct dense point clouds with high speed and high precision. Therefore, it can be well applied in industrial inspection and measurement, digital-analog comparison, 3D printing, reverse engineering and other fields.
[0045] In the embodiment of the present invention, the three-dimensional point cloud constructed by binocular structured light technology has the characteristics of high precision, high density and high quality, and at the same time has the characteristics of low power consumption, high stability and low latency of the system-on-a-chip (SoC). The system can be used for high-end industrial manufacturing, such as printing of three-dimensional circuit boards, manufacturing of high-precision workpieces, etc.
[0046] Among the many coding schemes, grating phase shift coding schemes, such as Gray code combined with phase shift method, encode each pixel in the structured light coverage area, greatly improving the density and accuracy of point cloud construction. The key to binocular structured light technology is to use structured light coding values instead of traditional image grayscale values for stereo matching to obtain high-precision, high-density, high-quality depth information of the target plane.
[0047] Binocular structured light technology increases the texture of the image through coded structured light, encodes each pixel in the camera's field of view, decodes the structured light coded pattern after the left and right cameras capture the image, and then only needs to compare the decoded values to obtain the parallax. For areas of the workpiece that lack texture (such as complex circuit boards), dense point clouds can be constructed, which can be applied to high-end industrial manufacturing with high-precision requirements.
[0048] like Figure 1a, the point cloud construction system based on binocular structured light can mainly include an off-chip storage module located at the front end (including binocular camera, structured light encoding and projection module), connected to the front end, and a data processing and point cloud construction module connected to the off-chip storage module, and outputs a three-dimensional point cloud through off-chip storage. Specifically, the structured light encoding and projection module projects the grating structured light encoded by Gray code and phase shift to the target plane, and the camera collects the Gray code structured light image (8 images for the left and right views respectively) and the phase shift structured light image (4 images for the left and right views respectively, with phase shifts of 0 degrees, 90 degrees, 180 degrees, and 270 degrees respectively) of the left and right views, and the image accuracy is 5 million pixels. The system stores these pixels collected by the camera in off-chip storage for subsequent processing by the three-dimensional point cloud construction module.
[0049] In an example of the present invention, the data processing and point cloud construction module may include a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module.
[0050] Reference Figure 1b , shows a step flow chart of a data processing method based on structured light provided by an embodiment of the present invention, which can be applied to a system on a chip.
[0051] Among them, the system on chip can be deployed with a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module, such as Figure 1c For the binocular image collected by structured light, a parallel Gray code decoding module, a phase shift decoding module, and a phase correction module can be set for the left view and the right view in the binocular image respectively, and they are connected with the depth calculation modules respectively to obtain the three-dimensional point cloud data for the pixel points in the binocular image.
[0052] Specifically, Figure 1d The system on chip may include a processing system (PS) and a programmable logic unit (PL). The two parts may be interconnected through an on-chip bus. The on-chip bus may be based on the AXI (Advanced eXtensible Interface) protocol, which may provide an on-chip bus oriented to high performance, high bandwidth, and low latency.
[0053] Among them, the programmable logic unit can be composed of a field programmable gate array (FPGA), and the Gray code decoding module, the phase shift decoding module, and the depth calculation module are implemented based on the field programmable gate array in the on-chip system.
[0054] The processing system can be configured with a memory unit and a microprocessor connected to the memory unit. The memory unit is connected to a module based on a field programmable gate array through an on-chip bus. The memory unit can be a synchronous dynamic random access memory (SDRAM), such as DDR (Double Data Rate SDRAM), and the microprocessor can be a microprocessor based on a reduced instruction set computer (RISC), such as ARM (Advanced RISC Machine).
[0055] In one example, the internal operations of the module and the operations between the modules follow the pipeline principle and have high real-time performance. Figure 1e As shown, the horizontal axis is time t, and the vertical axis is module operation.
[0056] First moment: Gray code decoding 1 is executed for the first image data.
[0057] At the second moment: the phase shift decoding 1 is executed for the first image data, and the Gray code decoding 2 is executed for the second image data.
[0058] At the third moment: a phase correction 1 is executed for the first image data, a phase shift decoding 2 is executed for the second image data, and a Gray code decoding 3 is executed for the third image data.
[0059] At the fourth moment: depth calculation 1 is executed for the first image data, phase correction 2 is executed for the second image data, phase shift decoding 3 is executed for the third image data, and Gray code decoding 4 is executed for the fourth image data.
[0060] Fifth moment: executing depth calculation 2 for the second image data, executing phase correction 3 for the third image data, and executing phase shift decoding 4 for the fourth image data.
[0061] Sixth time point: Depth calculation 3 is executed for the third image data, and phase correction 4 is executed for the fourth image data.
[0062] Time 7: Run depth calculation 4 for the fourth image data.
[0063] Specifically, the following steps may be included:
[0064] Step 101 : A Gray code decoding module reads and processes a binocular image from a memory unit in parallel according to a first parallelism corresponding to a bus width of an intra-chip bus to obtain a Gray code result.
[0065] For structured light, the same scene is photographed by two cameras at different positions. The cameras use structured light to form a three-dimensional light pattern on the object being measured. The degree of distortion of the light projected on the object being measured depends on the depth of the object's surface. After the captured image is collected, the three-dimensional coordinate value of the point in space, i.e., the depth information, is obtained by calculating the time difference between the two images (i.e., binocular images).
[0066] Specifically, the binocular image may include a binocular Gray code structured light image and a phase shift structured light image, which may be stored in a memory unit of the on-chip system.
[0067] The Gray code decoding module can be used to obtain the Gray code result of phase shift decoding. Specifically, the Gray code decoding module can read the Gray code structured light image from the memory unit through the on-chip bus, and obtain a coarse-grained Gray code result, that is, the Gray code result of phase shift decoding, through threshold comparison and shift XOR operation. The Gray code result may include a decoding cycle for phase shift decoding.
[0068] Since the Gray code decoding module reads data from the memory unit through the on-chip bus, in order to make full use of the high parallelism characteristics of the field programmable logic gate array, the first parallelism can be determined according to the bus width of the on-chip bus. According to the first parallelism, the binocular image can be read and the pixels can be processed in parallel from the memory unit, thereby significantly improving the efficiency of system operations and reducing the initial delay of the system without affecting the calculation accuracy.
[0069] In one embodiment of the present invention, the method may further include: applying the first data bit width during the process of the Gray code decoding module reading the binocular image from the memory unit.
[0070] Since the image input to the Gray code decoding module is a grayscale image, the requirement for data accuracy is not high, and a lower first data bit width, such as 8 bits, can be used to avoid occupying resources.
[0071] In one embodiment of the present invention, the first degree of parallelism may be the maximum degree of parallelism corresponding to the bus width of the on-chip bus, and the maximum degree of parallelism may be negatively correlated with the first data width, that is, when the bus width of the on-chip bus remains unchanged, the larger the data width of the grayscale image read and processed, the smaller the maximum degree of parallelism.
[0072] For example, the Gray code decoding module reads 8 Gray code grayscale images through 8 on-chip buses. Assuming that the on-chip bus width of the on-chip system is 128 bits, for each on-chip bus, since each pixel of the grayscale image is 8 bits wide, the maximum parallelism can be 16 degrees of parallelism.
[0073] Step 102, the phase shift decoding module reads and processes the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus and the Gray code result, obtains the first absolute phase value, and stores the first absolute phase value in the memory unit.
[0074] The phase shift decoding module can be used to decode the sinusoidal periodic fringe pattern in the phase shift structured light image to obtain a relative phase value, and superimpose it with the Gray code result to obtain a first absolute phase value.
[0075] Specifically, the phase-shift decoding module can read the phase-shifted structured light image from the memory unit through the on-chip bus, and can read the Gray code result obtained by the Gray code decoding module through the internal data stream of the field programmable logic gate array, and then calculate the relative phase value of each pixel in the image, and then superimpose the Gray code results to obtain a globally unique first absolute phase value. After obtaining the first absolute phase value, it can be transmitted to the memory unit for storage through the on-chip bus.
[0076] For example, the cordic algorithm can be used to calculate the relative phase value. It is a method that replaces multiplication operations with basic addition and shift operations to achieve advanced operations such as trigonometric functions, logarithms, and exponentials. The cordic algorithm continuously approximates the target value through iteration. The more iterations, the higher the calculation accuracy. Based on this, the initial relative phase value calculation formula of the pixel point can be as follows:
[0077]
[0078] Among them, φ(x, y) can be a relative phase value, I1 to I4 can correspond to the grayscale value of each pixel of four grating images with a phase difference of 90 degrees, and x and y can represent the row and column where the pixel is located, respectively.
[0079] Since the phase-shift decoding module reads data from the memory unit through the on-chip bus, in order to make full use of the high parallelism of the field programmable logic gate array, the second parallelism can be determined according to the bus width of the on-chip bus. The binocular images can be read and processed in parallel from the memory unit according to the second parallelism, thereby significantly improving the efficiency of system operations and reducing the initial delay of the system without affecting the calculation accuracy.
[0080] In one embodiment of the present invention, the method may further include:
[0081] In the process of the phase shift decoding module reading the binocular image from the memory unit, the first data bit width is applied; in the process of the phase shift decoding module calculating the first absolute phase value, the applied data bit width is dynamically adjusted to the second data bit width, and the second data bit width is greater than the first data bit width.
[0082] Since the image input to the phase shift decoding module is a grayscale image, which has low requirements on data accuracy, a lower first data bit width, such as 8 bits, can be used to avoid occupying resources.
[0083] When the phase shift decoding module calculates the relative phase value and the first absolute phase value, the phase accuracy requirement is high, so the data bit width can be dynamically adjusted to dynamically adjust the application data bit width to the second data bit width, such as 32 bits, to ensure data accuracy.
[0084] In one embodiment of the present invention, the second degree of parallelism may be the maximum degree of parallelism corresponding to the bus width of the on-chip bus, and the maximum degree of parallelism may be negatively correlated with the first data width, that is, when the bus width of the on-chip bus remains unchanged, the larger the data width of the grayscale image read and processed, the smaller the maximum degree of parallelism.
[0085] For example, the phase shift decoding module reads four Gray code grayscale images through four on-chip buses. Assuming that the bus width of the on-chip bus of the system on chip is 128 bits, for each on-chip bus, since each pixel of the grayscale image is 8 bits wide, the maximum parallelism can be 16 degrees of parallelism.
[0086] Step 103: The phase correction module performs phase correction on the first absolute phase value obtained from the memory unit to obtain a second absolute phase value, and writes the second absolute phase value into the memory unit.
[0087] The phase correction module can be used to perform coordinate transformation on the first absolute phase value according to the calibrated camera distortion parameter to obtain a corrected second absolute phase value.
[0088] In one embodiment of the present invention, since the binocular views are both high-pixel, such as 5 million pixels, the phase correction module requires a large cache resource to complete the cache of the correction address. When implementing the necessary phase correction algorithm in 3D structured light, if an address buffer + phase value random access memory is used on a single field programmable logic gate array, a large amount of hardware resources inside the field programmable logic gate array will be consumed, and the operating efficiency will be significantly reduced. The phase correction module can be implemented based on a microprocessor in the on-chip system, and the field programmable logic gate array and the microprocessor interact with each other using an on-chip bus and a memory unit, which fully utilizes the on-chip resources and significantly improves the operating efficiency of the system, avoiding the occupation of resources caused by deploying all algorithms on a single field programmable logic gate array.
[0089] Specifically, the microprocessor can read the first absolute phase value and the correction position matrix obtained according to the camera distortion parameters from the memory unit, complete the phase correction, obtain the second absolute phase value, and then write the second absolute phase value into the memory unit through the on-chip bus.
[0090] Step 104, the depth calculation module determines a third degree of parallelism according to the current available resources in the field programmable logic gate array, and determines in parallel the three-dimensional point cloud data corresponding to the pixel points in the binocular image according to the third degree of parallelism in combination with the second absolute phase value.
[0091] The depth calculation module can be used to find the corresponding pixel points of the left and right images in the binocular image according to the second absolute phase value, calculate the depth value of each two-dimensional coordinate and output it to obtain three-dimensional point cloud data.
[0092] Specifically, the depth calculation module can read the second absolute phase value from the memory unit, and combine it with the second absolute phase value to find the position with the closest phase in the left and right views of the binocular view through a traversal method, and use the linear difference of the position and the triangulation algorithm to obtain the three-dimensional point cloud data of the corresponding point.
[0093] Since traversal-based matching and searching consumes a lot of time and resources, the degree of parallelism is flexibly configured according to the current available resources in the field programmable logic gate array to obtain the third degree of parallelism. According to the third degree of parallelism, the three-dimensional point cloud data corresponding to the pixel points in the binocular image are determined in parallel, thereby maximizing the acceleration of the depth calculation module.
[0094] In one embodiment of the present invention, the method may further include:
[0095] In the process of determining the three-dimensional point cloud data by the depth calculation module, the applied data bit width is dynamically adjusted to a third data bit width, and the third data bit width is greater than the first data bit width.
[0096] In the process of determining the three-dimensional point cloud data by the depth calculation module, if the accuracy of data such as depth information is required to be high, the data bit width can be dynamically adjusted to the second data bit width, such as 32 bits, to ensure the accuracy of the data.
[0097] In an embodiment of the present invention, the Gray code decoding module reads and processes the binocular image from the memory unit in parallel according to the first parallelism corresponding to the bus width of the on-chip bus, and obtains the Gray code result. The phase shift decoding module reads and processes the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus, combined with the Gray code result, obtains the first absolute phase value, and stores the first absolute phase value in the memory unit. The phase correction module performs phase correction on the first absolute phase value obtained from the memory unit to obtain the second absolute phase value, and writes the second absolute phase value into the memory unit. The depth calculation module determines the third parallelism according to the current available resources in the field programmable logic gate array, and determines the three-dimensional point cloud data corresponding to the pixel points in the binocular image in parallel according to the third parallelism and combined with the second absolute phase value, so as to achieve parallel optimization of obtaining three-dimensional point cloud data according to the binocular image, improve the efficiency of system processing, and reduce delay.
[0098] Moreover, by applying the first data bit width in the process of reading the binocular image from the memory unit in the phase shift decoding module, dynamically adjusting the applied data bit width to the second data bit width in the process of calculating the first absolute phase value in the phase shift decoding module, and dynamically adjusting the applied data bit width to the third data bit width in the process of determining the three-dimensional point cloud data in the depth calculation module, the third data bit width is larger than the first data bit width, thereby realizing dynamic adjustment of the data bit width according to the requirements of different modules for calculation accuracy, and avoiding the waste of storage resources and bandwidth resources due to the use of fixed data bit width.
[0099] Reference Figure 2a , shows a step flow chart of another data processing method based on structured light provided by an embodiment of the present invention, which can be applied to a system on a chip.
[0100] Among them, the system on chip can be deployed with a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module, such as Figure 1c For the binocular image collected by structured light, a parallel Gray code decoding module, a phase shift decoding module, and a phase correction module can be set for the left view and the right view in the binocular image respectively, and they are connected with the depth calculation modules respectively to obtain the three-dimensional point cloud data for the pixel points in the binocular image.
[0101] Specifically, Figure 1d The system on chip may include a processing system and a programmable logic unit. The two parts may be interconnected through an on-chip bus. The on-chip bus may be based on the AXI protocol, which may provide an on-chip bus oriented to high performance, high bandwidth, and low latency.
[0102] The programmable logic unit may be composed of a field programmable logic gate array, and the Gray code decoding module, the phase shift decoding module, and the depth calculation module are implemented based on the field programmable logic gate array in the system on chip. The processing system may be configured with a memory unit and a microprocessor connected to the memory unit, the memory unit is connected to the module based on the field programmable logic gate array through an on-chip bus, the memory unit may be a synchronous dynamic random access memory, such as DDR, and the microprocessor may be a microprocessor based on a reduced instruction set computer, such as ARM.
[0103] Specifically, the following steps may be included:
[0104] Step 201 : A Gray code decoding module reads and processes a binocular image from a memory unit in parallel according to a first parallelism corresponding to a bus width of an intra-chip bus to obtain a Gray code result.
[0105] Step 202, the phase shift decoding module reads and processes the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus and the Gray code result, obtains the first absolute phase value, and stores the first absolute phase value in the memory unit.
[0106] Step 203: The phase correction module performs phase correction on the first absolute phase value obtained from the memory unit to obtain a second absolute phase value, and writes the second absolute phase value into the memory unit.
[0107] Step 204: The depth calculation module determines a third degree of parallelism according to currently available resources in the field programmable logic gate array, and determines a plurality of first pixel points in the first view of the binocular image according to the third degree of parallelism.
[0108] The first view and the second view of the binocular image may be left and right views of the binocular image.
[0109] Since the third degree of parallelism is used to process the pixels, multiple first pixels can be determined in the first view of the binocular image according to the third degree of parallelism. For example, if the third degree of parallelism is 8, 8 pixels can be determined from the first view as the first pixels to be processed in parallel.
[0110] Step 205: The depth calculation module performs pixel matching in the second view of the binocular image for each first pixel in combination with the second absolute phase value to obtain a successfully matched second pixel.
[0111] For each first pixel, the depth calculation module can combine the second absolute phase value and use a traversal method to find the position with the closest phase in the second view of the binocular view, and obtain a second pixel that successfully matches the first pixel.
[0112] In one embodiment of the present invention, step 205 may include:
[0113] Sub-step 11, in the second view, determining the third pixel point that was successfully matched most recently, and determining the pixel point range for this pixel point matching based on the third pixel point.
[0114] Since most adjacent pixels on a two-dimensional plane are also adjacent in the depth dimension, when traversing and searching for the point with the best phase match between the left and right views, it is preferred to start traversing from the vicinity of the pixel point where the last match was successful, which can significantly shorten the traversal time.
[0115] Based on this, the third pixel point of the most recent successful pixel matching can be determined (the third pixel point can be randomly selected for the first pixel matching), and then the third pixel point can be used as a reference to determine the pixel point range for pixel matching in the second view this time.
[0116] In an embodiment of the present invention, determining the pixel range for pixel matching this time based on the third pixel may include:
[0117] According to the third degree of parallelism, the size of the range for pixel point matching is determined this time; according to the size of the range, the range of pixels for pixel point matching this time is determined with the third pixel point as a reference.
[0118] Since the degree of parallelism is greater, that is, more pixels are processed in parallel, such as from 8 to 16, the corresponding search range must also be expanded, so that the search success rate will be relatively high. The range size of the pixel point matching this time can be determined according to the third degree of parallelism. The range size can be positively correlated with the third degree of parallelism. Then, based on the range size, the pixel point range for the pixel point matching this time can be determined with the third pixel point as a reference.
[0119] In one example, the range size may be associated with the detection environment. For example, if depth detection is to be performed on a plane with dramatic vertical fluctuations, the search range should be expanded accordingly.
[0120] Sub-step 12: for each first pixel, in combination with the second absolute phase value, pixel matching is performed in a pixel range of the second view to obtain a successfully matched second pixel.
[0121] After determining the pixel range, for each first pixel, pixel matching can be performed in the pixel range of the second view to find a second pixel with the same or similar second absolute phase value in the pixel range, which is the second pixel that is successfully matched.
[0122] like Figure 2b, the third parallelism is 8, that is, 8 pixels are processed in parallel, and the pixel range is 40 pixels. Then, in the nth search, 8 first pixels are determined from the left image of the binocular view, that is, pixel 1 to pixel 8. In the nth search, pixel 20 in the right image of the binocular view is used as the search basis, and 40 pixels are determined in the right image of the binocular view, that is, pixel 1 to pixel 40, and then the second pixel matching pixel 1 to pixel 8 can be searched therefrom.
[0123] In the n+1th search, 8 first pixel points are determined from the left image of the binocular view, namely pixel point 9 to pixel point 16. Since in the nth search, the last pixel point 8 in the left image is successfully matched with pixel point 37 in the right image, pixel point 37 in the right image can be used as the search basis in the n+1th search, namely the third pixel point. Combined with the pixel point range of 40 pixels, 40 pixels can be determined in the right image, namely pixel point 17 to pixel point 57, and then the second pixel point matching pixel point 17 to pixel point 57 can be searched therefrom.
[0124] Compared with the common serial search method, this parallel search method can significantly improve the search efficiency. Taking 8 degrees of parallelism as an example, the module's operation speed can be increased by nearly 8 times. Although sometimes the match fails due to untimely update of the search range, the probability is small. After practical testing, it is found that the loss of accuracy is about 0.0013%, which can be ignored.
[0125] Step 206: The depth calculation module determines the depth information corresponding to the pixel points in the binocular image according to the successfully matched pixel point pairs, and determines the three-dimensional point cloud data according to the depth information.
[0126] For successfully matched pixel pairs, the linear difference of the position and the triangulation algorithm can be used to obtain the depth information of the corresponding points, and then the three-dimensional point cloud data can be determined based on the depth information.
[0127] For example, the depth information calculation method in the triangulation algorithm can use the following formula:
[0128]
[0129] Among them, f is the normalized focal length of the camera, L is the baseline distance between the left and right cameras, and d is the difference in the ordinate coordinates of the principal points of the left and right cameras. The above parameters are fixed after the camera is calibrated. L is the horizontal coordinate of the pixel point in the left view, x R is the horizontal coordinate of the right view pixel corresponding to the phase, and Z is the depth information.
[0130] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0131] An embodiment of the present invention also provides a system on chip, which is deployed with a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module. The Gray code decoding module, the phase shift decoding module, and the depth calculation module are implemented based on a field programmable gate array in the system on chip. The system on chip is configured with a memory unit, and the memory unit is connected to the module implemented based on the field programmable gate array through an on-chip bus.
[0132] The Gray code decoding module is used to read and process the binocular image from the memory unit in parallel according to the first parallelism corresponding to the bus width of the on-chip bus to obtain a Gray code result.
[0133] The phase shift decoding module is used to read and process the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus and the Gray code result, obtain the first absolute phase value, and store the first absolute phase value in the memory unit.
[0134] The phase correction module is used to perform phase correction on the first absolute phase value obtained from the memory unit to obtain a second absolute phase value, and write the second absolute phase value into the memory unit.
[0135] The depth calculation module is used to determine the third degree of parallelism according to the current available resources in the field programmable logic gate array, and to determine the three-dimensional point cloud data corresponding to the pixel points in the binocular image in parallel according to the third degree of parallelism and in combination with the second absolute phase value.
[0136] In one embodiment of the present invention, the system on chip may also be used for:
[0137] In the process of the Gray code decoding module and the phase shift decoding module reading the binocular image from the memory unit, the first data bit width is applied.
[0138] In the process of the phase shift decoding module calculating the first absolute phase value, the applied data bit width is dynamically adjusted to a second data bit width, and the second data bit width is greater than the first data bit width.
[0139] In the process of determining the three-dimensional point cloud data by the depth calculation module, the applied data bit width is dynamically adjusted to a third data bit width, and the third data bit width is greater than the first data bit width.
[0140] In one embodiment of the present invention, according to the third degree of parallelism, in combination with the second absolute phase value, three-dimensional point cloud data corresponding to the pixel points in the binocular image are determined in parallel, including:
[0141] According to the third degree of parallelism, a plurality of first pixel points are determined in the first view of the binocular image.
[0142] For each first pixel point, in combination with the second absolute phase value, pixel matching is performed in the second view of the binocular image to obtain a second pixel point with successful matching.
[0143] According to the successfully matched pixel pairs, the depth information corresponding to the pixel points in the binocular image is determined, and the three-dimensional point cloud data is determined based on the depth information.
[0144] In one embodiment of the present invention, for each first pixel point, in combination with the second absolute phase value, pixel matching is performed in the second view to obtain a successfully matched second pixel point, including:
[0145] In the second view, the third pixel point of the most recent successful pixel matching is determined, and the pixel point range for this pixel matching is determined based on the third pixel point.
[0146] For each first pixel point, pixel matching is performed in a pixel point range of the second view in combination with the second absolute phase value to obtain a successfully matched second pixel point.
[0147] In one embodiment of the present invention, the pixel range for pixel matching is determined based on the third pixel, including:
[0148] The size of the range for pixel matching this time is determined according to the third degree of parallelism.
[0149] According to the size of the range, the pixel range for pixel matching this time is determined with the third pixel as a reference.
[0150] In one embodiment of the present invention, the phase correction module is implemented based on a microprocessor in a system on chip, and the microprocessor is connected to a memory unit.
[0151] In one embodiment of the present invention, the first degree of parallelism and the second degree of parallelism are the maximum degrees of parallelism corresponding to the bus bit width of the on-chip bus, and the maximum degree of parallelism is negatively correlated with the first data bit width.
[0152] In an embodiment of the present invention, the Gray code decoding module reads and processes the binocular image from the memory unit in parallel according to the first parallelism corresponding to the bus width of the on-chip bus, and obtains the Gray code result. The phase shift decoding module reads and processes the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus, combined with the Gray code result, obtains the first absolute phase value, and stores the first absolute phase value in the memory unit. The phase correction module performs phase correction on the first absolute phase value obtained from the memory unit to obtain the second absolute phase value, and writes the second absolute phase value into the memory unit. The depth calculation module determines the third parallelism according to the current available resources in the field programmable logic gate array, and determines the three-dimensional point cloud data corresponding to the pixel points in the binocular image in parallel according to the third parallelism and combined with the second absolute phase value, thereby realizing parallel optimization of obtaining three-dimensional point cloud data according to the binocular image, improving the efficiency of system processing, and reducing delay.
[0153] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above-mentioned data processing method based on structured light is implemented.
[0154] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method based on structured light is implemented.
[0155] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0156] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0157] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0158] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.
[0159] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0161] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0162] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0163] The above provides a detailed introduction to a data processing method and system on chip based on structured light. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A data processing method based on structured light, characterized in that: Applied to a system on chip, the system on chip is deployed with a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module, the Gray code decoding module, the phase shift decoding module, and the depth calculation module are implemented based on a field programmable logic gate array in the system on chip, the system on chip is configured with a memory unit, and the memory unit is connected to a module implemented based on the field programmable logic gate array through an on-chip bus, the method comprising: The Gray code decoding module reads and processes the binocular image from the memory unit in parallel according to a first parallelism corresponding to the bus width of the on-chip bus to obtain a Gray code result; The phase shift decoding module reads and processes the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus and in combination with the Gray code result to obtain a first absolute phase value, and stores the first absolute phase value in the memory unit; The phase correction module performs phase correction on the first absolute phase value obtained from the memory unit to obtain a second absolute phase value, and writes the second absolute phase value into the memory unit; The depth calculation module determines a third degree of parallelism based on currently available resources in the field programmable logic gate array, and determines in parallel the three-dimensional point cloud data corresponding to the pixel points in the binocular image according to the third degree of parallelism in combination with the second absolute phase value.
2. The method according to claim 1, characterized in that Also includes: In the process of the Gray code decoding module and the phase shift decoding module reading the binocular image from the memory unit, applying a first data bit width; In the process of the phase shift decoding module calculating the first absolute phase value, dynamically adjusting the applied data bit width to a second data bit width, wherein the second data bit width is greater than the first data bit width; In the process of determining the three-dimensional point cloud data by the depth calculation module, the applied data bit width is dynamically adjusted to a third data bit width, and the third data bit width is greater than the first data bit width.
3. The method according to claim 1 or 2, characterized in that: The method of determining the three-dimensional point cloud data corresponding to the pixel points in the binocular image in parallel according to the third parallelism and in combination with the second absolute phase value includes: According to the third degree of parallelism, determining a plurality of first pixel points in a first view of the binocular image; For each first pixel point, in combination with the second absolute phase value, pixel matching is performed in the second view of the binocular image to obtain a second pixel point that is successfully matched; Determine the depth information corresponding to the pixel points in the binocular image based on the successfully matched pixel pairs, and determine the three-dimensional point cloud data based on the depth information.
4. The method according to claim 3, characterized in that The step of performing pixel matching in the second view in combination with the second absolute phase value for each first pixel to obtain a successfully matched second pixel includes: In the second view, a third pixel point for which pixel matching was successfully performed most recently is determined, and a pixel point range for pixel matching this time is determined based on the third pixel point; For each first pixel point, pixel matching is performed in a pixel point range of the second view in combination with the second absolute phase value to obtain a successfully matched second pixel point.
5. The method according to claim 4, characterized in that The step of determining the pixel range for pixel matching this time based on the third pixel point as a reference includes: Determine the size of the range of pixel point matching this time according to the third parallelism; According to the range size, the pixel point range for pixel point matching this time is determined with the third pixel point as a reference.
6. The method according to claim 1, characterized in that The phase correction module is implemented based on a microprocessor in the system on chip, and the microprocessor is connected to the memory unit.
7. The method according to claim 2, characterized in that The first degree of parallelism and the second degree of parallelism are maximum degrees of parallelism corresponding to the bus bit width of the on-chip bus, and the maximum degree of parallelism is negatively correlated with the first data bit width.
8. A system on chip, characterized in that: The system on chip is deployed with a Gray code decoding module, a phase shift decoding module, a phase correction module, and a depth calculation module, the Gray code decoding module, the phase shift decoding module, and the depth calculation module are implemented based on a field programmable logic gate array in the system on chip, and the system on chip is configured with a memory unit, and the memory unit is connected to the module implemented based on the field programmable logic gate array through an on-chip bus; The Gray code decoding module is used to read and process the binocular image from the memory unit in parallel according to a first parallelism corresponding to the bus width of the on-chip bus to obtain a Gray code result; The phase shift decoding module is used to read and process the binocular image from the memory unit in parallel according to the second parallelism corresponding to the bus width of the on-chip bus and in combination with the Gray code result, to obtain a first absolute phase value, and store the first absolute phase value in the memory unit; The phase correction module is used to perform phase correction on the first absolute phase value obtained from the memory unit to obtain a second absolute phase value, and write the second absolute phase value into the memory unit; The depth calculation module is used to determine a third degree of parallelism according to the current available resources in the field programmable logic gate array, and to determine in parallel the three-dimensional point cloud data corresponding to the pixel points in the binocular image according to the third degree of parallelism in combination with the second absolute phase value.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for data processing based on structured light as claimed in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method based on structured light according to any one of claims 1 to 7 is implemented.
