A real-time three-dimensional reconstruction system, design method and edge computing hardware

Abstract

This invention provides a real-time 3D reconstruction system, a design method, and edge computing hardware. The edge computing hardware is a dedicated processing chip such as an ASIC or FPGA that can be adapted to the real-time 3D reconstruction system. The real-time 3D reconstruction system adopts a real-time streaming hardware acceleration architecture, including: a gradient generation module that converts surface normal vectors into gradient field data streams; a two-dimensional forward transform module that converts the gradient field data streams into frequency domain data; a frequency domain solving module that solves for frequency domain depth data; and an inverse transform and reconstruction module that performs a discrete orthogonal inverse transform on the frequency domain depth data. This invention achieves low latency and power consumption. Furthermore, through the hardening of the frequency domain fast Poisson equation algorithm, the hardware logic design is highly versatile and easily ported to dedicated chips such as FPGAs, ASICs, and ISPs to achieve low power consumption. It can be widely applied to various 3D reconstruction scenarios such as industrial inspection, medical tactile sensing, and robot dexterity manipulation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a real-time 3D reconstruction system, design method, and edge computing hardware. Background Technology

[0002] Currently, high-precision 3D reconstruction technology generally relies on general-purpose computing platforms such as CPUs / GPUs, which results in problems such as high power consumption, large size, and high latency, making it difficult to meet the needs of edge devices such as mobile robots and wearable devices for battery life and real-time response. If the computing tasks are offloaded to the cloud, they are limited by bandwidth and communication latency.

[0003] Therefore, the existing high-energy-consuming and bulky hardware architecture cannot be integrated as an embedded module into electronic devices such as edge computing hardware, which severely limits the application of high-precision 3D reconstruction technology in portable and low-power scenarios. Summary of the Invention

[0004] This invention is made to solve the above-mentioned problems, and aims to provide a real-time 3D reconstruction system, design method and edge computing hardware, which adopts a real-time streaming hardware acceleration architecture and has the advantages of low power consumption, low latency and hardware-deployable features.

[0005] This invention provides a real-time 3D reconstruction system, characterized by a real-time streaming hardware acceleration architecture, comprising: a gradient generation module for receiving raw image data streams, generating surface normal vectors through a pre-constructed illumination-normal mapping model, and converting the surface normal vectors into a gradient field data stream; a 2D forward transform module connected to the gradient generation module for performing a discrete orthogonal transform on the gradient field data stream, converting the gradient field data stream into frequency domain data; a frequency domain solution module connected to the 2D forward transform module for performing algebraic operations on the frequency domain data to obtain frequency domain depth data; and an inverse transform and reconstruction module connected to the frequency domain solution module for performing a discrete orthogonal inverse transform on the frequency domain depth data to generate a spatial domain 3D surface topography data stream.

[0006] In one embodiment of the present invention, the two-dimensional forward transformation module includes: a row transformation unit, which performs a one-dimensional discrete orthogonal transformation on each row of the gradient field data stream to obtain an intermediate data stream after row transformation; a transpose buffer unit, which stores the intermediate data stream; and a column transformation unit, which performs a one-dimensional discrete orthogonal transformation on each column of the intermediate data stream.

[0007] In one embodiment of the present invention, the frequency domain solution module includes a parallel dot product operation unit and a frequency domain filter. The dot product operation unit performs algebraic operations on the frequency domain data and the spectral feature parameters based on the Laplace operator in the transform domain through a hardware multiplier, thereby obtaining the frequency domain depth data.

[0008] In one embodiment of the present invention, the inverse transformation and reconstruction module includes: a row inverse transformation unit, which performs a one-dimensional discrete orthogonal inverse transformation on each row of the frequency domain depth data to obtain a row inverse transformation data stream; and a column inverse transformation unit, which performs a one-dimensional discrete orthogonal inverse transformation on each column of the row inverse transformation data stream.

[0009] In one embodiment of the present invention, the illumination-normal mapping model is any one of a theoretical mathematical model, a high-dimensional lookup table, or a neural network. The theoretical mathematical model directly constructs mathematical formulas through circuits and maps the surface normal vector pixel by pixel.

[0010] In one embodiment of the present invention, the implementation steps of the high-dimensional lookup table include: the original image data stream contains multiple input pixels input in sequence; when the illumination-normal mapping model is a high-dimensional lookup table, the high-dimensional lookup table uses the combination of multi-channel light intensity values ​​of the input pixels as the address and outputs the surface normal vector corresponding to the input pixels.

[0011] In one embodiment of the present invention, the implementation steps of the neural network include: after fitting and approximating the illumination-normal mapping model, directly inferring the surface normal vector using the method of computational logic, converting the surface normal vector into a gradient field, and outputting the gradient field data stream, wherein the method of computational logic is formula inference, numerical lookup, or neural network prediction.

[0012] In one embodiment of the present invention, the gradient generation module, the two-dimensional forward transform module, the frequency domain solution module, and the inverse transform and reconstruction module are connected to each other via on-chip streaming interfaces.

[0013] This invention also provides a design method for a real-time 3D reconstruction system, characterized by the following steps: receiving an original image data stream; generating surface normal vectors through a pre-constructed illumination-normal mapping model; converting the surface normal vectors into a gradient field data stream; converting the gradient field data stream into frequency domain data by performing a discrete orthogonal transformation; performing algebraic operations on the frequency domain data to obtain frequency domain depth data; and generating a spatial domain 3D surface topography data stream by performing a discrete orthogonal inverse transformation on the frequency domain depth data.

[0014] The present invention also provides edge computing hardware, characterized by including the above-described real-time 3D reconstruction system, wherein the edge computing hardware is an application-specific integrated circuit, a field-programmable gate array, or an image signal processor.

[0015] The role and effect of invention

[0016] According to the real-time 3D reconstruction system, design method, and edge computing hardware involved in this invention, the invention has the beneficial effects of extremely low latency and power consumption. Due to the adoption of a streaming data processing structure, the number of times intermediate results are stored and accessed is reduced, thereby lowering system latency and power consumption. Millisecond-level end-to-end reconstruction can be achieved, meeting the power consumption requirements of robot edge devices and the needs of high-dynamic closed-loop control systems such as haptic feedback. The hardware logic design of this invention has strong versatility, not only suitable for field-programmable gate array (FPGA) prototype verification but also easily ported to dedicated chips such as application-specific integrated circuits (ASICs) and image signal processors (ISS). It can be widely applied to various 3D reconstruction scenarios such as industrial inspection, medical haptic sensing, and robot dexterity manipulation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a real-time 3D reconstruction system for dedicated hardware acceleration in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart illustrating the process of applying the intelligent visual-tactile sensor to low-power edge computing hardware in an embodiment of the present invention.

[0020] Figure 3 These are the specific steps S4 and S5 in the workflow of the intelligent visual-touch sensor in the embodiments of the present invention.

[0021] Figure 4 This is a flowchart illustrating the workflow of the computational photography 3D reconstruction device based on an embedded motherboard of a SoC in an embodiment of the present invention.

[0022] Explanation of reference numerals in the attached figures: 100 - Real-time 3D reconstruction system; 110 - Gradient generation module; 120 - 2D forward transform module; 130 - Frequency domain solution module; 140 - Inverse transform and reconstruction module. Detailed Implementation

[0023] The technical solutions disclosed in this invention will be described in detail below with reference to specific embodiments.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0026] In this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used only for descriptive and distinguishing purposes and should not be construed as indicating or implying relative importance.

[0027] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the real-time 3D reconstruction system, design method and edge computing hardware of this invention.

[0028] like Figure 1As shown, this invention discloses a real-time 3D reconstruction system 100 employing a real-time streaming hardware acceleration architecture, comprising at least a gradient generation module 110, a 2D forward transform module 120, a frequency domain solver module 130, and an inverse transform and reconstruction module 140, sequentially connected via an on-chip streaming interface. The real-time 3D reconstruction system 100 of this invention can be deployed in low-power edge computing hardware as a hardware intellectual property core or an independent chip. Preferably, the low-power edge computing hardware can be a dedicated processing chip, such as a hardware intellectual property core, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), an image signal processor (Image Signal Processor), and a haptic processing unit (HSP) chip. Specifically, the real-time 3D reconstruction system 100 can be licensed to other chip design companies as a hardware intellectual property core, or it can exist as an independent coprocessor chip, connected to the main central processing unit (CPU) via a PCIe interface or a USB interface, offloading the host's 3D reconstruction computation tasks. All these variations, as long as they include the relevant modules of this invention and their connection relationships, fall within the protection scope of this invention. The design method of the real-time three-dimensional reconstruction system of the present invention includes the following steps: receiving the original image data stream, generating surface normal vectors through a pre-constructed illumination-normal mapping model, and converting the surface normal vectors into a gradient field data stream; converting the gradient field data stream into frequency domain data by performing discrete sine or discrete cosine transform; performing algebraic operations on the frequency domain data to directly obtain frequency domain depth data; and generating a three-dimensional surface topography data stream in the spatial domain by performing inverse discrete sine or discrete cosine transform on the frequency domain depth data.

[0029] Therefore, the application scope of this invention can cover multiple fields such as intelligent manufacturing, healthcare, consumer electronics, and robotics, solving the perception bottleneck problems encountered by various industries during intelligent transformation. In the field of intelligent manufacturing, this invention can be integrated with visual and tactile sensors at the end effector of a robotic arm to perform three-dimensional surface defect detection on parts. The millisecond-level reconstruction speed can keep up with the operation of automated production lines, while its low power consumption makes it suitable for deployment in compact industrial robot control cabinets. In the field of healthcare, this invention can be integrated into the end effector of surgical instruments for application in minimally invasive surgery or telemedicine, providing doctors with real-time three-dimensional deformation and texture information of the surgical area to assist doctors in precise operations and achieve true tactile feedback. In the field of consumer electronics, this invention can be embedded in smartphones, tablets, or AR / VR devices to achieve low-power, real-time 3D object scanning and facial recognition, providing core technical support for mobile 3D content creation, augmented reality games, and biometric payments. In the field of robotics, this invention can improve the perception speed of a robot's dexterous hand when grasping unknown objects, increasing the perception process from milliseconds to microseconds, enabling the robot to handle fragile, deformable, or smooth objects, and significantly improving the stability and safety of operation.

[0030] like Figure 1 As shown, the gradient generation module 110 receives the original image data stream, generates surface normal vectors through a pre-constructed illumination-normal mapping model, and converts the surface normal vectors into a gradient field data stream. It is worth noting that this invention does not limit the color format and bit depth of the original image data stream. The original image data stream can be in RAW format, RGB format, YUV format, or any other custom format, and the bit depth of each channel can be 8 bits, 10 bits, 12 bits, or higher. The input interface of the gradient generation module 110 includes preprocessing logic to ensure that the quality of the input data meets the reconstruction requirements. This invention also does not limit the specific construction method of the illumination-normal mapping model. In one variant, the illumination-normal mapping model can be a physically based high-dimensional lookup table generation method, that is, by simulating the reflected light intensity under different normals through ray tracing, high-dimensional lookup table data is generated offline. As an example, the high-dimensional lookup table can be a color-normal lookup table. In another variation, the illumination-normal mapping model can be based on a data-driven learning method. This involves collecting a large amount of paired data from the light intensity of real objects to surface normal vectors, training a multilayer perceptron or convolutional neural network, and then embedding the trained network weights into a hardware inference engine. Regardless of the generation method used, as long as low-latency, deterministic output from the original image data stream to the surface normal vector can be achieved, it falls within the scope of this invention. The original image data stream contains multiple input pixels input sequentially. When the illumination-normal mapping model is a high-dimensional lookup table, the high-dimensional lookup table uses the combination of multi-channel light intensity values ​​of the input pixels as the address, outputting the surface normal vector corresponding to the input pixel. When the illumination-normal mapping model is a convolutional neural network, the surface normal vector is directly inferred using convolutional operation logic, converting the surface normal vector into a gradient field, and outputting the gradient field data stream. It is worth noting that this invention does not limit the specific implementation of the mapping model. As an example, the mapping model can be implemented using neural network operation logic, such as convolution or matrix multiplication.

[0031] like Figure 1As shown, the two-dimensional forward transform module 120 is connected to the gradient generation module 110 and is used to perform a discrete orthogonal transform on the gradient field data stream, which can be a discrete sine transform or a discrete cosine transform, thereby converting the gradient field data stream into frequency domain data. The two-dimensional forward transform module 120 includes at least a row transform unit, a transpose buffer unit, and a column transform unit. The row transform unit performs a one-dimensional discrete sine transform on each row of the gradient field data stream to obtain an intermediate data stream after row transformation. The transpose buffer unit stores the intermediate data stream. The column transform unit performs a one-dimensional discrete sine transform on each column of the intermediate data stream. It is worth noting that this invention does not impose specific limitations on the specific computational architecture of the discrete sine transform. As a variation, the two-dimensional forward transform module 120 can be directly implemented using a two-dimensional discrete sine transform IP core. As another variation, the discrete sine transform can also be indirectly implemented through a fast Fourier transform. Through pre / post-processing steps, the discrete sine transform is mapped to the computational core of the Fourier transform.

[0032] like Figure 1 As shown, the frequency domain solving module 130 is connected to the two-dimensional forward transform module 120 and is used to perform algebraic operations on the frequency domain data to directly obtain the frequency domain depth data. It is worth noting that this invention does not limit the specific operations of the frequency domain algebraic operations. The frequency domain solving module 130 can be understood as performing preset algebraic operation logic on the transform coefficients in the frequency domain. This algebraic operation logic can be pre-stored in a configuration register and dynamically configured through software, which makes this invention highly versatile. For example, the frequency domain solving module 130 may include a parallel dot product operation unit and an optional frequency domain filter. The dot product operation unit performs a dot product operation on the frequency domain data and the reciprocal of the pre-calculated Laplacian operator eigenvalues ​​using a hardware multiplier, thereby obtaining the frequency domain depth data.

[0033] like Figure 1As shown, the inverse transform and reconstruction module 140 is connected to the frequency domain solution module 130 and is used to perform a discrete inverse sinusoidal transform on the frequency domain depth data to generate a three-dimensional surface topography data stream in the spatial domain. The inverse transform and reconstruction module 140 includes at least a row inverse transform unit and a column inverse transform unit. The row inverse transform unit performs a one-dimensional discrete inverse sinusoidal transform on each row of the frequency domain depth data to obtain a row inverse transform data stream. The column inverse transform unit performs a one-dimensional discrete inverse sinusoidal transform on each column of the row inverse transform data stream. In one embodiment, the present invention can be applied to an intelligent visual-tactile sensor with an embedded depth reconstruction chip. The intelligent visual-tactile sensor directly outputs high-precision tactile surface deformation at the edge, achieving millisecond-level tactile feedback without the need for complex calculations by a host computer. The intelligent visual-tactile sensor includes at least a sensor assembly and a dedicated processing chip. It is worth noting that the present invention does not limit the specific composition, shape, quantity, and functional configuration of the sensor assembly. The number of components in the sensor assembly can be one or more. They can have the same function in the system, or they can complement each other or independently perform signal acquisition tasks of different dimensions. Specifically, the goal is to achieve real-time, high-fidelity acquisition of high dynamic range optical signals modulated by external force contact. Those skilled in the art can flexibly configure the physical structure and photoelectric parameters of the sensor assembly according to the actual application scenario. Preferably, the sensor assembly in this invention can be, for example, but not limited to: a flexible elastomer coated with a reflective coating, a three-color LED illumination array located inside the elastomer, and a miniature CMOS image sensor, wherein the three colors are red (R), green (G), and blue (B).

[0034] In this embodiment, the dedicated processing chip is edge computing hardware integrating the real-time 3D reconstruction system 100 of the present invention. It is worth noting that the present invention does not limit the specific physical form of the edge computing hardware; examples include, but are not limited to, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), system-on-a-chip (SoCs), image signal processors (ISPs), or tactile processing unit chips. In a preferred embodiment, to achieve low power consumption and miniaturized integration, an application-specific integrated circuit is preferred, directly flip-chip or surface-mounted onto the back of the sensor assembly via a PCB board, forming an integrated sensing and computing intelligent sensing module. Internally, this dedicated processing chip sequentially and pipelines a gradient generation module 110, a two-dimensional forward transform module 120, a frequency domain solution module 130, and an inverse transform and reconstruction module 140 through an on-chip streaming interface.

[0035] like Figures 2-3 As shown, the workflow of the intelligent visual-tactile sensor specifically includes the following steps: Step S1: Multispectral image acquisition to obtain the raw image data stream. A three-color LED illuminates the interior of the elastomer at a specific preset angle. When an external object contacts the surface of the elastomer, causing deformation, the sensor assembly simultaneously acquires a single-frame color image. The intensity changes of the R, G, and B color channels in this image encode the reflection information under different illumination directions.

[0036] Step S2: Hardware-based normal estimation. Surface normal vectors are generated using a pre-built illumination-normal mapping model, and then converted into a gradient field data stream. This invention does not impose specific restrictions on the specific implementation of the illumination-normal mapping model; its core lies in achieving a hard logic mapping from light intensity to surface normal vectors through a pre-built model. It is worth noting that this invention does not limit the dimension or addressing accuracy of the lookup table. A high-dimensional lookup table can be a three-dimensional lookup table based on RGB three channels, or it can be a higher-dimensional lookup table constructed by introducing more spectral channels or neighborhood pixel features, specifically based on achieving a unique and fast mapping from light intensity to surface normal vectors. As an example, when the lighting-normal mapping model is implemented using a high-dimensional lookup table, the on-chip memory within the dedicated processing chip pre-configures a color-normal lookup table (LUT). For each input pixel (r, g, b) with its multi-channel light intensity value combination, the hardware addressing logic directly concatenates these values ​​as an address bus signal, accesses the color-normal lookup table within one or several clock cycles, and outputs the corresponding surface normal vector (nx, ny, nz) in parallel. This process completely avoids the complex floating-point lighting equation calculations.

[0037] Step S3: Perform streaming gradient calculation. After obtaining the surface normal vector, the gradient (p, q) calculation logic in the gradient generation module 110 performs division operations p = -nx / nz and q = -ny / nz, outputting a two-dimensional gradient field data stream.

[0038] Step S4: Hardware Poisson solution based on discrete sine transform. For example... Figure 3 As shown, step S4 specifically includes the following sub-steps: Step S4-1: The gradient field data stream is converted into frequency domain data by the two-dimensional forward transform module 120. A two-dimensional forward transform is performed based on hardware Poisson solving using discrete sine transform. The gradient field data stream enters the two-dimensional forward transform module 120. Considering the solution characteristics of the Poisson equation under Dirichlet boundary conditions, the two-dimensional forward transform module 120 employs discrete sine transform. In one embodiment, to adapt to a streaming hardware processing architecture, the two-dimensional forward transform module 120 adopts a pipelined structure of row and column decomposition. First, the row transform unit performs a one-dimensional discrete sine transform on each row of input pixels, generating an intermediate data stream after row transform. Second, this intermediate data stream is stored in a transpose buffer unit, which can adopt a double buffer structure to achieve data write and read without waiting. Finally, the column transform unit reads data from the transpose buffer unit and performs a one-dimensional discrete sine transform on each column, thereby completing the two-dimensional discrete sine transform and converting the spatial domain gradient field data into frequency domain data.

[0039] Step S4-2: The frequency domain data is processed by the frequency domain solution module 130 using algebraic operations to directly obtain the frequency domain depth data. The frequency domain data stream enters the frequency domain solution module 130. This module utilizes the diagonalization property of the Poisson equation in the sinusoidal transform domain to transform the solution of complex partial differential equations into scalar algebraic operations. The frequency domain solution module 130 internally contains parallel dot product units and frequency domain filter banks. The dot product unit performs a dot product operation on the input frequency domain data and the reciprocal of the pre-calculated Laplace operator eigenvalues ​​using a hardware multiplier. This process is purely combinational logic or extremely short pipelined logic, and the solution can be completed without iteration, directly outputting the frequency domain depth data.

[0040] Step S5: Inverse Transformation and Reconstruction. The inverse transformation and reconstruction module 140 restores the frequency domain depth data into a three-dimensional surface topography data stream in the spatial domain. The frequency domain depth data stream enters the inverse transformation and reconstruction module 140. The inverse transformation and reconstruction module 140 performs a two-dimensional discrete inverse sinusoidal transform (2D-IDST). Symmetrical to the aforementioned two-dimensional forward transform module 120, the inverse transformation and reconstruction module 140 includes at least row inverse transform units and column inverse transform units, adopting a row-column decomposition pipeline structure to progressively restore the frequency domain data into a height map in the spatial domain, i.e., a three-dimensional surface topography data stream.

[0041] like Figure 3 As shown, specifically, step S5 includes the following sub-steps: Step S5-1: The row inverse transform unit performs a one-dimensional discrete sine inverse transform on each row of the frequency domain depth data to obtain the row inverse transform data stream.

[0042] Step S5-2: The row inverse transformation data stream is stored in the transpose buffer unit, and the data is transposed using the ping-pong buffer.

[0043] In step S5-3, the column inverse transform unit performs a one-dimensional discrete sine inverse transform on each column of the transposed data stream, ultimately generating a three-dimensional surface topography data stream in the spatial domain.

[0044] Step S6: Output Results. The final reconstructed 3D surface topography data stream is output directly to the robot's main controller through the standard communication interface of a dedicated processing chip. It is worth noting that the standard communication interface is, for example, but not limited to, SPI, I2C, or MIPI.

[0045] This embodiment achieves fully hardware-based streaming processing from optical signal input to 3D shape output. The entire process is completed within the sensor assembly, with end-to-end reconstruction latency below 30ms and typical power consumption controlled at the hundreds of milliwatts level. Compared to traditional solutions that rely on PC hosts for software Poisson calculations, this embodiment completely solves the problems of cable constraints, high latency, and high power consumption, significantly improving the tactile perception capabilities of the robot's dexterous hand.

[0046] In another embodiment, the present invention can also be applied to other algorithms that require 3D reconstruction using photometric stereo methods. For example, the present invention can be applied to computational photogrammetric 3D reconstruction equipment based on a dedicated depth reconstruction intellectual property core. The present invention does not limit the specific type of 3D reconstruction equipment; it can be handheld or embedded. The 3D reconstruction equipment utilizes a 3D reconstruction system to achieve real-time 3D reconstruction of object surfaces. The computational photogrammetric 3D reconstruction equipment includes at least an imaging module and a computing platform. The imaging module contains a high-resolution camera and an externally controlled programmable ring light source. The computing platform is based on an embedded motherboard of a system-on-a-chip, on which the hardware intellectual property core of the present invention is programmed.

[0047] It is worth noting that this invention does not limit the specific configuration, number of light sources, arrangement geometry, or illumination function mode of the programmable light source device. The number of programmable ring light sources can be one or more, and the arrangement of multiple light source units in space can be a complete ring, a non-complete arc, a rectangular array, a multi-ring stacked ring structure, or any other preset geometric topology. The function of each light source unit can be the same, for example, all providing single-color-temperature white light illumination, or it can be different, for example, some units are responsible for providing the main illumination direction, some units are responsible for filling shadow areas, and some units are used to emit structured light encoding information. Specifically, it should be able to provide at least two distinguishable, calibrable, and time-controllable multi-directional incident light illuminating methods for the target scene or contact surface. Preferably, the programmable ring light source of this invention can be eight monochromatic LEDs in different orientations. This invention also does not limit the delivery form of the hardware intellectual property core, which can be register-transfer level code, netlist files, or synthesizable hard macros, specifically based on its ability to be integrated into the target chip or logic unit.

[0048] like Figure 4 As shown, the workflow of the computational photogrammetry 3D reconstruction device in this embodiment specifically includes the following steps: Step I: Time-series illumination acquisition using photometric stereo method. The system controls a programmable ring light source to perform high-speed time-series switching, and coordinates with the camera to synchronously trigger shooting, thereby acquiring a sequence of multiple frames of images illuminated from the same viewpoint under different lighting directions. For example, this could be 4 to 8 images.

[0049] Step II: Preprocessing via normal estimation. The acquired image sequence is input to the gradient generation module 110. It should be noted that this invention does not impose specific limitations on the specific implementation of the normal estimation algorithm. For example, the specific algorithm implementation of the gradient generation module 110 can be least squares solution, high-dimensional lookup table mapping, or deep learning-based neural network inference, as long as it can achieve the goal of quickly generating surface normal vectors. This embodiment is compatible with various existing normal estimation techniques, aiming to quickly calculate and output the surface normal vector data (NormalMap) for each pixel as input to the subsequent hardware solver.

[0050] Step III: High-Resolution Streaming Poisson Solution. For high-resolution (e.g., 4K / 8K) applications involving industrial scanning or computational photography, the gradient field data stream corresponding to the normal vector is accelerated by the Discrete Sine Transform (DST) or Discrete Cosine Transform (DCT) hardware acceleration engine at the core of this invention. The Discrete Sine Transform hardware acceleration engine employs a deeply pipelined optimization design for large-size matrix operations, fully utilizing the DSP Slice resources within the field-programmable gate array (FPGA) for high-concurrency frequency domain data processing, effectively overcoming the computational bottleneck faced by general-purpose processors at high resolutions. It is worth noting that this invention does not limit the fixed-point bit width of the Discrete Sine Transform. In the FPGA implementation, to balance resource consumption and reconstruction accuracy, the transform core can be quantized using 16-bit or 24-bit fixed-point numbers, and the reciprocal of the pre-calculated eigenvalues ​​is also stored using the same bit width.

[0051] Step IV: 3D Model Generation and Output. After processing by the inverse transformation and reconstruction module 140, the frequency domain depth data is restored in real time to a continuous and smooth height field in the spatial domain. The real-time 3D reconstruction system 100 can further convert this depth map into a 3D point cloud or triangular mesh model, and perform texture mapping in conjunction with the original image, ultimately outputting a complete 3D reconstruction result.

[0052] In summary, this invention can generate high-precision 3D models in real time on-site without transmitting massive amounts of original images to the cloud or workstations for processing. Compared to traditional CPU-based software algorithms, this embodiment improves 3D reconstruction speed by one to two orders of magnitude while maintaining reconstruction accuracy and significantly reduces device battery consumption, making it ideal for scenarios such as mobile 3D scanning, facial recognition payment terminals, and defect detection of industrial parts.

[0053] From a technological standpoint, Poisson surface reconstruction is a key computational method connecting discrete perceptual data with continuous geometric models, possessing irreplaceable application value in cutting-edge fields such as computer vision, tactile perception, and robot interaction. This technology can reconstruct globally optimal closed surfaces based on noisy or even non-uniformly sampled normal vector field data by solving the Poisson equation, playing a crucial role in addressing data gaps and geometric topology repair.

[0054] Specifically, the algorithm of this invention can process volumetric mesh data, such as obtaining depth from 2D images. In the field of tactile interaction, especially in the application of visual-tactile sensors, the system first calculates the collected light intensity distribution into a surface normal vector field based on the photometric stereo method. Then, it must be solved using the Poisson equation to accurately estimate the three-dimensional deformation and texture details of the contact surface. Therefore, the solution of Poisson surface reconstruction directly determines the perception accuracy of the robot's fine manipulation.

[0055] Although the algorithms for solving the Poisson equation are theoretically mature, current engineering implementations still heavily rely on general-purpose computing platforms such as CPUs or GPUs. This computing model faces severe challenges in practical deployment. First, there are bottlenecks in power consumption and energy efficiency. The high power consumption of general-purpose processors makes them unsuitable for mobile robots, dexterous hands, or wearable haptic devices with extremely high battery life requirements. Second, there is the conflict between real-time performance and edge computing. In large-scale high-resolution reconstruction tasks, the computational latency of embedded processors cannot meet the millisecond-level response requirements of closed-loop control, while deploying them in the cloud is limited by bandwidth and communication latency. Therefore, the high power consumption and large size of existing general-purpose hardware architectures make them difficult to integrate as embedded modules at the edge, severely limiting the application of high-precision 3D reconstruction technology in portable and low-power scenarios.

[0056] Therefore, the present invention has conducted certain research, such as... Figures 1-4As shown, this invention achieves lower end-to-end reconstruction latency and system power consumption, thus filling the application gap of high-precision 3D reconstruction technology in edge computing and closed-loop control. Specifically, this invention adopts a full-stream hardware data processing structure. From the input of the original image data stream to the output of the final 3D surface topography data stream, data is directly transferred between the gradient generation module 110, the 2D forward transform module 120, the frequency domain solution module 130, and the inverse transform and reconstruction module 140 through an on-chip streaming interface. This eliminates the need to frequently write intermediate calculation results back to off-chip memory or repeatedly read them, fundamentally reducing the huge overhead generated during data storage and access. The streaming data processing structure adopted in this invention allows modules at each level to work in parallel: while a subsequent module is processing the current data block, a preceding module can simultaneously receive and process new data. This architecture significantly shortens the end-to-end reconstruction latency of the real-time 3D reconstruction system 100, fully meeting the stringent real-time requirements of robot tactile feedback and dexterous hand grasping control, thus playing a crucial role in practical applications. Meanwhile, by greatly reducing the reliance on off-chip memory, the system power consumption of this invention under typical workloads can be controlled at the level of hundreds of milliwatts, which is only a fraction of that of traditional GPU solutions. This makes it possible to integrate high-precision 3D reconstruction capabilities into battery-powered portable devices, mobile robots, and wearable sensors.

[0057] Meanwhile, the hardware logic design of this invention possesses strong versatility, scalability, and portability, laying a solid foundation for the large-scale industrial application of the technology. The core of the real-time 3D reconstruction system 100 of this invention lies in a series of modular hardware logic descriptions, including a hardware intellectual property core. This design gives the real-time 3D reconstruction system 100 inherent cross-platform adaptability, allowing it to be prototyped and deployed first on a field-programmable gate array (FPGA). Because FPGAs are reconfigurable and have short development cycles, developers can quickly verify the functionality and performance of this invention on an FPGA and perform parameter optimization for specific application scenarios, meeting the needs of small-batch, diversified, or scientific research applications. The clear modular interfaces and pipelined design of this invention enable the hardware intellectual property core to be integrated into more complex on-chip system designs in a plug-and-play manner. For example, it can be integrated as an independent computing unit into the pipeline backend of an image signal processor (Image Signal Processor) to directly reconstruct 3D information from the image data stream output by the Image Signal Processor; it can also be integrated into a robot main control chip as a coprocessor dedicated to processing tactile or visual information. This portability means that there are virtually no technical barriers to the leap from field-programmable gate array (FPGA) prototypes to application-specific integrated circuit (ASIC) mass production, enabling rapid response to market demands ranging from scientific research to large-scale distribution of consumer electronics products.

[0058] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0059] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A real-time three-dimensional reconstruction system, characterized in that, It adopts a real-time streaming hardware acceleration architecture, including: The gradient generation module is used to receive the raw image data stream, generate surface normal vectors through a pre-built illumination-normal mapping model, and convert the surface normal vectors into a gradient field data stream. A two-dimensional forward transform module, connected to the gradient generation module, is used to perform a discrete orthogonal transform on the gradient field data stream, converting the gradient field data stream into frequency domain data; The frequency domain solution module is connected to the two-dimensional forward transform module and is used to perform algebraic operations on the frequency domain data to obtain frequency domain depth data. The inverse transformation and reconstruction module, connected to the frequency domain solution module, is used to perform a discrete orthogonal inverse transformation on the frequency domain depth data to generate a three-dimensional surface topography data stream in the spatial domain.

2. The real-time three-dimensional reconstruction system according to claim 1, characterized in that, The two-dimensional forward transformation module includes: The row transformation unit performs a one-dimensional discrete orthogonal transformation on each row of the gradient field data stream to obtain the intermediate data stream after row transformation. A transpose buffer unit stores the intermediate data stream; The column transformation unit performs a one-dimensional discrete orthogonal transformation on each column of the intermediate data stream.

3. The real-time three-dimensional reconstruction system according to claim 1, characterized in that: The frequency domain solution module includes a parallel dot product operation unit and a frequency domain filter. The dot product operation unit performs algebraic operations on the frequency domain data and the spectral feature parameters based on the Laplacian operator in the transform domain through a hardware multiplier, thereby obtaining the frequency domain depth data.

4. The real-time three-dimensional reconstruction system according to claim 1, characterized in that, The inverse transformation and reconstruction module includes: The row inverse transform unit performs a one-dimensional discrete orthogonal inverse transform on each row of the frequency domain depth data to obtain the row inverse transform data stream; The column inverse transformation unit performs a one-dimensional discrete orthogonal inverse transformation on each column of the row inverse transformation data stream.

5. The real-time three-dimensional reconstruction system according to claim 1, characterized in that: The illumination-normal mapping model can be any one of the following: a theoretical mathematical model, a high-dimensional lookup table, or a neural network. The theoretical mathematical model is constructed directly through circuits to generate mathematical formulas, and the surface normal vector is obtained by mapping pixel by pixel.

6. The real-time three-dimensional reconstruction system according to claim 5, characterized in that, The implementation steps of the high-dimensional lookup table include: The original image data stream contains multiple input pixels input in sequence. When the illumination-normal mapping model is the high-dimensional lookup table, the high-dimensional lookup table uses the combination of multi-channel light intensity values ​​of the input pixels as the address and outputs the surface normal vector corresponding to the input pixels.

7. The real-time three-dimensional reconstruction system according to claim 5, characterized in that, The implementation steps of the neural network include: After approximating the illumination-normal mapping model, the surface normal vector is directly inferred using computational logic. The surface normal vector is then converted into a gradient field, and the gradient field data stream is output. The computational logic is implemented through formula deduction, numerical lookup, or neural network prediction. The method for converting the surface normal vector into a gradient field data stream is as follows: After obtaining the surface normal vector, the gradient generation module outputs the gradient field data stream by performing division operations p = -nx / nz and q = -ny / nz. The gradient (p, q) and the surface normal vector (nx, ny, nz) are mentioned.

8. The real-time three-dimensional reconstruction system according to claim 1, characterized in that: The gradient generation module, the two-dimensional forward transform module, the frequency domain solution module, and the inverse transform and reconstruction module are connected via on-chip streaming interfaces.

9. A design method for a real-time three-dimensional reconstruction system, characterized in that, The design methodology for a real-time 3D reconstruction system, used to accelerate the implementation of intellectual property cores, includes the following steps: Receive the raw image data stream, generate surface normal vectors through a pre-constructed illumination-normal mapping model, and convert the surface normal vectors into a gradient field data stream; The gradient field data stream is converted into frequency domain data by performing a discrete orthogonal transform. Perform algebraic operations on the frequency domain data to obtain the frequency domain depth data; By performing a discrete orthogonal transformation on the frequency domain depth data, a three-dimensional surface topography data stream in the spatial domain is generated.

10. An edge computing hardware, characterized in that, include: The real-time three-dimensional reconstruction system according to any one of claims 1 to 8, Edge computing hardware includes application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or image signal processors.

Images