3D morphological reconstruction system based on active light shadow

Abstract

The application provides a 3D shape reconstruction system based on active light and shadow, which does not depend on the texture of an object. The system controls a small point light source at the end of a mechanical arm, moves on the surface of a target object and projects clear shadows or highlights. Another fixed camera captures the dynamic changes of the light and shadow features, and through a learning model, the surface normal and depth information of the object which is invisible to both the light source and the camera are inverted. Passive observation is changed to active detection, and the dynamic light and shadow are projected by the mechanical arm to analyze the three-dimensional geometry of the textureless or highly reflective object. Without relying on the texture of the object itself, the reconstruction problem of structured light and multi-view stereo vision on textureless objects is solved. The active movement of the mechanical arm driving the point light source increases the dimension of light and shadow information from 2D to 3D, provides more abundant constraint conditions and significantly improves the inversion accuracy of the normal and depth. The double-branch deep learning model based on the attention mechanism, combined with bilateral filtering and guided filtering post-processing, improves the reconstruction accuracy.

Description

Technical Field

[0001] This invention relates to the field of 3D optical reconstruction technology, and in particular to a 3D morphology reconstruction system based on active lighting and shadow, a 3D morphology reconstruction method based on active lighting and shadow, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Existing 3D shape reconstruction technologies mainly rely on the object's own texture (such as structured light, multi-view stereo vision) or passive lighting and shadows (such as monocular / binocular shadow shape recovery), but they have significant limitations on textureless objects (such as plastic and ceramic) or highly reflective objects (such as metal and glass).

[0003] Texture dependency problem: Structured light extracts texture features by projecting encoded patterns, but textureless objects cannot effectively reflect the patterns, resulting in feature point matching failure; multi-view stereo vision requires texture information from different angles, and it is difficult to establish the correspondence between views of textureless objects.

[0004] Limitations of passive lighting and shadows: Traditional shadow shape recovery (SFS) relies on static shadows from a fixed light source, which has limited information dimensions and is difficult to analyze complex curved surfaces; the specular reflection of highly reflective objects can overwhelm shadow features, resulting in large depth estimation errors.

[0005] Insufficient flexibility: Existing active light sources (such as structured light projectors) are mostly in fixed positions, which cannot adapt to the surface coverage requirements of objects of different shapes. For large or irregularly shaped objects, the coverage of light and shadow information is incomplete. Summary of the Invention

[0006] To address the technical problems existing in the prior art, the present invention provides the following technical solution:

[0007] On the one hand, a 3D shape reconstruction system based on active lighting and shadow is provided, including:

[0008] A robotic arm with a movable point light source at its end;

[0009] A camera is fixed relative to the point light source and is used to capture the light and shadow image of the surface of an object illuminated by the point light source.

[0010] The processor, which is communicatively connected to the robotic arm and the camera, is configured to:

[0011] The robotic arm is controlled to move the point light source along a preset trajectory to project dynamically changing shadows and / or highlights onto the surface of the object.

[0012] Control the camera to synchronously acquire a sequence of light and shadow images of the object's surface;

[0013] Process the light and shadow image sequence to extract light and shadow features;

[0014] The light and shadow features and the corresponding point light source position information are input into a pre-trained deep learning model to obtain the normal map and depth map of the object surface.

[0015] A three-dimensional morphological model of the object is generated based on the normal map and depth map.

[0016] Preferably, the robotic arm is a six-degree-of-freedom industrial robotic arm, the point light source is a high-brightness LED light source with a spot diameter of less than or equal to 5 mm, and the camera is an industrial camera that supports external triggering mode. It is connected to the robotic arm through a hardware synchronization signal to achieve strict synchronization between light and shadow acquisition and the movement of the light source position.

[0017] Preferably, when controlling the robotic arm, the processor is specifically configured to: generate a trajectory that avoids object boundaries using a motion planning method based on a random tree expansion algorithm, and perform real-time closed-loop control of the robotic arm using a proportional-integral-derivative controller to ensure the motion accuracy of the point light source.

[0018] Preferably, when extracting light and shadow features, the processor is specifically configured to: preprocess the acquired image, use an edge detection algorithm to extract the shadow boundary information of the object surface, and use a scale-invariant feature transformation algorithm to extract the highlight feature point information of the object surface.

[0019] Preferably, the processor is further configured to: encode the extracted shadow boundary information and highlight feature point information into a first feature map and a second feature map, respectively; expand the three-dimensional position coordinates of the point light source in the preset trajectory into three coordinate maps of the same size as the first feature map; and merge the first feature map, the second feature map, and the three coordinate maps into a multi-channel input tensor as the input of the deep learning model.

[0020] Preferably, the pre-trained deep learning model is a dual-branch network model that includes an attention mechanism, comprising a shared feature extraction layer, an independent normal prediction branch, an independent depth prediction branch, and an attention module for fusing features from both branches.

[0021] Preferably, the processor is further configured to: perform bilateral filtering on the normal map output by the deep learning model and guided filtering on the output depth map to remove noise and preserve geometric edges.

[0022] Preferably, when the processor generates a three-dimensional morphological model based on the normal map and the depth map, it is specifically configured to: convert the two-dimensional coordinates of each pixel and its corresponding depth value into a three-dimensional point cloud based on the camera imaging model, wherein the normal map provides the normal vector information of each point in the three-dimensional point cloud.

[0023] Preferably, it also includes a user interface connected to the processor, for receiving trajectory parameters, camera parameters or model parameters input by the user, and for displaying the three-dimensional morphological model in real time.

[0024] On the other hand, a 3D shape reconstruction method based on active lighting and shadow is provided, which is implemented based on the above system and includes the following steps:

[0025] S1: Control the robotic arm to drive the point light source at its end to move on the surface of the target object according to a preset trajectory, so as to project dynamically changing shadows and / or highlights;

[0026] S2: While the point light source moves, a fixed camera is used to synchronously acquire a sequence of light and shadow images of the surface of the target object.

[0027] S3: Process the light and shadow image sequence and extract the light and shadow features of the target object surface;

[0028] S4: Input the light and shadow features and the position information of the point light source during the movement into the pre-trained deep learning model to obtain the normal map and depth map of the target object surface;

[0029] S5: Reconstruct the three-dimensional shape of the target object based on the normal map and depth map.

[0030] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0031] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0032] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0033] The core advantages of this solution compared to the prior art are as follows:

[0034] 1. Texture-free: Dynamic features are obtained through active lighting and shadow projection without relying on the object's own texture, solving the reconstruction problem of structured light and multi-view stereo vision on textureless objects.

[0035] 2. Active Information Enhancement: The robotic arm drives the point light source to move actively, enhancing the dimension of light and shadow information from 2D to 3D, providing richer constraints, and significantly improving the accuracy of normal and depth inversion.

[0036] 3. High adaptability: Adaptive trajectory planning can cover objects of different shapes and sizes, and is especially suitable for the reconstruction of highly reflective (metal, glass) and complex curved surface objects.

[0037] 4. Real-time and efficient: The external trigger synchronization mechanism ensures that the light and shadow acquisition and the position of the light source are strictly corresponded. GPU-accelerated inference enables real-time reconstruction with a processing speed of up to 30fps, meeting the needs of industrial applications.

[0038] 5. High-precision output: The dual-branch deep learning model based on the attention mechanism, combined with bilateral filtering and guided filtering post-processing, achieves a reconstruction accuracy of ±0.02mm, which is superior to traditional passive lighting methods. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the system deployment architecture provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the hardware system composition structure provided in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart of the data interaction and synchronization mechanism provided in the embodiments of the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0045] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0046] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0047] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0048] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0049] This system does not rely on the object's own texture. It controls a small point light source at the end of a robotic arm to move across the target object's surface and project sharp shadows or highlights. Another fixed camera captures the dynamic changes in these lighting features, and a learning model inversely retrieves the object's surface normals and depth information, which are invisible to both the light source and the camera. This transforms passive observation into active detection, using the robotic arm to project dynamic light and shadow to resolve the 3D geometry of textureless or highly reflective objects.

[0050] This invention proposes a 3D shape reconstruction system based on active lighting and shadow. A robotic arm drives a point light source to actively move, projecting dynamic lighting and shadow. A fixed camera captures changes in lighting and shadow characteristics, and a deep learning model is used to invert surface normals and depth. This system transforms passive observation into active detection, does not rely on object texture, and can effectively reconstruct the 3D geometry of textureless / highly reflective objects. It boasts advantages such as high flexibility, high accuracy, and strong generalization ability.

[0051] The following section will provide a detailed introduction to the system's composition and its software algorithms.

[0052] I. System Architecture

[0053] This system adopts an architecture of "active projection by robotic arm + passive capture by camera + deep learning inversion," and is divided into three core layers: hardware, software, and algorithm (e.g., Figure 1 (as shown)

[0054] Hardware layer: includes robotic arm system (driving the movement of point light source), vision perception system (capturing light and shadow images), control system (coordinating hardware to work together), and power and communication module (providing power and data transmission).

[0055] The software layer includes robotic arm control software (the core of which is the RRT-Connect motion planning algorithm and PID real-time control to achieve obstacle avoidance trajectory generation and high-precision position tracking), vision acquisition software (based on external trigger acquisition from Basler Pylon SDK and OpenCV preprocessing to achieve low-latency capture and noise reduction of light and shadow images), light and shadow reconstruction algorithm software (based on a dual-branch deep learning model with attention mechanism to achieve joint inversion of normal and depth), and user interaction software (using an OpenGL visualization engine and Qt graphical interface to support real-time parameter adjustment and 3D model interaction).

[0056] Algorithm layer: The core consists of active lighting and shadow projection strategy, lighting and shadow feature extraction, and attention-based normal-depth joint inversion model to realize the mapping from lighting and shadow data to 3D geometry.

[0057] II. Hardware System

[0058] like Figure 2 As shown, the hardware system is the physical basis for active light and shadow detection and needs to meet the requirements of high precision, high synchronization, and high flexibility. The specific components are as follows:

[0059] 1. Robotic arm system

[0060] Function: Drives point light sources to move along a planned trajectory to achieve active light and shadow projection.

[0061] Components and parameters:

[0062] Robotic arm body: ABB IRB 1200 six-degree-of-freedom industrial robotic arm was selected, with a load capacity of 5kg (meeting the weight requirements of the point light source + end effector), repeatability of ±0.02mm (ensuring the accuracy of the light source position), and a working space radius of 1.45m (covering small and medium-sized objects).

[0063] End effector: Custom lightweight fixture made of aluminum alloy, weighing ≤0.5kg, used to fix point light source; the fixture is equipped with a fine adjustment mechanism (±1mm stroke) to adjust the direction of the light spot.

[0064] Point light source: Select OSRAM LZ4-00R100 high-brightness LED, spot diameter ≤5mm (small spot ensures clear light and shadow features), brightness 1000lm (high brightness penetrates highly reflective surfaces), color temperature 6500K (close to natural light, reducing color deviation), DC drive (12V / 0.5A), lifespan ≥50000 hours.

[0065] Functional Description: The high repeatability of the robotic arm is key to active light and shadow projection, ensuring that the position of each light source is consistent with the planned trajectory; small-spot LEDs can project clear shadow / highlight boundaries, avoiding feature extraction errors caused by light and shadow blur.

[0066] 2. Visual perception system

[0067] Function: Captures dynamic changes in light and shadow on the surface of an object and extracts shadow / highlight features.

[0068] Components and parameters:

[0069] Industrial camera: Basler acA1280-20gm, 12 megapixels (4096×3072), 20fps (meets the needs of dynamic light and shadow capture), 1.1-inch sensor size (large sensor improves image quality in low light environment), 3.75μm pixel size (high pixel density improves detail resolution), supports external triggering (to ensure synchronization with robotic arm).

[0070] Lens: Select Basler C125-0618-5M prime lens, focal length 6mm (medium angle of view covering the whole object), aperture F1.8 (large aperture increases light intake), distortion ≤0.1% (reduces the impact of image distortion on feature extraction), working distance 0.3-2m (adapts to different object sizes).

[0071] Image acquisition card: Basler Pylon Capture Card is selected, which supports USB 3.0 interface (transmission rate of 5Gbps, which meets the requirements of real-time transmission of high-resolution images), supports external triggering (receiving synchronization signals from the robotic arm), and has a built-in FPGA (for preprocessing images and reducing the load on the main controller).

[0072] Function Description: The high resolution and high frame rate of the industrial camera ensure that subtle changes in light and shadow are captured; the low distortion characteristics of the fixed-focus lens avoid distortion of shadow / highlight boundaries and improve the accuracy of feature extraction.

[0073] 3. Control System

[0074] Function: Coordinates the collaborative work of the robotic arm, camera, and point light source to achieve synchronization of motion and data acquisition.

[0075] Components and parameters:

[0076] Main controller: Advantech IPC-610L industrial computer, Intel Core i7-10700 (8 cores and 16 threads, meeting the needs of deep learning inference), 16GB DDR4 memory (caching image and feature data), 512GB SSD (storing model and data), operating system Windows 10 IoT (stable industrial-grade system).

[0077] Robotic arm controller: Adopts ABB IRC5 controller, supports Ethernet / IP protocol (for communication with the main controller), has built-in motion planning algorithm (RRT-Connect), and real-time control cycle of 1ms (to ensure motion accuracy).

[0078] Vision controller: Integrated into the image acquisition card, using a Xilinx Zynq-7000 FPGA (handling image preprocessing and feature extraction, reducing the load on the main controller), and supporting TCP / IP protocol (for communication with the main controller).

[0079] Functional Description: The main controller acts as the system's central hub, sending motion trajectory commands (including joint angle sequences and velocity parameters) to the robotic arm controller via the Ethernet / IP protocol. The robotic arm controller provides real-time feedback on position errors through joint encoders (10,000 pulses / revolution resolution), and the main controller dynamically adjusts the commands based on a PID algorithm. Simultaneously, the main controller sends a data acquisition trigger signal (a 5V pulse synchronized with the robotic arm's movement) to the vision controller via the TCP / IP protocol. The vision controller then transmits pre-processed light and shadow feature data (shadow boundary map, highlight feature point map) back to the main controller at a rate of 1Gbps. The robotic arm controller and vision controller directly transmit synchronization pulses (delay ≤1ms) through the GPIO interface, ensuring strict alignment between the motion position and light and shadow acquisition, and preventing data misalignment.

[0080] 4. Power Supply and Communication Module

[0081] Function: Provides stable power and enables data transmission between hardware components.

[0082] Components and parameters:

[0083] Power supply: Select Mean Well MW-200-12 switching power supply, output 12V / 16.7A (powering the robotic arm end effector and point light source); Advantech PS-250-12 power supply, output 12V / 20.8A (powering the main controller, camera, and image acquisition card), both with overvoltage and overcurrent protection.

[0084] Communication: The robotic arm controller and the main controller use the Ethernet / IP protocol (transmission rate 100Mbps, latency ≤1ms) to transmit joint angle commands (32-bit floating-point type, sampling frequency 1kHz) and position feedback data (16-bit integer type, resolution 10000 pulses / revolution) in real time; the image acquisition card and the main controller use the TCP / IP protocol (transmission rate 1Gbps, MTU=1500 bytes) to transmit preprocessed light and shadow feature data (shadow boundary map is a 1080×1920 single-channel binary stream, and highlight feature points are N×2 floating-point coordinate arrays); the synchronization signal between the robotic arm and the camera uses 5VTTL pulses (frequency 30Hz, duty cycle 50%, rising edge triggered), directly connected through the GPIO interface, with a latency ≤0.1ms, ensuring that the light source position is strictly aligned with the image frame.

[0085] Functional Description: A stable power supply is the foundation for normal hardware operation, avoiding motion errors of the robotic arm or blurry camera images caused by voltage fluctuations; a low-latency communication protocol ensures real-time transmission of synchronization signals and image data.

[0086] The connection and interaction methods between hardware and software are as follows:

[0087] hardware components Corresponding software module Physical connection method Communication protocol / interface Core data flow ABB IRB1200 robotic arm body robotic arm control software Ethernet cable Ethernet / IP Software → Robotic arm controller: trajectory commands; Robotic arm → Software: joint position feedback, synchronization pulse signals End effector + point light source Robotic arm control software (fine-tuning module) GPIO signal lines Analog signal output Software → End effector: Fine-tuning commands for light spot direction; Point light source → Software: Status feedback (brightness, temperature) Baslerac A1280 camera Visual acquisition software USB 3.0 cable USB 3.0 Camera → Capture Card: Raw Image; Software → Camera: Exposure Time, Frame Rate Settings Basler data acquisition card Visual acquisition software (preprocessing module) TCP / IP network cable TCP / IP Acquisition card → Main controller: Preprocessed image, light and shadow feature map; Software → Acquisition card: Filtering parameters, feature thresholds Advantech IPC-610L Main Controller Light and shadow reconstruction algorithm software, user interaction software PCIe slot (GPU) CUDA (GPU acceleration) Main controller → Model: Fusion of input tensors; Model → Main controller: Normal / depth map; Main controller → Interactive software: 3D point cloud ABB IRC5 robotic arm controller Robotic arm control software (real-time control module) Ethernet cable Ethernet / IP Software → Controller: PID parameters; Controller → Software: Joint encoder feedback signal

[0088] III. Software System

[0089] The software system is the "brain" of hardware collaboration, and needs to realize functions such as motion control, image acquisition, feature extraction, and model inference. The specific modules are as follows:

[0090] 1. Robotic arm control software

[0091] Function: To realize motion planning, trajectory generation and real-time control of the robotic arm.

[0092] Module:

[0093] Motion planning module: The RRT-Connect algorithm is adopted. This algorithm generates a feasible path when two trees meet by bidirectionally expanding a random tree (generating trees simultaneously from the starting point S and the target region G). For the object boundary (obtained from the user-input CAD model or pre-scanned point cloud), the algorithm adds collision detection (fast intersection test based on AABB bounding boxes) when expanding the tree to ensure that the minimum distance between the trajectory and the object surface is ≥0.05m. The technical advantages are that the convergence speed is 2-3 times faster than the traditional RRT, it can handle the high-dimensional space obstacle avoidance of 6-DOF robotic arms, and it is suitable for trajectory planning of irregularly shaped objects.

[0094] The trajectory generation module converts the planned path into a trajectory that the robotic arm can execute. It uses a cubic spline interpolation algorithm to ensure smooth motion (acceleration ≤ 0.5 m / s²) and avoids light source position deviation caused by sudden stops.

[0095] Real-time control module: Employs a PID controller to calculate the control output of the joint motor. The core formula is:

[0096] ,

[0097] in:

[0098] : The output current of the motor at time t (unit: amperes, A). This is the final output of the controller, which directly acts on the motor driver.

[0099] : Proportional gain. Here it is set to 0.5, determining the strength of the response to the current error. A larger value results in a faster response, but may lead to overshoot or oscillation.

[0100] The position error at time t (unit: millimeters, mm) is defined as:

[0101] ,in For the target location, This represents the actual position fed back by the encoder.

[0102] Integral gain. Here it is set to 0.1, which determines the system's ability to eliminate historical accumulated errors (i.e., steady-state errors).

[0103] Position error from the initial moment to the current time t The integral over time. It accumulates all past errors and is used to eliminate steady-state deviations caused by constant disturbances such as load.

[0104] Differential gain. Here, it is set to 0.02, determining the damping effect on the error variation trend. It is used to predict future error changes, thereby suppressing overshoot and increasing system stability.

[0105] Position error The first derivative with respect to time, i.e., the rate of change of the error (unit: mm / s), reflects whether the error is increasing or decreasing and the speed at which it is decreasing.

[0106] Operating mechanism: The proportional term quickly responds to position errors, the integral term accumulates errors to eliminate steady-state deviations (such as drift caused by changes in the robot arm load), and the derivative term predicts the trend of error changes to suppress overshoot; the actual position is fed back in real time through the joint encoder (resolution 10,000 pulses / revolution, converted to position accuracy ±0.001mm), and the controller updates the output every 1ms to ensure motion error ≤ ±0.01mm; the technical advantages are strong robustness, simple adjustment, and suitability for high-precision position tracking of industrial robot arms.

[0107] Synchronization signal module: Outputs a 5V pulse signal at each sampling point on the trajectory (0.1s interval) to trigger the camera to capture images, ensuring that the position of the light source is synchronized with the image acquisition.

[0108] Functional Description: The RRT-Connect algorithm has the advantage of quickly planning complex trajectories, making it suitable for irregularly shaped objects; cubic spline interpolation ensures smooth movement of the robotic arm and reduces light and shadow jitter; PID control achieves high-precision motion, ensuring that the position of the light source is consistent with the plan.

[0109] 2. Visual acquisition software

[0110] Function: Enables real-time image capture, preprocessing, and extraction of light and shadow features.

[0111] Module:

[0112] Image capture module: Based on Basler Pylon SDK, it adopts external trigger mode (receiving synchronization signals from the robotic arm). It captures one frame of image for each pulse signal received and stores it in BMP format (uncompressed, retaining the original information).

[0113] Preprocessing module:

[0114] Noise reduction: Gaussian filtering is used (a two-dimensional Gaussian function (kernel) is used to generate the weights of each neighboring pixel during filtering). The core formula is:

[0115] ,

[0116] in, This represents the Gaussian kernel function value (weight) at the relative coordinates (x, y). It is a positive real number representing the degree of influence of the center pixel on pixels at that location within its neighborhood. The sum of values ​​(within the effective neighborhood) is usually normalized to 1.

[0117] Standard deviation of the Gaussian distribution (in pixels). Here it is set to 1.4. It controls the "smoothing strength" or "blur radius" of the filter. The larger the Gaussian kernel, the "flatter" it is, and the stronger the smoothing effect. The smaller the Gaussian kernel, the "sharper" it is, and the weaker the smoothing effect.

[0118] The integer coordinate offset (in pixels) of any pixel in the neighborhood relative to the center pixel. Typically, this is calculated within a finite window, for example... ∈[−3 ,3 ].when When the value is 1.4, the commonly used window size is 9×9 (that is, from -4 to 4, covering approximately 3). scope).

[0119] exp( ): Natural exponential function.

[0120] Brief description of the operating mechanism: For each pixel in the image, its surrounding N×N neighborhood (e.g., 9×9) is taken. The gray value of each pixel in the neighborhood is multiplied by the Gaussian weight corresponding to its coordinates. Then, all product results are summed to obtain the new grayscale value of the pixel after denoising. This process effectively smooths high-frequency noise, and because the weight distribution is continuous and centered, it can better preserve edge information. The advantages of this technique are good denoising effect and small edge loss, avoiding blurred shadow boundaries.

[0121] Distortion correction: Zhang Zhengyou calibration method is adopted, and the core camera projection model is (describing how a point in the three-dimensional world is projected onto the two-dimensional image plane under ideal distortion-free conditions):

[0122] ,

[0123] ,

[0124] in:

[0125] ( , , ): The three-dimensional coordinates of the point in the world coordinate system (unit: meters, m). This is the point's actual spatial location.

[0126] ( , ): The two-dimensional coordinates (in pixels) of the point in the image pixel coordinate system. This is the position of the point projected onto the image. u: Pixel column index (horizontal direction, usually in the range [0, width-1]); v: Pixel row index (vertical direction, usually in the range [0, height-1]).

[0127] ( , ): The focal length of the camera in the x and y directions (unit: pixels). The value here is... =1920, =1080. They convert lengths (meters) in world coordinates to pixel coordinates. Typically... and They are very close; if the camera pixels are square, then they are equal.

[0128] ( , ): The coordinates of the camera's principal point (optical center) in the pixel coordinate system (unit: pixels). Here, the value is taken as... =960, =540. Usually close to the image center.

[0129] Image distortion caused by lens imperfections, and how to calculate the actual distortion coordinates from ideal coordinates (positive distortion model). During correction, the actual distortion coordinates are derived from the observed coordinates (…). , The inverse solution for the ideal (u,v) is a nonlinear optimization process, and the distortion correction formula is:

[0130] ,

[0131] ,

[0132] ,

[0133] ,

[0134] in:

[0135] ( , () represents the actual, distorted pixel coordinates captured in the image;

[0136] The square of the distance from a point to the principal point on the normalized image plane is usually calculated in pixel coordinates as follows:

[0137] = + ;

[0138] In the normalized coordinate system, it is calculated as follows: = + .

[0139] , Radial distortion coefficient. The value here is... =0.12, =−0.34. These describe the distortion caused by changes in the radial curvature of the lens, resulting in a radial offset of image points (away from or near the center). Typically... The dominant value is positive for pincushion distortion and negative for barrel distortion.

[0140] , Tangential distortion coefficient (dimensionless). Here, the value is taken as... =0.001, =0.0005. These values ​​describe the distortion caused by the lens optical center not being parallel to the image plane, resulting in a tangential shift of image points.

[0141] Brief description of the operating mechanism (calibration process): Using a chessboard of known size (e.g., 0.02m × 0.02m squares), images of it are captured from multiple angles (e.g., 10 images). The algorithm automatically detects the pixel coordinates of the chessboard corner points in each image. , ), and combined with its known world coordinates ( , , (Usually, the checkerboard plane is set to Z=0). The optimal camera intrinsic parameters are solved iteratively using nonlinear optimization (such as the least squares method). , , , ) and distortion coefficient ( , , , This minimizes the error between the coordinates reprojected from the model and the actual observed coordinates. Once these parameters are obtained, distortion correction can be applied to any newly captured image.

[0142] Grayscale conversion: A weighted average method is used, with the core formula being Y=0.299R+0.587G+0.114B, where R / G / B are the pixel values ​​of the red / green / blue channels of the color image (range 0-255), and 0.299 / 0.587 / 0.114 are the sensitivity weights of the human eye to the three channels of light (the human eye is most sensitive to green, followed by red, and least sensitive to blue); Operating mechanism: The three channel values ​​of each pixel are summed according to their weights and converted into a single-channel grayscale value (range 0-255); The technical advantage is that it preserves the image brightness information, conforms to the characteristics of human vision, and reduces the computational load of subsequent processing (from 3 channels to 1 channel).

[0143] Feature extraction module:

[0144] Shadow features: Canny edge detection is used, with the following steps: ① Gaussian filtering for noise reduction → ② Sobel operator to calculate gradient magnitude and direction → ③ Non-maximum suppression (preserving local maxima in the gradient direction to refine edges) → ④ Dual threshold detection (min=50, max=150, preserving strong edges and connecting weak edges); output binary edge map (1 channel, boundary pixel value is 255, the rest are 0); the advantages of this technique are accurate edge localization, strong noise resistance, and suitability for extracting clear boundaries of shadows.

[0145] Highlight features: The SIFT algorithm is used, with the following steps: ① Scale-space extremum detection (constructing a Gaussian difference pyramid to find key points at multiple scales) → ② Key point localization (removing low-contrast and weak edge response points) → ③ Orientation assignment (calculating the gradient orientation histogram of the key point's neighborhood to determine the main orientation) → ④ Descriptor generation (extracting 128-dimensional gradient feature vectors around the key points); output feature point map (1 channel, feature point pixel value of 255, others 0); the technical advantages are scale invariance and rotation invariance, and stable matching of highlight feature points under different light source positions.

[0146] Function Description: The external trigger mode ensures a one-to-one correspondence between the image and the light source position; the preprocessing step removes image noise and distortion, improving the accuracy of feature extraction; Canny edge detection and SIFT algorithms are designed for the characteristics of shadows (clear edges) and highlights (obvious point features) respectively, achieving efficient feature extraction.

[0147] 3. Light and shadow reconstruction algorithm software

[0148] Function: Input the light and shadow features and light source position data into the deep learning model to invert the normal and depth of the object's surface.

[0149] Module:

[0150] Data preprocessing module: This module fuses the shadow edge map (1080×1920 single-channel binary image), the highlight feature point map (1080×1920 single-channel binary image), and the light source position data (world coordinates (x,y,z) at the end of the robotic arm). First, the light source coordinates are normalized to the range [0,1] (x'=x / 1.45, y'=y / 1.45, z'=z / 1.45, 1.45m is the radius of the robotic arm's workspace). Then, the normalized x' / y' / z' are used as three single-channel images (1080×1920) and superimposed on the shadow / highlight map to generate the model input tensor (size 1080×1920×5). The purpose is to allow the model to simultaneously acquire spatial light and shadow features and light source position information, establish the correlation between light and shadow changes and light source movement, and provide multi-dimensional constraints for normal-depth inversion.

[0151] Model Inference Module:

[0152] Model Construction: A dual-branch deep learning model based on an attention mechanism (implemented using the TensorFlow framework) is adopted. The structure includes a shared convolutional layer (3 convolutional blocks, extracting multi-scale lighting and shadow features), a normal branch (4 layers of convolution + deconvolution, outputting a 1080×1920×3 normal map), a depth branch (same structure as the normal branch, outputting a 1080×1920×1 depth map), and an attention module (fusion of dual-branch features to improve the accuracy of edge regions). The training data consists of a synthetic lighting and shadow dataset (100,000 frames of untextured / highly reflective object lighting and shadow images generated by Blender) + a real-world dataset (50,000 frames of industrial part lighting and shadow images). The loss function adopts normal MSE loss + depth L1 loss + joint regularization loss (weight ratio 1:1:0.5). After 100 training iterations, the accuracy reached 98.2%.

[0153] Model deployment: The pre-trained model is frozen in .pb format, and FP16 quantization optimization is performed through TensorRT to reduce the GPU memory usage (from 10GB to 5GB). The inference engine is bound to the computing core of the NVIDIA GeForce RTX 3080 GPU (8704 CUDA cores, 10GB of GPU memory), and Tensor Cores are enabled to accelerate matrix operations. It is deployed as a C++ inference interface (latency ≤0.05s / frame).

[0154] Application process: Receive the fused 5-channel input tensor (shadow edge map, specular feature map, and light source x / y / z coordinates, all normalized to [0,1]), load the model through the inference interface for forward computation, and output a normalized normal map (x / y / z component range [-1,1]) and a depth map (range [0,1]). Then, convert it into actual physical quantities through inverse normalization (normal components remain [-1,1], and depth values ​​are multiplied by 0.5m to convert to meters), and pass it to the post-processing module.

[0155] Post-processing module: Filters the normal map (1080×1920×3) and depth map (1080×1920×1) output by the model.

[0156] Normal map: Bilateral filtering is used, and the core formula is:

[0157] ,

[0158] Components and normalized weights:

[0159] ,

[0160] ,

[0161] ,

[0162] in:

[0163] in:

[0164] : Input the 3D normal vector of the normal map at pixel position (i,j) (original output of the model).

[0165] : The output normal vector at pixel position (u,v) after bilateral filtering.

[0166] Ω: A square neighborhood window centered on the target pixel (u,v).

[0167] Spatial domain weight kernel. Measures the spatial distance correlation between a neighboring pixel (i,j) and the center pixel (u,v). The closer the distance, the greater the weight.

[0168] Spatial domain standard deviation. Controlling the rate at which weights decay with spatial distance determines the effective neighborhood size of the filter. In this paper, a value of [value to be filled in] is used. =5.

[0169] : Range (or color range) weight kernel. Measures the vector similarity between the normal vectors N(i,j) of neighboring pixels and the normal vector N(u,v) of the center pixel. The smaller the vector difference, the greater the weight.

[0170] ∥ ∥2: The L2 norm (Euclidean distance) of a vector.

[0171] : Standard deviation of the range. The rate at which the control weight decays with the difference in the normal vector determines the sensitivity of edge preservation. In this paper, the value is taken as... =0.1.

[0172] Normalization factor. Ensures that the sum of all weights is 1, maintaining stable output brightness.

[0173] Operating mechanism: It combines spatial domain (the closer the distance, the greater the weight) and value domain (the closer the gray levels, the greater the weight) filtering to smooth noise while preserving normal edges; the technical advantage is that its edge preservation ability is better than Gaussian filtering, making it suitable for processing the detail preservation requirements of normal maps.

[0174] Depth map: Utilizes guided filtering; the core formula is:

[0175] qi = ak * Ii + bk (i ∈ ωk, where ωk is the neighborhood of pixel i).

[0176] Where: qi is the output depth value, Ii is the guide map (here, the depth map itself), ak and bk are local linear coefficients (solved by minimizing ||qi-Ii||²+εak², ε=0.01 is the regularization parameter);

[0177] Operating mechanism: Based on the local linearity assumption, coefficients are calculated for each neighborhood and averaged to obtain the final output; the technical advantages are strong edge preservation capability and fast calculation speed, which is suitable for handling the boundary preservation requirements of depth maps.

[0178] Function Description: Data fusion associates light and shadow features with light source positions, providing multi-dimensional information for the model; GPU-accelerated inference improves model running speed and meets real-time reconstruction requirements; post-processing filtering improves the smoothness and accuracy of the output results.

[0179] 4. User interaction software

[0180] Functionality: Enables parameter setting, result visualization, and data storage.

[0181] Module:

[0182] Parameter setting module: Provides a graphical interface that allows users to set robotic arm motion parameters (trajectory type, speed, acceleration), camera parameters (frame rate, exposure time), and model parameters (inference threshold, filtering parameters).

[0183] Visualization module: Using the OpenGL library, it displays 3D point cloud models in real time (generated from normal maps and depth maps), and supports rotation (±360°), scaling (0.1-10 times), and translation (±0.5m) operations, making it convenient for users to view model details.

[0184] Data storage module: Stores original images, light and shadow feature maps, normal maps, depth maps, and 3D point cloud data into an SQLite database (supporting multi-user access) and generates reports (including information such as reconstruction accuracy and time consumption).

[0185] Function Description: The graphical interface reduces the difficulty of operation for users and is suitable for non-professionals; OpenGL visualization enables real-time interaction with 3D models, making it easy for users to verify the reconstruction results; database storage facilitates data traceability and analysis.

[0186] IV. Introduction to Core Algorithms

[0187] The core algorithm is the "soul" of the system, and it needs to solve three key problems: active lighting and shadow casting strategy, lighting and shadow feature extraction, and normal-depth inversion, as detailed below:

[0188] 1. Active Light and Shadow Projection Strategy

[0189] Objective: To ensure that the light source trajectory covers the object's surface and obtain sufficient light and shadow information.

[0190] Strategy:

[0191] Trajectory type: Using a spiral trajectory (e.g.) Figure 2 As shown), the starting point is 0.3m above the center of the object (camera coordinate system), the ending point is 0.1m below the center of the object, the pitch is 0.05m (to ensure that the overlap rate of adjacent light spots is ≥30%), and the number of rotations is 10 (covering the top, bottom, left and right surfaces of the object).

[0192] Sampling interval: A light source position is sampled every 0.1s (corresponding to a camera frame rate of 30fps), and each position is paused for 0.05s (to ensure that the camera captures a clear image of light and shadow).

[0193] Adaptive adjustment: Based on the curvature changes of the object's surface, the trajectory density is dynamically adjusted (e.g., increasing sampling points in areas of high curvature and decreasing sampling points in areas of low curvature) to reduce unnecessary movement and improve efficiency.

[0194] Function Description: Spiral trajectory is more efficient than grid trajectory (continuous movement, no back and forth) and can cover the entire surface of the object; the adaptive adjustment strategy balances reconstruction accuracy and time cost, and is suitable for complex curved objects.

[0195] 2. Light and Shadow Feature Extraction Algorithm

[0196] Objective: To extract dynamic change features of shadows / highlights from images and provide input for the model.

[0197] algorithm:

[0198] Shadow features:

[0199] Extract the positional changes (Δx, Δy) and shape changes (boundary length L, curvature C) of the shadow boundary. Use Canny edge detection to obtain the binary image of the shadow boundary, and then extract the pixel coordinate sequence of the boundary using the Suzuki contour tracking algorithm.

[0200] ,in, Let n be the ordered set of shadow boundary points obtained by contour tracking in frame t, where n is the total number of points.

[0201] Calculate the offset between adjacent frames:

[0202] , i is the boundary point index, ( , ) represents the pixel displacement of the i-th boundary point from frame t-1 to frame t;

[0203] Calculate the boundary length (closed profile):

[0204] ,in ;

[0205] Calculating curvature (discrete curvature (three-point approximation method, using an approximate formula)):

[0206] , Let be the curvature estimate of point i, where a positive value indicates local convexity and a negative value indicates local concavity; is the average arc length, and is the Euclidean distance between adjacent points (averaged). Its function is to quantify the dynamic changes of the shadow boundary and provide shape constraints for normal inversion.

[0207] Highlight Features: Extract feature point displacements (Δu, Δv) and density changes (D = number of feature points / area of ​​highlight region) in the highlight region. The SIFT algorithm is used to extract feature points in the highlight region (low-contrast points are filtered with a threshold of 0.04, points with weak edge response are filtered with an edge threshold of 10, and irrelevant background points are excluded); the FLANN feature matching algorithm (fast nearest neighbor search library, using kd-tree indexing to accelerate matching, matching threshold of 0.7) is used to obtain the correspondence between feature points in adjacent frames; feature point displacements are calculated.

[0208]

[0209] in,( , ) represents the displacement coordinates of the feature point in the current frame. , ( ) is the displacement coordinate of the matching point in the previous frame); calculate the density of the highlight region: D=N / S (N is the number of feature points in the highlight region, S is the pixel area of ​​the highlight region, obtained through contour detection); its function is to quantify the dynamic changes of the highlight region and reflect the depth changes of the object surface (large displacement indicates rapid depth changes, and high density indicates large surface curvature).

[0210] Functional Description: The position and shape changes of the shadow boundary reflect the changes in the normal direction of the object's surface (e.g., in areas with high boundary curvature, the normal direction changes rapidly); the displacement and density changes of the specular feature points reflect the depth changes of the object's surface (e.g., in areas with large feature point displacement, the depth changes rapidly).

[0211] 3. Normal-Depth Inversion Model

[0212] Objective: To establish the mapping relationship between lighting and shadow features and the surface normals and depth of an object.

[0213] Model structure: A dual-branch deep learning model is adopted (refer to the structure of existing convolutional neural network models), which shares a feature extraction layer and outputs normal maps and depth maps respectively. Finally, the two features are fused through an attention mechanism to improve the inversion accuracy.

[0214] Input layer: size is (1080, 1920, 5), where the 5 channels are shadow boundary map (1), highlight feature point map (1), light source x coordinate (1), light source y coordinate (1), and light source z coordinate (1).

[0215] Shared convolutional layer: Contains 3 convolutional blocks, each block has 2 3×3 convolutional layers (stride=1, padding=1), ReLU activation function, and 2×2 max pooling layer (stride=2) to extract multi-scale light and shadow features.

[0216] Normal branch: It contains four 3×3 convolutional layers (stride=1, padding=1), ReLU activation function, and then a 2×2 deconvolutional layer (stride=2) to enlarge the feature map to the input size, outputting a normal map (1080, 1920, 3), where each pixel represents the x, y, and z components of the normal (range [-1,1]).

[0217] Depth branch: The structure is similar to the normal branch, outputting a depth map (1080, 1920, 1), where each pixel represents a depth value (range [0, 0.5m]).

[0218] Attention module: Receives feature maps (size (540, 960, 64)) from the normal and depth branches, and calculates the attention weight for each pixel: First, the normal feature map... With depth feature map The feature maps are concatenated along the channel dimension (resulting in a 540×960×128 feature map), and an attention weight map is generated using a 1×1 convolutional layer (outputting channel 1) and a sigmoid activation function. (Each pixel value ranges from [0,1]), the formula is:

[0219] ,

[0220] Where σ is the sigmoid function. This compresses the output value to the [0,1] range;

[0221] ∈R H×W×C : Normal feature map (H=540,W=960,C=64);

[0222] ∈R H×W×C : Depth feature map;

[0223] ∈R H×W×2C : Feature map after concatenation of channel dimensions (2C=128);

[0224] ∈R 1×1×2C×1 b∈R: weights and biases of a 1×1 convolutional layer;

[0225] ∈R H×W×1 : The generated attention weight map;

[0226] Then the feature maps are fused:

[0227] ( The larger the value, the more attention is paid to the normal features; conversely, the smaller the value, the more attention is paid to the depth features.

[0228] in:

[0229] ⊙ indicates element-wise multiplication (Hadamard product);

[0230] 1 is a full-1 tensor with the same dimension as AA (used for broadcasting);

[0231] ∈R H×W×C The fused feature map. Weight A indicates the degree of dependence of each spatial location on the normal and depth features;

[0232] Finally, the fused feature map is input into the deconvolution layer to restore it to the input size. The advantage of this technique is that it adaptively weights and fuses normal and depth features, which improves the inversion accuracy of edge regions and complex surfaces.

[0233] Functional Description: The dual-branch structure processes normal and depth information separately to avoid information confusion; the attention mechanism integrates the features of both, allowing the model to focus more on features that are more important for inversion (such as normal features of edge regions and depth features of planar regions).

[0234] V. Data Interaction and Synchronization Mechanism

[0235] Data interaction and synchronization are crucial for the normal operation of the system. It is necessary to ensure real-time coordination between robotic arm movement, camera acquisition, and model inference. The specific process is as follows (e.g.) Figure 3 (as shown)

[0236] 1. Command Issuance: The user sets parameters (trajectory type, speed, frame rate, etc.) through the Qt interactive interface. The main controller encapsulates the parameters in JSON format (e.g., {"trajectory_type":"spiral","speed":0.1,"fps":30}) and sends them to the robotic arm control software (an embedded system running on an ABB IRC5 controller) and the vision acquisition software (a Linux system running on an FPGA) via TCP / IP protocol (port 5000). The robotic arm control software parses the parameters to generate motion commands, and the vision acquisition software adjusts the camera exposure time (10ms) and frame rate (30fps).

[0237] 2. Motion Planning: The robotic arm control software uses the RRT-Connect algorithm to generate a spiral trajectory in Cartesian space (starting point (0,0,0.3), ending point (0,0,-0.1), pitch 0.05m), which is converted into an angle sequence in joint space through inverse kinematics solution (sampling frequency 1kHz) and sent to the robotic arm controller via Ethernet / IP protocol. The controller drives the joint motors to execute (real-time control cycle 1ms).

[0238] 3. Synchronization signal: The robotic arm control software outputs a 5VTTL pulse (rising edge triggered, duty cycle 50%) through the GPIO interface at each trajectory sampling point (0.1s interval). The pulse signal is transmitted to the external trigger input port of the camera through the shielded wire. The camera immediately captures an image after receiving the pulse, ensuring that the position of the light source is strictly synchronized with the image frame (delay ≤ 0.1ms).

[0239] 4. Image Acquisition: Images captured by the camera (4096×3072 resolution, 24-bit RGB) are transmitted to the image acquisition card via USB 3.0 interface (transmission rate 5Gbps, latency ≤1ms); the FPGA performs preprocessing on the images: Gaussian filtering (3×3 kernel, σ=1.4), distortion correction (based on parameters calibrated by Zhang Zhengyou, k1=0.12, k2=-0.34, p1=0.001, p2=0.0005), and grayscale conversion (weighted average method Y=0.299R+0.587G+0.114B), with a processing time ≤0.05s / frame.

[0240] 5. Feature Extraction: The FPGA performs feature extraction on the preprocessed grayscale image: Canny edge detection (min=50, max=150) is used to obtain the shadow boundary map, and the SIFT algorithm (threshold 0.04) is used to obtain the highlight feature point map; at the same time, the robotic arm controller sends the current light source position (Cartesian coordinates (x,y,z)) to the FPGA via Ethernet / IP protocol; the FPGA packages the shadow map, highlight map, and light source position data into a binary stream (format: [shadow map size][shadow map data][highlight map size][highlight map data][x][y][z]), and transmits it to the main controller via TCP / IP protocol (port 6000), with a transmission time ≤0.02s / frame.

[0241] 6. Model Inference: The main controller parses the received binary stream into a model input tensor (1080×1920×5), loads the pre-trained TensorFlow model (stored on SSD), and calls the CUDA kernel function of the NVIDIA RTX3080 for inference (batch size=1, inference time ≤0.1s / frame); after obtaining the normal map (1080×1920×3) and depth map (1080×1920×1), the main controller performs bilateral filtering on the normal map (…). =5, =0.1), guide filtering is applied to the depth map (radius=5, ε=0.01), and the post-processing time is ≤0.03s / frame.

[0242] 7. Visualization and Storage: The main controller converts the post-processed normal map and depth map into a 3D point cloud (via camera projection model: x=(u-cx)*z / fx, y=(v-cy)*z / fy, where cx=960, cy=540, fx=1920, fy=1080), with the point cloud format being PCD; it is rendered in real time using the OpenGL library (frame rate ≥30fps), supporting rotation, scaling, and translation operations; simultaneously, the original image, lighting feature map, normal map, depth map, and point cloud data are stored in an SQLite database (table structure: id, timestamp, raw_image_path, shadow_map_path, highlight_map_path, normal_map_path, depth_map_path, point_cloud_path), with a storage time ≤0.05s / frame.

[0243] 8. Synchronization: An external trigger mode is adopted (the robotic arm outputs a pulse signal to trigger the camera), with a pulse signal delay of ≤0.1ms, ensuring that each light source position corresponds to a unique image frame; the camera frame rate (30fps) and pulse frequency (30Hz) are strictly synchronized; the main controller marks each data frame with a timestamp (accuracy 1ms) to ensure the time consistency of light and shadow features, light source positions, and model output; the technical advantage is to avoid data misalignment and improve reconstruction accuracy (synchronization error ≤0.5%).

[0244] VI. Technical Principle Analysis

[0245] 1. Active lighting and shadow information enhancement mechanism

[0246] Traditional passive lighting methods rely on ambient light sources and have limited information dimensions (only static shadows can be obtained). This system uses a robotic arm to drive point light sources to actively move and obtain dynamic lighting sequences (the position and shape of shadows / highlights change as the light source moves). The information dimension is improved from 2D (static shadows) to 3D (dynamic lighting + light source position), providing richer constraints for normal-depth inversion.

[0247] 2. The mapping relationship between shadows and normals

[0248] Points on the shadow boundary satisfy the condition that the normal is perpendicular to the direction of the light source:

[0249] Let the coordinates of point light source S be (Sx, Sy, Sz), the coordinates of point P on the object surface be (Px, Py, Pz), and the normal be n = (nx, ny, nz). Then, the points on the shadow boundary satisfy the following:

[0250] That is, the normal direction is perpendicular to the vector from the light source to point P.

[0251] =(nx,ny,nz)T: The unit normal vector at point P on the surface of the object;

[0252] S=(Sx,Sy,Sz)T: The position coordinates of the point light source in the world coordinate system;

[0253] P=(Px,Py,Pz)T: The three-dimensional coordinates of the shadow boundary point on the object surface;

[0254] At the shadow boundary, the surface normal direction n is perpendicular to the vector SP from the surface point to the light source, which means that the light just grazes the surface of the object, neither entering nor leaving the surface.

[0255] When the light source moves, S changes, and the point P on the shadow boundary also changes. By detecting the change in P, the change in n can be inferred (e.g., when the light source moves from left to right, the shadow boundary moves from right to left; in areas with large boundary curvature, the normal direction changes rapidly).

[0256] 3. The mapping relationship between highlights and depth

[0257] The highlight satisfies the condition that the normal direction is the angle bisector of the direction of the light source and the direction of the camera:

[0258] Let the coordinates of the camera center C be (Cx, Cy, Cz), then the normal to the highlight point P is:

[0259] ,

[0260] C = (Cx, Cy, Cz) T The coordinates of the camera center (projection center) in the world coordinate system;

[0261] : The unit direction vector from a surface point to the light source;

[0262] : The unit direction vector from a surface point to the camera;

[0263] normalize ): Vector normalization operation. ;

[0264] ∥ ∥: The Euclidean norm (length) of a vector;

[0265] For ideal specular reflection (highlight), the surface normal direction is exactly the angle bisector of the incident direction (from the light source to the surface point) and the reflection direction (from the surface point to the camera). When the light source moves, S changes, and the position of the highlight P also changes. By detecting the change in P, the depth of P can be deduced (e.g., on a plane, the highlight moves linearly with the light source, and the depth change is small; on a curved surface, the movement of the highlight is non-linear, and the depth change is large).

[0266] Example 1

[0267] 1. System calibration

[0268] Objective: To ensure the coordinate systems of the robotic arm, camera, and point light source are consistent. Steps are as follows:

[0269] Robotic arm and camera extrinsic parameter calibration: A checkerboard pattern (0.02m × 0.02m) is fixed to the end of the robotic arm. The robotic arm is moved to 10 different positions (covering the camera's field of view). The camera captures images of the checkerboard pattern. The rotation matrix R and translation vector T for each position are calculated using OpenCV's calibrateCamera function. Then, the global extrinsic parameters are optimized using the least squares method (e.g., R = [[0.999, 0.01, 0.005], [-0.01, 0.999, 0.002], [-0.005, -0.002, 0.999]], T = [[0.1], [0.05], [0.2]]).

[0270] Point light source position calibration: Using a cube (side length 0.1m), move the point light source 0.2m above the cube to cast a shadow. Take a picture of the shadow with the camera and measure the shadow length L=0.05m. Calculate the position of the light source based on similar triangles (L = a·h / (H - h), where a=0.1m, h=0.2m, solving for H=0.6m). Adjust the position of the robotic arm to ensure that the height of the light source from the ground is 0.6m.

[0271] 2. Target placement and parameter setting steps:

[0272] Target placement: Place the object (such as a textureless plastic gear) on the worktable, ensuring it is within the robotic arm's workspace and the camera's field of view (the center of the object coincides with the camera's optical axis).

[0273] Parameter settings: Configured via interactive software:

[0274] Robotic arm: trajectory type (spiral), speed (0.1m / s), acceleration (0.2m / s²);

[0275] Camera: Frame rate (30fps), Exposure time (10ms);

[0276] Model: Inference threshold (0.5), filtering parameters (bilateral filter sigma_color=0.1, sigma_space=5).

[0277] 3. Active lighting and image acquisition steps:

[0278] The robotic arm control software generates a helical trajectory (starting point (0,0,0.3), ending point (0,0,-0.1), pitch 0.05m, number of rotations 10).

[0279] The robotic arm drives the point light source to move along the trajectory, outputting a pulse signal every 0.1s to trigger the camera to take a picture;

[0280] The camera captures 300 frames of images (10 rotations x 30fps) and transmits them to the image acquisition card via USB 3.0.

[0281] 4. Light and shadow feature extraction and preprocessing steps:

[0282] The image acquisition card performs the following preprocessing steps: noise reduction (Gaussian filtering σ=1.4), distortion correction (using the distortion coefficients obtained from calibration), and grayscale conversion (weighted average method).

[0283] Shadow feature extraction: Use Canny edge detection (threshold min=50, max=150) to obtain shadow boundaries, and calculate the positional change (Δx, Δy) and shape change (length L, curvature C) of the boundaries.

[0284] Spectral feature extraction: Use the SIFT algorithm (threshold 0.04, edge threshold 10) to obtain spectral feature points, and calculate the displacement (Δu, Δv) and density change (D) of the feature points;

[0285] The shadow feature map, highlight feature map, and light source position data (coordinates of the robotic arm's end effector) are packaged and transmitted to the main controller.

[0286] 5. Normal-depth inversion and post-processing steps:

[0287] The main controller inputs data into the deep learning model and infers the normal map (1080×1920×3) and the depth map (1080×1920×1).

[0288] Apply bilateral filtering (sigma_color=0.1, sigma_space=5) to the normal map to remove noise;

[0289] Apply guided filtering to the depth map (radius=5, epsilon=0.01) to preserve depth boundaries;

[0290] Generate a 3D point cloud: the coordinates of each pixel are (X, Y, Z):

[0291] ,

[0292] ,

[0293] .

[0294] By summing the 3D coordinates of all pixels, a complete point cloud model of the object is obtained; finally, a triangular mesh model is generated using the Poisson surface reconstruction algorithm to complete the 3D shape reconstruction.

[0295] Therefore, this system can acquire dynamic features through active lighting and shadow projection without relying on the object's own texture, solving the reconstruction problem of structured light and multi-view stereo vision on textureless objects. A robotic arm drives the active movement of point light sources, increasing the dimension of lighting and shadow information from 2D to 3D, providing richer constraints and significantly improving the accuracy of normal and depth inversion. Adaptive trajectory planning can cover objects of different shapes and sizes, and is particularly suitable for the reconstruction of highly reflective (metal, glass) and complex curved surfaces. An external trigger synchronization mechanism ensures a strict correspondence between lighting and shadow acquisition and the light source position. GPU-accelerated inference enables real-time reconstruction with a processing speed of up to 30fps, meeting the needs of industrial applications. A dual-branch deep learning model based on an attention mechanism, combined with bilateral filtering and guided filtering post-processing, achieves a reconstruction accuracy of ±0.02mm, superior to traditional passive lighting and shadow methods.

[0296] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, electronic device 410 may include a first processor 2001.

[0297] Optionally, the electronic device 410 may also include a memory 2002 and a transceiver 2003.

[0298] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0299] The following is combined Figure 4 A detailed description of each component of electronic device 410 is provided below:

[0300] The first processor 2001 is the control center of the electronic device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0301] Optionally, the first processor 2001 can perform various functions of the electronic device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0302] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0303] In a specific implementation, as one example, the electronic device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0304] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0305] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0306] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0307] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0308] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0309] It should be noted that, Figure 4 The structure of the electronic device 410 shown does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0310] Furthermore, the technical effects of the electronic device 410 can be referred to the technical effects of the XXX method described in the above method embodiments, and will not be repeated here.

[0311] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0312] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A 3D shape reconstruction system based on active lighting and shadow, characterized in that, include: A robotic arm with a movable point light source at its end; A camera is fixed relative to the point light source and is used to capture the light and shadow image of the surface of an object illuminated by the point light source. The processor, which is communicatively connected to the robotic arm and the camera, is configured to: The robotic arm is controlled to move the point light source along a preset trajectory to project dynamically changing shadows and / or highlights onto the surface of the object. Control the camera to synchronously acquire a sequence of light and shadow images of the object's surface; Process the light and shadow image sequence to extract light and shadow features; The light and shadow features and the corresponding point light source position information are input into a pre-trained deep learning model to obtain the normal map and depth map of the object surface. A three-dimensional morphological model of the object is generated based on the normal map and depth map.

2. The system according to claim 1, characterized in that, The robotic arm is a six-degree-of-freedom industrial robotic arm, the point light source is a high-brightness LED light source with a spot diameter of less than or equal to 5 mm; the camera is an industrial camera that supports external triggering mode, and it is connected to the robotic arm through a hardware synchronization signal to achieve strict synchronization between light and shadow acquisition and the movement of the light source position.

3. The system according to claim 1, characterized in that, When controlling the robotic arm, the processor is specifically configured to: generate a trajectory that avoids object boundaries using a motion planning method based on a random tree expansion algorithm, and perform real-time closed-loop control of the robotic arm through a proportional-integral-derivative controller to ensure the motion accuracy of the point light source.

4. The system according to claim 1, characterized in that, When extracting light and shadow features, the processor is specifically configured to: preprocess the acquired image, use an edge detection algorithm to extract the shadow boundary information of the object surface, and use a scale-invariant feature transformation algorithm to extract the highlight feature point information of the object surface.

5. The system according to claim 4, characterized in that, The processor is further configured to: encode the extracted shadow boundary information and highlight feature point information into a first feature map and a second feature map, respectively; expand the three-dimensional position coordinates of the point light source in the preset trajectory into three coordinate maps of the same size as the first feature map; and merge the first feature map, the second feature map, and the three coordinate maps into a multi-channel input tensor as the input of the deep learning model.

6. The system according to claim 1, characterized in that, The pre-trained deep learning model is a dual-branch network model with an attention mechanism, which includes a shared feature extraction layer, an independent normal prediction branch, an independent depth prediction branch, and an attention module for fusing features from both branches.

7. The system according to claim 1, characterized in that, The processor is further configured to: perform bilateral filtering on the normal map output by the deep learning model and guided filtering on the output depth map to remove noise and preserve geometric edges.

8. The system according to claim 1, characterized in that, When the processor generates a three-dimensional morphological model based on the normal map and the depth map, it is specifically configured to: convert the two-dimensional coordinates of each pixel and its corresponding depth value into a three-dimensional point cloud based on the camera imaging model, wherein the normal map provides the normal vector information of each point in the three-dimensional point cloud.

9. The system according to claim 1, characterized in that, It also includes a user interface connected to the processor, used to receive trajectory parameters, camera parameters or model parameters input by the user, and to visualize the three-dimensional morphological model in real time.

10. A 3D morphology reconstruction method based on active lighting and shadow, characterized in that, include: S1: Control the robotic arm to drive the point light source at its end to move on the surface of the target object according to a preset trajectory, so as to project dynamically changing shadows and / or highlights; S2: While the point light source moves, a fixed camera is used to synchronously acquire a sequence of light and shadow images of the surface of the target object. S3: Process the light and shadow image sequence and extract the light and shadow features of the target object surface; S4: Input the light and shadow features and the position information of the point light source during the movement into the pre-trained deep learning model to obtain the normal map and depth map of the target object surface; S5: Reconstruct the three-dimensional shape of the target object based on the normal map and depth map.

Images