Image distortion removal and optimization system

Abstract

The invention relates to the technical field of image processing, in particular to an image distortion removal and optimization system which comprises an intelligent calibration engine, a multi-sensor fusion module, an FPGA hardware acceleration module, a NeRF distortion correction module and a lightweight image enhancement module. Generating a distortion parameter model; compared with the defects that in the prior art, manual calibration board placement and manual parameter adjustment are dependent, the process is tedious and time-consuming, and the precision is restricted by the operation level, the scheme adopts an intelligent calibration engine driven by reinforcement learning, and through dynamic pose optimization and mixing precision calculation, the accuracy of the calibration board is improved. The calibration time is shortened to 45 seconds, the re-projection error is stably lower than 0.3 pixel, and the deployment efficiency of an industrial detection scene is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image dedistortion and optimization system. Background Art

[0002] Current image dedistortion technology mainly relies on traditional methods based on geometric models and end-to-end correction schemes based on deep learning.

[0003] Traditional methods use the Zhang Zhengyou calibration method to obtain camera intrinsic parameters and distortion coefficients. This requires manual acquisition of multi-angle calibration plate images and the screening of valid samples. The entire process can take over 30 minutes, and calibration accuracy is significantly affected by operator experience, with errors often exceeding 3 pixels in edge areas. The fourth-order polynomial distortion model used in existing tools like OpenCV is insufficient to fit the complex distortion of fisheye lenses with a field of view exceeding 180°, resulting in ineffective correction of wide-angle scenes. While deep learning methods can automatically learn distortion features, they rely on large-scale annotated data and GPU computing power, making them difficult to meet real-time requirements on embedded devices. For example, the Bundle Adjustment algorithm only achieves a frame rate of 12fps when processing 1080P video on a Cortex-A55 chip.

[0004] Furthermore, differences in optical properties across different application scenarios lead to limited algorithm versatility. High reflections from medical endoscopes and high-frequency textures in industrial inspections require customized development, and parameter tuning relies heavily on specialized personnel. These technical bottlenecks severely restrict the large-scale application of high-precision vision systems in areas such as industrial inspection and autonomous driving. Summary of the Invention

[0005] In order to overcome the problems raised in the above background technology, the present invention proposes an image dedistortion and optimization system.

[0006] The technical solution of the present invention is: an image dedistortion and optimization system, comprising:

[0007] Intelligent calibration engine, which optimizes the calibration process through reinforcement learning, performs automatic calibration, and generates distortion parameter models;

[0008] Multi-sensor fusion module for integrating camera, IMU and LiDAR data to achieve dynamic environment perception;

[0009] FPGA hardware acceleration module, used to provide real-time computing acceleration for NeRF inference and image processing;

[0010] NeRF distortion correction module, which is used to build an implicit scene model through neural radiance field to handle complex optical distortion;

[0011] A lightweight image enhancement module is used to perform denoising, sharpening, and contrast optimization on the rectified image.

[0012] Preferably, the intelligent calibration engine optimizes the calibration process through reinforcement learning, performs automatic calibration, and generates a distortion parameter model, specifically including:

[0013] S11: Environment detection and initialization: Automatically detect key parameters of the working environment, including illumination uniformity and calibration plate flatness, while loading the improved YOLOv5s spatial attention network model.

[0014] S12: Adaptive calibration plate recognition: After the camera captures the current scene image, it analyzes and processes the image through the improved YOLOv5s spatial attention network model to accurately locate and identify the calibration plate type. It then performs sub-pixel corner detection to obtain high-precision feature point locations, and then uses the EPnP algorithm to calculate the precise pose parameters of the calibration plate in three-dimensional space.

[0015] S13: Reinforcement learning optimization sampling. Based on the current calibration parameters and reprojection error state, the reinforcement learning agent analyzes and generates the optimal camera pose adjustment plan. It automatically adjusts the camera position and collects new images, updates the calibration parameters and re-evaluates the error. This optimization process is repeated until the reprojection error is stable below the 0.3 pixel threshold.

[0016] S14: Mixed-precision parameter solving, using a multi-precision collaborative computing strategy to accelerate Jacobian matrix calculations at FP16 precision and use INT8 precision for fast error evaluation. It also applies the Levenberg-Marquardt optimization algorithm to accurately solve the distortion coefficients and complete parameter updates at FP32 precision.

[0017] S15: Output the results and initialize NeRF. After parameter optimization is completed, a JSON file containing the complete camera parameters is generated. At the same time, the camera pose and image sequence obtained during the acquisition process are transmitted to the NeRF distortion correction module.

[0018] Preferably, when the intelligent calibration engine automatically detects key parameters of the working environment and loads the improved YOLOv5s spatial attention network model, the structure of the improved YOLOv5s spatial attention network model includes:

[0019] A11: Input layer: This layer receives a 640×640 pixel RGB three-channel image, normalizes the pixel values ​​to the 0–1 range, uses bilinear interpolation to maintain the image ratio, and has a built-in automatic contrast adjustment module.

[0020] A12: Focus slicing layer. The Focus layer performs feature reconstruction. It first divides the input image into a 2×2 grid. The pixels in each grid are concatenated according to the channel dimension to form a 320×320 resolution feature map, with the number of channels expanded to 12. The Focus layer then compresses the image to 32 channels using a 3×3 convolution kernel.

[0021] A13: Backbone network primary feature extraction, including Stage 1 at 160×160 resolution: downsampling through 3×3 convolution with a stride of 2, followed by the C3 module to extract basic features. The C3 module uses a cross-stage residual structure, including one 1×1 convolution and a 3×3 depthwise separable convolution, focusing on capturing the primary features of the calibration plate; Stage 2 at 80×80 resolution: after further downsampling in the convolutional layer, two C3 modules are connected in series to enhance feature expression capabilities;

[0022] A14: Backbone network advanced feature extraction, including three key components: standard convolutional layers that reduce the resolution to 40×40; three cascaded C3 modules to enhance feature reuse; and a spatial attention submodule that performs channel-wise average / max pooling on feature maps, generates a spatial weight map through 7×7 convolution, and multiplies it with the original features after Sigmoid activation, increasing the weight of the calibration plate area to 0.8-1.0 and attenuating the background area to below 0.2.

[0023] A15: Backbone network final feature extraction, including: downsampling convolution layer output 512-channel features; SPPF module: captures multi-scale contextual information through maximum pooling of 5, 9, and 13 scales; ASPP module: uses parallel convolution with dilation rates of 6, 12, and 18, covering receptive fields of 45×45 to 85×85 pixels; channel attention submodule: performs global pooling on feature maps and generates channel weight vectors through fully connected layers to enhance the expressive power of key feature channels;

[0024] A16: Neck network, which implements multi-scale feature fusion: Upsampling path: Upsamples 20×20 deep features to 40×40, fuses them with Stage 3 output, and transmits high-level semantic information; Downsampling path: Downsamples 40×40 features to 20×20, fuses them with Stage 4 output, and supplements detailed features; Concat operation is used to splice features of different levels, and 1×1 convolution is used to adjust the number of channels, and finally outputs dual-resolution feature maps of 40×40×256 and 20×20×512;

[0025] A17: Decoupled detection head, consisting of three independent branches: Classification branch: Outputs a 17×17 grid of calibration plate category probabilities through 3×3 convolution → 1×1 convolution → Sigmoid activation; Regression branch: Outputs bounding box coordinates with the same structure; Corner heatmap branch: Adds a corner prediction channel and outputs a 17×17×4 heatmap, where each channel corresponds to the probability distribution of a corner position on the calibration plate;

[0026] A18: Output layer. The output layer performs post-processing: threshold filtering is applied to the classification branch; the regression branch output is aspect-ratio checked and dynamically matched to the optimal anchor type; the corner heat map is weighted by Gaussian neighborhood to calculate the sub-pixel offset; and finally, the calibration plate type, bounding box coordinates, and four corner coordinates are output.

[0027] As an optimal choice, the multi-sensor fusion module integrates camera, IMU and LiDAR data to achieve dynamic environment perception:

[0028] S21: Hardware-level spatiotemporal synchronization, achieving μs-level precision synchronization through FPGA hardware triggering. Using the image sensor exposure signal as the primary clock source, it synchronously triggers IMU sampling and LiDAR pulses, attaching nanosecond-level timestamps to all sensor data to establish a unified spatiotemporal reference.

[0029] S22: Sensor data preprocessing: lens correction and feature point extraction for image data, Kalman filtering for IMU data and calculation of 6-DoF attitude matrix, ground filtering and dynamic object removal for LiDAR point cloud to generate depth map;

[0030] S23: Multimodal data alignment: image feature points, IMU pose, and LiDAR depth map are converted to a unified coordinate system using a pre-calibrated extrinsic matrix, and pose offset during image exposure is compensated using IMU motion data.

[0031] S24: Adaptive fusion engine, using a three-layer fusion strategy: the data layer aligns the original input, the feature layer matches ORB points and depth information, and the decision layer dynamically assigns weights based on confidence and generates a joint estimate through weighted voting;

[0032] S25: Dynamic environment modeling, constructing a cost function that includes photometric error, geometric error, and motion smoothing terms, using the Gauss-Newton method to optimize and solve the dense 3D scene model, and generating a dynamic distortion compensation matrix that is transmitted to the NeRF distortion correction module. The principle formula of the cost function that includes photometric error, geometric error, and motion smoothing terms is:

[0033] E=ω img E photo +ω depth E geom +ωimu E motion ;

[0034] Among them, E is the total cost function, ω img is the image weight coefficient, E photo is the photometric error term, ω depth is the depth weight coefficient, E geom is the geometric error term, ω imu is the IMU weight coefficient, E motion is the motion smoothing term;

[0035] S26: Exception handling mechanism, real-time monitoring of sensor status, enabling HDR mode when the image is overexposed, switching to monocular depth estimation when the LiDAR point cloud is sparse, starting the motion prediction model when the IMU is abnormal, and triggering hardware resynchronization when the time deviation is >1ms;

[0036] S27: Output and feedback: Output the registered depth map, camera pose, and distortion parameters for each frame. Extract key frames every 10 frames and upload them to the cloud via MQTT. Incrementally update the calibration parameters and optimize the weight strategy.

[0037] As a preference, a three-layer fusion strategy is adopted: the data layer aligns the original input, the feature layer matches the ORB points and depth information, and the decision layer dynamically assigns weights based on the confidence level, and generates a joint estimate through weighted voting, specifically including:

[0038] S31: Feature-level fusion: First, extract the ORB feature points of the image and calculate the descriptors. Then, establish the feature trajectory through optical flow tracking. Then, align the LiDAR depth map with the feature points and assign a depth value to each feature point. Finally, use the 6-DoF pose data of the IMU to predict the motion trajectory of the feature points and generate a 3D feature point cloud with depth and velocity vectors.

[0039] S32: Decision-level fusion, based on a Bayesian filtering framework, implements multi-hypothesis fusion: generates three scenario hypotheses: image features, LiDAR point cloud, and IMU motion, and calculates the joint posterior distribution using a probabilistic formula; performs triple checks on geometric consistency, motion consistency, and temporal consistency;

[0040] S33: Dynamic weight adjustment and real-time evaluation of the confidence of each sensor: The image calculates the camera's confidence variance based on the feature matching success rate, the LiDAR calculates the IMU sensor's confidence variance based on the point cloud density and reflection intensity, and the IMU calculates the LiDAR sensor's confidence variance through the accelerometer and gyroscope variance. The weights are dynamically assigned. The principle formula for dynamic weight assignment is:

[0041]

[0042] Among them, ωi is the fusion weight of the i-th sensor, σ i is the confidence variance of the i-th sensor, j is the sensor index, and j=1 represents a camera, j=2 represents a LiDAR sensor, j=3 represents an IMU sensor, and n is the total number of sensors;

[0043] S34: Construct a cost function that includes photometric error, geometric error, and motion smoothing terms, with the IMU pose as the initial value; calculate the image brightness gradient, LiDAR point-to-surface distance, and the Jacobian matrix of the IMU Lie algebra; solve the state update through incremental equations, iterate and optimize until convergence, and output the optimal scene structure and camera pose.

[0044] S35: Output generation: Outputs dense 3D point clouds, precise camera poses, and dynamic object motion vectors for each frame. Quality indicators such as reprojection error, point cloud void rate, and pose jumps are monitored simultaneously, triggering local optimization and depth completion when anomalies occur.

[0045] As a preferred option, the FPGA hardware acceleration module provides real-time computing acceleration for NeRF reasoning and image processing, specifically including:

[0046] S41: Data input stage: Receive image data through the MIPI CSI-2 interface, synchronously obtain IMU attitude and LiDAR point cloud, transfer them to memory through the DMA engine, and perform preprocessing;

[0047] S42: Ray generation phase: 256 rays / cycle are generated in parallel based on camera parameters. The NeRF distortion layer is applied to correct the ray directions, achieving a processing capacity of 4096 rays / millisecond. The ray parameter package with the starting point and direction vector is output.

[0048] S43: Hash code query, store 512 in BRAM 3 Resolution hash grid, optimized layout via Z-order curve, parallel query of 8 neighboring voxels of ray position and trilinear interpolation;

[0049] S44: Neural radiance field inference, using 128 parallel processing units to accelerate an 8-layer MLP network, supporting mixed-precision computation with FP16 input, INT8 weights, and INT32 accumulation, and outputting RGB σ values.

[0050] S45: Volume rendering integral, marching rays with adaptive step size, using Q8.24 fixed-point format to calculate the volume rendering integral equation, triggering early termination mechanism when transparency > 0.99;

[0051] S46: Image post-processing, performing CLAHE contrast enhancement, NLM denoising, and guided filter sharpening in parallel, and dynamically adjusting gain parameters based on image entropy.

[0052] Preferably, the NeRF distortion correction module, when processing complex optical distortion by establishing an implicit scene model through the neural radiation field, specifically includes:

[0053] S51: Scene encoding stage: input multi-view calibration images and initial distortion parameters, and discretize the 3D space into a 512×512×512 voxel grid through a hash grid encoder. Each voxel stores a 64-dimensional feature vector. A spatial hash function is used to achieve efficient query and establish an implicit radiance field representation of the scene.

[0054] S52: Joint optimization training, constructing a cost function that includes RGB reconstruction loss and distortion smoothing constraints, synchronously optimizing the radiance field MLP and the learnable distortion layer, and alternatingly updating parameters using the Adam optimizer. Training is performed until the peak signal-to-noise ratio exceeds 35dB.

[0055] S53: Dynamic correction inference, real-time processing of input images: generating ray vectors based on pixel coordinates, applying learned distortion parameters to correct the ray direction, sampling 64 points along the corrected ray for volume rendering, querying hash grid features and predicting color and density through MLP, and finally integrating to synthesize the undistorted image;

[0056] S54: Edge optimization mechanism, enabling special processing for image edge areas: sampling density increased to 128 points / ray, activation of high-order distortion compensation items, and addition of edge smoothing constraints.

[0057] Preferably, when constructing a cost function including RGB reconstruction loss and distortion smoothing constraint, the constructed cost function including RGB reconstruction loss and distortion smoothing constraint is expressed as follows:

[0058]

[0059] Among them, L total is the total loss function, I pred is the pixel color predicted by the model, I gt is the true pixel color, λ is the regularization coefficient, is the gradient operator, θ dist is the distortion parameter vector.

[0060] Preferably, when the lightweight image enhancement module performs denoising, sharpening and contrast optimization processing on the corrected image, the lightweight image enhancement module specifically includes:

[0061] S51: Input preprocessing, receiving YUV420 format input stream, separating brightness and color components, and allocating three independent buffers: the Y area stores brightness data, the UV area stores downsampled chroma data, and the RGB area is ready to receive the conversion results;

[0062] S52: Multi-task parallel processing, performing three operations simultaneously: CLAHE contrast enhancement of the luminance component using 32×32 blocks, NLM non-local means denoising using a 7×7 window on the chrominance components, and guided filter sharpening with a radius of 3 pixels on the RGB data;

[0063] S53: Dynamic parameter adjustment, calculating the brightness entropy value based on a 64×64 sliding window and adjusting the enhancement intensity parameters in real time;

[0064] S54: Output optimization, converting the processed YUV data into RGB888 format, and applying the BT.601→BT.709 color gamut conversion matrix for color space mapping;

[0065] S55: Pipeline control, using a double buffer mechanism to process frame data, CLAHE is divided into 16 blocks for parallel processing, NLM is divided into 4 areas for execution, and guided filtering is performed for full frame processing;

[0066] S56: Abnormal protection, automatically reduces gain when overexposed pixels are detected, switches to bilateral filtering in low-light scenes, and disables the sharpening module at high temperatures.

[0067] As an option, it also includes:

[0068] A21: Cloud collaboration module, used to manage the model incremental training and parameter update process through the MQTT protocol;

[0069] A22: Dynamic deployment module, used to provide a cross-platform deployment interface to support heterogeneous hardware environments.

[0070] Beneficial effects of the present invention:

[0071] 1. Compared with existing technologies that rely on manual calibration plate placement and parameter adjustment, which have the disadvantages of being cumbersome and time-consuming, and accuracy limited by operator proficiency, this solution uses a reinforcement learning-driven intelligent calibration engine. Through dynamic pose optimization and mixed-precision solution, it shortens calibration time to 45 seconds and maintains a stable reprojection error below 0.3 pixels, significantly improving the deployment efficiency of industrial inspection scenarios.

[0072] 2. Compared to existing technologies that use fixed-order polynomial models, which suffer from the inability to handle ultra-wide-angle distortion and high computational latency, this solution combines implicit neural radiance field modeling with FPGA hardware acceleration. Through hash-coded volume rendering and a learnable distortion layer, it supports real-time correction of 190° fisheye lenses, with edge error controlled within 0.8 pixels and a 4K resolution processing latency of only 1.2 milliseconds, meeting the high real-time requirements of drone panoramic photography.

[0073] 3. Compared with existing technologies that rely on a single visual sensor and have the disadvantages of being sensitive to motion blur and poorly adaptable to lighting, this solution innovatively introduces a multi-sensor spatiotemporal fusion mechanism. Relying on hardware-level synchronization and adaptive weighting strategies, it maintains sub-pixel accuracy even under harsh conditions such as vibration, rain and fog, with a position error of less than 0.3 meters, providing all-weather reliable protection for the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 Shown is a schematic diagram of the structure of the image dedistortion and optimization system of the present invention;

[0075] Figure 2 Shown is a schematic diagram of the workflow of the intelligent calibration engine in the image dedistortion and optimization system of the present invention. DETAILED DESCRIPTION

[0076] The present invention will be further described below with reference to the accompanying drawings and examples.

[0077] See also Figure 1-Figure 2 The present invention provides an embodiment: an image dedistortion and optimization system, comprising:

[0078] Intelligent calibration engine, which optimizes the calibration process through reinforcement learning, performs automatic calibration, and generates distortion parameter models;

[0079] Multi-sensor fusion module for integrating camera, IMU and LiDAR data to achieve dynamic environment perception;

[0080] FPGA hardware acceleration module, used to provide real-time computing acceleration for NeRF inference and image processing;

[0081] NeRF distortion correction module, which is used to build an implicit scene model through neural radiance field to handle complex optical distortion;

[0082] A lightweight image enhancement module is used to perform denoising, sharpening, and contrast optimization on the rectified image.

[0083] As described above, this invention uniquely integrates five core modules: intelligent calibration, multi-sensor fusion, hardware acceleration, neural radiation field correction, and image enhancement, to create an end-to-end, full-process processing chain. Compared to the separate calibration and correction steps in traditional solutions, this architecture achieves seamless data flow through modular collaboration, reducing the calibration time of complex lenses on industrial inspection production lines from 45 minutes to less than 1 minute. It also supports real-time processing of 60 frames at 4K resolution, achieving a breakthrough in edge distortion correction accuracy reaching sub-pixel level, completely solving the industry problem of inefficient deployment of high-precision vision systems.

[0084] Preferably, the intelligent calibration engine optimizes the calibration process through reinforcement learning, performs automatic calibration, and generates a distortion parameter model, specifically including:

[0085] S11: Environment detection and initialization: Automatically detect key parameters of the working environment, including illumination uniformity and calibration plate flatness, while loading the improved YOLOv5s spatial attention network model.

[0086] S12: Adaptive calibration plate recognition: After the camera captures the current scene image, it analyzes and processes the image through the improved YOLOv5s spatial attention network model to accurately locate and identify the calibration plate type. It then performs sub-pixel corner detection to obtain high-precision feature point locations, and then uses the EPnP algorithm to calculate the precise pose parameters of the calibration plate in three-dimensional space.

[0087] S13: Reinforcement learning optimization sampling. Based on the current calibration parameters and reprojection error state, the reinforcement learning agent analyzes and generates the optimal camera pose adjustment plan. It automatically adjusts the camera position and collects new images, updates the calibration parameters and re-evaluates the error. This optimization process is repeated until the reprojection error is stable below the 0.3 pixel threshold.

[0088] S14: Mixed-precision parameter solving, using a multi-precision collaborative computing strategy to accelerate Jacobian matrix calculations at FP16 precision and use INT8 precision for fast error evaluation. It also applies the Levenberg-Marquardt optimization algorithm to accurately solve the distortion coefficients and complete parameter updates at FP32 precision.

[0089] S15: Output the results and initialize NeRF. After parameter optimization is completed, a JSON file containing the complete camera parameters is generated. At the same time, the camera pose and image sequence obtained during the acquisition process are transmitted to the NeRF distortion correction module.

[0090] As described above, this invention addresses the core pain points of traditional calibration, which relies on manual intervention and has low sampling efficiency. This method introduces a reinforcement learning agent to dynamically plan the camera pose. By autonomously determining the optimal shooting angle and distance, the calibration sampling frame count is reduced by 70%. Combined with a mixed-precision computing engine to accelerate parameter calculation, this system completes the entire calibration process in 45 seconds, utilizing FP16 / INT8 / FP32 collaborative computing. It outputs a high-precision model with a reprojection error of less than 0.3 pixels, providing plug-and-play rapid calibration capabilities for drone inspection equipment.

[0091] Preferably, when the intelligent calibration engine automatically detects key parameters of the working environment and loads the improved YOLOv5s spatial attention network model, the structure of the improved YOLOv5s spatial attention network model includes:

[0092] A11: Input layer: This layer receives a 640×640 pixel RGB three-channel image, normalizes the pixel values ​​to the 0–1 range, uses bilinear interpolation to maintain the image ratio, and has a built-in automatic contrast adjustment module.

[0093] A12: Focus slicing layer. The Focus layer performs feature reconstruction. It first divides the input image into a 2×2 grid. The pixels in each grid are concatenated according to the channel dimension to form a 320×320 resolution feature map, with the number of channels expanded to 12. The Focus layer then compresses the image to 32 channels using a 3×3 convolution kernel.

[0094] A13: Backbone network primary feature extraction, including Stage 1 at 160×160 resolution: downsampling through 3×3 convolution with a stride of 2, followed by the C3 module to extract basic features. The C3 module uses a cross-stage residual structure, including one 1×1 convolution and a 3×3 depthwise separable convolution, focusing on capturing the primary features of the calibration plate; Stage 2 at 80×80 resolution: after further downsampling in the convolutional layer, two C3 modules are connected in series to enhance feature expression capabilities;

[0095] A14: Backbone network advanced feature extraction, including three key components: standard convolutional layers that reduce the resolution to 40×40; three cascaded C3 modules to enhance feature reuse; and a spatial attention submodule that performs channel-wise average / max pooling on feature maps, generates a spatial weight map through 7×7 convolution, and multiplies it with the original features after Sigmoid activation, increasing the weight of the calibration plate area to 0.8-1.0 and attenuating the background area to below 0.2.

[0096] A15: Backbone network final feature extraction, including: downsampling convolution layer output 512-channel features; SPPF module: captures multi-scale contextual information through maximum pooling of 5, 9, and 13 scales; ASPP module: uses parallel convolution with dilation rates of 6, 12, and 18, covering receptive fields of 45×45 to 85×85 pixels; channel attention submodule: performs global pooling on feature maps and generates channel weight vectors through fully connected layers to enhance the expressive power of key feature channels;

[0097] A16: Neck network, which implements multi-scale feature fusion: Upsampling path: Upsamples 20×20 deep features to 40×40, fuses them with Stage 3 output, and transmits high-level semantic information; Downsampling path: Downsamples 40×40 features to 20×20, fuses them with Stage 4 output, and supplements detailed features; Concat operation is used to splice features of different levels, and 1×1 convolution is used to adjust the number of channels, and finally outputs dual-resolution feature maps of 40×40×256 and 20×20×512;

[0098] A17: Decoupled detection head, consisting of three independent branches: Classification branch: Outputs a 17×17 grid of calibration plate category probabilities through 3×3 convolution → 1×1 convolution → Sigmoid activation; Regression branch: Outputs bounding box coordinates with the same structure; Corner heatmap branch: Adds a corner prediction channel and outputs a 17×17×4 heatmap, where each channel corresponds to the probability distribution of a corner position on the calibration plate;

[0099] A18: Output layer. The output layer performs post-processing: threshold filtering is applied to the classification branch; the regression branch output is aspect-ratio checked and dynamically matched to the optimal anchor type; the corner heat map is weighted by Gaussian neighborhood to calculate the sub-pixel offset; and finally, the calibration plate type, bounding box coordinates, and four corner coordinates are output.

[0100] As described above, this invention deeply optimizes the YOLOv5s network architecture and embeds a dual-path attention mechanism within the backbone network. The spatial attention submodule generates weighted heatmaps through 7x7 convolutions, significantly improving the response of regional features on the calibration plate. The channel attention submodule adaptively selects key feature channels. This design maintains a 99.8% calibration plate recognition rate and a false positive rate below 0.02% even in the strong spatter interference environment of a welding workshop, significantly lowering the deployment threshold for complex industrial scenarios.

[0101] As an optimal choice, the multi-sensor fusion module integrates camera, IMU and LiDAR data to achieve dynamic environment perception:

[0102] S21: Hardware-level spatiotemporal synchronization, achieving μs-level precision synchronization through FPGA hardware triggering. Using the image sensor exposure signal as the primary clock source, it synchronously triggers IMU sampling and LiDAR pulses, attaching nanosecond-level timestamps to all sensor data to establish a unified spatiotemporal reference.

[0103] S22: Sensor data preprocessing: lens correction and feature point extraction for image data, Kalman filtering for IMU data and calculation of 6-DoF attitude matrix, ground filtering and dynamic object removal for LiDAR point cloud to generate depth map;

[0104] S23: Multimodal data alignment: image feature points, IMU pose, and LiDAR depth map are converted to a unified coordinate system using a pre-calibrated extrinsic matrix, and pose offset during image exposure is compensated using IMU motion data.

[0105] S24: Adaptive fusion engine, using a three-layer fusion strategy: the data layer aligns the original input, the feature layer matches ORB points and depth information, and the decision layer dynamically assigns weights based on confidence and generates a joint estimate through weighted voting;

[0106] S25: Dynamic environment modeling, constructing a cost function that includes photometric error, geometric error, and motion smoothing terms, using the Gauss-Newton method to optimize and solve the dense 3D scene model, and generating a dynamic distortion compensation matrix that is transmitted to the NeRF distortion correction module. The principle formula of the cost function that includes photometric error, geometric error, and motion smoothing terms is:

[0107] E=ω img E photo +ω depth E geom +ω imu E motion ;

[0108] Among them, E is the total cost function, ω img is the image weight coefficient, E photo is the photometric error term, ω depth is the depth weight coefficient, E geom is the geometric error term, ω imu is the IMU weight coefficient, E motion is the motion smoothing term;

[0109] S26: Exception handling mechanism, real-time monitoring of sensor status, enabling HDR mode when the image is overexposed, switching to monocular depth estimation when the LiDAR point cloud is sparse, starting the motion prediction model when the IMU is abnormal, and triggering hardware resynchronization when the time deviation is >1ms;

[0110] S27: Output and feedback: Output the registered depth map, camera pose, and distortion parameters for each frame. Extract key frames every 10 frames and upload them to the cloud via MQTT. Incrementally update the calibration parameters and optimize the weight strategy.

[0111] As described above, this invention innovatively utilizes an FPGA hardware trigger mechanism, using the image sensor exposure signal as a benchmark, to achieve microsecond-level synchronization between the IMU and LiDAR. Nanosecond-level timestamp alignment and motion compensation algorithms eliminate image smearing caused by timing misalignment during sharp turns, limiting multi-source data registration errors to within 0.1 pixel, ensuring centimeter-level positioning accuracy for high-speed autonomous driving.

[0112] As a preference, a three-layer fusion strategy is adopted: the data layer aligns the original input, the feature layer matches the ORB points and depth information, and the decision layer dynamically assigns weights based on the confidence level, and generates a joint estimate through weighted voting, specifically including:

[0113] S31: Feature-level fusion: First, extract the ORB feature points of the image and calculate the descriptors. Then, establish the feature trajectory through optical flow tracking. Then, align the LiDAR depth map with the feature points and assign a depth value to each feature point. Finally, use the 6-DoF pose data of the IMU to predict the motion trajectory of the feature points and generate a 3D feature point cloud with depth and velocity vectors.

[0114] S32: Decision-level fusion, based on a Bayesian filtering framework, implements multi-hypothesis fusion: generates three scenario hypotheses: image features, LiDAR point cloud, and IMU motion, and calculates the joint posterior distribution using a probabilistic formula; performs triple checks on geometric consistency, motion consistency, and temporal consistency;

[0115] S33: Dynamic weight adjustment and real-time evaluation of the confidence of each sensor: The image calculates the camera's confidence variance based on the feature matching success rate, the LiDAR calculates the IMU sensor's confidence variance based on the point cloud density and reflection intensity, and the IMU calculates the LiDAR sensor's confidence variance through the accelerometer and gyroscope variance. The weights are dynamically assigned. The principle formula for dynamic weight assignment is:

[0116]

[0117] Among them, ω i is the fusion weight of the i-th sensor, σ i is the confidence variance of the i-th sensor, j is the sensor index, and j=1 represents a camera, j=2 represents a LiDAR sensor, j=3 represents an IMU sensor, and n is the total number of sensors;

[0118] S34: Construct a cost function that includes photometric error, geometric error, and motion smoothing terms, with the IMU pose as the initial value; calculate the image brightness gradient, LiDAR point-to-surface distance, and the Jacobian matrix of the IMU Lie algebra; solve the state update through incremental equations, iterate and optimize until convergence, and output the optimal scene structure and camera pose.

[0119] S35: Output generation: Outputs dense 3D point clouds, precise camera poses, and dynamic object motion vectors for each frame. Quality indicators such as reprojection error, point cloud void rate, and pose jumps are monitored simultaneously, triggering local optimization and depth completion when anomalies occur.

[0120] As described above, the present invention constructs a three-level fusion framework: feature-level, decision-level, and weight optimization. Feature-level fusion correlates visual features with depth information, decision-level fusion integrates multi-sensor hypotheses through Bayesian inference, and a weighting module dynamically assigns confidence based on the environment. This mechanism automatically increases the LiDAR weight to 0.85 when the camera suddenly loses focus in a dimly lit tunnel, maintaining a lane detection error of less than 0.9 pixels, thus overcoming the bottleneck of single sensor failure under extreme conditions.

[0121] As a preferred option, the FPGA hardware acceleration module provides real-time computing acceleration for NeRF reasoning and image processing, specifically including:

[0122] S41: Data input stage: Receive image data through the MIPI CSI-2 interface, synchronously obtain IMU attitude and LiDAR point cloud, transfer them to memory through the DMA engine, and perform preprocessing;

[0123] S42: Ray generation phase: 256 rays / cycle are generated in parallel based on camera parameters. The NeRF distortion layer is applied to correct the ray directions, achieving a processing capacity of 4096 rays / millisecond. The ray parameter package with the starting point and direction vector is output.

[0124] S43: Hash code query, store 512 in BRAM 3 Resolution hash grid, optimized layout via Z-order curve, parallel query of 8 neighboring voxels of ray position and trilinear interpolation;

[0125] S44: Neural radiance field inference, using 128 parallel processing units to accelerate an 8-layer MLP network, supporting mixed-precision computation with FP16 input, INT8 weights, and INT32 accumulation, and outputting RGB σ values.

[0126] S45: Volume rendering integral, marching rays with adaptive step size, using Q8.24 fixed-point format to calculate the volume rendering integral equation, triggering early termination mechanism when transparency > 0.99;

[0127] S46: Image post-processing, performing CLAHE contrast enhancement, NLM denoising, and guided filter sharpening in parallel, and dynamically adjusting gain parameters based on image entropy.

[0128] As described above, this invention utilizes a fully pipelined acceleration engine designed based on programmable logic. The ray generation unit processes 256 rays in parallel per cycle, hash code queries optimize storage access via Z-order curves, and the mixed-precision MLP inference unit integrates 128 parallel processors. 4K real-time rendering is achieved using a 28nm process with a power consumption of only 5.5 watts, achieving an energy efficiency seven times that of GPU solutions with the same computing power requirements, successfully implementing neural radiation field technology on an in-vehicle embedded platform.

[0129] Preferably, the NeRF distortion correction module, when processing complex optical distortion by establishing an implicit scene model through the neural radiation field, specifically includes:

[0130] S51: Scene encoding stage: input multi-view calibration images and initial distortion parameters, and discretize the 3D space into a 512×512×512 voxel grid through a hash grid encoder. Each voxel stores a 64-dimensional feature vector. A spatial hash function is used to achieve efficient query and establish an implicit radiance field representation of the scene.

[0131] S52: Joint optimization training, constructing a cost function that includes RGB reconstruction loss and distortion smoothing constraints, synchronously optimizing the radiance field MLP and the learnable distortion layer, and alternatingly updating parameters using the Adam optimizer. Training is performed until the peak signal-to-noise ratio exceeds 35dB.

[0132] S53: Dynamic correction inference, real-time processing of input images: generating ray vectors based on pixel coordinates, applying learned distortion parameters to correct the ray direction, sampling 64 points along the corrected ray for volume rendering, querying hash grid features and predicting color and density through MLP, and finally integrating to synthesize the undistorted image;

[0133] S54: Edge optimization mechanism, enabling special processing for image edge areas: sampling density increased to 128 points / ray, activation of high-order distortion compensation items, and addition of edge smoothing constraints.

[0134] As described above, this invention overcomes the limitations of traditional polynomial models by constructing a joint optimization framework for a learnable distortion layer and radiation field. The distortion layer dynamically fits high-order nonlinear optical distortion, while the radiation field uses hash coding to establish an implicit representation of the scene. In calibration tests of a 190-degree fisheye lens, the distortion correction error in the edge region was reduced from 3.2 pixels in traditional solutions to 0.78 pixels, providing distortion-free surround-view images for panoramic monitoring systems.

[0135] Preferably, when constructing a cost function including RGB reconstruction loss and distortion smoothing constraint, the constructed cost function including RGB reconstruction loss and distortion smoothing constraint is expressed as follows:

[0136]

[0137] Among them, L total is the total loss function, I pred is the pixel color predicted by the model, I gt is the true pixel color, λ is the regularization coefficient, is the gradient operator, θ dist is the distortion parameter vector.

[0138] As described above, this invention introduces a gradient sparsity constraint into the loss function, forcing smooth variations in distortion parameters. By suppressing parameter abruptness, it effectively addresses the overfitting problem in small-sample training, enabling the medical endoscope to maintain 99% generalization capability with only 50 calibration images. This reduces the number of model parameters by 40% while increasing inference speed by 2.3 times.

[0139] Preferably, when the lightweight image enhancement module performs denoising, sharpening and contrast optimization processing on the corrected image, the lightweight image enhancement module specifically includes:

[0140] S51: Input preprocessing, receiving YUV420 format input stream, separating brightness and color components, and allocating three independent buffers: the Y area stores brightness data, the UV area stores downsampled chroma data, and the RGB area is ready to receive the conversion results;

[0141] S52: Multi-task parallel processing, performing three operations simultaneously: CLAHE contrast enhancement of the luminance component using 32×32 blocks, NLM non-local means denoising using a 7×7 window on the chrominance components, and guided filter sharpening with a radius of 3 pixels on the RGB data;

[0142] S53: Dynamic parameter adjustment, calculating the brightness entropy value based on a 64×64 sliding window and adjusting the enhancement intensity parameters in real time;

[0143] S54: Output optimization, converting the processed YUV data into RGB888 format, and applying the BT.601→BT.709 color gamut conversion matrix for color space mapping;

[0144] S55: Pipeline control, using a double buffer mechanism to process frame data, CLAHE is divided into 16 blocks for parallel processing, NLM is divided into 4 areas for execution, and guided filtering is performed for full frame processing;

[0145] S56: Abnormal protection, automatically reduces gain when overexposed pixels are detected, switches to bilateral filtering in low-light scenes, and disables the sharpening module at high temperatures.

[0146] As described above, the present invention innovatively designs a three-step parallel processing pipeline: CLAHE blocks enhance local contrast, NLM filtering suppresses chromatic noise, and guided filtering sharpens details. Dynamically adjusts gain parameters based on image entropy, automatically increasing contrast enhancement by 2.3 times in foggy scenes. The entire process completes 4K image processing in under 1 millisecond, meeting the stringent latency requirements of real-time image enhancement for surgical robots.

[0147] As an option, it also includes:

[0148] A21: Cloud collaboration module, used to manage the model incremental training and parameter update process through the MQTT protocol;

[0149] A22: Dynamic deployment module, used to provide a cross-platform deployment interface to support heterogeneous hardware environments.

[0150] As described above, this invention establishes a collaborative end-to-end optimization ecosystem. Terminal devices upload keyframe data via the lightweight MQTT protocol. The cloud uses incremental training to update distortion model parameters, dynamically adapting to different computing platforms through the hardware abstraction layer. When the new catadioptric lens is put into use, the system deployment cycle is shortened from two weeks to three days, significantly improving cross-platform and cross-device adaptability.

[0151] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge of those skilled in the art without departing from the spirit of the present invention.

Claims

1. An image dedistortion and optimization system, characterized by: include: Intelligent calibration engine, which optimizes the calibration process through reinforcement learning, performs automatic calibration, and generates distortion parameter models; Multi-sensor fusion module for integrating camera, IMU and LiDAR data to achieve dynamic environment perception; FPGA hardware acceleration module, used to provide real-time computing acceleration for NeRF inference and image processing; NeRF distortion correction module, which is used to build an implicit scene model through neural radiance field to handle complex optical distortion; A lightweight image enhancement module is used to perform denoising, sharpening, and contrast optimization on the rectified image.

2. The image dedistortion and optimization system according to claim 1, wherein: The intelligent calibration engine optimizes the calibration process through reinforcement learning, performs automatic calibration, and generates a distortion parameter model. Specifically, it includes: S11: Environment detection and initialization: Automatically detect key parameters of the working environment, including illumination uniformity and calibration plate flatness, while loading the improved YOLOv5s spatial attention network model. S12: Adaptive calibration plate recognition: After the camera captures the current scene image, it analyzes and processes the image through the improved YOLOv5s spatial attention network model to accurately locate and identify the calibration plate type. It then performs sub-pixel corner detection to obtain high-precision feature point locations, and then uses the EPnP algorithm to calculate the precise pose parameters of the calibration plate in three-dimensional space. S13: Reinforcement learning optimization sampling. Based on the current calibration parameters and reprojection error state, the reinforcement learning agent analyzes and generates the optimal camera pose adjustment plan. It automatically adjusts the camera position and collects new images, updates the calibration parameters and re-evaluates the error. This optimization process is repeated until the reprojection error is stable below the 0.3 pixel threshold. S14: Mixed-precision parameter solving, using a multi-precision collaborative computing strategy to accelerate Jacobian matrix calculations at FP16 precision and use INT8 precision for fast error evaluation. It also applies the Levenberg-Marquardt optimization algorithm to accurately solve the distortion coefficients and complete parameter updates at FP32 precision. S15: Output the results and initialize NeRF. After parameter optimization is completed, a JSON file containing the complete camera parameters is generated. At the same time, the camera pose and image sequence obtained during the acquisition process are transmitted to the NeRF distortion correction module.

3. The image dedistortion and optimization system according to claim 2, wherein: The intelligent calibration engine automatically detects key parameters of the working environment and loads the improved YOLOv5s spatial attention network model. The structure of the improved YOLOv5s spatial attention network model includes: A11: Input layer: This layer receives a 640×640 pixel RGB three-channel image, normalizes the pixel values ​​to the 0–1 range, uses bilinear interpolation to maintain the image ratio, and has a built-in automatic contrast adjustment module. A12: Focus slicing layer. The Focus layer performs feature reconstruction. It first divides the input image into a 2×2 grid. The pixels in each grid are concatenated according to the channel dimension to form a 320×320 resolution feature map, with the number of channels expanded to 12. The Focus layer then compresses the image to 32 channels using a 3×3 convolution kernel. A13: Backbone network primary feature extraction, including Stage 1 at 160×160 resolution: downsampling through 3×3 convolution with a stride of 2, followed by the C3 module to extract basic features. The C3 module uses a cross-stage residual structure, including one 1×1 convolution and a 3×3 depthwise separable convolution, focusing on capturing the primary features of the calibration plate; Stage 2 at 80×80 resolution: after further downsampling in the convolutional layer, two C3 modules are connected in series to enhance feature expression capabilities; A14: Backbone network advanced feature extraction, including three key components: standard convolutional layers that reduce the resolution to 40×40; three cascaded C3 modules to enhance feature reuse; and a spatial attention submodule that performs channel-wise average / max pooling on feature maps, generates a spatial weight map through 7×7 convolution, and multiplies it with the original features after Sigmoid activation, increasing the weight of the calibration plate area to 0.8-1.0 and attenuating the background area to below 0.

2. A15: Backbone network final feature extraction, including: downsampling convolution layer output 512-channel features; SPPF module: captures multi-scale contextual information through maximum pooling of 5, 9, and 13 scales; ASPP module: uses parallel convolution with dilation rates of 6, 12, and 18, covering receptive fields of 45×45 to 85×85 pixels; channel attention submodule: performs global pooling on feature maps and generates channel weight vectors through fully connected layers to enhance the expressive power of key feature channels; A16: Neck network, which implements multi-scale feature fusion: Upsampling path: Upsamples 20×20 deep features to 40×40, fuses them with Stage 3 output, and transmits high-level semantic information; Downsampling path: Downsamples 40×40 features to 20×20, fuses them with Stage 4 output, and supplements detailed features; Concat operation is used to splice features of different levels, and 1×1 convolution is used to adjust the number of channels, and finally outputs dual-resolution feature maps of 40×40×256 and 20×20×512; A17: Decoupled detection head, consisting of three independent branches: Classification branch: Outputs a 17×17 grid of calibration plate category probabilities through 3×3 convolution → 1×1 convolution → Sigmoid activation; Regression branch: Outputs bounding box coordinates with the same structure; Corner heatmap branch: Adds a corner prediction channel and outputs a 17×17×4 heatmap, where each channel corresponds to the probability distribution of a corner position on the calibration plate; A18: Output layer. The output layer performs post-processing: threshold filtering is applied to the classification branch; the regression branch output is aspect-ratio checked and dynamically matched to the optimal anchor type; the corner heat map is weighted by Gaussian neighborhood to calculate the sub-pixel offset; and finally, the calibration plate type, bounding box coordinates, and four corner coordinates are output.

4. The image dedistortion and optimization system according to claim 3, wherein: When the multi-sensor fusion module integrates camera, IMU and LiDAR data to achieve dynamic environment perception: S21: Hardware-level spatiotemporal synchronization, achieving μs-level precision synchronization through FPGA hardware triggering. Using the image sensor exposure signal as the primary clock source, it synchronously triggers IMU sampling and LiDAR pulses, attaching nanosecond-level timestamps to all sensor data to establish a unified spatiotemporal reference. S22: Sensor data preprocessing: lens correction and feature point extraction for image data, Kalman filtering for IMU data and calculation of 6-DoF attitude matrix, ground filtering and dynamic object removal for LiDAR point cloud to generate depth map; S23: Multimodal data alignment: image feature points, IMU pose, and LiDAR depth map are converted to a unified coordinate system using a pre-calibrated extrinsic matrix, and pose offset during image exposure is compensated using IMU motion data. S24: Adaptive fusion engine, using a three-layer fusion strategy: the data layer aligns the original input, the feature layer matches ORB points and depth information, and the decision layer dynamically assigns weights based on confidence and generates a joint estimate through weighted voting; S25: Dynamic environment modeling, constructing a cost function that includes photometric error, geometric error, and motion smoothing terms, using the Gauss-Newton method to optimize and solve the dense 3D scene model, and generating a dynamic distortion compensation matrix that is transmitted to the NeRF distortion correction module; S26: Exception handling mechanism, real-time monitoring of sensor status, enabling HDR mode when the image is overexposed, switching to monocular depth estimation when the LiDAR point cloud is sparse, starting the motion prediction model when the IMU is abnormal, and triggering hardware resynchronization when the time deviation is >1ms; S27: Output and feedback: Output the registered depth map, camera pose, and distortion parameters for each frame. Extract key frames every 10 frames and upload them to the cloud via MQTT. Incrementally update the calibration parameters and optimize the weight strategy.

5. The image dedistortion and optimization system according to claim 4, wherein: When adopting a three-layer fusion strategy: the data layer aligns the original input, the feature layer matches ORB points and depth information, and the decision layer dynamically assigns weights based on confidence, and generates a joint estimate through weighted voting, specifically including: S31: Feature-level fusion: First, extract the ORB feature points of the image and calculate the descriptors. Then, establish the feature trajectory through optical flow tracking. Then, align the LiDAR depth map with the feature points and assign a depth value to each feature point. Finally, use the 6-DoF pose data of the IMU to predict the motion trajectory of the feature points and generate a 3D feature point cloud with depth and velocity vectors. S32: Decision-level fusion, based on a Bayesian filtering framework, implements multi-hypothesis fusion: generates three scenario hypotheses: image features, LiDAR point cloud, and IMU motion, and calculates the joint posterior distribution using a probabilistic formula; performs triple checks on geometric consistency, motion consistency, and temporal consistency; S33: Dynamic weight adjustment and real-time evaluation of the confidence of each sensor: The camera's confidence variance is calculated based on the image feature matching success rate, the LiDAR's confidence variance is calculated based on the point cloud density and reflection intensity, and the IMU's confidence variance is calculated based on the accelerometer and gyroscope variance. Weights are dynamically assigned. S34: Construct a cost function that includes photometric error, geometric error, and motion smoothing terms, with the IMU pose as the initial value; calculate the image brightness gradient, LiDAR point-to-surface distance, and the Jacobian matrix of the IMU Lie algebra; solve the state update through incremental equations, iterate and optimize until convergence, and output the optimal scene structure and camera pose. S35: Output generation: Outputs dense 3D point clouds, precise camera poses, and dynamic object motion vectors for each frame. Quality indicators such as reprojection error, point cloud void rate, and pose jumps are monitored simultaneously, triggering local optimization and depth completion when anomalies occur.

6. The image dedistortion and optimization system according to claim 5, wherein: The FPGA hardware acceleration module provides real-time computing acceleration for NeRF inference and image processing, specifically including: S41: Data input stage: Receive image data through the MIPI CSI-2 interface, synchronously obtain IMU attitude and LiDAR point cloud, transfer them to memory through the DMA engine, and perform preprocessing; S42: Ray generation phase: 256 rays / cycle are generated in parallel based on camera parameters. The NeRF distortion layer is applied to correct the ray directions, achieving a processing capacity of 4096 rays / millisecond. The ray parameter package with the starting point and direction vector is output. S43: Hash code query, store 512 in BRAM 3 Resolution hash grid, optimized layout via Z-order curve, parallel query of 8 neighboring voxels of ray position and trilinear interpolation; S44: Neural radiance field inference, using 128 parallel processing units to accelerate an 8-layer MLP network, supporting mixed-precision computation with FP16 input, INT8 weights, and INT32 accumulation, and outputting RGB σ values. S45: Volume rendering integral, marching rays with adaptive step size, using Q8.24 fixed-point format to calculate the volume rendering integral equation, triggering early termination mechanism when transparency > 0.99; S46: Image post-processing, performing CLAHE contrast enhancement, NLM denoising, and guided filter sharpening in parallel, and dynamically adjusting gain parameters based on image entropy.

7. The image dedistortion and optimization system according to claim 6, wherein: The NeRF distortion correction module builds an implicit scene model through the neural radiation field to handle complex optical distortions. Specifically, it includes: S51: Scene encoding stage: input multi-view calibration images and initial distortion parameters, and discretize the 3D space into a 512×512×512 voxel grid through a hash grid encoder. Each voxel stores a 64-dimensional feature vector. A spatial hash function is used to achieve efficient query and establish an implicit radiance field representation of the scene. S52: Joint optimization training, constructing a cost function that includes RGB reconstruction loss and distortion smoothing constraints, synchronously optimizing the radiance field MLP and the learnable distortion layer, and alternatingly updating parameters using the Adam optimizer. Training is performed until the peak signal-to-noise ratio exceeds 35dB. S53: Dynamic correction inference, real-time processing of input images: generating ray vectors based on pixel coordinates, applying learned distortion parameters to correct the ray direction, sampling 64 points along the corrected ray for volume rendering, querying hash grid features and predicting color and density through MLP, and finally integrating to synthesize the undistorted image; S54: Edge optimization mechanism, enabling special processing for image edge areas: sampling density increased to 128 points / ray, activation of high-order distortion compensation items, and addition of edge smoothing constraints.

8. The image dedistortion and optimization system according to claim 7, wherein: When constructing a cost function including RGB reconstruction loss and distortion smoothness constraint, the expression of the cost function including RGB reconstruction loss and distortion smoothness constraint is: Among them, L total is the total loss function, I pred is the pixel color predicted by the model, I gt is the true pixel color, λ is the regularization coefficient, is the gradient operator, θ dist is the distortion parameter vector.

9. The image dedistortion and optimization system according to claim 8, wherein: The lightweight image enhancement module performs denoising, sharpening, and contrast optimization on the corrected image, specifically including: S51: Input preprocessing, receiving YUV420 format input stream, separating brightness and color components, and allocating three independent buffers: the Y area stores brightness data, the UV area stores downsampled chroma data, and the RGB area is ready to receive the conversion results; S52: Multi-task parallel processing, performing three operations simultaneously: CLAHE contrast enhancement of the luminance component using 32×32 blocks, NLM non-local means denoising using a 7×7 window on the chrominance components, and guided filter sharpening with a radius of 3 pixels on the RGB data; S53: Dynamic parameter adjustment, calculating the brightness entropy value based on a 64×64 sliding window and adjusting the enhancement intensity parameters in real time; S54: Output optimization, converting the processed YUV data into RGB888 format, and applying the BT.601→BT.709 color gamut conversion matrix for color space mapping; S55: Pipeline control, using a double buffer mechanism to process frame data, CLAHE is divided into 16 blocks for parallel processing, NLM is divided into 4 areas for execution, and guided filtering is performed for full frame processing; S56: Abnormal protection, automatically reduces gain when overexposed pixels are detected, switches to bilateral filtering in low-light scenes, and disables the sharpening module at high temperatures.

10. The image dedistortion and optimization system according to claim 9, characterized in that: Also includes: A21: Cloud collaboration module, used to manage the model incremental training and parameter update process through the MQTT protocol; A22: Dynamic deployment module, used to provide a cross-platform deployment interface to support heterogeneous hardware environments.

Images