Container surface damage detection method and device based on machine vision

Abstract

The invention relates to the technical field of intelligent detection, in particular to a container surface damage detection method and device based on machine vision, and the device consists of a multi-modal data acquisition module, a dynamic compensation processing module, a multi-modal data fusion module and a damage classification and positioning module. The multi-modal data acquisition module generates a high-density three-dimensional point cloud through double-laser line scanning, and acquires a multispectral image, an infrared thermogram and a real-time motion state. The dynamic compensation processing module integrates an optical flow method and acceleration data to realize sub-pixel-level motion compensation, and combines adaptive exposure control to optimize the imaging quality under complex illumination. The multi-modal data fusion module strengthens defect feature expression through space-time alignment and a double-branch collaborative attention mechanism, and model parameters are reduced through lightweight network design. And the damage classification and positioning module adopts an improved deep network to realize defect classification and complete millimeter-level three-dimensional positioning. The automatic detection requirement of a port is met, and the problems of poor container detection precision, poor adaptability and the like are solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection technology, and in particular to a method and device for detecting container surface damage based on machine vision. Background Art

[0002] As the core vehicle for logistics and transportation, the surface integrity of containers is directly related to cargo safety and transportation efficiency. Traditional manual inspection methods rely on workers visually checking each container, which is subject to low efficiency, high subjectivity, and high labor intensity.

[0003] The existing machine vision inspection solutions have the following technical bottlenecks:

[0004] 1. Insufficient adaptability to multiple scenarios: Traditional monocular or binocular vision systems are prone to image distortion in complex lighting conditions (such as alternating cloudy and bright sunlight) and high-contrast shadows, leading to feature extraction failure. When the container surface is oily or rusty, the image quality of visible light cameras deteriorates significantly.

[0005] 2. Lack of dynamic compensation mechanism: Existing systems mostly adopt a fixed installation method, which cannot cope with the image blur caused by vibration during container transportation and conveyor belt speed fluctuations.

[0006] 3. Insufficient multimodal data fusion: Current detection devices mostly rely on a single sensor (such as a visible light camera), which makes it difficult to effectively distinguish similar defects such as surface cracks and scratches.

[0007] 4. Limited damage classification accuracy: Existing deep learning models (such as YOLOv5) perform poorly in identifying small sample defects (such as pinhole corrosion) and cannot meet industrial-grade detection needs. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to propose a method and device for detecting container surface damage based on machine vision, so as to solve the problem of inaccurate container surface damage detection in the prior art.

[0009] Based on the above objectives, the present invention provides a method for detecting container surface damage based on machine vision, comprising the following steps:

[0010] S1. Obtain multimodal data of the container, including 3D point cloud, visible light image, infrared thermal image, and motion parameters of the container surface;

[0011] S2, motion compensation and exposure control based on multimodal data fusion to optimize image quality;

[0012] S3. Construct a cross-modal feature fusion network to enhance the feature expression of the defect area, and achieve deep fusion of multimodal data through multi-scale feature extraction and cross-modal feature fusion;

[0013] S4. Defect classification is achieved through an improved deep neural network, and three-dimensional positioning of damage is achieved based on point cloud data.

[0014] Preferably, in step S1, the multimodal data is calibrated using a spatiotemporal joint calibration method to achieve high-precision spatiotemporal alignment of multi-sensor data.

[0015] Preferably, step S2 specifically includes composite motion estimation, adaptive exposure control and motion compensation;

[0016] The composite motion estimation includes extracting the motion vector of the image sequence by the optical flow method, predicting the motion trend according to the motion parameters, and dynamically adjusting the fusion weight of the optical flow and acceleration data by the adaptive Kalman filter to achieve motion compensation;

[0017] The adaptive exposure control includes real-time adjustment of exposure time and gain based on image entropy analysis using fuzzy PID double closed-loop control. The outer loop of the double closed-loop control adjusts the exposure time based on the image entropy value, and the inner loop optimizes the gain through Bayesian estimation.

[0018] Vibration compensation involves predicting vibration displacement using motion parameters, using acceleration and gyroscope data to predict vibration displacement and adjust the camera posture in advance. A hidden Markov model is used to distinguish between translation, rotation, and impact vibration, suppressing the effects of vibration and reducing image blur.

[0019] Preferably, in step S3, the deep fusion of modal data includes lightweight feature fusion, cross-modal feature fusion strategy and dual-branch collaborative attention mechanism;

[0020] The lightweight feature fusion extracts texture, thermal, and geometric features from visible light images, infrared thermal images, and point cloud data, respectively. It uses hierarchical feature extraction and differentiated feature extraction strategies for different modal data, fully preserving details and semantic information. It also compresses model parameters through channel pruning and knowledge distillation techniques.

[0021] The cross-modal feature fusion strategy adopts a spatiotemporal alignment algorithm to upsample the infrared thermal map to the same resolution as the visible light image based on the homography matrix, uses multimodal feature stitching to fuse visible light, infrared and point cloud features along the channel dimension, and introduces a dual-branch collaborative attention mechanism to enhance the feature expression of key areas.

[0022] Preferably, step S4 specifically includes: using an improved D-FINE network to implement defect classification, introducing multi-scale feature fusion, dynamic convolution and adaptive attention mechanism;

[0023] The improved D-FINE network uses a multi-task learning approach to classify damage, simultaneously predicting the category and confidence level of container damage. The classification loss function uses a combination of focal loss and label smoothing loss.

[0024] Based on the 2D detection results of the improved D-FINE network, the 3D point cloud data is combined to accurately locate the damage in 3D, including mapping the 2D detection box to the 3D point cloud coordinate system, and then determining the 3D coordinate location of the damaged area through minimum bounding box fitting.

[0025] Preferably, the network structure of the D-FINE network includes a backbone network, a neck network and a detection head;

[0026] The backbone network adopts an improved ResNeXt network. It adds depthwise separable convolutional layers to the original ResNeXt network and introduces cross-stage local connections. The cross-stage local connection structure divides the input feature map into two parts. One part passes through a series of convolutional layers, and the other part is directly connected to the output across stages. The two parts of features are then spliced ​​together.

[0027] The neck network introduces PANet and an improved dynamic convolution module. PANet aggregates features of different scales through bottom-up and top-down paths, and the improved dynamic convolution module uses dynamic convolution to adaptively generate convolution kernel weights based on input features.

[0028] The detection head adopts a multi-scale detection head with an adaptive attention mechanism to detect feature maps of different scales. The adaptive attention mechanism enhances the expression of key features by calculating the channel attention and spatial attention of the feature map.

[0029] The present invention also provides a container surface damage detection device based on machine vision, comprising:

[0030] Multimodal data acquisition module, used to obtain multimodal data of the container, including 3D point cloud of the container surface, visible light image, infrared thermal image and motion parameters;

[0031] Motion compensation processing module, which implements motion compensation and exposure control based on multimodal data fusion to optimize image quality;

[0032] Multimodal data fusion module, which builds a cross-modal feature fusion network to enhance the feature expression of defect areas;

[0033] The damage classification and positioning module implements defect classification through an improved deep neural network and realizes three-dimensional positioning of damage based on point cloud data.

[0034] Preferably, the multimodal data acquisition module includes:

[0035] Cross laser line scanning device for generating 3D point clouds and supporting dynamic focus adjustment;

[0036] Multispectral imaging unit, synchronously collecting visible light and near-infrared images;

[0037] Lock-in thermal imaging unit detects the thermal distribution characteristics of hidden defects through periodic thermal excitation;

[0038] Six-axis motion sensor to monitor the box's motion status in real time;

[0039] The spatiotemporal joint calibration unit achieves high-precision spatiotemporal alignment of multi-sensor data through hardware-level synchronous triggering and software compensation algorithms.

[0040] Preferably, the cross laser scanning device uses asymmetric dual laser line cross projection to form laser stripes with an angle of 90°, generate a three-dimensional point cloud and dynamically adjust the spot diameter.

[0041] Preferably, the phase-locked thermal imaging unit extracts the thermal diffusion characteristics of the defect through 0.1Hz periodic thermal excitation and dynamically adjusts the detection field based on the visible light image.

[0042] Beneficial effects of the present invention:

[0043] 1. This invention improves the real-time performance of container surface damage detection in multiple port scenarios. It employs an adaptive exposure control algorithm to adjust exposure time and gain in real time, stabilizing image entropy within an appropriate bit range. This effectively reduces image quality fluctuations during sudden changes in light intensity (e.g., from sunny to shaded), ensuring detection reliability under varying lighting conditions.

[0044] 2. The present invention adopts the fusion detection of infrared thermal imaging and laser line scanning, combined with the heat diffusion characteristics extracted by phase-locked thermal imaging technology, to improve the detection rate of internal structural damage (such as weld fatigue) and meet the detection requirements of hidden defects.

[0045] 3. The oil of the present invention uses the near-infrared channel of the multispectral camera to effectively penetrate surface pollution, combined with the thermal response analysis of the infrared thermal imager, which can effectively improve the defect detection rate in the oil-covered area.

[0046] 4. The present invention can effectively overcome the difficulties in detecting surface damage in moving containers. The present invention uses a composite motion estimation model to fuse the optical flow method with six-axis sensor data to control the image displacement compensation error to within ±0.5 pixels. When the conveyor belt speed fluctuates greatly, a high detection frame rate is still maintained to meet real-time requirements and ensure clear imaging under high-speed movement. The present invention is based on a pre-compensation algorithm for a six-axis acceleration sensor, which can effectively offset high-frequency and high-amplitude vibration displacements, reduce the three-dimensional point cloud reconstruction error, and improve the accuracy of three-dimensional shape reconstruction.

[0047] 5. The intelligent exposure control strategy adopted by the present invention can expand the range of light intensity for effective detection, support continuous operation in all weather conditions, and adapt to the complex lighting environment of the port.

[0048] 6. This invention uses a combined criterion based on visible light image texture features and infrared thermal images to reduce the misclassification rate of cracks and scratches, significantly improving the accuracy of container defect classification. A multi-scale feature fusion network, using a dual-branch collaborative attention mechanism, enhances the representation of small target features, improving the detection performance of small crack widths. The lightweight network design significantly reduces the number of detection model parameters compared to traditional fusion models, improving actual inference speed, meeting the requirements of high-speed pipelines, and reducing hardware costs.

[0049] 7. The present invention adopts a millimeter-level calibration algorithm based on point cloud data, combined with minimum bounding box fitting, to keep the XYZ axis error of the defect position at a low level, improving the positioning accuracy compared to traditional two-dimensional detection solutions, and providing a basis for precise maintenance.

[0050] 8. This invention improves the accuracy of classification tasks through dynamic convolution and adaptive attention mechanism, and effectively distinguishes 12 types of defects such as cracks, rust, and dents. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 This is a schematic diagram of the overall process of the container surface damage detection device and method based on machine vision of the present invention;

[0053] Figure 2 1 is a flow chart of a multimodal data acquisition module in an embodiment of the present invention;

[0054] Figure 3 is a schematic flow chart of a dynamic compensation processing module in an embodiment of the present invention;

[0055] Figure 4 1 is a flow chart of a multimodal data fusion module in an embodiment of the present invention;

[0056] Figure 5 2 is a flow chart of the damage classification and positioning module in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments.

[0058] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0059] In the following embodiments, a device and method for detecting container surface damage based on machine vision are provided. Figure 1 This is a schematic diagram of the overall process of the container surface damage detection device and method based on machine vision of the present invention, as shown in FIG. Figure 1 As shown, the following steps are included:

[0060] S1. The multimodal data acquisition module synchronously acquires the three-dimensional point cloud, multispectral image, infrared thermal image, and motion parameters of the cabinet surface. A cross-laser line scanner combined with a global shutter camera is used to reconstruct the three-dimensional topography of the cabinet surface. An infrared thermal imager is configured to detect hidden defects. An integrated six-axis accelerometer monitors the cabinet's motion status in real time. This module overcomes the limitations of traditional single-sensor detection through three major innovations: asymmetric dual-laser scanning geometry optimization, phase-locked thermal imaging defect enhancement, and multimodal data synchronous spatiotemporal acquisition. This provides a high-quality data source for subsequent dynamic compensation and defect classification, laying the foundation for the device's overall performance.

[0061] In specific implementation, the multimodal data acquisition module implementation process in step S1 is mainly divided into three parts: sensor array construction, synchronous triggering and data acquisition, and data preprocessing and transmission. Figure 2 FIG. 1 is a flow chart of a multimodal data acquisition module in an embodiment of the present invention. Figure 2 shown.

[0062] In specific implementation, the sensor array construction in step S1 mainly includes the specific installation of the cross laser line scanning device, global shutter camera, infrared thermal imager and six-axis acceleration sensor. Specifically:

[0063] Cross laser line scanning devices are installed at 45° and 135° on both sides of the gantry to ensure that the laser stripes cover the container surface and generate a density of 1000 points / cm 2 The laser divergence angle is dynamically adjusted (5°-20°) by a servo motor, adapting to a detection distance of 1.2-3m.

[0064] The specific geometric model is:

[0065]

[0066] Where Z represents the vertical distance from the measured point to the camera lens, X represents the horizontal position of the measured point in the camera coordinate system, and Y represents the vertical position of the measured point in the camera coordinate system. b = 120 mm is the baseline distance (horizontal spacing) between the laser emitter and the camera. f = 8 mm is the camera focal length. (u0, v0) are the coordinates of the camera principal point (image center).

[0067] The multispectral imaging unit utilizes a global shutter camera to simultaneously capture visible and near-infrared images. Intelligent gain control dynamically adjusts the ISO based on histogram entropy. This maintains a high signal-to-noise ratio under varying light intensities, ensuring high image quality under varying lighting conditions. Furthermore, the camera utilizes an adaptive image response mechanism, optimizing image quality under various lighting conditions by adaptively adjusting wavelength and exposure time. The global shutter camera is coaxially mounted with the laser emitter, and a focal plane filter is installed in front of the lens. The wavelength range from 450 to 900 nm is switched by a stepper motor. An intelligent gain control module is also included to automatically adjust the ISO based on the histogram entropy value.

[0068] The phase-locked thermal imaging unit detects the thermal distribution characteristics of hidden defects through periodic thermal excitation. Through phase-locked thermal imaging technology and intelligent field of view tracking, efficient detection of hidden defects in containers can be achieved. Among them, phase-locked thermal imaging mainly applies periodic thermal excitation (0.1Hz) to the surface of the box, causing a temperature phase difference between the defective area and the base material due to the difference in thermal diffusion. The thermal response characteristics of defects such as cracks are extracted through phase analysis, the thermal contrast of cracks is improved, and the size of detectable defects is reduced. Intelligent field of view tracking is mainly based on the visible light image detection results, and dynamically adjusts the field of view of the thermal imager to the suspicious area, thereby reducing the acquisition of invalid areas, improving detection efficiency, and avoiding complex background interference. The infrared thermal imaging system complements visible light and three-dimensional point cloud data to achieve multi-dimensional defect verification of "surface morphology-thermal response-spatial position" and reduce the missed detection rate.

[0069] An infrared thermal imager can be installed directly above the laser scanning area, with a field of view covering the entire scanning range. An integrated 0.1Hz periodic thermal excitation device applies a controllable heat source to the surface of the box.

[0070] A six-axis motion sensor monitors the container's motion status in real time. A six-axis acceleration sensor provides a basis for motion compensation for multimodal data acquisition. Combined with optical flow, it reduces image blur and 3D reconstruction errors, enabling adaptation to a wide range of vibration environments. This subsystem achieves high-precision monitoring of container motion through multi-sensor fusion filtering and intelligent motion pattern recognition, providing real-time data support for dynamic compensation. The multi-sensor fusion filter primarily employs an extended Kalman filter to fuse acceleration, gyroscope, and magnetometer data to suppress vibration noise. Its state equation is:

[0071] x k =Fx k-1 +Ba k

[0072] Where: x k Contains position, velocity, acceleration and attitude. F is the state transfer matrix (describing motion continuity). B is the control matrix (associated acceleration input). a k Represents the acceleration measurement value at the kth time step, usually containing acceleration components in three directions.

[0073] The intelligent motion pattern recognition system uses a hidden Markov model to distinguish between translation, rotation, and impact vibration. It dynamically adjusts filter parameters to improve the robustness of motion estimation under complex working conditions.

[0074] The six-axis accelerometer is rigidly fixed to the camera bracket and monitors vibration and displacement in real time at a sampling rate of 1000 Hz.

[0075] The joint spatiotemporal calibration method, through hardware-level synchronous triggering and software compensation algorithms, achieves high-precision spatiotemporal alignment of multi-sensor data, ensuring the synergy of multimodal detection, providing a unified spatiotemporal reference for multimodal feature fusion, and guaranteeing defect location accuracy. Hardware-level synchronous triggering uses an FPGA to generate synchronization pulses with a 10ms period and ±1μs accuracy, strictly controlling the sensor triggering sequence. The sequence is laser emission, camera exposure, and infrared acquisition. Hardware synchronization eliminates spatiotemporal misalignment caused by asynchronous sensor acquisition, supporting industrial-grade real-time requirements. Software compensation includes temporal offset compensation and spatial calibration. Temporal offset compensation uses a cross-correlation algorithm to calculate sensor timestamp offsets, compensate for temporal synchronization errors, and ensure timestamp consistency across multimodal data. Spatial calibration primarily involves camera-laser extrinsic calibration and infrared-visible light registration. Camera-laser extrinsic calibration uses Zhang's calibration method to solve the transformation matrix, achieving spatial alignment between the 3D point cloud and the image. Infrared-visible light registration uses SIFT feature matching to calculate the homography matrix, effectively reducing registration accuracy errors.

[0076] During specific implementation, the synchronous triggering and data acquisition in step S1 adopt an FPGA synchronous trigger system to generate three synchronous pulses (laser emission → camera exposure → infrared acquisition) with an interval of ≤50μs. Differential signal transmission is adopted, and the anti-interference ability reaches 40dB. The main division of labor for data acquisition is as follows: the line laser projects cross lasers at a frequency of 20kHz to generate a three-dimensional point cloud (XYZ coordinates + normal vector). The global shutter camera synchronously captures dual-channel images at 20fps. The infrared thermal imager collects phase heat maps at a frame rate of 50Hz. The six-axis sensor outputs acceleration, angular velocity and attitude data in real time.

[0077] In specific implementation, during data preprocessing and transmission in step S1, point cloud data is transferred to an industrial computer via a USB interface, and images and thermal maps are transmitted via a GigE Vision interface. The industrial computer is equipped with a PCIe 4.0 bus, ensuring a data throughput of ≥1.5GB / s and real-time storage of raw data.

[0078] The S2 motion compensation processing module achieves sub-pixel motion compensation and image quality optimization under multiple lighting conditions through multi-sensor fusion motion estimation, dynamic exposure control and vibration compensation technology. It overcomes the limitations of traditional motion compensation technology, reduces motion compensation errors, maintains high precision on the conveyor belt, and improves image quality stability under sudden changes in lighting.

[0079] In specific implementation, the process of implementing the dynamic compensation processing module in step S2 is mainly divided into a motion compensation process, an exposure optimization process and a vibration compensation process, which are respectively implemented by a composite motion estimation unit, an adaptive exposure control unit and a vibration compensation unit. Figure 3 FIG. 1 is a flow chart of a dynamic compensation processing module in an embodiment of the present invention. Figure 3 shown.

[0080] The composite motion estimation unit fuses optical flow and acceleration sensor data to estimate motion vectors, achieving sub-pixel motion compensation. Optical flow extracts motion vectors from image sequences, while the six-axis sensor predicts motion trends. Furthermore, the composite motion estimation unit dynamically adjusts the weighting of the optical flow and acceleration data using an adaptive Kalman filter to achieve sub-pixel motion compensation. This unit improves motion estimation accuracy and adapts to dynamic compensation requirements in complex working conditions.

[0081] In the specific implementation, the motion compensation process in step S2 uses the ORB algorithm to extract 500-800 feature points per frame from the image sequence. The six-axis sensor data is then filtered through a Kalman filter to predict the motion trend, generate compensation displacement, and complete motion vector prediction.

[0082] The adaptive exposure control unit uses fuzzy PID dual closed-loop control based on image entropy analysis to adjust exposure time and gain in real time, optimizing image quality, adapting to a wide lighting range, suppressing interference from sudden changes in lighting, and improving response speed. The outer loop of this dual closed-loop control adjusts exposure time based on image entropy, while the inner loop optimizes gain using Bayesian estimation.

[0083] In the specific implementation, in step S2 of the exposure optimization process, image entropy is calculated using real-time statistics of the image grayscale distribution to calculate the information entropy. Fuzzy PID control dynamically adjusts the exposure time based on the deviation between the entropy value and the target value. Bayesian estimation is used to analyze noise variance and automatically adjust ISO to suppress noise (SNR ≥ 40dB).

[0084] The vibration compensation unit uses a six-axis sensor to predict vibration displacement based on acceleration and gyroscope data, adjusting the camera's posture in advance. It uses a hidden Markov model to distinguish between translational, rotational, and impact vibrations, suppressing vibration effects and reducing image blur pixels. It supports stable detection in environments with multiple vibration ranges.

[0085] In the vibration compensation process in step S2, the six-axis sensor data is filtered through an extended Kalman filter to estimate the vibration displacement (error < 0.5mm). The camera posture is adjusted in advance to offset high-frequency vibration (5-20Hz) with an amplitude of ±3mm, ensuring image blur less than 0.5 pixels.

[0086] S3. Build a cross-modal feature fusion network through the multimodal data fusion module to enhance the feature expression of the defect area. Through multi-scale feature extraction and cross-modal feature fusion, deep fusion of multimodal data is achieved.

[0087] In specific implementation, the implementation process of the multimodal data fusion module in step S3 is mainly divided into spatiotemporal alignment processing, feature extraction and fusion, and lightweight network inference, which is achieved through a lightweight feature fusion network, cross-modal feature fusion strategy, and dual-branch collaborative attention mechanism. Figure 4 FIG. 1 is a flow chart of a multimodal data fusion module in an embodiment of the present invention. Figure 4 shown.

[0088] A lightweight feature fusion network extracts texture, thermal, and geometric features from visible light, infrared, and point cloud data. It employs hierarchical feature extraction and differentiated feature extraction strategies for different modal data types (visible light, infrared, and point cloud), fully preserving details and semantic information. Using a lightweight network structure, channel pruning, and knowledge distillation techniques significantly reduce model parameters, effectively improving computational efficiency.

[0089] The cross-modal feature fusion strategy employs a spatiotemporal alignment algorithm to upsample the infrared thermal image to the same resolution as the visible light image based on a homography matrix, ensuring spatial consistency between the different modal data. Multimodal feature concatenation is used to fuse features from the three modalities (visible light, infrared, and point cloud) along the channel dimension, and an innovative dual-branch collaborative attention mechanism is introduced to enhance feature representation in key regions.

[0090] In the spatiotemporal alignment algorithm, the infrared thermal image is upsampled by bilinear interpolation, specifically F ir-up =BilinearUpsample(F ir ,H), where BilinearUpsample is the bilinear upsampling function. Multimodal feature concatenation, concatenates the three modal features (visible light, infrared and point cloud) to obtain F concat =Concat(F vis ,F ir-up ,MaxPool(F pc )), where F vis is the visible light characteristic, F ir-up is the upsampled infrared feature, F pc Point cloud features.

[0091] In specific implementation, step S3 involves spatiotemporal alignment and extrinsic calibration, which primarily involves obtaining camera intrinsic parameters using a checkerboard target. Nonlinear optimization is used to solve the laser and camera transformation matrices (rotation error < 0.5°). SIFT features are used to match infrared and visible light images, and the homography matrix is ​​used to upsample the infrared thermal image to visible light resolution, ensuring an RMS error of ≤ 1.5 pixels.

[0092] The dual-branch collaborative attention mechanism strengthens the complementarity between global context and local features. The dual-branch collaborative attention mechanism (DB-CAM) includes a global context branch, a local feature branch, and collaborative fusion. The global context branch calculates the global context weight W through global average pooling and multi-layer perceptron. gc =MLP(GlobalAvgPool(F concat )), where GlobalAvgPool is the global average pooling operation. The local feature branch uses depthwise separable convolution to extract local feature information and obtain the local feature weight W lf =DepthwiseConv(F concat ), where DepthwiseConv is a depth-wise separable convolution operation. Collaborative fusion is the fusion of global context weights and local feature weights to obtain the final attention weight W att =σ(W gc+W lf ), where σ is the sigmoid activation function. The concatenated features are weighted by the attention weight to obtain the enhanced feature F enhanced =W att ⊙F concat , ⊙ represents channel-by-channel multiplication.

[0093] In specific implementation, feature extraction and fusion in step S3 involve extracting texture features from visible light features using the first five layers of VGG-16. Thermal features are extracted from infrared features using the fourth stage of ResNet-34. Geometric features are extracted from point cloud features using PointNet++ through farthest point sampling and sphere querying. Feature fusion is achieved by concatenating the three modal features along the channel dimension, and a dual-branch collaborative attention mechanism (DB-CAM) is used to enhance response to defect areas.

[0094] In specific implementation, the lightweight network inference in step S3 mainly designs the multi-scale feature fusion network (MSFFNet) through channel pruning and knowledge distillation to compress the model parameters and improve the inference speed.

[0095] The S4 damage classification and localization module uses an improved deep neural network to classify defects and achieves 3D localization based on point cloud data. The improved D-FINE network, which incorporates multi-scale feature fusion, dynamic convolution, and an adaptive attention mechanism, enhances the detection capabilities of damage of different types and scales.

[0096] The network structure mainly includes: backbone network, neck network and detection head.

[0097] The backbone network uses an improved ResNeXt network, adding depthwise separable convolutional layers to the original architecture to reduce parameter count and improve computational efficiency. Furthermore, a Cross-Stage Local Connection (CSLC) structure is introduced to enhance feature propagation and reuse. The CSLC structure splits the input feature map into two parts: one part passes through a series of convolutional layers, and the other part is directly connected to the output across stages. These two parts are then concatenated.

[0098] The neck network introduces PANet (Path Aggregation Network) and an improved dynamic convolution module. PANet fuses features of different scales through bottom-up and top-down path aggregation. The improved dynamic convolution module uses dynamic convolution to adaptively generate convolution kernel weights based on input features.

[0099] The detection head uses a multi-scale detection head with an adaptive attention mechanism to detect feature maps of different scales. The adaptive attention mechanism enhances the expression of key features by calculating the channel attention and spatial attention of the feature map.

[0100] Channel Attention M c =σ(MLP(GlobalAvgPool(F in ))), where σ is the sigmoid function, MLP is a multi-layer perceptron, and GlobalAvgPool is global average pooling.

[0101] The spatial attention is M s =σ(Conv(Concat(AvgPool(F in ),MaxPool(F in )))) , where AvgPool is average pooling and MaxPool is maximum pooling. The final attention feature F att =M c ⊙F in ⊙M s , ⊙ represents element-by-element multiplication.

[0102] In specific implementation, the damage classification and positioning module implementation process in step S4 is mainly divided into defect detection and classification, three-dimensional positioning process, result output and report generation. Figure 5 This is a flow chart of the damage classification and positioning module in an embodiment of the present invention. Figure 5 shown.

[0103] In specific implementation, defect detection and classification in step S4 uses the designed improved D-FINE network, which integrates dynamic convolution and adaptive attention mechanisms to output two-dimensional detection boxes. A joint loss optimization of Focal Loss and Dice Loss is used to improve the classification of small objects.

[0104] Joint loss function optimization, combining Focal Loss with label smoothing loss, improves small object detection capabilities. The improved D-FINE network uses a multi-task learning approach for damage classification, simultaneously predicting the container damage category and confidence level. The classification loss function combines Focal Loss with label smoothing loss to address data imbalance and overfitting, improving classification accuracy.

[0105] Among them, Focal Loss is Where N is the number of samples, y i is the class balancing factor, p i is the predicted probability and γ is the adjustment factor.

[0106] Label smoothing loss, let the true label be y, and the smoothed label be Where C is the number of categories and ò is the smoothing coefficient. Label smoothing loss

[0107] Total classification loss: L cls =λ1L focal +λ2L ls , where λ1 and λ2 are weight coefficients.

[0108] In the specific implementation, in step S4, the 3D positioning process uses triangulation to map the 2D detection box to a 3D point cloud coordinate system and calculate the coordinates of the defect centroid. Covariance analysis is performed on the mapped point cloud data to determine the spatial extent of the defect and achieve minimum bounding box fitting.

[0109] The 3D positioning unit calculates the millimeter-level spatial coordinates of defects based on point cloud data. The 2D detection frame is mapped to the 3D point cloud coordinate system through triangulation, and the coordinates of the defect's centroid are determined based on minimum bounding box fitting, reducing XY axis positioning errors. Based on the 2D detection results of the improved D-FINE network, the 3D point cloud data is combined for precise 3D positioning of the damage. First, the 2D detection frame is mapped to the 3D point cloud coordinate system, and then the 3D coordinates of the damaged area are determined through minimum bounding box fitting. This gives the module a significant performance advantage in the task of classifying and locating container surface damage.

[0110] The two-dimensional to three-dimensional mapping method is: let the intrinsic parameter matrix of the camera be The external parameter matrix is Among them, (f x ,f y ) is the focal length of the camera, (u0, v0) is the coordinate of the camera's principal point, R is the rotation matrix, and t is the translation vector. The pixel coordinates of the 2D detection box are (u, v), and the corresponding 3D point cloud coordinates (X, Y, Z) can be calculated using the following formula: By inversely solving the equation, we can get (X, Y, Z).

[0111] The damaged area point cloud P = {p1, p2, ..., p n}, calculate its minimum bounding box. First, calculate the covariance matrix of the point cloud in is the centroid of the point cloud. Then, perform eigenvalue decomposition on the covariance matrix C = UΣU T , where U is the eigenvector matrix and Σ is the eigenvalue matrix. A new coordinate system is established with the eigenvectors as coordinate axes. The boundary of the point cloud is calculated in the new coordinate system to obtain the parameters of the minimum bounding box.

[0112] In specific implementation, the results are output and a report is generated in step S4. The inspection results display the defect type, location, and confidence level in real time. A standardized inspection report is generated, including the defect's 3D coordinates, classification information, and repair recommendations. The report supports data integration with the port's IoT platform.

[0113] During implementation, the device and method require systematic installation, calibration, and maintenance. Checkerboard calibration is automatically performed upon initial daily operation to compensate for temperature drift. A cross-correlation algorithm is used to calibrate sensor timestamps, achieving an error of ≤50μs after compensation.

[0114] In practice, the dual laser transmitters serve as backups for each other, ensuring the system maintains high detection capabilities even if a single laser fails. The industrial computer integrates an NVIDIA GPU, supporting local inference and cloud data synchronization.

[0115] Those skilled in the art will understand that the discussion of any of the above embodiments is merely illustrative and is not intended to limit the scope of the present invention to these examples. Within the spirit and principles of the present invention, the technical features of the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and many other variations exist for the various aspects of the present invention described above, which are not provided in detail for the sake of clarity. Any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A container surface damage detection method based on machine vision, characterized in that: The method comprises the following steps: S1. Obtain multimodal data of the container, including 3D point cloud, visible light image, infrared thermal image, and motion parameters of the container surface; S2, motion compensation and exposure control based on multimodal data fusion to optimize image quality; S3. Construct a cross-modal feature fusion network to enhance the feature expression of the defect area, and achieve deep fusion of multimodal data through multi-scale feature extraction and cross-modal feature fusion; S4. Defect classification is achieved through an improved deep neural network, and three-dimensional positioning of damage is achieved based on point cloud data.

2. The method for detecting container surface damage based on machine vision according to claim 1, characterized in that: In step S1, multimodal data is calibrated using a spatiotemporal joint calibration method to achieve high-precision spatiotemporal alignment of multi-sensor data.

3. The method for detecting container surface damage based on machine vision according to claim 1, characterized in that: Step S2 specifically includes composite motion estimation, adaptive exposure control and motion compensation; The composite motion estimation includes extracting the motion vector of the image sequence by the optical flow method, predicting the motion trend according to the motion parameters, and dynamically adjusting the fusion weight of the optical flow and acceleration data by the adaptive Kalman filter to achieve motion compensation; The adaptive exposure control includes real-time adjustment of exposure time and gain based on image entropy analysis using fuzzy PID double closed-loop control. The outer loop of the double closed-loop control adjusts the exposure time based on the image entropy value, and the inner loop optimizes the gain through Bayesian estimation. Vibration compensation involves predicting vibration displacement using motion parameters, using acceleration and gyroscope data to predict vibration displacement and adjust the camera posture in advance. A hidden Markov model is used to distinguish between translation, rotation, and impact vibration, suppressing the effects of vibration and reducing image blur.

4. The method for detecting container surface damage based on machine vision according to claim 1, characterized in that: In step S3, the deep fusion of modal data includes lightweight feature fusion, cross-modal feature fusion strategy and dual-branch collaborative attention mechanism; The lightweight feature fusion extracts texture, thermal, and geometric features from visible light images, infrared thermal images, and point cloud data, respectively. It uses hierarchical feature extraction and differentiated feature extraction strategies for different modal data, fully preserving details and semantic information. It also compresses model parameters through channel pruning and knowledge distillation techniques. The cross-modal feature fusion strategy adopts a spatiotemporal alignment algorithm to upsample the infrared thermal map to the same resolution as the visible light image based on the homography matrix, uses multimodal feature stitching to fuse visible light, infrared and point cloud features along the channel dimension, and introduces a dual-branch collaborative attention mechanism to enhance the feature expression of key areas.

5. The method for detecting container surface damage based on machine vision according to claim 1, characterized in that: Step S4 specifically includes: using an improved D-FINE network to implement defect classification, introducing multi-scale feature fusion, dynamic convolution, and adaptive attention mechanism; The improved D-FINE network uses a multi-task learning approach to classify damage, simultaneously predicting the category and confidence level of container damage. The classification loss function uses a combination of FocalLoss and label smoothing loss. Based on the 2D detection results of the improved D-FINE network, the 3D point cloud data is combined to accurately locate the damage in 3D, including mapping the 2D detection box to the 3D point cloud coordinate system, and then determining the 3D coordinate location of the damaged area through minimum bounding box fitting.

6. The method for detecting container surface damage based on machine vision according to claim 5, characterized in that: The network structure of the D-FINE network includes a backbone network, a neck network, and a detection head; The backbone network adopts an improved ResNeXt network. It adds depthwise separable convolutional layers to the original ResNeXt network and introduces cross-stage local connections. The cross-stage local connection structure divides the input feature map into two parts. One part passes through a series of convolutional layers, and the other part is directly connected to the output across stages. The two parts of features are then spliced ​​together. The neck network introduces PANet and an improved dynamic convolution module. PANet aggregates features of different scales through bottom-up and top-down paths, and the improved dynamic convolution module uses dynamic convolution to adaptively generate convolution kernel weights based on input features. The detection head adopts a multi-scale detection head with an adaptive attention mechanism to detect feature maps of different scales. The adaptive attention mechanism enhances the expression of key features by calculating the channel attention and spatial attention of the feature map.

7. A container surface damage detection device based on machine vision, characterized in that: include: Multimodal data acquisition module, used to obtain multimodal data of the container, including 3D point cloud of the container surface, visible light image, infrared thermal image and motion parameters; Motion compensation processing module, which implements motion compensation and exposure control based on multimodal data fusion to optimize image quality; Multimodal data fusion module, which builds a cross-modal feature fusion network to enhance the feature expression of defect areas; The damage classification and positioning module implements defect classification through an improved deep neural network and realizes three-dimensional positioning of damage based on point cloud data.

8. The container surface damage detection device based on machine vision according to claim 7 is characterized in that: The multimodal data acquisition module includes: Cross laser line scanning device for generating 3D point clouds and supporting dynamic focus adjustment; Multispectral imaging unit, synchronously collecting visible light and near-infrared images; Lock-in thermal imaging unit detects the thermal distribution characteristics of hidden defects through periodic thermal excitation; Six-axis motion sensor to monitor the box's motion status in real time; The spatiotemporal joint calibration unit achieves high-precision spatiotemporal alignment of multi-sensor data through hardware-level synchronous triggering and software compensation algorithms.

9. The container surface damage detection device based on machine vision according to claim 8, characterized in that: The cross laser scanning device uses asymmetric dual laser lines to cross-project, forming laser stripes with an angle of 90°, generating a three-dimensional point cloud and dynamically adjusting the spot diameter.

10. The container surface damage detection device based on machine vision according to claim 8, characterized in that: The lock-in thermal imaging unit extracts the thermal diffusion characteristics of the defect through 0.1 Hz periodic thermal excitation and dynamically adjusts the detection field of view based on the visible light image.

Images