A machine vision-based fresh food packaging carton damage detection method

Abstract

The application relates to the technical field of quality detection, in particular to a fresh food packaging carton damage detection method based on machine vision, which comprises image synchronous acquisition, submodality image preprocessing, double-branch multi-modality defect detection, double-modality time sequence comprehensive detection, and detection result fusion and output; through the double-modality time sequence comprehensive detection link, based on a TCN-LSTM double-modality time sequence fusion network, in combination with the time sequence change characteristics of visible light color and structure and the time sequence change law of infrared heat characteristics, special determination logic is designed for the physical characteristics of hidden corrugated damage and water seepage; the special determination logic can not only effectively distinguish the fixed printed patterns, static stains and dynamic water seepage defects of the carton, and control the overall false detection rate to be less than 2.2%, but also can accurately identify the hidden water seepage of the inner wall without surface visual marks and the hidden corrugated damage without surface deformation.

Description

Technical Field

[0001] This invention relates to the field of quality inspection technology, and in particular to a method for detecting damage to fresh produce packaging cartons based on machine vision. Background Technology

[0002] Currently, the industry still relies mainly on manual visual sampling for damage detection of fresh produce packaging cartons. This method has inherent drawbacks such as low detection efficiency, high labor costs, insufficient full inspection coverage, and the test results being greatly affected by the subjective state of the personnel. It is completely unsuitable for the large-volume, high-speed online full inspection requirements of cold chain logistics. At the same time, the human eye cannot detect hidden defects such as water seepage on the inner wall of the carton or damage to the corrugated layer without surface deformation, resulting in an extremely high overall missed detection rate, which can easily lead to subsequent fresh produce losses and logistics safety accidents.

[0003] Chinese invention patent application CN115294517A discloses a method for detecting damage to corrugated cardboard boxes based on YOLOv5. This method only uses visible light single-modal imaging, which can only identify visible defects such as tears and holes on the surface of the cardboard box, and cannot detect hidden defects such as corrugated damage and water seepage in the inner wall that have no visible surface traces. Chinese utility model patent CN218865827U discloses a corrugated cardboard box defect detection device based on infrared thermal imaging. This method only uses a single infrared modality, which is highly susceptible to temperature fluctuations in the cold chain environment and interference from heat radiation on the surface of the cardboard box, resulting in a high false detection rate. Moreover, it cannot simultaneously cover visible defects such as surface structural damage, mold, and oil stains, and cannot meet the online full inspection requirements of fresh food cold chain logistics. In summary, neither of the existing technologies can simultaneously achieve full-category coverage of visible defects + accurate identification of hidden defects + low false detection rate online detection, and cannot meet the full-dimensional quality control requirements of fresh food cold chain scenarios. Summary of the Invention

[0004] To overcome the problems mentioned in the background art, the present invention proposes a machine vision-based method for detecting damage to fresh produce packaging cartons.

[0005] The technical solution of this invention is: a method for detecting damage to fresh produce packaging cartons based on machine vision, comprising the following steps: S1: Image synchronous acquisition. Through a visible light camera and an infrared thermal imager triggered by the same source, a continuous series of visible light images and infrared images of the fresh food packaging carton to be inspected are acquired simultaneously. Visible light time-series image sequence and infrared time-series image sequence are constructed respectively, and the frames of the two sets of sequences correspond one-to-one and are registered at the pixel level. S2: Modal image preprocessing, performing adaptive preprocessing on each frame of visible light image in the visible light time-series image sequence and each frame of infrared image in the infrared time-series image sequence to obtain the preprocessed visible light time-series image sequence and the preprocessed infrared time-series image sequence; S3: Dual-branch multimodal defect detection. A visible light-infrared dual-branch detection model is constructed. The preprocessed single-frame visible light image is input into the visible light image branch of the dual-branch detection model to detect the first detection result of structural deformation defects and color change defects of the carton to be detected. The preprocessed infrared image of the frame corresponding to the single-frame visible light image is input into the infrared image branch of the dual-branch detection model to detect the second detection result of the thermal anomaly area of ​​the carton to be detected. S4: Dual-modal temporal integrated detection. Based on the preprocessed visible light temporal image sequence and the preprocessed infrared temporal image sequence, dual-modal temporal features are extracted respectively. The dual-modal temporal features are input into the pre-trained temporal fusion network, and the third detection result of hidden corrugated damage and water leakage defects of the carton to be detected is obtained by comprehensive judgment. S5: Detection result fusion and output. The first, second and third detection results are fused at the decision level to output the damage detection grading results of the fresh food packaging cartons to be tested and the corresponding execution instructions.

[0006] Preferably, in step S1, the visible light camera and the infrared thermal imager are installed coaxially and in the same field of view, and the trigger signals are from the same source, so as to ensure that the visible light image and the infrared image collected at the same time correspond to the same physical area of ​​the carton to be inspected; for the same fresh food packaging carton to be inspected, the number of consecutive frames collected N≥10, forming a time sequence image sequence corresponding to a single carton.

[0007] Preferably, in step S2, the adaptation preprocessing is as follows: for visible light images, lens distortion correction, Gaussian filtering for noise reduction, ROI region extraction, and contrast-limited adaptive histogram equalization are performed in sequence; for infrared images, non-uniformity correction, blind pixel filling removal, reference temperature normalization, and median filtering for noise reduction are performed in sequence.

[0008] Preferably, in step S3, the visible light image branch is configured with a structural deformation detection subhead and a color change detection subhead; the structural deformation detection subhead is used to extract the edge, contour, and texture features of the visible light image, and to detect and locate structural deformation defects such as tears, holes, corner deformation, delamination, dents, and indentations; the color change detection subhead is used to extract color abnormalities in the visible light image through color space conversion and semantic segmentation, and to detect and locate color change defects such as water stains, mold spots, and oil stains.

[0009] Preferably, in step S3, the infrared image branch is used to extract the temperature distribution features of the infrared image. Based on the difference in thermal conductivity between the intact area and the defective area of ​​the carton, the thermal anomaly area is detected, and the coordinates, temperature gradient and thermal conductivity characteristic parameters of the thermal anomaly area are output. The thermal anomaly area corresponds to the hidden corrugated damage and water leakage hazard area of ​​the carton to be tested.

[0010] As a preferred embodiment, in step S3, the backbone network of the dual-branch detection model adopts an improved YOLOv8s network. A cross-modal feature interaction unit is set in the C2f module and the PAN-FPN feature fusion layer of the backbone network to perform multi-scale bidirectional fusion of the texture and color features extracted from the visible light image branch and the thermal distribution features extracted from the infrared image branch, so as to achieve feature complementarity and interference filtering.

[0011] Preferably, in step S4, before extracting the dual-modal temporal features, pixel-level registration is performed on all frames in the visible light temporal image sequence and the infrared temporal image sequence based on the ORB feature point matching algorithm and homography transformation, aligning the carton ROI regions of different temporal frames to the same pixel coordinate system, ensuring that the temporal features of the same physical location can be traced and compared.

[0012] Preferably, in step S4, the dual-modal temporal features include visible light temporal features and infrared temporal features; the visible light temporal features are the area of ​​the color anomaly region, the rate of change of color depth, and the expansion trend of the structural deformation region in the continuous frames; the infrared temporal features are the average temperature, temperature gradient, and the rate of change of temperature difference between the thermal anomaly region and the surrounding intact region in the continuous frames.

[0013] As a preferred option, in step S4, the temporal fusion network adopts a TCN-LSTM dual-modal temporal fusion network. The specific logic for comprehensive judgment is as follows: if the dual-branch detection does not find surface structure damage, but the thermal anomaly characteristics of the corresponding area in the infrared temporal features remain stable and the temperature change response speed is faster than that of the surrounding intact area, and the confidence level is ≥ the preset threshold, it is determined that there is hidden corrugated damage; if the color anomaly area in the visible light temporal features continues to spread and the color depth increases over time, while the temperature of the corresponding area in the infrared temporal features remains lower than that of the surrounding dry area, and the temperature difference change satisfies the general thermophysical rules of water evaporation—the temperature difference shows a monotonically non-increasing trend, the coefficient of variation of the inter-frame temperature difference attenuation rate is ≤0.3, and the cumulative temperature difference attenuation rate within the temporal period is ≥30%, and the confidence level is ≥ the preset threshold, it is determined that there is water seepage in the box.

[0014] As a preferred option, in step S5, the decision-level fusion adopts a three-level graded early warning mechanism, specifically: Level 1 non-compliance corresponds to obvious severe damage with a confidence level ≥ 0.85 in the first test result, directly triggering a rejection execution instruction; Level 2 non-compliance corresponds to hidden corrugated damage with a confidence level ≥ 0.75 in the third test result, triggering rejection and early warning instructions; Level 3 non-compliance corresponds to water leakage defects in the enclosure with a confidence level ≥ 0.75 in the third test result, triggering early warning and re-inspection instructions.

[0015] The beneficial effects of this invention are: 1. This invention employs a scene-specific visible light-infrared dual-branch detection model. The visible light branch simultaneously detects two types of visible defects: structural deformation and color change. The infrared branch overcomes the visual limitations of visible light. Based on the difference in thermal conductivity between defective and intact areas of the cardboard box, it enables the preliminary identification of hidden defects such as corrugated damage without surface deformation and water seepage in the inner wall. This expands the scope of defects, which existing technologies can only detect, to include internal structural defects and potential water seepage. The defect coverage is greatly improved, fundamentally reducing the core risks of fresh food spoilage and stacking collapse caused by cardboard box defects in fresh food cold chain logistics.

[0016] 2. This invention sets up a cross-modal feature interaction unit in the dual-branch detection model, and realizes multi-scale bidirectional fusion of visible light texture, color features and infrared thermal distribution features through the channel attention mechanism. It not only filters out thermal interference caused by cold chain environment heat radiation and cardboard box obstruction in infrared images through the texture features of the visible light branch, but also supplements the internal structural information that visible light cannot capture through the thermal distribution features of the infrared branch. This solves the inherent problems of single-modal detection being susceptible to environmental interference and having a high false detection rate. The explicit defect detection rate of this scheme is ≥99.2%, which is significantly better than the existing single-modal detection scheme.

[0017] 3. This invention utilizes a dual-modal temporal integrated detection process. Based on the TCN-LSTM dual-modal temporal fusion network, it combines the temporal variation characteristics of visible light color and structure with the temporal variation patterns of infrared thermal characteristics. It designs a dedicated judgment logic for the physical characteristics of hidden corrugated damage and water seepage. This not only effectively distinguishes between fixed printed patterns on cardboard boxes, static stains, and dynamic water seepage defects, keeping the overall false detection rate below 2.2%, but also accurately identifies hidden water seepage on the inner wall without surface visual traces and hidden corrugated damage without surface deformation. The detection rate of hidden corrugated damage is ≥96.8%, and the detection rate of water seepage defects is ≥97.5%, completely solving the industry problem of high missed detection rates for hidden defects in existing static detection solutions.

[0018] 4. The detection method of this invention is fully adapted to the online detection requirements of fresh food cold chain logistics conveyor lines. The hardware system can be directly embedded into existing logistics conveyor lines without modifying the original production line structure. The detection process takes into account both real-time performance and accuracy, and can adapt to the conventional conveyor line operating speed of 0.5m / s, meeting the full inspection requirements of large batches and fast cycle time. At the same time, a three-level graded early warning mechanism that fits the quality control requirements of fresh food logistics is designed, which can be directly connected to existing logistics PLC control systems and quality control management systems, and has extremely high industrial application value. Attached Figure Description

[0019] Figure 1 The diagram shown is a flowchart of the machine vision-based method for detecting damage to fresh produce packaging cartons according to the present invention. Figure 2The diagram shows the architecture of the visible light-infrared dual-branch detection model in the machine vision-based fresh food packaging carton damage detection method of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Please see Figure 1 This invention provides an embodiment: a machine vision-based method for detecting damage to fresh produce packaging cartons, including synchronous image acquisition, modal image preprocessing, dual-branch multimodal defect detection, dual-modal temporal integrated detection, and detection result fusion and output. The specific implementation methods of each step are as follows: S1: Synchronous Image Acquisition This step uses a visible light camera and an infrared thermal imager triggered by the same source to simultaneously acquire multiple consecutive frames of visible light and infrared images of the fresh food packaging carton to be inspected, and constructs visible light time-series image sequences and infrared time-series image sequences respectively. The frames of the two sets of sequences correspond one-to-one and are registered at the pixel level.

[0022] The specific implementation details of this step are as follows: Hardware System Setup: This method is suitable for online inspection scenarios in fresh food cold chain logistics conveyor lines. A coaxial, same-field-of-view visible light-infrared synchronous acquisition system is built directly above the inspection station on the conveyor line. The visible light camera used is a Baslerra L2048-48km line-scan industrial camera with a resolution of 2048×2048, a frame rate of 48fps, and an f=16mm fixed-focus industrial lens. The infrared thermal imager used is an FLIRA655sc uncooled infrared thermal imager with a temperature measurement accuracy of ±0.2℃, a spatial resolution of 640×512, and a response time of 8ms. The two imaging devices are coaxially mounted using a beam splitter prism, ensuring complete overlap of their fields of view. The installation height is 1200mm from the conveyor belt surface, covering the standard size range of fresh food cartons (300mm×200mm~600mm×400mm).

[0023] The system triggering unit uses a conveyor line PLC controller, paired with a diffuse reflection photoelectric sensor. The photoelectric sensor is installed at the entrance of the inspection station. When the carton to be inspected passes the photoelectric sensor, the PLC simultaneously outputs the same trigger pulse signal to the visible light camera and the infrared thermal imager, ensuring that the two sets of equipment complete image acquisition at the same time and eliminating timing deviation.

[0024] Temporal image sequence construction: For the same fresh food packaging carton to be inspected, in this embodiment, the conveyor line runs at a speed of 0.5 m / s, and the camera frame rate is set to 20 fps. During the entire process of a single carton passing through the inspection station, 12 frames of synchronous visible light-infrared image pairs are continuously acquired to form the visible light temporal image sequence corresponding to that carton. Infrared time-series image sequences ,in and These are the corresponding frames captured at the same time.

[0025] Inter-frame pixel-level registration: A 200mm×200mm checkerboard calibration board is used in advance to jointly calibrate the visible light camera and the infrared thermal imager, obtaining the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficients of the two devices, and completing the pixel-level registration of corresponding frames in the two sequences. The registration error is ≤0.5 pixels, ensuring that the pixel coordinates of the same physical location in the visible light image and the infrared image correspond one-to-one. In particular, this step completes the cross-modal pixel-level pre-registration of the visible light image and the infrared image acquired at the same time, ensuring that the coordinates of the same physical location correspond in the two modal images.

[0026] Specifically, the dual-camera joint calibration steps are as follows: ① A calibration plate of 200mm×200mm with a checkerboard pattern of 20mm was used. The calibration plate was placed within the detection field of view, and 15 pairs of visible light-infrared calibration images at different angles and distances were collected. ② Single-target calibration was performed on the visible light camera and the infrared thermal imager respectively, and the intrinsic parameter matrix and distortion coefficients of the two sets of equipment were obtained by solving. ③ Based on the corresponding corner points of the calibration board, perform dual-target calibration, solve for the extrinsic parameter matrix (rotation matrix R, translation vector T) between the visible light camera and the infrared thermal imager, and complete pixel-level pre-registration with a registration error ≤ 0.5 pixels.

[0027] The coaxial optical axis calibration method is as follows: Coaxial installation is achieved using a beam splitter prism. The beam splitter prism is a visible / infrared dual-band beam splitter prism with a splitting ratio of 50:50 and a working wavelength of 400nm~14μm. During the calibration process, the installation angle of the infrared thermal imager is adjusted with the optical axis of the visible light camera as a reference until the center deviation of the field of view of the two sets of equipment is ≤1 pixel, thus completing the optical axis coaxial calibration.

[0028] The trigger timing requirements are as follows: the rising edge of the same trigger pulse signal output by the PLC synchronously triggers the two sets of equipment, the pulse width is ≥100μs, the difference in the length of the trigger signal transmission line is ≤0.5m, and the acquisition timing deviation of the two sets of equipment is ≤1ms.

[0029] S2: Modal Image Preprocessing This step performs adaptive preprocessing on each frame of the visible light temporal image sequence and each frame of the infrared temporal image sequence to obtain the preprocessed visible light temporal image sequence and the preprocessed infrared temporal image sequence.

[0030] This step performs dedicated preprocessing procedures for the imaging characteristics of the two modalities of images. The specific implementation details are as follows: The visible light image preprocessing workflow performs the following operations sequentially for each frame of a visible light image: (1) Lens distortion correction: Based on the intrinsic distortion coefficients obtained by calibration in step S1, the original image is distorted to eliminate radial and tangential distortion of the lens and restore the true geometric structure of the cardboard box. (2) Gaussian filtering noise reduction: A 5×5 Gaussian kernel with a standard deviation σ=1.2 is used to filter the corrected image to eliminate Gaussian noise during image acquisition and retain the edge and texture features of the carton; (3) ROI region extraction: First, convert the RGB image to grayscale, obtain the binary mask of the carton region through OTSU adaptive threshold segmentation, then extract the outer contour of the carton through Canny edge detection, remove the background of the conveyor line, and obtain the ROI region containing only the carton to be detected. (4) Limit contrast adaptive histogram equalization (CLAHE): Set the block size to 8×8 and the contrast limit threshold to 2.0 to equalize the grayscale image of the ROI area and enhance the contrast of texture and color features such as carton edges, creases, and water stains.

[0031] The infrared image preprocessing workflow performs the following operations sequentially for each frame of infrared image: (1) Non-uniformity correction: The two-point correction method is adopted. The infrared thermal imager is calibrated in advance at blackbody temperatures of 20℃ and 40℃ to obtain the correction coefficient. The original infrared image is then non-uniformly corrected to eliminate the inherent non-uniform noise of the infrared focal plane array. (2) Blind pixel removal: The median filtering method with a 3×3 window is used to identify and replace dead pixels, overheated pixels and other blind pixels in the infrared image to eliminate imaging bad pixel interference; (3) Reference temperature normalization: The ambient temperature of the detection station is collected in real time by temperature sensor. The raw pixel temperature values ​​of the infrared image To unify and normalize to a reference ambient temperature of 25℃ and eliminate interference from ambient temperature fluctuations, the normalization formula is as follows: ,in This is the normalized pixel temperature value; (4) Median filtering for noise reduction: A 3×3 median filter kernel is used to filter the normalized infrared image to eliminate impulse noise while retaining temperature gradient edge information.

[0032] Specifically, the steps of the two-point correction method for infrared images are as follows: ① Set the blackbody to two calibration temperatures of 20℃ and 40℃ in advance. After the blackbody temperature stabilizes, the infrared thermal imager collects 100 frames of blackbody images respectively, and takes the average value to obtain the blackbody response images Y1 and Y2 at the two temperatures. ② Calculate the correction coefficients: Gain coefficient G = (40-20) / (Y2-Y1), Offset coefficient O = 20-G×Y1; ③ The non-uniformity correction of the acquired raw infrared image Y_raw is completed by the formula Y_corrected=G×Y_raw+O; ④ For every ±5℃ fluctuation in ambient temperature, perform the two-point calibration again and update the correction coefficients.

[0033] S3: Dual-branch multimodal defect detection This step constructs a visible light-infrared dual-branch detection model. The preprocessed single-frame visible light image is input into the visible light image branch of the dual-branch detection model to detect the first detection result of structural deformation defects and color change defects of the carton to be detected. The preprocessed infrared image of the frame corresponding to the single-frame visible light image is input into the infrared image branch of the dual-branch detection model to detect the second detection result of the thermal anomaly area of ​​the carton to be detected.

[0034] The core of this step is the visible light-infrared dual-branch detection model. The backbone network adopts an improved YOLOv8s network. The two branches share the bottom-level feature extraction backbone network, while the higher layers have dedicated detection heads. At the same time, cross-modal feature interaction units are set in the C2f module and the PAN-FPN feature fusion layer of the backbone network. The specific implementation details are as follows: Visible light image branch: This branch takes a preprocessed 3-channel RGB visible light image as input and sets up two parallel detection units: a structural deformation detection subhead and a color change detection subhead, to detect two types of explicit defects respectively. (1) Structural Deformation Detection Subheader: The input is the visible light multi-scale feature map extracted by the backbone network. Based on the native YOLOv8s detection head, an edge feature enhancement module is added. The Sobel operator is used to extract the edge information of the feature map and perform channel splicing with the original feature map to enhance the edge features of the structural defects. The detection categories of this subheader include six types of structural deformation defects: tear, hole, corner deformation, glue separation, dent, and indentation. The output is the category, boundingbox coordinates (x1, y1, x2, y2) and confidence of each defect, forming the structural deformation defect part in the first detection result.

[0035] (2) Color change detection subheader: The improved U-Net semantic segmentation architecture is adopted. The input is the low-level features of the visible light image. The RGB image is first converted into the HSV color space, and the hue, saturation and brightness channels are separated to extract color abnormal features. The segmentation categories of this subheader include three types of color change defects: water stains, mold spots and oil stains. The segmentation mask, area ratio and confidence of each abnormal region are output to form the color change defect part in the first detection result.

[0036] Infrared Image Branch: The input to this branch is the preprocessed infrared image. The core function is to extract the temperature distribution features of the infrared image and complete the thermal anomaly detection based on the difference in thermal conductivity between the intact and defective areas of the carton. The single-channel infrared image is copied into 3 channels according to the channel dimension, which is consistent with the input dimension and numerical normalization range of the visible light image, and matches the input requirements of the backbone network.

[0037] In this embodiment, the cardboard box to be tested is a type B corrugated cardboard box commonly used in fresh food cold chain logistics. The thermal conductivity of an intact cardboard box is approximately 0.06 W / (m·K), the thermal conductivity of the corrugated crushed / broken area is approximately 0.2 W / (m·K), and the thermal conductivity of the water-permeable area is approximately 0.58 W / (m·K). The difference in thermal conductivity will form a stable temperature difference and temperature gradient anomaly in the infrared image. This branch calculates the temperature gradient map on the infrared feature map, obtains the temperature change rate of adjacent pixels, detects the thermal anomaly area, and outputs the coordinates, average temperature, temperature gradient, temperature difference value with the surrounding intact area, and thermal conductivity characteristic parameters of the thermal anomaly area, forming the second detection result, corresponding to the hidden corrugated damage and water seepage potential areas of the cardboard box to be tested.

[0038] Cross-modal feature interaction unit: Cross-modal feature interaction units are set in the three C2f modules of the backbone network (corresponding to 8x, 16x, and 32x downsampling respectively) and the upsampling and downsampling nodes of PAN-FPN. Specifically, the feature maps of the visible light branch and the infrared branch at the same scale are weighted and fused using a channel attention mechanism (CBAM). First, channel attention weights are applied to the feature maps of the two branches separately, and then element-wise summation is performed to obtain a fused feature map, which is then input into the high-level networks of the two branches respectively. This unit achieves bidirectional feature complementarity: the texture features of the visible light branch filter out thermal interference in the infrared image caused by environmental heat radiation and cardboard box obstruction; the thermal distribution features of the infrared branch supplement internal structural information that visible light cannot capture.

[0039] Pre-training of the dual-branch detection model: (1) Dataset construction: Cardboard box samples were collected in the fresh food cold chain scenario, including intact cardboard boxes, cardboard boxes with various structural deformation defects, cardboard boxes with color change defects, cardboard boxes with hidden corrugated damage, and cardboard boxes with water leakage. A total of 12,000 pairs of visible light-infrared images were collected, with the ratio of training set, validation set and test set being 7:2:1. All samples were labeled, with structural defects labeled with bounding boxes and categories, color defects labeled with segmentation masks and categories, and infrared thermal anomaly regions labeled with corresponding defect types.

[0040] (2) Training parameter settings: The optimizer used was AdamW, with an initial learning rate of 1e-4, a batch size of 16, and 300 training epochs. A cosine annealing learning rate decay strategy was adopted. The loss function was a weighted sum of the CIoU loss for structure detection, the classification loss, the Dice loss for color segmentation, and the MSE loss for the infrared branch. After training, the model achieved an explicit defect detection rate of ≥99.2% on the test set, meeting the accuracy requirements for online detection.

[0041] Specifically, the cross-modal feature interaction unit is implemented as follows: ① Same-scale feature maps of the visible light branch and the infrared branch outputs , The weighted features are obtained by inputting them into the channel attention module CBAM. ②The channel attention branch of the CBAM module first performs global average pooling and global max pooling on the feature map, and then outputs channel weights through two shared fully connected layers. The channel weights are multiplied by the original feature map to complete channel weighting. The spatial attention branch performs channel-dimensional average pooling and max pooling on the channel-weighted feature map, and after concatenation, outputs spatial weights through a 7×7 convolution to complete spatial weighting. ③ Add the weighted feature maps element by element to obtain the fused feature map, and input it into the high-level networks of the two branches to complete the feature interaction.

[0042] Detect sub-head network parameters: ① Structural Deformation Detection Subheader: The Sobel operator uses a 3×3 convolution kernel to extract edge features in the x and y directions respectively. After concatenation, the features are concatenated with the original feature map in the channel dimension. The concatenated feature map is then input into the YOLOv8s native detection head. The number of output channels of the detection head is set to 14 (6 types of defects + 4 coordinate parameters + 1 confidence parameter + 3 class probability parameters). ② Color change detection subhead: The improved U-Net architecture adopts an encoder-decoder structure. The encoder is set to 4 downsampling layers, each layer using 2 3×3 convolutions + ReLU activation function + max pooling. The decoder is set to 4 upsampling layers, using transposed convolution to achieve upsampling. The final output channel number is 4 (background + 3 types of color defects), and the Sigmoid activation function is used to output the segmentation probability map.

[0043] Loss function and training parameters: ① Total loss function ,in The loss is CIoU for structure detection plus cross-entropy for classification. The loss is Dice loss plus cross-entropy loss for color segmentation. MSE loss in infrared thermal anomaly detection; ② Training hyperparameters: The optimizer used is AdamW, with a weight decay coefficient of 1e-4, an initial learning rate of 1e-4, a batch size of 16, 300 training epochs, and a cosine annealing learning rate decay strategy with a minimum learning rate of 1e-6. During training, Mosaic, random flipping, and brightness perturbation data augmentation methods are used.

[0044] S4: Dual-modal timing synthesis detection This step extracts bimodal temporal features from the preprocessed visible light temporal image sequence and the preprocessed infrared temporal image sequence, respectively. The bimodal temporal features are then input into a pre-trained temporal fusion network to comprehensively judge and obtain the third detection result of hidden corrugated damage and water leakage defects in the carton to be detected.

[0045] The specific implementation details of this step are as follows: Time-series frame pixel-level registration: Before extracting temporal features, the first frame of the time-series sequence is used as the reference frame. Based on the ORB feature point matching algorithm, the ORB feature points of each frame image are extracted and matched with the feature points of the reference frame. Mismatched points are eliminated by the Random Sampling Consensus Algorithm (RANSAC), and the homography transformation matrix H is calculated. Based on matrix H, the current frame is subjected to perspective transformation to align the carton ROI regions of all time-series frames to the same pixel coordinate system of the reference frame. The registration error is ≤0.3 pixels, ensuring that the pixel coordinates of the same physical location are consistent in different time-series frames, eliminating pixel offset errors caused by conveyor line movement and slight carton shaking. Among them, time-series frame pixel-level registration is used to achieve pixel alignment between different time-series frames under the same modality, ensuring traceable comparison of temporal features.

[0046] Among them, the ORB feature point matching parameters are set as follows: the FAST corner threshold for ORB feature point extraction is set to 20, the number of pyramid layers is set to 8, and the scaling factor is set to 1.2; the number of RANSAC algorithm iterations is set to 1000, the inlier threshold is set to 1.0 pixel, and mismatched points with reprojection errors greater than 1.0 pixel are removed.

[0047] Dual-modal temporal feature extraction: For the registered temporal sequence, visible light temporal features and infrared temporal features are extracted separately, specifically as follows: (1) Visible light temporal features: For each frame of visible light image, extract the area of ​​the color abnormality region. Mean color depth (The average saturation channel value in HSV space), the rate of change between adjacent frames was calculated frame by frame, resulting in 11 sets of rate of change features: , , Simultaneously, the contour expansion trend features of the structural deformation region are extracted, and a visible light temporal feature vector of length 11 is constructed. Single visible light feature dimension = 2, time series length = 11; (2) Infrared temporal characteristics: For each frame of infrared image, extract the average temperature of the thermal anomaly region. mean temperature gradient Temperature difference with the surrounding intact area ( (The average temperature of the surrounding intact area) was used to calculate the rate of change between adjacent frames, resulting in 11 sets of rate of change features: , Finally, an infrared temporal feature vector of length 11 is constructed. Single infrared feature dimension = 3, time sequence length = 11.

[0048] (3) The temporal fusion network and the input tensor shape temporal fusion network adopt the TCN-LSTM dual-modal temporal fusion network, and the dimension of the network input tensor is uniquely determined: Visible light branch input shape: (11,2) (temporal length 11, feature dimension 2); Infrared branch input shape: (11,3) (temporal length 11, feature dimension 3); The total input shape after fusion is (11,5) (temporal length 11, total feature dimension after concatenation 5).

[0049] Temporal fusion network and comprehensive judgment: The temporal fusion network adopts a TCN-LSTM dual-modal temporal fusion network, specifically: firstly, it is processed by two parallel TCN modules (temporal convolutional networks) to... and For local temporal feature extraction, the TCN module uses a 3-layer causal convolution with a kernel size of 3 and dilation coefficients of 1, 2, and 4 to capture features at different time scales. Then, the output features of the two TCN modules are concatenated and input into a bidirectional LSTM module to capture long-distance temporal dependencies. Finally, through a fully connected layer and a Softmax activation function, the classification confidence scores of the hidden corrugated damage and box leakage categories are output.

[0050] The specific logic of the comprehensive judgment is as follows: (1) Hidden corrugated damage determination: If no surface structure damage defect with a confidence level ≥ 0.5 is detected in the first detection result of the dual-branch multimodal defect detection, but the thermal anomaly characteristics of the corresponding area are consistently and stably present in the entire time series in the infrared time series, and the temperature change response speed (absolute value of temperature change rate) is more than 3 times that of the surrounding intact area, and the hidden corrugated damage classification confidence level output by the time series fusion network is ≥ 0.75, it is determined that there is hidden corrugated damage. The calculation formula for the temperature change response speed is: ,in. The temperature change values ​​are for two consecutive frames. The time interval between two frames is used; the surrounding intact area is a ring-shaped area 20 pixels outside the defective area, and the average response speed of this area is taken as the baseline value. (2) Judgment of water seepage in the box: If the area of ​​the color abnormality region in the visible light time sequence characteristics continues to expand over time, the area change rate is... In the frame, and simultaneously in the infrared temporal features, the temperature of the corresponding region is consistently lower than that of the surrounding dry region, with a temperature difference. Furthermore, the temperature difference change conforms to the thermophysical law of water evaporation. When the confidence level of the enclosure water seepage classification output by the time-series fusion network is ≥0.75, it is determined that there is enclosure water seepage. Even if the visible light branch does not detect abnormal surface color areas, but thermal anomalies consistent with the above-mentioned water seepage characteristics appear in the infrared time-series sequence, it is also determined that there is hidden water seepage on the inner wall. The above confidence threshold is determined based on the ROC curve of the test set. At a threshold of 0.75, the optimal balance between defect detection rate and false detection rate can be achieved. In practical applications, it can be adjusted within the range of 0.6~0.9 according to quality control requirements. The formula for calculating the area change rate is as follows: ,in Let n be the area of ​​the color anomaly region in the nth frame. The area of ​​the (n-1)th frame is given, and the color depth continues to increase.

[0051] The rule for determining whether temperature difference changes conform to the thermophysical laws of water evaporation is that the following three general quantitative conditions must be met simultaneously: Monotonic temperature difference decay characteristic: In continuous time frames, the inter-frame temperature difference between the suspected water seepage area and the surrounding dry area shows a monotonic non-increasing trend, with no rebound increase in temperature difference. Stable attenuation without sudden changes: The coefficient of variation (CV) of the inter-frame temperature difference attenuation rate is less than or equal to a preset threshold (coefficient of variation = standard deviation of attenuation rate / mean attenuation rate, used to quantify the stability of the attenuation process), ensuring that evaporation is a natural and continuous process without abnormal fluctuations such as sudden rises or falls. Effective cumulative attenuation characteristics: During the entire time-series detection period, the cumulative attenuation rate of temperature difference is ≥ the preset ratio (cumulative attenuation rate = (initial temperature difference - final temperature difference) / initial temperature difference × 100%), ensuring that the temperature difference attenuation is the actual thermal effect generated by water evaporation, rather than the minor fluctuations caused by environmental interference.

[0052] Preferably, the threshold calibration method is as follows: for different conveyor line speeds, for every 5fps increase in frame rate, the area change rate threshold ΔS is decreased by 0.01 / frame; for different ambient temperatures, for every 5℃ decrease in ambient temperature, the temperature difference threshold ΔT is increased by 0.2℃.

[0053] The pre-training process of the temporal fusion network is as follows: (1) Dataset construction: Collect 5000 sets of time-series images of cardboard boxes, including 1200 sets of hidden corrugated damaged samples, 1500 sets of water-seeped samples, and 2300 sets of intact samples. Label the defect category of each sequence. The ratio of training set, validation set and test set is 7:2:1.

[0054] (2) Training parameter settings: The optimizer used was Adam, with an initial learning rate of 5e-5, a batch size of 8, and 200 training epochs. An early stopping strategy was adopted, and data augmentation methods including time-series sliding window and Gaussian noise were used. Training was stopped if the validation set loss did not decrease for 10 consecutive epochs. After training, the model achieved a hidden corrugation damage detection rate of ≥96.8%, a water seepage defect detection rate of ≥97.5%, and a false detection rate of ≤2.2% on the test set.

[0055] Specifically, the complete structural parameters of the TCN-LSTM dual-modal temporal fusion network are as follows: ① The visible light feature TCN module and the infrared feature TCN module have the same structure: both use 3-layer causal convolution, with a kernel size of 3, dilation coefficients of 1, 2 and 4 respectively, the number of convolution channels set to 64, the activation function used is ReLU, and each layer is set with residual connections and Dropout layers, with the Dropout ratio set to 0.2; ② Bidirectional LSTM module: The number of hidden layers is set to 2, the number of neurons in each layer is set to 128, the time step is set to 11, the dropout ratio is set to 0.2, and the output dimension is set to 64; ③ Fully connected layer: Two fully connected layers are set, the number of neurons in the first layer is 64, and the number of neurons in the second layer is 2. Finally, the classification confidence of the two categories of corrugated damage and box leakage is output through the Softmax activation function.

[0056] The temporal feature normalization method is as follows: For the extracted visible light temporal feature vector and infrared temporal feature vector, min-max normalization is applied respectively, mapping the feature values ​​to the [0,1] interval. The normalization formula is: ,in The minimum value of the characteristic sequence. This represents the maximum value of the characteristic sequence.

[0057] S5: Detection Result Fusion and Output This step integrates the first, second, and third detection results at the decision level, and outputs the damage detection grading results of the fresh food packaging cartons to be tested and the corresponding execution instructions.

[0058] This step employs a three-tiered early warning mechanism to meet the quality control requirements of fresh food cold chain logistics. The specific implementation logic is as follows: Level 1 Non-conformance: If the first inspection result shows obvious severe damage such as tearing, holes, or large-area glue separation, and the defect confidence level is ≥0.85, it is directly judged as Level 1 Non-conformance, and a rejection execution command is sent to the conveyor line PLC system to remove the carton from the conveyor line; Level 2 non-compliance: If the first inspection result does not show any obvious serious damage that would be considered a Level 1 non-compliance, but the third inspection result indicates the presence of hidden corrugated damage with a confidence level ≥ 0.75, it is judged as a Level 2 non-compliance. A rejection and warning instruction is sent, and the defect information of the carton is recorded and uploaded to the logistics quality control system. Level 3 Non-conformity: The first inspection result does not show any obvious serious damage that would be considered a Level 1 non-conformity, but the third inspection result indicates that there is a water leakage defect in the box, with a confidence level of ≥0.75. Therefore, it is judged as a Level 3 non-conformity, and an early warning and re-inspection instruction are sent. The carton is then transported to the manual re-inspection station, and the defect information is recorded and uploaded to the quality control system. Specific Implementation This example is applied to an online inspection scenario of a conveyor line in a fresh food e-commerce cold chain logistics center. The test environment temperature is 10℃~15℃, the conveyor line speed is 0.5m / s, and the cardboard box to be tested is a 400mm×300mm×200mm B-type corrugated cardboard box commonly used in fresh food cold chains, containing fresh fruits and vegetables. The complete inspection process is as follows: Image synchronous acquisition: When the carton passes the photoelectric sensor at the detection station, the PLC outputs a synchronous trigger signal to trigger the visible light camera and infrared thermal imager simultaneously, continuously acquiring 12 frames of synchronous visible light-infrared image pairs, constructing the visible light time sequence image sequence and infrared time sequence image sequence of the carton, and achieving pixel-level registration of the two sets of sequences through pre-calibrated registration parameters. Modal image preprocessing: Following the aforementioned preprocessing process, each frame of visible light image and infrared image is preprocessed separately to obtain a preprocessed time-series image sequence; Dual-branch multimodal defect detection: The pre-processed single-frame image is input into the pre-trained dual-branch detection model. The visible light image branch detects that the cardboard box has no structural deformation defects, but there is a color abnormality area in the lower right corner of the surface, with a confidence level of 0.62. The infrared image branch detects that there is a thermal abnormality in the area corresponding to the color abnormality, with a temperature difference of 1.5℃ from the surrounding area. Dual-modal temporal integrated detection: After registering 12 frames of temporal sequence, dual-modal temporal features were extracted. Visible light temporal features showed that the area of ​​the color abnormality region continued to expand over time, with an average area change rate of 0.08 per frame, and the color depth continued to deepen. Infrared temporal features showed that the temperature of this region was consistently lower than that of the surrounding area, and the temperature difference change rate conformed to the thermophysical law of water evaporation. The confidence level of the box leakage output by the temporal fusion network was 0.92, indicating that there was a box leakage defect. Results Integration and Output: The carton has no obvious severe damage that would be considered a Level 1 defect, so it is judged to be a Level 3 defect. An early warning and re-inspection instruction is sent to the conveyor line, and the carton is transported to the manual re-inspection station. At the same time, the defect information is uploaded to the quality control system. The inspection time for a single carton is ≤80ms, which is fully compatible with the online inspection requirements of a 0.5m / s conveyor line.

[0060] During the testing process, 1,000 sets of cardboard box samples were collected, including 300 sets of intact samples, 200 sets of samples with obvious structural defects, 150 sets of samples with color change defects, 150 sets of samples with hidden corrugation damage, and 200 sets of samples with water seepage (including 50 sets of samples with hidden water seepage on the inner wall). The evaluation indicators are calculated using the following formula: Detection rate = (Number of correctly detected defective samples / Total number of actual defective samples) × 100% False positive rate = (Number of intact samples falsely identified as defects / Total number of actual intact samples) × 100% Test results: ① The detection rate of visible structural defects was 99.5%, the detection rate of color change defects was 98.9%, and the detection rate of overall visible defects was 99.2%; ②The detection rate of hidden corrugated damage was 97.3%, the detection rate of water seepage defects in the enclosure was 98.0%, and the detection rate of hidden water seepage in the inner wall was 96.0%; ③ The overall false detection rate was 1.8%, which meets the design target of ≤2.2%.

[0061] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A machine vision-based method for detecting damage to fresh produce packaging cartons, characterized in that: Includes the following steps: S1: Image synchronous acquisition. Through a visible light camera and an infrared thermal imager triggered by the same source, a continuous series of visible light images and infrared images of the fresh food packaging carton to be inspected are acquired simultaneously. Visible light time-series image sequence and infrared time-series image sequence are constructed respectively, and the frames of the two sets of sequences correspond one-to-one and are registered at the pixel level. S2: Modal image preprocessing, performing adaptive preprocessing on each frame of visible light image in the visible light time-series image sequence and each frame of infrared image in the infrared time-series image sequence to obtain the preprocessed visible light time-series image sequence and the preprocessed infrared time-series image sequence; S3: Dual-branch multimodal defect detection. A visible light-infrared dual-branch detection model is constructed. The preprocessed single-frame visible light image is input into the visible light image branch of the dual-branch detection model to detect the first detection result of structural deformation defects and color change defects of the carton to be detected. The preprocessed infrared image of the frame corresponding to the single-frame visible light image is input into the infrared image branch of the dual-branch detection model to detect the second detection result of the thermal anomaly area of ​​the carton to be detected. S4: Dual-modal temporal integrated detection. Based on the preprocessed visible light temporal image sequence and the preprocessed infrared temporal image sequence, dual-modal temporal features are extracted respectively. The dual-modal temporal features are input into the pre-trained temporal fusion network, and the third detection result of hidden corrugated damage and water leakage defects of the carton to be detected is obtained by comprehensive judgment. S5: Detection result fusion and output. The first, second and third detection results are fused at the decision level to output the damage detection grading results of the fresh food packaging cartons to be tested and the corresponding execution instructions.

2. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 1, characterized in that: In step S1, the visible light camera and the infrared thermal imager are installed coaxially and in the same field of view, and the trigger signals are from the same source, ensuring that the visible light image and the infrared image acquired at the same time correspond to the same physical area of ​​the carton to be inspected; for the same fresh food packaging carton to be inspected, the number of consecutive frames acquired N≥10, forming a time-series image sequence corresponding to a single carton.

3. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 2, characterized in that: In step S2, the adaptation preprocessing is as follows: for visible light images, lens distortion correction, Gaussian filtering for noise reduction, ROI region extraction, and contrast-limited adaptive histogram equalization are performed in sequence; for infrared images, non-uniformity correction, blind pixel filling removal, reference temperature normalization, and median filtering for noise reduction are performed in sequence.

4. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 3, characterized in that: In step S3, the visible light image branch is configured with a structural deformation detection subhead and a color change detection subhead; The structural deformation detection subhead is used to extract edge, contour, and texture features from visible light images, and to detect and locate structural deformation defects such as tears, holes, corner deformation, delamination, dents, and indentations. The color change detection subhead is used to extract color abnormalities in visible light images through color space conversion and semantic segmentation, and to detect and locate color change defects such as water stains, mold spots, and oil stains.

5. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 4, characterized in that: In step S3, the infrared image branch is used to extract the temperature distribution features of the infrared image. Based on the difference in thermal conductivity between the intact area and the defective area of ​​the carton, the thermal anomaly area is detected, and the coordinates, temperature gradient and thermal conductivity characteristic parameters of the thermal anomaly area are output. The thermal anomaly area corresponds to the hidden corrugated damage and water leakage potential area of ​​the carton to be tested.

6. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 5, characterized in that: In step S3, the backbone network of the dual-branch detection model adopts an improved YOLOv8s network. Cross-modal feature interaction units are set in the C2f module and PAN-FPN feature fusion layer of the backbone network to perform multi-scale bidirectional fusion of texture and color features extracted from the visible light image branch and thermal distribution features extracted from the infrared image branch, so as to achieve feature complementarity and interference filtering.

7. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 6, characterized in that: In step S4, before extracting the dual-modal temporal features, pixel-level registration is performed on all frames in the visible light temporal image sequence and the infrared temporal image sequence based on the ORB feature point matching algorithm and homography transformation. The ROI regions of the carton in different temporal frames are aligned to the same pixel coordinate system to ensure that the temporal features of the same physical location can be traced and compared.

8. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 7, characterized in that: In step S4, the dual-modal timing features include visible light timing features and infrared timing features; The visible light temporal characteristics are the area of ​​the color anomaly region, the rate of change of color depth, and the expansion trend of the structural deformation region in consecutive frames; the infrared temporal characteristics are the average temperature, temperature gradient, and the rate of change of temperature difference between the thermal anomaly region and the surrounding intact region in consecutive frames.

9. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 8, characterized in that: In step S4, the temporal fusion network adopts a TCN-LSTM dual-modal temporal fusion network. The specific logic for comprehensive judgment is as follows: if the dual-branch detection does not find surface structural damage, but the thermal anomaly characteristics of the corresponding area in the infrared temporal features remain stable and the temperature change response speed is faster than that of the surrounding intact area, and the confidence level is ≥ the preset threshold, it is determined that there is hidden corrugated damage; if the color anomaly area in the visible light temporal features continues to spread and the color depth increases over time, while the temperature of the corresponding area in the infrared temporal features remains lower than that of the surrounding dry area, and the temperature difference change satisfies the general thermophysical rules of water evaporation—the temperature difference shows a monotonically non-increasing trend, the coefficient of variation of the inter-frame temperature difference attenuation rate is ≤0.3, and the cumulative temperature difference attenuation rate within the temporal period is ≥30%, and the confidence level is ≥ the preset threshold, it is determined that there is water seepage in the box.

10. The method for detecting damage to fresh produce packaging cartons based on machine vision according to claim 9, characterized in that: In step S5, the decision-level fusion adopts a three-level hierarchical early warning mechanism, specifically: Level 1 non-compliance corresponds to explicit severe damage with a confidence level ≥ 0.85 in the first test result, which directly triggers the rejection execution instruction; Level 2 non-compliance corresponds to hidden corrugated damage with a confidence level ≥ 0.75 in the third test result, which triggers rejection and early warning instructions. Level 3 non-compliance corresponds to a water leakage defect in the enclosure with a confidence level ≥ 0.75 in the third test result, triggering an early warning and re-inspection instruction.

Images