Soft package sealing detection equipment and method based on vibration and infrared image fusion

Abstract

The present invention relates to a soft package sealing detection method and equipment based on the fusion of vibration and infrared images. The detection method includes: step S1, obtaining the soft package vibration signal and infrared image data; step S2, performing adaptive ROI area extraction on the soft package seal infrared image; step S3, preprocessing the soft package seal vibration signal and infrared image, including noise filtering and signal enhancement; step S4, inputting the extracted vibration signal and infrared image features into a multimodal fusion model for training; step S5, using the trained multimodal fusion model to perform joint analysis and defect detection on the vibration signal and infrared image to be tested; step S6, evaluating the detection results and locating defects. The present invention comprehensively utilizes the information of vibration signals and infrared images to improve the accuracy and robustness of detection; adopts an unsupervised learning method, does not require a large number of training samples to be labeled in advance, and simplifies the model training process; through multimodal fusion, compared with traditional algorithms, it has higher detection accuracy and stronger adaptability.

Description

Technical Field

[0001] The present invention belongs to the field of detection technology, and in particular relates to a soft package sealing detection device and method based on vibration and infrared image fusion. Background Art

[0002] Testing the leak-tightness of soft packages is crucial for a variety of industries, particularly in the food, pharmaceutical, and chemical industries. Traditional leak-tightness testing methods are often based on artificial feature engineering or rule-based approaches. However, these methods can be susceptible to interference from environmental noise, are sensitive to parameter settings, and perform poorly in complex scenarios. To address these issues, in recent years, an increasing number of studies have begun to incorporate multimodal data fusion technology into soft package leak-tightness testing. Vibration signals and infrared images, as two important information sources, have attracted widespread attention.

[0003] Vibration signals and infrared images are two commonly used non-contact inspection methods, each with unique advantages. Vibration signals can reveal the integrity and internal state of the seal structure, while infrared images can detect temperature changes at the seal, revealing possible sealing issues. Therefore, fusing vibration signals and infrared images can provide more comprehensive and accurate information, thereby improving the robustness and performance of flexible package sealing inspection.

[0004] Unsupervised learning methods have been widely used in multimodal data fusion in recent years. Unsupervised learning methods, such as autoencoders and generative adversarial networks (GANs), can automatically learn features from data and effectively learn representations. These methods do not require labeled training data, making them more suitable for multimodal data fusion tasks. Unsupervised learning methods can effectively extract and fuse features from vibration signals and infrared images, thereby improving the accuracy and robustness of soft package seal testing.

[0005] However, in the field of soft-package sealing testing, research on methods based on the fusion of vibration signals and infrared images is relatively limited. Therefore, it is of great significance to propose a soft-package sealing testing method based on the fusion of vibration signals and infrared images. This method can comprehensively utilize information from vibration signals and infrared images to achieve accurate and rapid testing of soft-package sealing, providing more reliable production quality assurance for related industries. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a soft package sealing detection device based on vibration and infrared image fusion, which includes a camera bracket, an infrared thermal imager, a heat sealing fixture, a photoelectric sensor, a PLC, a switch, an industrial computer, a vibration sensor, a spring, a fixing bracket, an extrusion device, a conveyor belt and a display;

[0007] The infrared thermal imager is mounted on a camera bracket and is aligned with the heat sealing fixture to capture infrared images of the seal and transmit the data to the switch. A photoelectric sensor is also mounted on the heat sealing fixture, which is connected in sequence to the PLC and the switch, and the switch is connected to the industrial computer.

[0008] The conveyor belt is provided with an extrusion device, which is fixed to the conveyor belt through a fixed bracket. The extrusion device is elastically supported by a spring. A vibration sensor is also provided on the extrusion device, which transmits data to an industrial computer, and the industrial computer is connected to a display.

[0009] The present invention also provides a soft package sealing detection method based on the fusion of vibration signals and infrared images, using the above-mentioned device, including the following steps:

[0010] Step S1, obtaining soft bag vibration signal and infrared image data;

[0011] Step S2, performing adaptive ROI region extraction on the soft package sealing infrared image;

[0012] Step S3, preprocessing the soft package sealing vibration signal and infrared image, including noise filtering and signal enhancement;

[0013] Step S4, inputting the extracted vibration signal and infrared image features into a multimodal fusion model for training;

[0014] Step S5, using the trained multimodal fusion model to perform joint analysis and defect detection on the vibration signal and infrared image to be tested;

[0015] Step S6: Evaluate the test results and determine whether there is a defect in the flexible package seal based on the abnormality score. If there is a defect, locate the defective part.

[0016] The step of performing adaptive ROI region extraction on the soft package seal infrared image in step S2 includes:

[0017] Step S21, performing three-channel weighted averaging on the acquired infrared image to achieve grayscale processing;

[0018] Step S22: smoothing and denoising the grayscale image to remove fine textures and irregular noise points. The smoothing and denoising method is Gaussian filtering, and its expression is:

[0019]

[0020] Where (x, y) represents the pixel coordinates in the image, G(x, y) represents the Gaussian function value calculated at that location, σ represents the standard of the Gaussian kernel, and e is the constant base of the natural logarithm function;

[0021] Step S23, using Canny edge detection to extract the edge contour of the smoothed grayscale image and obtain a binarized edge image;

[0022] In step S24, a contour extraction algorithm is used to first traverse the binary image to find all completely closed edge contours. For each closed contour, its circumscribed rectangle is calculated—that is, the smallest rectangle enclosing the contour. Then, the largest rectangle is selected from all the rectangles to ensure that the entire target area is covered as much as possible. After selecting the largest rectangle, the corresponding area is cropped from the original image based on its position and size. This area is considered the region of interest (ROI) of the infrared image of the aluminum foil seal.

[0023] The step of pre-processing the soft package sealing infrared image and the vibration signal in step S3 includes:

[0024] Step S31: perform image enhancement on the extracted ROI image. First, the contrast of the image is enhanced to improve the visibility of image details. Specifically, adaptive histogram equalization is used, and the formula is as follows:

[0025] Assume that the input image is I and the output image is I eq , the size of the image is M×N, and the size of the local region is p×q.

[0026] For each pixel (x, y), its corresponding local area is R xy , then the calculation formula of adaptive histogram equalization is:

[0027] I eq (x,y)=HE(R xy )(I(x,y))

[0028] Among them, HE(R xy ) represents the local area R xy Perform histogram equalization operation, where I(x,y) is the grayscale value of the pixel in the input image.

[0029] Local area R xy The histogram equalization can be achieved by the following steps:

[0030] First, calculate the local region R xy The cumulative distribution function (CDF) of :

[0031]

[0032] Among them, n i represents the number of pixels with pixel value i, and pq is the total number of pixels in the local area.

[0033] Secondly, according to the cumulative distribution function, the pixel value after histogram equalization is calculated:

[0034] I eq (x,y)=(L-1)×CDF xy (I(x,y))

[0035] Where L is the number of gray levels of the image.

[0036] Through such processing, histogram equalization of the local area of the image can be achieved, thereby enhancing the contrast and details of the image.

[0037] Then, the Laplacian filter is used to sharpen the image and enhance the edges and details of the image;

[0038] In step S32, the enhanced image samples are interpolated and upsampled to increase the resolution of the infrared image and provide more image information for subsequent deep learning model training. Biquadratic interpolation is used. Biquadratic interpolation is a simple interpolation method that estimates the value of a new pixel at each pixel position in the image by taking a weighted average of the four surrounding pixels. These four pixels are usually the four pixels closest to the position to be interpolated. The interpolation formula is as follows:

[0039]

[0040] Where f(x,y) is the pixel value at position x and y, and a ij are the coefficients of the biquadratic interpolation.

[0041] Step S33, randomly rotating the image to expand the training data set and increase the robustness of the model;

[0042] Step S34: normalize the image pixel values to between 0 and 1, and standardize the normalized image pixel values to have a mean of 0 and a variance of 1 to accelerate model convergence. The formula is:

[0043]

[0044] Where x norm is the normalized result, x std is the standardized result, x org is the pixel value of the image, μ and σ are the mean and standard deviation of the pixel values of a single channel;

[0045] In step S35, the vibration signal s(t) is analyzed in the time domain to calculate the mean μ and standard deviation σ of the vibration signal data. Then, according to the 3σ principle, the data outside the range of μ±3σ is regarded as outliers or noise and is eliminated.

[0046]

[0047] Where N is the total number of data points, s i represents the value of the i-th data point, μ represents the mean of the vibration signal data, and σ is the standard deviation.

[0048] Step S36, perform frequency domain analysis on the vibration signal, use Fourier transform to convert the signal s(t) into the frequency domain, S(f) and calculate the energy distribution P of the signal at different frequencies in the frequency domain. norm .

[0049]

[0050] P(f)=|S(f) 2

[0051] Where S(f) is the Fourier transform of the vibration signal, which represents the complex representation of the signal in the frequency domain, and P(f) is the energy spectral density, which represents the signal energy distribution at frequency f.

[0052] Step S37, normalize the vibration signal s(t) to the range of [0,1], S norm Right now.

[0053]

[0054] Where s norm represents the normalized vibration signal value, s represents a value of the original vibration signal, s min Represents the minimum value in the original vibration signal data, s max Indicates the maximum value in the original vibration signal data.

[0055] In step S38, the preprocessed image dataset and vibration signal dataset are divided into training and test sets, with normal sample data accounting for 75% as the training set and the remaining 25% of normal samples and defective samples as the test set.

[0056] In step S4, the multimodal fusion network mainly includes a vibration signal feature extraction module, an infrared image feature extraction module, a feature fusion module and a discrimination module.

[0057] The vibration signal feature extraction module uses a one-dimensional convolutional neural network (CNN) as the feature extractor of the vibration signal. First, the input layer receives data of the shape of (signal length, number of signal channels). Subsequently, three convolutional layers are used, each followed by a ReLU activation function, containing 64, 32 and 16 convolution kernels, respectively, with sizes of 3, 5 and 7, and a step size of 1. This is followed by three maximum pooling layers with a pooling window size of 2 and a step size of 2 to reduce the feature dimension. Then there is a flattening layer that flattens the output of the pooling layer into a one-dimensional vector. This is followed by a fully connected layer containing 128 neurons and a ReLU activation function. Finally, the output layer uses a sigmoid activation function for the probability output of the binary classification problem. This design can be adjusted according to actual conditions to adapt to different data and task requirements.

[0058] The infrared image feature extraction module uses the classic convolutional neural network (CNN) architecture ResNet50 as the feature extractor for infrared images. The ResNet-50 model consists of four residual blocks, each of which contains several convolutional layers and skip connections.

[0059] Specifically, each residual block contains three convolutional layers. Except for the first residual block, each residual block is followed by a downsampling layer to ensure dimensionality reduction in the feature map size. In ResNet-50, the number of these residual blocks is [3, 4, 6, 3], where the first number represents the number of convolutional layers in the first residual block, and so on. Within each residual block, the convolutional layer has a size of 3x3, a stride of 1, and padding of "same" to maintain the size of the feature map. The pooling layer uses max pooling of 3x3 with a stride of 2 to reduce the size of the feature map. This design fully extracts image features and mitigates the vanishing gradient problem through skip connections, ensuring effective model training.

[0060] The feature fusion module first normalizes and scales the feature vectors from the vibration signal and infrared image feature extractors to ensure they have similar scales and ranges. Next, an attention mechanism is considered to regulate the contribution of each modal feature to dynamically adjust the degree of feature fusion, thereby better capturing the relevance and importance between different modalities. After determining the order and ranking of the features, the processed and adjusted feature vectors of the vibration signal and infrared image are concatenated along the feature dimension, stacking them together on the feature dimension. Finally, the feature information of the two modalities is fused together through a weighted summation to form the final fused feature representation. Through this design, the feature fusion module can fully integrate the information of the vibration signal and infrared image, and perform flexible feature fusion on this basis to produce a more comprehensive and rich feature representation.

[0061] The step of using the trained multimodal fusion model to perform joint analysis and defect detection on the vibration signal and infrared image in step S5 includes:

[0062] In step S51, the processed image and vibration signal are input into the model for parameter training. The cross entropy loss function is used as the classification loss function. For two categories (qualified and unqualified), the cross entropy loss between the model output and the true label is calculated; the square loss function is introduced to compare the difference between the defect position output by the model and the true defect position as an auxiliary loss function for the defect localization task.

[0063] In step S52, the trained network weights are fixed and, for a given number of clusters K, the feature vectors of K cluster centers are randomly initialized. For each input feature vector P, its distance to all cluster centers is calculated, and the feature vector is assigned to the cluster center with the closest distance. For each cluster, the mean of all samples in the cluster is calculated, and this mean is used as the new cluster center. The above two steps are repeated until the stopping condition is met (e.g., the cluster center no longer changes significantly or the maximum number of iterations is reached).

[0064] Among them, the distance C between the feature vector P and the cluster center is calculated i The Euclidean distance formula is usually used. For a given cluster center C i And the feature vector P, the Euclidean distance calculation formula is as follows:

[0065]

[0066] Where n is the dimension of the feature vector, P j represents the jth eigenvalue of the eigenvector P, C ij Represents the cluster center C i The j-th eigenvalue of .

[0067] In step S6, the test results are evaluated and the abnormality score is used to determine whether there is a defect in the flexible packaging seal. If there is a defect, the step of locating the defective part includes:

[0068] Step S61, determining the abnormality score of the input data by calculating the distance between each sample and its cluster center;

[0069] Step S62: Set an appropriate abnormality score threshold based on the statistical information of the training data to distinguish normal samples from abnormal samples.

[0070] Step S63: Each sample is judged according to the set abnormality score threshold, and samples with abnormality scores higher than the threshold are marked as abnormal samples, indicating that there are defects;

[0071] In step S64, for samples determined to be abnormal samples, their features are further analyzed to locate the defective parts. This is achieved by visualizing the feature representation of the abnormal samples and using auxiliary technology, namely, region-based networks.

[0072] Compared with the prior art, the beneficial effect of the present invention is that the present invention adopts an unsupervised learning method and utilizes the multimodal data of vibration signals and infrared images for fusion to realize soft package sealing detection. Compared with traditional methods, this method can not only automatically learn the characteristics of vibration signals and infrared images, but also effectively fuse them, thereby improving the accuracy and robustness of detection. By integrating the information of vibration signals and infrared images, the present invention can more comprehensively capture the characteristics of sealing problems, thereby reducing the risk of missed detection and false detection, and improving the reliability of detection. In addition, since the unsupervised learning method does not require a large amount of labeled training data and does not require manual design of feature extractors, the cost of the system is reduced. By processing and analyzing vibration signals and infrared images in real time, the present invention is suitable for online monitoring and control on production lines, and is expected to provide more reliable production quality assurance for related industries.

[0073] Other features and advantages of the present invention will be described in detail in the following description, and some of them can be directly seen from the description or specific embodiments.

[0074] In order to make the above-mentioned objects, advantages and features of the invention clearer and more intuitive, the following is a detailed description of preferred implementation cases of the invention and related drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the specific implementation cases of the present invention or the existing technical solutions, the specific implementation plans of the present invention will be described with relevant drawings below.

[0076] Figure 1 This is a schematic flow chart of a specific embodiment of the method for detecting the sealing performance of a soft package according to the present invention;

[0077] Figure 2 Schematic diagram of the steps of the method for testing the sealing performance of a soft package according to the present invention;

[0078] Figure 3 This is a schematic diagram of a flexible package sealing test device according to the present invention.

[0079] In the figure: 1. Camera bracket; 2. Infrared thermal imager; 3. Heat sealing fixture; 4. Photoelectric sensor; 5. PLC; 6. Flexible packaging bag; 7. Switch; 8. Industrial computer; 9. Vibration sensor; 10. Spring; 11. Fixed bracket; 12. Extrusion device; 13. Conveyor belt; 14. Display. DETAILED DESCRIPTION

[0080] To more clearly illustrate the objectives, technical solutions, and advantages of the various embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the examples described herein represent only some of the embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0081] Example:

[0082] Figure 2 This is a schematic diagram of the steps of the soft package sealing detection method based on the fusion of vibration signal and infrared image provided by an embodiment of the present invention, with reference to Figure 1 The specific steps of the method are as follows:

[0083] Step S1, obtaining soft bag vibration signal and infrared image data;

[0084] Step S2, performing adaptive ROI region extraction on the soft package sealing infrared image;

[0085] Step S3, preprocessing the soft package sealing vibration signal and infrared image, including noise filtering and signal enhancement;

[0086] Step S4, inputting the extracted vibration signal and infrared image features into a multimodal fusion model for training;

[0087] Step S5, using the trained multimodal fusion model to perform joint analysis and defect detection on the vibration signal and infrared image to be tested;

[0088] Step S6: Evaluate the test results and determine whether there is a defect in the flexible packaging seal based on the abnormality score. If there is a defect, locate the defective part.

[0089] In this embodiment, a soft package sealing detection device based on multimodal fusion of vibration and infrared images is used, including a camera bracket 1, an infrared thermal imager 2, a heat sealing fixture 3, a photoelectric sensor 4, a PLC 5, a switch 7, an industrial computer 8, a vibration sensor 9, a spring 10, a fixing bracket 11, an extrusion device 12, a conveyor belt 13, and a display 14;

[0090] The infrared thermal imager 2 is mounted on the camera bracket 1 and is aligned with the heat sealing fixture 3 to capture the seal infrared image and transmit the data to the switch 7. The heat sealing fixture 3 is also equipped with a photoelectric sensor 4, which is connected to the PLC 5 and the switch 7 in sequence. The switch 7 is connected to the industrial computer 8.

[0091] The conveyor belt 13 is provided with an extrusion device 12, which is fixed to the conveyor belt 13 through a fixing bracket 11. The extrusion device 12 is elastically supported by a spring 10. A vibration sensor 9 is also provided on the extrusion device 12, which transmits data to the industrial computer 8. The industrial computer 8 is connected to the display 14.

[0092] An infrared thermal imager 2 captures infrared images of the seals, while a squeeze device 12 squeezes the soft packages 6 on a conveyor belt 13. A vibration sensor 9 mounted on the device captures vibration signals. The infrared images and vibration signals are then used to perform non-destructive defect detection on the soft package seals. This method utilizes a neural network based on unsupervised learning, eliminating the need for defective samples and associated manual annotation, significantly reducing preparation time and labor costs. Furthermore, a feature-level fusion method is used to fuse the feature information of the vibration signals and infrared images.

[0093] First, features are extracted from both the vibration signal and the infrared image. These features are then fused into a common feature space, directly concatenating the two features using a splicing method to form a larger feature vector. This method fully utilizes information from both the vibration signal and the infrared image, improving the detection system's ability to detect soft-package sealing issues.

[0094] Through feature-level fusion, the system can more comprehensively describe the state and characteristics of an object, thereby enhancing the robustness and accuracy of the detection system. As two distinct information sources, vibration signals and infrared images are highly complementary. Fusion can compensate for each other's shortcomings, improving the performance of the soft-package seal inspection system. This fusion method is highly applicable and practical, and can be widely used in production areas such as bottling lines, providing a reliable quality control tool for the production process.

[0095] In this embodiment, the infrared image of the soft package seal in step S1 can be obtained using an infrared thermal imager 32mini with a resolution of 384×288, located on a camera bracket 1, as shown in Figure 2. This is installed 0.5m to 1m behind the heat-sealing clamp 3, and in conjunction with the photoelectric signal transmitted by the photoelectric sensor 4 at the clamping position, an infrared image of the seal is obtained on the front of the soft package. The vibration signal acquisition process is as follows: First, the conveyor belt 13 transports the soft package 6 to the extrusion device 12, which then applies appropriate pressure to the soft package, and the vibration sensor 9 near the extrusion device acquires the vibration signal. Throughout this process, the infrared thermal imager 2 controls the PLC 5, which receives the photoelectric signal from the photoelectric sensor. Real-time communication between the host and slave computers is achieved through the switch 7. The operation interface consists of a display 14 and an industrial computer 8.

[0096] The step of performing adaptive ROI region extraction on the aluminum foil seal infrared image in step S2 includes:

[0097] Step S21, performing three-channel weighted averaging on the acquired infrared image to achieve grayscale processing;

[0098] For example, the weighted average formula for the three channels may be:

[0099] Gray=0.1140B+0.5870G+0.2989R

[0100] Where B, G, and R are the pixel values of the blue, green, and red channels, and the weights before the channels are calculated from the infrared image dataset in this embodiment;

[0101] Step S22: smoothing and denoising the grayscale image to remove fine textures and irregular noise points. The smoothing and denoising method is Gaussian filtering, and its expression is:

[0102]

[0103] Where (x, y) represents the pixel coordinates in the image, G(x, y) represents the Gaussian function value calculated at that location, σ represents the standard of the Gaussian kernel, and e is the constant base of the natural logarithm function;

[0104] Step S23, using Canny edge detection to extract the edge contour of the smoothed grayscale image and obtain a binarized edge image;

[0105] In step S24, a contour extraction algorithm is used to traverse the pixel values of the binary image, extract all completely closed edge contours in the edge image, and use the bounding box with the largest width, that is, the circumscribed rectangle, to mark the target area. According to the position and size of the circumscribed rectangle, the corresponding area is cropped from the original image. The corresponding area obtained is the ROI area of the aluminum foil seal infrared image.

[0106] The steps of pre-processing the extracted infrared images and vibration signals and preparing normal samples into a data set in step S3 include:

[0107] Step S31: perform image enhancement on the extracted ROI image. First, the contrast of the image is enhanced to improve the visibility of image details. Specifically, adaptive histogram equalization is used, and the formula is as follows:

[0108] Assume that the input image is I and the output image is I eq , the size of the image is M×N, and the size of the local region is p×q.

[0109] For each pixel (x, y), its corresponding local area is R xy , then the calculation formula of adaptive histogram equalization is:

[0110] I eq (x,y)=HE(R xy )(I(x,y))

[0111] Among them, HE(R xy ) represents the local area R xy Perform histogram equalization operation, where I(x,y) is the grayscale value of the pixel in the input image.

[0112] Local area R xy The histogram equalization can be achieved by the following steps:

[0113] First, calculate the local region R xy The cumulative distribution function (CDF) of :

[0114]

[0115] Among them, n i represents the number of pixels with pixel value i, and pq is the total number of pixels in the local area.

[0116] Secondly, according to the cumulative distribution function, the pixel value after histogram equalization is calculated:

[0117] I eq (x,y)=(L-1)×CDF xy (I(x,y))

[0118] Where L is the number of gray levels of the image.

[0119] Through such processing, histogram equalization of the local area of the image can be achieved, thereby enhancing the contrast and details of the image.

[0120] Then, the Laplacian filter is used to sharpen the image and enhance the edges and details of the image;

[0121] In step S32, the enhanced image samples are interpolated and upsampled to increase the resolution of the infrared image and provide more image information for subsequent deep learning model training. Biquadratic interpolation is used. Biquadratic interpolation is a simple interpolation method that estimates the value of a new pixel at each pixel position in the image by taking a weighted average of the four surrounding pixels. These four pixels are usually the four pixels closest to the position to be interpolated. The interpolation formula is as follows:

[0122]

[0123] Where f(x,y) is the pixel value at position x and y, and a ij are the coefficients of the biquadratic interpolation.

[0124] In this example, the upsampled image size is 224×224;

[0125] Step S33, randomly rotating the image to expand the training data set and increase the robustness of the model;

[0126] In this example, to ensure the integrity of the image edges after rotation, background pixel values are padded on all four sides of the image to a size of 256×256 before rotation. After a random clockwise or counterclockwise rotation of 10 degrees, the image is cropped along the center to a size of 224×224 to complete the dataset expansion.

[0127] Step S34: normalize the image pixel values to between 0 and 1, and standardize the normalized image pixel values to have a mean of 0 and a variance of 1 to accelerate model convergence. The formula is:

[0128]

[0129] Where x norm is the normalized result, x std is the standardized result, x org is the pixel value of the original image, μ and σ are the mean and standard deviation of the pixel values of a single channel;

[0130] In step S35, the vibration signal s(t) is analyzed in the time domain to calculate the mean μ and standard deviation σ of the vibration signal data. Then, according to the 3σ principle, the data outside the range of μ±3σ is regarded as outliers or noise and is eliminated.

[0131]

[0132] Where N represents the total number of data points, s irepresents the value of the i-th data point, μ represents the mean of the vibration signal data, and σ is the standard deviation.

[0133] Step S36, perform frequency domain analysis on the vibration signal, use Fourier transform to convert the signal s(t) into the frequency domain, S(f) and calculate the energy distribution P of the signal at different frequencies in the frequency domain. norm .

[0134]

[0135] P(f)=|S(f) 2

[0136] Where S(f) is the Fourier transform of the vibration signal, which represents the complex representation of the signal in the frequency domain, and P(f) is the energy spectral density, which represents the signal energy distribution at frequency f.

[0137] Step S37, normalize the vibration signal s(t) to the range of [0,1], S norm Right now.

[0138]

[0139] Where s norm represents the normalized vibration signal value, s represents a value of the original vibration signal, s min Represents the minimum value in the original vibration signal data, s max Indicates the maximum value in the original vibration signal data.

[0140] In step S38, the preprocessed image dataset and vibration signal dataset are divided into training and test sets, with normal sample data accounting for 75% as the training set and the remaining 25% of normal samples and defective samples as the test set.

[0141] In step S4, the multimodal fusion network mainly includes a vibration signal feature extraction module, an infrared image feature extraction module, a feature fusion module and a discrimination module.

[0142] Step S41, extract the vibration signal features, and select a one-dimensional convolutional neural network (CNN) as the feature extractor of the vibration signal. First, the input layer receives data of shape (signal length, number of signal channels). Subsequently, three convolutional layers are used, each followed by a ReLU activation function, containing 64, 32 and 16 convolution kernels, respectively, with sizes of 3, 5 and 7, and a step size of 1. This is followed by three maximum pooling layers with a pooling window size of 2 and a step size of 2, which are used to reduce the feature dimension. This is followed by a flattening layer, which flattens the output of the pooling layer into a one-dimensional vector. This is followed by a fully connected layer containing 128 neurons and a ReLU activation function. Finally, the output layer uses a sigmoid activation function for the probability output of the binary classification problem.

[0143] The overall calculation of vibration signal feature extraction can be divided into the following three stages:

[0144] The first is the convolution and activation function stage:

[0145] Z (1) =Conv(X,W (1) ,b (1) )

[0146] A (1) =ReLU(Z (1) )

[0147] Next is the pooling stage:

[0148] P (1) =MaxPooling(A (1) ,pool_size=2,strides=2)

[0149] Finally, there is the full connection and output stage:

[0150] Y=Sigmoid(ReLU(Flatten(P (1) )·W (FC) +b (FC) ))

[0151] Step S42 extracts infrared image features. This step uses the classic convolutional neural network (CNN) architecture, the ResNet50 network model, as the infrared image feature extractor. The ResNet-50 model consists of four residual blocks, each of which contains several convolutional layers and skip connections. Specifically, each residual block contains three convolutional layers. Except for the first residual block, each residual block is followed by a downsampling layer to ensure dimensionality reduction of the feature map. In ResNet-50, the number of residual blocks is [3, 4, 6, 3], where the first number represents the number of convolutional layers in the first residual block, and so on. In each residual block, the convolutional layer has a size of 3x3, a stride of 1, and padding to "same" to maintain the size of the feature map. The pooling layer uses max pooling of 3x3 with a stride of 2 to reduce the size of the feature map. This hierarchical feature extraction helps capture important information in the image and provides rich semantic expression capabilities for subsequent feature fusion.

[0152] In step S43, feature fusion is performed by concatenating the feature vectors of the vibration signal and infrared image into a single, longer feature vector, forming a common feature space. This method fuses the information of the vibration signal and infrared image, resulting in a more comprehensive and rich feature representation. This feature fusion approach can enhance the model's ability to understand data from different modalities, improving the performance and robustness of subsequent classification tasks.

[0153] Step S44, classify and discriminate the fused data input into the network model. In the discrimination module, the K-means clustering algorithm is first used to divide the spliced feature vectors into two clusters, representing two different categories or states. K-means clustering is an unsupervised learning algorithm that can effectively assign data points to K different clusters so that each data point belongs to the cluster closest to it. Then, for each cluster, it is judged whether it belongs to category 1 or category 2 based on the characteristic distribution or other attributes of the samples in the cluster. This is achieved by analyzing the true labels of the samples near the cluster center or the characteristic distribution of the cluster center. The clustering results are thus matched with the true labels, providing an important basis for subsequent model evaluation and performance improvement. The discrimination module designed in this way can effectively classify and judge the sealing problems of soft packages, thereby improving the accuracy and robustness of detection.

[0154] In the embodiment, the step of using the trained multimodal fusion model to jointly analyze the vibration signal and infrared image to be measured and perform defect detection in step S5 includes:

[0155] Step S51: fix the trained network weights, and for a given number of clusters K, randomly initialize the feature vectors of K cluster centers.

[0156] Step S52: For each input feature vector P, calculate its distance from all cluster centers, and assign the feature vector to the cluster center with the closest distance.

[0157] Among them, the distance C between the feature vector P and the cluster center is calculated i The Euclidean distance formula is usually used. For a given cluster center C i And the feature vector P, the Euclidean distance calculation formula is as follows:

[0158]

[0159] Where n is the dimension of the feature vector, P j represents the jth eigenvalue of the eigenvector P, C ij Represents the cluster center C i The j-th eigenvalue of .

[0160] Step S53: For each cluster, calculate the mean of all samples in the cluster, and use the mean as the new cluster center.

[0161] Repeat the above two steps until the stopping condition is met (such as the cluster center no longer changes significantly or the maximum number of iterations is reached).

[0162] In the embodiment, the test results are evaluated in step S6, and whether there is a defect in the flexible package seal is determined based on the abnormality score. If there is a defect, the step of locating the defective part includes:

[0163] Step S61, determining the abnormality score of the input data by calculating the distance between each sample and its cluster center;

[0164] Step S62: Set an appropriate abnormality score threshold based on the statistical information of the training data to distinguish normal samples from abnormal samples.

[0165] Step S63: Each sample is judged according to the set abnormality score threshold, and samples with abnormality scores higher than the threshold are marked as abnormal samples, indicating that there are defects;

[0166] In step S64, for samples determined to be abnormal samples, their features are further analyzed to locate the defective parts. This is achieved by visualizing the feature representation of the abnormal samples and using auxiliary technology, namely, region-based networks.

[0167] In the embodiments provided herein, it is apparent that the disclosed methods may also be implemented in other ways and are not limited to the methods described above. For example, in the steps described in the detection method flow chart in the accompanying drawings, certain processes may be performed simultaneously, or the order of the steps may be changed, all of which may depend on the specific functional requirements. Furthermore, each block shown in the flow chart may be implemented using a dedicated hardware system that performs the relevant function or action, or through a combination of a computer program and hardware.

[0168] If the functions are implemented in the form of software modules and sold or used as separate products, the system can be stored in a readable storage medium. Based on this, the technical solution of the present invention, or part of the technical solution, can essentially be a software product, which contains a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to implement all or part of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard drives, RAM, ROM, optical disks, and other media that can store programs.

[0169] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A method for detecting the sealing properties of soft packages based on the fusion of vibration signals and infrared images, utilizing a soft package sealing detection device based on the fusion of vibration signals and infrared images. The detection device includes a camera bracket, an infrared thermal imager, a heat sealing fixture, a photoelectric sensor, a PLC, a switch, an industrial computer, a vibration sensor, a spring, a fixing bracket, an extrusion device, a conveyor belt, and a display. The infrared thermal imager is mounted on a camera bracket and is aligned with the heat sealing fixture to capture infrared images of the seal and transmit the data to the switch. A photoelectric sensor is also mounted on the heat sealing fixture, which is connected in sequence to the PLC and the switch, and the switch is connected to the industrial computer. The conveyor belt is provided with an extrusion device, which is fixed to the conveyor belt by a fixing bracket. The extrusion device is elastically supported by a spring. The extrusion device is also provided with a vibration sensor, which transmits data to an industrial computer, and the industrial computer is connected to a display; It is characterized in that The steps include: Step S1, obtaining soft bag vibration signal and infrared image data; Step S2, performing adaptive ROI region extraction on the soft package sealing infrared image; Step S3, preprocessing the soft package sealing vibration signal and infrared image, including noise filtering and signal enhancement; Step S4, inputting the extracted vibration signal and infrared image features into a multimodal fusion model for training; The multimodal fusion network includes a vibration signal feature extraction module, an infrared image feature extraction module, a feature fusion module and a discrimination module; The vibration signal feature extraction module uses a one-dimensional convolutional neural network as a feature extractor for vibration signals. First, the input layer receives data in the shape of signal length and number of signal channels. Subsequently, three convolutional layers are used, each followed by a ReLU activation function, containing 64, 32, and 16 convolution kernels, sizes of 3, 5, and 7, respectively, with a step size of 1. This is followed by three maximum pooling layers with a pooling window size of 2 and a step size of 2 to reduce the feature dimension. This is followed by a flattening layer to flatten the output of the pooling layer into a one-dimensional vector. This is followed by a fully connected layer containing 128 neurons and a ReLU activation function. Finally, the output layer uses a sigmoid activation function for the probability output of the binary classification problem. The infrared image feature extraction module uses the classic convolutional neural network architecture ResNet50 as the feature extractor for infrared images. The ResNet-50 model consists of four residual blocks, each of which contains several convolutional layers and skip connections. The feature fusion module first normalizes and scales the feature vectors from the vibration signal and infrared image feature extractors to ensure that they have similar scales and ranges. Next, an attention mechanism is introduced to regulate the contribution of each modal feature to dynamically adjust the degree of feature fusion, thereby better capturing the relevance and importance between different modalities. After determining the order and ranking of the features, the processed and adjusted feature vectors of the vibration signal and infrared image are concatenated along the feature dimension, that is, they are stacked together in the feature dimension. Finally, the feature information of the two modalities is fused together through weighted summation to form the final fused feature representation. Step S5, using the trained multimodal fusion model to perform joint analysis and defect detection on the vibration signal and infrared image to be tested; Step S6: Evaluate the test results and determine whether there is a defect in the flexible package seal based on the abnormality score. If there is a defect, locate the defective part.

2. The soft package sealing detection method based on the fusion of vibration signal and infrared image according to claim 1 is characterized in that: The step of performing adaptive ROI region extraction on the soft package seal infrared image in step S2 includes: Step S21, performing three-channel weighted averaging on the acquired infrared image to achieve grayscale processing; Step S22: smoothing and denoising the grayscale image to remove fine textures and irregular noise points. The smoothing and denoising method is Gaussian filtering, and its expression is: Where (x, y) represents the pixel coordinates in the image, G(x, y) represents the Gaussian function value calculated at that location, σ represents the standard of the Gaussian kernel, and e is the constant base of the natural logarithm function; Step S23, using Canny edge detection to extract the edge contour of the smoothed grayscale image and obtain a binarized edge image; In step S24, a contour extraction algorithm is used to first traverse the binary image to find all completely closed edge contours. For each closed contour, its circumscribed rectangle, that is, the minimum rectangle enclosing the contour, is calculated; then, the one with the largest area is selected from all the rectangles to ensure that the entire target area can be covered as much as possible; after selecting the largest rectangle, the corresponding area is cropped out from the original image according to its position and size. This area is considered to be the region of interest (ROI) of the infrared image of the aluminum foil seal.

3. The soft package sealing detection method based on the fusion of vibration signal and infrared image according to claim 1 is characterized in that: The step of pre-processing the soft package sealing infrared image and the vibration signal in step S3 includes: Step S31: perform image enhancement on the extracted ROI image. First, the contrast of the image is enhanced to improve the visibility of image details. Specifically, adaptive histogram equalization is used, and the formula is as follows: Assume that the input image is I and the output image is I eq , the size of the image is M×N, and the size of the local region is p×q; For each pixel (x, y), its corresponding local area is R xy , then the calculation formula of adaptive histogram equalization is: I eq (x,y)=HE(R xy )(I(x,y)) Among them, HE(R xy ) represents the local area R xy Perform histogram equalization operation, I(x,y) is the grayscale value of the pixel in the input image; Local area R xy The histogram equalization can be achieved by the following steps: First, calculate the local area R xy The cumulative distribution function (CDF) of : Among them, n i represents the number of pixels with pixel value i, and pq is the total number of pixels in the local area; Secondly, according to the cumulative distribution function, the pixel value after histogram equalization is calculated: I eq (x,y)=(L-1)×CDF xy (I(x,y)) Where L is the grayscale level of the image; Through such processing, the histogram equalization of the local area of the image is achieved, thereby enhancing the contrast and details of the image; Then, the Laplacian filter is used to sharpen the image and enhance the edges and details of the image; In step S32, an interpolation upsampling operation is performed on the enhanced image samples to improve the resolution of the infrared image and provide more image information for subsequent deep learning model training. Specifically, a biquadratic interpolation method is used. At each pixel position in the image, the value of the new pixel is estimated by taking the weighted average of the four surrounding pixels; the four pixels are the four pixels closest to the position to be interpolated. The interpolation formula is as follows: Where f(x,y) is the pixel value at position x and y, and a ij are the coefficients of biquadratic interpolation; Step S33, randomly rotating the image to expand the training data set and increase the robustness of the model; Step S34: normalize the image pixel values to between 0 and 1, and standardize the normalized image pixel values to have a mean of 0 and a variance of 1 to accelerate model convergence. The formula is: Where x norm is the normalized result, x std is the standardized result, x org is the pixel value of the image, μ and σ are the mean and standard deviation of the pixel values of a single channel; Step S35: Perform time domain analysis on the vibration signal s(t), calculate the mean μ and standard deviation σ of the vibration signal data, and then, according to the 3σ principle, treat data outside the range of μ±3σ as outliers or noise and remove them; Where N represents the total number of data points, s i represents the value of the i-th data point, μ represents the mean of the vibration signal data, and σ is the standard deviation; Step S36, perform frequency domain analysis on the vibration signal, use Fourier transform to convert the signal s(t) into the frequency domain, S(f) and calculate the energy distribution P of the signal at different frequencies in the frequency domain norm ; P(f)=|S(f)| 2 Where S(f) is the Fourier transform of the vibration signal, which represents the complex representation of the signal in the frequency domain; P(f) is the energy spectral density, which represents the signal energy distribution at frequency f; Step S37, normalize the vibration signal s(t) to the range of [0,1], S norm Right now Where s norm represents the normalized vibration signal value, s represents a value of the original vibration signal, s min Represents the minimum value in the original vibration signal data, s max Indicates the maximum value in the original vibration signal data; In step S38, the preprocessed image dataset and vibration signal dataset are divided into training and test sets, with normal sample data accounting for 75% as the training set and the remaining 25% of normal samples and defective samples as the test set.

4. The soft package sealing detection method based on the fusion of vibration signal and infrared image according to claim 1 is characterized in that: The specific steps of step S5 include: In step S51, the processed image and vibration signal are input into the model for parameter training. The cross entropy loss function is used as the classification loss function. For the two categories: qualified and unqualified, the cross entropy loss between the model output and the true label is calculated. The square loss function is introduced to compare the difference between the defect location output by the model and the true defect location as an auxiliary loss function for the defect localization task. Step S52: Fix the trained network weights and, for a given number of clusters K, randomly initialize the feature vectors of K cluster centers. For each input feature vector P, calculate its distance to all cluster centers and assign the feature vector to the cluster center with the closest distance. For each cluster, calculate the mean of all samples in the cluster and use the mean as the new cluster center. Repeat the above two steps until the stopping condition is met. Among them, the distance C between the feature vector P and the cluster center is calculated i The Euclidean distance formula is usually used; for a given cluster center C i And the feature vector P, the Euclidean distance calculation formula is as follows: Where n is the dimension of the feature vector, P j represents the jth eigenvalue of the eigenvector P, C ij Represents the cluster center C i The j-th eigenvalue of .

5. The soft package sealing detection method based on the fusion of vibration signal and infrared image according to claim 1 is characterized in that: The specific steps of step S6 include: Step S61, determining the abnormality score of the input data by calculating the distance between each sample and its cluster center; Step S62: setting an appropriate abnormality score threshold based on the statistical information of the training data to distinguish normal samples from abnormal samples; Step S63: Each sample is judged according to the set abnormality score threshold, and samples with abnormality scores higher than the threshold are marked as abnormal samples, indicating that there are defects; In step S64, for samples determined to be abnormal samples, their features are further analyzed to locate the defective parts, which is achieved by visualizing the feature representation of the abnormal samples and using auxiliary technology, namely, the region localization network.

Images