Real-time defect detection method, device and equipment in plastic uptake carrier tape forming process

Abstract

The invention relates to the technical field of image detection, and discloses a real-time defect detection method, device and equipment for a blister carrier tape forming process, and the method comprises the steps: collecting first image data of the blister carrier tape forming process, and constructing a multi-source sensing data matrix; training the first image data to obtain a defect identification deep learning model; deploying the defect identification deep learning model to an edge computing device for multi-thread concurrent processing, and performing model reasoning on the second image data to obtain real-time defect detection data; establishing a state space model, and calculating the process parameters to obtain process parameter optimization data; according to the technological parameter optimization data, multi-scale space division is conducted on the historical machining track, the optimal machining track parameters are obtained through a dynamic programming algorithm, intelligent optimization of the machining track is achieved, and the forming precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a method, device and equipment for real-time defect detection in a blister carrier tape forming process. Background Art

[0002] As an important packaging material for electronic components, the molding quality of blister tapes directly affects the protection and transportation safety of electronic components. During the blister tape molding process, due to the high-speed operation of the production line and equipment limitations, it is very challenging to obtain accurate information from blister tapes with different molding parameters, especially when capturing tiny defects.

[0003] At present, traditional methods for defect detection of blister carriers mainly rely on manual visual inspection or a single machine vision system. These methods are difficult to meet the needs of real-time detection of high-speed production lines and are easily disturbed by environmental factors, resulting in unstable detection accuracy. At the same time, existing detection systems often only focus on the identification of surface defects, ignoring the coordinated analysis of multi-source information such as pressure, temperature, and displacement during the molding process, making it difficult to prevent defects from the process parameter level. In addition, existing processing trajectory optimization methods usually use fixed process parameter models, lack the ability to dynamically respond to real-time conditions, and cannot adjust and optimize process parameters in real time based on the detected defect information. This static optimization method is difficult to adapt to the dynamic changes in process parameters during the molding process of blister carriers, affecting the stability of product quality. Summary of the invention

[0004] The present invention provides a method, a device and equipment for real-time defect detection in a blister carrier forming process. The present invention realizes intelligent optimization of the processing trajectory and improves the forming accuracy.

[0005] In a first aspect, the present invention provides a method for real-time defect detection in a blister carrier forming process, the method comprising: Collect the first image data of the blister carrier tape forming process, and pre-process the pressure data, temperature data and displacement data of the blister carrier tape forming production line to construct a multi-source sensor data matrix; Performing a three-layer one-dimensional convolution operation of a time domain branch and a two-layer two-dimensional convolution operation of a frequency domain branch on the first image data, and obtaining a defect recognition deep learning model through fusion and training of a fully connected layer; The defect recognition deep learning model is deployed to an edge computing device for multi-threaded concurrent processing, and feature extraction and model inference are performed on the second image data collected in real time to obtain real-time defect detection data; Establishing a state space model based on the real-time defect detection data and the multi-source sensor data matrix, optimizing and calculating the process parameters, and obtaining process parameter optimization data; The historical processing trajectory is divided into multi-scale spaces according to the process parameter optimization data to obtain the optimal processing trajectory parameters.

[0006] In a second aspect, the present invention provides a real-time defect detection device for a blister carrier tape forming process, the real-time defect detection device for a blister carrier tape forming process comprising: An acquisition module is used to acquire the first image data of the blister carrier tape forming process, and pre-process the pressure data, temperature data and displacement data of the blister carrier tape forming production line to construct a multi-source sensor data matrix; A training module, used to perform a three-layer one-dimensional convolution operation of a time domain branch and a two-layer two-dimensional convolution operation of a frequency domain branch on the first image data, and obtain a defect recognition deep learning model through fusion and training of a fully connected layer; A deployment module, used to deploy the defect recognition deep learning model to an edge computing device for multi-threaded concurrent processing, and perform feature extraction and model inference on the second image data collected in real time to obtain real-time defect detection data; An establishment module is used to establish a state space model based on the real-time defect detection data and the multi-source sensor data matrix, optimize and calculate the process parameters, and obtain process parameter optimization data; The solution module is used to perform multi-scale spatial division of the historical processing trajectory according to the process parameter optimization data to solve and obtain the optimal processing trajectory parameters.

[0007] The third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned real-time defect detection method for the blister carrier forming process.

[0008] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned method for real-time defect detection in a blister carrier tape molding process.

[0009] In the technical solution provided by the present invention, by constructing a multi-source sensor data matrix, multi-dimensional information such as pressure, temperature, and displacement are integrated, and combined with the time domain and frequency domain feature extraction of image data, a comprehensive defect detection system is formed, which improves the accuracy and reliability of defect detection; a deep learning model architecture of three-layer one-dimensional convolution with time domain branch and two-layer two-dimensional convolution with frequency domain branch is adopted to effectively extract the temporal characteristics and frequency domain characteristics of the image, and enhance the model's recognition ability for different types of defects; through the multi-threaded concurrent processing mechanism of edge computing devices, the detection response speed is improved to meet the real-time monitoring needs of high-speed production lines; the process parameter optimization method based on the interactive multi-model Kalman filtering algorithm realizes the dynamic estimation and optimization of parameters such as temperature, pressure, and displacement, and enhances the system's adaptability to changes in process parameters; the processing trajectory optimization scheme using multi-scale space division and Markov mobile model realizes intelligent optimization of the processing trajectory through precise calculation of the transfer probability between grid units, thereby improving the forming accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0011] Figure 1 Schematic diagram of the steps of a method for real-time defect detection in a blister carrier forming process according to an embodiment of the present invention; Figure 2 It is a structural schematic diagram of a real-time defect detection device for a blister carrier tape forming process according to an embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] Embodiments of the present invention provide a method, device and equipment for real-time defect detection in a blister carrier forming process. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0013] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the method for real-time defect detection in the blister carrier forming process in the embodiment of the present invention includes: Step S1, collecting first image data of the blister carrier tape forming process, and pre-processing the pressure data, temperature data and displacement data of the blister carrier tape forming production line to construct a multi-source sensor data matrix; It is understandable that the execution subject of the present invention may be a real-time defect detection device for the blister carrier forming process, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0014] Specifically, the surface of the blister carrier is continuously photographed by an industrial camera to obtain a sequence of original images covering the entire molding process. These original images have different resolutions and brightness distributions, and grayscale processing is performed on them to simplify the image data structure and reduce the computational complexity, while eliminating the interference caused by color information; then the grayscale images are size-uniformed so that all images have the same resolution and pixel scale, which is convenient for subsequent algorithm modeling and feature extraction, and the first image data is generated. At the same time, the original signal collected by the pressure sensor in the blister carrier molding production line is sampled, and the continuous signal is discretized and converted into a form suitable for digital processing. The discrete signal is spectrally analyzed by Fourier transform, the original pressure data is converted from the time domain to the frequency domain, and its spectral features are extracted to reflect the energy distribution of the pressure signal at different frequencies. On the basis of frequency domain analysis, the spectral features are bandpass filtered to eliminate high-frequency noise and low-frequency interference, and only the frequency components that are of practical significance to the molding process are retained to generate pressure frequency domain data. The signals collected by the temperature sensor and the displacement sensor in the blister carrier molding production line are sampled, and the signals are smoothed by Gaussian filtering. Gaussian filtering is a denoising method that effectively removes random noise from the signal while retaining the main trends and important features of the signal. After filtering, the basic temperature displacement data is obtained, which reflects the changes in the key process parameters of the production line. The pressure frequency domain data is restored to time domain data through inverse Fourier transform so that it can be processed in the same time dimension as the basic temperature displacement data. Since there is a certain deviation in the timestamps of data collected by different sensors, all data are timestamp aligned to ensure the consistency and comparability of the data. The aligned data is segmented according to the different stages of the molding process to extract the characteristics of the corresponding process state. The pre-processed data sequence is matrix spliced ​​in the order of pressure data, temperature data and displacement data, and the sensor data from different sources are integrated into a matrix form. In order to eliminate the differences in dimensions and scales between different data sources, the spliced ​​data matrix is ​​normalized so that all eigenvalues ​​fluctuate within the same numerical range to obtain a multi-source sensor data matrix.

[0015] Step S2, performing a three-layer one-dimensional convolution operation of a time domain branch and a two-layer two-dimensional convolution operation of a frequency domain branch on the first image data, and obtaining a defect recognition deep learning model through fusion and training of a fully connected layer; Specifically, the first image data is processed by time series segmentation, and the time window length is set to 32 frames and the sliding step size is set to 16 frames, so as to effectively capture the dynamic change characteristics in the molding process. The continuously collected image data is divided into multiple image segments with time series information, that is, time series image sequences. The time series image sequence is input into the first one-dimensional convolution layer of the time domain branch for processing. In this layer, the size of the convolution kernel is set to 5 and the number of convolution kernels is set to 32, which can effectively extract the local features between adjacent pixels in each image segment. After the convolution operation, the generated feature map is normalized by the BatchNormalization layer to accelerate the convergence of the model and alleviate the problem of gradient disappearance. After the nonlinear mapping of the ReLU activation function, the first feature map of the time domain is obtained. This feature map contains the preliminary extracted time series related information. The first feature map of the time domain is input into the second one-dimensional convolution layer of the time domain branch. In this layer, the size of the convolution kernel is set to 3 and the number of convolution kernels is increased to 64, so that the convolution layer can capture more detailed feature patterns. After the convolution operation is completed, the BatchNormalization layer and the ReLU activation function are used for normalization and nonlinear processing to enhance the feature expression ability. The feature map is processed by the maximum pooling layer. The pooling operation can reduce the size of the feature map, thereby reducing the computational complexity, while retaining the key feature information and generating the second feature map in the time domain. The second feature map in the time domain is input into the third one-dimensional convolution layer of the time domain branch. In this layer, the size of the convolution kernel is 3, and the number of convolution kernels is increased to 128, aiming to extract higher-level features. After processing by convolution, BatchNormalization, and ReLU activation function, the final feature map in the time domain is generated. This feature map integrates time series features from low to high levels and can effectively describe the dynamic changes in the time series image sequence. At the same time, the fast Fourier transform is applied to the first image data to convert the time domain image data into frequency domain image data, revealing the frequency components in the image data. The frequency domain image data is input into the first two-dimensional convolution layer of the frequency domain branch for processing. In this layer, the convolution kernel size is set to 3×3, and the number of convolution kernels is 64. The convolution operation extracts the local frequency features in the frequency domain data. After the nonlinear mapping of the LeakyReLU activation function, the first feature map in the frequency domain is generated. The first feature map in the frequency domain is input into the second two-dimensional convolution layer of the frequency domain branch. The convolution kernel size is 3×3, and the number of convolution kernels is increased to 128 to extract more fine-grained frequency features. In this process, the LeakyReLU activation function is used for nonlinear mapping to enhance the expressive power of the model. At the same time, the Dropout layer is introduced to prevent overfitting and improve the generalization performance of the model. After processing, the final feature map in the frequency domain is obtained. The final feature map in the time domain and the final feature map in the frequency domain are spliced ​​in the channel dimension. The spliced ​​feature map contains information in two dimensions, time and frequency. The spliced ​​feature map is input into the fully connected layer for processing.The number of hidden layer nodes in the fully connected layer is set to 256, which can further nonlinearly combine the spliced ​​features to generate a fused feature vector. In this process, the Dropout layer is used to reduce the risk of overfitting, and the ReLU activation function is used to improve the feature expression ability to ensure that the final fused feature vector can fully represent the defect characteristics in the input image data. The fused feature vector is input into the classification layer. The number of output nodes of the classification layer is consistent with the number of defect categories. The probability distribution of each type of defect is calculated by the Softmax activation function to achieve defect classification. The cross entropy loss function is used for optimization during the training process, and the model parameters are continuously adjusted through the back propagation algorithm, and finally a deep learning model that can effectively identify defects is obtained.

[0016] Step S3: deploy the defect recognition deep learning model to the edge computing device for multi-threaded concurrent processing, and perform feature extraction and model inference on the second image data collected in real time to obtain real-time defect detection data; Specifically, the model structure of the defect recognition deep learning model is optimized, and the network structure is simplified by removing redundant layers and parameters to adapt to the computing power and storage limitations of the edge computing device. On this basis, in order to improve the computing efficiency and reduce the inference delay of the model, the parameter quantization technology is used to convert the floating point precision of the model from FP32 to FP16. Through this quantization process, the storage requirements and computational complexity of the model are reduced, while the accuracy of the model is maintained, and a lightweight deployment model suitable for edge device operation is generated. In order to efficiently run the model on the edge computing device, a multi-threaded processing system is designed. By constructing data acquisition threads, feature extraction threads and model inference threads, and using the thread pool management mechanism to schedule these threads, efficient multi-threaded collaboration is achieved. The thread pool management mechanism can dynamically allocate and recycle thread resources, avoid the additional overhead of thread creation and destruction, and ensure the task coordination between different threads, thereby improving the overall concurrent processing capability of the system. When running the multi-threaded processing system on the edge computing device, the image data of the surface of the blister carrier is collected in real time through the industrial camera. These image data will be stored in the data cache queue, which can support the dynamic update of image data to meet the needs of real-time processing. The image data collected in real time is the second image data. A time series window is constructed based on the second image data, where the length and sliding step of the time series window are determined by the previous model design. In order to improve processing efficiency, the number of parallel processing units is set to 4. By preprocessing the image data in batches in parallel, multiple image segments can be processed simultaneously to generate data to be inferred. The data to be inferred is input into the optimized edge deployment model, and feature extraction and inference are performed through the time domain branch and frequency domain branch of the model respectively. In the time domain branch, the data to be inferred is calculated by three layers of one-dimensional convolutional networks to extract time domain features reflecting the changes in the time series. After each layer of convolution operation, the feature map is processed by activation function and normalization to gradually extract more high-order and complex time domain features and obtain time domain inference features. At the same time, the data to be inferred is fast Fourier transformed and input into the frequency domain branch of the edge deployment model. Through the calculation of two layers of two-dimensional convolutional networks, frequency domain inference features describing frequency distribution and periodicity characteristics are extracted. The extraction of frequency domain features is also processed by activation function and normalization to ensure the nonlinear enhancement and stability of feature expression. The time domain reasoning features and frequency domain reasoning features are integrated in the feature fusion module. Feature fusion is achieved through a fully connected layer, which can map the features from the two branches into a unified feature space, comprehensively reflecting the defect information in the time domain and frequency domain. The Softmax function is applied to the fused feature vector to calculate the probability distribution of each defect category, and finally output the defect detection results. The defect detection results are associated with the location information and timestamp information of the image. The associated data is integrated through the result summary thread to form complete real-time defect detection data, including the category, location and specific timestamp of each defect.

[0017] Step S4: establishing a state space model based on the real-time defect detection data and the multi-source sensor data matrix, optimizing and calculating the process parameters, and obtaining process parameter optimization data; Specifically, a state space model is constructed based on real-time defect detection data and multi-source sensor data matrix, and the state vector is designed to include temperature parameters, pressure parameters, displacement parameters and defect characteristic parameters, which can fully describe the dynamic state of the equipment during the blister carrier molding process. At the same time, the measurement vector is set to include the real-time data collected by the sensor and the detection results output by the defect detection model to ensure the observation ability of the state space model. Based on the state vector and the measurement vector, the dynamic behavior of the system in the molding process is fully described by constructing the state equation and the measurement equation to form the system state equation. On the basis of the system state equation, three sub-models are constructed to describe different state evolution characteristics. Among them, the first sub-model assumes that the system state mainly changes at a uniform speed, and its state transfer matrix is ​​designed as a diagonal matrix to simplify the calculation, which is suitable for the relatively stable stage in the production process; the second sub-model introduces the velocity term, assuming that the system state has the characteristics of uniform acceleration change, and its state transfer matrix contains the velocity component, which can capture the situation where the process parameters change rapidly; the third sub-model introduces nonlinear terms to describe the complex changes of the state, especially the nonlinear characteristics in the molding process, thereby improving the expression ability of the model. The three sub-models together constitute a hybrid model system, which can flexibly cope with the diversity and complexity of the blister carrier forming process. The state of the hybrid model system is predicted according to the calculation formula of Kalman filtering. In each time step, the state vector of the previous moment is multiplied by the corresponding state transfer matrix, and process noise is added to simulate the uncertainty of the dynamic change of the system to obtain the predicted state vector. The predicted state vector reflects the expected state of the system under the current conditions. The new information sequence is calculated by combining the measurement equation and the actual measurement data. The new information sequence is used to reflect the difference between the predicted state and the actual observation. The Kalman gain is calculated by the measurement noise covariance matrix, and the predicted state is corrected by the Kalman gain to obtain the state update value of each sub-model. The model probability is calculated for the state update value of each sub-model. The likelihood function is constructed based on the new information sequence and the prediction covariance matrix. The likelihood function is used to evaluate the explanatory ability of each sub-model for the current state to obtain the model weight coefficient. The state update value of each sub-model is weighted fused by the weight coefficient, and the interactive resampling method is used to process the fusion result to avoid the model weight distribution being too biased towards a single sub-model, and the optimized state vector is obtained. On the basis of optimizing the state vector, the process parameters are optimized in combination with the process constraints. A Lagrangian function is constructed, in which the process objectives are decomposed into three aspects: defect rate, molding accuracy, and workstation coordination, and weight coefficients are set for each part, namely, the defect rate weight coefficient is 0.6, the molding accuracy weight coefficient is 0.3, and the workstation coordination weight coefficient is 0.1. Through weight setting, the Lagrangian function can comprehensively reflect the priority of different process objectives. The weight coefficient is substituted into the augmented Lagrangian equation, and the process constraints are handled by introducing penalty terms, and the stability and convergence of the optimization problem are improved.In order to solve the optimization problem of the augmented Lagrangian equation, the KKT condition is constructed based on its first-order derivative and second-order derivative, and the optimization problem is transformed into the problem of solving the optimal control quantity. The KKT condition is numerically solved by the Newton iteration method, and the optimal solution is gradually approached through multiple iterations to obtain the final process parameter optimization data. The optimized process parameters can dynamically adjust the operating state of the blister carrier molding equipment, reduce the defect rate, improve the molding accuracy and realize the coordinated optimization of the workstations.

[0018] Step S5: perform multi-scale spatial division on the historical processing trajectory according to the process parameter optimization data, and solve to obtain the optimal processing trajectory parameters.

[0019] Specifically, coordinate transformation is performed on the historical processing trajectory in the process parameter optimization data to unify its reference system to adapt to the subsequent grid segmentation. According to the preset grid scale, the processing area is multi-scale spatially segmented according to three different density thresholds of 16×16 grid, 8×8 grid and 4×4 grid to generate multi-scale grid data. The trajectory point density is calculated for each grid cell in the multi-scale grid data. According to the statistical results of the trajectory point density, grids with a density greater than 80% are divided into high-density areas, which represent the main working paths of the processing trajectory; grids with a density between 50% and 80% are divided into medium-density areas, reflecting auxiliary processing paths; and grids with a density less than 50% are divided into low-density areas, which are areas with fewer processing paths. Through this step, complete grid density distribution data is formed to describe the spatial distribution characteristics of the processing trajectory under multi-scale grids. A Markov mobile state space is established for adjacent grid cells in the grid density distribution data. This model can describe the dynamic evolution characteristics of the trajectory between adjacent grid cells. By counting the number of transfers and transfer times of trajectories between adjacent grid cells, the transfer data of the trajectory is constructed. These data reflect the frequency and time characteristics of the trajectory from one grid to another. In order to eliminate the influence of different grid distributions and trajectory lengths on the transfer data, it is normalized by dividing each transfer number by the total transfer number to obtain the normalized grid unit transfer probability matrix. Based on the transfer probability matrix, the residence time distribution and movement direction distribution of the trajectory in each grid cell are analyzed. These characteristics are used as state transfer characteristics to fully describe the dynamic behavior of the processing trajectory. The residence time distribution can reflect the degree of processing concentration of processing equipment in a certain area, while the movement direction distribution reveals the main migration direction and preference of the trajectory in space. According to the state characteristic data, a dynamic programming objective function is constructed to optimize the trajectory path. In the dynamic programming objective function, factors such as trajectory smoothness, processing efficiency and path length are comprehensively considered, and these objectives are balanced by designing reasonable weights. In order to solve the dynamic programming objective function, the state value of each stage is calculated step by step by forward recursion. In the recursive process, the optimal state transfer path of each stage is recorded to form a complete state path sequence. This sequence represents an optimal trajectory from the starting point to the end point, taking into account the processing technology requirements and path characteristics. The state path sequence is traced back, and the center position coordinates of each grid unit are extracted by backtracking to ensure the uniform distribution and rationality of the path points, and the center position coordinates of each grid unit are extracted. The extracted center position coordinates are subjected to trajectory fitting. In order to ensure the smoothness and accuracy of the trajectory, a cubic spline curve is used for fitting. The cubic spline curve can strictly pass through all the center position points while ensuring a smooth transition of the trajectory, avoiding the deviation caused by the traditional fitting method. After fitting, the optimal processing trajectory parameters are obtained.

[0020] Each grid cell in the grid density distribution data is assigned a continuous number from 1 to N to provide a unique identifier for subsequent processing, while ensuring that the spatial position relationship between grid cells can be directly indexed by the number. By analyzing the spatial layout of the grid cells, the eight-neighborhood connection relationship of each grid cell is calculated, that is, in the two-dimensional plane, the adjacent grid cells in the upper, lower, left, right, and diagonal directions of each grid cell are taken as its neighbors, and a grid connection matrix is ​​generated to describe the adjacency relationship between each grid cell. Bidirectional connection edges are established for adjacent grid cell pairs in the grid connection matrix, which means that the trajectory can be transferred in any direction. In order to reflect the importance of different regions to trajectory transfer, a weight coefficient is assigned to each connection edge, where the connection edge weight of the high-density area is set to 1.0, the connection edge weight of the medium-density area is set to 0.8, and the connection edge weight of the low-density area is set to 0.6. The weight value reflects the transfer priority of the trajectory in different density areas, which can effectively guide the strategy optimization of the model in regional division. After weighted processing, a weighted connection graph is generated. This graph is the core data structure of trajectory transfer modeling and can describe the mutual influence and transfer trend between grid cells. Based on the weighted connection graph, all transfer events from one grid unit to another are recorded, including the starting grid and target grid numbers of each transfer and the time when the transfer occurs. At the same time, the transfer time series is generated by calculating the transfer time intervals between adjacent moments. On the basis of marking the starting grid and target grid numbers of each transfer event in the transfer time series, the number of transfers between each grid unit pair is counted to construct a transfer count matrix. The transfer count matrix is ​​segmented according to the time window, and the transfer frequency distribution in each time window is calculated to generate a time-varying transfer frequency matrix. The time-varying transfer frequency matrix can capture the changing trend of the trajectory in different time periods. A sliding average filter is applied to the time-varying transfer frequency matrix. Through the sliding average filter, the interference caused by short-term fluctuations is eliminated, and a smoother transfer frequency matrix is ​​obtained. Based on the smoothed transfer frequency matrix, a Markov chain state transfer model is constructed to calculate the transfer probability and average transfer time between any two adjacent grid units to generate a transfer feature matrix. The transfer feature matrix is ​​combined with the spatial position information of the grid unit to integrate the key information such as position, transfer probability and transfer time contained in the trajectory transfer process to generate complete trajectory transfer data.

[0021] In the embodiment of the present invention, by constructing a multi-source sensor data matrix, multi-dimensional information such as pressure, temperature, and displacement is integrated, and the time domain and frequency domain feature extraction of image data is combined to form a comprehensive defect detection system, thereby improving the accuracy and reliability of defect detection; a deep learning model architecture of three-layer one-dimensional convolution with time domain branch and two-layer two-dimensional convolution with frequency domain branch is adopted to effectively extract the temporal characteristics and frequency domain characteristics of the image, thereby enhancing the model's recognition ability for different types of defects; through the multi-threaded concurrent processing mechanism of the edge computing device, the detection response speed is improved, and the real-time monitoring requirements of the high-speed production line are met; the process parameter optimization method based on the interactive multi-model Kalman filtering algorithm realizes the dynamic estimation and optimization of parameters such as temperature, pressure, and displacement, thereby enhancing the system's adaptability to changes in process parameters; a processing trajectory optimization scheme using multi-scale space division and Markov mobile model is adopted, and through the precise calculation of the transfer probability between grid units, the intelligent optimization of the processing trajectory is realized, thereby improving the forming accuracy.

[0022] In a specific embodiment, the process of executing step S1 may specifically include the following steps: The surface of the blister carrier is continuously photographed by an industrial camera to obtain an original image sequence, and the original image sequence is grayed and resized to obtain first image data; The original signal collected by the pressure sensor in the blister carrier forming production line is sampled, and the spectrum is analyzed by Fourier transform to obtain the basic pressure data, and the spectrum characteristics of the basic pressure data are band-pass filtered to obtain the pressure frequency domain data; The signals collected by the temperature sensor and displacement sensor in the blister carrier molding production line are sampled, and the signals are smoothed by Gaussian filtering to obtain the basic temperature and displacement data; The pressure frequency domain data is restored to time domain data through inverse Fourier transform, and the time stamp is aligned and segmented with the temperature displacement basic data to obtain the preprocessed data sequence; According to the order of pressure data, temperature data and displacement data, the preprocessed data sequence is matrix concatenated and normalized to obtain a multi-source sensing data matrix.

[0023] Specifically, the surface of the blister carrier is continuously photographed by an industrial camera to capture a complete image sequence of surface details and dynamic changes. Assuming that the industrial camera has a frame rate of The original image sequence is obtained as ,in Indicates the time frame index. Since the original image contains multi-channel color information (such as RGB three channels), in order to simplify calculation and processing, Perform grayscale processing. The grayscale formula is as follows: ; in are the red, green, and blue channel pixel values ​​of the original image, respectively. is the pixel coordinate. The grayscale image It only contains single channel information, which significantly reduces the data dimension. To unify the image size, the grayscale image Perform size standardization and set the target size to , the image resolution is adjusted by interpolation to obtain the first image data of the same size At the same time, in order to obtain the production environment information during the blister carrier forming process, the original signal collected by the pressure sensor Processing. At sampling frequency The continuous pressure signal is discretized and represented as a discrete time series: ; in is the sampling period, represents the discrete time index. Then the discrete pressure signal Apply fast Fourier transform to convert the signal from time domain to frequency domain to extract spectral features. The Fourier transform formula is as follows: ; in Indicates frequency components The amplitude of is the sequence length, is an imaginary unit. After obtaining the spectrum, in order to suppress high-frequency noise and low-frequency interference, it is subjected to bandpass filtering. Assume that the passband of the filter is , the spectrum filtering process is as follows: ; in is the actual frequency value corresponding to the frequency index. The frequency domain data after filtering The effective frequency components related to the molding process are extracted. and , also with sampling frequency Discretize and get the discrete sequences and In order to remove noise and smooth the signal, Gaussian filtering is used. The convolution formula of Gaussian filtering is: ; ; in is the Gaussian kernel, is half the length of the filter window, is the standard deviation, indicating the filtering strength. After processing, the smoothed temperature basic data is obtained. and displacement basic data The pressure frequency domain data is transformed into Restore to time domain signal The inverse transformation formula is: ; The restored pressure time domain data is time-stamped and aligned with the temperature and displacement basic data to ensure that each sequence is synchronized under the same time reference. The aligned data is segmented according to the process cycle, and each segment of data contains complete molding process information to generate a pre-processed data sequence. ,in The preprocessed data sequence is matrix spliced ​​in the order of pressure data, temperature data and displacement data to construct a multi-source sensor data matrix .matrix Each column of corresponds to the data of one time step, and each row corresponds to a sensor data source. In order to unify the dimensions and eliminate the influence of numerical differences, Perform normalization processing, the normalization formula is: ; in Representation Matrix Middle Row, No. The elements of the column, and are the minimum and maximum values ​​of the matrix rows respectively. The multi-source sensor data matrix finally generated is Contains normalized pressure, temperature, and displacement information.

[0024] In a specific embodiment, the process of executing step S2 may specifically include the following steps: Perform time series segmentation on the first image data, set the time window length to 32 frames, and the sliding step length to 16 frames to obtain a time series image sequence; The time-series image sequence is input into the first one-dimensional convolution layer of the time domain branch, the convolution kernel size is set to 5, the number of convolution kernels is set to 32, and processed by the BatchNormalization layer and the ReLU activation function to obtain the first feature map in the time domain; The first time-domain feature map is input into the second one-dimensional convolution layer of the time-domain branch, the convolution kernel size is set to 3, the number of convolution kernels is set to 64, and it is processed by the BatchNormalization layer and the ReLU activation function, and then processed by the maximum pooling layer to obtain the second time-domain feature map; The second feature map in the time domain is input into the third one-dimensional convolution layer of the time domain branch, and the convolution kernel size is set to 3 and the number of convolution kernels is set to 128. After being processed by the BatchNormalization layer and the ReLU activation function, the final feature map in the time domain is obtained. Perform fast Fourier transform on the first image data to obtain frequency domain image data, and input the frequency domain image data into the first two-dimensional convolution layer of the frequency domain branch, set the convolution kernel size to 3×3, the number of convolution kernels to 64, and process it through the LeakyReLU activation function to obtain the first feature map in the frequency domain; The first feature map in the frequency domain is input into the second two-dimensional convolution layer of the frequency domain branch. The convolution kernel size is set to 3×3 and the number of convolution kernels is set to 128. After being processed by the LeakyReLU activation function and the Dropout layer, the final feature map in the frequency domain is obtained. The final feature map in the time domain and the final feature map in the frequency domain are concatenated in the channel dimension and input into the fully connected layer. The number of hidden layer nodes is set to 256. After being processed by the Dropout layer and the ReLU activation function, a fused feature vector is obtained. The fused feature vector is input into the classification layer, the number of output nodes is set to the number of defect categories, processed by the Softmax activation function, and trained and optimized using the cross entropy loss function to obtain a deep learning model for defect recognition.

[0025] Specifically, for the first image data Perform time segmentation processing to decompose the continuously acquired image sequence into time windows of length 32 frames, and generate overlapping window sequences with a sliding step of 16 frames. Assume that the number of frames in the input image sequence is , then it can be split into windows, each window represents an independent temporal image segment. Each segment is represented by Indicates that is the timeframe index, is the spatial coordinate of the image. Input to the first one-dimensional convolution layer of the time domain branch. Assume the convolution kernel size is , the number of convolution kernels is , the convolution operation extracts the features of the time dimension for each time segment. The convolution calculation formula is as follows: ; in Represents the first layer of convolution output feature map at time and Channel The value on It is The convolution kernel is The weight of the position, The convolution result passes through the batch normalization layer to standardize the feature values ​​to speed up training, and passes through the ReLU activation function Introduce nonlinear mapping to generate the first feature map in the time domain. Input to the second one-dimensional convolution layer of the time domain branch, the convolution kernel size is set to , the number of convolution kernels increases to The convolution formula is similar to the first layer, but the feature dimension is increased, which can capture more complex temporal features. After batch normalization and ReLU activation, the convolution result is further processed by the maximum pooling layer. Assume the pooling window size is , then the pooling operation formula is: ; in is the feature map after pooling, which can reduce the size of the feature map while retaining key information. The result is output as the second feature map in the time domain. The second feature map in the time domain is input to the third one-dimensional convolution layer of the time domain branch. The convolution kernel size of this layer is , the number of convolution kernels is After similar convolution, normalization and activation processing as the first two layers, the final feature map in the time domain is output , whose structure contains rich time series information. At the same time, the first image data Perform fast Fourier transform and convert to frequency domain image data The fast Fourier transform formula is as follows: ; in is the frequency index, is the image width and height, is an imaginary unit. The frequency domain data is input into the first two-dimensional convolution layer of the frequency domain branch, and the convolution kernel size is , the number of convolution kernels is The formula for two-dimensional convolution is: ; The result is processed by the LeakyRelU activation function to obtain the first feature map in the frequency domain. It is input into the second two-dimensional convolution layer, and the convolution kernel size is still , the number of convolution kernels is After similar convolution and activation processing, the Dropout layer is combined to reduce overfitting, and the final feature map in the frequency domain is output. . The final feature map in the time domain And the final feature map in the frequency domain Splice in the channel dimension to get the fused feature map .Will Input the fully connected layer, the number of hidden layer nodes is set to 256, and the fused feature vector is generated after the ReLU activation function is processed The formula for fusing feature vectors is: ; in is the weight of the fully connected layer, is the bias. The fused feature vector Input classification layer, the number of output nodes is consistent with the number of defect categories, the probability distribution of each category is calculated by Softmax function, and the model parameters are optimized by cross entropy loss function. The cross entropy loss formula is: ; in is the true category label, is the predicted probability, is the total number of categories, is the number of samples. The model weights are adjusted through back propagation and optimizers (such as Adam), and finally a complete defect recognition deep learning model is obtained.

[0026] In a specific embodiment, the process of executing step S3 may specifically include the following steps: The defect recognition deep learning model is optimized in terms of model structure and parameter quantization, and the floating point precision is converted from FP32 to FP16 to obtain an edge deployment model. Construct data collection threads, feature extraction threads, and model inference threads for edge deployment models, and use thread pool management mechanism for scheduling to obtain a multi-threaded processing system; The multi-threaded processing system is run on the edge computing device, the surface image of the blister carrier is collected in real time by an industrial camera, and the image data is updated based on the data cache queue to obtain the second image data; Building a time series window based on the second image data, setting the number of parallel processing units to 4, and performing parallel preprocessing of the image data in batches to obtain data to be inferred; The data to be inferred is input into the time domain branch of the edge deployment model, and features are calculated through a three-layer one-dimensional convolutional network to obtain time domain inference features. The data to be inferred is subjected to fast Fourier transform and then input into the frequency domain branch of the edge deployment model. The features are calculated through a two-layer two-dimensional convolutional network to obtain frequency domain inference features. The time domain reasoning features and frequency domain reasoning features are fused, and the defect category probability is calculated using the fully connected layer and Softmax function to obtain the defect detection results. The defect detection results are then associated with the image location information and timestamp information, and data integration is performed through the result summary thread to obtain real-time defect detection data.

[0027] Specifically, the structure of the defect recognition deep learning model is optimized, and redundant network layers and weights are removed through pruning and sparsification techniques to reduce computational complexity and storage requirements. For example, for a convolutional layer weight matrix (in is the number of output channels, is the number of input channels, and are the height and width of the convolution kernel respectively), by setting the threshold The weight value is less than The elements of are set to zero, thus sparseening the weight matrix: ; The optimized model uses parameter quantization technology to reduce the floating point precision of weights and activation functions from FP32 to FP16. Quantization converts the original 32-bit floating point numbers into The representation range is reduced to 16-bit floating point numbers The specific conversion formula is: ; in Indicates the number of digits of precision for quantization. represents the range. Through quantization, the memory usage and computing delay of the model are greatly reduced while maintaining the prediction accuracy of the model, and a quantized deployment model suitable for edge devices is obtained. In order to realize the real-time multi-threaded processing system, data acquisition threads, feature extraction threads and model inference threads are constructed. These threads are scheduled through the thread pool management mechanism, and the size of the thread pool is set to , which represents the maximum number of parallel threads. The thread pool management calculates the allocation priority of each thread through the following formula : ; in It is a thread The thread resource allocation strategy is dynamically adjusted based on the task importance and computing load. After the multi-threaded processing system is deployed to the edge computing device, the surface image of the blister carrier is collected in real time through the industrial camera. The collected original image is stored in the form of a FIFO data cache queue. The image resolution is set to , then each frame image The order of storage in the queue is To ensure real-time performance, the cache queue capacity is set to , when the queue length exceeds When , the oldest image is automatically cleared. A time series window is constructed based on the cached image data. Each window contains 32 frames of images, and the sliding step is set to 16 frames. Assume that the total number of windows is , the window construction formula is: ; To improve the parallel processing capability, the number of parallel processing units is set to 4, and the image window is divided into 4 batches. Each batch is preprocessed in parallel to obtain the data to be inferred. In the reasoning process of the time domain branch, the data to be inferred is input into the three-layer one-dimensional convolutional network for feature calculation. Assume that the input feature is

[0029] (in is the length of the time series, is the number of channels), The layer convolution operation is: ; in For the The convolution kernel of the layer, is the bias, * is a one-dimensional convolution operation, and the output is the time domain inference feature. At the same time, the data to be inferred is converted into frequency domain representation through fast Fourier transform ,in and is the number of rows and columns of the frequency component. The frequency domain data is input into the two-dimensional convolutional network of the frequency domain branch for feature extraction. Assume that the convolution kernel size is , the output features are: ; in is a two-dimensional convolution kernel, The result is the frequency domain reasoning feature. After the time domain reasoning feature and the frequency domain reasoning feature are concatenated in the channel dimension, the defect category probability is calculated through the fully connected layer and the Softmax function. The Softmax function formula is: ; in For the The score of the class, is the number of categories. By selecting the category with the maximum probability as the prediction result, the defect detection result is generated. The defect detection result is associated with the image location information and timestamp and stored as ,in is the defect category, is the defect location, The result summary thread integrates the data and outputs real-time defect detection data.

[0030] In a specific embodiment, the process of executing step S4 may specifically include the following steps: Based on real-time defect detection data and multi-source sensor data matrix, state space modeling is performed, the state vector is set to include temperature parameters, pressure parameters, displacement parameters and defect characteristic parameters, and the measurement vector is set to include sensor data and defect detection results to obtain the system state equation; Construct three sub-models for the system state equation: set the first sub-model as a uniform velocity model and the state transfer matrix as a diagonal matrix, set the second sub-model as a uniform acceleration model and the state transfer matrix includes velocity terms, set the third sub-model as a conversion model and the state transfer matrix includes nonlinear terms, and obtain a hybrid model system; According to the Kalman filter formula, the state of the first sub-model, the second sub-model and the third sub-model in the hybrid model system is predicted, the state vector of the previous moment is multiplied by the corresponding state transfer matrix and the process noise is added to obtain the predicted state vector; The new information sequence is calculated based on the predicted state vector and the actual measurement data, and the Kalman gain is calculated according to the measurement equation and the measurement noise covariance matrix to obtain the state update value of each sub-model; Perform model probability calculation on the state update value of each sub-model, construct the likelihood function based on the new information sequence and the predicted covariance matrix, obtain the model weight coefficient, and perform weighted fusion on the state update value of each sub-model according to the model weight coefficient, and perform interactive resampling calculation to obtain the optimized state vector; The Lagrangian function is constructed for the optimization state vector and process constraints, and the defect rate weight coefficient is set to 0.6, the molding accuracy weight coefficient is set to 0.3, and the station coordination weight coefficient is set to 0.1, and the augmented Lagrangian equation is obtained; The KKT conditions are constructed based on the first-order derivative and the second-order derivative of the augmented Lagrangian equation, and the Newton iteration method is used to solve the optimal control quantity to obtain the process parameter optimization data.

[0031] Specifically, a state space model is constructed based on real-time defect detection data and multi-source sensor data matrix to fully describe the dynamic evolution characteristics of the system. Assume that the state vector of the system is , including temperature parameters , pressure parameters , displacement parameters and defect characteristic parameters ,Right now: ; At the same time, the measurement vector It includes real-time data from sensors and defect category information output by defect detection models, expressed as: ; The subscript Represents the measured value. The system state equation and measurement equation are expressed as: ; ; in, is the state transition matrix, describing the state from arrive evolution; is the measurement matrix, describing the relationship between the state and the observed value; and They are process noise and measurement noise, which satisfy Gaussian distribution and ,in and is the noise covariance matrix. Based on the system state equation, three sub-models are constructed to capture the diverse dynamic characteristics of the system. The first sub-model is the uniform speed model, and its state transfer matrix For a diagonal matrix: ; in is the unit matrix, indicating that the system state evolves at a constant speed. The second submodel is the uniform acceleration model, and the state transfer matrix Contains velocity terms to reflect accelerated changes in state: ; in is the time step. The third sub-model is a nonlinear conversion model, and its state transfer matrix Contains nonlinear terms such as sin or , to describe complex nonlinear behavior. Through the hybrid model system, the Kalman filter formula is used to predict the state of each sub-model. Assume that the state vector at the previous moment is , the state prediction formula is: ; in It is a sub-model Combined with actual measurement data , calculate the new information sequence : ; Based on the innovation sequence and measurement noise covariance matrix , calculate the Kalman gain: ; in is the predicted state covariance matrix. The state update formula is: ; The model probability is calculated for each sub-model state update value, and the likelihood function is constructed based on the new information sequence: ; Normalize the likelihood function to get the model weight coefficient : ; Optimize the state vector by weighted fusion calculation : ; The augmented Lagrangian equation is constructed based on the optimization state vector and process constraints. Let the objective function be the defect rate , Molding accuracy Collaboration with workstations , the weight coefficients are 0.6, 0.3 and 0.1 respectively, and the augmented Lagrangian equation is: ; in represents the process constraints, is the Lagrange multiplier, is the penalty parameter. Based on the first-order derivative of the augmented Lagrangian equation and the second derivative , solve the optimal control quantity by Newton iteration method: ; Finally, the process parameter optimization data is obtained.

[0032] In a specific embodiment, the process of executing step S5 may specifically include the following steps: The historical processing trajectory in the process parameter optimization data is transformed into coordinates, and spatial segmentation is performed according to the density thresholds of three scales: 16×16 grid, 8×8 grid, and 4×4 grid, to obtain multi-scale grid data; The trajectory point density is calculated for each grid cell in the multi-scale grid data, and the area with a trajectory point density greater than 80% is divided into a high-density area, the area with a trajectory point density between 50% and 80% is divided into a medium-density area, and the area with a trajectory point density less than 50% is divided into a low-density area, and the grid density distribution data is obtained; Establishing Markov mobile state space for adjacent grid cells in grid density distribution data, calculating the number and time of trajectory transfer between adjacent grid cells, and obtaining trajectory transfer data; Normalize the trajectory transfer data, divide the number of transfers by the total number of transfers, and obtain the transition probability matrix between grid units; According to the transfer direction and transfer probability in the transfer probability matrix, the residence time distribution and movement direction distribution of the trajectory in each grid unit are calculated and used as the state transfer feature to obtain the state feature data; Construct a dynamic programming objective function for the state feature data, solve the dynamic programming objective function, calculate the state value of each stage through forward recursion, and record the optimal state transfer path to obtain the state path sequence; The state path sequence is traced back to extract the center position coordinates of each grid unit, and the trajectory is fitted through a cubic spline curve to obtain the optimal processing trajectory parameters.

[0033] Specifically, the historical processing trajectory in the process parameter optimization data includes the coordinate information of each sampling point on the processing path , these points represent the machining trajectory in a continuous form. In order to unify the coordinate systems of different scales, the original coordinate system is converted into a standardized coordinate system, assuming that the coordinate range is and , then the standardized coordinates of each point Calculated by the following formula: ; in represents the original coordinate point, is the boundary range of the trajectory. The normalized coordinates are distributed in the unit interval [0,1], providing consistent input for subsequent grid division. The normalized trajectory is segmented at multiple scales according to different grid sizes of 16×16, 8×8, and 4×4. The side lengths of each grid unit are and ,in Indicates the resolution of the grid (16, 8, or 4). Track Points Assign to the corresponding grid unit according to its coordinate value , the calculation formula is: ; in Indicates a round-down operation. The trajectory point density is calculated by counting the number of trajectory points contained in each grid cell. , defined as: ; in For Grid The number of trajectory points in is the total number of all trajectory points. According to the density threshold, the trajectory point density The grids with more than 80% of the grids are divided into high-density areas, the grids with 50%-80% of the grids are divided into medium-density areas, and the grids with less than 50% of the grids are divided into low-density areas. The grid density distribution data is obtained. After obtaining the grid density distribution data, the Markov mobile state space is constructed by analyzing the transfer behavior between adjacent grid units. Assume that the adjacent grid units are and , record the trajectory from Transfer to The number of and transfer time By counting the transfer behaviors of all trajectories, the trajectory transfer matrix is ​​constructed and the time matrix . Normalize the number of transitions and calculate the transition probability matrix : ; Based on the transition probability matrix and time matrix, calculate the residence time distribution of the trajectory in each grid cell and movement direction distribution . Assume that each stay time is ,but: ; ; in and Represents the displacement component of the grid unit. Enter the dynamic programming objective function. Assume that the objective function is to minimize the total trajectory transfer time and trajectory deviation: ; in is the transition time of the current state, is the current position of the track, is the offset penalty coefficient. The state value of each stage is calculated by the forward recursive algorithm: ; in is the transfer cost at the current stage, is the optimal value of the previous stage. Record the optimal state transfer path , perform backward tracing to extract the center position coordinates of the grid unit , the extracted trajectory points are fitted by a cubic spline curve. The fitting formula of the cubic spline curve is: ; in are the spline coefficients, is the horizontal coordinate of the fitting point. By constraining the continuity and smoothness of the fitting points, the optimal machining trajectory parameters are generated.

[0034] In a specific embodiment, the execution step establishes a Markov mobile state space for adjacent grid cells in the grid density distribution data, calculates the number of transfers and transfer time of the trajectory between adjacent grid cells, and obtains the trajectory transfer data, which may specifically include the following steps: Assign a continuous number from 1 to N to each grid unit in the grid density distribution data, and calculate the eight-neighborhood connection relationship of each grid unit through the spatial position relationship to obtain a grid connection matrix; Establish bidirectional connection edges for adjacent grid unit pairs in the grid connection matrix, and assign a weight coefficient to each connection edge. The connection edge weight in the high-density area is set to 1.0, the connection edge weight in the medium-density area is set to 0.8, and the connection edge weight in the low-density area is set to 0.6, so as to obtain a weighted connection graph; Based on the weighted connection graph, all transfer events and their occurrence times of the trajectory from one grid unit to another are recorded, and the transfer time intervals between adjacent moments are calculated to obtain the transfer time series; Mark the number of the starting grid and the target grid for each transfer event in the transfer time series, and count the number of transfers between each grid unit pair to obtain a transfer count matrix. Then segment the transfer count matrix, calculate the transfer frequency distribution in each time window, and obtain a time-varying transfer frequency matrix. Apply sliding average filtering to the time-varying transfer frequency matrix to obtain a smooth transfer frequency matrix, and build a Markov chain state transfer model based on the smooth transfer frequency matrix to calculate the transfer probability and average transfer time between any two adjacent grid cells to obtain a transfer feature matrix. The transition feature matrix is ​​combined with the spatial position information of the grid cells to construct trajectory transition data including position, transition probability and transition time.

[0035] Specifically, a unique number is assigned to each grid cell in the grid density distribution data. Assume that the size of the grid is , then by traversing the rows and columns of the grid cells, assigning numbers to each grid in row priority order The specific numbering formula is: ; in and are the row index and column index of the grid cell respectively. This numbering method ensures that each grid cell is uniquely numbered and the numbering order is consistent with the grid layout. The eight-neighborhood connectivity of the grid cell is calculated through the spatial position relationship to construct the grid connectivity matrix If the number is The grid cells are numbered The grid cells of are adjacent, then they are defined in the connectivity matrix: ; otherwise The eight-neighborhood connection relationship includes the adjacent relationship in the up-down, left-right and four diagonal directions. For example, , Grid, Grid The eight neighborhoods include After obtaining the grid connection matrix, establish bidirectional connection edges for adjacent grid cell pairs, and assign weights to each connection edge according to the density attribute of the grid cell. Assume that the density attribute of the grid cell is classified into high-density, medium-density and low-density areas, and their weights are , and For the connection matrix The position of the corresponding weighted connection graph By Grid and The specific calculation is: ; in and The grid cells are and This rule ensures that low-density areas weaken the connection weight. , records all the transfer events and their occurrence times of the trajectory from one grid unit to another. Let the time series of the trajectory points be , the position number of the trajectory point in the grid is For any continuous trajectory points and ,like , then record once from arrive The time interval of the transfer event is: ; By traversing the trajectory point sequence, count the number of transitions between each pair of grid cells and time interval , construct the transfer count matrix and the time matrix . For the transition count matrix Segmentation, according to fixed time windows Divide the time series and calculate the transition frequency in each time window as: ; in Represents a time window This process generates a time-varying transition frequency matrix , which is used to describe the dynamic changes of the transfer frequency between grid cells. Apply sliding average filtering to remove high-frequency noise and smooth transition features. Set the filter window size to , then the smoothed transfer frequency matrix Calculated as: ; Based on the smoothed frequency matrix , construct a Markov chain state transition model and calculate the transition probability between adjacent grid cells and the average transfer time The transition probability is calculated as: ; The average transfer time is: ; Transfer the feature matrix and The center position of the grid cell The combination generates trajectory transfer data including spatial position information, transfer probability and transfer time, which is used to analyze the dynamic characteristics of the machining path.

[0036] The above describes the real-time defect detection method for the blister carrier forming process in the embodiment of the present invention. The following describes the real-time defect detection device for the blister carrier forming process in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a real-time defect detection device for a blister carrier forming process includes: An acquisition module is used to acquire the first image data of the blister carrier tape forming process, and pre-process the pressure data, temperature data and displacement data of the blister carrier tape forming production line to construct a multi-source sensor data matrix; A training module is used to perform a three-layer one-dimensional convolution operation of a time domain branch and a two-layer two-dimensional convolution operation of a frequency domain branch on the first image data, and obtain a defect recognition deep learning model through fusion and training of a fully connected layer; A deployment module is used to deploy the defect recognition deep learning model to the edge computing device for multi-threaded concurrent processing, and perform feature extraction and model inference on the second image data collected in real time to obtain real-time defect detection data; Establish a module for establishing a state space model based on real-time defect detection data and multi-source sensor data matrix, optimizing and calculating process parameters, and obtaining process parameter optimization data; The solution module is used to perform multi-scale spatial division of the historical processing trajectory according to the process parameter optimization data, and solve the optimal processing trajectory parameters.

[0037] Through the collaborative cooperation of the above-mentioned components, by constructing a multi-source sensor data matrix, integrating multi-dimensional information such as pressure, temperature, and displacement, and combining the time domain and frequency domain feature extraction of image data, a comprehensive defect detection system is formed, which improves the accuracy and reliability of defect detection; the deep learning model architecture of three-layer one-dimensional convolution of time domain branch and two-layer two-dimensional convolution of frequency domain branch is adopted to effectively extract the temporal characteristics and frequency domain characteristics of the image, and enhance the model's recognition ability for different types of defects; through the multi-threaded concurrent processing mechanism of edge computing devices, the detection response speed is improved to meet the real-time monitoring needs of high-speed production lines; the process parameter optimization method based on the interactive multi-model Kalman filter algorithm realizes the dynamic estimation and optimization of parameters such as temperature, pressure, and displacement, and enhances the system's adaptability to changes in process parameters; the processing trajectory optimization scheme using multi-scale space division and Markov mobile model realizes intelligent optimization of the processing trajectory through the precise calculation of the transfer probability between grid units, thereby improving the forming accuracy.

[0038] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0039] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0040] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0041] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0042] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0043] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0044] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time defect detection method for a blister carrier forming process, characterized in that: The method comprises: Collect the first image data of the blister carrier tape forming process, and pre-process the pressure data, temperature data and displacement data of the blister carrier tape forming production line to construct a multi-source sensor data matrix; Performing a three-layer one-dimensional convolution operation of a time domain branch and a two-layer two-dimensional convolution operation of a frequency domain branch on the first image data, and obtaining a deep learning model for defect recognition through fusion and training of a fully connected layer; The defect recognition deep learning model is deployed to an edge computing device for multi-threaded concurrent processing, and feature extraction and model inference are performed on the second image data collected in real time to obtain real-time defect detection data; Establishing a state space model based on the real-time defect detection data and the multi-source sensor data matrix, optimizing and calculating the process parameters, and obtaining process parameter optimization data; The historical processing trajectory is divided into multi-scale spaces according to the process parameter optimization data to obtain the optimal processing trajectory parameters.

2. The real-time defect detection method for the blister carrier forming process according to claim 1 is characterized in that: The first image data of the blister carrier tape forming process is collected, and the pressure data, temperature data and displacement data of the blister carrier tape forming production line are preprocessed to construct a multi-source sensor data matrix, including: Continuously photographing the surface of the blister carrier tape with an industrial camera to obtain an original image sequence, and graying and resizing the original image sequence to obtain first image data; The original signal collected by the pressure sensor in the blister carrier forming production line is sampled, and the spectrum is analyzed by Fourier transform to obtain the pressure basic data, and the spectrum characteristics of the pressure basic data are subjected to bandpass filtering to obtain the pressure frequency domain data; The signals collected by the temperature sensor and displacement sensor in the blister carrier molding production line are sampled, and the signals are smoothed by Gaussian filtering to obtain the basic temperature and displacement data; The pressure frequency domain data is restored to time domain data by inverse Fourier transform, and timestamp alignment and segmentation processing are performed with the temperature displacement basic data to obtain a preprocessed data sequence; According to the order of pressure data, temperature data and displacement data, matrix splicing and normalization processing are performed on the pre-processed data sequence to obtain a multi-source sensing data matrix.

3. The real-time defect detection method for the blister carrier forming process according to claim 2 is characterized in that: The first image data is subjected to a three-layer one-dimensional convolution operation of a time domain branch and a two-layer two-dimensional convolution operation of a frequency domain branch, and a defect recognition deep learning model is obtained through fusion and training of a fully connected layer, including: Performing time series segmentation on the first image data, setting the time window length to 32 frames and the sliding step length to 16 frames, to obtain a time series image sequence; Input the time-series image sequence into the first one-dimensional convolution layer of the time domain branch, set the convolution kernel size to 5, the number of convolution kernels to 32, and process it through the BatchNormalization layer and the ReLU activation function to obtain the first feature map in the time domain; Input the first time-domain feature map into the second one-dimensional convolution layer of the time-domain branch, set the convolution kernel size to 3, the number of convolution kernels to 64, process it through the BatchNormalization layer and the ReLU activation function, and process it through the maximum pooling layer to obtain the second time-domain feature map; Input the second time-domain feature map into the third one-dimensional convolution layer of the time-domain branch, set the convolution kernel size to 3, the number of convolution kernels to 128, and process it through the BatchNormalization layer and the ReLU activation function to obtain the final time-domain feature map; Performing a fast Fourier transform on the first image data to obtain frequency domain image data, and inputting the frequency domain image data into the first two-dimensional convolution layer of the frequency domain branch, setting the convolution kernel size to 3×3 and the number of convolution kernels to 64, and processing with a LeakyReLU activation function to obtain a first feature map in the frequency domain; Input the first frequency domain feature map into the second two-dimensional convolution layer of the frequency domain branch, set the convolution kernel size to 3×3, the number of convolution kernels to 128, and process it through the LeakyReLU activation function and the Dropout layer to obtain the final frequency domain feature map; The final feature map in the time domain and the final feature map in the frequency domain are concatenated in the channel dimension, and input into the fully connected layer, the number of hidden layer nodes is set to 256, and processed by the Dropout layer and the ReLU activation function to obtain a fused feature vector; The fused feature vector is input into the classification layer, the number of output nodes is set to the number of defect categories, processed by the Softmax activation function, and trained and optimized using the cross entropy loss function to obtain a defect recognition deep learning model.

4. The real-time defect detection method for the blister carrier forming process according to claim 3 is characterized in that: The defect recognition deep learning model is deployed to the edge computing device for multi-threaded concurrent processing, and feature extraction and model inference are performed on the second image data collected in real time to obtain real-time defect detection data, including: Performing model structure optimization and parameter quantization processing on the defect recognition deep learning model, converting the floating point number precision from FP32 to FP16, and obtaining an edge deployment model; Constructing a data collection thread, a feature extraction thread, and a model reasoning thread for the edge deployment model, and using a thread pool management mechanism for scheduling to obtain a multi-threaded processing system; The multi-threaded processing system is run on an edge computing device, an industrial camera is used to collect a surface image of the blister carrier in real time, and image data is updated based on a data cache queue to obtain second image data; Building a time series window based on the second image data, setting the number of parallel processing units to 4, and performing parallel preprocessing of the image data in batches to obtain data to be inferred; Input the data to be inferred into the time domain branch of the edge deployment model, perform feature calculation through a three-layer one-dimensional convolutional network, and obtain time domain reasoning features; The data to be inferred is subjected to a fast Fourier transform and then input into the frequency domain branch of the edge deployment model, and feature calculation is performed through a two-layer two-dimensional convolutional network to obtain frequency domain reasoning features; The time domain reasoning features and the frequency domain reasoning features are fused, and the defect category probability is calculated using the fully connected layer and the Softmax function to obtain the defect detection result. The defect detection result is associated with the image location information and timestamp information, and data integration is performed through the result summary thread to obtain real-time defect detection data.

5. The real-time defect detection method for the blister carrier forming process according to claim 4 is characterized in that: The step of establishing a state space model based on the real-time defect detection data and the multi-source sensor data matrix, optimizing and calculating the process parameters, and obtaining process parameter optimization data includes: Based on the real-time defect detection data and the multi-source sensor data matrix, state space modeling is performed, a state vector is set to include temperature parameters, pressure parameters, displacement parameters and defect characteristic parameters, and a measurement vector is set to include sensor data and defect detection results, to obtain a system state equation; Constructing three sub-models for the state equation of the system: setting the first sub-model as a uniform speed model and the state transfer matrix as a diagonal matrix, setting the second sub-model as a uniform acceleration model and the state transfer matrix including a speed term, setting the third sub-model as a conversion model and the state transfer matrix including a nonlinear term, and obtaining a hybrid model system; Performing state prediction on the first sub-model, the second sub-model and the third sub-model in the hybrid model system according to the Kalman filter formula, multiplying the state vector at the previous moment by the corresponding state transfer matrix and adding process noise to obtain a predicted state vector; Calculate the new information sequence based on the predicted state vector and the actual measurement data, and calculate the Kalman gain according to the measurement equation and the measurement noise covariance matrix to obtain the state update value of each sub-model; Performing model probability calculation on the state update values ​​of each sub-model, constructing a likelihood function based on the new information sequence and the prediction covariance matrix to obtain a model weight coefficient, and performing weighted fusion on the state update values ​​of each sub-model according to the model weight coefficient, and performing interactive resampling calculation to obtain an optimized state vector; A Lagrangian function is constructed for the optimization state vector and process constraints, and the defect rate weight coefficient is set to 0.6, the molding accuracy weight coefficient is set to 0.3, and the station coordination weight coefficient is set to 0.1, thereby obtaining an augmented Lagrangian equation; The KKT condition is constructed based on the first-order derivative and the second-order derivative of the augmented Lagrangian equation, and the optimal control quantity is solved by the Newton iteration method to obtain the process parameter optimization data.

6. The real-time defect detection method for the blister carrier forming process according to claim 5 is characterized in that: The multi-scale spatial division of the historical processing trajectory according to the process parameter optimization data to obtain the optimal processing trajectory parameters includes: Coordinate transformation is performed on the historical processing trajectory in the process parameter optimization data, and spatial segmentation is performed according to density thresholds of three scales: 16×16 grid, 8×8 grid, and 4×4 grid, to obtain multi-scale grid data; Calculating the trajectory point density for each grid cell in the multi-scale grid data, and dividing the area with a trajectory point density greater than 80% into a high-density area, the area with a trajectory point density between 50% and 80% into a medium-density area, and the area with a trajectory point density less than 50% into a low-density area, to obtain grid density distribution data; Establishing a Markov mobile state space for adjacent grid cells in the grid density distribution data, calculating the number of transfers and transfer time of the trajectory between adjacent grid cells, and obtaining trajectory transfer data; Normalizing the trajectory transfer data, dividing the number of transfers by the total number of transfers, and obtaining a transfer probability matrix between grid units; Calculate the residence time distribution and movement direction distribution of the trajectory in each grid unit according to the transition direction and transition probability in the transition probability matrix, and use them as state transition features to obtain state feature data; Constructing a dynamic programming objective function for the state characteristic data, solving the dynamic programming objective function, calculating the state value of each stage by forward recursion, and recording the optimal state transfer path to obtain a state path sequence; The state path sequence is backtracked to extract the center position coordinates of each grid unit, and the trajectory is fitted through a cubic spline curve to obtain the optimal processing trajectory parameters.

7. The real-time defect detection method for the blister carrier forming process according to claim 6 is characterized in that: The step of establishing a Markov mobile state space for adjacent grid cells in the grid density distribution data, calculating the number of transfers and transfer time of trajectories between adjacent grid cells, and obtaining trajectory transfer data includes: Assigning a continuous number from 1 to N to each grid unit in the grid density distribution data, and calculating the eight-neighborhood connection relationship of each grid unit through the spatial position relationship to obtain a grid connection matrix; Establishing bidirectional connection edges for adjacent grid unit pairs in the grid connection matrix, and assigning a weight coefficient to each connection edge, setting the connection edge weight of the high-density area to 1.0, the connection edge weight of the medium-density area to 0.8, and the connection edge weight of the low-density area to 0.6, to obtain a weighted connection graph; Based on the weighted connection graph, all transfer events and their occurrence times of the trajectory from one grid unit to another grid unit are recorded, and the transfer time intervals between adjacent moments are calculated to obtain a transfer time series; Marking the numbers of the starting grid and the target grid for each transfer event in the transfer time series, and counting the number of transfers between each grid unit pair to obtain a transfer count matrix, and segmenting the transfer count matrix to calculate the transfer frequency distribution in each time window to obtain a time-varying transfer frequency matrix; Applying sliding average filtering to the time-varying transfer frequency matrix to obtain a smoothed transfer frequency matrix, and constructing a Markov chain state transfer model based on the smoothed transfer frequency matrix, calculating the transfer probability and average transfer time between any two adjacent grid units, and obtaining a transfer feature matrix; The transfer feature matrix is ​​combined with the spatial position information of the grid unit to construct trajectory transfer data including position, transfer probability and transfer time.

8. A real-time defect detection device for a blister carrier forming process, characterized in that: Used to perform the real-time defect detection method for the blister carrier forming process according to any one of claims 1 to 7, the real-time defect detection device for the blister carrier forming process comprises: An acquisition module is used to acquire the first image data of the blister carrier tape forming process, and pre-process the pressure data, temperature data and displacement data of the blister carrier tape forming production line to construct a multi-source sensor data matrix; A training module, used to perform a three-layer one-dimensional convolution operation of a time domain branch and a two-layer two-dimensional convolution operation of a frequency domain branch on the first image data, and obtain a defect recognition deep learning model through fusion and training of a fully connected layer; A deployment module, used to deploy the defect recognition deep learning model to an edge computing device for multi-threaded concurrent processing, and perform feature extraction and model inference on the second image data collected in real time to obtain real-time defect detection data; An establishment module is used to establish a state space model based on the real-time defect detection data and the multi-source sensor data matrix, optimize and calculate the process parameters, and obtain process parameter optimization data; The solution module is used to perform multi-scale spatial division of the historical processing trajectory according to the process parameter optimization data to solve and obtain the optimal processing trajectory parameters.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and is characterized in that when the processor executes the computer program, the real-time defect detection method for the blister carrier forming process described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is enabled to execute the method for real-time defect detection in a blister carrier tape molding process according to any one of claims 1 to 7.

Images