A method for identifying cold-rolled plate shape defects based on multimodal data

Abstract

The present invention discloses a method for identifying cold-rolled flat plate defects based on multimodal data, comprising: collecting flatness information of cold-rolled steel plates, the flatness information including flatness image information, process timing parameter information, and motion timing parameter information of the cold-rolled steel plates; constructing a cold-rolled flat plate defect recognition model, inputting the flatness image information of the cold-rolled steel plates into a lightweight CNN module of the cold-rolled flat plate defect recognition model to extract image features, inputting the process timing parameter information and motion timing parameter information of the cold-rolled steel plates into a bidirectional LSTM module of the cold-rolled flat plate defect recognition model to extract timing features, inputting the extracted image features and timing features into a spatiotemporal cross-attention layer to extract fused features, and inputting the extracted fused features into a defect classification module and a defect ratio prediction module, respectively, to predict the defect type and defect ratio of the cold-rolled steel plates. The present invention can achieve high-precision and low-latency cold-rolled flat plate defect recognition.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent manufacturing and industrial detection technology, and specifically relates to a cold-rolled plate shape defect recognition method based on multimodal data. Background Art

[0002] Cold-rolled plate defects, such as edge waves, center waves, and warping, directly affect steel quality and production efficiency. Traditional manual visual inspection is limited by human subjectivity and fatigue, is inefficient, and cannot guarantee accurate identification. Real-time defect detection can reduce downtime and adjustment time, improving the continuity of the production line.

[0003] However, current methods for extracting defect outlines using Canny edge detection and threshold segmentation are insufficient for detecting tiny defects such as pinholes in cold-rolled sheet and lack adaptability to complex textures such as thermal imaging temperature fields, resulting in low accuracy in identifying cold-rolled sheet defects. Shallow machine learning methods such as random forests and support vector machines require manual feature extraction of cold-rolled sheet shapes, making it difficult to model the dynamic relationship between process parameters and defects and failing to meet the real-time requirements of production lines. The development of deep learning technology, particularly the application of CNNs and LSTMs, has provided new solutions for cold-rolled sheet defect identification. However, existing deep learning models often suffer from high computational complexity and resource consumption, making them difficult to meet the real-time requirements of industrial sites, and their accuracy also needs to be improved. Summary of the Invention

[0004] In response to the problems existing in the prior art, the present invention provides a cold-rolled plate shape defect identification method based on multimodal data. By constructing a lightweight cold-rolled plate shape defect detection model, the feature extraction capability of cold-rolled steel plates and the ability to capture key information are enhanced, thereby achieving high-precision and low-latency cold-rolled plate shape defect identification.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: a method for identifying cold-rolled plate shape defects based on multimodal data, specifically comprising the following steps:

[0006] Step S1, collecting the shape information of the cold-rolled steel plate, wherein the shape information includes: shape image information, process timing parameter information, and motion timing parameter information of the cold-rolled steel plate;

[0007] Step S2: constructing a cold-rolled plate shape defect recognition model consisting of a lightweight CNN module, a bidirectional LSTM module, a spatiotemporal cross attention layer, a defect classification module, and a defect ratio prediction module;

[0008] Step S3: Input the plate image information of the cold-rolled steel plate into the lightweight CNN module of the cold-rolled plate shape defect recognition model to extract image features, input the process timing parameter information and motion timing parameter information of the cold-rolled steel plate into the bidirectional LSTM module of the cold-rolled plate shape defect recognition model to extract timing features, input the extracted image features and timing features into the spatiotemporal cross attention layer to extract fusion features, and input the extracted fusion features into the defect classification module and the defect proportion prediction module respectively to predict the defect type and defect proportion of the cold-rolled steel plate respectively.

[0009] Furthermore, the plate shape image information of the cold-rolled steel plate is obtained by collecting plate shape radial force information and plate shape infrared thermal imaging images, encoding the plate shape radial force information into a 2D pseudo image using a Gram angle difference field, mapping it to the red channel, mapping the corresponding plate shape infrared thermal imaging image to the green channel, and setting the blue channel to 0 to obtain three-channel plate shape image information.

[0010] Furthermore, the process timing parameter information of the cold-rolled steel plate includes: rolling force, bending roll force and tension of the cold-rolled steel plate; the motion timing parameter information of the cold-rolled steel plate includes: running distance, speed and thickness of the cold-rolled steel plate.

[0011] Furthermore, the lightweight CNN module includes: a multi-scale depth-separable convolution module, a dynamic channel reorganization residual connection module, an adaptive receptive field module and a global average pooling layer;

[0012] The multi-scale depth-separable convolution module captures the multi-scale features of cold-rolled flatness defects in the flatness image information through parallel depth-separable convolution at different scales, and splices the multi-scale features to obtain a fused feature map;

[0013] The dynamic channel reorganization residual connection module is used to dynamically suppress the noise channel on the fusion feature map and enhance the cold-rolled plate shape defect characteristics;

[0014] The adaptive receptive field module dynamically adjusts the receptive field size of the convolution kernel according to the fusion feature map enhanced by the cold-rolled plate shape defect feature, thereby enhancing the ability to capture cold-rolled plate shape defects of different sizes and obtaining image spatial features that fuse multi-scale receptive fields;

[0015] The global average pooling layer compresses the global spatial information of the image spatial features fused with the multi-scale receptive fields to obtain image features.

[0016] Furthermore, the dynamic channel reorganization residual connection module is used to dynamically suppress the noise channel on the fusion feature map and enhance the cold-rolled plate shape defect feature. The specific process is as follows:

[0017] i. Calculate the global statistics of each channel based on the feature sub-graph on each channel of the fused feature map ,in, Indicates the height of the fused feature map, i express The index of represents the width of the fused feature map, j express The index of Represents the fusion feature map c The height of the channel is i , width is j The characteristic subgraph of

[0018] ii. Generate the importance weight of each channel through the fully connected layer and normalize it by the Sigmoid function to obtain the channel attention weight ,in, represents the weight matrix of the fully connected layer, represents the bias of the fully connected layer;

[0019] iii. According to channel attention weight Filter out important channels and dynamically prune the fused feature map: ,in, represents the fused feature map after dynamic pruning, represents the fused feature map, represents the indicator function, , represents the channel pruning threshold, Indicates the number of channels;

[0020] iv. Perform residual connection on the fused feature map after dynamic pruning and expand the channels to the original number of channels.

[0021] Furthermore, the acquisition process of the image spatial features of the fused multi-scale receptive field is as follows: the fused feature map with enhanced cold-rolled plate shape defect features is subjected to global average pooling to obtain channel-level statistics, the channel-level statistics are mapped to the expansion rate weight of the corresponding receptive field through linear transformation, the convolution kernel of the receptive field is adjusted according to the expansion rate weight to perform convolution operation, and the convolution results of different expansion rate weights are weighted to obtain the image spatial features after the fusion of the multi-scale receptive field.

[0022] Furthermore, the bidirectional LSTM module includes: a forward LSTM and a backward LSTM, the forward LSTM extracts the process timing characteristics of the cold-rolled steel plate based on the process timing parameter information of the cold-rolled steel plate, and the backward LSTM extracts the motion timing characteristics of the cold-rolled steel plate based on the motion timing parameter information of the cold-rolled steel plate, and the process timing characteristics and motion timing characteristics of the cold-rolled steel plate are reduced in dimension along the time dimension through average pooling and spliced into the timing characteristics of the cold-rolled steel plate.

[0023] Furthermore, the extraction process of the fusion feature is as follows:

[0024]

[0025] in, represents the fusion feature, represents the fusion operation, represents the extracted image features, represents the features fused through the spatiotemporal cross attention mechanism, , represents the query vector generated by linear transformation of time series features, Represents the key vector generated by linear transformation of image features, Represents the value vector generated by linear transformation of image features, represents the attention mechanism scaling factor, Represents causal mask The resulting lower triangular matrix, , represents the current time step, Represents a historical time step.

[0026] Furthermore, the process of predicting the defect type of the cold-rolled steel sheet is as follows:

[0027]

[0028] in, represents the predicted probability of defect type of cold-rolled steel sheet, 、 Represent the weights and biases of the fully connected layer in the defect classification module respectively;

[0029] The prediction process of the defect ratio of the cold-rolled steel sheet is as follows:

[0030]

[0031] in, Indicates the defect ratio of cold-rolled steel sheets, S Indicates the defect type of cold rolled steel sheet, s express S The index of 、 They represent the weight matrix and bias of the fully connected layer in the defect ratio prediction module respectively.

[0032] Furthermore, the cold-rolled flat shape defect recognition model needs to be trained until the loss function of the cold-rolled flat shape defect recognition model converges, thereby completing the training of the cold-rolled flat shape defect recognition model.

[0033] The loss function of the cold-rolled plate shape defect recognition model for:

[0034]

[0035] in, represents the adaptive loss function for the classification task, , represents the category balance factor for the classification task loss, represents the adjustment factor of the classification task loss; represents the quantile loss function for the regression task, , 、 Represent the true value and predicted value of the cold rolled plate shape defect type, The quantile parameter representing the quantile loss for regression tasks, is the regularization coefficient, represents the training parameters in the cold-rolled plate shape defect recognition model, for of norm.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention adopts a cold-rolled plate shape defect recognition method based on multimodal data, which adopts a cold-rolled plate shape defect recognition model composed of a lightweight CNN module, a bidirectional LSTM module, a spatiotemporal cross attention layer, a defect classification module and a defect proportion prediction module. It can not only recognize the type of cold-rolled plate shape defect, but also quantitatively analyze the proportion of cold-rolled plate shape defect types, provide a quantitative basis for rolling parameter optimization, and promote the upgrading of intelligent manufacturing. At the same time, the lightweight CNN module, when extracting image features, enhances the cold-rolled plate shape defect features in the multi-scale feature fusion feature map to extract richer detail features, providing a basis for accurate recognition of cold-rolled plate shape defects, and can reduce computational difficulty and delay. The bidirectional LSTM module is used to capture the historical dependency and future correlation of temporal features, obtain more comprehensive temporal features, and fuse the extracted image features with the temporal features through the spatiotemporal cross attention mechanism to improve the expression ability of multimodal features and the recognition ability of cold-rolled plate shape defects. The present invention can enhance the feature extraction capability of cold-rolled steel sheets and the ability to capture key information, and realize high-precision and low-latency cold-rolled plate shape defect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the cold-rolled plate shape defect identification method based on multimodal data of the present invention;

[0039] Figure 2 Schematic diagram of the cold-rolled plate shape defect detection model in the present invention;

[0040] Figure 3 Schematic diagram of the comparison between the cold-rolled plate shape defects predicted by the cold-rolled plate shape defect identification method based on multimodal data of the present invention and the actual defects. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be further explained below with reference to the accompanying drawings.

[0042] like Figure 1 Flowchart of the cold-rolled flat shape defect recognition method based on multimodal data of the present invention, the cold-rolled flat shape defect recognition method comprises the following steps:

[0043] Step S1: collecting shape information of the cold-rolled steel plate, including shape image information, process timing parameter information, and motion timing parameter information of the cold-rolled steel plate. The shape image information of the cold-rolled steel plate is obtained by collecting radial force information and infrared thermal imaging images of the cold-rolled steel plate, encoding the radial force information into a 2D pseudo-image using a Gramian Angular Difference Field (GADF) and mapping it to a red channel, mapping the corresponding infrared thermal imaging image to a green channel, and setting the blue channel to 0, thereby obtaining three-channel shape image information. The process timing parameter information of the cold-rolled steel plate includes rolling force, bending roll force, and tension of the cold-rolled steel plate. The motion timing parameter information of the cold-rolled steel plate includes running distance, speed, and thickness of the cold-rolled steel plate.

[0044] In one technical solution of the present invention, the collected shape information of the cold-rolled steel plate also needs to be denoised and smoothed, wherein the process of denoising the shape information of the cold-rolled steel plate is as follows: for periodic interference or noise with obvious frequency band separation such as electromagnetic interference and rolling mill vibration noise, an N-order Butterworth low-pass filter is used to filter, and the low-frequency shape signal is retained by frequency domain truncation, and the high-frequency mechanical vibration noise is filtered out; for low-frequency long-period interference such as temperature drift, a moving average filter is used to suppress low-frequency noise. The process of smoothing the shape information of the cold-rolled steel plate is as follows: each time series parameter information is smoothed and normalized based on the sliding window length. ,in, Represents the time series parameter information after smoothing and normalization, , 、 Represent the mean and standard deviation of the timing parameter information based on the sliding window length respectively.

[0045] Step S2: Figure 2, construct a cold-rolled plate shape defect recognition model consisting of a lightweight CNN module, a bidirectional LSTM module, a spatiotemporal cross attention layer, a defect classification module and a defect ratio prediction module, enhance the feature extraction capability of cold-rolled steel plates and the ability to capture key information, and achieve high-precision and low-latency cold-rolled plate shape defect recognition.

[0046] The lightweight CNN module in the present invention includes: a multi-scale depth-separable convolution module, a dynamic channel reorganization residual connection module, an adaptive receptive field module and a global average pooling layer;

[0047] The multi-scale depthwise separable convolution module captures the multi-scale features of cold-rolled plate defects in plate image information, such as microcracks, edge waves, and mid-wave information, through parallel depthwise separable convolutions at different scales, and splices the multi-scale features to obtain a fused feature map. Among them, the depthwise separable convolution first performs a depthwise convolution operation on each channel on the plate image information through the depthwise convolution kernel, and then performs a point-by-point convolution operation through the convolution kernel, which greatly reduces the number of parameters of the cold-rolled plate defect recognition model.

[0048] The dynamic channel reorganization residual connection module is used to dynamically suppress the noise channel on the fusion feature map and enhance the cold-rolled plate shape defect characteristics; specifically,

[0049] i. Calculate the global statistics of each channel based on the feature sub-graph on each channel of the fused feature map ,in, Indicates the height of the fused feature map, i express The index of represents the width of the fused feature map, j express The index of Represents the fusion feature map c The height of the channel is i , width is j The characteristic subgraph of

[0050] ii. Generate the importance weight of each channel through the fully connected layer and normalize it by the Sigmoid function to obtain the channel attention weight ,in, represents the weight matrix of the fully connected layer, The data is initialized to a mean of 0 and a variance of 1 through the Xavier normal distribution, denoted as ; represents the bias of the fully connected layer, Initialize to 0 vector;

[0051] iii. According to channel attention weight By screening out important channels and dynamically pruning the fused feature map, redundant channels can be closed, further reducing redundant calculations. The calculation process of dynamic pruning is as follows: , represents the fused feature map after dynamic pruning, represents the fused feature map, represents the indicator function, , Indicates the number of channels, represents the channel pruning threshold, , Indicates the number of channels to be retained after pruning, the value is ; is the value function, To retain The top 50% of the importance channels are used to focus on defect-related areas; is the weight matrix, which is initialized using Kaiming normal distribution. ;

[0052] iv. Perform residual connection on the fused feature map after dynamic pruning to alleviate the gradient disappearance and expand the channel to the original number of channels. The output is , For the fused feature map after pruning conduct Convolution operation.

[0053] The adaptive receptive field module dynamically adjusts the receptive field size of the convolution kernel based on the fused feature map enhanced with cold-rolled flat defects, enhancing the ability to capture cold-rolled flat defects of different sizes and obtaining image spatial features that fuse the multi-scale receptive field. Specifically, the fused feature map enhanced with cold-rolled flat defects undergoes global average pooling to obtain channel-level statistics. These channel-level statistics are then mapped to the dilation rate weights of the corresponding receptive field through linear transformation. The convolution kernel of the receptive field is adjusted based on the dilation rate weights and convolution operations are performed. The convolution results with different dilation rate weights are then weighted to obtain image spatial features after the multi-scale receptive field fusion. Through the adaptive receptive field module, if the input features contain large-scale defects, the dilation rate weight is increased, otherwise, it is reduced, thereby achieving adaptive adjustment of the dilation rate weights and enhancing the ability to capture defects of different sizes.

[0054] The global average pooling layer compresses the global spatial information of the image spatial features that fuse the multi-scale receptive fields to obtain image features.

[0055] The bidirectional LSTM module in the present invention includes: a forward LSTM and a backward LSTM. The forward LSTM extracts the process timing characteristics of the cold-rolled steel plate according to the process timing parameter information of the cold-rolled steel plate, and the backward LSTM extracts the motion timing characteristics of the cold-rolled steel plate according to the motion timing parameter information of the cold-rolled steel plate. The process timing characteristics and the motion timing characteristics of the cold-rolled steel plate are reduced in dimension along the time dimension by average pooling and spliced into the timing characteristics of the cold-rolled steel plate. The bidirectional LSTM module is used to capture the historical dependency and future correlation of the timing characteristics, obtain more comprehensive timing characteristics, reduce redundant parameters, and improve the efficiency of timing modeling.

[0056] The spatiotemporal cross attention layer in the present invention dynamically fuses image features and temporal features through the spatiotemporal cross attention mechanism to improve the correlation analysis capability of the causes of shape defects of cold-rolled steel sheets. The extraction process of fused features is as follows:

[0057]

[0058] in, represents the fusion feature, represents the fusion operation, represents the extracted image features, represents the features fused through the spatiotemporal cross attention mechanism, , Indicates that the time series features are linearly transformed The generated query vector, , Represents the extracted temporal features; Represents image features after linear transformation The generated key vector, ; Represents image features after linear transformation The generated value vector, ; represents the attention mechanism scaling factor to prevent gradient explosion, ; Represents causal mask The resulting lower triangular matrix, , represents the current time step, represents the historical time step, when When , it means that access to historical information is allowed. , it means shielding future information, thereby ensuring that the current time step only depends on historical information and effectively preventing future information leakage.

[0059] The defect classification module linearizes the fused features through a fully connected layer and activates the Softmax function to predict the probability distribution of the defect type of the cold-rolled steel plate:

[0060]

[0061] in, represents the predicted probability of defect type of cold-rolled steel sheet, 、 denote the weight and bias of the fully connected layer in the defect classification module, Initialized by He normal distribution , is a matrix the number of rows, Initialize to 0 vector;

[0062] The defect ratio prediction module linearizes the fused features through a fully connected layer, activates and normalizes them using the Softmax function, and predicts the defect ratio of various cold-rolled steel sheets:

[0063]

[0064] in, Indicates the defect ratio of cold-rolled steel sheets, S Indicates the defect type of cold rolled steel sheet, s express S The index of 、 Represent the weight matrix and bias of the fully connected layer in the defect ratio prediction module, Initialized by Xavier uniform distribution , 、 They are The number of rows and columns, Initialized to a 0 vector.

[0065] Step S3: Input the plate image information of the cold-rolled steel plate into the lightweight CNN module of the cold-rolled plate shape defect recognition model to extract image features, input the process timing parameter information and motion timing parameter information of the cold-rolled steel plate into the bidirectional LSTM module of the cold-rolled plate shape defect recognition model to extract timing features, input the extracted image features and timing features into the spatiotemporal cross attention layer to extract fusion features, and input the extracted fusion features into the defect classification module and the defect proportion prediction module respectively to predict the defect type and defect proportion of the cold-rolled steel plate respectively.

[0066] In one technical solution of the present invention, it is necessary to train the cold-rolled flat shape defect recognition model until the loss function of the cold-rolled flat shape defect recognition model converges, thereby completing the training of the cold-rolled flat shape defect recognition model:

[0067] The first stage is pre-training. First, the bidirectional LSTM module is fixed and only the board image information is used to train the lightweight CNN module, spatiotemporal cross attention layer and defect classification module. The discard rate is , the number of hidden units is 64, and the batch size is , set the adaptive loss function for the classification task , and use progressive warm-up and cosine annealing to design the learning rate:

[0068] ,

[0069] in, Represents the category balance factor of the classification task loss to dynamically adjust the category weights, , Indicates the The number of class samples, Represents the adjustment factor of the classification task loss, taking ; For current training epoch number, Warm-up phase epoch Number, take , the learning rate increases linearly from 0 to the maximum learning rate ; For total training epoch Number, take ; is the minimum learning rate in the cosine annealing stage, and .

[0070] Then, the lightweight CNN module is fixed, and the process timing parameter information and motion timing parameter information are used to train the bidirectional LSTM module, the spatiotemporal cross attention layer and the defect ratio prediction module. , the number of hidden units is 64, and the batch size is The loss function is designed through median regression to reduce the noise impact caused by process parameter fluctuations. The quantile loss function of the regression task is: , and combine cyclic learning rate and gradient clipping to design learning rate to accelerate training and avoid the algorithm falling into local optimum, each epoch At the end, the updated learning rate is: ,in, 、 Represent the true value and predicted value of the cold rolled plate shape defect type, is the quantile parameter of the quantile loss of the regression task, which represents the median adjustment factor. To balance median regression with outlier robustness, is the basic learning rate, ; is the maximum learning rate, take ; is the cycle length, take ; Indicates the current epoch Position within the cycle, Represents the modulo operation, that is, calculating the current training round epoch Divide by The remainder of , then the remainder is 0, 1, 2, ..., 9. When the remainder is 0, the learning rate returns to the starting point , starting a new cycle.

[0071] The optimizer clips the gradient before each parameter update. The optimizer updates the parameters based on the clipped gradient to prevent gradient explosion and improve training stability:

[0072]

[0073] in, It is the gradient set of all trainable parameters of the cold-rolled flat defect recognition model, which is calculated by back propagation; for The L2 norm of is the clipping threshold, which is set to 1.0.

[0074] When 5 consecutive epoch When the verification loss function does not decrease, the pre-training ends.

[0075] The second stage is joint fine-tuning, which unfreezes all parameters in the cold-rolled flatness defect recognition model. The Nadam optimizer with Nesterov momentum is used, and a dynamically self-adjusted learning rate is introduced: , the total loss function consists of classification loss, regression loss and regularization term: ,in, Cold rolled plate shape defect recognition model The learning rate at each moment; is the basic learning rate of the cold-rolled plate shape defect recognition model, and ; is the number of warm-up steps, which is taken as 1000; is the minimum value function; is the regularization coefficient, take ; represents the training parameters in the cold-rolled plate shape defect recognition model, for of norm.

[0076] When 10 consecutive epoch When the total validation loss does not decrease, the training ends.

[0077] Convert the trained cold-rolled plate defect recognition model into ONNX format. Use a certain number of calibration sets, here we take 500 representative samples, covering different defect types and process conditions. Use entropy calibration to minimize the KL (Kullback-Leibler Divergence) divergence between FP32 and INT8 activation distributions to minimize the distribution difference before and after quantization. Determine the minimum KL divergence. Generate quantization parameters for the optimal threshold, record the FP32 activation value of each layer of the cold-rolled plate shape defect recognition model, load the quantization parameters and start INT8 quantization through the TensorRT engine to achieve a balance between accuracy and speed of the cold-rolled plate shape defect recognition model. , FP32 distribution refers to the numerical distribution of the activation value output by a certain layer of the cold-rolled plate defect recognition model under full-precision floating-point conditions. The number of and normalized probability distribution; is the candidate threshold; INT8 distribution, which refers to the numerical distribution of activation values of the same layer after quantization; For the indivual FP32 activation value probability; is the threshold Next indivual The probability of the INT8 activation value is obtained by quantizing the INT8 value Construct a histogram and get: is the rounding function, is the scaling factor; each corresponds to a numerical interval, for The maximum value of is 1000. By traversing different candidate thresholds , select the threshold that minimizes the KL divergence as the optimal value , the symmetric quantization method is used to generate the quantized scaling factor and zero point, the zero point is taken as 0, the optimal scaling factor Finally, the engine is optimized by automatically selecting the optimal computing kernel, such as Winograd convolution, memory reuse, and layer fusion. The optimized engine is loaded and memory allocation and inference are performed, outputting the results of plate shape defect classification and defect composition ratio.

[0078] Taking 12,000 sets of samples from a 1450mm cold rolling production line in actual continuous operation as an example, the defect detection rate recall and false alarm rate (FAR) were used as indicators to evaluate the cold-rolled steel plate defect type prediction accuracy of the cold-rolled steel plate shape defect recognition model. The basic defect mean absolute error (BD-MAE) was used as an indicator to evaluate the cold-rolled steel plate defect type proportion prediction accuracy of the cold-rolled steel plate shape defect recognition model. The inference latency, throughput, and resource usage were used as evaluation indicators to evaluate the real-time performance of the cold-rolled steel plate shape defect recognition model.

[0079]

[0080]

[0081]

[0082] in, is the number of defect samples correctly identified; is the number of normal samples correctly identified; The number of samples that are incorrectly identified as defects as correct samples; is the number of defective samples that were missed; is the total number of samples, ; is the number of defect types, here we take ; For the In the sample The actual percentage of defects; The first prediction of the cold rolled plate shape defect recognition model In the sample The proportion of defects.

[0083] The real-time evaluation results are shown in Table 1. Inference latency is the time from data input to the cold-rolled plate defect recognition model to the output of the results. Throughput is the number of samples processed per unit time. Video memory usage is the amount of video memory used. INT8 quantization reduces latency to 35ms, a 3.4x improvement over FP32, and reduces video memory usage by 50%.

[0084] Table 1 Real-time evaluation results

[0085]

[0086] Common defect types at the production site are generally divided into five categories: edge waves, middle waves, warping, cracked edges, and wrinkles. The performance indicators of cold-rolled steel plate defects predicted by the cold-rolled plate shape defect recognition model of the present invention are shown in Table 2. It can be seen that the defect detection rate of the present invention reaches more than 95%, greatly reducing the risk of missed detection; at the same time, the defect false alarm rate is less than 3%, which can reduce the risk of downtime; the defect ratio analysis prediction error is less than 5%, which can assist decision makers in more accurately analyzing the causes of defects and provide a high-precision basis for process adjustments.

[0087] Table 2 Performance index results of the cold-rolled plate shape defect recognition model of the present invention

[0088]

[0089] like Figure 3 As can be seen, the comparison curves of actual defects and predicted defects are basically consistent, indicating that the present invention has a high degree of defect prediction accuracy. The present invention's cold-rolled plate shape defect recognition method based on multimodal data achieves high-precision, low-latency cold-rolled plate shape defect recognition by enhancing the ability to extract features from cold-rolled steel sheets and capture key information. It can not only identify cold-rolled plate shape defect types, but also quantitatively analyze the proportion of cold-rolled plate shape defect types, providing a quantitative basis for rolling parameter optimization and promoting the upgrading of intelligent manufacturing.

[0090] In one technical solution of the present invention, a computer-readable storage medium is further provided, storing a computer program, wherein the computer program enables a computer to execute the cold-rolled plate shape defect identification method based on multimodal data.

[0091] In one technical solution of the present invention, an electronic device is also provided, comprising: a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the cold-rolled plate shape defect identification method based on multimodal data is implemented.

[0092] In the embodiments disclosed herein, computer storage media may be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media may include electrical connections based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0094] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A cold-rolled plate shape defect recognition method based on multimodal data, characterized in that: The specific steps include: Step S1, collecting the shape information of the cold-rolled steel plate, wherein the shape information includes: shape image information, process timing parameter information, and motion timing parameter information of the cold-rolled steel plate; Step S2: constructing a cold-rolled plate shape defect recognition model consisting of a lightweight CNN module, a bidirectional LSTM module, a spatiotemporal cross attention layer, a defect classification module, and a defect ratio prediction module; The lightweight CNN module includes: a multi-scale depth-separable convolution module, a dynamic channel reorganization residual connection module, an adaptive receptive field module and a global average pooling layer; The multi-scale depth-separable convolution module captures the multi-scale features of cold-rolled flatness defects in the flatness image information through parallel depth-separable convolution at different scales, and splices the multi-scale features to obtain a fused feature map; The dynamic channel reorganization residual connection module is used to dynamically suppress the noise channel on the fusion feature map and enhance the cold-rolled plate shape defect characteristics; The adaptive receptive field module dynamically adjusts the receptive field size of the convolution kernel according to the fusion feature map enhanced by the cold-rolled plate shape defect feature, thereby enhancing the ability to capture cold-rolled plate shape defects of different sizes and obtaining image spatial features that fuse multi-scale receptive fields; The global average pooling layer compresses the global spatial information of the image spatial features of the multi-scale receptive field to obtain image features; Step S3: Input the plate image information of the cold-rolled steel plate into the lightweight CNN module of the cold-rolled plate shape defect recognition model to extract image features, input the process timing parameter information and motion timing parameter information of the cold-rolled steel plate into the bidirectional LSTM module of the cold-rolled plate shape defect recognition model to extract timing features, input the extracted image features and timing features into the spatiotemporal cross attention layer to extract fusion features, and input the extracted fusion features into the defect classification module and the defect proportion prediction module respectively to predict the defect type and defect proportion of the cold-rolled steel plate respectively.

2. The cold-rolled plate shape defect recognition method based on multimodal data according to claim 1, characterized in that: The plate shape image information of the cold-rolled steel plate is obtained by collecting plate shape radial force information and plate shape infrared thermal imaging images, encoding the plate shape radial force information into a 2D pseudo image using a Gram angle difference field, and mapping it to the red channel; mapping the corresponding plate shape infrared thermal imaging image to the green channel, setting the blue channel to 0, and obtaining three-channel plate shape image information.

3. The cold-rolled plate shape defect recognition method based on multimodal data according to claim 2, characterized in that: The process timing parameter information of the cold-rolled steel plate includes: rolling force, bending roll force and tension of the cold-rolled steel plate; the motion timing parameter information of the cold-rolled steel plate includes: running distance, speed and thickness of the cold-rolled steel plate.

4. The method for identifying cold-rolled plate shape defects based on multimodal data according to claim 1, characterized in that: The dynamic channel reorganization residual connection module is used to dynamically suppress the noise channel on the fusion feature map and enhance the cold-rolled plate shape defect feature. The specific process is: i. Calculate the global statistics of each channel based on the feature sub-graph on each channel of the fused feature map ,in, Indicates the height of the fused feature map, i express The index of represents the width of the fused feature map, j express The index of Represents the fusion feature map c The height of the channel is i , width is j The characteristic subgraph of ii. Generate the importance weight of each channel through the fully connected layer and normalize it by the Sigmoid function to obtain the channel attention weight ,in, represents the weight matrix of the fully connected layer, represents the bias of the fully connected layer; iii. According to channel attention weight Filter out important channels and dynamically prune the fused feature map: ,in, represents the fused feature map after dynamic pruning, represents the fused feature map, represents the indicator function, , represents the channel pruning threshold, Indicates the number of channels; iv. Perform residual connection on the fused feature map after dynamic pruning and expand the channels to the original number of channels.

5. The cold-rolled plate shape defect recognition method based on multimodal data according to claim 4, characterized in that: The process of acquiring the image spatial features of the fused multi-scale receptive field is as follows: the fused feature map enhanced with the cold-rolled plate shape defect features is subjected to global average pooling to obtain channel-level statistics, the channel-level statistics are mapped to the dilation rate weight of the corresponding receptive field through linear transformation, the convolution kernel of the receptive field is adjusted according to the dilation rate weight, and the convolution operation is performed, and the convolution results of different dilation rate weights are weighted to obtain the image spatial features after the fusion of the multi-scale receptive field.

6. The method for identifying cold-rolled plate shape defects based on multimodal data according to claim 5, characterized in that: The bidirectional LSTM module includes: a forward LSTM and a backward LSTM. The forward LSTM extracts the process timing characteristics of the cold-rolled steel plate based on the process timing parameter information of the cold-rolled steel plate, and the backward LSTM extracts the motion timing characteristics of the cold-rolled steel plate based on the motion timing parameter information of the cold-rolled steel plate. The process timing characteristics and motion timing characteristics of the cold-rolled steel plate are reduced in dimension along the time dimension through average pooling and spliced into the timing characteristics of the cold-rolled steel plate.

7. The method for identifying cold-rolled plate shape defects based on multimodal data according to claim 6, characterized in that: The extraction process of the fusion feature is as follows: in, represents the fusion feature, represents the fusion operation, represents the extracted image features, represents the features fused through the spatiotemporal cross attention mechanism, , represents the query vector generated by linear transformation of time series features, Represents the key vector generated by linear transformation of image features, Represents the value vector generated by linear transformation of image features, represents the attention mechanism scaling factor, Represents causal mask The resulting lower triangular matrix, , represents the current time step, Represents a historical time step.

8. The method for identifying cold-rolled plate shape defects based on multimodal data according to claim 7, characterized in that: The prediction process of the defect type of the cold-rolled steel plate is as follows: in, represents the predicted probability of defect type of cold-rolled steel sheet, 、 Represent the weights and biases of the fully connected layer in the defect classification module respectively; The prediction process of the defect ratio of the cold-rolled steel sheet is as follows: in, Indicates the defect ratio of cold-rolled steel sheets, S Indicates the defect type of cold rolled steel sheet, s express S The index of 、 They represent the weight matrix and bias of the fully connected layer in the defect ratio prediction module respectively.

9. A method for identifying cold-rolled flat shape defects based on multimodal data according to any one of claims 1 to 8, characterized in that: The cold-rolled flat shape defect recognition model needs to be trained until the loss function of the cold-rolled flat shape defect recognition model converges, thereby completing the training of the cold-rolled flat shape defect recognition model. The loss function of the cold-rolled plate shape defect recognition model for: in, represents the adaptive loss function for the classification task, , represents the category balance factor for the classification task loss, represents the adjustment factor of the classification task loss; represents the predicted probability of defect type of cold-rolled steel plate; represents the quantile loss function for the regression task, , 、 Represent the true value and predicted value of the cold rolled plate shape defect type, The quantile parameter representing the quantile loss for regression tasks, is the regularization coefficient, represents the training parameters in the cold-rolled plate shape defect recognition model, for of norm.

Images