Online production quality detection system based on multi-modal data

Abstract

The application discloses an online production quality detection system based on multi-modal data and belongs to the technical field of industrial detection. The physical semantic unification of multi-modal features is completed from a feature input layer, the problem that each modal feature only retains statistical semantics and has no physical semantic correlation in a traditional scheme can be avoided, a generated feature map with a physical label directly provides a physical semantic benchmark for a subsequent physical information neural network detection model, all features are aligned according to spatial coordinates, the same position of each modal feature corresponding to a workpiece is ensured when multi-modal fusion is performed subsequently, and fusion errors caused by spatial misplacement are avoided. Through three types of collaborative technical means, i.e., a physical property anchoring mechanism, a hidden layer physical alignment constraint and a physical consistency verification threshold, and the core optimization direction of a physical information neural network detection model embedded with a physical loss, cooperation is formed, and the semantic gap problem can be solved from the root through the physical label unified multi-modal semantic benchmark.

Description

Technical Field

[0001] This invention relates to the field of industrial testing technology, and more specifically to an online production quality testing system based on multimodal data. Background Technology

[0002] Industrial online production quality inspection is a technical method for real-time and automatic monitoring and evaluation of product quality during industrial production. Its core objective is to promptly detect defects, deviations, or anomalies during product manufacturing, thereby improving yield, reducing scrap rate, and optimizing process parameters.

[0003] While existing technical solutions based on multimodal data have made significant progress, in addition to common problems such as difficulty in data fusion, high cost, and poor real-time performance, there are still some less-mentioned but far-reaching atypical problems. Although multimodal systems integrate multiple signals such as vision, infrared, ultrasound, and vibration, different modalities represent the same defect in vastly different ways, forming a semantic gap of "seeing but not being able to explain." For example, a microcrack may appear as a linear dark line in an optical image, show a temperature gradient change in thermal imaging due to local thermal conduction anomalies, and be reflected or attenuated in ultrasonic echoes. Without a unified physical mechanism for modeling, it is difficult for the model to establish a consistent understanding across modalities. Even if the individual detection accuracy of each modality is high, "voting contradictions" may still occur during fusion decision-making, leading to missed detections or false alarms. The semantic gap between modalities leads to the problem of collaborative failure. Summary of the Invention

[0004] The purpose of this invention is to provide an online production quality inspection system based on multimodal data, which solves the technical problem of collaborative failure caused by the semantic gap between modalities in existing solutions.

[0005] The objective of this invention can be achieved through the following technical solutions: An online production quality inspection system based on multimodal data includes: Tensor format feature map processing module: Pre-calibration is performed and a modality-physical quantity mapping knowledge base is built. Defect regions are identified and processed in the visible light image of the visual modality of the workpiece to be inspected. Infrared feature vector, ultrasonic feature vector, and vibration feature vector of each defect are obtained and fitted and analyzed. Labels are dynamically bound according to the analysis results. Tensor format feature map is built based on the features bound to the physical labels. Physical law constraint filtering output module: Input the tensor format feature map with physical labels into the pre-built physical information neural network detection model. While embedding physical property alignment constraints in the hidden layer of the model, the stress-strain relationship of the material and the heat conduction equation are added as physical loss terms to the total loss function of the model. The single-mode prediction results that do not conform to physical laws are filtered through physical consistency verification. Finally, a unified quality detection conclusion that conforms to physical law constraints is output.

[0006] Furthermore, when performing pre-calibration and constructing a modal-physical quantity mapping knowledge base, M calibration stamping parts with known defects are pre-selected, where M is a positive integer. All calibration stamping parts are subjected to destructive testing to obtain the true physical parameters of the defects. The true physical parameters include stress concentration values, thermal conductivity anomaly coefficients, and deformation. Multimodal signals were collected from M calibration stamping parts. The multimodal signals included the original visual, infrared, ultrasonic, and vibration signals of the calibration stamping parts. The collected multimodal signals were preprocessed and feature extracted. The mapping relationship between the features of each modal signal and the real physical parameters was fitted to obtain the mapping coefficients and bias terms of different modes. After fitting is completed, all mapping coefficients, bias terms, and the calibrated intermodal homography matrix, ultrasonic channel spatial mapping table, and vibration sensitive frequency band library are uniformly stored in the modal-physical quantity mapping knowledge base.

[0007] Furthermore, the visible light image of the visual modality of the workpiece to be inspected is used to detect the defect region and extract features to obtain the two-dimensional pixel coordinates of the defect region and generate a defect ID. At the same time, the two-dimensional pixel coordinates are mapped to the actual physical coordinate space of the workpiece to be inspected to obtain the physical coordinate range.

[0008] Furthermore, for the physical coordinate range of each defect, the infrared-visual homography matrix in the modality-physical quantity mapping knowledge base is called, and the corresponding pixel coordinates of the physical coordinate range on the infrared image are calculated through the transformation formula; Furthermore, the average temperature rise rate and maximum temperature gradient corresponding to each defective pixel region are calculated and combined to obtain the infrared feature vector.

[0009] Furthermore, for the physical coordinate range of each defect, the ultrasonic channel spatial mapping table in the modal-physical quantity mapping knowledge base is called to filter out ultrasonic channels that meet the requirement that the physical interval corresponding to the channel overlaps with the defect coordinate range. The raw echoes of all selected channels are denoised, and the echo peak value, defect depth, and attenuation coefficient of each channel are obtained. The mean values ​​of echo peak value, defect depth, and attenuation coefficient for each channel are calculated and combined to obtain the ultrasonic feature vector.

[0010] Furthermore, for the physical coordinate range of each defect, the vibration sensitive frequency band library in the modal-physical quantity mapping knowledge base is called to match the simulation sensitive frequency band range of the mesh corresponding to that coordinate; The acquired vibration acceleration signal is subjected to a fast Fourier transform to obtain the spectrum. The natural frequency offset, damping ratio change rate, and resonant peak change rate within the simulated sensitive frequency band are extracted and combined to obtain the vibration feature vector.

[0011] Furthermore, when initializing the physical information neural network detection model and embedding physical constraints, the pre-trained physical information neural network detection model is loaded, the model input layer is adapted to tensor format, and physical attribute alignment constraints are first embedded in the 3rd and 5th hidden layers of the model, requiring all features of the same type of physical label to be bound, and the feature cosine similarity in the hidden layer is ≥0.8, so as to ensure the consistency of semantic features from the feature space level.

[0012] Furthermore, the constructed physical information neural network detection model adopts a three-stage architecture of feature extraction, physical constraint embedding, and multi-branch output, which contains a total of 12 structured network layers.

[0013] Furthermore, when performing forward inference and outputting preliminary single-modal predictions, the previously generated feature map is input into the initialized physical information neural network detection model, and preliminary prediction results for the four modalities are output through the feature extraction branch.

[0014] Furthermore, physical consistency verification is performed on the prediction results of each single mode. All single-mode prediction results that pass the physical verification are weighted and fused according to the fitting determination coefficients corresponding to each mode. After the fusion is completed, the final detection conclusion is generated.

[0015] Compared to existing solutions, the beneficial effects achieved by this invention are: This invention achieves physical semantic unification of multimodal features at the feature input layer, avoiding the problem in traditional schemes where each modality feature only retains statistical semantics without physical semantic association, thus fundamentally narrowing the semantic gap between different modalities. The generated feature map with physical labels directly provides a physical semantic benchmark for the subsequent physical information neural network detection model, allowing the physical loss constraint of the physical information neural network detection model to directly correspond to the feature dimension without additional semantic matching, significantly improving the training convergence speed and inference accuracy of the physical information neural network detection model. All features are aligned according to spatial coordinates, ensuring that each modality feature corresponds to the same position of the workpiece during subsequent multimodal fusion, avoiding fusion errors caused by spatial misalignment.

[0016] This invention employs three collaborative techniques—physical attribute anchoring mechanism, hidden layer physical alignment constraint, and physical consistency verification threshold—in conjunction with the core optimization direction of embedding physical loss into the physical information neural network detection model. This effectively avoids conventional solutions that merely apply physical constraints to the surface of the loss function, achieving end-to-end physical constraints from the feature input layer to the model's hidden layer. By unifying multimodal semantic benchmarks through physical labels, it addresses the semantic gap problem at its root. Furthermore, by actively filtering modal prediction results that do not conform to physical laws, it avoids voting conflicts from a mechanism perspective, effectively improving accuracy compared to conventional weighted voting schemes. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating the operation of the online production quality inspection system based on multimodal data according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, the present invention is an online production quality inspection system based on multimodal data, including a tensor format feature map processing and construction module and a physical law constraint screening and output module; Tensor format feature map processing module: Pre-calibration and construction of a modality-physical quantity mapping knowledge base are performed. Defect regions are identified and processed in the visible light image of the visual modality of the workpiece to be inspected, obtaining infrared, ultrasonic, and vibration feature vectors for each defect, which are then fitted and analyzed. Tags are dynamically bound based on the analysis results, and a tensor format feature map is constructed based on the bound physical tags. Specific steps include: When performing pre-calibration and building a modal-physical quantity mapping knowledge base, M calibration stamping parts with known defects are pre-selected, where M is a positive integer and can be 100. All calibration stamping parts are subjected to destructive testing to obtain the true physical parameters of the defects. The true physical parameters include stress concentration value, thermal conductivity anomaly coefficient, and deformation. Multimodal signals were collected from M calibration stamping parts. The multimodal signals included visual, infrared, ultrasonic, and vibration raw signals of the calibration stamping parts. The collected multimodal signals were preprocessed and feature extracted, and the mapping relationship between the features of each modal signal and the real physical parameters was fitted. The preprocessing and feature extraction of the multimodal signals are existing conventional technical solutions, and the specific implementation steps are not described here. Furthermore, the expression involved in the fitting is: ;in, The predicted value of the i-th type of physical parameter; These are the eigenvalues ​​of the i-th type of physical parameters corresponding to the m-th type of mode; These are the mapping coefficients and bias terms for the m-th mode, respectively; m takes values ​​from 1 to 4, corresponding to visual, infrared, ultrasonic, and vibration modes, respectively; i takes values ​​from 1 to 3, corresponding to three types of physical parameters: stress correlation, heat conduction correlation, and structural deformation correlation, respectively. After fitting, The modal homography matrix, ultrasonic channel spatial mapping table, and vibration sensitive frequency band library obtained from calibration are all stored in the modal-physical quantity mapping knowledge base. In the calibration process for obtaining the intermodal homography matrix, visual and infrared images of the calibration stamped part at the same detection position are acquired. The checkerboard corner pixel coordinates of the visual and infrared images are extracted using a sub-pixel corner detection algorithm. The coordinates of the nth corner point of the visual image are then labeled as follows: And mark the actual observed coordinates of the nth corner point of the infrared image as ; Substitute the corner coordinates of all visual images into the obtained homography matrix H to perform a coordinate transformation. The transformation formula is as follows: The predicted coordinates of each visual corner point mapped to the infrared image are calculated. , The infrared coordinate prediction value of the nth corner point output by the homography matrix H is given by H, which is a 3×3 homography transformation matrix obtained by solving through 30 sets of calibration images. For each set of corner points, the Euclidean distance between the predicted coordinates and the true values ​​is calculated as the mapping error for a single corner point, using the following formula: , The mapping error is for the nth corner point; The average mapping error is obtained by summing the single-corner errors of all corner points and dividing by the total number of points. When the average mapping error is ≤0.5 pixels, it is judged as qualified and the corresponding homography matrix H is stored in the knowledge base. Otherwise, it is judged as unqualified and recalibrated. When calibrating and obtaining the ultrasonic channel spatial mapping table, a standard ultrasonic calibration test block with the same material and thickness as the stamping part to be tested is used. 128 φ0.8mm transverse hole defects with known physical coordinates are pre-machined on the test block. The spatial position of each defect corresponds exactly to the detection center position of each channel of the 128-channel ultrasonic phased array. Each ultrasonic channel is activated sequentially to emit ultrasonic waves. The peak echo amplitude of each channel when it detects the corresponding transverse hole defect is recorded. When the peak echo amplitude is ≥60% of the full scale, the physical coordinate interval corresponding to the channel is recorded. Each channel covers a 5mm×5mm workpiece area, and finally a spatial mapping table of 128 channel numbers and physical coordinate intervals is formed. It should be explained that φ0.8mm is the smallest detectable defect size for ultrasonic testing commonly used in the industry, 60% full scale is the threshold for defect detection, a single channel covers a 5mm×5mm area, and 128 channels just completely cover the entire inspection range of the stamped parts. This mapping table establishes a direct correspondence between ultrasonic channels and the physical coordinates of the workpiece. Subsequently, the corresponding channels can be quickly filtered directly by the physical coordinates of the defects, eliminating the need to process all 128 channels of data and effectively improving the speed of ultrasonic feature extraction.

[0021] When calibrating and acquiring the vibration-sensitive frequency band library, finite element modal simulation is performed based on the 3D CAD model of the stamping part to be tested. The workpiece surface is divided into 10mm×10mm mesh elements, and a simulated impact load is applied to each mesh element. The simulated defect-free resonant center frequency is output for each mesh element. And the simulated sensitive frequency band range corresponding to 3dB bandwidth. ; Furthermore, several calibration samples were used for field verification, specifically 120 calibration samples. Each grid cell was subjected to an impact load and struck three times, with vibration acceleration responses collected simultaneously for 200 ms. A fast Fourier transform was performed on each collected response signal to obtain the spectrum, and the measured center frequency corresponding to the resonant peak was extracted. And the measured frequency band range corresponding to 3dB bandwidth. ; Calculate the average center frequency of all measured data for all grid cells. Average measured frequency range ; Measured center frequency for each grid Corresponding to the simulated defect-free resonant center frequency Perform linear regression fitting; the fitted expression is: , denoted as the fitted frequency; a is the frequency correction coefficient, and b is the frequency correction bias. And, the range of simulated sensitive frequency bands Substituting the upper and lower limits into the same correction formula, we obtain the corrected simulated sensitive frequency band. ; Calculate the relative error of the center frequency Frequency band overlap The expression involved is: ; len() is a function that calculates the length of a frequency band interval. like and If the frequency band of the grid is deemed acceptable, the physical coordinates of the grid are bound to the corrected simulated sensitive frequency band, forming the grid physical coordinates-sensitive frequency band range and storing it in the vibration sensitive frequency band library; Conversely, if the frequency band of the mesh is deemed unqualified, the input parameters of the finite element simulation for the mesh are adjusted and the simulation and fitting are repeated until the qualified conditions are met; the input parameters include, but are not limited to, material parameters such as elastic modulus and damping coefficient. It should be explained that the 10mm×10mm grid is the optimal granularity for balancing detection accuracy and computing power. 5% and 95% are the maximum allowable frequency band errors. If the error is exceeded, the simulation calibration will be re-performed. This vibration-sensitive frequency band library realizes the correspondence between different physical positions of the workpiece and the vibration-sensitive frequency bands. Subsequently, the vibration characteristics of the corresponding frequency band can be directly extracted based on the defect coordinates, avoiding noise interference from irrelevant frequency bands, which can effectively improve the accuracy of vibration characteristics in identifying defects.

[0022] It should be noted that a pre-defined mapping knowledge base for a specific workpiece can effectively reduce errors compared to general mapping rules, providing a precise benchmark reference for subsequent physical property anchoring and avoiding generalization errors across workpieces and materials.

[0023] The visible light image of the workpiece to be inspected is used to detect defect regions and extract features to obtain the two-dimensional pixel coordinates of the defect regions. The defect ID is generated, and the two-dimensional pixel coordinates are mapped to the actual physical coordinate space of the workpiece to be inspected, thus obtaining the physical coordinate range. The expression involved is: ; ; Where x and y are the pixel coordinates of the visual image; The width and height of the visual image pixels are 2448×2048 in this embodiment of the invention; The actual physical dimensions of the workpiece to be inspected are 1200mm × 800mm in this embodiment of the invention; X and Y are the actual physical coordinates after mapping. YOLOv8 is a deep learning network capable of quickly locating multiple objects in an image and identifying their categories. Through its anchor-free design, decoupled head, and efficient feature fusion, it successfully addresses the pain point of traditional models struggling to balance speed and accuracy. For AI vision projects that require rapid deployment, are resource-constrained, or have extremely high real-time requirements, YOLOv8 is often the preferred solution. Detecting defect regions and extracting features using YOLOv8 is a conventional technical solution, and the specific implementation steps are not detailed here.

[0024] For each defect's physical coordinate range, the infrared-visual homography matrix H from the modality-physical quantity mapping knowledge base is invoked, and the corresponding pixel coordinates on the infrared image are calculated using a transformation formula. ; In addition, calculate the average temperature rise rate corresponding to each defective pixel region. and maximum temperature gradient And combine them to obtain the infrared feature vector. The expression involved in calculating the average temperature rise rate is as follows: ; The average temperature of the defective pixel area in the last frame. The average temperature of the defective pixel area in the first frame; This represents the total collection time. The expression involved in calculating the maximum temperature gradient is: ; The temperature value of adjacent pixels within the defective pixel region; The physical spacing between adjacent pixels in the defective pixel region; The numbers of different pixels within the defective pixel region; For each defect's physical coordinate range, the ultrasonic channel spatial mapping table in the modal-physical quantity mapping knowledge base is called to filter out ultrasonic channels that meet the requirement of overlap between the physical interval corresponding to the channel and the defect coordinate range. For example, ultrasonic channels with an overlap rate of ≥80% between the physical interval corresponding to the channel and the defect coordinate range are usually selected, typically 6-12. If the number of channels selected is <3, re-acquisition is triggered. Three-layer wavelet soft thresholding denoising was performed on the raw echoes of all selected channels to eliminate industrial electromagnetic interference, and the echo peak value A, defect depth d, and attenuation coefficient of each channel were obtained. ; Among them, the echo peak value A is the maximum amplitude of the defect reflection signal; Defect depth d is obtained through the formula Calculations show that The speed at which ultrasonic waves propagate in the workpiece being inspected; This is the time difference between the emission of the ultrasonic wave and the reception of the reflected signal. Attenuation coefficient Through formula Calculations show that lg() represents the standard echo peak value of the defect-free region stored in the modal-physical quantity mapping knowledge base; lg() is a logarithmic function with base 10. Calculate the echo peak value A, defect depth d, and attenuation coefficient for each channel. The corresponding mean values ​​are combined to obtain the ultrasound feature vector. , These represent the average echo peak value, average defect depth, and average attenuation coefficient for all channels, respectively. For each defect's physical coordinate range, the vibration-sensitive frequency band library in the modal-physical quantity mapping knowledge base is invoked to match the simulation-sensitive frequency band range of the mesh corresponding to that coordinate. ; The collected vibration acceleration signal was subjected to a Fast Fourier Transform to obtain the spectrum, and the simulation sensitive frequency band range was extracted. Intrinsic frequency offset Damping ratio change rate , Resonance peak change rate By combining these elements, we obtain the vibration characteristic vector. ; Among them, natural frequency offset Through formula Calculations show that This is the measured peak resonant frequency; The defect-free standard resonant frequency stored in the knowledge base; Damping ratio change rate Through formula Calculations show that This is the measured damping ratio; The standard damping ratio is the one without defects. Resonance peak change rate Through formula Calculations show that This represents the measured peak resonance value. The peak value is the standard resonant peak value without defects; Substitute the infrared feature vector, ultrasonic feature vector, and vibration feature vector obtained from each defect treatment into the fitting expression to calculate the predicted physical parameter value corresponding to each feature. The predicted physical parameter value corresponding to each feature is compared with the corresponding physical parameter threshold in the modal-physical quantity mapping knowledge base; the physical parameter threshold can be determined based on the production design data and previous test data of the workpiece to be tested. If the predicted value of the physical parameter is less than the corresponding physical parameter threshold, the feature will be bound to the normal and defect-free label. Conversely, the features are bound to associated dimension labels; the associated dimension labels specifically include stress associated dimension labels, heat conduction associated dimension labels, and structural deformation associated dimension labels. Using the 3D CAD model of the workpiece to be inspected as a spatial reference, all features bound to physical labels are mapped to a 3D spatial grid according to physical coordinates, generating a tensor format feature map of dimension H×W×(C+P), which is directly used as the input of the subsequent physical information neural network detection model; where H and W are the height and width of the feature map, corresponding to the physical dimensions of the workpiece, C is the number of feature channels, containing all extracted features of 4 modalities, and P is the physical label dimension, corresponding to the encoded values ​​of 3 physical labels; It should be explained that the process of constructing the tensor format feature map involves the initialization of the three-dimensional spatial grid reference, tensor normalization preprocessing, spatial mapping and filling of labeled features, and splicing of feature channels and physical label channels. These are all existing conventional technical solutions, and the specific implementation steps will not be elaborated here.

[0025] In this embodiment of the invention, the physical semantics of multimodal features are unified from the feature input layer, which avoids the problem in traditional schemes where each modality feature only retains statistical semantics and has no physical semantic association, thus fundamentally narrowing the semantic gap between different modalities. The generated feature map with physical labels directly provides a physical semantic benchmark for the subsequent physical information neural network detection model, so that the physical loss constraint of the physical information neural network detection model can be directly mapped to the feature dimension without additional semantic matching, which greatly improves the training convergence speed and inference accuracy of the physical information neural network detection model. All features are aligned according to spatial coordinates to ensure that each modality feature corresponds to the same position of the workpiece during subsequent multimodal fusion, avoiding fusion errors caused by spatial misalignment.

[0026] The physical law constraint filtering output module: Tensor format feature maps with physical labels are input into a pre-built physical information neural network detection model. While embedding physical property alignment constraints in the model's hidden layers, the material stress-strain relationship and heat conduction equation are added as physical loss terms to the model's total loss function. Single-mode prediction results that do not conform to physical laws are filtered out through physical consistency verification. Finally, a unified quality detection conclusion that conforms to physical law constraints is output. Specific steps include: When initializing the physical information neural network detection model and embedding physical constraints, the pre-trained physical information neural network detection model is loaded. The model input layer is adapted to the tensor format. First, physical attribute alignment constraints are embedded in the 3rd and 5th hidden layers of the model, requiring all features of the same type of physical label to be bound. The cosine similarity of the features in the hidden layer is ≥0.8, which ensures the consistency of semantic features from the feature space level. The constructed physical information neural network detection model adopts a three-stage architecture of feature extraction, physical constraint embedding, and multi-branch output, which contains a total of 12 structured network layers, specifically: input layer, first convolutional feature extraction layer, first pooling dimensionality reduction layer, second convolutional feature extraction layer, third hidden layer physical attribute alignment constraint embedding, second pooling dimensionality reduction layer, third convolutional feature extraction layer, fifth hidden layer physical attribute alignment constraint embedding, global average pooling layer, multi-branch output layer, physical loss calculation layer, and total loss aggregation layer. All constraint parameters are derived from the previous calibration results to ensure that the physical rules of the model inference are completely matched with the actual workpiece characteristics, avoiding the adaptation error caused by general physical rules. The spatial dimension of the input tensor is 80×120, corresponding to the grid division of the preceding feature map. The channel dimension is 12, corresponding to 8 feature channels, specifically infrared temperature rise rate, temperature gradient, ultrasonic echo peak value, defect depth, attenuation coefficient, vibration frequency shift, damping ratio change rate, and resonant peak change rate, as well as 4 physical label thermally encoded channels, specifically stress correlation, thermal conduction correlation, structural deformation correlation, and defect-free. The input layer dimension is completely aligned with the output of the preceding feature map generation step, requiring no additional dimension transformation operation. The first convolutional feature extraction layer is a two-dimensional convolutional layer with a kernel size of 3×3, 64 kernels, a stride of 1, and ReLU activation function. This layer is responsible for extracting low-level spatial features from the original feature map, including basic features such as edge texture, local temperature anomaly regions, and vibration energy distribution. The first pooling dimensionality reduction layer is a max pooling layer with a pooling window size of 2×2 and a stride of 2. This layer downsamples the spatial dimension by 2 times, reducing computational complexity while expanding the receptive field, so that each feature point can cover a larger physical area. The second convolutional feature extraction layer is a two-dimensional convolutional layer with a kernel size of 3×3, 128 kernels, a stride of 1, and ReLU activation function. This layer is responsible for extracting intermediate semantic features, including defect morphology patterns, multimodal feature association patterns, and physical parameter distribution patterns. The third hidden layer is a physical attribute alignment constraint embedding layer, which is a custom physical alignment constraint layer. This layer adds a physical attribute alignment constraint mechanism after the regular convolution operation. Specifically, it performs a convolution operation with 128 3×3 convolution kernels. It reads the last four physical label channels of the input feature map and divides all features into four groups according to the labels: stress correlation group, heat conduction correlation group, structural deformation correlation group, and defect-free group. For all features in each group, it calculates the cosine similarity between all pairs of hidden layer output vectors. The values ​​range from [-1, 1], where p and q are the indices of the hidden layer output vector; an alignment constraint loss is constructed. N is the total number of feature pairs in the same group; For the set of feature pairs in the same group; construct the alignment constraint loss. Add to the model's total loss function; It should be noted that this constraint forces heterogeneous features bound to the same physical label to align in the hidden layer representation space, which can eliminate the multimodal semantic gap and provide a unified semantic basis for subsequent physical loss calculation.

[0027] The second pooling dimensionality reduction layer is a max pooling layer with a pooling window size of 2×2 and a stride of 2. This layer further reduces the spatial dimension and expands the receptive field to a physical region of 40mm×40mm. The third convolutional feature extraction layer is a two-dimensional convolutional layer with a kernel size of 3×3, 256 kernels, a stride of 1, and ReLU activation function. This layer is responsible for extracting high-level semantic features, including defect type discrimination features, physical parameter prediction features, and quality level classification features. The fifth hidden layer embeds physical property alignment constraints. The layer type is a custom physical alignment constraint layer, and its implementation logic is completely identical to that of the third hidden layer, while incorporating the constructed alignment constraint loss. Add to the model's total loss function; The global average pooling layer is a global average pooling layer that averages all feature values ​​in a spatial dimension of 20×30. This layer compresses the spatial features into a global feature vector, with each feature channel corresponding to a scalar value. The multi-branch output layer contains four parallel output branches: Branch 1 - Defect Classification Branch: The layer type is a fully connected layer with Softmax activation, with an input dimension of 256 and an output dimension of 5. It corresponds to 5 defect types: cracks, dents, hidden damage, thermal stress concentration, and no defects. The output is a probability distribution vector. The category with the highest probability is taken as the final classification result. Branch 2 - Stress Prediction Branch: Layer type is fully connected layer + ReLU activation, input dimension 256, output dimension 1, output is the predicted value of stress concentration factor. The value range is [0, 1000]. Branch 3 - Heat Conduction Prediction Branch: The layer type is a fully connected layer with ReLU activation, input dimension 256, output dimension 1, and outputs the predicted value of heat conduction anomaly rate. The value range is [0, 2], with 1 as the baseline value; Branch 4 - Deformation Prediction Branch: The layer type is a fully connected layer with ReLU activation, input dimension 256, output dimension 1, output deformation prediction value. The value range is [-5, 5]; The physical loss calculation layer is a custom loss calculation layer. This layer does not participate in forward inference; it only calculates the physical constraint loss during the training phase. Specifically, it receives the predicted values ​​from branches 2, 3, and 4, and loads the physical constant E from the knowledge base. k ; Calculate stress-strain residuals Thermal conductivity residual and construct physical loss , Different physical weights, ,and ; It should be noted that the prediction results of the physical loss forced model conform to the basic physical laws of materials, and can avoid prediction results that violate common sense physics, such as large deformation and low stress.

[0028] The total loss aggregation layer, which is a loss aggregation layer, sums the classification loss, physical alignment loss, and physical loss by weight, as shown in the formula. ;in, This is the total loss function; This is the weight for the physical alignment loss, with a default value of 0.2. This is the weight for physical loss, with a default value of 0.3; In addition, the Adam optimizer was used during the training phase, with an initial learning rate of 0.001, which decayed to 0.5 times the original rate every 100 epochs. The total number of training epochs was 500. The early stopping strategy was to stop training if the validation set loss did not decrease for 20 consecutive epochs. In this embodiment of the invention, low-level, mid-level, and high-level features are extracted stepwise through three convolutional layers, with the number of feature channels increasing from 12 to 256, achieving a hierarchical progression in feature expression capabilities, which can effectively improve the accuracy of defect identification. Physical attribute alignment constraints are doubly embedded in the 3rd and 5th hidden layers, achieving dual embedding of physical constraints, which can effectively improve the accuracy of similarity between label features and multimodal feature fusion. The multi-branch output architecture can simultaneously output classification results and predicted physical parameters, providing complete data support for subsequent physical consistency verification. All detection results can be obtained in a single inference, which can effectively improve inference efficiency compared to serial prediction. The physical loss calculation layer forces the model prediction to conform to the basic physical laws of materials, which can effectively improve the physical consistency compliance rate and avoid false detection results that violate physical common sense. When performing forward inference and outputting the preliminary prediction for a single mode, the feature map generated in the previous step is input into the initialized physical information neural network detection model. The preliminary prediction results for four modes are output through the feature extraction branch. The prediction result for each mode includes: defect ID, corresponding physical coordinate range, defect type, and predicted physical parameter value. All preliminary prediction results for the four modes are temporarily stored and enter the next verification stage. The predicted physical parameter values ​​specifically include stress, heat conduction anomaly rate, and deformation. When performing physical consistency verification on the prediction results for each single mode, the residuals are calculated by substituting them sequentially into the physical constraint equations. The verification formula is as follows: ;in, The value represents the residual; max() is the function to get the maximum value. like If the result does not conform to physical laws, it is determined that the single-mode prediction result does not conform to physical laws and is directly filtered and discarded; otherwise, it is determined to be a valid result and proceeds to the next fusion stage. For all single-mode prediction results that pass physical validation, the fitting determination coefficients for each mode are used as the basis for the predictions. Weighted fusion, the fusion formula is: ;in, The final fused physical parameter prediction values; M is the total number of modes that passed the verification; Here are the predicted physical parameters for the m-th mode; m is the index of the mode. This is the determination coefficient for the m-th mode, with a value ranging from 0 to 1. The higher the value, the higher the prediction reliability of that modality, and the greater the weight. In this embodiment, visual, infrared, ultrasonic, and vibration modalities are used. The values ​​were 0.92, 0.87, 0.95, and 0.83 respectively, with the confidence levels from highest to lowest being ultrasound > vision > infrared > vibration. The final fusion physical parameter prediction values ​​are identified and analyzed to generate the final detection conclusion, which includes the ID, location, type, physical parameter value, and pass / fail judgment of all defects. Among these steps, the final fused physical parameter prediction values ​​are identified and analyzed. First, the physical parameter type corresponding to the final fused physical parameter prediction value is identified. The identification rule is: based on the modal combinations that passed physical verification in the previous steps, the type of physical parameter is determined. Which category of physical parameters does it belong to? Based on the identified physical parameter type, By comparing with the corresponding threshold, a quality determination result for a single physical parameter is generated: When multiple physical parameter predictions exist for the same defect area, a comprehensive judgment is made according to their priority order, with the priority rule being: excessive stress > abnormal heat conduction > excessive deformation. The specific judgment logic is as follows: Comprehensive Judgment Rule 1 - Stress Priority Principle: If any physical parameter is judged to be unqualified, and that parameter is the stress concentration factor, then the final quality grade is judged to be unqualified, the defect type is marked as stress exceeding standard defect, and the severity level of the defect is taken as the severity level of the stress judgment; Comprehensive Judgment Rule 2 - Principle of Multiple Defects: If multiple physical parameters are judged to be unqualified at the same time, the final quality grade is judged to be unqualified, the defect type is marked as a composite defect, the severity level of the defect is taken as the highest severity level among the judgment results of each parameter, and all unqualified items are listed in detail in the inspection conclusion; Comprehensive Judgment Rule 3 - Early Warning Priority Principle: If all physical parameters are judged to be qualified, but any parameter is judged to be qualified but needs to be monitored, then the final quality level is judged to be qualified but needs to be monitored, the defect type is marked as early warning defect, and the defect severity level is the highest severity level among the early warning parameters; Comprehensive Judgment Rule 4 - Full Compliance Principle: If all physical parameters are judged to be qualified and there are no warning items, the final quality level is judged to be qualified, the defect type is marked as no defect or minor defect, and the defect severity level is none; The final test results are simultaneously output to the production line PLC system, triggering the sorting of non-conforming parts and providing a simultaneous early warning.

[0029] In this embodiment of the invention, three collaborative techniques—physical attribute anchoring mechanism, hidden layer physical alignment constraint, and physical consistency verification threshold—are combined with the core optimization direction of embedding physical loss into the physical information neural network detection model. This effectively avoids conventional solutions that merely apply physical constraints to the surface of the loss function, achieving full-link physical constraints from the feature input layer to the model's hidden layer. By unifying multimodal semantic benchmarks through physical labels, the semantic gap problem can be solved at its root. By actively filtering modal prediction results that do not conform to physical laws, voting conflicts are avoided from a mechanism perspective, effectively improving accuracy compared to conventional weighted voting schemes.

[0030] In the several embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0031] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0032] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0033] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An online production quality inspection system based on multimodal data, characterized in that, include: Tensor format feature map processing module: Pre-calibration is performed and a modality-physical quantity mapping knowledge base is built. Defect regions are identified and processed in the visible light image of the visual modality of the workpiece to be inspected. Infrared feature vector, ultrasonic feature vector, and vibration feature vector of each defect are obtained and fitted and analyzed. Labels are dynamically bound according to the analysis results. Tensor format feature map is built based on the features bound to the physical labels. Physical law constraint filtering output module: Input the tensor format feature map with physical labels into the pre-built physical information neural network detection model. While embedding physical property alignment constraints in the hidden layer of the model, the stress-strain relationship of the material and the heat conduction equation are added as physical loss terms to the total loss function of the model. The single-mode prediction results that do not conform to physical laws are filtered through physical consistency verification. Finally, a unified quality detection conclusion that conforms to physical law constraints is output.

2. The online production quality inspection system based on multimodal data according to claim 1, characterized in that, When performing pre-calibration and building a modal-physical quantity mapping knowledge base, M calibration stamping parts with known defects are pre-selected, where M is a positive integer. All calibration stamping parts are subjected to destructive testing to obtain the true physical parameters of the defects. The true physical parameters include stress concentration values, thermal conductivity anomaly coefficients, and deformation. Multimodal signals were collected from M calibration stamping parts. The multimodal signals included the original visual, infrared, ultrasonic, and vibration signals of the calibration stamping parts. The collected multimodal signals were preprocessed and feature extracted. The mapping relationship between the features of each modal signal and the real physical parameters was fitted to obtain the mapping coefficients and bias terms of different modes. After fitting is completed, all mapping coefficients, bias terms, and the calibrated intermodal homography matrix, ultrasonic channel spatial mapping table, and vibration sensitive frequency band library are uniformly stored in the modal-physical quantity mapping knowledge base.

3. The online production quality inspection system based on multimodal data according to claim 2, characterized in that, The visible light image of the visual modality of the workpiece to be inspected is used to detect the defect area and extract features to obtain the two-dimensional pixel coordinates of the defect area and generate a defect ID. At the same time, the two-dimensional pixel coordinates are mapped to the actual physical coordinate space of the workpiece to be inspected to obtain the physical coordinate range.

4. The online production quality inspection system based on multimodal data according to claim 3, characterized in that, For each defect's physical coordinate range, the infrared-visual homography matrix in the modality-physical quantity mapping knowledge base is called, and the corresponding pixel coordinates of the physical coordinate range on the infrared image are calculated through the transformation formula. Furthermore, the average temperature rise rate and maximum temperature gradient corresponding to each defective pixel region are calculated and combined to obtain the infrared feature vector.

5. The online production quality inspection system based on multimodal data according to claim 4, characterized in that, For each defect's physical coordinate range, the ultrasonic channel spatial mapping table in the modal-physical quantity mapping knowledge base is called to filter out ultrasonic channels that meet the requirement that the physical interval corresponding to the channel overlaps with the defect's coordinate range. The raw echoes of all selected channels are denoised, and the echo peak value, defect depth, and attenuation coefficient of each channel are obtained. The mean values ​​of echo peak value, defect depth, and attenuation coefficient for each channel are calculated and combined to obtain the ultrasonic feature vector.

6. The online production quality inspection system based on multimodal data according to claim 5, characterized in that, For each defect's physical coordinate range, the vibration sensitive frequency band library in the modal-physical quantity mapping knowledge base is called to match the simulation sensitive frequency band range of the mesh corresponding to that coordinate; The acquired vibration acceleration signal is subjected to a fast Fourier transform to obtain the spectrum. The natural frequency offset, damping ratio change rate, and resonant peak change rate within the simulated sensitive frequency band are extracted and combined to obtain the vibration feature vector.

7. The online production quality inspection system based on multimodal data according to claim 1, characterized in that, When initializing the physical information neural network detection model and embedding physical constraints, the pre-trained physical information neural network detection model is loaded. The model input layer is adapted to the tensor format. First, physical attribute alignment constraints are embedded in the 3rd and 5th hidden layers of the model, requiring all features of the same type of physical label to be bound. The cosine similarity of the features in the hidden layer is ≥0.8, which ensures the consistency of semantic features from the feature space level.

8. The online production quality inspection system based on multimodal data according to claim 7, characterized in that, The constructed physical information neural network detection model adopts a three-stage architecture of feature extraction, physical constraint embedding, and multi-branch output, which contains a total of 12 structured network layers.

9. The online production quality inspection system based on multimodal data according to claim 7, characterized in that, When performing forward inference and outputting preliminary single-modal predictions, the feature map generated in the preceding step is input into the initialized physical information neural network detection model, and the preliminary prediction results of the four modalities are output through the feature extraction branch respectively.

10. The online production quality inspection system based on multimodal data according to claim 9, characterized in that, Physical consistency verification is performed on the prediction results of each single mode. All single mode prediction results that pass the physical verification are weighted and fused according to the fitting determination coefficients corresponding to each mode. After fusion, the final detection conclusion is generated.

Images