Industrial quality inspection method based on multi-mode and few-sample learning

Abstract

The invention provides an industrial quality inspection method based on multi-mode and few-sample learning. The method comprises the following steps: acquiring at least two types of modal samples such as images, geometry and process signals of a reference production line and a new production line, constructing a cross-modal pairing training set according to a workpiece identifier, and training a SigLIP2 type contrast learning model comprising modal encoders and a unified projection layer to obtain 512-dimensional unified representation; constructing a category prototype library based on the standard production line labeling samples; forming production line calibration transformation by using the difference between a small number of labeled normal samples of the new production line and the reference normal prototype, calibrating the prototype library, and updating in combination with the labeled samples of the new production line to obtain a new production line prototype library; and calculating the cosine similarity between the calibrated representation of the sample to be detected and each prototype, and determining a threshold output defect category or a normal category according to the calibrated normal sample similarity quantile, thereby realizing the few-sample cross-production-line industrial quality inspection.

Description

Industrial quality inspection methods based on multimodal and few-shot learning Technical Field

[0001] This invention relates to the fields of industrial product quality inspection and multimodal machine learning, and particularly to an industrial quality inspection method based on multimodal and few-shot learning. Background Technology

[0002] Quality inspection in industrial manufacturing typically requires a comprehensive assessment of workpiece appearance, dimensions, and process status. Common data sources include camera images, 3D geometric point clouds, and equipment process signals. Because defect morphologies are diverse and related to process fluctuations, a single modality is often insufficient to cover all anomaly clues. Furthermore, production line changes, equipment calibration, and process parameter adjustments introduce data distribution differences, necessitating consistent judgment criteria and feasible deployment processes for the quality inspection system under multi-source information conditions.

[0003] In existing technical solutions, industrial quality inspection often employs rule-based visual inspection, supervised learning-based image classification / detection networks, or fusion modeling of images with 3D and signal information. Fusion methods include feature-level fusion, decision-level fusion, and training models separately for different modalities before ensemble judgment. Some solutions also use metric learning or prototype / nearest neighbor classification, using class center vectors as the judgment criterion. In cross-scenario applications, some methods use transfer learning, domain adaptation, or contrastive learning for representation learning to improve usability and reusability under different acquisition conditions.

[0004] The above-mentioned solutions still face challenges in adapting to production line migration and multimodal collaboration: First, new production lines often only have a small number of labeled samples, and existing methods often rely on sufficient labels for retraining or fine-tuning; second, there is a lack of a unified alignment mechanism between different modalities based on the "same workpiece," making it difficult to maintain consistent reference after production line changes; third, the representational shift caused by production line differences makes it difficult to directly reuse judgment strategies based on fixed thresholds or fixed category centers, requiring recalibration and updating of the judgment benchmark.

[0005] Therefore, an industrial quality inspection method that can overcome the shortcomings of the existing technology is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an industrial quality inspection method based on multimodal and few-sample learning. The core technical problem to be solved by this application is: under the condition that a new production line has only a small number of labeled samples and there are differences between production lines, how to map multimodal information such as images, geometry and process signals to an alignable unified representation space, and complete the stable output of workpiece defect category or normal category through transferable calibration and prototype judgment mechanism.

[0007] An industrial quality inspection method based on multimodal and few-shot learning according to embodiments of the present invention includes:

[0008] S1. Obtain the multimodal sample set of workpieces from the benchmark production line, the annotation sample set of the benchmark production line, the annotation sample set of the new production line, and the multimodal sample of the workpiece to be inspected. Organize the multimodal sample set of workpieces from the benchmark production line into a cross-modal paired training sample set according to the workpiece identification.

[0009] S2. Train the SigLIP2 contrastive learning multimodal base model based on the cross-modal paired training sample set. The SigLIP2 contrastive learning multimodal base model consists of an image encoder, a geometric encoder, a process signal encoder, and a unified projection layer. Generate a unified representation vector for each workpiece multimodal sample. Construct cross-modal positive pairing based on the same workpiece identifier and constrain the unified representation vectors of the positive pairing to be close. Construct negative pairing based on different workpiece identifiers and constrain the unified representation vectors of the negative pairing to be separated, thus obtaining a unified representation model.

[0010] S3. Input the baseline production line labeled sample set into the unified representation model to obtain the baseline production line unified representation vector set. Aggregate the baseline production line unified representation vector set according to defect category and normal category to obtain the baseline production line prototype library.

[0011] S4. Input the new production line labeled sample set into the unified representation model to obtain the new production line unified representation vector set. Aggregate the normal samples of the new production line unified representation vector set to obtain the new production line normal prototype. Calculate the difference between the new production line normal prototype and the benchmark production line normal prototype in the benchmark production line prototype library to obtain the production line benchmark difference. Based on the production line benchmark difference, form a production line calibration transformation. Apply the production line calibration transformation to the benchmark production line prototype library and perform category aggregation and update in combination with the new production line unified representation vector set to obtain the new production line prototype library. Apply the production line calibration transformation to the normal samples of the new production line unified representation vector set to obtain the calibration normal vector set.

[0012] S5. Input the multimodal sample of the workpiece to be inspected into the unified representation model to obtain the unified representation vector to be inspected, and apply the production line calibration transformation to the unified representation vector to be inspected to obtain the calibrated vector to be inspected. Calculate the similarity between the calibrated vector to be inspected and the new production line prototype library to obtain the category matching score.

[0013] S6. Determine the judgment threshold based on the similarity statistics between the calibration normal vector set and the normal prototype of the new production line. When the maximum value of the category matching score meets the judgment threshold, output the defect category of the defect prototype corresponding to the maximum value. When the maximum value of the category matching score does not meet the judgment threshold, output the normal category and obtain the quality inspection judgment result.

[0014] Optionally, S1 includes:

[0015] Multimodal samples of workpieces are collected on the benchmark production line according to the workpiece identification. The multimodal samples of workpieces include at least two types of image modal input, geometric modal input, and process signal modal input. The workpiece identification is bound to each multimodal sample of workpieces to form a multimodal sample set of workpieces on the benchmark production line.

[0016] Samples with defect or normal category labels are selected from the multimodal sample set of workpieces from the benchmark production line to form the benchmark production line labeled sample set. In the new production line, workpiece identification rules consistent with those of the benchmark production line are used to collect and label samples to form the new production line labeled sample set. At the same time, unlabeled multimodal samples of workpieces to be inspected are collected.

[0017] The multimodal sample set of the benchmark production line workpieces is grouped by workpiece identifier as index. Different modal inputs under the same workpiece identifier are used to generate cross-modal paired training samples according to modal combination relationship. All cross-modal paired training samples are then collected into a cross-modal paired training sample set.

[0018] Optionally, S2 includes:

[0019] Using a cross-modal paired training sample set as input, a SigLIP2-style contrastive learning multimodal basic model is established. The SigLIP2-style contrastive learning multimodal basic model includes an image encoder, a geometric encoder, a process signal encoder, and a unified projection layer, with the unified projection layer set as the common output interface of the three types of encoders.

[0020] The image modal input of each workpiece multimodal sample in the cross-modal pairing training sample set is fed into the image encoder. The image intermediate feature map is formed through multiple layers of two-dimensional convolutional layers, nonlinear activation layers and downsampling layers, and then globally converged to obtain a 1024-dimensional image feature vector.

[0021] The geometric modal input of the multimodal sample of the workpiece is sent to the geometric encoder. Local edge features are constructed based on the point neighborhood and transformed and aggregated layer by layer by the neurons of the multilayer perceptron to obtain a 1024-dimensional geometric feature vector.

[0022] The process signal modal input of the multimodal sample of the workpiece is sent to the process signal encoder. The multi-scale time pattern is extracted by the temporal convolution module composed of one-dimensional convolutional neurons and then globally converged to obtain a 1024-dimensional process signal feature vector.

[0023] Image feature vectors, geometric feature vectors, and process signal feature vectors are respectively input into a unified projection layer composed of shared fully connected neurons, and uniformly mapped into a 512-dimensional unified representation vector. The unified representation vector is then normalized to serve as the input for cross-modal similarity calculation.

[0024] Within the same training batch, cross-modal positive pairings are constructed based on the workpiece identifier. The cross-modal positive pairings are obtained by combining the unified representation vectors generated by the same workpiece identifier in different modes. Negative pairings are constructed based on different workpiece identifiers. The negative pairings are obtained by combining the unified representation vectors generated by different workpiece identifiers in pairs.

[0025] The similarity between cross-modal positive and negative pairings is calculated, and the model parameters are iteratively updated using a SigLIP2-style contrastive learning objective. This ensures that the similarity of cross-modal positive pairings satisfies the training constraint that the similarity of negative pairings is greater than that of negative pairings, thus obtaining a unified representation model.

[0026] Optionally, S3 includes:

[0027] The baseline production line annotation sample set is input into the unified representation model one by one. The image modal input, geometric modal input, and process signal modal input contained in each sample are respectively processed by the corresponding encoder and the unified projection layer to output a unified representation vector of 512 dimensions, forming a unified representation vector set of the baseline production line.

[0028] When the same sample contains multiple modal inputs, the multiple unified representation vectors corresponding to the sample are aggregated to obtain the sample unified representation vector, and the sample unified representation vector is replaced and written into the benchmark production line unified representation vector set.

[0029] Based on the defect category and normal category labels of the baseline production line labeled sample set, the unified representation vector set of the baseline production line is grouped by category, and the vectors in each category group are aggregated to obtain the category prototype of that category.

[0030] The category prototypes of each defect category and the category prototypes of the normal category are combined to form a baseline production line prototype library.

[0031] Optionally, S4 includes:

[0032] The new production line labeled sample set is input into the unified representation model one by one. The image modal input, geometric modal input, and process signal modal input contained in each sample are respectively processed by the corresponding encoder and the unified projection layer to output a unified representation vector of 512 dimensions, forming a unified representation vector set for the new production line.

[0033] When the same sample contains multiple modal inputs, the multiple unified representation vectors corresponding to the sample are aggregated to obtain the sample unified representation vector, and the sample unified representation vector is replaced and written into the new production line unified representation vector set.

[0034] Based on the normal category labels of the new production line labeled sample set, the vectors corresponding to normal samples are selected from the unified representation vector set of the new production line, and the vectors corresponding to the normal samples are aggregated to obtain the normal prototype of the new production line.

[0035] Read the normal prototype of the benchmark production line from the benchmark production line prototype library, calculate the difference between the new normal prototype of the production line and the normal prototype of the benchmark production line to obtain the benchmark difference of the production line, and define the benchmark difference of the production line as the production line calibration transformation acting on the 512-dimensional unified representation vector space.

[0036] The calibration prototype library is obtained by applying the production line calibration transformation to each category of prototypes in the baseline production line prototype library.

[0037] Based on the defect category and normal category labels of the new production line labeled sample set, the unified characterization vector set of the new production line is grouped by category. Vector aggregation is performed on the vectors in each category group to obtain the category aggregation result. The calibration prototype library is updated by category aggregation based on the category aggregation result to obtain the new production line prototype library.

[0038] The calibration normal vector set is obtained by applying the production line calibration transformation to the normal sample corresponding vector of the new production line unified characterization vector set.

[0039] Optionally, S5 includes:

[0040] The multimodal samples of the workpiece to be inspected are represented by a unified model based on the sample input. The image modal input, geometric modal input, and process signal modal input contained in the multimodal samples of the workpiece to be inspected are respectively encoded by the corresponding encoder and the unified projection layer to output a unified representation vector of 512 dimensions. When the multimodal samples of the workpiece to be inspected contain multiple types of modal inputs, the multiple unified representation vectors are aggregated to obtain the unified representation vector to be inspected.

[0041] The calibration vector to be tested is obtained by applying a production line calibration transformation to the unified characterization vector to be tested.

[0042] The similarity between the calibration vector to be tested and each category of prototype in the new production line prototype library is calculated, and cosine similarity is used as the similarity measure.

[0043] The similarity scores corresponding to the prototypes of each category are aggregated to form a category matching score.

[0044] Optionally, step S6 includes:

[0045] The cosine similarity of each calibration normal vector set with the normal prototype of the new production line is calculated to obtain the normal similarity sequence.

[0046] Statistical analysis was performed on normal similarity sequences, and the 5th percentile value of normal similarity sequences was determined as the judgment threshold.

[0047] The defect matching score is composed of scores that correspond to the defect category in the category matching score, and the maximum value of the defect matching score is taken as the maximum defect score.

[0048] When the maximum defect score is not less than the judgment threshold, the defect category of the defect prototype corresponding to the maximum defect score is output as the quality inspection judgment result. When the maximum defect score is less than the judgment threshold, the normal category is output as the quality inspection judgment result.

[0049] The beneficial effects of this invention are:

[0050] (1) This proposal proposes an improved multimodal unified representation learning method. By constructing cross-modal positive and negative pairings with workpiece identifiers as the link, SigLIP2-style contrastive learning is used in the same training batch to constrain the representations of different modalities for the same workpiece to be close in a unified space and the representations of different workpieces to be separated. Compared with the existing approach of simply splicing or training and then fusing multimodalities, this scheme introduces an image encoder, a geometric encoder, a process signal encoder and a shared unified projection layer in the model structure, so that the three types of modalities are mapped to a normalized 512-dimensional unified representation vector, thereby providing a consistent semantic coordinate system for subsequent cross-modal similarity calculation, category prototype aggregation and cross-production line calibration; at the same time, vector aggregation is performed on the multimodal output of the same sample to avoid the inconsistent impact of single modality missing or noise on the final representation, and to meet the acquisition constraint of "at least two types of modalities are available" in industrial sites.

[0051] (2) This proposal proposes a novel production line calibration and prototype migration mechanism. Addressing scenarios where new production lines have only a small number of labeled samples and exhibit production line differences, the mechanism aggregates normal labeled samples from the new production line to obtain a normal prototype. The difference between this prototype and the normal prototype of the baseline production line is then calculated to obtain the production line baseline difference. This difference is defined as the production line calibration transformation applied to a unified representation space. Unlike existing schemes that rely on whole-model retraining, fine-tuning, or empirical normalization at the input, this mechanism characterizes the production line domain offset within a unified representation space using "normal class alignment." This allows the calibration transformation to simultaneously apply to prototypes of various defect categories and the representation of samples to be inspected, reducing dependence on the scale of defect class labels. Based on this, the calibrated baseline prototype library is updated by combining a small number of labeled samples from the new production line, enabling rapid construction and completion of the new production line prototype library. This ensures that the category determination criteria can be consistently migrated as the production line changes.

[0052] (3) This proposal puts forward a quality inspection judgment method based on prototype similarity and quantile threshold statistics. After the multimodal samples of the workpiece to be inspected are characterized by a unified representation model, a production line calibration transformation is applied. The cosine similarity with the new production line prototype library is calculated class by class to form a category matching score. The judgment threshold (take the 5th percentile value) is determined by the similarity distribution between the calibration normal vector set and the normal prototype of the new production line. Based on this, the corresponding defect category is output when the "maximum defect matching score reaches the threshold", otherwise the normal category is output. This judgment link is different from the method of relying on a fixed threshold or relying on a large number of defect samples to fit the classification boundary. It binds the threshold determination with the statistics of normal data of the new production line, so that the threshold can be adjusted synchronously with the production line calibration and prototype update. Thus, under the condition that the new production line label is scarce and the production line difference exists, it provides a reusable judgment basis for multimodal quality inspection output. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 is a flowchart of an industrial quality inspection method based on multimodal and few-shot learning proposed in this invention;

[0055] Figure 2 is a flowchart of the process for obtaining and organizing cross-modal paired training sample sets for an industrial quality inspection method based on multimodal and few-shot learning proposed in this invention.

[0056] Figure 3 is a flowchart of the training SigLIP2 contrastive learning multimodal basic model for an industrial quality inspection method based on multimodal and few-shot learning proposed in this invention.

[0057] Figure 4 is a flowchart of the construction of a benchmark production line prototype library for an industrial quality inspection method based on multimodal and few-shot learning proposed in this invention.

[0058] Figure 5 is a flowchart of the construction of a new production line prototype library and production line calibration for an industrial quality inspection method based on multimodal and few-shot learning proposed in this invention.

[0059] Figure 6 is a flowchart of the workpiece category matching score calculation for an industrial quality inspection method based on multimodal and few-shot learning proposed in this invention.

[0060] Figure 7 is a flowchart of the determination of the judgment threshold and the output of the quality inspection judgment result of the industrial quality inspection method based on multimodal and few-sample learning proposed in this invention.

[0061] Figure 8 is a schematic diagram of production line calibration in the unified representation space of the industrial quality inspection method based on multimodal and few-sample learning proposed in this invention. The figure shows the cross-production line migration process in the unified representation space: the left side is the normal prototype N0 and several defect prototypes of the benchmark production line; the right side is the normal sample cloud and the new normal prototype N1 of the new production line. The production line calibration transformation is constructed by calculating the difference Δ between N1 and N0, and the benchmark prototype library is calibrated as a whole to the new production line reference system. After applying Δ to the sample vector to be inspected, the similarity with the calibrated defect prototype is calculated and the category is determined. This mechanism uses normal class alignment to characterize the differences between production lines, so that a small number of new production line annotations can be reused and the prototype library can be updated, reducing the dependence on cross-production line retraining, while maintaining the consistent judgment criteria of multimodal under the same semantic coordinate. Detailed Implementation

[0062] In Example 1, referring to Figures 1 to 8, an industrial quality inspection method based on multimodal and few-shot learning includes:

[0063] S1. Obtain the multimodal sample set of workpieces from the benchmark production line, the annotation sample set of the benchmark production line, the annotation sample set of the new production line, and the multimodal sample of the workpiece to be inspected. Organize the multimodal sample set of workpieces from the benchmark production line into a cross-modal paired training sample set according to the workpiece identification.

[0064] S2. Train the SigLIP2 contrastive learning multimodal base model based on the cross-modal paired training sample set. The SigLIP2 contrastive learning multimodal base model consists of an image encoder, a geometric encoder, a process signal encoder, and a unified projection layer. Generate a unified representation vector for each workpiece multimodal sample. Construct cross-modal positive pairing based on the same workpiece identifier and constrain the unified representation vectors of the positive pairing to be close. Construct negative pairing based on different workpiece identifiers and constrain the unified representation vectors of the negative pairing to be separated, thus obtaining a unified representation model.

[0065] S3. Input the baseline production line labeled sample set into the unified representation model to obtain the baseline production line unified representation vector set. Aggregate the baseline production line unified representation vector set according to defect category and normal category to obtain the baseline production line prototype library.

[0066] S4. Input the new production line labeled sample set into the unified representation model to obtain the new production line unified representation vector set. Aggregate the normal samples of the new production line unified representation vector set to obtain the new production line normal prototype. Calculate the difference between the new production line normal prototype and the benchmark production line normal prototype in the benchmark production line prototype library to obtain the production line benchmark difference. Based on the production line benchmark difference, form a production line calibration transformation. Apply the production line calibration transformation to the benchmark production line prototype library and perform category aggregation and update in combination with the new production line unified representation vector set to obtain the new production line prototype library. Apply the production line calibration transformation to the normal samples of the new production line unified representation vector set to obtain the calibration normal vector set.

[0067] S5. Input the multimodal sample of the workpiece to be inspected into the unified representation model to obtain the unified representation vector to be inspected, and apply the production line calibration transformation to the unified representation vector to be inspected to obtain the calibrated vector to be inspected. Calculate the similarity between the calibrated vector to be inspected and the new production line prototype library to obtain the category matching score.

[0068] S6. Determine the judgment threshold based on the similarity statistics between the calibration normal vector set and the normal prototype of the new production line. When the maximum value of the category matching score meets the judgment threshold, output the defect category of the defect prototype corresponding to the maximum value. When the maximum value of the category matching score does not meet the judgment threshold, output the normal category and obtain the quality inspection judgment result.

[0069] In this embodiment, step S1 includes:

[0070] The benchmark production line is equipped with image acquisition devices, geometric acquisition devices, and process signal acquisition interfaces. A unique workpiece identifier is generated when a workpiece enters the acquisition station; this identifier is recorded as follows: The acquired image modal input is denoted as The acquired geometric modal input is denoted as The acquired process signal modal input is denoted as ,in For size take take Image tensors, The number of points is Each point contains a point cloud matrix with three-dimensional coordinates. For length is And each moment contains The timing matrix of each process quantity channel, when the same When at least two types of modal inputs are obtained, corresponding workpiece multimodal samples are formed. ;

[0071] Assigning each workpiece continuously produced on the baseline production line with... And trigger the collection, and collect the results. , , With the same Binding was performed to obtain a multimodal sample set of workpieces from the benchmark production line. When a certain mode is in If no valid input is collected, set the modality to empty and count the results. The number of valid modal inputs is determined by the number of valid modal inputs of at least two types. Write When the number of valid modal inputs is less than two types, the corresponding... Write This ensures that subsequent cross-modal pairing training samples can provide a pairing basis for at least two types of modal inputs for step S2;

[0072] from Multimodal samples of workpieces with defect or normal category labels are selected to form a baseline production line labeled sample set. Each labeled sample carries its corresponding category label. and One-to-one correspondence, using the same workpiece identification rules as the baseline production line to generate identification on the new production line. The same modal acquisition interface is used to acquire multimodal samples of workpieces from the new production line, and the samples are labeled with defect categories or normal categories to form a new production line labeled sample set. Simultaneously, unlabeled multimodal samples of workpieces are continuously collected on the new production line as the input source for multimodal samples of workpieces to be inspected. These multimodal samples of workpieces to be inspected only contain [data missing] before entering step S5. With at least two modal inputs without carrying category labels;

[0073] by For index pairs Grouping, putting the same Different modal inputs generate cross-modal paired training samples, when the same The following includes and Simultaneous generation of cross-modal paired training samples of image modal input and geometric modal input, when the same The following includes and Simultaneously generate cross-modal paired training samples of image modal input and process signal modal input, when the same The following includes and Cross-modal paired training samples of geometric modal input and process signal modal input are generated in real time. Each cross-modal paired training sample is organized into a set containing the first modal input, the second modal input, and a common modal input. The recording units are then collected to obtain a cross-modal paired training sample set. This enables step S2 to construct cross-modal positive and negative pairings within the unified representation vector space output by the unified projection layer and complete SigLIP2-style contrastive learning training.

[0074] In this embodiment, step S2 includes:

[0075] Training sample set with cross-modal pairing As training input Each record unit contains a workpiece identifier. A first modal input and a second modal input, the first modal input and the second modal input being derived from an image modal input. Geometric modal input Process signal modal input Selected from, and the same Different modal inputs constitute cross-modal positive pairing candidates, different The modal inputs below form negative pairing candidates, establishing a SigLIP2-style contrastive learning multimodal basic model. The SigLIP2-style contrastive learning multimodal foundation model Image encoder Geometric encoder Process signal encoder With the unified projection layer Composition, and will unify the projection layer It is set as the common output interface of the three types of encoders, so that the output of the three types of encoders can be mapped from the 1,024-dimensional feature space to the 512-dimensional unified representation vector space through the same set of shared fully connected neurons.

[0076] Image modal input The processing is as follows: Input to image encoder In the image encoder Internally, a two-dimensional convolutional layer, a non-linear activation layer, and a downsampling layer are executed sequentially to obtain the intermediate feature map of the image. For the intermediate feature map of the image Perform global convergence to obtain image feature vectors The image feature vector Each dimension is a pair of intermediate feature maps of the image. The spatial aggregation result of the corresponding channels, image feature vector The dimension is 1024, for geometric modal input The processing is as follows: Feed into geometric encoder Constructing local edge features based on point neighborhood The local edge features It is obtained by concatenating the coordinate difference between point pairs with the original point coordinates, thus capturing local edge features. Inputting multiple layers of neurons into a multilayer perceptron and aggregating them layer by layer yields geometric feature vectors. The geometric feature vector The system has 1024 dimensions, with each dimension representing a channel aggregation response to local edge features, and provides modal input to the process signal. The processing is as follows: Input process signal encoder In the process signal encoder Internally, a temporal convolutional module composed of one-dimensional convolutional neurons is executed sequentially to extract multi-scale temporal patterns and obtain intermediate feature sequences of the process signal. For the intermediate feature sequence of the process signal Performing global convergence yields the process signal feature vector. The process signal feature vector The dimension is 1024, and each dimension is the channel response after aggregation of the time axis;

[0077] Image feature vectors Geometric eigenvectors Process signal feature vector Input uniform projection layer separately A unified representation vector is obtained. , , The unified representation vectors all have a dimension of 512. Normalization is performed on each unified representation vector to make its magnitude one, allowing cross-modal similarity calculation to directly obtain the cosine similarity input using the vector inner product. (Unified projection layer) By adopting the structural constraint of shared fully connected neurons, the three modalities output a unified representation vector under the same semantic coordinates, so that the production line calibration transformation in the subsequent step S4 can be effective for different modalities in the 512-dimensional unified representation vector space.

[0078] Regarding the construction of training batches, from Select a training batch Training batch Contains multiple recording units and covers multiple For training batches Each recording unit calculates the unified representation vector corresponding to the modal inputs on both sides, and based on... Perform paired organization: Organize the same The unified representation vectors from different modalities are combined pairwise to form a cross-modal positive pairing set, which combines different... The unified representation vectors under the given conditions are combined pairwise to form a negative pairing set. When constructing the SigLIP2-style contrastive learning objective, the following total loss is used. Iterative updates to the model parameters:

[0079] ;

[0080] in, This represents the total loss used for parameter updates. , Indicates the modal type, Represents image modality, Represents geometric modes, Represents the process signal mode, for any given mode direction , Indicates training batch It simultaneously possesses modalities Input and Modality The input set of workpiece identifiers, Indicates training batch Zhongyu corresponding modes Side candidate workpiece identifier set, Represents a set Number of workpiece identifiers included. Indicates workpiece identification. Representing modes Candidate workpiece identification on the side, Indicates that the workpiece is identified as And the mode is The normalized unified representation vector, Indicates that the workpiece is identified as And the mode is The normalized unified representation vector, The dot product of two 512-dimensional vectors is represented by the symbol... This represents the transpose of a vector. This represents the temperature parameter, which is a preset constant. Represents an exponential function. Represents the natural logarithm function;

[0081] Employing gradient descent-type optimizers with total loss For the target image encoder Geometric encoder Process signal encoder With the unified projection layer The parameters are iteratively updated until the preset number of training rounds or the total loss convergence criterion is met, resulting in a unified representation model for subsequent steps S3 to S6. .

[0082] In this embodiment, step S3 includes:

[0083] The baseline production line labeled sample set is denoted as... Each sample is identified by a workpiece. As an index, and containing image modal input. Geometric modal input Process signal modal input At least two categories are included, both of which contain defect category or normal category labels. Let the unified representation model obtained from step S2 be denoted as The unified representation model Includes image encoder Geometric encoder Process signal encoder With the unified projection layer Unified projection layer This is the common output interface for all three types of encoders;

[0084] Will according to Sample-by-sample input unified representation model When the sample contains At that time, through and We obtain a 512-dimensional unified representation vector. When the sample contains At that time, through and We obtain a 512-dimensional unified representation vector. When the sample contains At that time, through and We obtain a 512-dimensional unified representation vector. , will the same The unified representation vector obtained below is written into the unified representation vector set of the benchmark production line. And associate each vector with its corresponding and Used for subsequent aggregation;

[0085] When the same When dealing with multiple unified representation vectors, for this Perform vector aggregation on all unified representation vectors to obtain the sample unified representation vector. The vector aggregation is achieved by summing the vectors dimension by dimension and then dividing by the number of vectors to obtain the mean value for each dimension. Perform normalization, the normalization being to... Divide each dimension of the components by The vector magnitude is such that the normalized vector... With a modulus of one, the normalized value is... Replace and write the unified representation vector set of the benchmark production line ,make Each Only one unified representation vector corresponds to each sample, thus ensuring that cross-modal samples are in the unified projection layer. The output forms a unified representation that can be directly aggregated under the same semantic coordinates;

[0086] according to Uniform representation vector for each sample Related category labels Group by category, and group all categories with the same label value. Samples are grouped into the same category, and vector aggregation is performed on the unified representation vectors of samples within each category group to obtain the category prototype. ,in Category labeling The value of the vector aggregation is obtained by summing the vectors dimensionally and then dividing by the number of vectors within the category group to obtain the dimensionally averaged value. This is applied to the category prototype. Perform normalization, the normalization being to... Divide each dimension of the components by The vector magnitude is such that the normalized vector... The module length is one.

[0087] The category prototypes corresponding to all defect categories Category prototypes corresponding to normal categories This collection constitutes the baseline production line prototype library. ,in This indicates the normal category label values ​​and maintains a one-to-one correspondence between the category prototype and the category label, enabling step S4 to retrieve data from the baseline production line prototype library. The normal prototype of the baseline production line is directly read for calculation of the baseline difference of the production line.

[0088] In this embodiment, step S4 includes:

[0089] The sample set labeled for the new production line is denoted as... Each sample is identified by a workpiece. As an index, it contains image modal input. Geometric modal input Process signal modal input It contains at least two categories, and includes either a defect category or a normal category label. Let the unified representation model obtained from step S2 be denoted as The unified representation model Includes image encoder Geometric encoder Process signal encoder With the unified projection layer Unified projection layer Output a 512-dimensional unified representation vector, and the normalized unified representation vector has a magnitude of one. The baseline production line prototype library constructed in step S3 is denoted as... This includes the baseline production line normal prototype. Prototypes of each defect category ;

[0090] Will according to Sample-by-sample input unified representation model When the sample contains At that time, through and Obtain the unified representation vector When the sample contains At that time, through and Obtain the unified representation vector When the sample contains At that time, through and Obtain the unified representation vector , will the same The unified representation vector obtained below is written into the unified representation vector set of the new production line. And associate each vector with its corresponding and Used for subsequent aggregation;

[0091] When the same When dealing with multiple unified representation vectors, for this Perform vector aggregation on all unified representation vectors to obtain the sample unified representation vector. The vector aggregation is achieved by summing the vectors dimension by dimension and then dividing by the number of vectors to obtain the mean value for each dimension. Perform normalization so that the normalized result is... With a modulus of one, the normalized value is... Replace and write the unified representation vector set of the new production line ,make Each It corresponds to only one sample unified representation vector;

[0092] according to Medium-sized sample unified representation vector Related category labels Select the vectors corresponding to the normal category labels to form a normal vector subset. ,right Performing vector aggregation yields the normal prototype of the new production line. And in this vector aggregation, a unified representation vector of the samples is introduced. Compared with the dimensional mean vector of the normal vector subset The aggregation weights are determined by the inner product similarity to complete the aggregation of... Numerical allocation of internal vector contributions;

[0093] From the baseline production line prototype library Read the baseline production line normal prototype Calculate the normal prototype of the new production line Compared with the normal prototype of the baseline production line The difference is obtained as the production line baseline difference. and production line baseline difference Defined as a production line calibration transformation acting on a 512-dimensional unified representation vector space. The method of applying any input vector is to combine the input vector with... The new production line prototype is obtained by adding the results dimension by dimension and then normalizing the results. Difference from production line benchmark The calculation uses the following formula:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] in, Represents a normal vector subset The one-dimensional mean vector, This represents a unified representation vector set from the new production line. The normal vector subset obtained by filtering Represents a normal vector subset The number of vectors contained. Indicates that the workpiece is identified as The unified representation vector of the samples, Represents the unified representation vector of samples The corresponding aggregate weights, Indicates the candidate workpiece identifier used for weight normalization. The dot product of two 512-dimensional vectors is represented by the symbol... This represents the transpose of a vector. This represents the temperature parameter, which is a preset constant. Represents an exponential function. This represents the normalization operator, which divides each component of the input vector by the magnitude of the input vector. This indicates that the new production line is in normal prototype condition. This indicates a normal prototype of the baseline production line. Indicates the production line baseline difference;

[0099] Baseline prototype library Apply production line calibration transformation to each category of prototypes one by one. Obtain the calibrated category prototypes, and compile all the calibrated category prototypes to form a calibration prototype library. ;

[0100] according to Uniform representation vector for each sample Related category labels Grouping by category yields a vector set for each category group. For each Performing vector aggregation yields the category aggregation result. The vector aggregation is achieved by summing the vectors dimension by dimension and then dividing by the number of vectors to obtain the mean value. The aggregation result is then normalized. For calibration prototype library The execution category aggregation update forms a new production line prototype library. , where the category exist When a calibration category prototype exists, the calibration category prototype is compared with... The updated category prototype is obtained by weighting and summing the samples according to their respective sample sizes and then normalizing. exist When no category prototype exists, the method is to use... The method of writing the prototype of the new category completes the category completion;

[0101] Subset of normal vectors Uniform representation vector for each sample Apply production line calibration transformation one by one Obtain the calibration normal vectors and aggregate all calibration normal vectors to form a calibration normal vector set. The threshold statistics are used for the subsequent step S6.

[0102] In this embodiment, step S5 includes:

[0103] The set of multimodal samples of the workpieces to be inspected collected from the new production line is denoted as . Each multimodal sample of the workpiece to be inspected is identified by the workpiece identifier. Index, and includes image modal input. Geometric modal input Process signal modal input At least two classes in the model, the unified representation model is denoted as The unified representation model Includes image encoder Geometric encoder Process signal encoder With the unified projection layer Unified projection layer Output a 512-dimensional, normalized unified representation vector, and denote the production line calibration transformation obtained in step S4 as... ,in This represents the production line baseline difference. Let the new production line prototype library obtained in step S4 be a 512-dimensional vector. New production line prototype library Contains multiple category prototypes ,in The category label value indicates the defect category or normal category;

[0104] Will according to Sample-by-sample input unified representation model When the multimodal sample of the workpiece to be inspected contains At that time, through and Obtain the unified representation vector When the multimodal sample of the workpiece to be inspected contains At that time, through and Obtain the unified representation vector When the multimodal sample of the workpiece to be inspected contains At that time, through and Obtain the unified representation vector The unified representation vector is generated by the unified projection layer. The output values ​​of the 512-dimensional output neurons constitute the output values, with each dimension representing the numerical output of the corresponding output neuron. When the multimodal sample of the same workpiece under inspection contains multiple modal inputs, the output values ​​of the 512-dimensional output neurons are used to determine the output values ​​of the corresponding output neurons. Perform vector aggregation on all unified representation vectors to obtain the unified representation vector to be tested. The vector aggregation is achieved by summing the vectors dimension by dimension and then dividing by the number of vectors to obtain the mean value. The aggregation result is then normalized to a magnitude of one. When only one unified representation vector is output for the multimodal samples of the same workpiece under inspection, this unified representation vector is directly used as the mean value. Thus, each It corresponds to only one unified representation vector to be detected. ;

[0105] The unified representation vector to be inspected Apply production line calibration transformation Obtain the calibration vector to be tested ,in This indicates that the vector to be calibrated is being applied, and the application process is to... Difference from production line benchmark The calibration intermediate vector is obtained by adding the vectors dimension by dimension, and then normalization is performed on the calibration intermediate vector to obtain the calibration intermediate vector. ,make With the new production line prototype library Medium category prototype They are under the same unified representation vector space reference;

[0106] Read the new production line prototype library Each category prototype calibrate the vector to be tested and Similarity calculations were performed, and cosine similarity was used as the similarity metric. and All vectors are normalized. Cosine similarity is achieved using the vector dot product, which involves multiplying the corresponding dimensional components of the 512-dimensional vector pairwise and summing the results over the 512 dimensions to obtain the similarity scalar. ,in Indicates that the workpiece is identified as Calibration of the vector to be tested With category prototype Similarity;

[0107] The same All corresponding similarity scalars Values ​​are labeled by category. Aggregate to form category matching scores ,in For multiple The record structure consists of category matching scores. As the input for the quality inspection result in the subsequent step S6, step S6 can be based on the category matching score. Complete the category determination process.

[0108] In this embodiment, step S6 includes:

[0109] The calibration normal vector set obtained in step S4 is denoted as... superscript This indicates that the vectors in this vector set have undergone production line calibration transformation and normalization. The new normal production line prototype obtained in step S4 is denoted as... The category matching score obtained in step S5 is recorded as ,in Indicates the identification of the workpiece to be inspected, and the category matching score. Includes multiple category label values Corresponding similarity scalar Category label value Includes normal category annotation values Values ​​can be assigned to multiple defect categories;

[0110] right Each calibration normal vector in the process is calculated one by one to compare with the normal prototype of the new production line. The cosine similarity is used to obtain the normal similarity sequence. ,Will A single calibration normal vector in the vector is denoted as ,in It is a 512-dimensional vector, where each dimension is a numerical component of the output from the unified projection layer after production line calibration transformation and normalization. and The cosine similarity is denoted as The calculation method is as follows: for dimension index from arrive Dimensional extraction The Dimensional components and The Multiply the components by their dimensionality and apply to all. Summing the products yields , each Write in the order of calculation ,make It is a one-dimensional sequence composed of multiple similarity scalars;

[0111] For normal similarity sequences Perform statistics to determine the judgment threshold. ,Will The sorted sequence is obtained by sorting all similarity scalars in ascending order of their values. superscript This represents the sorted sequence. The length is denoted as , in Given a positive integer, calculate the position value. And round up to get the index ,when season ,Pick The Middle A similarity scalar is used as the 5th percentile value, and this 5th percentile value is determined as the decision threshold. ;

[0112] Category matching score The defect matching score is composed of the scores corresponding to the defect categories selected in the screening process. superscript This represents the set of scores corresponding to defect categories, and the filtering is a traversal. All category labels within the text are categorized and their values ​​are retrieved. Only retain those that meet the requirements. of ,right The maximum defect score is obtained by performing a maximum value operation on the scalar similarity of all defect categories. And record The defect category label value that achieves the maximum value ,in For a single value, Assign values ​​to a single defect category;

[0113] Maximum defect score With the judgment threshold When comparing, Not less than When, output As a result of quality inspection, when Less than When, output As the result of quality inspection.

[0114] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An industrial quality inspection method based on multimodal and few-shot learning, characterized in that, include: S1. Obtain the multimodal sample set of workpieces from the baseline production line, the labeled sample set of the baseline production line, the labeled sample set of the new production line, and the multimodal sample set of workpieces to be inspected. Organize the multimodal sample set of workpieces from the baseline production line into a cross-modal paired training sample set according to the workpiece identification. S2. Train the SigLIP2 contrastive learning multimodal basic model based on the cross-modal paired training sample set. The SigLIP2 contrastive learning multimodal basic model consists of an image encoder, a geometric encoder, a process signal encoder, and a unified projection layer. Generate a unified representation vector for each workpiece multimodal sample. Construct cross-modal positive pairings based on the same workpiece identification and constrain the unified representation vectors of the positive pairings to be close. Construct negative pairings based on different workpiece identifications and constrain the unified representation vectors of the negative pairings to be separated, thus obtaining a unified representation model. S3. Input the labeled sample set of the baseline production line into the unified representation model to obtain the unified representation vector set of the baseline production line. Aggregate the unified representation vector set of the baseline production line according to the defect category and the normal category to obtain the baseline production line prototype library. S4. Input the new production line labeled sample set into the unified representation model to obtain the new production line unified representation vector set. Aggregate the normal samples of the new production line unified representation vector set to obtain the new production line normal prototype. Calculate the difference between the new production line normal prototype and the benchmark production line normal prototype in the benchmark production line prototype library to obtain the production line benchmark difference. Based on the production line benchmark difference, form a production line calibration transformation. Apply the production line calibration transformation to the benchmark production line prototype library and perform category aggregation and update in conjunction with the new production line unified representation vector set to obtain the new production line prototype library. Apply the production line calibration transformation to the normal samples of the new production line unified representation vector set to obtain the calibrated normal prototype. Vector set; S5, Input the multimodal sample of the workpiece to be inspected into the unified representation model to obtain the unified representation vector to be inspected, and apply the production line calibration transformation to the unified representation vector to be inspected to obtain the calibrated vector to be inspected. Calculate the similarity between the calibrated vector to be inspected and the new production line prototype library to obtain the category matching score; S6, Determine the judgment threshold based on the similarity statistics between the calibrated normal vector set and the new production line normal prototype. When the maximum value of the category matching score meets the judgment threshold, output the defect category of the defect prototype corresponding to the maximum value. When the maximum value of the category matching score does not meet the judgment threshold, output the normal category to obtain the quality inspection judgment result.

2. The industrial quality inspection method based on multimodal and few-shot learning according to claim 1, characterized in that, S1 includes: Multimodal samples of workpieces are collected on the benchmark production line according to the workpiece identification. The multimodal samples of workpieces include at least two types of image modal input, geometric modal input, and process signal modal input. The workpiece identification is bound to each multimodal sample of workpieces to form a multimodal sample set of workpieces on the benchmark production line. Samples with defect or normal category labels are selected from the multimodal sample set of workpieces from the benchmark production line to form the benchmark production line labeled sample set. In the new production line, workpiece identification rules consistent with those of the benchmark production line are used to collect and label samples to form the new production line labeled sample set. At the same time, unlabeled multimodal samples of workpieces to be inspected are collected. The multimodal sample set of the benchmark production line workpieces is grouped by workpiece identifier as index. Different modal inputs under the same workpiece identifier are used to generate cross-modal paired training samples according to modal combination relationships. All cross-modal paired training samples are then collected into a cross-modal paired training sample set.

3. The industrial quality inspection method based on multimodal and few-shot learning according to claim 1, characterized in that, S2 includes: Using a cross-modal paired training sample set as input, a SigLIP2-style contrastive learning multimodal foundation model is established. This model includes an image encoder, a geometric encoder, a process signal encoder, and a unified projection layer, with the unified projection layer serving as the common output interface for all three encoder types. The image modal input of each workpiece multimodal sample from the cross-modal paired training sample set is fed into the image encoder, where it passes through multiple 2D convolutional layers, nonlinear activation layers, and downsampling layers to form an intermediate image feature map. This map is then globally converged to obtain a 1024-dimensional image feature vector. The geometric modal input of the workpiece multimodal samples is fed into the geometric encoder. Based on point neighborhoods, local edge features are constructed and transformed and aggregated layer by layer by multi-layer perceptron neurons to obtain a 1024-dimensional geometric feature vector. The process signal modal input of the multi-modal sample of the workpiece is sent to the process signal encoder, and multi-scale time patterns are extracted by a temporal convolution module composed of one-dimensional convolutional neurons and then globally converged to obtain a 1024-dimensional process signal feature vector. The image feature vector, geometric feature vector, and process signal feature vector are respectively input into a unified projection layer composed of shared fully connected neurons, and uniformly mapped to a 512-dimensional unified representation vector. The unified representation vector is then normalized to serve as the input for cross-modal similarity calculation. Within the same training batch, cross-modal positive pairings are constructed based on the workpiece identifier. The cross-modal positive pairings are obtained by combining the unified representation vectors generated by the same workpiece identifier in different modes. Negative pairings are constructed based on different workpiece identifiers. The negative pairings are obtained by combining the unified representation vectors generated by different workpiece identifiers in pairs. The similarity between cross-modal positive and negative pairings is calculated, and the model parameters are iteratively updated using a SigLIP2-style contrastive learning objective. This ensures that the similarity of cross-modal positive pairings satisfies the training constraint that the similarity of negative pairings is greater than that of negative pairings, thus obtaining a unified representation model.

4. The industrial quality inspection method based on multimodal and few-shot learning according to claim 1, characterized in that, S3 includes: inputting the baseline production line labeled sample set into the unified representation model one by one; outputting a 512-dimensional unified representation vector for each sample's image modal input, geometric modal input, and process signal modal input through the corresponding encoder and unified projection layer, forming a baseline production line unified representation vector set; when the same sample contains multiple modal inputs, aggregating the multiple unified representation vectors corresponding to the sample to obtain a sample unified representation vector, and replacing the sample unified representation vector in the baseline production line unified representation vector set; grouping the baseline production line unified representation vector set into categories according to the defect category and normal category labels of the baseline production line labeled sample set, aggregating the vectors within each category group to obtain the category prototype of that category; and compiling the category prototypes of each defect category and the category prototypes of the normal category to form a baseline production line prototype library.

5. The industrial quality inspection method based on multimodal and few-shot learning according to claim 1, characterized in that, S4 includes: The new production line labeled sample set is input into the unified representation model one by one. For each sample, the image modal input, geometric modal input, and process signal modal input are respectively encoded by the corresponding encoder and the unified projection layer to output a 512-dimensional unified representation vector, forming a unified representation vector set for the new production line. When the same sample contains multiple modal inputs, the multiple unified representation vectors corresponding to the sample are aggregated to obtain a unified representation vector for the sample, and the unified representation vector for the sample is replaced and written into the unified representation vector set for the new production line. According to the normal category label of the new production line labeled sample set, the vectors corresponding to normal samples are selected from the unified representation vector set for the new production line, and the vectors corresponding to the normal samples are aggregated to obtain the normal prototype of the new production line. Read the normal prototype of the benchmark production line from the benchmark production line prototype library, calculate the difference between the normal prototype of the new production line and the normal prototype of the benchmark production line to obtain the benchmark difference of the production line, and define the benchmark difference of the production line as the production line calibration transformation acting on the 512-dimensional unified representation vector space. The calibration prototype library is obtained by applying the production line calibration transformation to each category of prototypes in the baseline production line prototype library; the new production line unified representation vector set is grouped into categories according to the defect category and normal category labels of the new production line labeled sample set, and vector aggregation is performed on the vectors in each category group to obtain the category aggregation result, and the calibration prototype library is updated by category aggregation using the category aggregation result to obtain the new production line prototype library; the calibration normal vector set is obtained by applying the production line calibration transformation to the vectors corresponding to the normal samples in the new production line unified representation vector set.

6. The industrial quality inspection method based on multimodal and few-shot learning according to claim 1, characterized in that, S5 include: The multimodal samples of the workpiece to be inspected are represented by a unified model based on the sample input. The image modal input, geometric modal input, and process signal modal input contained in the multimodal samples of the workpiece to be inspected are respectively encoded by the corresponding encoders and the unified projection layer to output a 512-dimensional unified representation vector. When the multimodal samples of the workpiece to be inspected contain multiple types of modal inputs, the multiple unified representation vectors are aggregated to obtain the unified representation vector to be inspected. The unified representation vector to be inspected is subjected to a production line calibration transformation to obtain a calibrated inspection vector. The similarity between the calibrated inspection vector and each category of prototype in the new production line prototype library is calculated, and cosine similarity is used as the similarity measure. The similarity scores corresponding to each category of prototype are aggregated to form a category matching score.

7. The industrial quality inspection method based on multimodal and few-shot learning according to claim 1, characterized in that, Step S6 includes: calculating the cosine similarity between each calibration normal vector set and the normal prototype of the new production line to obtain a normal similarity sequence; performing statistics on the normal similarity sequence and determining the 5th percentile value of the normal similarity sequence as the judgment threshold; selecting the scores corresponding to the defect category from the category matching scores to form a defect matching score, and taking the maximum value of the defect matching score as the maximum defect score; when the maximum defect score is not less than the judgment threshold, outputting the defect category of the defect prototype corresponding to the maximum defect score as the quality inspection judgment result; when the maximum defect score is less than the judgment threshold, outputting the normal category as the quality inspection judgment result.

Images