Manufacturing quality detection method and device, electronic equipment and storage medium
By leveraging cross-modal attention mechanisms and data-augmented AI models, combined with a case database, the problem of low efficiency and poor consistency in traditional manufacturing quality inspection has been solved. This enables fast, intelligent, and accurate quality inspection, is compatible with both mobile and PC platforms, and supports large-scale intelligent quality control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG LEAPENERGY TECH CO LTD
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-14
Smart Images
Figure CN122390528A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of quality inspection technology, specifically to a manufacturing quality inspection method, apparatus, electronic device, and storage medium. Background Technology
[0002] During vehicle manufacturing, factors such as process fluctuations, material properties, and equipment condition inevitably lead to production defects in parts, including dimensional deviations, surface defects, or assembly errors. To ensure that the overall vehicle quality meets standards, the quality inspection workshop needs to perform manufacturing quality inspections on each component to determine whether its quality status meets the standards and to generate targeted improvement suggestions when anomalies are detected. However, traditional manufacturing quality inspection relies heavily on the subjective experience of quality inspectors, resulting in low inspection efficiency, poor consistency, and difficulty in scaling up, failing to meet the urgent needs of modern automobile manufacturing for intelligent and efficient quality control. Summary of the Invention
[0003] This application provides a manufacturing quality inspection method, apparatus, electronic device, and storage medium, which can quickly, intelligently, accurately, and systematically inspect the manufacturing quality of the object under test, improve inspection efficiency, and help achieve large-scale, high-efficiency intelligent quality control in application fields such as vehicles.
[0004] In a first aspect, embodiments of this application provide a manufacturing quality inspection method, the method comprising: performing cross-modal feature processing on multimodal data of an object under test to obtain cross-modal interaction feature data of the object under test, wherein the multimodal data includes text data, image data, and indicator data for indicating the manufacturing quality of the object under test; determining a target sample and a reference sample from the multiple case samples based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in a case database, wherein the manufacturing quality of the target sample is similar to the manufacturing quality of the object under test, and the reference sample belongs to the same object type as the object under test and the manufacturing quality of the reference sample is normal; and determining the manufacturing quality inspection result of the object under test based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object under test.
[0005] In some embodiments, performing cross-modal feature processing on the multimodal data of the test object to obtain cross-modal interaction feature data of the test object includes: performing feature encoding on the text data, image data, and indicator data in the multimodal data of the test object to obtain text-encoded data, image-encoded data, and indicator-encoded data, respectively; performing linear transformation on the text-encoded data, image-encoded data, and indicator-encoded data, respectively, to obtain text feature data, image feature data, and indicator feature data; and performing attention mechanism calculation based on the text feature data, image feature data, and indicator feature data to obtain the cross-modal interaction feature data of the test object.
[0006] In some embodiments, the step of calculating the cross-modal interaction feature data of the test object based on the text feature data, the image feature data, and the indicator feature data includes: calculating first cross-modal attention feature data using the text feature data as a query vector, the image feature data as a key vector, and the indicator feature data as a value vector; calculating second cross-modal attention feature data using the image feature data as a query vector, the text feature data as a key vector, and the indicator feature data as a value vector; and performing a weighted summation of the first cross-modal attention feature data and the second cross-modal attention feature data to obtain the cross-modal interaction feature data of the test object.
[0007] In some embodiments, determining a target sample and a reference sample from multiple case samples based on the cross-modal interaction feature data of the test object and the cross-modal interaction feature data of multiple case samples in the case database includes: inputting the cross-modal interaction feature data of the test object and the cross-modal interaction feature data of multiple case samples into a feature analysis model, so that the feature analysis model determines the target sample and the reference sample from the multiple case samples; wherein the processing performed by the feature analysis model on the cross-modal interaction feature data includes, in sequence: retrieval, recall, similarity calculation, and rearrangement.
[0008] In some embodiments, the method is performed based on a lightweight model, which is obtained by distilling a large model that has been trained; wherein the distillation includes at least one of the following: feature extraction distillation, attention pruning, and sample selection distillation.
[0009] In some embodiments, the method further includes: if the manufacturing quality inspection result of the object under test meets the target conditions, then the object under test is used as a candidate sample; the multimodal data of at least one candidate sample is input into the large model, so that the large model outputs the manufacturing quality inspection result of the candidate sample; if the manufacturing quality inspection result of the candidate sample output by the lightweight model does not match the manufacturing quality inspection result of the candidate sample output by the large model, then the candidate sample is used as an incremental sample; the lightweight model is corrected based on at least one incremental sample.
[0010] In some embodiments, the method further includes: performing data augmentation on the multimodal data of the case sample to obtain augmented multimodal data; performing cross-modal feature processing on the augmented multimodal data to obtain cross-modal interaction feature data of the case sample; wherein, the data augmentation for the text data includes at least one of the following: synonym replacement, sentence restructuring, and definition generation; the data augmentation for the image data includes at least one of the following: view expansion, rotation, cropping, scaling, brightness adjustment, and blur adjustment; the data augmentation for the indicator data includes at least one of the following: data rewriting with equivalent values and data unit conversion.
[0011] Secondly, embodiments of this application provide a manufacturing quality inspection device, the device comprising: a cross-modal feature processing module, used to perform cross-modal feature processing on the multimodal data of the object under test to obtain cross-modal interaction feature data of the object under test, the multimodal data including text data, image data, and indicator data for indicating the manufacturing quality of the object under test; a sample acquisition module, used to determine a target sample and a reference sample from multiple case samples based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in a case database, wherein the manufacturing quality of the target sample is similar to the manufacturing quality of the object under test, the reference sample belongs to the same object type as the object under test and the manufacturing quality of the reference sample is normal; and a manufacturing quality determination module, used to determine the manufacturing quality inspection result of the object under test based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object under test.
[0012] Thirdly, embodiments of this application provide an electronic device, including: a memory storing computer programs or instructions thereon; and a processor for executing the computer programs or instructions in the memory to implement the manufacturing quality inspection method as described above.
[0013] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the manufacturing quality inspection method as described above.
[0014] In summary, the technical solution provided in this application involves performing cross-modal feature processing on the multimodal data of the object under test to obtain cross-modal interaction feature data of the object under test; determining target samples and reference samples from multiple case samples based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in the case database; and determining the manufacturing quality inspection result of the object under test based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object under test. The multimodal data includes text data, image data, and indicator data used to indicate the manufacturing quality of the object under test. This data can simulate the multi-dimensional data relied upon in the manual inspection process, ensuring comprehensive and reliable data to achieve all-round manufacturing quality inspection of the object under test. The cross-modal feature processing of multimodal data in this application to obtain cross-modal interaction feature data allows for the interactive supplementation and fusion enhancement of various modal data and their inherent relationships, overcoming the detection bias caused by the partial or missing information of a single modality. This achieves a comprehensive and consistent characterization of the manufacturing quality of the object under test, providing a reliable feature basis for subsequent comparison with case samples and manufacturing quality judgment. Furthermore, target and reference samples are obtained from the case database based on cross-modal interaction feature data. The manufacturing quality of the target sample is similar to that of the object under test, thus the quality inspection label of the object under test can be determined based on the quality inspection label of the target sample. The reference sample belongs to the same object type as the object under test, and the manufacturing quality of the reference sample is normal. Therefore, improvement suggestions can be provided for the object under test by comparing the multimodal data of the reference sample with the multimodal data of the object under test. Based on this, the embodiments of this application can quickly, intelligently, accurately, and systematically realize the manufacturing quality inspection of the object under test, improve inspection efficiency, and help achieve large-scale, high-efficiency intelligent quality control in application fields such as vehicles. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart of a manufacturing quality inspection method provided in an embodiment of this application; Figure 2This is a schematic diagram of a cross-modal feature processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating a data augmentation method provided in an embodiment of this application; Figure 4 This is a schematic diagram of a distillation process provided in an embodiment of this application; Figure 5 This is a schematic diagram of a manufacturing quality inspection method provided in an embodiment of this application; Figure 6 This is a schematic diagram of a manufacturing quality inspection device provided in an embodiment of this application; Figure 7 This is a schematic diagram of another manufacturing quality inspection device provided in an embodiment of this application. Detailed Implementation
[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.
[0018] Traditional manufacturing quality inspection relies heavily on the subjective experience of quality inspectors, resulting in low efficiency, poor consistency, and difficulty in large-scale implementation. This fails to meet the urgent needs of modern automobile manufacturing for intelligent and efficient quality control. To address this, AI (Artificial Intelligence) algorithm-based manufacturing quality inspection methods have been proposed. However, most of these technologies are limited to detecting based on only one or two features from vehicle-related image, text, or indicator data, failing to simulate the multi-dimensional data required for manual inspection. Furthermore, most of these technologies lack model distillation and pruning to train a lightweight model with fewer parameters and lower resource consumption, making them unusable on mobile devices and PCs.
[0019] In view of this, embodiments of this application provide a manufacturing quality inspection method, apparatus, electronic device, and storage medium, which perform manufacturing quality inspection based on technologies such as cross-modal attention mechanism, data augmentation, large AI model, and model distillation, and provide manufacturing quality inspection results. The preliminary quality inspection data can be organized into case samples in a certain format. Each case sample can include text data, image data, indicator data, and quality inspection labels. Then, data augmentation can be performed on the case samples, including text augmentation, image augmentation, and indicator augmentation. Next, cross-modal fusion is performed on the text data, image data, and indicator data of each case sample to obtain cross-modal interaction feature data of the case samples. Similarly, cross-modal fusion is performed on the text data, image data, and indicator data of the test object to obtain cross-modal interaction feature data of the test object. Through a large AI model, the cross-modal interaction feature data of the case samples and the cross-modal interaction feature data of the test object are retrieved, recalled, similarity calculated, and rearranged to output the quality inspection labels of the test object. Improvement suggestions are given based on the comparison results. Furthermore, a lightweight model with small parameters and low resource consumption can be trained through model distillation and pruning operations to ensure that it can be deployed and used on ordinary mobile or PC terminals. The lightweight model with low resource consumption can be periodically corrected based on the model recognition results.
[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating a manufacturing quality inspection method provided in an embodiment of this application. This manufacturing quality inspection method can be used for manufacturing quality inspection in various application fields such as vehicles, aircraft, ships, robots, and home appliances; however, this embodiment does not limit its application to these fields. Figure 1 As shown, the manufacturing quality inspection method may include the following steps: Step S100: Perform cross-modal feature processing on the multimodal data of the test object to obtain cross-modal interaction feature data of the test object; wherein, the multimodal data includes text data, image data, and indicator data used to indicate the manufacturing quality of the test object; Step S200: Based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in the case database, determine the target sample and the reference sample from the multiple case samples; wherein, the manufacturing quality of the target sample is similar to that of the object under test, and the reference sample belongs to the same object type as the object under test and the manufacturing quality of the reference sample is normal. Step S300: Determine the manufacturing quality inspection result of the object under test based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object under test.
[0021] The specific type of the test object may vary depending on the application area, and can be flexibly set according to requirements in practical applications. For example, in the field of vehicle applications, the test object may include, but is not limited to, components such as batteries, brake discs, brake pads, motors, transmissions, gears, engines, pistons, suspensions, pedals, airbags, seats, audio systems, vehicle bodies, and screens.
[0022] To more comprehensively and accurately reflect the manufacturing quality of the object under test, embodiments of this application acquire multimodal data of the object under test from multiple dimensions. For example, the multimodal data may include text data, image data, and indicator data used to indicate the manufacturing quality of the object under test. Image data may be data obtained by photographing or scanning the object under test, such as, but not limited to, appearance images, wiring images, and soldering images. Text data may be descriptive descriptions of the image data, such as text obtained and recorded through visual inspection or automatic detection and recognition, which can complement the image data. Indicator data may be status data or operational data recorded during the corresponding testing of the object under test, including but not limited to voltage sequences, current sequences, temperature sequences, frequency sequences, and rotational speed sequences.
[0023] This application embodiment performs cross-modal feature processing on the multimodal data of the test object to obtain cross-modal interaction feature data of the test object. Taking multimodal data including text data, image data, and indicator data as an example, cross-modal feature processing refers to establishing a correlation mapping between different modal data, fusing heterogeneous text data, image data, and indicator data into a feature vector of a unified representation space, so that various modal data and their inherent correlations are interactively supplemented and enhanced. By obtaining cross-modal interaction feature data, the detection bias caused by the partial or missing information of a single modality can be overcome, and a comprehensive and consistent characterization of the manufacturing quality of the test object can be achieved, providing a reliable feature basis for subsequent comparison with case samples and manufacturing quality judgment. For other descriptions of cross-modal feature processing, please refer to the following embodiments, which will not be repeated here.
[0024] The case database can be a pre-configured database of multiple case samples related to manufacturing quality inspection. The initial case samples in the case database can be compiled based on historical quality inspection cases and / or expert experience. During actual quality inspection, case samples can be continuously added to expand the case database based on the inspection results. Each case sample in the case database can include corresponding multimodal data, cross-modal interaction feature data, and quality inspection tags. The cross-modal interaction feature data of the case sample can be obtained by performing the aforementioned cross-modal feature processing on the multimodal data of the case sample. The cross-modal interaction feature data of each case sample can be pre-acquired and stored in the case database to facilitate subsequent rapid comparison and manufacturing quality inspection. The quality inspection tags of the case samples are used to indicate whether the manufacturing quality of the case sample is normal or abnormal, and can also be used to indicate the type or cause of manufacturing quality abnormalities. This application embodiment does not limit the specific meaning of the quality inspection tags; in practical applications, they can be flexibly set according to requirements.
[0025] It should be understood that the case database may include case samples with normal manufacturing quality as well as case samples with abnormal manufacturing quality. Furthermore, the case database may include case samples of manufacturing quality abnormalities corresponding to different types of abnormalities, such as battery overvoltage, motor system abnormalities, high-voltage interlock circuit failures, controller failures, vehicle starting difficulties, and OTA (Over-the-Air Technology) upgrade failures, to ensure that various possible object types and manufacturing quality abnormality types can be dealt with during actual quality inspection.
[0026] Based on the cross-modal interaction feature data of the test object and the cross-modal interaction feature data of multiple case samples, target samples and reference samples can be determined from the multiple case samples. The cross-modal interaction feature data of the target sample is similar to that of the test object, thus the manufacturing quality of the target sample is similar to that of the test object. The manufacturing quality of the reference sample is normal, and the reference sample belongs to the same object type as the test object, such as batteries. Therefore, improvement suggestions can be provided for the test object based on the reference sample. For further details on the selection methods for target and reference samples, please refer to the following embodiments; they will not be elaborated upon here.
[0027] This application embodiment, based on a target sample and a reference sample, can quickly and accurately determine the manufacturing quality inspection results of the object under test. These results may include, but are not limited to, quality inspection labels, improvement suggestions, and test result confidence levels. Since the manufacturing quality of the target sample is similar to that of the object under test, the quality inspection label of the target sample can be used as the quality inspection label of the object under test. Because the reference sample and the object under test belong to the same object type and the reference sample has normal manufacturing quality, improvement suggestions can be provided for the object under test based on a comparison between the multimodal data of the reference sample and the multimodal data of the object under test, so as to subsequently optimize the manufacturing quality of the object under test.
[0028] In summary, the technical solution provided in this application involves performing cross-modal feature processing on the multimodal data of the object under test to obtain cross-modal interaction feature data of the object under test; determining target samples and reference samples from multiple case samples based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in the case database; and determining the manufacturing quality inspection result of the object under test based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object under test. The multimodal data includes text data, image data, and indicator data used to indicate the manufacturing quality of the object under test. This data can simulate the multi-dimensional data relied upon in the manual inspection process, ensuring comprehensive and reliable data to achieve all-round manufacturing quality inspection of the object under test. The cross-modal feature processing of multimodal data in this application to obtain cross-modal interaction feature data allows for the interactive supplementation and fusion enhancement of various modal data and their inherent relationships, overcoming the detection bias caused by the partial or missing information of a single modality. This achieves a comprehensive and consistent characterization of the manufacturing quality of the object under test, providing a reliable feature basis for subsequent comparison with case samples and manufacturing quality judgment. Furthermore, target and reference samples are obtained from the case database based on cross-modal interaction feature data. The manufacturing quality of the target sample is similar to that of the object under test, thus the quality inspection label of the object under test can be determined based on the quality inspection label of the target sample. The reference sample belongs to the same object type as the object under test, and the manufacturing quality of the reference sample is normal. Therefore, improvement suggestions can be provided for the object under test by comparing the multimodal data of the reference sample with the multimodal data of the object under test. Based on this, the embodiments of this application can quickly, intelligently, accurately, and systematically realize the manufacturing quality inspection of the object under test, improve inspection efficiency, and help achieve large-scale, high-efficiency intelligent quality control in application fields such as vehicles.
[0029] In some embodiments, the cross-modal feature processing of the multimodal data of the test object in step S100 above to obtain the cross-modal interaction feature data of the test object may include: performing feature encoding on the text data, image data, and indicator data in the multimodal data of the test object to obtain text-encoded data, image-encoded data, and indicator-encoded data respectively; performing linear transformation on the text-encoded data, image-encoded data, and indicator-encoded data respectively to obtain text feature data, image feature data, and indicator feature data; and performing attention mechanism calculation based on the text feature data, image feature data, and indicator feature data to obtain the cross-modal interaction feature data of the test object.
[0030] The feature encoding methods may differ for different data modalities, and can be flexibly set according to requirements in practical applications. For example, feature encoding for text data may include, but is not limited to, embedding encoding or one-hot encoding; feature encoding for image data may be implemented based on CNN convolution or autoencoders; and feature encoding for indicator data may be implemented based on Min-Max normalization or softmax normalization.
[0031] After feature encoding, linear transformations can be applied to the encoded data in each dimension to facilitate subsequent cross-modal fusion. This linear transformation can be achieved by multiplying the encoded data by the corresponding parameter matrix, where the dimension of the parameter matrix corresponds to the dimension of the encoded data. During training, the internal parameters of this parameter matrix can be initialized in any manner, such as random initialization (normal or uniform distribution), or orthogonal initialization, but its dimension must correspond to the dimension of the encoded data. The internal parameters of this parameter matrix are then iteratively optimized. After training reaches the optimal solution and convergence, the parameter matrix can be used in the actual quality inspection process. It should be understood that the parameter matrices corresponding to encoded data in different dimensions can be the same or different, and can be determined in practice by combining iterative optimization during the training process. This embodiment does not impose such limitations.
[0032] Based on the linearly transformed text feature data, image feature data, and indicator feature data, an attention mechanism can be calculated to achieve cross-modal fusion and obtain the cross-modal interaction feature data of the object under test. This application embodiment does not limit the number of heads or layers in the attention mechanism calculation; a multi-head attention mechanism or a multi-layer attention mechanism can be used. In practical applications, it can be flexibly set according to requirements to ensure that cross-modal features can be fused.
[0033] In some embodiments, the above steps, which calculate the attention mechanism based on text feature data, image feature data, and indicator feature data to obtain cross-modal interaction feature data of the object under test, may include: calculating first cross-modal attention feature data using text feature data as a query vector, image feature data as a key vector, and indicator feature data as a value vector; calculating second cross-modal attention feature data using image feature data as a query vector, text feature data as a key vector, and indicator feature data as a value vector; and performing a weighted summation of the first and second cross-modal attention feature data to obtain the cross-modal interaction feature data of the object under test.
[0034] In the cross-modal fusion process, this application embodiment uses text feature data and image feature data alternately as query vector and key vector, while indicator feature data is always used as value vector. This ensures that text feature data and image feature data always focus on each other's key information and supplement relevant indicator information.
[0035] Please see Figure 2 , Figure 2 This is a schematic diagram of a cross-modal feature processing method provided in an embodiment of this application. For example... Figure 2 As shown, firstly, the text data, image data, and indicator data in the multimodal data are respectively encoded to obtain text encoded data Et, image encoded data Ep, and indicator encoded data Ev. Then, linear transformations are performed on the text encoded data Et, image encoded data Ep, and indicator encoded data Ev respectively. For example, the text encoded data Et is multiplied by the corresponding parameter matrix Wt to obtain the text feature data Et·Wt, the image encoded data Ep is multiplied by the corresponding parameter matrix Wp to obtain the image feature data Ep·Wp, and the indicator encoded data Ev is multiplied by the corresponding parameter matrix Wv to obtain the indicator feature data Ev·Wv.
[0036] like Figure 2 As shown, in the calculation of the attention mechanism, on the one hand, the text feature data Et·Wt can be used as the query vector Query. tex Using image feature data Ep·Wp as the key vector Key pic Using the indicator feature data Ev·Wv as the value vector Value, calculate the first cross-modal attention feature data Att. tpv As shown in Formula 1 below; on the other hand, the image feature data Ep·Wp can be used as the query vector Query. pic Using text feature data Et·Wt as the key vector Key tex Using the indicator feature data Ev·Wv as the value vector Value, calculate the second cross-modal attention feature data Att.ptv As shown in Formula 2 below.
[0037] Formula 1:
[0038] Formula 2:
[0039] like Figure 2 As shown, during cross-modal fusion, the cross-modal attention feature data calculated in two different ways are weighted and summed. The first cross-modal attention feature data Att is then weighted and summed. tpv Second cross-modal attention feature data Att ptv Different weights are assigned to ensure that the weights can be dynamically adjusted according to the actual detection items. Image and text attention features are merged to make the extraction effect of cross-modal interaction feature data more consistent with reality. Among them, the cross-modal interaction feature data Att... sum It can be calculated using the following formula 3, where α and β are the first cross-modal attention feature data Att, respectively. tpv Second cross-modal attention feature data Att ptv The weight.
[0040] Formula 3: Att sum =α×Att tpv +β×Att ptv It should be understood that, for Figure 2 The cross-modal feature processing shown can select an appropriate multi-layer multi-head attention mechanism for computation based on the actual detection target, and then fuse them after computation. The number of layers and heads can be dynamically adjusted according to the actual situation.
[0041] In some embodiments, the manufacturing quality inspection method described above may further include: performing data augmentation on the multimodal data of the case sample to obtain augmented multimodal data; and performing cross-modal feature processing on the augmented multimodal data to obtain cross-modal interaction feature data of the case sample.
[0042] This application's embodiments, through data augmentation, can increase the case database at low cost while enhancing the data's expressiveness, improving the model's generalization ability, and reducing the possibility of identification errors due to different data representation methods. The data augmentation method may differ for different data modalities, and in practical applications, it can be flexibly set according to requirements.
[0043] For example, such as Figure 3As shown, data augmentation for text data can include, but is not limited to, at least one of the following: synonym replacement, sentence restructuring, and definition generation. For example, "internal component positioning is accurate" can be augmented to "internal component positioning is accurate, with no risk of misalignment or loosening," and "welding quality is reliable" can be augmented to "the internal structure of the sample is normal, and the welding quality is reliable." By augmenting text data, as many language expressions as possible can be transformed, ensuring that in the subsequent quality inspection process, situations such as insufficient feature extraction or label recognition errors caused by different expressions can be reduced.
[0044] For example, such as Figure 3 As shown, data augmentation for image data can include, but is not limited to, at least one of the following: view expansion, rotation, cropping, scaling, brightness adjustment, and blur adjustment. For example, it can reduce image brightness, increase image blur, rotate the image at random angles, or crop the image. Through image data augmentation, insufficient feature extraction and recognition errors caused by factors such as angle, size, brightness, and sharpness can be reduced.
[0045] For example, such as Figure 3 As shown, data augmentation for indicator data can include, but is not limited to, at least one of the following: data rewriting with equivalent values, data unit conversion, etc. For example, rewriting 10,000 as 10. 4 Data augmentation of indicator data aims to cover as many common ways of expressing data as possible, thus avoiding issues such as abnormal feature extraction and label recognition errors caused by different data formats during subsequent quality inspection.
[0046] After augmenting the multimodal data of the case samples, cross-modal feature processing can be performed on the augmented multimodal data to obtain cross-modal interaction feature data of the case samples. The cross-modal feature processing for the case samples can be performed in the same manner as the cross-modal feature processing for the test object to ensure consistency.
[0047] Based on this, the above steps involve cross-modal feature processing of the augmented multimodal data to obtain cross-modal interaction feature data of the case samples. This can include: performing feature encoding on the text data, image data, and indicator data in the augmented multimodal data to obtain text-encoded data, image-encoded data, and indicator-encoded data, respectively; performing linear transformation on the text-encoded data, image-encoded data, and indicator-encoded data, respectively, to obtain text feature data, image feature data, and indicator feature data; and performing attention mechanism calculation based on the text feature data, image feature data, and indicator feature data to obtain the cross-modal interaction feature data of the case samples. The process involves calculating cross-modal interaction feature data for the case sample using attention mechanisms based on text feature data, image feature data, and indicator feature data. This calculation may include: calculating first cross-modal attention feature data using text feature data as the query vector, image feature data as the key vector, and indicator feature data as the value vector; calculating second cross-modal attention feature data using image feature data as the query vector, text feature data as the key vector, and indicator feature data as the value vector; and finally, weighted summing of the first and second cross-modal attention feature data to obtain the cross-modal interaction feature data for the case sample. For further details regarding the cross-modal feature processing of the case sample, please refer to the above embodiments and the above... Figure 2 I won't go into details here.
[0048] In some embodiments, the step S200 above, which determines the target sample and reference sample from multiple case samples based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in the case database, may include: inputting the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples into the feature analysis model so that the feature analysis model can determine the target sample and reference sample from multiple case samples.
[0049] The feature analysis model can be a sample retrieval and matching model built based on a vector space similarity measurement mechanism. It is used to quickly locate target samples in a case database that are similar in manufacturing quality to the object under test, and to select reference samples for comparison. For example, the feature analysis model can be an AI model. This application does not limit the specific type of feature analysis model; in practical applications, it can be flexibly selected based on requirements. For example, the feature analysis model can include, but is not limited to, Tongyi Qianwen, DeepSeek, LLaMA, GPT, etc.
[0050] The feature analysis model processes cross-modal interaction feature data in the following ways: retrieval, recall, similarity calculation, and re-ranking. Retrieval refers to quickly locating candidate sample subsets from the full set of case samples in the case database based on a vector index structure, narrowing the scope of subsequent precise comparisons and reducing computational overhead. Recall involves extracting the Top-K case samples from the candidate sample subset whose vector distance to the cross-modal interaction feature data of the target object is less than a preset distance, serving as preliminary matching case samples to ensure that highly similar case samples are not missed. Similarity calculation involves accurately calculating the similarity score between the recalled case samples and the target object's cross-modal interaction feature data, quantifying the degree of similarity between their manufacturing quality. Re-ranking involves sorting the recalled case samples according to the similarity score, identifying the case sample with the highest score as the target sample to determine the quality detection label for the target object; simultaneously, from case samples belonging to the same object type and with normal manufacturing quality as the target object, the case sample with the highest similarity to the target object is selected as a reference sample to generate improvement suggestions.
[0051] This application embodiment achieves efficient and accurate matching of target samples and reference samples from a massive case database by sequentially performing retrieval, recall, similarity calculation and rearrangement through a feature analysis model, providing data support for the subsequent generation of quality detection labels and improvement suggestions for the objects to be tested.
[0052] In some embodiments, the manufacturing quality inspection method described above can be implemented based on a lightweight model, which is obtained by distilling a large, trained model. By distilling the large, trained model, the number of model parameters and computational resource overhead can be reduced while maintaining the accuracy of manufacturing quality inspection. The resulting lightweight model is more suitable for deployment on mobile devices or ordinary PCs.
[0053] Based on this, the large model can be used to build a case database, pre-compute and store cross-modal interaction feature data of each case sample, and has the characteristics of large number of parameters, complex network structure and strong representation ability; the lightweight model uses the large model as the teacher model and learns the core detection capabilities of the large model through distillation.
[0054] The distillation process includes at least one of the following: feature extraction distillation, attention pruning, and sample selection distillation. Feature extraction distillation learns the process by which a large model encodes features and performs linear transformations on multimodal data to generate feature data. This allows a lightweight model to output feature representations semantically aligned with the large model with a simplified network structure, achieving parameter compression and forward shifting of capabilities in the encoding layer. Attention pruning learns the process by which a large model calculates and fuses feature data from different modalities through an attention mechanism to generate cross-modal interactive feature data. This allows a lightweight model to output cross-modal interactive feature data with a reduced number of attention layers and heads, aligning with the distribution of the large model, achieving structural lightweighting and knowledge preservation in the fusion layer. Sample selection distillation learns the process by which a large model performs similarity calculations and sample selection based on cross-modal interactive feature data to determine target and reference samples. This allows a lightweight model to output selection results aligned with the large model's results with a simplified index structure and scoring mechanism, achieving efficiency optimization and performance approximation of the retrieval layer. During model distillation, the weighted summation process of cross-modal attention feature data can be omitted to allow for dynamic adjustment of the image and text attention weights based on the items to be detected later.
[0055] Please see Figure 4 , Figure 4 This is a schematic diagram of a distillation process provided in an embodiment of this application. For example... Figure 4 As shown, for feature encoding and linear transformation of multimodal data, the feature data (Et·Wt, Ep·Wp, Ev·Wv) output by the large learning model is distilled and labeled as (T, P, V). This simplifies the feature encoding and linear transformation processes, thereby reducing the processing steps and parameters. The large learning model calculates cross-modal attention feature data (Att) through an attention mechanism. tpv Att ptv ), and label the feature as (Att) tpv ', Att ptv This approach prunes the model by simplifying the computation of cross-modal attention feature data. The weighted summation of cross-modal attention feature data is not distilled, allowing for dynamic adjustment of the image-text attention weights based on the target item. For the retrieval, recall, similarity calculation, ranking, and output processes, pruning is learned through feature vector space alignment, dynamic adjustment of hidden layers, and hard-negative example distillation recognition, reducing model parameters while achieving more accurate recognition results. Based on this, by learning the features of each step in the large model's operation through distillation, a lightweight model can be obtained for use in actual quality inspection processes to identify quality inspection labels for target objects and provide improvement suggestions.
[0056] In some embodiments, the above manufacturing quality inspection method may further include: if the manufacturing quality inspection result of the object to be tested meets the target conditions, then the object to be tested is used as a candidate sample; the multimodal data of at least one candidate sample is input into a large model so that the large model outputs the manufacturing quality inspection result of the candidate sample; if the manufacturing quality inspection result of the candidate sample output by the lightweight model does not match the manufacturing quality inspection result of the candidate sample output by the large model, then the candidate sample is used as an incremental sample; and the lightweight model is corrected based on at least one incremental sample.
[0057] This application does not limit the specific content of the target conditions in its embodiments. In practical applications, they can be flexibly set according to requirements. For example, the target conditions may include, but are not limited to, at least one of the following: the confidence level of the manufacturing quality inspection result is lower than a preset threshold, the manufacturing quality inspection result is reported as an error by the user, and the object type of the object to be tested is different from the object type of any case sample in the case database. If the manufacturing quality inspection result of the object to be tested meets the target conditions, the object to be tested is used as a candidate sample, and candidate samples can be collected during the actual quality inspection process.
[0058] The multimodal data of candidate samples are processed using a large model, such as by executing steps S100 to S300 above, to obtain the manufacturing quality inspection results of the candidate samples output by the large model. The manufacturing quality inspection results of the candidate samples output by the lightweight model are compared with those output by the large model. If they do not match (e.g., there is a difference or the difference is too large), the candidate samples are used as incremental samples. In the actual quality inspection process, incremental samples can be collected. These incremental samples are used as difficult examples to incrementally distill the lightweight model, correcting its parameters, such as freezing stable layer parameters or fine-tuning layer parameters with large differences.
[0059] In this embodiment, the manufacturing quality inspection results actually output by the lightweight model are periodically compared with the manufacturing quality inspection results output by the large model in order to periodically optimize the pruning process and learning parameters and ensure the accuracy of the lightweight model.
[0060] Please see Figure 5 , Figure 5 This is a schematic diagram of a manufacturing quality inspection method provided in an embodiment of this application. This manufacturing quality inspection method can be used for manufacturing quality inspection in various application fields such as vehicles, aircraft, ships, robots, and home appliances; however, this embodiment of the application does not limit its application to these fields.
[0061] like Figure 5As shown, in the quality inspection workshop, multimodal data of case samples and test objects can be organized according to needs. For case samples, historical quality inspection cases can be organized into (text data, image data, indicator data) formats, and the multimodal data of test objects can be organized in the same format. Quality inspection labels also need to be organized for case samples. For example, for a specific case sample, the organized text data may include: visual inspection of the battery appearance as normal, internal condition as normal, welding as correct, wiring as correct, etc.; image data may include: battery appearance images, battery pack images, welding images, wiring images, etc.; indicator data may include: battery voltage sequence, current sequence, battery temperature sequence, etc. during charging and discharging; and quality inspection labels may include battery overvoltage.
[0062] like Figure 5 As shown, multimodal data of case samples can be augmented to increase the case database at low cost, while enhancing the expressiveness of the data, improving the generalization ability of the model, and reducing the possibility of identification errors due to different data representation methods.
[0063] like Figure 5 As shown, for case samples, cross-modal feature processing can be performed on their multimodal data to obtain cross-modal interaction feature data of the case samples; for the test object, the same cross-modal feature processing can also be used to obtain the cross-modal interaction feature data of the test object; through cross-modal feature processing, text data, image data, indicator data and their corresponding relationships can be interacted, fused and completed.
[0064] like Figure 5 As shown, through a large model (including the aforementioned feature analysis model), cross-modal interaction feature data of multiple case samples and cross-modal interaction feature data of the test object are retrieved, recalled, similarity calculated, and rearranged to screen target samples that are highly similar to the test object. Based on the quality inspection labels of the target samples, the quality inspection labels of the test object are determined. At the same time, reference samples with normal manufacturing quality are screened, and the reference samples and the test object are compared to generate improvement suggestions for the test object.
[0065] like Figure 5 As shown, for a large model that has completed training, a lightweight model with lower resource overhead can be distilled out. Distillation processes can include, but are not limited to, feature extraction distillation, attention pruning, and sample selection distillation. The distilled lightweight model can be deployed on mobile devices and ordinary PCs to output quality detection labels and improvement suggestions based on multimodal data during actual quality inspection. At the same time, the lightweight model can be periodically compared with the output of the large model and learned to correct the parameters of the lightweight model.
[0066] related Figure 5For further descriptions of the steps and their beneficial effects in the embodiments, please refer to the above embodiments; they will not be repeated here.
[0067] In summary, the improvements and beneficial effects of the embodiments of this application include at least the following aspects: First, it acquires all information about the object under test, including text data, image data, and indicator data, to ensure that the detection information is comprehensive and reliable, and can simulate manual detection to conduct a comprehensive detection of the object under test. Secondly, augmenting the case samples before using them in the case database can not only reduce the limitations of the initial data collection process, but also increase the amount of data, ensuring a significant increase in the number of case samples. Third, multimodal data includes not only image data, but also text data and indicator data. These information do not exist in isolation, but are interconnected and complementary. Therefore, when the attention mechanism is calculated, text feature data and image feature data are used as the query and key in turn, while indicator feature data is always used as the value. This ensures that when processing cross-modal features, image data, text data, and indicator data can focus on and locate each other's key information and make supplements. Fourth, through feature analysis models, the cross-modal interaction feature data of case samples and the cross-modal interaction feature data of the target object are retrieved, recalled, similarity calculated, and rearranged to screen case samples that are highly similar to the target object, ensuring that reasonable quality detection labels are output and providing improvement suggestions. Fifth, after training the large model, the large model is distilled in layers and pruned to train a lightweight model with fewer parameters and less resource consumption that can achieve the same function. This lightweight model can be deployed to mobile and PC. The lightweight model only needs to be corrected periodically based on the large model, without consuming a lot of resources all the time. Sixth, the process of weighted summation of cross-modal attention feature data extracted by two different methods does not involve distillation or pruning, which makes it easy to adjust the weights for different detection items at any time, ensuring that the importance of images and text can be adjusted at any time.
[0068] Please see Figure 6 , Figure 6 This is a schematic diagram of a manufacturing quality inspection device provided in an embodiment of this application. This manufacturing quality inspection device can be used to perform the manufacturing quality inspection method described in the above embodiments. Figure 6 As shown, the manufacturing quality inspection device 600 may include: a cross-modal feature processing module 610, a case sample screening module 620, and a manufacturing quality determination module 630.
[0069] The cross-modal feature processing module 610 is used to: perform cross-modal feature processing on the multimodal data of the test object to obtain cross-modal interaction feature data of the test object; wherein, the multimodal data includes text data, image data, and indicator data used to indicate the manufacturing quality of the test object.
[0070] The case sample screening module 620 is used to: determine target samples and reference samples from multiple case samples based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in the case database; wherein, the manufacturing quality of the target sample is similar to that of the object under test, and the reference sample belongs to the same object type as the object under test and the manufacturing quality of the reference sample is normal.
[0071] The manufacturing quality determination module 630 is used to determine the manufacturing quality inspection result of the object under test based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object under test.
[0072] In some embodiments, the cross-modal feature processing module 610 is further configured to: encode the text data, image data, and indicator data in the multimodal data of the test object to obtain text-encoded data, image-encoded data, and indicator-encoded data respectively; perform linear transformations on the text-encoded data, image-encoded data, and indicator-encoded data respectively to obtain text feature data, image feature data, and indicator feature data; and perform attention mechanism calculations based on the text feature data, image feature data, and indicator feature data to obtain cross-modal interaction feature data of the test object.
[0073] In some embodiments, the cross-modal feature processing module 610 is further configured to: calculate first cross-modal attention feature data using text feature data as a query vector, image feature data as a key vector, and indicator feature data as a value vector; calculate second cross-modal attention feature data using image feature data as a query vector, text feature data as a key vector, and indicator feature data as a value vector; and perform a weighted summation of the first cross-modal attention feature data and the second cross-modal attention feature data to obtain cross-modal interaction feature data of the object to be tested.
[0074] In some embodiments, the case sample screening module 620 is further configured to: input the cross-modal interaction feature data of the object to be tested, and the cross-modal interaction feature data of multiple case samples, into the feature analysis model, so that the feature analysis model can determine the target sample and the reference sample from the multiple case samples; wherein, the processing performed by the feature analysis model on the cross-modal interaction feature data includes: retrieval, recall, similarity calculation, and rearrangement.
[0075] In some embodiments, such as Figure 7As shown, the manufacturing quality inspection device 600 also includes a model distillation module 640, which is used to distill the large model that has been trained to obtain a lightweight model. The cross-modal feature processing module 610, the case sample screening module 620 and the manufacturing quality determination module 630 are applied to the lightweight model. The distillation process includes at least one of the following: feature extraction distillation, attention pruning and sample screening distillation.
[0076] In some embodiments, such as Figure 7 As shown, the above-mentioned model distillation module 640 is further configured to: if the manufacturing quality inspection result of the object to be tested meets the target conditions, then use the object to be tested as a candidate sample; input the multimodal data of at least one candidate sample into the large model so that the large model outputs the manufacturing quality inspection result of the candidate sample; if the manufacturing quality inspection result of the candidate sample output by the lightweight model does not match the manufacturing quality inspection result of the candidate sample output by the large model, then use the candidate sample as an incremental sample; and correct the lightweight model based on at least one incremental sample.
[0077] In some embodiments, such as Figure 7 As shown, the manufacturing quality inspection device 600 also includes a data augmentation module 650, used for: augmenting the multimodal data of the case sample to obtain augmented multimodal data; and performing cross-modal feature processing on the augmented multimodal data to obtain cross-modal interaction feature data of the case sample. Data augmentation for text data includes at least one of the following: synonym replacement, sentence restructuring, and definition generation; data augmentation for image data includes at least one of the following: view expansion, rotation, cropping, scaling, brightness adjustment, and blur adjustment; data augmentation for indicator data includes at least one of the following: data rewriting and data unit conversion.
[0078] For further descriptions of the functions and beneficial effects of each module in the aforementioned manufacturing quality inspection device, please refer to the embodiments of the aforementioned manufacturing quality inspection method; these will not be elaborated upon here.
[0079] This application also provides a computer-readable storage medium storing a computer program or instructions. When the computer program or instructions are executed by a processor, they implement the above-described manufacturing quality inspection method and have all the beneficial effects of the above-described manufacturing quality inspection method. Further details will not be elaborated here.
[0080] This application also provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, they implement the above-described manufacturing quality inspection method and have all the beneficial effects of the above-described manufacturing quality inspection method. Further details will not be elaborated here.
[0081] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program or instructions; the processor executes the computer program or instructions in the memory to implement the steps of the above-described manufacturing quality inspection method. This electronic device has all the beneficial effects of the above-described manufacturing quality inspection method, which will not be elaborated further here.
[0082] Computer-readable storage media can be, for example, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof, without particular limitation herein. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0083] In some embodiments, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.
[0084] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0085] Computer program code for performing operations of some embodiments of this application can be written in one or more programming languages or a combination thereof. These programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a Local Area Network (LAN) or a Wide Area Network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function.
[0087] It should also be noted that in some alternative implementations, the functions marked in the box may occur in a different order than those marked in the attached figures.
[0088] For example, two consecutively represented blocks can actually be executed in substantially parallel order, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, as well as combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.
[0089] The units described in some embodiments of this application can be implemented in software or in hardware. The described units can also be located in a processor.
[0090] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.
[0091] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0092] The embodiments, implementation methods, and related technical features of this application can be combined and substituted for each other without conflict.
[0093] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments and may not be to scale. The modules or processes shown in the drawings are not necessarily essential for implementing this application and therefore should not be used to limit the scope of protection of this application.
[0094] The manufacturing quality inspection method, apparatus, electronic device, and storage medium provided in the embodiments of this application have been described in detail above, and specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the technical solutions and core ideas of this application. Those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A manufacturing quality inspection method, characterized in that, The method includes: Cross-modal feature processing is performed on the multimodal data of the test object to obtain the cross-modal interaction feature data of the test object; wherein, the multimodal data includes text data, image data, and indicator data used to indicate the manufacturing quality of the test object; Based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in the case database, a target sample and a reference sample are determined from the multiple case samples; wherein, the manufacturing quality of the target sample is similar to the manufacturing quality of the object under test, and the reference sample belongs to the same object type as the object under test and the manufacturing quality of the reference sample is normal; Based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object to be tested, the manufacturing quality inspection result of the object to be tested is determined.
2. The method according to claim 1, characterized in that, The multimodal data of the object under test is subjected to cross-modal feature processing to obtain the cross-modal interaction feature data of the object under test, including: The text data, image data, and indicator data in the multimodal data of the object under test are respectively feature-encoded to obtain text-encoded data, image-encoded data, and indicator-encoded data; Linear transformations are performed on the text encoded data, the image encoded data, and the indicator encoded data respectively to obtain text feature data, image feature data, and indicator feature data. Based on the text feature data, the image feature data, and the indicator feature data, an attention mechanism is used to calculate the cross-modal interaction feature data of the object under test.
3. The method according to claim 2, characterized in that, The step of calculating the cross-modal interaction feature data of the test object based on the text feature data, the image feature data, and the indicator feature data using an attention mechanism includes: Using the text feature data as the query vector, the image feature data as the key vector, and the indicator feature data as the value vector, calculate the first cross-modal attention feature data; Using the image feature data as the query vector, the text feature data as the key vector, and the indicator feature data as the value vector, calculate the second cross-modal attention feature data; The first cross-modal attention feature data and the second cross-modal attention feature data are weighted and summed to obtain the cross-modal interaction feature data of the object under test.
4. The method according to claim 1, characterized in that, The step of determining the target sample and reference sample from the multiple case samples based on the cross-modal interaction feature data of the test object and the cross-modal interaction feature data of multiple case samples in the case database includes: The cross-modal interaction feature data of the object under test, and the cross-modal interaction feature data of multiple case samples are input into the feature analysis model so that the feature analysis model can determine the target sample and the reference sample from the multiple case samples; The processing performed by the feature analysis model on the cross-modal interaction feature data includes, in sequence: retrieval, recall, similarity calculation, and rearrangement.
5. The method according to claim 1, characterized in that, The method is based on a lightweight model, which is obtained by distilling a large model that has been trained. The distillation process includes at least one of the following: feature extraction distillation, attention pruning, and sample screening distillation.
6. The method according to claim 5, characterized in that, The method further includes: If the manufacturing quality inspection result of the object to be tested meets the target conditions, then the object to be tested will be used as a candidate sample. The multimodal data of at least one of the candidate samples is input into the large model so that the large model outputs the manufacturing quality inspection results of the candidate samples; If the manufacturing quality inspection result of the candidate sample output by the lightweight model does not match the manufacturing quality inspection result of the candidate sample output by the large model, then the candidate sample is used as an incremental sample. The lightweight model is modified based on at least one of the incremental samples.
7. The method according to claim 1, characterized in that, The method further includes: Data augmentation is performed on the multimodal data of the case samples to obtain augmented multimodal data; The augmented multimodal data is subjected to cross-modal feature processing to obtain the cross-modal interaction feature data of the case samples; The data augmentation for the text data includes at least one of the following: synonym replacement, sentence restructuring, and definition generation; the data augmentation for the image data includes at least one of the following: view expansion, rotation, cropping, scaling, brightness adjustment, and blur adjustment; the data augmentation for the indicator data includes at least one of the following: data rewriting and data unit conversion.
8. A manufacturing quality inspection device, characterized in that, The device includes: A cross-modal feature processing module is used to: perform cross-modal feature processing on the multimodal data of the test object to obtain cross-modal interaction feature data of the test object; wherein, the multimodal data includes text data, image data, and indicator data used to indicate the manufacturing quality of the test object; The sample acquisition module is used to: determine a target sample and a reference sample from multiple case samples based on the cross-modal interaction feature data of the object under test and the cross-modal interaction feature data of multiple case samples in the case database; wherein the manufacturing quality of the target sample is similar to the manufacturing quality of the object under test, and the reference sample belongs to the same object type as the object under test and the manufacturing quality of the reference sample is normal; The manufacturing quality determination module is used to: determine the manufacturing quality inspection result of the object under test based on the quality inspection label of the target sample, the multimodal data of the reference sample, and the multimodal data of the object under test.
9. An electronic device, characterized in that, include: A memory on which computer programs or instructions are stored; A processor for executing the computer program or instructions in the memory to implement the manufacturing quality inspection method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program or instructions thereon, which, when executed by a processor, implement the manufacturing quality inspection method as described in any one of claims 1 to 7.