Multi-mode large model automobile part defect detection method
By combining a multimodal large model with visual and textual cue vectors and optimizing the visual cue vectors, the problem of machine vision systems struggling to identify defects in automotive parts under complex working conditions is solved, achieving higher recognition accuracy and system stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2026-03-16
- Publication Date
- 2026-05-08
AI Technical Summary
Existing machine vision systems struggle to accurately identify defects in automotive parts under complex conditions, especially in situations with uneven lighting, strong reflections, or shadows. This can lead to some defects not being accurately identified or normal textures being mistakenly identified as defects.
By employing a multimodal large model and combining visual and textual cue vectors, and through visual guidance and textual supplementation, the visual cue vectors are optimized to improve the accuracy of defect identification.
It improves the accuracy and stability of multimodal large model in identifying defects in automotive parts, enhances adaptability to complex working conditions, and ensures the accuracy of defect identification and the operational stability of the system.
Smart Images
Figure CN121998971A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of component defect detection technology, and in particular to a method for detecting defects in multimodal large-scale automotive components. Background Technology
[0002] In modern automotive manufacturing, the quality of auto parts directly affects the performance, safety, and reliability of the entire vehicle. With the rapid development of the automotive industry, consumers' demands for vehicle quality are increasing, prompting automakers to implement increasingly stringent quality control over auto parts. During the production process, various defects inevitably arise due to the combined effects of raw material characteristics, fluctuations in processing parameters, and equipment wear, among which surface defects are particularly common and critical. These defects not only affect the aesthetic appearance of the parts but, more importantly, can lead to functional failure, thereby causing safety hazards and shortening the product's lifespan.
[0003] Most widely used machine vision systems currently rely solely on optical imaging to acquire two-dimensional image information of the surface of parts for defect analysis. Although high-resolution cameras can capture fine surface details, in complex real-world conditions such as uneven lighting, strong reflections, and shadow occlusion, feature extraction based solely on vision is easily interfered with, leading to the inaccurate identification of some defects or misclassification of normal textures as defects, thus requiring improvement. Summary of the Invention
[0004] To improve the accuracy of defect detection and identification, this application provides a multimodal large-scale automotive component defect detection method.
[0005] This application provides a multimodal large-scale automotive component defect detection method, which adopts the following technical solution: Multimodal large-scale automotive component defect detection methods include: Real car component images and visual cue vectors are input into a multimodal large model. The multimodal large model processes the input component images based on the input visual cue vectors to obtain image feature data. The text cue vector and task-related category labels are input into the multimodal large model. The multimodal large model processes the category labels based on the text cue vector to obtain text feature data. The extracted image feature data is matched with the text feature data to obtain the predicted defect data of the automotive parts images; The system acquires real defect data from component images and compares the predicted defect data with the real defect data based on the built-in loss function to calculate the prediction error data. The visual cue vector is calculated based on the prediction error data to obtain the gradient data of the corresponding visual cue vector. The visual cue vector is then corrected based on the gradient data until the corrected cue vector can accurately assist the multimodal large model in identifying defects in automotive parts. The well-trained multimodal large model is denoted as the defect detection model. Image data of the automotive parts to be identified is acquired and input into the defect detection model. The defect detection model identifies the image data, obtains defect identification data, and outputs it.
[0006] Preferably, the learnable visual cue vector is initialized to obtain an initial visual vector; The input component images are segmented to obtain a sequence of image blocks; The initial visual vector is concatenated with the image patch sequence to obtain composite image data; The composite image data is input into the visual encoder, which guides the encoder's attention based on the initial visual vector to obtain the guided image. Image features are then extracted from the guided image to obtain the guided image feature data.
[0007] Preferably, the image patch sequence is labeled based on the initial visual vector to obtain an important distribution map of the image patch sequence; Image features are extracted from non-key areas in the important distribution map to obtain basic feature data; Image features are extracted from key areas within important distribution regions to obtain second feature data; The second feature data is filtered based on the basic feature data to obtain image features unique to the key area, and these image features are marked as the image feature data of the guide image.
[0008] Preferably, the text prompt vector is initialized to obtain an initial text vector, and the initial text vector is matched and concatenated with the category label to obtain composite label data; The text encoder refines and supplements the category labels based on the initial text vector in the composite label data to obtain detailed label data; Text feature extraction is performed on the detailed labels to obtain text feature data.
[0009] Preferably, image feature data is matched based on text feature data to determine the similarity between each text feature data and image feature data, thereby obtaining similar data; By using the category labels corresponding to text feature data with similar data as the defects of the corresponding component images, the predicted defect data of the corresponding automotive component images can be obtained.
[0010] Preferably, the predicted defect data is compared with the actual defect data to determine whether there is actual defect data in the predicted defect data of the component image data. If real defect data is determined to exist, then similar data in the predicted defect data are compared to determine whether the similar data is greater than the built-in evaluation threshold. If the similarity data is determined to be less than the built-in evaluation threshold or there is no real defect data, then the error between the predicted defect data and the real defect data is calculated based on the built-in loss function to obtain the prediction error data.
[0011] Preferably, if the number of similar data is greater than the built-in evaluation threshold, the initial visual vector corresponding to the image of the component is labeled to obtain labeled vector data; The number of defects in the component images is judged. If only real defects are present, the marker vector data is not corrected. If there are multiple defects, then the non-real defects among the multiple predicted defects are recorded as defect data to be corrected. The defect data to be corrected is compared with the actual defect data to determine the data values of the indicator items that are similar between the defect data to be corrected and the actual defect data, thus obtaining the indicator data to be corrected. Based on the interaction between cue terms and indicator terms in the multimodal large model and the indicator data to be corrected for the corresponding indicator terms, data prediction is performed on the labeled vector data to obtain the corresponding pseudo gradient data, and the labeled vector data is adjusted based on the pseudo gradient data.
[0012] Preferably, when it is determined that there is an error between the predicted defect data and the actual defect data, a corresponding visual cue vector is determined based on the index item with the difference value, and recorded as the cue item; Obtain the interaction relationship between cue items and indicator items in the multimodal large model, and perform data prediction on cue items based on the interaction relationship and the prediction error data of indicator items to obtain gradient data.
[0013] In summary, this application includes at least one of the following beneficial technical effects: 1. By using textual cue vectors to transform manually set category labels, the multimodal model's understanding of defect categories is improved, thereby enhancing defect identification accuracy. Visual cue vectors are used to visually guide the multimodal model, directing the encoder's attention to regions containing defect features, ensuring image recognition accuracy. The model's prediction error is determined by comparing the identified defect results with actual defect results. Based on this prediction error, the visual cue vectors are iteratively optimized to further improve the accuracy of the multimodal model in identifying automotive component defects. Simultaneously, adjusting the visual cue vectors ensures both the stability of the multimodal model's operation and enhances its auxiliary recognition effect, thus improving defect identification accuracy. 2. The learnable visual cue vectors are initialized to randomly generate untrained initial visual vectors, thereby avoiding overfitting of the system to the visual cue vectors and improving the system's adaptability to actual needs, thus increasing its universality. By using the initial visual vectors to divide the image block sequence that needs image feature extraction into regions, image feature extraction is performed on both key and non-key regions after division. The secondary feature data of the key regions is filtered and selected based on the basic feature data of the non-key regions, so that the selected image features are more similar to the defect features indicated by the initial visual vectors, thereby improving the accuracy of defect identification by the multimodal large model. 3. By utilizing text feature data and image feature data to convert between text and images, the data dimension of the two types of feature data is guaranteed to be the same when comparing them. Through mutual conversion and comparison, the accuracy of similarity judgment is improved, thereby improving the accuracy of defect identification. Attached Figure Description
[0014] Figure 1 This is a flowchart illustrating the steps of the multimodal large-scale automotive component defect detection method in this embodiment. Detailed Implementation
[0015] The following is in conjunction with the appendix Figure 1 This application will be described in further detail.
[0016] This application discloses a method for detecting defects in multimodal large-scale automotive parts.
[0017] Example: Figure 1 As shown, the multimodal large-scale automotive component defect detection method of the present invention includes: S1, input real car component images and visual cue vectors into the multimodal large model. The multimodal large model performs data processing on the input component images based on the input visual cue vectors to obtain image feature data; S2, input the text cue vector and task-related category labels into the multimodal large model. The multimodal large model processes the category labels based on the text cue vector to obtain text feature data; S3, the extracted image feature data is matched with the text feature data to obtain the predicted defect data of the automotive parts image; S4: Obtain the actual defect data of the component image, and compare the predicted defect data obtained by judgment with the actual defect data based on the built-in loss function to calculate the prediction error data. S5. Calculate the visual cue vector based on the prediction error data to obtain the gradient data of the corresponding visual cue vector, and correct the visual cue vector according to the gradient data until the corrected cue vector can accurately assist the multimodal large model in identifying defects in automotive parts. The well-trained multimodal large model is denoted as the defect detection model. S6: Acquire image data of the automotive parts to be identified, and input the image data into the defect detection model. The defect detection model identifies the image data, obtains defect identification data, and outputs it.
[0018] In this embodiment, textual cue vectors are used to transform manually set category labels to help the multimodal model understand the accuracy of defect categories, thereby improving the accuracy of defect identification. Visual cue vectors are used to visually guide the multimodal large model to the region containing defect features, ensuring the accuracy of image recognition. By comparing the defect results identified by the model with the actual defect results, the prediction error of the model is determined. The visual cue vectors are then continuously iteratively optimized based on the prediction error to further improve the accuracy of the multimodal large model in identifying defects in automotive parts. At the same time, by adjusting the visual cue vectors, the stability of the multimodal large model operation is ensured, and the auxiliary recognition effect of the visual cue vectors on the multimodal large model is improved, thus improving the accuracy of defect identification.
[0019] For example, before identifying automotive parts that require defect identification, an initial multimodal large model is trained by building a database until it can accurately identify defects.
[0020] To ensure the operational stability of the multimodal large model, visual cue vectors are constructed to improve the accuracy of defect identification in the input image without altering the model's operating mechanism. These visual cue vectors provide visual guidance to the image recognition module within the multimodal large model, indicating which regions are more important and which areas should be prioritized for recognition.
[0021] Meanwhile, to help the system accurately understand what constitutes a defect, some common defect category labels are manually defined. Text prompt vectors are then used to process these labels, refining, supplementing, and correcting their semantics, thus helping the system correctly understand the defects. For example, regarding scratches, humans can visually observe a metal surface and process the information to understand whether it has scratches. However, machines cannot accurately understand what a scratch is and might misinterpret it as a "line," etc. Therefore, text prompt vectors are needed to transform the manually defined category labels into language semantics that the system can understand.
[0022] By utilizing textual cue vectors, the system is able to understand various defects, while visual cue vectors assist the system in recognizing images. The combination of these two approaches improves the accuracy of defect recognition in the multimodal large model.
[0023] Meanwhile, by comparing the identified defects with the actual defects, the prompting effect of the visual cue vector is verified. If they are different, it indicates that the prompting effect is not good. Therefore, by determining the difference between the actual result and the identification result, the data difference required for the identification result to reach the actual result is determined. Then, the visual cue vector is corrected based on the data difference, and the visual cue vector is continuously iterated and optimized to make the prompting effect of the visual cue vector more and more accurate, thereby improving the accuracy of multimodal large model recognition.
[0024] After continuously iterating the visual cue vectors to improve the accuracy of multimodal large model recognition, the actual image data to be recognized is input into the trained defect detection model to achieve accurate recognition of the image data.
[0025] In step S1, real images of car parts and visual cue vectors are input into the multimodal large model. The multimodal large model processes the input part images based on the input visual cue vectors to obtain image feature data, including the following steps: S11, Initialize the learnable visual cue vector to obtain the initial visual vector; S12, perform image segmentation on the input component image to obtain an image block sequence; S13, the initial visual vector is concatenated with the image patch sequence to obtain composite image data; S14, input the composite image data into the visual encoder. The visual encoder guides the encoder's attention based on the initial visual vector to obtain the guided image. Then, image features are extracted from the guided image to obtain the guided image feature data.
[0026] S141, Based on the initial visual vector, the image patch sequence is marked to obtain the importance distribution map of the image patch sequence; S142, perform image feature extraction on non-key areas in the important distribution map to obtain basic feature data; S143, Image features are extracted from key areas in important distribution areas to obtain second feature data; S144, using the basic feature data as a benchmark, the second feature data is filtered to obtain image features unique to the key area, and these image features are marked as the image feature data of the guide image.
[0027] In this embodiment, by initializing the learnable visual cue vectors to randomly generate untrained initial visual vectors, the system avoids overfitting to the visual cue vectors and improves the system's adaptability to actual needs, thus enhancing its universality. By using the initial visual vectors to divide the image block sequence that needs image feature extraction into regions, and simultaneously extracting image features from the divided key and non-key regions, the second feature data of the key regions is filtered and selected based on the basic feature data of the non-key regions. This makes the selected image features more closely resemble the defect features indicated by the initial visual vectors, improving the accuracy of the multimodal large model in defect identification.
[0028] For example, when performing visual recognition on the input image, firstly, visual cue vectors are randomly set. These visual cue vectors are a string of numbers, such as [P1, P2, P3, ..., Pn]. These n vectors are initialized to obtain the initial, untrained "cues," which are essentially some operation instructions randomly and without any pattern written onto a blank instruction card. Simultaneously, the images of the car parts are preprocessed and segmented into a sequence of image patches. Then, the visual cue vectors are used in the feature space along with the image features of the parts images in the Transformer layer to guide the encoder's attention to regions containing defect features (e.g., edges, texture anomalies).
[0029] This process does not modify the image itself, but rather modifies the importance and distribution of the encoder's regions of interest, thereby improving the accuracy of image feature recognition.
[0030] Because visual cue vectors have different focuses, there will be both key and non-key regions on a plane. Key regions mainly refer to areas containing defect features as indicated by the visual cue vectors, while non-key regions mainly refer to areas that do not contain defect features. Therefore, when performing image feature recognition on key regions, non-key regions are used as a benchmark to filter key regions, thereby ensuring that the filtered feature results are more similar to defect features, thus improving the accuracy of defect feature recognition.
[0031] In step S2, the text cue vector and task-related category labels are input into the multimodal large model. The multimodal large model processes the category labels based on the text cue vector to obtain text feature data, including the following steps: S21, the text prompt vector is initialized to obtain an initial text vector, and then matched and concatenated with the category label to obtain composite label data. The matching and concatenation involves matching different initial text vectors with the same category label to obtain corresponding composite label data. Subsequent refinement of the category label based on the initial text vector results in diverse label descriptions, facilitating the system's understanding of defects and improving the accuracy of the system's understanding of defect concepts. In short, a defect is described in text form.
[0032] S22, The text encoder refines and supplements the category labels based on the initial text vector in the composite label data to obtain detailed label data; S23, extract text features from detailed labels to obtain text feature data.
[0033] For example, after improving the accuracy of multimodal large model in identifying defects in component images through visual cue vectors, text cue vectors are then used to provide detailed explanations of the preset category labels of the defects to be detected. This allows the multimodal large model to understand what each defect specifically looks like in text form. For instance, if "flower" is identified as the target to be identified, the multimodal large model needs to be told what constitutes a flower; for example, a flower has five petals, a six-petaled flower has six points, etc. The model will also be told what petals are, including their curvature, convergence, color, texture, etc., enabling the multimodal large model to automatically generate the corresponding "image" based on the text description. In other words, the multimodal large model can truly understand each defect.
[0034] For example, regarding the defect "scratches," if the multimodal large model initially interprets scratches as "lines," then misjudgments may occur when there are stains or lines on the car parts. Therefore, it is necessary to supplement the defect "scratches" with textual prompts, such as adding reflectivity, dark areas, length, and shape, so that the multimodal large model can accurately distinguish between textures, stains, and scratches, thereby improving the accuracy of the multimodal large model in defect identification.
[0035] In step S3, the extracted image feature data is matched with the text feature data to obtain predicted defect data for automotive component images, including the following steps: S31, Match image feature data based on text feature data to determine the similarity between each text feature data and image feature data, and obtain similar data; S32, the category labels corresponding to the text feature data with similar data are used as the defects of the corresponding component images to obtain the predicted defect data of the corresponding automotive component images.
[0036] In this embodiment, by utilizing text feature data and image feature data to convert between text and images, the data dimension of the two types of feature data is ensured to be the same when comparing them. This mutual conversion and comparison improves the accuracy of similarity judgment, thereby improving the accuracy of defect identification. For example, if text A and image a have a similarity of 80%, it can be determined that the component has an 80% probability of having the defect corresponding to text A.
[0037] For example, text can be converted into corresponding images by utilizing text features, and then the images can be compared to determine the similarity between the two images. Alternatively, image features can be used to convert images into text descriptions, and then the texts can be compared to determine the similarity between the two texts, thus achieving a detection effect of "1+1>2".
[0038] In step S4, the actual defect data of the component images is obtained, and the predicted defect data obtained is compared with the actual defect data based on the built-in loss function to calculate the prediction error data, including the following steps: S41, compare the predicted defect data with the actual defect data to determine whether there is actual defect data in the predicted defect data of the component image data; S42, If it is determined that there is real defect data, then compare the similar data in the predicted defect data to determine whether the similar data is greater than the built-in evaluation threshold. S43. If the similar data is determined to be less than the built-in evaluation threshold or there is no real defect data, then the error between the predicted defect data and the real defect data is calculated based on the built-in loss function to obtain the predicted error data.
[0039] In this embodiment, by comparing predicted defect data with actual defect data, it is determined whether the multimodal large model in the current training state can correctly identify the actual defects. Once it is determined that the actual defects can be correctly identified, the recognition accuracy of the identified defects is determined, thereby determining the recognition effect of the current training state on the corresponding actual defects. When the recognition effect is poor, the difference between the predicted defect data and the actual defect data is calculated, and the relevant visual cue vectors are adjusted accordingly to continuously correct the auxiliary effect of the visual cue vectors on the multimodal large model, thereby improving recognition accuracy. Simultaneously, by correcting the visual cue vectors rather than adjusting the system parameters of the multimodal large model, the operational stability of the multimodal large model is ensured.
[0040] For example, after identifying defects in automotive component images using visual cue vectors and text cue vectors, the accuracy of the defects identified is verified by comparing them with actual defects. Since defects identified by visual cue vectors in an imperfect state may have varying degrees of similarity—for instance, defects A, B, and C may all be similar—and assuming the similarity of defect A is 80%, defect B is 40%, and defect C is 30%, then when comparing them with actual defects, defects A, B, and C are further compared... The comparison is performed as follows: Assuming the real defect is B, it is determined that the multimodal large model in the corresponding state can identify the defect correctly. Then, the similar data of the identified real defect are compared. Assuming the evaluation threshold is 80%, it indicates that the multimodal large model in the corresponding state has low recognition accuracy, and the corresponding visual cue vector has a poor guiding effect on defect B. Therefore, it is necessary to make an error judgment on defect B in the real situation based on the recognition result of defect B, so as to adjust the visual cue vector in reverse according to the judgment error, thereby continuously correcting it until the visual cue vector reaches the evaluation threshold.
[0041] Suppose that the identified defect B data is [a=5, b=4, c=3], and the actual defect B data is [a=10, b=10, c=10], then the corresponding prediction error data is [a error=5, b error=6, c error=7]. Since the visual cue vector is used to visually guide a multimodal large model, there is a certain data relationship between the data values of each vector in the visual cue vector and the actual identified data. Therefore, the visual cue vector is adjusted according to this data relationship and the corresponding prediction error data. Suppose that in the visual cue vectors, vector P1 corresponds to a, vector P2 corresponds to b, and vector P3 corresponds to c. Suppose that if vector P1 is adjusted by one unit, the prediction error data of the collected result will have a difference of 4. Suppose that vector P1 is adjusted by another unit, the prediction error data of the collected result will have a difference of 1. After adjusting vector P1 by another unit, the prediction error data of the collected result will have a difference of -7. This indicates that the adjustment result of vector P1 is excessive, so a callback is required. Through multiple training sessions, the final adjusted result can correctly identify defect B, that is, the data of the identified defect B is [a=10, b=10, c=10].
[0042] Step S42, if it is determined that there is real defect data, then the similar data in the predicted defect data are compared to determine whether the similar data is greater than the built-in evaluation threshold, and the following steps are also included: S421, If the similar data is determined to be greater than the built-in evaluation threshold, the initial visual vector corresponding to the component image is marked to obtain the marked vector data; S422: Determine the number of defects in the part image. If only real defects are present, do not correct the marker vector data. S423, if multiple defects are present, then the non-real defects among the multiple predicted defects are recorded as defect data to be corrected; where, a non-real defect is a defect that is not present in the image of the component. For example, if three defects A, B, and C are detected, and the actual defect on the component is defect A, then B and C are non-real defects.
[0043] S434, compare the defect data to be corrected with the actual defect data, determine the data values of the indicator items that are similar between the defect data to be corrected and the actual defect data, and obtain the indicator data to be corrected. S435, based on the interaction relationship between cue terms and indicator terms in the multimodal large model and the indicator data to be corrected for the corresponding indicator terms, performs data prediction on the labeled vector data to obtain corresponding pseudo-gradient data, and adjusts the labeled vector data based on the pseudo-gradient data. Specifically, the pseudo-gradient data is used to adjust for deviation, and the gradient data is used to adjust for convergence.
[0044] In this embodiment, when it is determined that there is real defect data and the similar data is greater than the built-in evaluation threshold, the number of defects identified by the component image is judged to determine whether there are interference items. If interference items are determined to exist, the defect data to be corrected is compared with the real defect data to clarify the correction index data that needs to be eliminated for each interference item. Based on the correction index data, the pseudo gradient data of the corresponding label vector data is determined to adjust the label vector data so that the adjusted label vector data can only accurately identify specific defects, reduce the interference of interference items, and improve the defect recognition accuracy of the multimodal large model.
[0045] For example, if real defect data is determined to exist, it indicates that the multimodal large model in the current training state can correctly identify the real defect. If the number of similar data identified is greater than the evaluation threshold, it indicates that the multimodal large model in the current training state has achieved the required accuracy in identifying the real defect. At this point, to further reduce the possibility of misjudgment, the number of defects identified in the predicted defect data is determined. Suppose the true defect is A. If the predicted defect data identifies only A, it indicates that the multimodal large model does not have other defects interfering with the identification of defect A. However, if the predicted defect data identifies defects including A, B, and C, then defects B and C are interfering defects. That is, when predicting defect A, they may be incorrectly identified as defects B or C. Therefore, in order to further reduce the interference of other defects, it is necessary to eliminate the misjudgments of defects B and C. The amount of data to be eliminated is the data currently possessed by defects B and C. Assuming the defect data of defect B is [a=1, b=1, c=1], then [a=1, b=1, c=1] needs to be eliminated to make [a=0, b=0, c=0], so as to further improve the accuracy of defect identification by the multimodal large model.
[0046] In step S5, the visual cue vector is calculated based on the prediction error data to obtain the gradient data of the corresponding visual cue vector, and the visual cue vector is corrected according to the gradient data, including the following steps: S51, when it is determined that there is an error between the predicted defect data and the actual defect data, the corresponding visual cue vector is determined based on the index item with the difference value and recorded as the cue item; S52: Obtain the interaction relationship between the cue items and the indicator items in the multimodal large model, and perform data prediction on the cue items based on the interaction relationship and the prediction error data of the indicator items to obtain gradient data.
[0047] For example, after determining that there is an error between the predicted defect data and the actual defect data, the degree of error in the prediction result is determined by using a loss function. For instance, the predicted error data is calculated as [a_error = 5, b_error = 6, c_error = 7]. Since the actual difference data and the data conversion relationship between the system and the system differ (e.g., the system obtains data B based on instruction A, while the actual requirement is data C), there is a data difference between B and C. In this case, how to convert the data difference between B and C into an adjustment to instruction A is the gradient data. That is, determining the adjustment direction and amount for each data item in the visual cue vector.
[0048] For example, if there is a positive relationship between indicator 'a' and cue 'P1', and we need to increase the value of indicator 'a', we might need to adjust the value of P1 or its offset angle. Through repeated adjustments, we can gradually improve the accuracy of the visual cue vector's cues, thereby enhancing the accuracy of defect identification.
[0049] Compared with existing multimodal large-scale automotive component defect detection methods, this invention improves the accuracy of defect identification.
[0050] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.
Claims
1. A multimodal large-scale automotive component defect detection method, characterized in that, include: Real car component images and visual cue vectors are input into a multimodal large model. The multimodal large model processes the input component images based on the input visual cue vectors to obtain image feature data. The text cue vector and task-related category labels are input into the multimodal large model. The multimodal large model processes the category labels based on the text cue vector to obtain text feature data. The extracted image feature data is matched with the text feature data to obtain the predicted defect data of the automotive parts images; The system acquires real defect data from component images and compares the predicted defect data with the real defect data based on the built-in loss function to calculate the prediction error data. The visual cue vector is calculated based on the prediction error data to obtain the gradient data of the corresponding visual cue vector. The visual cue vector is then corrected based on the gradient data until the corrected cue vector can accurately assist the multimodal large model in identifying defects in automotive parts. The well-trained multimodal large model is denoted as the defect detection model. Image data of the automotive parts to be identified is acquired and input into the defect detection model. The defect detection model identifies the image data, obtains defect identification data, and outputs it.
2. The multimodal large-scale automotive component defect detection method according to claim 1, characterized in that: The process involves inputting real automotive component images and visual cue vectors into a multimodal large model. The multimodal large model then processes the input component images based on the input visual cue vectors to obtain image feature data, including: The learnable visual cue vectors are initialized to obtain the initial visual vectors; The input component images are segmented to obtain a sequence of image blocks; The initial visual vector is concatenated with the image patch sequence to obtain composite image data; The composite image data is input into the visual encoder, which guides the encoder's attention based on the initial visual vector to obtain the guided image. Image features are then extracted from the guided image to obtain the guided image feature data.
3. The multimodal large-scale automotive component defect detection method according to claim 2, characterized in that: The composite image data is input into the visual encoder, which guides the encoder's attention based on the initial visual vector to obtain a guided image. Image features are then extracted from the guided image to obtain guided image feature data, including: Based on the initial visual vector, the image patch sequence is labeled to obtain the importance distribution map of the image patch sequence; Image features are extracted from non-key areas in the important distribution map to obtain basic feature data; Image features are extracted from key areas within important distribution regions to obtain second feature data; The second feature data is filtered based on the basic feature data to obtain image features unique to the key area, and these image features are marked as the image feature data of the guide image.
4. The multimodal large-scale automotive component defect detection method according to claim 1, characterized in that: The text cue vector and task-related category labels are input into the multimodal large model. The multimodal large model processes the category labels based on the text cue vector to obtain text feature data, including: The text prompt vector is initialized to obtain an initial text vector, and then the initial text vector is matched and concatenated with the category label to obtain composite label data; The text encoder refines and supplements the category labels based on the initial text vector in the composite label data to obtain detailed label data; Text feature extraction is performed on the detailed labels to obtain text feature data.
5. The multimodal large-scale automotive component defect detection method according to claim 1, characterized in that: The step of matching the extracted image feature data with text feature data to obtain predicted defect data for automotive component images includes: By matching image feature data with text feature data, the similarity between each text feature data and image feature data is determined, and similar data is obtained. By using the category labels corresponding to text feature data with similar data as the defects of the corresponding component images, the predicted defect data of the corresponding automotive component images can be obtained.
6. The multimodal large-scale automotive component defect detection method according to claim 1, characterized in that: The process involves acquiring real defect data from component images and comparing the predicted defect data with the real defect data based on a built-in loss function to calculate prediction error data, including: The predicted defect data is compared with the actual defect data to determine whether there is actual defect data in the predicted defect data of the component image data. If real defect data is determined to exist, then similar data in the predicted defect data are compared to determine whether the similar data is greater than the built-in evaluation threshold. If the similarity data is determined to be less than the built-in evaluation threshold or there is no real defect data, then the error between the predicted defect data and the real defect data is calculated based on the built-in loss function to obtain the prediction error data.
7. The multimodal large-scale automotive component defect detection method according to claim 6, characterized in that: If it is determined that there is real defect data, then comparing similar data in the predicted defect data to determine whether the similar data is greater than the built-in evaluation threshold also includes: If the number of similar data is greater than the built-in evaluation threshold, the initial visual vector corresponding to the image of the component is labeled to obtain labeled vector data. The number of defects in the component images is judged. If only real defects are present, the marker vector data is not corrected. If there are multiple defects, then the non-real defects among the multiple predicted defects are recorded as defect data to be corrected. The defect data to be corrected is compared with the actual defect data to determine the data values of the indicator items that are similar between the defect data to be corrected and the actual defect data, thus obtaining the indicator data to be corrected. Based on the interaction between cue terms and indicator terms in the multimodal large model and the indicator data to be corrected for the corresponding indicator terms, data prediction is performed on the labeled vector data to obtain the corresponding pseudo gradient data, and the labeled vector data is adjusted based on the pseudo gradient data.
8. The multimodal large-scale automotive component defect detection method according to claim 1, characterized in that: The step of calculating the visual cue vector based on the prediction error data to obtain the gradient data of the corresponding visual cue vector, and correcting the visual cue vector according to the gradient data, includes: When it is determined that there is an error between the predicted defect data and the actual defect data, the corresponding visual cue vector is determined based on the index item with the difference value and recorded as the cue item. Obtain the interaction relationship between cue items and indicator items in the multimodal large model, and perform data prediction on cue items based on the interaction relationship and the prediction error data of indicator items to obtain gradient data.