Industrial defect detection and cause analysis method, device, equipment and storage medium

Abstract

The application provides an industrial defect detection and cause analysis method, device, equipment and storage medium, and relates to the technical field of industrial component manufacturing. The method comprises the following steps: selecting a suitable visual language large model as a basic model; introducing a modal fusion module to efficiently fine-tune the model, the module fuses visual and text information by using cross attention and a neural network based on quaternion operation, so as to enhance the attention of the model to the defect area and the information utilization efficiency, and obtain a model with basic defect positioning and cause analysis capability; and collecting actual production line data to continuously incrementally learn the model, so that the model is adapted to a specific production environment and maintains the detection capability for new and original products. The application realizes integrated output of defect positioning and cause analysis, and improves the automation and intelligent level of industrial quality inspection.

Description

Technical Field

[0001] This application relates to the field of industrial component manufacturing technology, and in particular to an industrial defect detection and cause analysis method, apparatus, equipment and storage medium. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technology, industrial manufacturing is undergoing a transformation towards intelligence and digitalization. This places higher demands on the quality monitoring, defect detection, and root cause analysis of industrial products during the production process. Deep learning-based visual inspection technology has become a mainstream solution widely used in the industrial quality inspection field. Although existing intelligent vision-based industrial defect detection can effectively identify and locate defects in most industrial products, compared to dedicated, offline quality analysis systems, vision-based industrial defect detection methods typically lack the analysis of the causes of industrial defects. Furthermore, deep vision models lack strong textual output capabilities, failing to directly output effective semantic information to analyze defect causes, and consequently, failing to provide effective guidance or assistance to industrial production lines. These factors collectively restrict the development of higher-level intelligent industrial production systems. Therefore, current industrial defect detection technology needs to further enhance its defect detection, diagnosis, and defect root cause localization capabilities. Simultaneously identifying product problems on the production line, it should be able to quickly pinpoint issues in the production process and provide timely feedback to the operators and maintenance personnel on the production line, thereby improving the controllability of production quality and reducing the defect rate and labor maintenance costs for enterprises.

[0003] In recent years, large language model technology has developed rapidly, demonstrating unprecedentedly powerful language and comprehension capabilities. Visual language models, in particular, integrate visual and linguistic abilities, enabling them to perform various visual tasks such as visual classification, visual description, and object localization. They also possess strong linguistic capabilities, directly outputting the model's reasoning results in human-understandable text. Therefore, visual language models can be well integrated into industrial production, which is of great significance for building intelligent, systematic, automated, and digital smart industrial production lines. However, before deploying a visual language model into the actual production process, it needs efficient fine-tuning. Existing methods for efficient fine-tuning large models often only apply to one modality of the visual language model, lacking effective interaction and fusion of information across all modalities, often failing to achieve satisfactory results. For example, the model may lack holistic perception of industrial defects across both visual and linguistic modalities, resulting in inaccurate defect localization and identification, which significantly affects the model's correct analysis and judgment of defects.

[0004] In summary, existing technologies for industrial defect detection and cause analysis mainly suffer from the following problems: 1. The problem of information fragmentation across different modalities during efficient fine-tuning of large visual-language models: Existing efficient fine-tuning methods for large visual-language models often only target a single visual or linguistic modality, failing to fully utilize the joint information between different modalities. This leads to modality mismatch or low information utilization during fine-tuning, consequently affecting the model's accuracy in locating defects. Therefore, efficient fine-tuning necessitates fusing information from different modalities to extract the joint information from the data across these modalities.

[0005] 2. Redundancy and difficulty in extracting key information in multimodal fusion: Most existing multimodal information fusion methods simply integrate information from different modalities without efficiently distinguishing key and redundant information in the fused information. This results in high information redundancy, which negatively impacts the model's reasoning process and leads to unsatisfactory model performance. Summary of the Invention

[0006] This application provides a method, apparatus, equipment, and storage medium for industrial defect detection and cause analysis. Based on efficient fine-tuning of multimodal information fusion using a large visual-language model, it achieves the fusion of visual and linguistic modal information during the model's efficient fine-tuning process. This fused information is then re-injected into the visual encoder of the visual-language model, guiding the encoder to better focus on the location of defects in the workpiece, achieving more accurate defect identification and detection. Simultaneously, the model can also analyze defects and generate cause analyses, outputting analysis results described in industrial terminology. With the defect location information and diagnostic analysis information from the large visual-language model, operators on the production line can make more timely adjustments to industrial production, significantly improving production efficiency.

[0007] Further, a feature extraction mechanism for fused information driven by quaternion operations is designed. Quaternion operations possess strong orthogonality; by performing quaternion operations on the fused information, relevant attributes can be effectively extracted, while irrelevant information can be isolated. Incorporating quaternion operations into the modal fusion of the visual-language large-scale model helps the model better capture key elements in the fused information, reduces redundancy, and thus enhances the model's understanding of industrial defects.

[0008] In a first aspect, this application provides a method for industrial defect detection and cause analysis, including: The system was constructed, and a large visual language model was selected as the base model. The large visual language model is fine-tuned using a multimodal fusion mechanism, including: mapping the images and text inputs of the industrial defect dataset to visual tokens and text tokens; fusing the visual tokens and text tokens through a modal fusion module to generate a fused token; re-injecting the fused token into the visual encoder to guide it to focus on the defect region in the image; and updating the model parameters using an efficient fine-tuning method. A high-quality dataset is built based on actual production line data. The model is then incrementally trained and fine-tuned to enable it to master the knowledge of industrial defect detection and cause analysis from actual production lines, thereby achieving integrated output of accurate location of industrial product defects and cause tracing.

[0009] In one possible design, the visual token and the text token are fused through a modal fusion module to generate a fused token, including: Adjust the dimensions of the text token to match those of the visual token to obtain the dimension-mapped text token, denoted as . ; The initial fusion token is calculated using the following formula. ; In the formula, Represents a visual token. and This indicates the number and dimensions of the image tokens. Represents the normalized exponential function, This represents the matrix transpose operation; Neural networks based on quaternion operations, for Redundancy removal is performed to obtain the final fusion token.

[0010] In one possible design, a neural network based on quaternion operations is used for... Redundancy removal is performed to obtain the final fusion token; Applying the operational properties of quaternions to linear neural networks, the computational formula for neural networks based on quaternion operations is expressed as follows: In the formula, As the input to the neural network, This represents the weights of a linear layer based on quaternions, while This indicates the network output.

[0011] In one possible design, the operational properties of quaternions are determined as follows: Let a quaternion be represented as: ,in , , It represents the imaginary unit and satisfies: Quaternions The matrix representation is as follows: In the formula, r , x , y and z These represent the real components of the quaternion; first quaternion With the second quaternion The Hamiltonian product between them can be expressed as follows: In the formula, , , and Represent quaternions respectively The real number components; , , and Represent quaternions respectively The real number components.

[0012] In one possible design, the number of modality fusion modules is 4 to 8, each connected to a different intermediate layer of the visual encoder.

[0013] In one possible design, the large model of the visual language is a Qwen2.5-VL or Qwen3-VL series model with 2B to 7B parameters.

[0014] In one possible design, the method of incrementally learning and fine-tuning the model based on actual production line data to build a high-quality dataset includes: collecting images and equipment parameter data from the actual production line; labeling the location, type, and cause of defects; and fine-tuning the model by mixing the new data with the original data in proportion.

[0015] Secondly, this application provides an industrial defect detection and cause analysis device, the device comprising: The basic building block is configured to build the system and selects a large visual language model as the base model. The model fine-tuning module is configured to fine-tune the large visual language model using a multimodal fusion mechanism, including: mapping the image and text input of the industrial defect dataset to visual tokens and text tokens; fusing the visual tokens and text tokens through the modal fusion module to generate a fused token; re-injecting the fused token into the visual encoder to guide it to focus on the defect region in the image; and updating the model parameters using an efficient fine-tuning method. The incremental learning module is configured to build a high-quality dataset based on actual production line data, and to perform incremental learning and fine-tuning on the fine-tuned model, so that the model can master the knowledge of industrial defect detection and cause analysis of actual production lines, and realize the integrated output of accurate location of industrial product defects and cause tracing.

[0016] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the industrial defect detection and cause analysis method as described in the first aspect and various possible designs of the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the industrial defect detection and cause analysis method described in the first aspect and various possible designs of the first aspect.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the industrial defect detection and cause analysis method described in the first aspect and various possible designs of the first aspect.

[0019] The industrial defect detection and cause analysis method, apparatus, equipment, and storage medium provided in this application have at least the following beneficial effects: 1) Existing defect detection technologies mainly rely on visual AI models. These models output limited information and operate independently of the cause analysis system. The model's output alone is insufficient to determine the cause of defects, requiring human intervention and resulting in a low overall level of system intelligence. While existing large-scale visual-language model-based industrial inspection technologies possess some intelligence, the disconnect between visual and linguistic modalities prevents them from meeting the demands of industrial production and thus hinders their practical application. This application proposes a modal fusion mechanism to facilitate the application of large-scale visual-language models in industrial quality inspection. Through the interactive fusion of multimodal information, the large-scale model can better understand the product information of workpieces during production, thereby more effectively capturing defect information from workpiece images. After being fine-tuned using this method, the model's defect detection capability has been greatly improved. It achieves integrated output of defect location and cause description for industrial products, upgrading the detection of traditional visual models to visual language modal collaborative reasoning based on visual language models. This not only identifies the location of defects but also traces the causes of defects, thereby better guiding industrial production, reducing the reliance on manual defect detection, and improving the overall intelligence and automation of industrial production.

[0020] 2) Existing visual language large-scale model modal fusion techniques merely fuse data from multiple modalities without deeply exploring the effective information components within the fused information, resulting in high redundancy. This application introduces a linear neural network based on quaternion operations to assist in extracting effective information from the multimodal fusion information, reducing the redundancy of the fused information and enabling the fused information to better facilitate the visual encoder in capturing information from industrial images. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 A flowchart of an industrial defect detection and cause analysis method provided in this application embodiment; Figure 2 A flowchart illustrating the construction process of the intelligent industrial quality inspection system provided in this application embodiment; Figure 3 A flowchart of model fine-tuning provided for embodiments of this application; Figure 4 A schematic diagram illustrating the reasoning process of the modified model provided in this application embodiment; Figure 5 A detailed structural diagram of the modal fusion module provided in this application embodiment; Figure 6A flowchart of incremental learning of industrial knowledge in an actual production line provided for embodiments of this application; Figure 7 This is a structural diagram of the industrial defect detection and cause analysis device provided in the embodiments of this application.

[0023] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0025] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0027] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0028] This application provides an industrial defect detection and cause analysis method, employing a highly efficient fine-tuning method based on multimodal fusion to enhance the capabilities of a large visual language model in industrial defect detection and cause analysis. The large visual language model fine-tuned using this method possesses integrated capabilities for industrial defect localization and defect cause tracing analysis, and is easy to deploy and use in industrial settings. Figure 1 As shown, the industrial defect detection and cause analysis method can be implemented through the following steps S10-S30.

[0029] S10: Construct the system and select a large visual language model as the base model.

[0030] In some embodiments, such as Figure 2 As shown, step S10 can achieve the basic construction of the intelligent industrial quality inspection system through the following steps S101-S102.

[0031] S101: Design hardware resources and software systems.

[0032] Since this embodiment is based on a large visual language model system, its hardware and software resources consumption is typically higher than that of traditional single-modal visual models. Therefore, the resources the system depends on must be estimated before building the overall system. This solution recommends at least one private server for training and deploying the model. The server should be equipped with at least one Intel Xeon Gold 6530 CPU, two RTX 4090 24 GB GPUs, 256 GB of DDR5 memory, and a 4TB hard drive.

[0033] The software system requires an Ubuntu operating system, a deep learning environment (Python, PyTorch, CUDA, etc.), a database system, and an operations and maintenance system.

[0034] S102: Select a suitable large visual language model as the core of the system.

[0035] Currently, many technology companies have developed their own large-scale visual language models and made them open source. However, these models are not necessarily suitable for industrial-grade deployment. Therefore, when selecting a model, the focus should be on its visual capabilities, prioritizing models that score high in visual capability benchmark tests to ensure they have good visual feature capture capabilities, thus capturing details in industrial images. Secondly, large-scale visual language models often have billions or even more parameters. Generally speaking, the larger the number of parameters, the stronger the basic capabilities and the stronger the reasoning ability. However, directly deploying a large-scale model on an industrial production line will face practical problems such as high hardware resource consumption and high model inference time costs. Therefore, selecting a large model with an appropriate number of parameters is also of great significance for industrial deployment.

[0036] This embodiment recommends using the Qwen2.5-VL or Qwen3-VL series of open-source models because these models use a large amount of visual text data for pre-training, resulting in models with strong basic visual localization and visual perception capabilities. Regarding the number of parameters, if computing power is insufficient, models with 2B-7B parameters are recommended. Models with this parameter range have sufficient inference capabilities and fast inference speed, while consuming relatively few hardware resources, making them suitable for industrial deployment. If computing power is sufficient, models with 32B or higher parameters are recommended because models with this parameter range have more comprehensive reasoning capabilities and more fundamental knowledge. In this solution, the Qwen2.5-VL 3B model is selected as the base model for illustration.

[0037] S20: Fine-tuning the large visual language model using a multimodal fusion mechanism includes: mapping the images and text inputs of the industrial defect dataset to visual tokens and text tokens; fusing the visual tokens and text tokens through a modal fusion module to generate a fused token; re-injecting the fused token into the visual encoder to guide it to focus on the defect region in the image; and updating the model parameters using an efficient fine-tuning method.

[0038] Step S20 is used to initially inject industrial defect detection knowledge into the model, and to perform efficient fine-tuning of the model based on modal fusion on the 3CAD dataset of real industrial scenarios. The training objective of this stage is to inject basic industrial knowledge into the model by introducing a modal fusion mechanism, that is, to train the model parameters to fit on real industrial images, and to enable the model to have a certain analytical and reasoning ability for industrial knowledge.

[0039] In some embodiments, such as Figure 3 As shown, step S20 can be implemented through the following steps S201-S203.

[0040] S201: Preprocess the 3CAD open-source dataset.

[0041] Specifically, the dataset is transformed from a format adapted for visual segmentation tasks to a format adapted for visual localization tasks, and only defective industrial image samples from the dataset are used to construct the training samples. ,in Indicates image input, Indicates text input. This refers to the labels used to supervise the model's output. The text input needs to include basic information about the workpiece, such as what kind of product it is and what materials it is made of. The labels need to indicate whether the workpiece has defects. If there are defects, further information such as the precise location of the defects in the image, the specific type of the defects, and the cause of the defects needs to be added.

[0042] S202: Adjust the structure of the pre-trained model and add a modality fusion module to the model to obtain the modified model.

[0043] The reasoning process of the modified model is as follows: Figure 4 As shown, Figure 4 The numbers ① to ⑤ in the middle of the arrows indicate the execution order of the reasoning process; the same number indicates that the data processing flow represented by the arrow is performed simultaneously. (Combined with...) Figure 4 As shown, after text input and image input enter the model, they are mapped into text tokens (text information) and image tokens (image information) by the visual encoder and text encoder, respectively, and then... and It means that among them and This indicates the number and dimensions of the text tokens. and This indicates the number and dimensions of the image tokens. The fused token obtained after modal fusion of the text and image tokens is re-entered into the visual encoder, which outputs a new fused image token. With the original text token Together, they are fed into the Big Prophecy model for inference, producing the final inference output. The image token of each fusion module comes from the output of a certain visual layer in the visual encoder. The number of new fusion modules and the selection of visual layers can be arbitrary. It is recommended to add 4 to 8 fusion modules and use the image tokens from the output of the visual layers that are relatively early in the model as the visual modal inputs of the fusion modules. In this embodiment, 4 modal fusion modules are actually added, and the image tokens from layers 1, 7, 12, and 18 of the visual encoder in Qwen2.5-VL 3B are selected as the inputs of the 4 fusion modules, respectively.

[0044] The specific structure of the modality fusion module is as follows: Figure 5 As shown, the specific data processing flow of the modal fusion module includes: First, the dimension of the text token is increased or decreased to the dimension of the image token. The dimension-mapped text token is denoted as... Then for the tokens of the two modalities and Calculate cross-attention to obtain the initial fusion token. : Then, a neural network based on quaternion operations was used to... A redundancy removal process is performed, where relevant information is retained and irrelevant information is removed from the initial fusion token, resulting in a final, highly relevant fusion token. Quaternions are a theoretical concept in quaternion algebra. A quaternion can be represented as: ,in , , It represents the imaginary unit and satisfies: Quaternions The matrix representation is as follows: , In the formula, r , x , y and z These represent the real components of the quaternion.

[0045] first quaternion With the second quaternion The Hamiltonian product between them can be expressed as follows: .

[0046] In the formula, , , and Represent quaternions respectively The real number components; , , and Represent quaternions respectively The real number components.

[0047] Applying the operational properties of quaternions to linear neural networks, we can obtain: , in As the input to the neural network, This represents the weights of a linear layer based on quaternions, while This represents the network output. Quaternion linear networks effectively separate relevant and irrelevant features from the input. If the input is fused multimodal information, the quaternion network will highlight the correlations between different modalities, while irrelevant information will be treated as redundant and its importance reduced. (Fusing token) The final fused token is obtained after passing through a quaternion neural network. ,use With the original image token Combining these elements can help visual encoders effectively utilize fused semantic information, enabling them to more accurately focus on defective areas in images and thus locate the defects.

[0048] S203: Select a suitable and efficient fine-tuning method for fine-tuning.

[0049] Fine-tuning is performed using the data from step S201 and the model from step S202. The purpose is to teach the model basic industrial defect detection knowledge and enable it to output text that meets requirements. During fine-tuning training, the model's visual encoder and large language model will be fine-tuned using efficient fine-tuning methods, while the newly added modality fusion module will be fine-tuned using full-parameter fine-tuning. Based on engineering experience, LoRA, Adapter, or DoRA efficient fine-tuning methods are recommended, and it is suggested that the learning rate of modules using efficient fine-tuning be set to 10. -5 The order of magnitude, while the learning rate of the fully parameter-tuned modal fusion module should be set to 10. -3 The order of magnitude, the number of fine-tuning rounds, and the batch size depend on the hardware. In this embodiment, the LoRA method is actually used, and the learning rate of the efficient fine-tuning module is set to... The learning rate of the modality fusion module is set to The number of rounds and batch size are set to 3 and 4 respectively.

[0050] S30: Construct a high-quality dataset based on actual production line data, and perform incremental learning and fine-tuning on the fine-tuned model to enable the model to master the knowledge of industrial defect detection and cause analysis of actual production lines, and realize the integrated output of accurate location of industrial product defects and cause tracing.

[0051] Step S30 mainly involves organizing data from the actual production line and using this data to further fine-tune the model. The goal is to enable the model to learn the characteristics of various workpieces and products on the actual production line and the corresponding defect detection knowledge. For example... Figure 6 As shown, step S30 specifically includes the following steps S301 to S304.

[0052] S301: Collect data from the actual production line.

[0053] Multiple sample images of various industrial products are collected during different production processes on the production line, including defect-free normal industrial product samples and defective industrial samples. For example, normal and defective images of steel plates during the welding process, and normal and defective images of laptop motherboards during the manufacturing process. While recording information about the workpiece itself, parameters of industrial equipment operation or factory environmental parameters are also recorded simultaneously, such as factory workshop temperature and humidity, voltage and current of engineering equipment, and basic parameters of manufacturing equipment operation.

[0054] After data collection is completed, professional workers or experts should organize and analyze the production data, especially the data containing defective samples. Professional technicians should mark the location and type of defects, analyze whether there is a causal relationship between the settings of different engineering parameters and defective samples, and give the accurate cause of the product defects.

[0055] S302: Organize high-quality datasets.

[0056] The data from step S301 is organized into a high-quality dataset for training a large model. For industrial images with defects, the labels should include the workpiece name, the exact location of the defect marked by the bounding box, the category, and the exact cause of the defect. For normal images without defects, the labels should include the workpiece name and a basic description of the absence of defects. The organized dataset can be represented as follows: in, Indicates the first An image of an industrial sample. Indicates the first The text of a sample.

[0057] For normal samples, their labels for , Indicates the first A basic description of the attributes of an industrial sample.

[0058] For defective anomalous samples, their labels for In the tags, In the formula Indicates the first element present in the input image. One defect, and , , , This indicates the location coordinates of the defect in the image. Indicates the first Defect cause analysis of an industrial sample.

[0059] S303: Data Standardization and Storage.

[0060] Images, annotation information (location, category), defect cause information, and production equipment parameters are integrated into a structured data record and persistently stored in a unified database. This serves as the data source for model fine-tuning and training, and can also provide data support for the continuous learning of subsequent models.

[0061] S304: Continuous learning of the model.

[0062] When a new type of industrial product is added to the production line, repeat steps S301 and S302 to generate a new training dataset and store it in the database. Then, randomly extract existing data from the database. and new product data , to use both The proportion of mixed data is used as the real training set to train the model. The purpose of this data mixing step is to allow the model to learn the feature attributes of newly added industrial products, while preventing the model from forgetting the feature attributes of existing industrial products. This avoids the model from becoming fickle after fine-tuning, i.e., performance degradation in the detection of old types of industrial products.

[0063] Through continuous learning, the model can maintain its ability to detect defects in older types of industrial products without degradation, while also learning the characteristics of new types of industrial products and performing defect detection and cause analysis on these new products.

[0064] This application also provides an industrial defect detection and cause analysis device for implementing the methods described in any of the above embodiments, such as... Figure 7 As shown, the industrial defect detection and cause analysis device includes: The basic building module 701 is configured to build the system and selects the visual language large model as the basic model. The model fine-tuning module 702 is configured to fine-tune the large visual language model using a multimodal fusion mechanism, including: mapping the image and text input of the industrial defect dataset to visual tokens and text tokens; fusing the visual tokens and text tokens through the modal fusion module to generate a fused token; re-injecting the fused token into the visual encoder to guide it to focus on the defect region in the image; and updating the model parameters using an efficient fine-tuning method. The incremental learning module 703 is configured to build a high-quality dataset based on actual production line data, and to perform incremental learning and fine-tuning on the fine-tuned model, so that the model can master the knowledge of industrial defect detection and cause analysis of actual production lines, and realize the integrated output of accurate location of industrial product defects and cause tracing.

[0065] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0066] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0067] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0068] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0069] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the industrial defect detection and cause analysis method described in the above embodiments.

[0070] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the industrial defect detection and cause analysis method in the above embodiments.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0072] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0073] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0074] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0075] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0076] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0077] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0078] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0079] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0080] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for industrial defect detection and cause analysis, characterized in that, The method includes: The system was constructed, and a large visual language model was selected as the base model. The large visual language model is fine-tuned using a multimodal fusion mechanism, including: mapping the images and text inputs of the industrial defect dataset to visual tokens and text tokens; fusing the visual tokens and text tokens through a modal fusion module to generate a fused token; re-injecting the fused token into the visual encoder to guide it to focus on the defect region in the image; and updating the model parameters using an efficient fine-tuning method. A high-quality dataset is built based on actual production line data. The model is then incrementally trained and fine-tuned to enable it to master the knowledge of industrial defect detection and cause analysis from actual production lines, thereby achieving integrated output of accurate location of industrial product defects and cause tracing.

2. The industrial defect detection and cause analysis method according to claim 1, characterized in that, The visual token and text token are fused using a modal fusion module to generate a fused token, including: Adjust the dimensions of the text token to match those of the visual token to obtain the dimension-mapped text token, denoted as . ; The initial fusion token is calculated using the following formula. ; In the formula, Represents a visual token. and This indicates the number and dimensions of the image tokens. Represents the normalized exponential function, This represents the matrix transpose operation; Neural networks based on quaternion operations, for Redundancy removal is performed to obtain the final fusion token.

3. The industrial defect detection and cause analysis method according to claim 2, characterized in that, Neural networks based on quaternion operations, for Redundancy removal is performed to obtain the final fusion token; Applying the operational properties of quaternions to linear neural networks, the computational formula for neural networks based on quaternion operations is expressed as follows: In the formula, As the input to the neural network, This represents the weights of a linear layer based on quaternions, while This indicates the network output.

4. The industrial defect detection and cause analysis method according to claim 3, characterized in that, The operational properties of quaternions are determined as follows: Let a quaternion be represented as: ,in , , It represents the imaginary unit and satisfies: Quaternions The matrix representation is as follows: In the formula, r , x , y and z These represent the real components of the quaternion; first quaternion With the second quaternion The Hamiltonian product between them can be expressed as follows: In the formula, , , and Represent quaternions respectively The real number components; , , and Represent quaternions respectively The real number components.

5. The industrial defect detection and cause analysis method according to claim 1, characterized in that, The modality fusion module consists of 4 to 8 modules, each connected to a different intermediate layer of the visual encoder.

6. The industrial defect detection and cause analysis method according to claim 1, characterized in that, The large visual language model is a Qwen2.5-VL or Qwen3-VL series model with 2B to 7B parameters.

7. The industrial defect detection and cause analysis method according to claim 1, characterized in that, The methods for incrementally learning and fine-tuning the model based on actual production line data include: collecting images and equipment parameter data from the actual production line; labeling the location, type, and cause of defects; and fine-tuning by mixing the new data with the original data in proportion.

8. An industrial defect detection and cause analysis device, characterized in that, The device includes: The basic building block is configured to build the system and selects a large visual language model as the base model. The model fine-tuning module is configured to fine-tune the large visual language model using a multimodal fusion mechanism, including: mapping the image and text input of the industrial defect dataset to visual tokens and text tokens; fusing the visual tokens and text tokens through the modal fusion module to generate a fused token; re-injecting the fused token into the visual encoder to guide it to focus on the defect region in the image; and updating the model parameters using an efficient fine-tuning method. The incremental learning module is configured to build a high-quality dataset based on actual production line data, and to perform incremental learning and fine-tuning on the fine-tuned model, so that the model can master the knowledge of industrial defect detection and cause analysis of actual production lines, and realize the integrated output of accurate location of industrial product defects and cause tracing.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the industrial defect detection and cause analysis method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the industrial defect detection and cause analysis method as described in any one of claims 1-7.

Images