Circuit defect detection method and device based on multi-modal model, equipment and medium

Abstract

The present application relates to the technical field of circuit defect detection based on a multi-modal model, and particularly relates to a circuit defect detection method, device and equipment based on a multi-modal model and a medium. The method first acquires an initial circuit board multi-source data set, performs data preprocessing on the multi-source data set to obtain a target multi-source feature set, then calls a pre-trained multi-modal model to perform semantic space mapping processing on the target multi-source feature set to obtain a multi-modal feature set, then calls a pre-trained defect detection model to perform circuit board defect prediction on the multi-modal feature set to obtain defect position information and a defect type label of the circuit board, and finally acquires natural language question data, calls a pre-trained visual language model to generate a natural language answer result based on the natural language question data, the defect position information and the defect type label, aiming to improve the circuit defect detection precision based on the multi-modal model and realize the explainability of the detection result.

Description

Technical Field

[0001] This invention relates to the field of circuit defect detection technology, and in particular to a circuit defect detection method, apparatus, equipment and medium based on a multimodal model. Background Technology

[0002] In the electronics manufacturing industry, circuit board defect detection is a core link in ensuring product reliability, and its accuracy and level of intelligence directly affect production efficiency and product quality. Current mainstream circuit board defect detection methods mostly rely on single vision models such as YOLO (You Only Look Once), Faster R-CNN (Fast Region-Based Convolutional Neural Network), or traditional rule-based algorithms. While these solutions can perform basic defect localization and classification, they generally have functional limitations. They can only achieve the basic task of defect detection and struggle to understand defects semantically, unable to engage in semantic interaction and reasoning with engineers, and thus failing to meet the needs of in-depth defect information analysis in intelligent industrial inspection.

[0003] Meanwhile, the field of PCB (Printed Circuit Board) defect detection has long faced the problem of a lack of semantically labeled data. Existing data constructions are mostly focused on single detection tasks, lacking the integration of image, semantic requirements, and analysis results, resulting in insufficient model generalization ability and limited adaptability in complex scenarios. In addition, the implementation of traditional detection solutions is relatively simple, mainly outputting independent detection results, lacking integrated interactive and visualization tools. Engineers need to manually sort out defect information, making it difficult to efficiently complete the closed-loop process of detection, analysis, and feedback. This situation is significantly different from the current industrial scenario's demand for intelligent and efficient detection tools. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a circuit defect detection method, device, equipment and medium based on a multimodal model, which aims to improve the accuracy of circuit board defect detection and realize the interpretability of detection results.

[0005] The first aspect of this invention provides a circuit defect detection method based on a multimodal model, comprising: acquiring an initial multi-source dataset of a circuit board; performing data preprocessing on the multi-source dataset to obtain a target multi-source feature set; calling a pre-trained multimodal model to perform semantic space mapping processing on the target multi-source feature set to obtain a multimodal feature set; calling a pre-trained defect detection model to predict circuit board defects on the multimodal feature set to obtain defect location information and defect type labels of the circuit board; acquiring natural language question data; calling a pre-trained visual language model to generate a natural language answer result based on the natural language question data, the defect location information, and the defect type labels.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the initial circuit board multi-source dataset includes an initial circuit board image, an initial circuit board design file, and initial circuit board text; the step of preprocessing the multi-source dataset to obtain a target multi-source feature set includes: performing channel normalization processing on the initial circuit board image using a channel normalization algorithm to obtain an optimized circuit board image; performing data augmentation processing on the optimized circuit board image based on a data augmentation algorithm to obtain an enhanced circuit board image; calling a pre-trained large visual model to extract features from the enhanced circuit board image to obtain image features; performing file parsing processing on the initial circuit board design file using a file parsing tool to obtain file features; performing semantic feature encoding processing on the initial circuit board text using an encoder to obtain text features; and performing common feature dimension mapping processing on the image features, the file features, and the text features using a modal linear layer mapping algorithm to obtain the target multi-source feature set.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the large visual model includes a patch partitioning module, an encoding module, and a feature aggregation module, wherein the patch partitioning module, the encoding module, and the feature aggregation module are connected sequentially; the step of calling the pre-trained large visual model to extract features from the enhanced circuit board image to obtain image features includes: performing patch partitioning processing on the enhanced circuit board image based on the patch partitioning module to obtain an image patch set; performing global feature modeling processing on the image patch set based on the encoding module to obtain a feature vector set; and performing feature aggregation processing on the feature vector set based on the feature aggregation module to obtain the image features.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the multimodal model includes an information extraction module and a feature fusion module, the information extraction module and the feature fusion module being connected; the step of calling the pre-trained multimodal model to perform semantic space mapping processing on the target multi-source feature set to obtain a multimodal feature set includes: acquiring multiple first query vectors; performing information extraction processing on the target multi-source feature set based on the information extraction module and the multiple first query vectors to obtain an intermediate fused feature set, the intermediate fused feature set including intermediate fused features corresponding to each first query vector; and performing feature fusion processing on the intermediate fused feature set based on the feature fusion module to obtain the multimodal feature set.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the defect detection model includes a self-attention module, a detection head decoder module, and a confidence calculation module, wherein the self-attention module, the detection head decoder module, and the confidence calculation module are connected sequentially; the step of calling the pre-trained defect detection model to predict circuit board defects on the multimodal feature set to obtain the defect location information and defect type label of the circuit board includes: acquiring multiple second query vectors; performing deformable attention focusing processing on each second query vector based on the self-attention module to obtain multiple third query vectors; performing candidate box prediction processing on the multimodal feature set based on the detection head decoder module and the multiple third query vectors to obtain a candidate box set, wherein the candidate box set includes multiple candidate boxes, and candidate box coordinates and defect category probabilities corresponding to each candidate box; performing confidence calculation on the multiple defect category probabilities based on the confidence calculation module to obtain the defect probability corresponding to each candidate box; and determining the defect location information and defect type label of the circuit board based on the multiple candidate box coordinates and the multiple defect probabilities.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the visual language model includes a question encoding module, a question decoding module, a knowledge base retrieval module, and a knowledge fusion generation module, wherein the question encoding module, the question decoding module, the knowledge base retrieval module, and the knowledge fusion generation module are connected sequentially; the step of acquiring natural language question data, calling a pre-trained visual language model, and generating a natural language answer result based on the natural language question data, the defect location information, and the defect type label includes: performing semantic encoding processing on the natural language question data, the defect location information, and the defect type label based on the question encoding module to obtain answer semantic features; performing sequence generation processing on the answer semantic features based on the question decoding module to obtain an initial answer sequence; performing PCB defect knowledge retrieval processing on the multimodal feature set based on the knowledge base retrieval module to obtain defect knowledge information; and performing integrated reasoning processing on the initial answer sequence and the defect knowledge information based on the knowledge fusion generation module to obtain a natural language answer result.

[0011] Optionally, in the sixth implementation of the first aspect of the present invention, after acquiring natural language question data, calling a pre-trained visual language model, and generating a natural language answer result based on the natural language question data, the defect location information, and the defect type label, the method further includes: using a visualization annotation tool to perform defect region selection and annotation processing on the initial circuit board image based on the defect location information to obtain a defect-annotated circuit board image; and performing visualization processing on the defect-annotated circuit board image and the natural language answer result to obtain visualization data.

[0012] A second aspect of the present invention provides a circuit defect detection device based on a multimodal model, comprising: a data processing module for acquiring an initial multi-source dataset of a circuit board, performing data preprocessing on the multi-source dataset to obtain a target multi-source feature set; a semantic space mapping module for calling a pre-trained multimodal model to perform semantic space mapping processing on the target multi-source feature set to obtain a multimodal feature set; a defect prediction module for calling a pre-trained defect detection model to predict circuit board defects on the multimodal feature set to obtain defect location information and defect type labels of the circuit board; and a natural language answer generation module for acquiring natural language question data, calling a pre-trained visual language model, and generating a natural language answer result based on the natural language question data, the defect location information, and the defect type labels.

[0013] A third aspect of the present invention provides a circuit defect detection device based on a multimodal model, the circuit defect detection device based on a multimodal model comprising: a memory and at least one processor, the memory storing instructions; at least one processor calling the instructions in the memory to cause the circuit defect detection device based on a multimodal model to perform each step of the circuit defect detection method based on a multimodal model as described above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the circuit defect detection method based on a multimodal model as described in any of the preceding claims.

[0015] In the technical solution of this invention, an initial multi-source dataset of a circuit board is first obtained. The multi-source dataset is then preprocessed to obtain a target multi-source feature set. Next, a pre-trained multimodal model is called to perform semantic space mapping on the target multi-source feature set to obtain a multimodal feature set. Then, a pre-trained defect detection model is called to predict circuit board defects using the multimodal feature set to obtain the defect location information and defect type label of the circuit board. Finally, natural language question data is obtained, and a pre-trained visual language model is called to generate natural language answer results based on the natural language question data, defect location information, and defect type label. This aims to improve the accuracy of circuit defect detection based on the multimodal model and to make the detection results interpretable, providing a more efficient, reliable, and practical PCB defect detection and auxiliary decision-making solution for the electronics manufacturing industry. Attached Figure Description

[0016] Figure 1 A logic flowchart of a circuit defect detection method based on a multimodal model provided in an embodiment of the present invention;

[0017] Figure 2A schematic diagram of the circuit defect detection device based on a multimodal model provided in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the circuit defect detection device based on a multimodal model provided in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a circuit defect detection method, apparatus, device, and medium based on a multimodal model. In this invention, the terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the circuit defect detection method based on a multimodal model in this invention includes:

[0021] 101. Obtain the initial circuit board multi-source dataset, perform data preprocessing on the multi-source dataset, and obtain the target multi-source feature set;

[0022] In this embodiment, the initial multi-source dataset for the circuit board encompasses circuit board image data, design documents, and related text information. The image data records the circuit board's appearance and potential defect characteristics, the design documents contain structured information such as circuit layout and wiring rules, and the text information includes semantic content such as product specifications and defect annotations. To ensure the uniformity of the input data, the system normalizes and color-corrects the circuit board image data, performs semantic parsing on the design documents and converts them into vectorized features, and segments, encodes, and vectorizes the related text information, thereby standardizing the multimodal data before it enters the deep model. The preprocessed target multi-source feature set possesses good standardization and compatibility, directly adapting to subsequent multimodal fusion and inference processes, significantly improving the efficiency and accuracy of model feature learning.

[0023] 102. Call the pre-trained multimodal model to perform semantic space mapping processing on the target multi-source feature set to obtain the multimodal feature set;

[0024] In this embodiment, when the pre-trained multimodal model is invoked to perform semantic space mapping on the target multi-source feature set, the target multi-source feature set has integrated image features, defect description text features, and design document features. These three types of features respectively carry the visual representation, semantic description, and structural information of the circuit board, exhibiting inherent modal heterogeneity. The pre-trained multimodal model draws on the structural design of CLIP (Contrastive Language-Image Pre-training), projecting different types of features into a unified semantic space in an orderly manner through modal feature mapping branches. This enables stable semantic associations to be established among images, text, and design documents, achieving effective integration of multimodal information. Based on this, the model introduces the Q-Former query mechanism (query transformer query mechanism), which dynamically interacts among multimodal features through a set of learnable query vectors. This accurately extracts key semantic components from information from different sources, capturing both explicit defect representations in images and deeply mining the implicit engineering semantics in text descriptions and design documents. This promotes the transformation of multi-source information from simple superposition to deep fusion, ultimately forming a structurally unified and semantically coherent multimodal feature set. The construction of a unified semantic space completely eliminates the heterogeneity barrier between different modal features, making multimodal information comparable and correlated, and providing a core foundation for multi-dimensional defect inference. The introduction of the Q-Former query mechanism enhances the efficiency of key semantic extraction, allowing the model to break through the information limitations of a single modality, and make comprehensive judgments by combining explicit visual features with implicit engineering semantics, thereby improving the ability to identify atypical defects in complex scenarios. At the same time, the semantically aligned multimodal feature set has both information completeness and representation consistency, which not only provides comprehensive and accurate feature support for subsequent defect detection models, but also ensures the semantic coherence of subsequent visual language interaction tasks, helping to achieve efficient connection of the entire process from feature fusion to defect detection to semantic question answering, significantly improving the intelligence level and industrial practical value of the overall technical solution.

[0025] 103. Call the pre-trained defect detection model to predict circuit board defects on the multimodal feature set, and obtain the defect location information and defect type label of the circuit board.

[0026] In this embodiment, the core of the pre-trained defect detection model adopts a DETR (DetectionTransformer)-based detection head structure, directly acting on semantically aligned multimodal features. Through an end-to-end inference mode, it synchronously completes defect category classification and bounding box regression, eliminating the need for manually designed anchor mechanisms and significantly reducing reliance on prior human knowledge, making the detection process more adaptable and generalizable. Addressing the technical challenges of detecting minute defects, the defect detection model further introduces a deformable attention mechanism. This mechanism adaptively focuses on key local regions related to defects within the multimodal features. Combining the implicit circuit structure semantics in the design document with the defect features described in the text, it accurately captures the representational information of fine-grained defects such as mouse bites and open circuits, achieving a more comprehensive and in-depth defect judgment. Ultimately, it outputs accurate defect location information and clear defect type labels. The synergistic effect of the deformable attention mechanism and multimodal features effectively overcomes the limitations of single-vision detection in adapting to minute defects and complex scenes, significantly improving the detection accuracy and recognition reliability of fine-grained defects.

[0027] 104. Obtain natural language question data, call the pre-trained visual language model, and generate natural language answer results based on the natural language question data, the defect location information, and the defect type label.

[0028] In this embodiment, the acquired natural language question data covers various user queries regarding circuit board defects, including basic questions such as defect existence, type, and location, as well as in-depth inquiries about defect impact and handling suggestions. The pre-trained visual language model, based on the LLaVA (Large Language Visual Assistant) framework, possesses mature visual-language interaction and reasoning capabilities. Upon receiving the natural language question data, it integrates it with the acquired defect location information and defect type labels through multi-source information association. The visual language model first uses a dedicated semantic mapping mechanism to synchronously project the question semantics, spatial features of the defect location, and category semantics of the defect type into a unified semantic space, achieving precise alignment of multimodal information and building a solid semantic foundation for subsequent reasoning. Based on this, the visual language model leverages the deep logical reasoning capabilities of the large language model, integrates defect risk level assessment standards and industry practice experience, and outputs response information that precisely matches the user's question output with defect features, forming a reasoning conclusion that is both professional and instructive. Finally, the reasoning result is transformed into standardized content conforming to natural language expression habits, generating a logically coherent and informationally complete natural language answer. Leveraging the visual-language fusion advantages of the LLaVA framework, the model breaks through the limitations of traditional defect detection methods that only output structured data. It achieves efficient transformation from technical inspection results to user-friendly interactive responses, significantly improving the intuitiveness and convenience of human-computer interaction. The construction of a unified semantic space ensures the effective integration of multi-source information, enabling responses to accurately address the core of user questions and avoiding semantic bias and information omissions. Simultaneously, the natural language responses balance professionalism and comprehensibility, meeting the professional demands of industrial scenarios while lowering the understanding threshold for non-professionals. This significantly enhances the practicality and promotional value of the technical solution, driving the upgrade of circuit board defect detection from simple technical inspection to intelligent, interactive analysis services.

[0029] In this embodiment of the invention, the initial circuit board multi-source dataset includes an initial circuit board image, an initial circuit board design file, and initial circuit board text. The preprocessing of the multi-source dataset to obtain a target multi-source feature set includes: applying a channel normalization algorithm to the initial circuit board image to obtain an optimized circuit board image; applying a data augmentation algorithm to the optimized circuit board image to obtain an enhanced circuit board image; using a pre-trained large visual model to extract features from the enhanced circuit board image to obtain image features; using a file parsing tool to parse the initial circuit board design file to obtain file features; using an encoder to perform semantic feature encoding on the initial circuit board text to obtain text features; and using a modal linear layer mapping algorithm to perform common feature dimension mapping on the image features, the file features, and the text features to obtain the target multi-source feature set.

[0030] In this embodiment, the initial circuit board multi-source dataset encompasses initial circuit board images, initial circuit board design files, and initial circuit board text. For the initial circuit board image, it is first represented as... ,in, This represents the initial color circuit board image. Indicates the height of the initial circuit board image. This indicates the width of the initial circuit board image; "3" represents the three RGB color channels. The data type used to represent numerical values ​​is real numbers; then, a channel normalization algorithm is used to perform channel normalization processing on the initial circuit board image to eliminate interference from illumination changes and sensor noise, resulting in an optimized circuit board image. The specific calculation expression is as follows:

[0031] ,

[0032] in, This indicates an optimized circuit board image. This represents the initial circuit board image. This represents the average value of the RGB (Red, Green, Blue) channels. This represents the standard deviation of the RGB channels. Based on this, the diversity of samples is expanded through data augmentation algorithms. Data augmentation processing is performed on the optimized circuit board image to obtain an enhanced circuit board image, which enhances the generalization ability of the subsequent model to complex industrial scenarios. Data augmentation processing includes random rotation, horizontal or vertical flipping, brightness and contrast adjustment, etc. The enhanced image is used to improve the generalization ability of the model. Finally, the pre-trained large visual model is called to extract features from the enhanced circuit board image to obtain image features.

[0033] For the initial circuit board design file, a file parsing tool is used to parse it. First, the Gerber file is converted into a binary mask for the circuit layers. The specific calculation expression is as follows:

[0034] ,

[0035] in, Indicates the first Gerber mask pattern of the multilayer circuit board. The value ranges from 1 to Each layer Corresponding to different copper foil or pad structures, it provides geometric constraint information that is not directly observable in the image. "1" represents a conductive area, and "0" represents an insulating or blank area. Indicates the height of the Gerber mask. This indicates the width of the Gerber mask, whose size is consistent with the initial board image size.

[0036] Next, to align with image features, the binary mask is encoded into file features via a convolutional network. The specific calculation expression is as follows:

[0037] ,

[0038] in, Indicates file characteristics, This represents the Gerber feature extraction network. This represents the size of the feature map after sampling the Gerber mask image. Indicates the height of the feature map. Indicates the width of the feature map. Indicates the first Gerber mask pattern of the multilayer circuit board. The numeric type used to represent data is real numbers. This represents the Gerber feature channel dimension. This process transforms geometric constraints, such as circuit topology, from the initial circuit board design file into computable features.

[0039] For the initial circuit board text, a file parsing tool is used to parse and process the initial circuit board design file. First, the initial circuit board text needs to be segmented and embedded to obtain the text input sequence. ,in, Indicates the first i Vector representation of each word T Indicates the length of the input text. d t This represents the dimension of each word embedding vector; then, based on the BERT (Bidirectional Encoder Representation Model) encoder, the text input sequence is semantically encoded to obtain text features. The specific calculation expression is as follows:

[0040] ,

[0041] in, Representing text features, Represents a BERT-based text encoding function; Represents a text input sequence. The numeric type used to represent data is real numbers. T Indicates the length of the input text. d t This represents the dimension of the embedding vector for each word.

[0042] After extracting features from each modality, a modal linear layer mapping algorithm is used to map the common feature dimensions of image features, file features, and text features, resulting in a target multi-source feature set. This eliminates the heterogeneity barrier between different modal features, enabling effective fusion of features of various types in a common space. This preserves the core information of each modality and provides a structurally consistent and informationally complementary feature foundation for subsequent multimodal model inference. The specific calculation expression is as follows:

[0043] ,

[0044] in, Representing image features, Indicates file characteristics, Representing text features, , and Represents the modal linear layer mapping function. This represents the dimension of the multimodal unified feature space.

[0045] In this embodiment of the invention, the large visual model includes a patch partitioning module, an encoding module, and a feature aggregation module, which are sequentially connected. The step of calling the pre-trained large visual model to extract features from the enhanced circuit board image to obtain image features includes: performing patch partitioning processing on the enhanced circuit board image based on the patch partitioning module to obtain an image patch set; performing global feature modeling processing on the image patch set based on the encoding module to obtain a feature vector set; and performing feature aggregation processing on the feature vector set based on the feature aggregation module to obtain the image features.

[0046] In this embodiment, the pre-trained large-scale visual model called is DINOv2 (a second-generation self-supervised visual Transformer model). This model adopts the Vision Transformer structure and is trained under large-scale self-supervision. It can learn the global and local feature representations of PCB images under unlabeled conditions. For complex circuit structures, DINOv2's global attention mechanism can capture long-distance dependencies across regions, which helps to distinguish between small cracks, local short circuits and normal traces in the circuit. While maintaining global information modeling, it can also enhance the ability to represent fine-grained defects, making the detection process more accurate and stable when facing small defects or atypical defects. Its built-in patch partitioning module, encoding module and feature aggregation module are connected in sequence to realize enhanced feature extraction of circuit board images.

[0047] First, the enhanced circuit board image is input into the model's patch segmentation module, which performs patch segmentation processing, dividing the image into N image patches, each corresponding to... ,in, Indicates the first One image patch, p This represents the side length of a single image patch, with 3 corresponding to the RGB color channels, thus obtaining the image patch set.

[0048] Subsequently, the image patch set is input into the model's encoding module for global feature modeling. This module uses the DINOv2 L-layer Transformer encoding structure, first performing linear embedding mapping and positional encoding operations on each image patch. The specific calculation expression is as follows:

[0049] ,

[0050] in, Indicates the first The initial representation vector of each image patch. Represents a linear embedding map, Indicates the first One image patch, Indicates the first The location encoding of each image patch.

[0051] Next, the obtained Transformer encoding is performed: global feature modeling is conducted through an L-layer Transformer to obtain high-level feature representations for each patch, i.e., feature vector sets. The specific calculation expression is as follows:

[0052] ,

[0053] ,

[0054] in, Indicates the first l After the first layer The output of each token Indicates the first l -1st floor The output of each token, FeedForward, represents the feedforward neural network in the Transformer architecture. L This indicates the number of layers in the Transformer, and MultiHeadAttention represents the multi-head attention mechanism, defined as:

[0055] ,

[0056] in, Q Represents the query matrix. K Represents the key matrix, V This represents a value matrix; softmax is used to normalize the attention weights. Represents the dimension of the key vector. K T Indicates to K Perform the transpose operation.

[0057] Finally, the feature vector set is input into the feature aggregation module of the model for feature aggregation processing. Average pooling is used to aggregate all patch feature vectors output from layer L in the feature vector set to obtain image features. The specific calculation expression is as follows:

[0058] ,

[0059] in, This represents image features; Aggregate represents average pooling. This represents the feature vector set, i.e., all feature vectors of the patch. N This indicates the number of feature vectors in the patch. The dimension representing the image features.

[0060] In this embodiment of the invention, the multimodal model includes an information extraction module and a feature fusion module, which are connected to each other. The step of calling the pre-trained multimodal model to perform semantic space mapping processing on the target multi-source feature set to obtain a multimodal feature set includes: acquiring multiple first query vectors; performing information extraction processing on the target multi-source feature set based on the information extraction module and the multiple first query vectors to obtain an intermediate fused feature set, the intermediate fused feature set including intermediate fused features corresponding to each first query vector; and performing feature fusion processing on the intermediate fused feature set based on the feature fusion module to obtain the multimodal feature set.

[0061] In this embodiment, a pre-trained multimodal model is invoked to perform semantic space mapping processing on the target multi-source feature set. First, multiple first query vectors are obtained, that is, a set of query vectors is introduced into the information extraction module. The information extraction process is performed on the target multi-source feature set to obtain the intermediate fused feature set. The specific calculation expression is as follows:

[0062] ,

[0063] in, h j Indicates the first j An intermediate fusion feature, Indicates the first j The first query vector, Representing image features, Indicates file characteristics, Representing text features, , and Let represent the modal linear layer mapping function, and Attn represent the multimodal attention function. This process allows the first query vector to extract key information from the target multi-source feature set, including explicit defects in the fused image, circuit structure constraints in the design file, and semantic descriptions in the text. h j Together they constitute the intermediate fusion feature set.

[0064] Finally, based on the feature fusion module of the multimodal model, feature fusion processing is performed on the intermediate fused feature set to obtain the multimodal feature set. The multimodal feature set obtained after feature fusion integrates the core features of each modality, which not only provides more comprehensive feature support for subsequent defect detection, but also guides the model to focus on key defect areas with the help of text information, effectively improving the adaptability to minor defects and complex circuit scenarios, and enhancing the accuracy and robustness of the overall detection process. The specific calculation expression is as follows:

[0065] ,

[0066] in, Represents a multimodal feature set. h j Indicates the first j The intermediate fusion feature, Concat, is used to fully preserve and integrate the multimodal key information extracted from each first query vector across different dimensions.

[0067] In this embodiment of the invention, the defect detection model includes a self-attention module, a detection head decoder module, and a confidence calculation module, which are sequentially connected. The step of calling the pre-trained defect detection model to predict circuit board defects on the multimodal feature set to obtain defect location information and defect type labels for the circuit board includes: acquiring multiple second query vectors; performing deformable attention focusing processing on each second query vector based on the self-attention module to obtain multiple third query vectors; performing candidate box prediction processing on the multimodal feature set based on the detection head decoder module and the multiple third query vectors to obtain a candidate box set, the candidate box set including multiple candidate boxes, and candidate box coordinates and defect category probabilities corresponding to each candidate box; calculating confidence on the multiple defect category probabilities based on the confidence calculation module to obtain a defect probability corresponding to each candidate box; and determining the defect location information and defect type labels for the circuit board based on the multiple candidate box coordinates and the multiple defect probabilities.

[0068] In this embodiment, when calling the pre-trained defect detection model to predict circuit board defects on a multimodal feature set, multiple second query vectors are first obtained and input into the self-attention module. This module introduces a deformable attention mechanism to perform focusing processing, that is, through deformable attention, each second query vector adaptively focuses on the local key feature points related to defects in the multimodal feature set, reducing interference from irrelevant information. After this processing, multiple third query vectors are obtained, which can more accurately capture the feature associations of fine-grained defects such as microcracks and local short circuits.

[0069] Subsequently, the third query vector and the multimodal feature set are input into the detection head decoder module for candidate box prediction processing to obtain a candidate box set. The candidate box set includes multiple candidate boxes, as well as the coordinates of each candidate box and the probability of the defect category. The detection head decoder module adopts an end-to-end candidate box prediction method, which reduces the dependence on manual prior design and enhances the model's generalization ability under different circuit scenarios. The specific calculation expression is as follows:

[0070] ,

[0071] in, Indicates the first k The prediction result, the first k The prediction results include the first... k The candidate box, and the one with the first candidate box. k The coordinates of each candidate bounding box and the probability of the defect category. Represents a multimodal feature set. Indicates the first k A third query vector, bk =( x k , y k , w k , h k () represents the candidate bounding box coordinates, which include the center point and dimensions of the bounding box. x k and y k Represent the x and y coordinates of the center point, respectively. w k and h k These represent the width and height of the bounding box, respectively. Indicates the probability of defect category. C Indicates the number of defect categories. The Decoder is the computational unit of the detection head decoder module, used for... as well as Perform candidate box prediction.

[0072] Next, the candidate box coordinates and multimodal feature set are input into the confidence calculation module for confidence calculation, obtaining the defect probability corresponding to each candidate box. The confidence calculation module reduces the false detection probability through probabilistic evaluation and category filtering, ensuring the reliability of the defect location and type output. The specific calculation expression is as follows:

[0073] ,

[0074] in, Indicates the relationship with the first k The probability of defects corresponding to each candidate box. Represents a multimodal feature set. b k Indicates the coordinates of the candidate box. For conditional probability, "defect" refers to an abnormal area on the circuit board that does not meet manufacturing process standards. Its calculation logic is as follows: the model first uses the coordinates of candidate boxes... b k Locate the target region and extract the multimodal feature set corresponding to the target region. Then The input is processed by a fully connected classification layer using function mapping, and the output is the probability value that the region belongs to a defect.

[0075] Finally, based on multiple candidate box coordinates and multiple defect probabilities, the defect location information and defect type label of the circuit board are determined. The specific calculation expression is as follows:

[0076] ,

[0077] in, Indicates the first k Each candidate box corresponds to a defect type label. `argmax` represents the independent variable that takes the maximum value. Its function is to select the category that maximizes the subsequent probability value from all possible defect categories. c Indicates the specific defect category (such as short circuit, open circuit, etc.). C Indicates the number of defect categories. Let be the conditional probability expression, representing the condition at the th... k Coordinates of the candidate boxes b k With multimodal feature sets Under the given conditions, the region corresponding to the candidate box belongs to category c The probability, based on the candidate box coordinates b k It can determine the location of defects on the circuit board.

[0078] In this embodiment of the invention, the visual language model includes a question encoding module, a question decoding module, a knowledge base retrieval module, and a knowledge fusion generation module, which are sequentially connected. The step of acquiring natural language question data, calling a pre-trained visual language model, and generating a natural language answer based on the natural language question data, the defect location information, and the defect type label includes: performing semantic encoding processing on the natural language question data, the defect location information, and the defect type label using the question encoding module to obtain answer semantic features; performing sequence generation processing on the answer semantic features using the question decoding module to obtain an initial answer sequence; performing PCB defect knowledge retrieval processing on the multimodal feature set using the knowledge base retrieval module to obtain defect knowledge information; and performing integrated reasoning processing on the initial answer sequence and the defect knowledge information using the knowledge fusion generation module to obtain a natural language answer result.

[0079] In this embodiment, the pre-trained visual language model is a multimodal model built around the LLaVA framework. Its core capability is to achieve multimodal semantic alignment, understanding, and generation of visual information and natural language. The LLaVA framework deeply integrates visual feature processing components with a large language model, enabling the model to not only parse the visual details of PCB defects and circuit structure information contained in the multimodal feature set, but also understand the semantic demands of natural language questions and generate professional answers that conform to human expression habits. Simultaneously, the model integrates a customized PCB defect knowledge base to enhance the professionalism and generalization ability of reasoning. In the process of generating natural language answer results, the natural language question data is first represented as a sequence of word vectors. Then, based on the question encoding module, the word vector sequence, defect location information, and defect type label are semantically encoded to obtain the answer semantic features. This process achieves multimodal semantic alignment between visual defect information and natural language questions. Subsequently, the question decoding module based on the model performs sequence generation processing on the answer semantic features to generate an initial answer sequence. Simultaneously, the question decoding module constrains the generation process through a cross-entropy loss function, relying on the pre-trained semantic association capabilities of the LLaVA framework to ensure accurate matching between the initial answer and defect information.

[0080] Next, the model-based knowledge base retrieval module performs PCB defect knowledge retrieval processing on the multimodal feature set to obtain defect knowledge information. This compensates for the limitations of the model training data and enhances its ability to handle novel defects. The pre-built PCB defect knowledge base built into the knowledge base retrieval module covers engineering experience such as defect types, common causes, potential risks, and repair suggestions. The specific calculation expression is as follows:

[0081] ,

[0082] in, K This represents a PCB defect knowledge base. This indicates the first item in the PCB defect knowledge base. i Each record contains fields such as defect type, cause, risk level, and remediation suggestions. Represents a multimodal feature set. R Representing defective knowledge information, Retrieve represents the knowledge base retrieval function. Its specific logic is as follows: first, a multimodal feature set is processed through a cross-modal encoder. The data is mapped to multimodal semantic feature vectors. At the same time, the text fields of each record in the PCB defect knowledge base are encoded into knowledge base semantic feature vectors. Then, PCB defect knowledge retrieval processing is performed based on the multimodal semantic feature vectors and the knowledge base semantic feature vectors.

[0083] Finally, the model-based knowledge fusion generation module integrates and reasones the initial answer sequence and defect knowledge information to obtain natural language answers. The knowledge base retrieval module and the knowledge fusion generation module infuse the model with professional experience from industrial scenarios, enabling the answers to not only include detection results but also practical value such as risk assessment and remediation suggestions. This enhances the professionalism of human-computer interaction and strengthens the model's generalization ability and practicality in complex and novel defect scenarios.

[0084] In this embodiment of the invention, after acquiring natural language question data, calling a pre-trained visual language model, and generating a natural language answer result based on the natural language question data, the defect location information, and the defect type label, the method further includes: using a visualization annotation tool to perform defect region selection and annotation processing on the initial circuit board image based on the defect location information to obtain a defect-annotated circuit board image; and performing visualization processing on the defect-annotated circuit board image and the natural language answer result to obtain visualization data.

[0085] In this embodiment of the invention, after obtaining the natural language response results, further visualization processing is required to achieve intuitive and efficient application of the detection results. First, a visualization annotation tool is used, with defect location information as the core basis. A coordinate mapping algorithm transforms abstract defect location parameters into pixel-level positioning data of the initial circuit board image. Then, precise bounding box annotation is performed on the defect areas in the initial circuit board image. The boundaries of the annotation boxes strictly conform to the actual distribution range of the defects, ensuring accurate visualization of the defect locations. Finally, a clearly labeled circuit board image with defect areas is obtained. Based on this, a systematic visualization process is performed on the labeled circuit board image and the natural language response results: For the labeled circuit board image, the visual presentation effect is optimized. Differentiated visual identifiers are used to enhance differentiation based on the differences in defect types. Simultaneously, defect confidence information is supplemented at the associated locations of the annotation boxes, making the location, range, and reliability of the defects intuitively identifiable. For the natural language response results, the information logic is organized through structured layout, precisely associating defect types, location details, core descriptions, and professional suggestions with the labeled circuit board image. Finally, this is integrated to form visualized data that combines visual intuitiveness with textual logic. The precise selection and annotation of the visualization tool transforms abstract defect location information into concrete visual identifiers, effectively reducing the cost of understanding defect locations and improving the readability of inspection results. The integrated visualization processing of defect-annotated circuit board images and natural language responses achieves the orderly integration of multi-dimensional information, preserving the professional semantic value of natural language responses while enhancing defect identifiability through visualization, providing efficient support for users to quickly grasp core inspection information. Simultaneously, standardized visualization data provides a unified carrier for archiving and comparing inspection results, supporting the clear tracing of defect changes through multiple comparisons of initial circuit board images. This provides intuitive and quantifiable evidence for production quality control and equipment anomaly analysis, further enhancing the practicality and adaptability of the PCB defect inspection process in industrial scenarios.

[0086] The circuit defect detection method based on a multimodal model in the embodiments of the present invention has been described above. The circuit defect detection device based on a multimodal model in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 2 One embodiment of the circuit defect detection device based on a multimodal model in this invention includes:

[0087] Data processing module 201: used to acquire an initial circuit board multi-source dataset, perform data preprocessing on the multi-source dataset, and obtain a target multi-source feature set;

[0088] Semantic space mapping module 202: used to call a pre-trained multimodal model to perform semantic space mapping processing on the target multi-source feature set to obtain a multimodal feature set;

[0089] Defect prediction module 203: used to call a pre-trained defect detection model to predict circuit board defects on the multimodal feature set, and obtain the defect location information and defect type label of the circuit board;

[0090] Natural language answer generation module 204: used to acquire natural language question data, call a pre-trained visual language model, and generate natural language answer results based on the natural language question data, the defect location information, and the defect type label.

[0091] Based on the same ideas as the methods in the above embodiments, the apparatus provided in this application can implement the methods in the above embodiments.

[0092] above Figure 2 The circuit defect detection device based on a multimodal model in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The circuit defect detection equipment based on a multimodal model in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0093] Figure 3 This is a schematic diagram of a circuit defect detection device based on a multimodal model provided in an embodiment of the present invention. The circuit defect detection device 300 based on the multimodal model can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the circuit defect detection device 300 based on the multimodal model. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the circuit defect detection device 300 based on the multimodal model to implement the steps of the circuit defect detection method based on the multimodal model provided in the above-described method embodiments.

[0094] The circuit defect detection device 300 based on a multimodal model may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The circuit defect detection device structure shown is not intended to limit the circuit defect detection device based on the multimodal model. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0095] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a circuit defect detection method based on a multimodal model.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A circuit defect detection method based on a multimodal model, characterized in that, include: Obtain an initial circuit board multi-source dataset, which includes initial circuit board images, initial circuit board design files, and initial circuit board text. The initial circuit board image is processed using a channel normalization algorithm to obtain an optimized circuit board image; the optimized circuit board image is then processed using a data augmentation algorithm to obtain an enhanced circuit board image. The pre-trained large visual model is invoked to extract features from the enhanced circuit board image to obtain image features; a file parsing tool is used to parse the initial circuit board design file to obtain file features; The initial circuit board text is semantically encoded using an encoder to obtain text features; the image features, file features, and text features are mapped using a modal linear layer mapping algorithm to obtain a target multi-source feature set. A pre-trained multimodal model is invoked to perform semantic space mapping on the target multi-source feature set to obtain a multimodal feature set; The pre-trained defect detection model is invoked to predict circuit board defects using the multimodal feature set, thereby obtaining the defect location information and defect type label of the circuit board. Acquire natural language question data, call a pre-trained visual language model, and generate a natural language answer result based on the natural language question data, the defect location information, and the defect type label.

2. The circuit defect detection method based on a multimodal model according to claim 1, characterized in that, The large-scale visual model includes a patch partitioning module, an encoding module, and a feature aggregation module, which are sequentially connected. The process of calling the pre-trained large-scale visual model to extract features from the enhanced circuit board image to obtain image features includes: The enhanced circuit board image is processed by patch segmentation module to obtain an image patch set. Based on the encoding module, global feature modeling is performed on the image patch set to obtain a feature vector set; The feature aggregation module performs feature aggregation processing on the feature vector set to obtain the image features.

3. The circuit defect detection method based on a multimodal model according to claim 1, characterized in that, The multimodal model includes an information extraction module and a feature fusion module, which are connected. The pre-trained multimodal model is invoked to perform semantic space mapping processing on the target multi-source feature set to obtain a multimodal feature set, including: Obtain multiple first query vectors; Based on the information extraction module and multiple first query vectors, the target multi-source feature set is processed to obtain an intermediate fusion feature set, which includes intermediate fusion features corresponding to each first query vector. The intermediate fused feature set is processed by the feature fusion module to obtain the multimodal feature set.

4. The circuit defect detection method based on a multimodal model according to claim 1, characterized in that, The defect detection model includes a self-attention module, a detection head decoder module, and a confidence calculation module, which are sequentially connected. The pre-trained defect detection model is used to predict circuit board defects on the multimodal feature set to obtain defect location information and defect type labels for the circuit board, including: Multiple second query vectors are obtained, and each second query vector is subjected to deformable attention focusing processing based on the self-attention module to obtain multiple third query vectors. Based on the detection head decoder module and multiple third query vectors, candidate box prediction processing is performed on the multimodal feature set to obtain a candidate box set. The candidate box set includes multiple candidate boxes, as well as the candidate box coordinates and defect category probability corresponding to each candidate box. The confidence calculation module calculates the confidence level of multiple defect category probabilities to obtain the defect probability corresponding to each candidate box. The defect location information and defect type label of the circuit board are determined based on multiple candidate box coordinates and multiple defect probabilities.

5. The circuit defect detection method based on a multimodal model according to claim 1, characterized in that, The visual language model includes a question encoding module, a question decoding module, a knowledge base retrieval module, and a knowledge fusion generation module, which are sequentially connected. The process of acquiring natural language question data, calling a pre-trained visual language model, and generating a natural language answer based on the natural language question data, the defect location information, and the defect type label includes: Based on the question encoding module, semantic encoding processing is performed on the natural language question data, the defect location information, and the defect type label to obtain the semantic features of the answer; Based on the question decoding module, the semantic features of the answer are processed to generate a sequence, resulting in an initial answer sequence; Based on the knowledge base retrieval module, PCB defect knowledge retrieval processing is performed on the multimodal feature set to obtain defect knowledge information; The knowledge fusion generation module integrates and reasons the initial answer sequence and the defective knowledge information to obtain natural language answer results.

6. The circuit defect detection method based on a multimodal model according to claim 1, characterized in that, After acquiring natural language question data, calling a pre-trained visual language model, and generating a natural language answer based on the natural language question data, the defect location information, and the defect type label, the process further includes: A visual annotation tool is used to select and annotate the defect area in the initial circuit board image based on the defect location information, resulting in a defect-annotated circuit board image. The defect-annotated circuit board image and the natural language response result are visualized to obtain visualized data.

7. A circuit defect detection device based on a multimodal model, characterized in that, include: Data processing module: used to acquire initial circuit board multi-source dataset, which includes initial circuit board image, initial circuit board design file and initial circuit board text; The initial circuit board image is processed using a channel normalization algorithm to obtain an optimized circuit board image; the optimized circuit board image is then processed using a data augmentation algorithm to obtain an enhanced circuit board image. The pre-trained large visual model is invoked to extract features from the enhanced circuit board image to obtain image features; a file parsing tool is used to parse the initial circuit board design file to obtain file features; The initial circuit board text is semantically encoded using an encoder to obtain text features; the image features, file features, and text features are mapped using a modal linear layer mapping algorithm to obtain a target multi-source feature set. Semantic space mapping module: used to call a pre-trained multimodal model to perform semantic space mapping processing on the target multi-source feature set to obtain a multimodal feature set; Defect prediction module: used to call a pre-trained defect detection model to predict circuit board defects on the multimodal feature set, and obtain the defect location information and defect type label of the circuit board; Natural Language Response Generation Module: This module acquires natural language question data, calls a pre-trained visual language model, and generates a natural language response based on the natural language question data, the defect location information, and the defect type label.

8. A circuit defect detection device based on a multimodal model, characterized in that, The circuit defect detection device based on the multimodal model includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the multimodal model-based circuit defect detection device to perform the steps of the multimodal model-based circuit defect detection method as described in any one of claims 1-6.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the various steps of the circuit defect detection method based on the multimodal model as described in any one of claims 1-6.

Images