Image selection method and device for model training, equipment and storage medium

Abstract

The invention discloses an image selection method and device for model training, equipment and a storage medium, and relates to the field of image detection. Receiving a new material image, extracting corresponding image source information, and extracting attribute information of a defect image for training a defect model from the image source information; the attribute information is used for extracting images meeting defect model training conditions from a defect material library; generating an image query instruction based on the selected target attribute information, and searching and returning a target defect map from the defect material library in an index manner; and constructing and generating a defect training set based on the target defect map. According to the scheme, the adaptive defect image is automatically screened by receiving the new material image, extracting the attribute information, generating the query instruction, indexing the target defect image and constructing the training set, and the method has the advantages that the adaptive defect image can be automatically and efficiently screened for the new material, so that the model training period is shortened, and the production efficiency is improved.

Description

Technical Field

[0001] This application relates to image detection, and more particularly to an image selection method, apparatus, device, and storage medium for model training. Background Technology

[0002] In the semiconductor manufacturing industry, defect detection of printed circuit boards (PCBs) and various other PCBs is a core step in ensuring product yield. Currently, the industry widely relies on artificial intelligence (AI) models to assist in defect identification tasks, thereby improving detection efficiency and accuracy. However, PCBs of different functions or types exhibit significant differences in physical characteristics, batch specifications, and process parameters, resulting in highly specific defect patterns. To meet the demands of high-precision detection, a vertical model design strategy is often adopted in engineering practice, which involves customizing a dedicated AI model for each specific PCB. While this strategy can optimize detection performance in a single scenario, it severely weakens the model's general adaptability, making it difficult to reuse existing model resources for detection tasks across different material types.

[0003] The key issue arising from this contradiction is that when new or unfamiliar material boards are introduced into the production line, engineers must rebuild dedicated models. Efficiently training such models heavily relies on massive and high-quality defect image datasets to support precise optimization of model parameters.

[0004] In traditional data preparation processes, engineers spend significant time and manpower. First, they use general AI models to initially screen raw images, extracting defect samples with high confidence levels. Then, they gradually accumulate sufficient training data through manual annotation. This process not only involves repetitive, labor-intensive operations but also significantly extends the model training cycle due to inefficient data screening. The entire chain, from acquiring new material images to final model deployment, often takes weeks or even months, severely disrupting the continuous operation of production lines and causing a decline in overall production efficiency. Especially in manufacturing environments with rapidly iterating materials, the lack of a mechanism to automatically associate and filter defect images based on the characteristics of new materials makes the construction of training datasets a major bottleneck restricting rapid model deployment. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for image selection in model training, which has the advantages of automatically and efficiently screening and adapting defect images for new materials, thereby shortening the model training cycle and improving production efficiency.

[0006] On the one hand, this application provides an image selection method for model training, the method comprising: The system receives new material images and extracts corresponding image source information. From the image source information, it extracts attribute information of defect images used to train a defect model. The attribute information is used to extract images from the defect material library that meet the conditions for training a defect model. Based on the selected target attribute information, an image query instruction is generated, and the target defect image is indexed, searched, and returned from the defect material library; A defect training set is generated based on the target defect map.

[0007] Specifically, all defect images in the defect material library contain corresponding image source information. The image source information is written into the image EXIF ​​or stored separately as a JSON file in a relational database, and a mapping relationship is established with the images, including an image table, an equipment table, and a part number table. The image table contains product part number, face number / layer, and defect size type information. The equipment table includes factory ID, equipment sequence, and camera shooting parameter information. The material table includes material process information, including category, solder resist, color, and surface treatment process.

[0008] Specifically, after obtaining the image query instruction, the target defect image is indexed, searched for, and returned from the defect material library, including: Parse the query conditions in the command and create a composite index; all query conditions are combined and filtered using AND logic. Sort and cache all images that meet the attribute filtering conditions to obtain an image collection.

[0009] Specifically, the image source information includes material information dimension, imaging parameter dimension, and image information dimension; The material information dimension is used to query defect images that meet the attributes of manufacturer, product, batch, and material, and has a mapping relationship with the material table; The imaging parameter dimension is used to query defect images that meet the camera shooting parameter attributes, and it has a mapping relationship with the device table; The image information dimension is used to query defect images that meet the attributes of material surface number, image area, defect size type, and CAM drawing file, and has a mapping relationship with the image table.

[0010] Specifically, when the number of sorted cached images exceeds the maximum threshold, vector extraction is performed on the new material image; Similarity matching calculation is performed on sorted cached images based on the extracted feature vectors; The target defect maps are reordered based on the returned similarity values ​​and the target number is output based on the TOP-K results.

[0011] Specifically, the defect images in the defect material library are processed by feature extraction and converted into fixed-dimensional embedding vectors, which are then stored in a feature vector library; and the embedding vectors are mapped to the defect images. The similarity matching calculation for the sorted cached images includes: The sorted cached images are imported into the Milvus vector database, and the imported vectors are clustered based on the Hierarchical Navigation Small World Graph (HNSW) or the Inverted File Index (IVF) to obtain the vector library index. The extracted feature vectors are spatially distanced with the vector library index, and the target defect map is output based on the TOP-K value index.

[0012] Specifically, the new material image is a panoramic image, and the vector extraction of the new material image includes: Each detection area of ​​the new material image is determined, and several images to be matched are cropped and extracted from it; wherein the number of images cropped from the defect-prone area is higher than that from other areas, and pixel overlap is allowed between different images to be matched; The images to be matched are assigned different label combinations based on their regional locations. Vectors are extracted based on these label combinations and then compared with the sorted cached images for similarity matching calculation.

[0013] On the other hand, this application provides an image selection device for model training, the device comprising: An extraction module is used to receive new material images and extract corresponding image source information, and extract attribute information of defect images for training defect models from the image source information; the attribute information is used to extract images that meet the conditions for training defect models from the defect material library; The output module is used to generate image query instructions based on the selected target attribute information, and to index and return the target defect image from the defect material library; A construction module is used to build a defect training set based on the target defect map.

[0014] In another aspect, this application provides a computer device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by the processor to implement the image selection method for model training described above.

[0015] In another aspect, this application provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image selection method for model training described above.

[0016] The beneficial effects of the technical solution provided in this application include at least the following: by receiving new material images in a structured manner and extracting image source information, effective management of new material data sources is achieved. By extracting key attributes from the image source information and generating accurate image query instructions based on these attributes, the system can directly perform efficient index searches from the defect material library. This approach avoids the inefficient steps of extensive manual screening and preliminary model screening in traditional methods. This efficient data preparation mechanism greatly shortens the training cycle of defect models, enabling factories to deploy customized AI detection models for new materials more quickly. This not only improves the adaptability and flexibility of the production line but also effectively reduces production delays and costs caused by long model iteration cycles. Attached Figure Description

[0017] Figure 1 This is a flowchart of the image selection method for model training provided in the embodiments of this application; Figure 2 This diagram illustrates the operation of an image retrieval interface under one possible scenario. Figure 3 A schematic diagram is shown illustrating one possible form of displaying defective images based on a query command index. Figure 4 This is a schematic diagram illustrating the cropping and extraction of images from areas with a high incidence of defects under one possible scenario; Figure 5 This is a structural block diagram of the image selection device for model training provided in the embodiments of this application; Figure 6 A structural block diagram of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0019] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0020] In traditional semiconductor manufacturing defect detection, model training for novel material boards faces the technical challenge of low data acquisition efficiency. Specifically, when introducing material boards with different physical properties or models and their variants, the defect image data screening process is dominated by manual intervention, making it impossible to quickly construct a high-quality defect training set that meets training conditions. Furthermore, the lack of structured processing of image source information leads to inaccurate attribute information extraction, thereby prolonging the iteration cycle from data collection to model deployment and limiting the model's adaptability to different materials. This negatively impacts the continuous operation of the production line and the real-time performance of defect detection.

[0021] For example, in a scenario where a wafer fab is processing new high-density interconnect PCBs, the initial defect images screened by a general AI model contain a large number of samples unrelated to the new material. Engineers need to manually verify each frame of the image to identify valid defects, a process prolonged by frame-by-frame analysis and susceptible to subjective interference. Furthermore, delays in the data preparation stage directly cause production line shutdowns, affecting the stability of the manufacturing cycle and preventing the defect detection process from responding to production demands in a timely manner.

[0022] If these issues are not addressed, the model training cycle will continue to be unable to adapt to the rapid production changeover pace, and the accuracy of defect detection will decrease due to insufficient training data, thereby increasing the risk of missed defects on the production line. In the long run, this will weaken the automation level and quality control capabilities of the semiconductor manufacturing process, increasing the likelihood of scrap.

[0023] Therefore, this application provides an image selection method for model training, such as... Figure 1 The flowchart shown illustrates the image selection method, which includes the following steps: S1. Receive the new material image and extract the corresponding image source information, and extract the attribute information of the defect image used to train the defect model from the image source information; In this embodiment, the "new material images" refer to images of new or specific batches of materials that require defect detection or are used to train defect models. These images typically contain visual information about the material to be inspected and form the basis for subsequent processing and analysis.

[0024] Image source information refers to metadata specific to images of new materials. It primarily describes the image's origin, shooting conditions, and material characteristics. This information may include, but is not limited to, the material category, production batch, imaging equipment parameters, and the image's own features, providing a basis for image classification, retrieval, and management.

[0025] The defect model used for training is designed for the material type or batch corresponding to the new material image, and is used to detect defects in the scanned images of the same material type or batch in the future.

[0026] The defect image library is a database that stores a large number of historical defect images and their corresponding source information. This library is an important resource for building defect training sets. By effectively querying it, specifically through attribute mapping, defect images that meet specific conditions can be quickly obtained.

[0027] In practical applications, images of new materials can be uploaded to a designated data storage area manually, with operators manually inputting or selecting image source information related to these images, such as material batch, production line number, and shooting date. Alternatively, images can be captured in real-time by a camera system on an automated production line and stored on a local server. In this case, the image source information may need to be manually reviewed and supplemented for association.

[0028] After receiving images of new materials and their source information, this source information needs to be analyzed to identify attributes valuable for training the defect model. For example, data analysts can manually review the text descriptions or tags in the source information to identify key attributes such as "FPC board," "OSP process," and "gold surface defects." These attribute information forms the basis for defining the image features required to train the defect model, and their accuracy directly affects the effectiveness of subsequent image selection.

[0029] S2. Generate an image query instruction based on the selected target attribute information, and search for and return the target defect image from the defect material library; The aforementioned attribute information is used to extract images from the defect material library that meet the conditions for training the defect model. Once the attribute information used to train the defect model is determined, this information can be used as filtering criteria to find matching images from the pre-established defect material library. For example, engineers can manually input or select these attributes as query parameters in the material library management interface, and the system will compare the images in the material library one by one based on these parameters to identify images that meet the conditions. This method can limit the filtering scope to defect images related to the characteristics of new material images, avoiding blind searching of the entire material library.

[0030] Based on this, an image query instruction is generated using the selected target attribute information. After determining the target attribute information to be queried, it needs to be converted into a query instruction that the system can recognize. For example, an SQL query statement can be manually constructed in the query interface, combining attributes such as "material type=FPC AND surface treatment=OSP AND defect area=gold surface" into a query string. This query instruction serves as a bridge for interaction with the defect material library, and its accuracy and completeness determine whether the required target defect image can be effectively retrieved.

[0031] Upon receiving an image query command, the defect image library retrieves data based on the query criteria specified in the command. For example, the library can employ a simple pagination and linear scanning method to traverse all stored defect images and their associated source information, comparing each image against all the criteria in the query command. Once a matching image is found, it is marked and collected, and finally, all images meeting the criteria are returned as the target defect images. This search method ensures that all images meeting the criteria are identified.

[0032] S3. Construct a defect training set based on the target defect map.

[0033] After obtaining the target defect images, these images will be used to build the training set for the defect model. For example, all returned target defect images can be manually copied to a designated folder, and data labelers can further annotate or classify these images to generate labeled training data. This organized and labeled image set constitutes the defect training set for training new defect models, providing basic data for model learning and optimization.

[0034] In summary, this method achieves effective management of new material data sources by receiving images of new materials in a structured manner and extracting image source information. By extracting key attributes from the image source information and generating precise image query instructions based on these attributes, the system can directly and efficiently index and search from the defect material library. This approach avoids the inefficient steps of extensive manual screening and preliminary model screening in traditional methods. For example, in the FPC board example mentioned above, traditional methods may require weeks or even months to accumulate sufficient high-quality "FPC", "OSP", and "gold surface" defect images, while the method of this application can obtain a large number of target defect images that meet the criteria in a short time through a single precise query.

[0035] Therefore, the quality and specificity of the defect training set generated based on these rapidly acquired target defect maps are improved. This efficient data preparation mechanism greatly shortens the training cycle of the defect model, enabling factories to deploy customized AI detection models for new materials more quickly. This not only improves the adaptability and flexibility of the production line but also effectively reduces production delays and costs caused by long model iteration cycles. The method in this application achieves rapid iteration from data to model through an automated and intelligent data screening process, providing effective technical support for the rapid deployment and performance optimization of AI models in the semiconductor manufacturing field.

[0036] In some embodiments of this application, although a defect material library is proposed to store defect images, the storage and mapping methods of image source information are unclear during its implementation, which may lead to low query efficiency and chaotic data management, thereby affecting the efficiency of model training.

[0037] Therefore, the defect material library of this application stores all defective images and their corresponding source information. The source information can be historically stored and written to the image EXIF ​​(Exchangeable Image File Format), or stored separately as a JSON (JavaScript Object Notation) file in a relational database, with a mapping relationship established between it and the image. JSON format, with its lightweight and easy-to-parse characteristics, is very suitable for the storage and exchange of structured data.

[0038] Relational databases offer powerful data management, querying, and indexing capabilities, efficiently handling large amounts of structured image source information and supporting complex query logic. They store unique identifiers for image files (such as file paths, hash values, or database primary keys) and associate them with corresponding image source information records; alternatively, when image source information is embedded in EXIF ​​data, the mapping is embedded. This mapping is crucial for achieving efficient retrieval and location of defective images based on image source information.

[0039] The attribute information in the image source annotation can be written into the image generation stage or manually annotated. For example, attributes such as image resolution, focal length, and camera parameters are automatically written into the image, while more detailed attributes, such as defect area, type, and angle, can be annotated in the model output from previous stages or manually. This information forms the basis for structured storage and retrieval of defect images, ensuring that each image comes with contextual data that can be queried and analyzed, thereby avoiding information silos and data ambiguity.

[0040] In some specific embodiments, the image source information may include material information dimension, imaging parameter dimension, and image information dimension; The material information dimension is a structured classification of the image source information, specifically used to describe the attributes of the material itself associated with the image, such as manufacturer, product model, production batch, material type, and process characteristics. Its function is to identify and filter defective images related to specific materials from a macroscopic perspective. In practice, it can be used to query defective images that meet the attributes of manufacturer, product, batch, and material, and it has a mapping relationship with the material list. The imaging parameter dimension is another structured classification of image source information, which focuses on recording the settings and environmental conditions of the imaging equipment used in the image acquisition process, such as camera model, lens focal length, light intensity, gain setting, resolution, shooting angle, etc. Its function is to ensure that when training the defect model, defect images taken under similar imaging conditions can be selected, and there is a mapping relationship between the equipment table. The image information dimension is the third structured classification of image source information. It is used to describe the inherent attributes and content characteristics of the image itself, such as specific areas of materials in the image (e.g., board surface, solder pads, gold fingers), types of defects (e.g., short circuit, open circuit, foreign objects), size range of defects, and association information with computer-aided manufacturing (CAM) drawings. Its function is to perform fine classification and retrieval of image content, and it has a mapping relationship with the image table.

[0041] Correspondingly, in some embodiments, for image source information data stored in JSON format files, its stored content may include the following attribute parameters: Includes an image table containing attribute information such as product part number, facet, layer, and defect size type; Equipment list: Includes factory ID, equipment sequence, camera shooting parameters (such as lens, gain, light source, channels, resolution, exposure, ISO) and other attribute information; Part number list; includes material process information, such as category (FPC, HDI, CSP), solder resist, color, and surface treatment process (OSP, electroplating, nickel-palladium-gold).

[0042] When a line scan image of a PCB board is captured, its relevant source information is extracted automatically or manually. For example, the image might originate from a camera with the serial number XYZ from "Factory A," with shooting parameters such as "Lens model L1, gain G1, light source S1, channel C1, and resolution R1"; the PCB board's part number is "P-001"; for defect images stored in the database, defect information is also recorded, such as the defect being located on the "top layer," the defect type being "short circuit," the material belonging to the "FPC" category, using "green solder mask," and the surface treatment process being "nickel-palladium-gold," etc. Some of this source information, such as the shooting date, camera model, and lens parameters, can be directly written into the EXIF ​​header of the defect image file. In this way, even if the image file is transmitted independently, its core metadata can be carried along. Meanwhile, more detailed and structured information is stored in a relational database.

[0043] For each new defect image, a new record is added to the image table, containing "Product Part Number P-001", "Surface / Layer Top", "Defect Size Type Short Circuit", and a unique identifier pointing to the image file storage path. In the equipment table, a new or updated record is added, containing "Factory ID Factory A", "Equipment Sequence XYZ", and "Camera Shooting Parameters Lens Model L1, Gain G1, Light Source S1, Channel C1, Resolution R1". In the material table, a new or updated record is added, containing "Category FPC", "Solder Resistance Green", "Color Green", and "Surface Treatment Process Nickel-Palladium-Gold". Records in these tables are linked through foreign keys; for example, the product part number in the image table can be linked to the material information in the material table, and the equipment ID in the image table can be linked to the equipment information in the equipment table, thus establishing a complete mapping relationship. When querying "all short-circuit defect images on FPC boards taken by equipment from Factory A", the system can quickly locate and return defect images that meet the criteria by querying the equipment table, material table, and image table.

[0044] The above design scheme not only reduces the workload of manual screening and labeling and reduces human error, but also ensures the quality and relevance of training data, thereby accelerating the iteration and deployment of AI models. Ultimately, it improves the adaptability and training efficiency of defect detection models in the semiconductor manufacturing field and effectively alleviates the problem of excessively long training time for new material board models. In some embodiments, the process of selecting specific attribute information of an image source can be controlled by the user through a visual UI interface, such as... Figure 2 The diagram illustrates one possible scenario for the image retrieval interface. The parameter adjustment interface allows for fine-tuning based on machine parameters, software attributes, region attributes, and defect attributes. To focus on searching for defect images in specific areas (such as text areas, copper surfaces, or gold lines), specific parameters can be selected. To specifically select defects of particular type, size, and coordinates, parameters such as "defect type," "area range," "shape," "aspect ratio range," and "layer" can be set, and the system will perform targeted filtering based on these selected attributes.

[0045] Furthermore, because the database is used to store images for filtering defective images, the data can be indexed using query commands to quickly retrieve images. Specifically: The query conditions in the parsing instruction are used to create a composite index, where all query conditions are combined and filtered using AND logic. All images that meet the attribute filtering conditions are sorted and cached to obtain an image set.

[0046] When attributes are input via a graphical interface, the instruction can be a JSON object containing query parameters, an SQL query statement, or a set of parameters for a predefined API call. This can be achieved through pattern matching, syntax analysis, or validation based on predefined data structures such as XML or JSON Schema. For example, a dedicated query parser can convert the instruction string into an internally operable data structure, such as a key-value dictionary or a conditional expression tree, for subsequent processing. Creating a composite index refers to an index structure created in a database or data storage system based on a combination of multiple attributes (fields). Its purpose is to optimize the performance of multi-condition queries, enabling the system to quickly locate records that simultaneously meet multiple conditions. Using AND logic to filter all query conditions means that an image is considered to meet the filtering requirements only if all specified attributes of the image completely match the conditions in the query instruction. This strict logical combination ensures high accuracy of the filtering results, preventing irrelevant or partially matching images from being included in the result set.

[0047] For example Figure 3 This diagram illustrates one possible form of displaying defect images based on a query command index. The left side shows various attribute parameters input according to actual needs, where selected parameters are used in an AND logical query, while unselected parameters are ignored. The right side shows the defect images output by the index that meet the query conditions.

[0048] Through the above technical solution, this application can significantly improve the efficiency and accuracy of indexing and retrieving target defect images from the defect material library. By parsing the query conditions in the instruction and building a composite index, the system can perform efficient data retrieval for complex query requests, avoiding full table scans or inefficient single-field index lookups, thereby greatly shortening the image retrieval time.

[0049] In some of the solutions described above in this application, index lookup and return of target defect images are proposed to construct a defect training set. However, in this process, because the defect images in the database are constantly increasing and massive, the number of sorted cached images selected is still excessive (e.g., setting a threshold of 5000 images) when the number of sorted cached images exceeds the maximum threshold. When the number of sorted cached images exceeds the maximum threshold, directly using all images may lead to processing redundancy and inefficiency, affecting the quality of the training set and the model iteration speed. At this point, a secondary selection mechanism needs to be triggered to extract target defect images that are more suitable for training the current new material model. To address this, this application proposes an improved image selection method, as follows: 1. When the number of sorted cached images exceeds the maximum threshold, perform vector extraction on the new material images; Of course, the maximum threshold can be a fixed value, such as 1,000 images; or, the maximum threshold can be a parameter that is dynamically adjusted according to system resources (such as memory and CPU load) to ensure that system performance is not affected by a large number of image processing.

[0050] Vector extraction from virgin material images can be performed using specific algorithms to convert them into fixed-dimensional numerical vectors that represent the image's visual features or semantic information. This process forms the basis for subsequent similarity matching, transforming image data into a computable mathematical form. For example, a pre-trained convolutional neural network (CNN) model (such as ResNet, VGG, etc.) can be used as a feature extractor. The virgin material image is input into a specific layer of the model, and the activation value of that layer is output as the image's feature vector. Alternatively, unsupervised learning methods such as autoencoders or variational autoencoders (VAEs) can be used to train the model to learn a low-dimensional representation of the image, thereby obtaining the feature vector.

[0051] 2. Calculate similarity matching for sorted cached images based on the extracted feature vectors; Similarity matching compares the feature vectors of new images with those of sorted cached images to quantify their similarity. This result serves as the basis for subsequent re-sorting and filtering. For example, cosine similarity can be used to calculate the cosine of the angle between two vectors; a value closer to 1 indicates higher similarity. Alternatively, Euclidean distance can be used to measure the distance between two vectors in multidimensional space; a smaller distance indicates higher similarity.

[0052] 3. Re-sort based on the returned similarity values ​​and output the target defect map based on the number of targets according to the TOP-K results.

[0053] Based on the similarity matching calculation results, the priority of the sorted cached images is adjusted so that images with higher similarity to the new material images are ranked first. This ensures that the images selected later are most relevant to the current needs. For example, the similarity value can be used as the sorting key to arrange all sorted cached images in descending order, with the image with the highest similarity value at the top; alternatively, different weights or scores can be assigned based on the similarity value, and then weighted and sorted by considering other factors (such as image quality, defect type, etc.).

[0054] Furthermore, from the reordered image list, the K images with the highest similarity are selected as the final target defect maps. Here, K is a preset number of targets, designed to precisely control the number of output images, avoid redundancy, and ensure the quality of the training set. This value of K can be dynamically determined based on the needs of training the model or user input, for example, adjusted according to the sample requirements of the current training phase.

[0055] Through the aforementioned mechanism, this method, when matching images of virgin materials with images in the defect material library, can perform deep matching and fine-tuning by initially screening a large number of candidate images and then utilizing the features of the virgin material images themselves. This dynamic, feature vector-based screening method enables the system to efficiently and accurately identify the defect samples most relevant to the current virgin material image from massive amounts of data, thereby avoiding the computational burden and inefficiency caused by processing a large number of low-relevance images. This not only ensures the quality and relevance of the defect training set but also significantly shortens the cycle from data collection to model iteration, improving the efficiency and accuracy of model training.

[0056] The essence of similarity matching calculation is to calculate the distance in the vector space. Therefore, for defective images in the defective material library, feature extraction is also required, and they need to be converted into fixed-dimensional embedding vectors and stored in the feature vector library. Furthermore, a mapping relationship is established between the embedding vectors and the defective images.

[0057] Performing similarity matching calculations on sorted cached images may include the following steps: A. Import the sorted cached images into the Milvus vector database, and cluster the imported vectors based on the Hierarchical Navigation Small World Graph (HNSW) or Inverted File Index (IVF) to obtain the vector library index. After importing sorted cached images into the Milvus vector database, the database organizes these vectors based on specific indexing algorithms to further accelerate the similarity search process. The Milvus vector database is a high-performance database system specifically designed for vector similarity search, capable of efficiently storing, indexing, and querying massive amounts of vector data. Hierarchical Navigation Small World Graph (HNSW) is a graph-based approximate nearest neighbor search algorithm that accelerates the search by constructing a multi-level graph structure, exhibiting excellent query performance and recall in high-dimensional spaces. Inverted File Index (IVF) is a clustering-based indexing method that first divides the vector space into multiple clusters, each corresponding to an inverted list. During a query, only a few relevant clusters need to be searched, thus reducing computational load. Clustering and organizing the imported vectors using these indexing algorithms significantly improves the efficiency of subsequent similarity matching.

[0058] B. Calculate the spatial distance between the extracted feature vectors and the vector library index, and output the target defect map based on the TOP-K value index.

[0059] In other embodiments, when the number of sorted cached images is below a minimum threshold, the method may fail to acquire enough target defect images, resulting in an insufficient training set and affecting model training efficiency. Therefore, it is necessary to utilize global defect images to expand the index range.

[0060] C. When the number of sorted cached images is lower than the minimum threshold, the image difference is determined based on the target number; D. Generate an index library vector based on all defect images in the defect material library, calculate the spatial distance between the extracted feature vector and the index library vector, and output the target defect map based on the image difference index.

[0061] The key feature of this approach is that, when the number of candidate sorted and cached images meets the requirements, spatial similarity matching is prioritized to select the most suitable defect image for model training on the current unfamiliar material board. When the number of candidate sorted and cached images is insufficient, a global similarity search mechanism is activated to ensure the sufficiency of the defect training set. This mechanism ensures that even when the initial screening results are insufficient, the most relevant supplementary images can be intelligently found from a broader material library, thereby guaranteeing the integrity and quality of the defect training set and avoiding the problem of insufficient data affecting model training efficiency.

[0062] In some embodiments, the virgin material image is typically a panoramic image acquired by a line scan camera, which includes several Pcs particles with the same structure. Therefore, one or more Pcs particle images can be selected for matching, and the size of the Pcs particle image is larger than the size of the defect image. The defect image is usually a cropped image of a specific defect point detected in the Pcs particle image. Therefore, when the virgin material image is a panoramic image, directly extracting vectors from the entire image may not distinguish the defect features of different regions, especially since the importance of high-defect areas is not prioritized, leading to reduced accuracy and efficiency in similarity matching. Therefore, vector extraction from the virgin material image can be further refined into the following steps: a. Determine the detection areas of the new material image, and crop and extract several images to be matched from them; the number of images cropped from the defect-prone areas is higher than that from other areas, and pixel overlap is allowed between different images to be matched; This step can be specified during cropping, automatically identifying and extracting functional areas from CAM images. Alternatively, image segmentation algorithms, such as deep learning-based semantic segmentation models, can be used to automatically identify and divide different functional or structural regions in the image as detection areas. Cropping and extracting several images to be matched transforms the determined detection areas into independent image patches that can be used for subsequent vector extraction and similarity matching.

[0063] The cropping can be performed by using a fixed-size sliding window to traverse and crop within each detection area, generating a series of image blocks with or without overlap; or, it can adaptively crop an image containing the entire detection area based on the shape and size of the detection area, or crop a local area image that may contain defects based on a preset defect size range.

[0064] The goal of cropping images from high-defect areas is to optimize image cropping strategies by allocating more computational resources and matching priorities to areas where historical data or experience indicates a higher probability of defect occurrence. This helps to discover potential defects more efficiently with limited computational resources, thereby improving the recall rate and overall efficiency of defect detection.

[0065] Figure 4 This diagram illustrates a possible scenario for cropping and extracting images from high-defect-rate areas. This strategy can identify regions with high defect density in different detection areas based on historical defect data statistics, and set higher sampling rates or smaller cropping steps for these regions during cropping. Alternatively, based on expert experience or process flow, certain key functional areas or vulnerable areas can be pre-defined as high-defect-rate areas, and the number of images cropped from these areas can be increased accordingly. Furthermore, allowing pixel overlap between different images to be matched aims to ensure that potential defect features are not segmented or missed due to their location at the cropping boundary during the cropping process, thereby improving the completeness and accuracy of defect detection.

[0066] b. Set different label combinations for the images to be matched according to their regional locations, extract vectors based on the label combinations, and perform similarity matching calculations with the sorted cached images.

[0067] By assigning region-based label combinations to the images to be matched, contextual information can be provided for subsequent vector extraction and similarity matching. This helps distinguish image features from different regions, making the matching process more targeted and better reflecting the regional characteristics of defects. The label combination can assign a unique region ID to each detection area and label the images to be matched cropped from that area with that region ID; alternatively, different labels can be assigned to the images to be matched based on the type of detection area (e.g., solder pad area, circuit area, gold finger area, etc.), and combined with their relative coordinate information in the panoramic image to form a label combination.

[0068] Vector extraction based on label combinations means that customized feature extraction strategies can be used for images in different regions, or feature vectors from different regions can be differentiated after extraction. This helps to capture the unique defect patterns of different regions and improve the accuracy and discriminativeness of feature representation.

[0069] For example, in areas with a high incidence of defects, images to be matched are assigned higher weight values, resulting in more defective images being matched. This concentrates more attention and computational resources on these key areas. Other areas are assigned lower weight values, resulting in fewer defective images being matched. Ultimately, these region-labeled images are matched using parameters, and similarity matching is calculated based on feature vectors and sorted cached defective images.

[0070] Through this partitioning, weighting, and labeled vector extraction and matching mechanism, the proposed solution can analyze the defect features in panoramic images more precisely, improve the accuracy and efficiency of similarity matching, and effectively solve the problem of insufficient matching accuracy and efficiency when directly extracting panoramic image vectors, especially when processing complex panoramic images.

[0071] Figure 5 This is a structural block diagram of the image selection device for model training provided in this application embodiment. The device includes: The extraction module 510 is used to receive new material images and extract corresponding image source information; extract attribute information of defect images used for training defect models from the image source information; the attribute information is used to extract images that meet the conditions for training defect models from the defect material library; Output module 520 is used to generate image query instructions based on selected target attribute information, and to index and return target defect images from the defect material library; Module 530 is used to build a defect training set based on the target defect map.

[0072] It should be noted that the image selection device for model training provided in this embodiment is only illustrated by the above-described division of functional modules / units. In practical applications, the above functions can be assigned to different functional modules / units as needed, that is, the internal structure of the image selection device for model training can be divided into different functional modules / units to complete all or part of the functions described above. Furthermore, the implementation method of the image selection method for model training provided in the above method embodiments and the implementation method of the image selection device for model training provided in this embodiment belong to the same concept. The specific implementation process of the image selection device for model training provided in this embodiment is detailed in the above method embodiments and will not be repeated here.

[0073] Figure 6 This application provides a structural block diagram of a computer device according to an exemplary embodiment. The computer device can be a desktop computer, laptop computer, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. The processor and memory can be connected via a bus or other means. The processor can be a central processing unit (CPU) or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a graphics processing unit (GPU), an embedded neural network processor (NPU) or other dedicated deep learning coprocessor, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0074] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0075] Those skilled in the art will understand that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0076] This application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, which, when executed by a processor, implements the methods described in the above-described method embodiments. Those skilled in the art will understand that implementing all or part of the processes in the methods described above can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0077] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. An image selection method for model training, characterized in that, The method includes: The system receives new material images and extracts corresponding image source information. From the image source information, it extracts attribute information of defect images used to train a defect model. The attribute information is used to extract images from the defect material library that meet the conditions for training a defect model. Based on the selected target attribute information, an image query instruction is generated, and the target defect image is indexed, searched, and returned from the defect material library; A defect training set is generated based on the target defect map.

2. The method according to claim 1, characterized in that, All defect images in the defect material library contain corresponding image source information. The image source information is written into the image EXIF ​​or stored separately as a JSON file in a relational database, and a mapping relationship is established with the images, including an image table, an equipment table, and a part number table. The image table contains product part number, face number / layer, and defect size type information. The equipment table includes factory ID, equipment sequence, and camera shooting parameter information. The material table includes material process information, including category, solder resist, color, and surface treatment process.

3. The method according to claim 1, characterized in that, After obtaining the image query instruction, the target defect image is indexed and retrieved from the defect material library, including: Parse the query conditions in the command and create a composite index; all query conditions are combined and filtered using AND logic. Sort and cache all images that meet the attribute filtering conditions to obtain an image collection.

4. The method according to claim 1, characterized in that, The image source information includes material information dimension, imaging parameter dimension, and image information dimension; The material information dimension is used to query defect images that meet the attributes of manufacturer, product, batch, and material, and has a mapping relationship with the material table; The imaging parameter dimension is used to query defect images that meet the camera shooting parameter attributes, and it has a mapping relationship with the device table; The image information dimension is used to query defect images that meet the attributes of material surface number, image area, defect size type, and CAM drawing file, and has a mapping relationship with the image table.

5. The method according to claim 1, characterized in that, When the number of sorted cached images exceeds the maximum threshold, vector extraction is performed on the new material image; Similarity matching calculation is performed on sorted cached images based on the extracted feature vectors; The target defect maps are reordered based on the returned similarity values ​​and the target number is output based on the TOP-K results.

6. The method according to claim 5, characterized in that, The defect images in the defect material library are processed by feature extraction and converted into fixed-dimensional embedding vectors, which are then stored in the feature vector library; and the embedding vectors are mapped to the defect images. The similarity matching calculation for the sorted cached images includes: The sorted cached images are imported into the Milvus vector database, and the imported vectors are clustered based on the Hierarchical Navigation Small World Graph (HNSW) or the Inverted File Index (IVF) to obtain the vector library index. The extracted feature vectors are spatially distanced with the vector library index, and the target defect map is output based on the TOP-K value index.

7. The method according to claim 5, characterized in that, The image of the new material is a panoramic view, and the vector extraction of the image of the new material includes: Each detection area of ​​the new material image is determined, and several images to be matched are cropped and extracted from it; wherein the number of images cropped from the defect-prone area is higher than that from other areas, and pixel overlap is allowed between different images to be matched; The images to be matched are assigned different label combinations based on their regional locations. Vectors are extracted based on these label combinations and then compared with the sorted cached images for similarity matching calculation.

8. An image selection device for model training, characterized in that, The device includes: An extraction module is used to receive new material images and extract corresponding image source information, and extract attribute information of defect images for training defect models from the image source information; the attribute information is used to extract images that meet the conditions for training defect models from the defect material library; The output module is used to generate image query instructions based on the selected target attribute information, and to index and return the target defect image from the defect material library; A construction module is used to build a defect training set based on the target defect map.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the image selection method for model training as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the image selection method for model training as described in any one of claims 1 to 7.

Images