Shell defect detection method and device and model training method and device

Abstract

The embodiment of the invention provides a tube shell defect detection method and device and a model training method and device, and relates to the technical field of computers. The tube shell defect detection method comprises the following steps: acquiring a region image to be detected and a material category corresponding to the region image; wherein the area image is an image of a to-be-detected area extracted from the chip tube shell image; inputting the regional image into a target module of a semantic segmentation model in a defect detection system, performing defect detection on each pixel in the regional image by the target module, and outputting a detection result; wherein the semantic segmentation model comprises a plurality of semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the regional image. Therefore, the target of detecting the defects of the tube shell can be well achieved, and meanwhile, the chip tube shell defect detection method has good compatibility with chip tube shells of various models.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method and device for detecting tube and shell defects, and a method and device for model training. Background Art

[0002] Current semiconductor production processes are constantly improving, and high-end processes place extremely high demands on product quality and defect rates. However, as process improvements continue to shrink product sizes, this negatively impacts the defect rate, which in turn increases.

[0003] At the same time, there are also significant differences in the packaging methods of integrated circuits. For example, the common CQFP (Ceramic Quad Flat Package), CSOP (Ceramic Small Outline Package), SMD (Surface-Mount Device), CFP (Ceramic Flat Package), CLCC (Ceramic Leadless Chip Carrier), and DIP (Dual In-line Package) packaging methods are very different. Each packaging method also has different pin packages, and the same pins can still have different styles of packages. Therefore, for the package shell and tube, it has the characteristics of a complex model. Traditional methods mainly rely on manual inspection methods for quality inspection.

[0004] With the advancement of semiconductor chip production independence, demand for chip packages is booming, and production is rapidly increasing. While traditional manual inspection methods can avoid the problem of excessive variations in package models, they still present other unavoidable issues. For example, manual inspection cannot provide a stable and clear view of the package area, making it difficult to effectively detect package defects. Purely manual inspection is no longer able to meet the growing demand for semiconductor chip packages. Summary of the Invention

[0005] The embodiments of the present application provide a method and apparatus for detecting tube and shell defects, and a method and apparatus for training a model. Taking into account the commonalities among the materials of chip tube and shells of different models, the semantic segmentation model used for tube and shell defect detection includes multiple semantic segmentation modules corresponding to different material categories. This allows for good compatibility with various types of chip tube and shells while achieving the goal of tube and shell defect detection.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting tube and shell defects, comprising:

[0007] Obtaining an image of the area to be inspected and a material category corresponding to the image of the area to be inspected; wherein the image of the area to be inspected is extracted from the chip shell image;

[0008] The above-mentioned area image is input into the target module of the semantic segmentation model in the defect detection system, and the above-mentioned target module performs defect detection on each pixel in the above-mentioned area image and outputs the detection result; wherein the above-mentioned semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the above-mentioned target module is a semantic segmentation module corresponding to the same material category as the above-mentioned area image.

[0009] In one embodiment, the defect detection system further includes an instance segmentation model;

[0010] Obtain the image of the area to be detected and the material category corresponding to the above area image, including:

[0011] Inputting the chip-shell image into the instance segmentation model for segmentation processing to obtain a segmentation result output by the instance segmentation model, wherein the segmentation result indicates the location information of the area to be detected and the material category to which it belongs;

[0012] The region image is extracted from the chip package image based on the position information, and it is determined that the region image corresponds to the material category.

[0013] In one embodiment, the detection result is a first segmentation mask having the same size as the regional image;

[0014] The above method further includes:

[0015] Based on the position information of the area to be inspected in the chip-shell image, the first segmentation mask is converted into a second segmentation mask having the same size as the chip-shell image.

[0016] In a second aspect, an embodiment of the present application provides a model training method, comprising:

[0017] Acquire multiple regional images, and the material categories and first annotation results corresponding to each of the multiple regional images; wherein the multiple regional images include images of the areas to be inspected in chip packages of various models, and the first annotation results include a defect category label for each pixel in the corresponding regional images;

[0018] Inputting a target region image from the plurality of region images into a target module of a semantic segmentation model, wherein the target module performs defect detection on each pixel in the target region image and outputs a detection result; wherein the semantic segmentation model includes a plurality of semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image;

[0019] Determining a first prediction loss based on a difference between the first annotation result corresponding to the target area image and the detection result, and determining a total loss based on the first prediction loss;

[0020] With the goal of minimizing the above total loss, adjust the parameters of the above objective modules.

[0021] In one embodiment, acquiring multiple regional images includes:

[0022] Acquire a plurality of chip-package images, wherein the plurality of chip-package images include images of the plurality of chip-package models;

[0023] Inputting the plurality of chip-shell images into the trained instance segmentation model for segmentation processing, respectively, to obtain a first segmentation result output by the instance segmentation model;

[0024] Based on the first segmentation result, extracting a first region image of the region to be inspected from the plurality of chip-shell images;

[0025] The plurality of region images are determined based on the first region images.

[0026] In one embodiment, determining the plurality of regional images based on each of the first regional images includes:

[0027] performing image enhancement processing on at least a portion of the first region images to generate second region images corresponding to at least a portion of the first region images;

[0028] A set of the first region images and the second region images is determined as the plurality of region images.

[0029] In one embodiment, each of the plurality of chip-shell images corresponds to a second annotation result, and the second annotation result includes position information and a material category label of the area to be detected in the corresponding chip-shell image;

[0030] The training process of the above instance segmentation model includes:

[0031] Inputting a target chip-shell image from the plurality of chip-shell images into the instance segmentation model for segmentation processing, obtaining a second segmentation result output by the instance segmentation model, wherein the second segmentation result indicates location information and a material category of a region to be detected in the target chip-shell image;

[0032] determining a second prediction loss based on a difference between the second annotation result and the second segmentation result corresponding to the target chip-shell image;

[0033] With the goal of minimizing the second prediction loss, the parameters of the instance segmentation model are adjusted.

[0034] In one embodiment, the instance segmentation model is pre-trained;

[0035] Adjust the parameters of the above instance segmentation model, including:

[0036] Adjust some parameters of the above instance segmentation model.

[0037] In one embodiment, the target module includes an encoder, which includes a first submodule and a second submodule. The first submodule is used to extract the feature vector of the target area image, and the second submodule is used to enhance the feature vector using an attention mechanism and output the enhanced feature vector.

[0038] In one embodiment, the defect category label indicates the true category to which the pixel belongs, and the detection result includes a target probability that each pixel in the target area image belongs to the corresponding true category;

[0039] The above method further includes:

[0040] Determining focus loss based on each of the above target probabilities in the above detection results;

[0041] The total loss is determined based on the first predicted loss, including:

[0042] Based on the first prediction loss and the focus loss, the total loss is determined.

[0043] In one embodiment, the target module has been pre-trained;

[0044] Adjust the parameters of the above target modules, including:

[0045] Adjust some parameters of the above target modules.

[0046] In a third aspect, an embodiment of the present application provides a device for detecting tube and shell defects, comprising:

[0047] an acquisition module configured to acquire an image of a region to be inspected and a material category corresponding to the image of the region; wherein the image of the region to be inspected is extracted from the chip package image;

[0048] The detection module is configured to input the above-mentioned area image into the target module of the semantic segmentation model in the defect detection system, and the above-mentioned target module performs defect detection on each pixel in the above-mentioned area image and outputs the detection result; wherein the above-mentioned semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the above-mentioned target module is a semantic segmentation module corresponding to the same material category as the above-mentioned area image.

[0049] In a fourth aspect, an embodiment of the present application provides a model training device, comprising:

[0050] an acquisition module configured to acquire a plurality of regional images, and a material category and a first annotation result corresponding to each of the plurality of regional images; wherein the plurality of regional images include images of regions to be inspected in chip packages of various models, and the first annotation result includes a defect category label for each pixel in the corresponding regional image;

[0051] a detection module configured to input a target region image from the plurality of region images into a target module of a semantic segmentation model, wherein the target module performs defect detection on each pixel in the target region image and outputs a detection result; wherein the semantic segmentation model includes a plurality of semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image;

[0052] a determination module configured to determine a first prediction loss based on a difference between the first annotation result corresponding to the target area image and the detection result, and to determine a total loss based on the first prediction loss;

[0053] The adjustment module is configured to adjust the parameters of the target module with the goal of minimizing the total loss.

[0054] In a fifth aspect, an embodiment of the present application provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory. When the processor executes the computer program, the tube and shell defect detection method described in any implementation method in the first aspect is implemented, or the model training method described in any implementation method in the second aspect is implemented.

[0055] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting tube and shell defects as described in any implementation method in the first aspect is implemented, or the model training method as described in any implementation method in the second aspect is implemented.

[0056] In the seventh aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the tube and shell defect detection method described in any implementation method in the first aspect, or implements the model training method described in any implementation method in the second aspect.

[0057] The solution provided in this embodiment of the application takes into account the commonalities among the materials of chip casings of different models. Therefore, the semantic segmentation model used for casing defect detection includes multiple semantic segmentation modules corresponding to different material categories. Furthermore, the solution provided in this embodiment of the application can be further divided into a model training solution and a casing defect detection solution.

[0058] In the model training scheme, multiple regional images, as well as the material categories and first annotation results corresponding to each of the multiple regional images, can be obtained; wherein, the multiple regional images include images of the area to be detected in chip tube shells of various models, and the first annotation results include the defect category label of each pixel in the corresponding regional image. Afterwards, the target regional image among the multiple regional images can be input into the target module of the semantic segmentation model, and the target module performs defect detection on each pixel in the target regional image and outputs the detection result; wherein, the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target regional image. Then, based on the difference between the first annotation result corresponding to the target regional image and the detection result, the first prediction loss can be determined, and the total loss can be determined based on the first prediction loss. Then, the parameters of the target module can be adjusted with the goal of minimizing the total loss. By performing pixel-level annotation on each regional image, distinguishing the material of the area to be inspected displayed by each regional image, and setting up multiple semantic segmentation modules corresponding to different material categories in the semantic segmentation model, the semantic segmentation modules corresponding to the same material category can be trained based on several regional images corresponding to the same material category. This allows the semantic segmentation modules to perform pixel-level defect detection on regional images corresponding to the same material category, thereby improving the defect detection performance of the semantic segmentation model. Applying this trained semantic segmentation model to the defect detection system can effectively achieve the goal of tube and shell defect detection while maintaining good compatibility with various chip tube and shell models.

[0059] In the tube and shell defect detection scheme, an image of the area to be detected and the material category corresponding to the area image can be obtained; wherein, the area image is an image of the area to be detected extracted from the chip tube and shell image. Afterwards, the area image can be input into the target module of the semantic segmentation model of the defect detection system, and the target module performs defect detection on each pixel in the area image and outputs the detection result; wherein, the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the area image. By setting multiple semantic segmentation modules corresponding to different material categories in the semantic segmentation model, this scheme can enable each semantic segmentation module to perform pixel-level defect detection on the area image of the same material category as it corresponds to, thereby improving the defect detection effect of the semantic segmentation model, and can well achieve the goal of tube and shell defect detection while having good compatibility with various types of chip tubes and shells. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings will make the technical solutions and other beneficial effects of the present application apparent.

[0061] Figure 1 This is a flow chart of the model training method provided in the embodiment of the present application;

[0062] Figure 2 is a flowchart of the training process of the instance segmentation model in an embodiment of the present application;

[0063] Figure 3 This is a schematic diagram of the YOLOv8 model;

[0064] Figure 4 This is a schematic diagram of the Unet network;

[0065] Figure 5 is a schematic diagram of the second submodule of the encoder in the semantic segmentation module in an embodiment of the present application;

[0066] Figure 6 This is a flow chart of a tube and shell defect detection method according to an embodiment of the present application;

[0067] Figure 7 Schematic diagram of the structure of the model training device provided in the embodiment of the present application;

[0068] Figure 8 Schematic diagram of the structure of the tube shell defect detection device provided in an embodiment of the present application.

[0069] Reference numerals: 701 - acquisition module, 702 - detection module, 703 - determination module, 704 - adjustment module, 801 - acquisition module, 802 - detection module. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0071] In the description of this application, it should be noted that, unless otherwise specified or limited, the term "and / or" herein is merely a description of an association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " herein, unless otherwise specified, generally indicates that the associated objects are in an "or" relationship.

[0072] As previously mentioned, with the advancement of semiconductor chip production independence, demand for chip housings is rapidly increasing, and production is also rapidly increasing. While traditional manual inspection methods can avoid the problem of excessive variations in housing models, other unavoidable issues remain. For example, manual inspections cannot provide a stable and clear view of the housing area, making it difficult to effectively detect housing defects. Furthermore, traditional manual inspection methods suffer from inconsistent and non-uniform standards, time constraints, and the difficulty of sustained quality inspection under strong light. These issues also hinder the effective detection of housing defects. Purely manual inspections are no longer able to meet the growing demand for semiconductor chip housings.

[0073] The embodiments of the present application provide a tube and shell defect detection method and device, and a model training method and device, which can well achieve the goal of tube and shell defect detection while having good compatibility with various models of chip tubes and shells.

[0074] Figure 1 This is a flow chart of the model training method provided in the embodiment of the present application. The method can be executed by any platform, server, or device cluster with computing and processing capabilities. The method includes the following steps:

[0075] S101: Acquire multiple regional images, and the material categories and first annotation results corresponding to the multiple regional images; wherein the multiple regional images include images of the areas to be inspected in chip packages of various models, and the first annotation results include defect category labels for each pixel in the corresponding regional images;

[0076] S103: Inputting a target region image from the multiple region images into a target module of a semantic segmentation model, wherein the target module performs defect detection on each pixel in the target region image and outputs a detection result; wherein the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image;

[0077] S105: determining a first prediction loss based on a difference between the first annotation result and the detection result corresponding to the target area image, and determining a total loss based on the first prediction loss;

[0078] S107: Adjust the parameters of the target module with the goal of minimizing the total loss.

[0079] Figure 1 The solution provided by the corresponding embodiment takes into account the commonalities of the materials of chip tube shells of different models, so that the semantic segmentation model used for tube shell defect detection includes multiple semantic segmentation modules corresponding to different material categories. In this solution, multiple regional images, as well as the material categories and first annotation results corresponding to each of the multiple regional images can be obtained; wherein, the multiple regional images include images of the area to be inspected in chip tube shells of multiple models, and the first annotation results include the defect category label of each pixel in the corresponding regional image. Afterwards, the target region image among the multiple region images can be input into the target module of the semantic segmentation model, and the target module performs defect detection on each pixel in the target region image and outputs the detection result; wherein, the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image. Then, based on the difference between the first annotation result corresponding to the target region image and the detection result, a first prediction loss can be determined, and a total loss can be determined based on the first prediction loss. Then, the parameters of the target module can be adjusted with the goal of minimizing the total loss. By performing pixel-level annotation on each regional image, distinguishing the material of the area to be inspected displayed by each regional image, and setting up multiple semantic segmentation modules corresponding to different material categories in the semantic segmentation model, the semantic segmentation modules corresponding to the same material category can be trained based on several regional images corresponding to the same material category. This allows the semantic segmentation modules to perform pixel-level defect detection on regional images corresponding to the same material category, thereby improving the defect detection performance of the semantic segmentation model. Applying this trained semantic segmentation model to the defect detection system can effectively achieve the goal of tube and shell defect detection while maintaining good compatibility with various chip tube and shell models.

[0080] Next, steps S101 to S107 will be described.

[0081] In step S101, multiple region images are acquired, along with the material categories and first annotation results corresponding to each of the region images. The region images include images of the regions to be inspected within various chip packages. The first annotation results include a defect category label for each pixel in the corresponding region images.

[0082] Among them, any area to be inspected in the chip shell can be, for example, one of the bonding fingers, pins and cavity bottom. The material category corresponding to the regional image is the material category to which the area to be inspected displayed by the regional image belongs. A single material category can be, for example, metal or porcelain. Optionally, in order to further improve the defect detection effect, the metal can also be divided into multiple subcategories. Among them, the multiple subcategories can be divided according to the different characteristics or uses of the metal, etc., which are not specifically limited here. It should be pointed out that the material category corresponding to the regional image can be manually set or determined by other means, which is not specifically limited here.

[0083] The first annotation result can be a segmentation mask of the same size as the corresponding regional image. Each pixel in the segmentation mask is used to record the defect category label of the pixel at the same position as the pixel in the regional image. The first annotation result can be manually annotated or annotated using other annotation methods, which are not specifically limited here.

[0084] The defect category label may correspond to a plurality of preset categories, and the plurality of categories may include at least one defect category. Furthermore, the plurality of categories may also include no defects. A single defect category may be, for example, any one of scratches, dirt, cracks, and white spots on porcelain. The defect category label may indicate the true category to which the pixel belongs, such as indicating the true category to which the pixel belongs among the plurality of categories. In one example, the defect category label may be a vector, the length of which is the same as the number of the plurality of categories. Moreover, one element in the vector has a value of 1, and the other element values are 0. The category corresponding to the element with a value of 1 in the vector is the true category to which the pixel corresponding to the vector belongs.

[0085] It should be noted that the aforementioned multiple chip housing models can be pre-selected chip housing samples based on actual housing defect detection requirements. These multiple models can cover all models that require testing. Furthermore, the various defect categories covered by these multiple chip housing models can cover all defect categories that require testing. Furthermore, the defective chip housings within these multiple models must exhibit sufficient diversity. For example, each chip housing should represent defects in various states, such as varying sizes, textures, depths, angles, shapes, and locations.

[0086] In practice, images of each selected chip-case sample can be captured to obtain a chip-case image of each chip-case sample. The plurality of region images can be determined based on at least a portion of the obtained chip-case image. For example, at least a portion of the obtained chip-case images can be used to form a training set, and the plurality of region images can be determined based on each chip-case image in the training set.

[0087] To maximize inspection results, a standardized sampling device can be used to capture images of the selected chip package samples. This ensures that the chip package images of each sample have a uniform screen size and minimum resolution, while also minimizing the effects of viewing angle and lighting variations. Furthermore, the sampling device must include a well-defined camera, lens, light source, and lighting angle.

[0088] In one embodiment, the method for acquiring the above-mentioned multiple region images may include: acquiring multiple chip-shell images, the multiple chip-shell images including images of multiple models of chip-shells; inputting the multiple chip-shell images into a trained instance segmentation model for segmentation processing, thereby obtaining a first segmentation result output by the instance segmentation model; extracting a first region image of the area to be detected from the multiple chip-shell images based on the first segmentation result; and determining the above-mentioned multiple region images based on each first region image.

[0089] The plurality of chip-shell images may be the chip-shell images in the training set described above. The first segmentation result may indicate location information of the region to be inspected in the corresponding chip-shell image. It should be understood that this location information is the location information of the region to be inspected in the chip-shell image. This location information may include, for example, the coordinates of a target corner (e.g., the upper left corner) of a bounding box of the region to be inspected, as well as the width and height of the bounding box.

[0090] Furthermore, the trained instance segmentation model can also be used to predict the material category of the region to be inspected in the chip-case image. Based on this, the first segmentation result can also indicate the material category of the region to be inspected in the corresponding chip-case image. After extracting a first region image of the region to be inspected from the chip-case image based on the location information of the region to be inspected indicated by the first segmentation result, the material category corresponding to the first region image can be determined.

[0091] When determining the above-mentioned multiple regional images based on each first regional image, as an implementation method, each first regional image can be directly used as the above-mentioned multiple regional images. In practice, defect samples are usually difficult to collect and the sample size is generally small. In order to expand the training samples and obtain more regional images for training the semantic segmentation model, as another implementation method, at least part of the first regional images can be subjected to image enhancement processing to generate second regional images corresponding to each of the at least part of the first regional images, and then the collection of each first regional image and each second regional image is determined as the above-mentioned multiple regional images. Among them, the at least part of the first regional images can be images that all have tube and shell defects. The image enhancement processing may include at least one of flipping, rotation, and grayscale transformation. In addition, for any second regional image, it is determined that the second regional image and its corresponding first regional image correspond to the same material category.

[0092] In one embodiment, each of the plurality of chip-case images corresponds to a second annotation result, which includes location information and a material category label of the region to be detected in the corresponding chip-case image. An instance segmentation model can be trained based on the plurality of chip-case images. The second annotation results corresponding to the chip-case images can be manually annotated or annotated using other methods, which are not specifically limited herein.

[0093] The material category label indicates the material category to which the area to be detected belongs. Furthermore, the material category label may correspond to multiple preset material categories, and the material category label indicates the material category to which the area to be detected belongs within the multiple material categories. In one example, the material category label may be a vector whose length is the same as the number of the multiple material categories. Furthermore, one element in the vector has a value of 1, and the other elements have values of 0. The material category corresponding to the element with a value of 1 in the vector is the material category to which the area to be detected corresponding to the vector belongs.

[0094] The training process of the instance segmentation model can be as follows Figure 2 As shown. Among them, Figure 2 Flowchart of the training process of the instance segmentation model in the embodiment of the present application. The training process includes the following steps:

[0095] Step S201: Inputting a target chip-case image from the plurality of chip-case images into an instance-based segmentation model for segmentation processing, obtaining a second segmentation result output by the instance-based segmentation model, wherein the second segmentation result indicates the location information and material category of the region to be detected in the target chip-case image;

[0096] Step S203: determining a second prediction loss based on a difference between the second annotation result corresponding to the target chip package image and the second segmentation result;

[0097] Step S205: Adjust the parameters of the instance segmentation model with the goal of minimizing the second prediction loss.

[0098] Figure 2 The solution provided by the corresponding embodiment not only enables the instance segmentation model to predict the position information of the area to be detected in the chip tube shell image, but also enables the instance segmentation model to predict the material category to which the area to be detected in the chip tube shell image belongs. This helps to automatically and accurately configure the corresponding material category for the regional image during the training data preparation stage of the semantic segmentation model. Compared with the manual configuration method, it can improve the efficiency of material category configuration and save labor costs.

[0099] Next, steps S201, S203, and S205 are described.

[0100] In step S201, a target chip-shell image among the plurality of chip-shell images may be input into an instance segmentation model for segmentation processing to obtain a second segmentation result output by the instance segmentation model. The second segmentation result indicates the position information and the material category of the region to be detected in the target chip-shell image.

[0101] As an implementation method, the base model of the instance segmentation model can be the YOLOv8 model. YOLOv8 is a version of the YOLO (You Only Look Once) series, which is a deep learning model for real-time object detection. The YOLO series is well-known for its efficient detection speed and good accuracy, and is widely used in various computer vision tasks. The model structure of the YOLOv8 model can be as follows: Figure 3 As shown. Among them, Figure 3 It is a schematic diagram of the YOLOv8 model.

[0102] It should be noted that the YOLOv8 model includes a backbone feature network. Figure 3 C1 to C5 are shown in the backbone feature network and can represent convolutional neural networks. In practice, the backbone feature network can use CSPDarknet as the base model. It consists of several convolutional neural networks and a cycle of C2f modules. Each C2f module also includes several convolution modules, pooling layers, and channel segmentation operation modules, ultimately resulting in three feature layers. CSPDarknet is an improved convolutional neural network architecture based on the Darknet framework. C2f is a modular design that is often used to improve the network's feature extraction capabilities.

[0103] In addition, the YOLOv8 model also includes a Feature Pyramid Network (FPN). Figure 3P3 to P7, shown in Figure 1, can each represent a feature layer and are part of a feature pyramid network. The feature pyramid network performs feature extraction. Using the C2f module and two-dimensional image convolution, the feature pyramid network continuously upsamples and downsamples the three feature layers extracted in the previous step, ultimately outputting a three-dimensional feature map and a prototype coefficient.

[0104] In addition, the YOLOv8 model also Figure 3 As shown in , it includes a Prediction head module, an NMS (Non-Maximum Suppression) module, a protonet module, a Crop module, and a Threshold module. In an example, the Prediction head module can be called a detection head module, the protonet module can be called a coefficient extraction module, the Crop module can be called a cropping module, and the Threshold module can be called a threshold module.

[0105] The Prediction head module can receive the feature map from the feature pyramid network, extract the mask coefficient of the target area from the feature map, and output the mask coefficient. The NMS module can receive the mask coefficient from the Prediction head module, perform non-maximum suppression on the mask coefficient, and output the real mask coefficient of the target area to be segmented. The protonet module can receive the prototype coefficient from the feature pyramid network, and output the image mask based on the prototype coefficient. The multiplication result of the image mask output by the protonet module and the mask coefficient output by the NMS module can be input into the Crop module. The Crop module can be used to clear the mask outside the boundary. The Threshold module can be used to filter the segmentation results. Since the YOLOv8 model is a well-known technology, it will not be elaborated on here.

[0106] Each time a chip-shell image is input into the instance segmentation model, one or more images can be selected from the multiple chip-shell images as target chip-shell images, and the target chip-shell images are input into the instance segmentation model for segmentation processing to obtain a second segmentation result output by the instance segmentation model. The second segmentation result indicates the location information of the area to be detected in the target chip-shell image and the material category to which it belongs.

[0107] In one embodiment, the second annotation result may further include a segmentation mask of the region to be inspected in the corresponding chip-case image. The segmentation mask is the same size as the chip-case image. In the segmentation mask, pixels belonging to the region to be inspected have a value of 1, while other pixels have a value of 0. Accordingly, the second segmentation result may further include a segmentation mask of the region to be inspected in the target chip-case image.

[0108] In step S203, a second prediction loss is determined based on the difference between the second annotation result and the second segmentation result corresponding to the target chip / package image. In practice, any loss function that can determine a prediction loss based on the difference between a true value and a predicted value can be used to determine the second prediction loss based on the difference between the second annotation result and the second segmentation result corresponding to the target chip / package image. For example, this loss function may include a cross-entropy loss or a binary cross-entropy loss.

[0109] In step S205, the parameters of the instance segmentation model are adjusted with the goal of minimizing the second prediction loss. As an implementation method, to improve model training efficiency, the instance segmentation model can be pre-trained. When adjusting the parameters of the instance segmentation model, some parameters of the instance segmentation model can be adjusted.

[0110] It's important to note that the parameters of the pre-trained instance segmentation model can be pre-trained using the COCO dataset. Some parameters of the instance segmentation model (such as those of the main architecture) can be fixed, allowing for network weight migration and ultimately achieving excellent instance segmentation results with small sample sizes. The COCO dataset is a widely used computer vision dataset, primarily used for tasks such as image recognition, object detection, and image segmentation.

[0111] Furthermore, when adjusting the parameters of the instance segmentation model, some parameters of the instance segmentation model can be adjusted in response to the current training round of the instance segmentation model being less than or equal to the preset round. The value of the preset round can be 45, 50, or 55, etc., and can be set according to actual needs and is not specifically limited here. Thus, some parameters can be frozen for training for the preset rounds. After the first preset rounds of training are completed, all parameters of the instance segmentation model can be updated. Then, training is continued until the loss function converges. This not only improves the model training efficiency, but also ensures that the instance segmentation model has a good instance segmentation effect.

[0112] As previously described, image acquisition can be performed on each selected chip-case sample to obtain chip-case images of each chip-case sample. At least a portion of these images can then be used to form a training set. To evaluate model performance and prevent overfitting, in one embodiment, a training set and a validation set can be determined based on the chip-case images of each chip-case sample. The training set and validation set can include different chip-case images. Furthermore, each chip-case image in the validation set corresponds to a second annotation result. Thus, during the training of the instance segmentation model, the validation set can be used to evaluate the performance of the instance segmentation model.

[0113] Next see Figure 1 In step S103, the target region image from the plurality of region images is input into the target module of the semantic segmentation model. The target module then performs defect detection on each pixel in the target region image and outputs a detection result. The semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image.

[0114] It should be noted that the material categories corresponding to the above-mentioned multiple semantic segmentation modules can cover all material categories that need to be detected. Since the network structure of the Unet network can meet the requirements of defect detection under small sample and small target conditions, in order to improve defect detection efficiency, the above-mentioned multiple semantic segmentation modules can be implemented based on the Unet network.

[0115] Figure 4 This is a schematic diagram of the Unet network. The Unet network can include an encoder and a decoder. Figure 4 The "Image input" in it can represent image input. "conv 3ⅹ3, ReLu" can represent a convolution layer followed by a ReLU (Rectified Linear Unit) activation function layer, and the convolution kernel size of the convolution layer is 3ⅹ3. Among them, the role of ReLU is to perform nonlinear transformation on the output of the convolution layer. "copy and crop" can represent copying and cropping. "max pool 2ⅹ2" can represent a maximum pooling layer with a convolution kernel size of 2ⅹ2. "up-conv 2ⅹ2" can represent a deconvolution layer with a convolution kernel size of 2ⅹ2. "conv 1ⅹ1" can represent a convolution layer with a convolution kernel size of 1ⅹ1. The encoder can be as follows Figure 4 As shown in [1], a five-layer convolutional neural network is used to continuously downsample, while simultaneously outputting feature extraction results at four scales, along with the resulting encoded feature vector. The decoder receives the encoder's final feature vector and continuously upsamples it through multiple rounds of convolution. After each upscaling, it is concatenated and fused with the encoder's output vector at the corresponding scale until the scale is restored to the same level as the input. Since the Unet network is well-known, it will not be elaborated on here.

[0116] In one embodiment, the above-mentioned multiple semantic segmentation modules all include an encoder, which can introduce an attention mechanism to enhance features, thereby improving the effectiveness of the extracted feature vectors and further improving the perception ability of the model. For example, the encoder can include a first submodule and a second submodule, the first submodule being used to extract the feature vectors of the regional image, and the second submodule being used to enhance the feature vectors using the attention mechanism and output the enhanced feature vectors. The output layer of the first submodule can be connected to the input layer of the second submodule, so that the output of the first submodule can serve as the input of the second submodule.

[0117] Furthermore, the attention mechanisms employed by the second submodule may include channel attention and / or spatial attention. Channel attention and spatial attention are attention mechanisms used in deep learning to enhance the feature extraction capabilities of convolutional neural networks. They focus on important information in the input feature map in different ways. The channel attention mechanism focuses on the channel dimension of the feature map, aiming to emphasize important features by assigning different weights to different channels. The spatial attention mechanism focuses on the spatial dimension of the feature map, aiming to emphasize important regions by assigning different weights to different spatial locations.

[0118] Below, we take the attention mechanism adopted by the second submodule as an example, including the channel attention mechanism and the spatial attention mechanism. Figure 5 The second submodule is further introduced. Figure 5 Schematic diagram of the second submodule of the encoder in the semantic segmentation module in the embodiment of the present application. The feature vector as the input of the second submodule can be as follows Figure 5 As shown in , the feature vector is input to the second submodule. The second submodule can first enhance the feature vector using a channel attention mechanism to generate a first processing result, and then enhance the fusion result of the feature vector and the first processing result using a spatial attention mechanism to generate a second processing result, and then output the fusion result of the first processing result and the second processing result as the enhanced feature vector.

[0119] Each time a region image is input to the semantic segmentation model, one or more images can be selected from the multiple region images as target region images. The target region images are then input into the target module of the semantic segmentation model. The target module then performs defect detection on each pixel in the target region image and outputs the detection results. The target module is a semantic segmentation module corresponding to the same material category as the target region image.

[0120] It should be noted that when the defect category labels correspond to multiple categories as described above, the target module can predict the category to which each pixel in the target area image belongs within the multiple categories, and the detection result output by the target module can indicate the category to which each pixel in the target area image belongs within the multiple categories. In one example, the detection result can include the probability that each pixel in the target area image belongs to each of the multiple categories. Furthermore, the detection result can be a segmentation mask of the same size as the target area image, and each pixel in the segmentation mask can record the probability that the pixel in the target area image at the same position as the pixel belongs to each of the multiple categories.

[0121] In step S105 , a first prediction loss may be determined based on a difference between the first annotation result and the detection result corresponding to the target area image, and a total loss may be determined based on the first prediction loss.

[0122] In practice, any loss function that can determine the prediction loss based on the difference between the true value and the predicted value can be used to determine the first prediction loss based on the difference between the first annotation result and the detection result corresponding to the target area image.

[0123] Furthermore, given that the Dice Loss loss function generally performs well in semantic segmentation tasks, the Dice Loss loss function can be used to determine the first prediction loss based on the difference between the first annotation result and the detection result corresponding to the target area image. Dice Loss is a loss function used to evaluate model performance in image segmentation tasks. Dice Loss is very effective for datasets with imbalanced categories because it directly measures the proportion of overlapping areas rather than simply calculating the error for each pixel.

[0124] The calculation formula of Dice Loss used in the embodiment of the present application can be the following formula (1):

[0125]

[0126] Here, X may represent the first annotation result corresponding to the target area image, and Y may represent the detection result corresponding to the target area image.

[0127] After determining the first predicted loss, the total loss can be determined based on the first predicted loss. As one implementation, the first predicted loss can be directly used as the total loss. Given that defects to be inspected typically occupy a small area and the ratio of positive and negative samples is relatively unbalanced, another implementation can be to determine the focus loss based on the inspection results, and then determine the total loss based on the first predicted loss and the focus loss. Determining the total loss based on the first predicted loss and the focus loss can improve the effectiveness of the determined total loss.

[0128] Furthermore, the defect category label can indicate the true category to which the pixel belongs, and the detection result can include the target probability that each pixel in the target area image belongs to the corresponding true category. When determining focus loss based on the detection result, the focus loss can be determined based on the target probabilities in the detection result.

[0129] In this embodiment of the present application, the Focal Loss function can be used to determine the focal loss. Focal Loss is a loss function used to address class imbalance, and is particularly effective in object detection tasks. Focal Loss is an improvement on the standard cross-entropy loss, designed to reduce the contribution of easily classified samples to the loss, thereby focusing more on difficult-to-classify samples.

[0130] The calculation formula of the Focal Loss function used in the embodiment of the present application can be as shown in the following formula (2):

[0131] Focal Loss(p t )=-(1-p t ) γ log(p t ) (2)

[0132] Among them, p t It can represent the target probability in the detection result. γ is a parameter that can be adjusted to reduce the loss contribution of the majority sample and increase the loss ratio of the minority sample.

[0133] When determining the total loss based on the first predicted loss and the focal loss, as one implementation, the sum of the first predicted loss and the focal loss can be used as the total loss. As another implementation, weight values associated with the first predicted loss and the focal loss can be preset, so that the first predicted loss and the focal loss can be weighted and summed based on the weight values, and the resulting calculation result can be used as the total loss.

[0134] In step S107, the parameters of the target module are adjusted with the goal of minimizing the total loss. As an implementation method, in order to improve the efficiency of model training, the above-mentioned multiple semantic segmentation modules may have been pre-trained. When adjusting the parameters of the target module, some parameters of the target module may be adjusted. Furthermore, in response to the current training round of the target module being less than or equal to the preset round, some parameters of the target module may be adjusted. Among them, the value of the preset round may be 45, 50 or 55, etc., and may be set according to actual needs, and is not specifically limited here. Therefore, some parameters may be frozen for training for the preset rounds first, and all parameters of the target module may be opened for update after the preset rounds of training are completed, and then training is continued until the loss function converges. This not only improves the efficiency of model training, but also ensures that the target module has a good defect detection effect.

[0135] In one embodiment, the validation set described above is further associated with a first set of region images. Each region image in the first set can be determined based on the segmentation results of each chip-shell image in the validation set using a trained instance segmentation model. Furthermore, each region image in the first set corresponds to a first annotation result. To evaluate model performance and prevent overfitting, in one embodiment, the first set can be used to evaluate the performance of the semantic segmentation model during its training.

[0136] After the semantic segmentation model is trained, it can be integrated into the defect detection system, allowing the defect detection system to use the semantic segmentation model to detect tube and shell defects. Furthermore, the trained instance segmentation model can be integrated into the defect detection system, allowing the defect detection system to use the instance segmentation model to perform instance segmentation processing.

[0137] Figure 6 This is a flow chart of a method for detecting tube and shell defects in an embodiment of the present application. The method can be executed by any platform, server, or device cluster with computing and processing capabilities. The method includes the following steps:

[0138] Step S601: Acquire an image of the region to be inspected and a material category corresponding to the image of the region; wherein the image of the region to be inspected is extracted from the chip package image;

[0139] Step S603: Input the regional image into the target module of the semantic segmentation model in the defect detection system, and the target module performs defect detection on each pixel in the regional image and outputs the detection result; wherein, the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the regional image.

[0140] Figure 6 The solution provided by the corresponding embodiment takes into account that the materials of chip tube shells of different models have certain commonalities. By setting multiple semantic segmentation modules corresponding to different material categories in the semantic segmentation model, each semantic segmentation module can perform pixel-level defect detection on the regional image corresponding to the same material category, thereby improving the defect detection effect of the semantic segmentation model. While being able to achieve the goal of tube shell defect detection well, it can have good compatibility with various models of chip tube shells.

[0141] Next, steps S601 and S603 are further described.

[0142] In step S601, an image of a region to be inspected and a material category corresponding to the image of the region are obtained, wherein the image of the region to be inspected is extracted from the chip package image.

[0143] It should be noted that the image of the region to be inspected and the material category corresponding to the region image may be provided by a user or other device. Alternatively, if the defect detection system includes an instance segmentation model, in step S601, sub-steps S6011 and S6013 may be executed to obtain the image of the region to be inspected and the material category corresponding to the region image.

[0144] like Figure 6 As shown, in sub-step S6011, the chip shell image mentioned in step S601 can be input into the instance segmentation model in the defect detection system for segmentation processing, and a segmentation result output by the instance segmentation model is obtained. The segmentation result indicates the location information of the area to be detected and the material category to which it belongs. Then, in sub-step S6013, a regional image can be extracted from the chip shell image based on the location information, and the material category corresponding to the regional image can be determined. It should be noted that by obtaining the regional image to be detected and the material category corresponding to the regional image through the segmentation result of the chip shell image based on the instance segmentation model, it is possible to quickly and accurately extract the area to be detected in the chip shell image and the material category to which the area to be detected belongs.

[0145] In one embodiment, the chip-case image mentioned in step S601 may be an image of a chip-case to be inspected for defects in real life, or an image of a chip-case sample as described above. For example, when determining a training set and a validation set based on the chip-case images of each chip-case sample, a test set may also be determined, where the training set, validation set, and test set include different chip-case images. The chip-case image mentioned in step S601 may be included in the test set.

[0146] It should be noted that each chip-shell image in the test set corresponds to a second annotation result. This test set can be used to test instance segmentation models. Furthermore, the test set is associated with a second set of region images. Each region image in the second set can be determined based on the segmentation results of each chip-shell image in the test set using a trained instance segmentation model. This second set can be used to test semantic segmentation models.

[0147] In step S603, the regional image is input into the target module of the semantic segmentation model in the defect detection system. The target module performs defect detection on each pixel in the regional image and outputs the detection results. The semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is the semantic segmentation module corresponding to the same material category as the regional image. The implementation process of step S603 can be found in the description of step S103 above and will not be repeated here.

[0148] In one embodiment, the inspection result is a first segmentation mask of the same size as the regional image. To facilitate rapid detection of defects in the chip-shell image, the first segmentation mask can be converted into a second segmentation mask of the same size as the chip-shell image based on the position of the inspected region in the chip-shell image.

[0149] As described above, the solution provided by the embodiments of this application takes into account the commonalities in the materials of chip packages of different models. This allows the semantic segmentation model used for package defect detection to include multiple semantic segmentation modules corresponding to different material categories. Furthermore, this solution can be further subdivided into a model training solution and a package defect detection solution.

[0150] The model training scheme provided in the embodiments of the present application distinguishes the materials of the areas to be detected displayed by each regional image by performing pixel-level annotation on each regional image, and sets multiple semantic segmentation modules corresponding to different material categories in the semantic segmentation model. Based on several regional images corresponding to the same material category, the semantic segmentation modules corresponding to the same material category can be trained. This enables the semantic segmentation modules to perform pixel-level defect detection on regional images corresponding to the same material category, thereby improving the defect detection effect of the semantic segmentation model. Applying the trained semantic segmentation model to the defect detection system can effectively achieve the goal of tube and shell defect detection while maintaining good compatibility with various chip tube and shell models.

[0151] In addition, this model training scheme also implements the training of the instance segmentation model, which not only enables the instance segmentation model to predict the location information of the area to be detected in the chip tube and shell image, but also enables the instance segmentation model to predict the material category to which the area to be detected in the chip tube and shell image belongs. This helps to automatically and accurately configure the corresponding material category for the regional image during the training data preparation stage of the semantic segmentation model. Compared with the manual configuration method, it can improve the efficiency of material category configuration and save labor costs.

[0152] The tube and shell defect detection solution provided in the embodiment of the present application, by setting multiple semantic segmentation modules corresponding to different material categories in the semantic segmentation model, can enable each semantic segmentation module to perform pixel-level defect detection on the regional image corresponding to the same material category, thereby improving the defect detection effect of the semantic segmentation model. While being able to well achieve the goal of tube and shell defect detection, it can have good compatibility with various models of chip tubes and shells.

[0153] Furthermore, this chip and housing defect detection solution integrates a trained instance-based segmentation model into the defect detection system. Based on the segmentation results of the chip and housing image, the system can obtain the image of the region to be inspected and the corresponding material category. This enables rapid and accurate extraction of the region to be inspected and the material category to which it belongs in the chip and housing image.

[0154] In summary, the solution provided in the embodiments of the present application has the following technical features:

[0155] 1. Based on actual production needs, this embodiment proposes a strategy for separately extracting different chip housing regions and materials, then using a specially trained defect detection algorithm for inspection. This strategy meets the accuracy requirements for detecting small defects while also enabling high-frequency product changeovers. Even if the incoming materials include previously untested product models, rapid and low-cost changeovers are possible.

[0156] 2. When constructing a semantic segmentation model, the embodiment of the present application introduces a plug-and-play attention mechanism based on the characteristics of material shooting, thereby enhancing the partial weight of the data extracted by the neural network and weakening the weight of other parts, thereby focusing the network's attention on the most important part.

[0157] 3. The embodiment of the present application selects a combination function in the loss function design of the semantic segmentation model, and adopts the combination method of Focal Loss and Dice Loss to reduce the impact of the mismatch of defect sample size and solve the problem of erroneous convergence under small area and small sample conditions.

[0158] It should be understood that the solution provided by the embodiments of this application can achieve the function of transforming complex backgrounds into pure backgrounds, while also preventing small defects from being missed due to compressed resolution. This can effectively improve detection results and meet the needs of manufacturers who frequently change models during actual production. In addition, it can detect tube and shell defects even when there are significant differences in packaging methods, ultimately achieving the effect of rapid model change at a low cost, meeting the small-batch, multi-batch, and multi-model production needs faced by some manufacturers in actual production.

[0159] Figure 7This is a schematic diagram of the structure of the model training device provided in an embodiment of the present application. The model training device can be applied to any platform, server, or device cluster with computing and processing capabilities. The model training device includes:

[0160] Acquisition module 701 is configured to acquire multiple regional images, as well as material categories and first annotation results corresponding to each of the multiple regional images; wherein the multiple regional images include images of the regions to be inspected in chip packages of various models, and the first annotation results include defect category labels for each pixel in the corresponding regional images;

[0161] The detection module 702 is configured to input a target region image from the plurality of region images into a target module of a semantic segmentation model, and have the target module perform defect detection on each pixel in the target region image and output a detection result; wherein the semantic segmentation model includes a plurality of semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image;

[0162] a determination module 703 configured to determine a first prediction loss based on a difference between the first annotation result corresponding to the target area image and the detection result, and to determine a total loss based on the first prediction loss;

[0163] The adjustment module 704 is configured to adjust the parameters of the target module with the goal of minimizing the total loss.

[0164] In one embodiment, the acquisition module 701 may be further configured to:

[0165] Acquire a plurality of chip-package images, wherein the plurality of chip-package images include images of the plurality of chip-package models;

[0166] Inputting the plurality of chip-shell images into the trained instance segmentation model for segmentation processing, respectively, to obtain a first segmentation result output by the instance segmentation model;

[0167] Based on the first segmentation result, extracting a first region image of the region to be inspected from the plurality of chip-shell images;

[0168] The plurality of region images are determined based on the first region images.

[0169] In one embodiment, the acquisition module 701 may be further configured to:

[0170] performing image enhancement processing on at least a portion of the first region images to generate second region images corresponding to at least a portion of the first region images;

[0171] A set of the first region images and the second region images is determined as the plurality of region images.

[0172] In one embodiment, each of the plurality of chip-shell images corresponds to a second annotation result, and the second annotation result includes position information and a material category label of the area to be detected in the corresponding chip-shell image;

[0173] The above-mentioned model training device may further include:

[0174] a segmentation module (not shown) configured to input a target chip-case image from the plurality of chip-case images into the instance-based segmentation model for segmentation processing, and obtain a second segmentation result output by the instance-based segmentation model, wherein the second segmentation result indicates location information and a material category of a target region in the target chip-case image;

[0175] The determination module 703 may also be configured to determine a second prediction loss based on a difference between the second annotation result and the second segmentation result corresponding to the target chip-shell image;

[0176] The adjustment module 704 may also be configured to adjust parameters of the instance segmentation model with the goal of minimizing the second prediction loss.

[0177] In one embodiment, the instance segmentation model is pre-trained;

[0178] The adjustment module 704 may be further configured to:

[0179] Adjust some parameters of the above instance segmentation model.

[0180] In one embodiment, the target module includes an encoder, which includes a first submodule and a second submodule. The first submodule is used to extract the feature vector of the target area image, and the second submodule is used to enhance the feature vector using an attention mechanism and output the enhanced feature vector.

[0181] In one embodiment, the defect category label indicates the true category to which the pixel belongs, and the detection result includes a target probability that each pixel in the target area image belongs to the corresponding true category;

[0182] The determination module 703 may also be configured to:

[0183] Determining focus loss based on each of the above target probabilities in the above detection results;

[0184] Based on the first prediction loss and the focus loss, the total loss is determined.

[0185] In one embodiment, the target module has been pre-trained;

[0186] The adjustment module 704 may be further configured to:

[0187] Adjust some parameters of the above target modules.

[0188] It should be noted that other aspects and implementation details of the model training device provided in the embodiment of the present application are the same or similar to the model training method described above and will not be repeated here.

[0189] Figure 8 This is a schematic diagram of the structure of the tube shell defect detection device provided in an embodiment of the present application. The tube shell defect detection device can be applied to any platform, server, or device cluster with computing and processing capabilities. The tube shell defect detection device includes:

[0190] The acquisition module 801 is configured to acquire an image of a region to be inspected and a material category corresponding to the image of the region; wherein the image of the region to be inspected is extracted from the chip package image;

[0191] The detection module 802 is configured to input the above-mentioned regional image into the target module of the semantic segmentation model in the defect detection system, and the above-mentioned target module performs defect detection on each pixel in the above-mentioned regional image and outputs the detection result; wherein the above-mentioned semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the above-mentioned target module is a semantic segmentation module corresponding to the same material category as the above-mentioned regional image.

[0192] In one embodiment, the defect detection system further includes an instance segmentation model;

[0193] The acquisition module 801 may be further configured to:

[0194] Inputting the chip-shell image into the instance segmentation model for segmentation processing to obtain a segmentation result output by the instance segmentation model, wherein the segmentation result indicates the location information of the area to be detected and the material category to which it belongs;

[0195] The region image is extracted from the chip package image based on the position information, and it is determined that the region image corresponds to the material category.

[0196] In one embodiment, the detection result is a first segmentation mask having the same size as the regional image;

[0197] The above-mentioned tube and shell defect detection device may further include:

[0198] The conversion module (not shown) is configured to convert the first segmentation mask into a second segmentation mask of the same size as the chip shell image based on the position information of the area to be detected in the chip shell image.

[0199] It should be noted that other aspects and implementation details of the tube and shell defect detection device provided in the embodiment of the present application are the same as or similar to the tube and shell defect detection method described above, and will not be repeated here.

[0200] The embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following is achieved: Figure 1 or Figure 2 The model training method described in Figure 6 A method for detecting tube and shell defects is described.

[0201] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the following is achieved: Figure 1 or Figure 2 The model training method described in Figure 6 A method for detecting tube and shell defects is described.

[0202] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the following Figure 1 or Figure 2 The model training method described in Figure 6 A method for detecting tube and shell defects is described.

[0203] The above description is only a partial implementation of the embodiments of the present application and does not constitute any form of limitation to the application. The protection scope of the embodiments of the present application is not limited thereto. Any simple modifications, equivalent changes and modifications that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed in the embodiments of the present application should be covered within the protection scope of the embodiments of the present application.

Claims

1. A method for detecting tube and shell defects, characterized in that: include: Acquire an image of a region to be inspected and a material category corresponding to the image of the region; wherein the image of the region to be inspected is extracted from the chip shell image; The regional image is input into the target module of the semantic segmentation model in the defect detection system, and the target module performs defect detection on each pixel in the regional image and outputs the detection result; wherein, the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the regional image.

2. The method according to claim 1, characterized in that The defect detection system also includes an instance segmentation model; Obtaining the image of the area to be detected and the material category corresponding to the image of the area, including: Inputting the chip-shell image into the instance segmentation model for segmentation processing to obtain a segmentation result output by the instance segmentation model, wherein the segmentation result indicates the location information of the area to be detected and the material category to which it belongs; The region image is extracted from the chip-package image based on the position information, and it is determined that the region image corresponds to the material category.

3. The method according to claim 1 or 2, characterized in that The detection result is a first segmentation mask having the same size as the regional image; The method further comprises: Based on the position information of the area to be inspected in the chip-shell image, the first segmentation mask is converted into a second segmentation mask having the same size as the chip-shell image.

4. A model training method, characterized in that: include: Acquire multiple regional images, and the material categories and first annotation results corresponding to each of the multiple regional images; wherein the multiple regional images include images of the areas to be inspected in chip packages of various models, and the first annotation results include a defect category label for each pixel in the corresponding regional images; Inputting a target region image from the multiple region images into a target module of a semantic segmentation model, wherein the target module performs defect detection on each pixel in the target region image and outputs a detection result; wherein the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image; determining a first prediction loss based on a difference between the first annotation result corresponding to the target area image and the detection result, and determining a total loss based on the first prediction loss; With the goal of minimizing the total loss, the parameters of the objective module are adjusted.

5. The method according to claim 4, characterized in that Acquire multiple area images, including: Acquire a plurality of chip-package images, wherein the plurality of chip-package images include images of the plurality of chip-package models; Inputting the plurality of chip-shell images into the trained instance segmentation model for segmentation processing respectively, and obtaining a first segmentation result output by the instance segmentation model; extracting a first region image of a to-be-detected region from the plurality of chip-shell images based on the first segmentation result; The plurality of area images are determined based on the first area images.

6. The method according to claim 5, characterized in that Determining the plurality of regional images based on the first regional images includes: performing image enhancement processing on at least a portion of the first region images to generate second region images corresponding to at least a portion of the first region images; A set of the first region images and the second region images is determined as the plurality of region images.

7. The method according to claim 5, characterized in that The plurality of chip-shell images each correspond to a second annotation result, wherein the second annotation result includes position information and a material category label of a region to be detected in the corresponding chip-shell image; The training process of the instance segmentation model includes: Inputting a target chip-shell image from the plurality of chip-shell images into the instance segmentation model for segmentation processing, obtaining a second segmentation result output by the instance segmentation model, wherein the second segmentation result indicates position information and a material category of a region to be detected in the target chip-shell image; determining a second prediction loss based on a difference between the second annotation result and the second segmentation result corresponding to the target chip-shell image; The parameters of the instance segmentation model are adjusted with the goal of minimizing the second prediction loss.

8. The method according to claim 7, characterized in that The instance segmentation model has been pre-trained; Adjusting parameters of the instance segmentation model includes: Adjust some parameters of the instance segmentation model.

9. The method according to claim 4, characterized in that The target module includes an encoder, and the encoder includes a first submodule and a second submodule. The first submodule is used to extract the feature vector of the target area image, and the second submodule is used to enhance the feature vector using an attention mechanism and output the enhanced feature vector.

10. The method according to claim 4, characterized in that The defect category label indicates the true category to which the pixel belongs, and the detection result includes a target probability that each pixel in the target area image belongs to the corresponding true category; The method further comprises: determining focus loss based on each of the target probabilities in the detection results; Determining a total loss based on the first predicted loss includes: The total loss is determined based on the first prediction loss and the focal loss.

11. The method according to any one of claims 4 to 10, characterized in that The target module has been pre-trained; Adjusting the parameters of the target module includes: Adjust some parameters of the target module.

12. A tube and shell defect detection device, characterized in that: include: an acquisition module configured to acquire an image of a region to be inspected and a material category corresponding to the image of the region; wherein the image of the region is an image of the region to be inspected extracted from the chip package image; A detection module is configured to input the area image into a target module of a semantic segmentation model in a defect detection system, and the target module performs defect detection on each pixel in the area image and outputs a detection result; wherein the semantic segmentation model includes multiple semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the area image.

13. A model training device, characterized in that: include: an acquisition module configured to acquire a plurality of regional images, and a material category and a first annotation result corresponding to each of the plurality of regional images; wherein the plurality of regional images include images of regions to be inspected in chip packages of various models, and the first annotation result includes a defect category label for each pixel in the corresponding regional image; a detection module configured to input a target region image from the plurality of region images into a target module of a semantic segmentation model, wherein the target module performs defect detection on each pixel in the target region image and outputs a detection result; wherein the semantic segmentation model includes a plurality of semantic segmentation modules corresponding to different material categories, and the target module is a semantic segmentation module corresponding to the same material category as the target region image; a determination module configured to determine a first prediction loss based on a difference between the first annotation result corresponding to the target area image and the detection result, and determine a total loss based on the first prediction loss; The adjustment module is configured to adjust the parameters of the target module with the goal of minimizing the total loss.

14. A computer device, characterized in that: The computer device includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the tube and shell defect detection method according to any one of claims 1 to 3 is implemented, or the model training method according to any one of claims 4 to 11 is implemented.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for detecting tube and shell defects according to any one of claims 1 to 3 is implemented, or the model training method according to any one of claims 4 to 11 is implemented.

Images