Defect detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a defect detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-detected image obtained by performing image acquisition on a to-be-detected object and at least one group of preset defect feature vectors in a defect feature base library; inputting the to-be-detected image into an image feature extraction module to obtain a to-be-detected image feature vector of the to-be-detected image; determining a defect detection result of the to-be-detected object according to the similarity between the to-be-detected image feature vector and each group of preset defect feature vectors in the at least one group of preset defect feature vectors; wherein the at least one group of preset defect feature vectors are text feature vectors obtained by inputting at least one defect sample image into a feature extraction network, and the defect sample image comprises a sample object with a defect to be detected. According to the scheme, the defect types contained in the adopted defect feature base library are rich, and effective defect prompts can be provided for the to-be-detected image.

Description

Technical Field

[0001] The present invention relates to the field of industrial detection, and in particular to a defect detection method, device, electronic equipment, storage medium and computer program product. Background Art

[0002] After the workpiece is produced, it is usually necessary to perform defect detection before it leaves the factory. At present, most industrial defect classification algorithms based on feature libraries can only classify the semantic level of the entire image to be tested. However, in actual industrial applications, the same type of defects (such as dust) occur in different areas of the workpiece, and the defect levels vary greatly. For example, in the field of wafer defect detection, dust is considered a defect only when it occurs in the circuit area. Therefore, most industrial defect classification algorithms will divide the image to be tested into multiple areas manually or through models, and judge the position of the defect in the workpiece to be tested by the positional relationship between the defect area and other areas. This method requires manual intervention and requires additional training of the background segmentation model. Summary of the invention

[0003] The present invention is proposed in view of the above problems. The present invention provides a defect detection method, device, electronic device, storage medium and computer program product, which can directly determine at one time whether there are defects to be detected in the image to be tested whose defect type and defect area meet the preset detection conditions based on the defect feature base library, without first identifying the defect type and then determining the defect area by manual or model segmentation.

[0004] According to one aspect of the present invention, a defect detection method is provided. The method comprises: obtaining an image to be tested obtained by performing image acquisition on an object to be tested and at least one set of preset defect feature vectors in a defect feature base library; inputting the image to be tested into an image feature extraction module to obtain the image feature vector of the image to be tested; determining a defect detection result of the object to be tested according to the similarity between the image feature vector to be tested and each set of preset defect feature vectors in at least one set of preset defect feature vectors, the defect detection result including information on whether there is a defect to be detected in the object to be tested, the defect to be detected being a defect whose defect type and defect area both meet the preset conditions to be detected; wherein at least one set of preset defect feature vectors is at least one defect sample image input feature extraction module. Take the text feature vector obtained by the network, the feature extraction network includes an image feature extraction module, a mapping module and a text feature extraction module connected in sequence, the image feature extraction module is used to extract the image feature vector of the input image of the input image feature extraction module, the mapping module is used to map the image feature vector into a text prediction vector, the text feature extraction module is used to extract the text feature vector of the defect description text, the defect description text is the text obtained by inserting the text prediction vector into a preset text template corresponding to the input image, the text information contained in the preset text template is used to reflect the positioning of the defect area of ​​the object in the input image, and the defect sample image contains a sample object with defects to be detected.

[0005] Optionally, at least one group of preset defect feature vectors corresponds one-to-one to at least one group of defect label information, the defect label information includes regional information of the defect area and / or type information of the defect type, and the regional information is used to indicate a preset positioning position of the defect area; based on the similarity between the feature vector of the image to be tested and each group of preset defect feature vectors in at least one group of preset defect feature vectors, the defect detection result of the object to be tested is determined, including: the defect label information corresponding to the preset defect feature vector whose similarity with the feature vector of the image to be tested meets the preset requirements is determined as the defect detection result.

[0006] Optionally, the process of creating a defect feature database is as follows: obtaining at least one defect sample image and one or more sets of preset text templates corresponding to each defect sample image and one or more sets of defect label information corresponding to each defect sample image, the one or more sets of preset text templates corresponding to the same defect sample image and the one or more sets of defect label information correspond one-to-one, and at least some of the preset text templates and the corresponding defect label information express at least partially the same semantic information; inputting each defect sample image in at least one defect sample image into a feature extraction network to obtain one or more sets of preset defect feature vectors corresponding to each defect sample image, the one or more sets of preset defect feature vectors corresponding to each defect sample image correspond one-to-one to the one or more sets of preset text templates corresponding to the defect sample image; and associating and saving the defect label information and the preset defect feature vectors in a one-to-one correspondence.

[0007] Optionally, when the preset text template contains text information reflecting a preset positioning position of the defect area, the defect label information contains area information or contains area information and type information; when the preset text template contains text information reflecting that the positioning of the defect area is unknown, the defect label information contains type information.

[0008] Optionally, the training process of the feature extraction network is as follows: inputting a first training image into the feature extraction network to obtain a first training image feature vector output by an image feature extraction module and a first training text feature vector output by a text feature extraction module; inputting a second training image into the feature extraction network to obtain a second training image feature vector output by an image feature extraction module and a second training text feature vector output by a text feature extraction module; optimizing a mapping module in the feature extraction network according to a first contrast loss between the first training image feature vector and the first training text feature vector, and a second contrast loss between the second training image feature vector and the second training text feature vector until preset requirements are met; wherein the first training image is an image block containing a training object in the second training image, and when the first training image is input, the corresponding preset text template includes a first preset text template for reflecting that the positioning of a defective area of ​​the training object is unknown, and when the second training image is input, the corresponding preset text template includes a second preset text template for reflecting a preset positioning position of a defective area of ​​the training object.

[0009] Optionally, when the second training image is input, the corresponding preset text template also includes a third preset text template for reflecting that the positioning of the defect area of ​​the training object is unknown, and the second training text feature vector includes a first sub-vector corresponding to the second preset text template, and a second sub-vector corresponding to the third preset text template; the second contrast loss includes the contrast loss between the second training image feature vector and the first sub-vector, and the contrast loss between the second training image feature vector and the second sub-vector.

[0010] Optionally, when the defect sample image is input into the feature extraction network, the preset text template includes a first preset text template for reflecting the unknown positioning of the defect area in the defect sample image and a second preset text template for reflecting the preset positioning position of the defect area in the defect sample image; or the preset text template includes a second preset text template for reflecting the preset positioning position of the defect area in the defect sample image.

[0011] Optionally, when the defect sample image is input into the feature extraction network, when the preset to-be-detected condition corresponding to the defect to be detected in the defect sample image is the first to-be-detected condition, the preset text template includes a first preset text template and a second preset text template; when the preset to-be-detected condition corresponding to the defect to be detected in the defect sample image is the second to-be-detected condition, the preset text template includes a second preset text template and a second preset text template; the first to-be-detected condition refers to that the defect type is the first preset type and the defect area falls within any area on the sample object in the defect sample image, and the second to-be-detected condition refers to that the defect type is the second preset type and the defect area falls within the area at the preset positioning position on the sample object in the defect sample image.

[0012] According to another aspect of the present invention, a defect detection device is provided, comprising: an acquisition module, used to acquire an image to be tested obtained by performing image acquisition on the object to be tested and at least one set of preset defect feature vectors in a defect feature base library; an input module, used to input the image to be tested into an image feature extraction module to obtain the image feature vector of the image to be tested; a determination module, used to determine a defect detection result of the object to be tested according to the similarity between the feature vector of the image to be tested and each set of preset defect feature vectors in at least one set of preset defect feature vectors, the defect detection result including information on whether there is a defect to be detected on the object to be tested, the defect to be detected being a defect whose defect type and defect area both meet the preset conditions to be detected; wherein at least one set of preset defect vectors The defect feature vector is a text feature vector obtained by inputting at least one defect sample image into a feature extraction network. The feature extraction network includes an image feature extraction module, a mapping module and a text feature extraction module connected in sequence. The image feature extraction module is used to extract the image feature vector of the input image of the image feature extraction module. The mapping module is used to map the image feature vector into a text prediction vector. The text feature extraction module is used to extract the text feature vector of the defect description text. The defect description text is the text obtained by inserting the text prediction vector into a preset text template corresponding to the input image. The text information contained in the preset text template is used to reflect the positioning of the defect area of ​​the object in the input image. The defect sample image contains a sample object with defects to be detected.

[0013] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned defect detection method when the processor is running.

[0014] According to yet another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned defect detection method when running.

[0015] According to yet another aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions are used to execute the defect detection method as described above when running.

[0016] The defect feature base library in the above technical solution is created by inputting a defect sample image into a feature extraction network. In each module of the feature extraction network, the corresponding image feature vector, text prediction vector and text feature vector of the defect description text can be obtained in sequence based on the defect sample image. The text feature vector can be used as a preset defect feature vector in the defect feature base library. The defect description text is a text obtained by inserting the text prediction vector into a preset text template corresponding to the input image, wherein the text information contained in the preset text template can be used to reflect the positioning of the defect area of ​​the object in the input image, and the defect sample image contains defects to be detected whose defect types and defect areas meet the preset conditions for detection. Through this defect feature base library, the image feature vector of the image to be tested can be compared with the preset defect feature vector in the defect feature base library. The preset defect feature vector is a text feature vector of a defect sample image containing the defect to be detected and the preset defect feature vector is generated at least partially based on a preset text template that can reflect the positioning of the defect area. In this way, it can be determined whether there is a defect to be detected in the image to be tested based on the comparison result with the preset defect feature vector. In this way, it is possible to directly determine at one time based on the defect feature database whether there are defects to be detected in the image to be tested whose defect type and defect area both meet the preset detection conditions, without having to first identify the defect type and then determine the defect area through manual or model segmentation. Therefore, this solution can effectively reduce manual operations and calculations.

[0017] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 A schematic flow chart of a defect detection method according to an embodiment of the present invention is shown;

[0020] Figure 2 A schematic flow chart of creating a defect feature base library according to an embodiment of the present invention is shown;

[0021] Figure 3 A schematic flow chart of training a feature extraction network according to an embodiment of the present invention is shown;

[0022] Figure 4 A schematic block diagram of a defect detection device according to an embodiment of the present invention is shown;

[0023] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0025] As mentioned above, in the related art, most industrial defect classification algorithms will divide the image to be tested into multiple areas manually or by models, and judge the position of the defect in the workpiece to be tested by the positional relationship between the defect area and other areas. This method requires manual intervention and requires additional training of the background segmentation model. In order to at least solve the above technical problems, the embodiments of the present invention provide a defect detection method, device, electronic device, storage medium and computer program product. This solution can directly determine at one time whether there is a defect to be detected in the image to be tested whose defect type and defect area meet the preset detection conditions based on the defect feature base library, without having to first identify the defect type and then determine the defect area by manual or model segmentation.

[0026] See also Figure 1As shown, it is a schematic flow chart of a defect detection method according to an embodiment of the present invention. According to one aspect of the present invention, a defect detection method is provided, the method comprising step S110, step S120 and step S130.

[0027] In step S110, an image to be tested obtained by performing image acquisition on the object to be tested and at least one set of preset defect feature vectors in a defect feature base library are obtained.

[0028] Exemplarily, the defect feature base library may include one or more groups of preset defect feature vectors, and each group of preset defect feature vectors may correspond to a defect. The defect may be a known defect, and the known defect may have a corresponding professional text description, such as scratches, cracks, etc. The defect corresponding to the preset defect feature vector may also be an unknown defect, and the unknown defect does not have a corresponding professional text description. Each group of preset defect feature vectors may correspond to a sample object, and the sample object may be any object that needs to be defect-detected in the industrial field, such as workpieces such as wafers, chips, and packaging materials. Each group of preset defect feature vectors contained in the defect feature base library may correspond to sample objects of the same category, for example, each group of preset defect feature vectors each corresponds to a certain defect of the wafer. Each group of preset defect feature vectors contained in the defect feature base library may also correspond to sample objects of different categories, for example, each preset defect feature vector of some preset defect feature vectors corresponds to a certain defect of the wafer, and each preset defect feature vector of another part of the preset defect feature vectors corresponds to a certain defect of the chip. Similarly, the object to be tested can be any object that requires defect detection in the industrial field, and its category can be consistent with the category of sample objects corresponding to one or more groups of preset defect feature vectors in the defect feature base library. For example, when the sample objects corresponding to each group of preset defect feature vectors in the defect feature base library are all wafers, the category of the object to be tested can be wafers.

[0029] In step S120, the image to be tested is input into an image feature extraction module to obtain an image feature vector of the image to be tested.

[0030] Exemplarily, the image feature extraction module may be a pre-trained image encoder, such as an open source image-text alignment model or an image-text description model, such as a Contrastive Language-Image Pre-training (CLIP) model, an open source image-text multimodal model (OpenCLIP), etc. After the image to be tested is input into the image feature extraction module, the image feature extraction module may extract and output a test image feature vector of the image to be tested.

[0031] In step S130, a defect detection result of the object to be tested is determined according to the similarity between the feature vector of the image to be tested and each group of preset defect feature vectors in at least one group of preset defect feature vectors, the defect detection result includes information on whether there are defects to be detected on the object to be tested, and the defects to be detected are defects whose defect types and defect areas meet preset conditions to be detected; wherein, at least one group of preset defect feature vectors is a text feature vector obtained by inputting at least one defect sample image into a feature extraction network, the feature extraction network includes an image feature extraction module, a mapping module and a text feature extraction module connected in sequence, the image feature extraction module is used to extract the image feature vector of the input image of the input image feature extraction module, the mapping module is used to map the image feature vector into a text prediction vector, the text feature extraction module is used to extract the text feature vector of the defect description text, the defect description text is the text obtained by inserting the text prediction vector into a preset text template corresponding to the input image, the text information contained in the preset text template is used to reflect the positioning of the defect area of ​​the object in the input image, and the defect sample image contains a sample object with the defect to be detected.

[0032] Exemplarily, the defect type refers to the type to which the defect belongs. All defect types that may exist in the current object to be tested may be known. For example, when the object to be tested is a wafer, the defect types that may exist may include scratches, dust, cracks, etc. The defect area refers to the area to which the defect belongs. The defect area may be an area surrounded by the outline of the defect, or it may be an area that includes the area surrounded by the outline of the defect and is larger than the area surrounded by the outline of the defect. All defect areas where defects that may exist in the current object to be tested may appear may be known. For example, when the current object to be tested is a wafer, the defect areas where defects that may exist may include cutting road areas, line areas, edge areas, etc. The defect to be detected is a defect whose defect type and defect area both meet the preset conditions to be detected. For example, the preset conditions to be detected may include that the defect type is dust and the defect area is a line area. In this case, the dust on the line area can be a defect to be detected. Accordingly, when the defect type is dust and the defect area is other areas except the circuit area (e.g., the cutting road area), it can be considered that the defect area does not meet the preset detection conditions, and the dust located in other areas except the circuit area is not a defect to be detected. For another example, the preset detection conditions can include that the defect type is scratches and the defect area is any area. In this case, scratches in any area can be defects to be detected.

[0033] Exemplarily, a defect sample image collected for a sample object with a defect to be detected can be obtained, and feature extraction is performed based on the defect sample image to create a defect feature base library. Exemplarily, each group of preset defect feature vectors in the defect feature base library can be obtained by inputting the defect sample image into a feature extraction network. The feature extraction network may include an image feature extraction module, a mapping module connected to the image feature extraction module, and a text feature extraction module connected to the mapping module. The image feature extraction module included in the feature extraction network is the image feature extraction module input by the image to be tested when executing step S120, and the image feature extraction module can extract the image feature vector of each input image including the image to be tested and the defect sample image. Exemplarily, the mapping module may include one or more linear layers, and the image feature vector output by the image feature extraction module may be input into the mapping module, and the linear layer of the mapping module may map the image feature vector into a text prediction vector. After obtaining the text prediction vector, the defect description text containing the text prediction vector may be input into the text feature extraction module to obtain the text feature vector of the defect description text. The text feature extraction module may perform text feature extraction on each group of texts input including the defect description text to obtain a text feature vector. The text feature extraction module can be a pre-trained text encoder, such as the text encoder in CLIP, or a text encoder of a large language model such as GPT. Specifically, for each defect sample image, it can correspond to one or more groups of defect description texts. The defect description text may include a text prediction vector and a preset text template corresponding to the defect sample image. The preset text template may be, for example, "This is a * photo", or, for example, "This is a * photo on the line area", etc. After obtaining the text prediction vector, the "*" in the preset text template can be replaced with the text prediction vector. For example, the obtained text prediction vector is f v For the preset text template “This is a photo of * on the line area”, the defect description text obtained by inserting the text prediction vector into the preset text template is “This is a photo of * on the line area”. vPhoto". Optionally, the symbol "*" in the above-mentioned preset text template may not actually exist. After obtaining the text prediction vector, the text prediction vector may be directly inserted into the position indicated by "*" to obtain the defect description text. The text information contained in the preset text template may reflect the positioning of the defect area of ​​the object in the input image. The positioning situation may include two situations: unknown positioning and known positioning. In the case of known positioning, the preset text template may include area information for indicating a preset positioning position of the defect area. The preset positioning position is a clear position of the defect area, for example, the defect area is a "cutting road area", "line area", etc., wherein the text information such as "cutting road area" and "line area" is the area information for indicating the preset positioning position of the defect area. In In some embodiments, if a defect located in any area on an object can be regarded as a defect to be detected of the object, the preset text template corresponding to the defect may include "This is a *photo", and may also include a text template with a description of the preset positioning position of the defect area, such as "This is a *photo on the line area". If a defect located in a specific area on an object can be regarded as a defect of the object, the preset text template corresponding to the defect may include a text template with a description of the preset positioning position of the defect area. In other embodiments, if the input defect sample image corresponds to multiple sample objects of different categories, the sample objects can also be described in the preset text template, for example, "This is a *wafer photo", and for example, "This is a *chip photo".

[0034] Exemplarily, after obtaining the feature vector of the image to be tested, the similarity between the feature vector of the image to be tested and each group of preset defect feature vectors in the defect feature base library can be calculated, and the defect detection result of the object to be tested can be determined based on the calculated similarity. The defect detection result includes information about whether the object to be tested has the defect to be detected (this information can be referred to as defect existence information). Exemplarily, in the case where the defect to be detected does not exist, the defect existence information can be the first information used to indicate that the defect to be detected does not exist, such as text information or voice information such as "no defect exists". In the case where the defect to be detected exists, the defect existence information can be the second information used to indicate the existence of the defect to be detected, such as text information or voice information such as "defect exists", or the following type information and / or area information. Exemplarily, in the case where the defect to be detected exists, the defect detection result can also include type information of the defect type and / or area information of the defect area. The second information can be information independent of the type information and the area information, and can also be represented by the type information and / or the area information. It can be understood that being able to determine the defect type and / or defect area means that the object to be tested itself has defects to be detected. At this time, only the type information of the defect type and / or the area information of the defect area can be obtained, and the type information of the defect type and / or the area information of the defect area are used to represent the second information. In some embodiments, a similarity threshold can be preset. If the similarity between the feature vector of the image to be tested and each group of preset defect vectors is less than the similarity threshold, the determined defect detection result may include that the object to be tested does not have defects. Conversely, the determined defect detection result may include that the object to be tested has defects. In other embodiments, if the similarity between the feature vector of the image to be tested and one or more groups of preset defect vectors is greater than or equal to the preset similarity threshold, the determined defect detection result may include that the object to be tested has defects, and may also include that the defect type and / or defect area of ​​the object to be tested is the defect type and / or defect area indicated by each group of preset defect vectors of one or more groups of preset defect vectors whose similarity is greater than or equal to the similarity threshold. In other embodiments, if the similarity between the feature vector of the image to be tested and one or more groups of preset defect vectors is greater than or equal to a preset similarity threshold, the determined defect detection result may include the presence of defects in the object to be tested, and may also include the defect type and / or defect area of ​​the object to be tested as the defect type and / or defect area indicated by each group of preset defect vectors with the preset defect vectors having the greatest similarity.

[0035] The defect feature base library in the above technical solution is created by inputting a defect sample image into a feature extraction network. In each module of the feature extraction network, the corresponding image feature vector, text prediction vector and text feature vector of the defect description text can be obtained in sequence based on the defect sample image. The text feature vector can be used as a preset defect feature vector in the defect feature base library. The defect description text is a text obtained by inserting the text prediction vector into a preset text template corresponding to the input image, wherein the text information contained in the preset text template can be used to reflect the positioning of the defect area of ​​the object in the input image, and the defect sample image contains defects to be detected whose defect types and defect areas meet the preset conditions for detection. Through this defect feature base library, the image feature vector of the image to be tested can be compared with the preset defect feature vector in the defect feature base library. The preset defect feature vector is a text feature vector of a defect sample image containing the defect to be detected and the preset defect feature vector is generated at least partially based on a preset text template that can reflect the positioning of the defect area. In this way, it can be determined whether there is a defect to be detected in the image to be tested based on the comparison result with the preset defect feature vector. In this way, it is possible to directly determine at one time based on the defect feature database whether there are defects to be detected in the image to be tested whose defect type and defect area both meet the preset detection conditions, without having to first identify the defect type and then determine the defect area through manual or model segmentation. Therefore, this solution can effectively reduce manual operations and calculations.

[0036] Optionally, at least one group of preset defect feature vectors corresponds one-to-one to at least one group of defect label information, the defect label information includes regional information of the defect area and / or type information of the defect type, and the regional information is used to indicate a preset positioning position of the defect area; based on the similarity between the feature vector of the image to be tested and each group of preset defect feature vectors in at least one group of preset defect feature vectors, the defect detection result of the object to be tested is determined, including: the defect label information corresponding to the preset defect feature vector whose similarity with the feature vector of the image to be tested meets the preset requirements is determined as the defect detection result.

[0037] Exemplarily, each set of preset defect feature vectors contained in the defect feature base library may correspond to a set of defect label information. The defect label information may be used to indicate a defect area, or to indicate a defect type, or to indicate a defect area and a defect type. In a specific embodiment, a defect sample image input to the feature extraction network is an image obtained by image acquisition of a cut road with scratches in a wafer. In this embodiment, the object to be tested is a wafer, the defect area is a cut road, and the defect type is a scratch. The preset text template corresponding to the defect sample image may include "This is a *photo" and "This is a *photo on the cut road". After the text feature vectors obtained based on the defect sample image are inserted into two preset text templates respectively, two sets of preset defect feature vectors can be obtained through the text feature extraction module. The defect label information corresponding to the preset defect feature vector generated based on "This is a *photo" may be "scratches", and the defect label information corresponding to the preset defect feature vector generated based on "This is a *photo on the cut road" may be "scratches on the cut road". In another specific embodiment, a defect sample image input to the feature extraction network is an image obtained by image acquisition of a circuit area with dust in a wafer. In this embodiment, the object to be tested is a wafer, the defect area is a circuit area, and the defect type is dust. The preset text template corresponding to the defect sample image may include "This is a *photo on the circuit area". After the text feature vector obtained based on the defect sample image is inserted into the preset text template, a set of preset defect feature vectors can be obtained through the text feature extraction module, and the corresponding defect label information can be "dust on the circuit area". In another specific embodiment, a defect sample image input into the feature extraction network is an image obtained by image acquisition of a cutting path with a certain unknown defect in the wafer. In this embodiment, the object to be tested is a wafer, the defect area is a cutting path, and the defect type is an unknown defect. The preset text template corresponding to the defect sample image may include "This is a *photo on the cutting path". After the text feature vector obtained based on the defect sample image is input into the preset text template, the corresponding preset defect feature vector can be obtained through the text feature extraction module, and the defect label information corresponding to the preset defect feature vector can be "cutting path", or, it can be "unknown defect on cutting path". If the sample objects contained in the input multiple defect sample images each correspond to a different type of unknown defect, the unknown defects can also be classified. For example, the unknown defects can be divided into: first category unknown defects, second category unknown defects, third category unknown defects, etc. Then the defect type indicated by the defect label information of the preset defect feature vector can be a first category unknown defect, a second category unknown defect, a third category unknown defect, etc.Exemplarily, the preset requirement may be, for example, that the similarity between the feature vector of the image to be tested is the maximum and the similarity is greater than a preset similarity threshold, or may be, for example, that the similarity between the feature vector of the image to be tested is greater than or equal to a preset similarity threshold. After obtaining the similarity between the feature vector of the image to be tested and each group of defect feature vectors, the defect label information corresponding to the preset defect feature vector whose similarity meets the preset requirement may be used as the defect detection result. The type information of the defect type contained in the defect detection result may be one or more groups, and the area information of the defect area contained may be one group. For example, taking the object to be tested as a wafer as an example, the defect detection result of the object to be tested may include scratches, or may include scratches and dust, or may include scratches on the circuit area, or may include scratches and cracks on the circuit area, or may include defects on the circuit area, or may include unknown defects / first type unknown defects on the circuit area.

[0038] The above technical solution determines the defect detection result based on the defect label information corresponding to the preset defect feature vector, which can more accurately and intuitively indicate the defect area and defect type of the object to be tested, so that the defect detection result has a strong reference. On the other hand, by obtaining the defect label information corresponding to the preset defect feature vector whose similarity with the feature vector of the image to be tested meets the preset requirements and is determined as the defect detection result, the preset defect feature vectors corresponding to the defects with a low degree of proximity to the defects of the object to be tested in the defect feature base library can be screened out, and the preset defect feature vector corresponding to the defect that is closest to the defect of the object to be tested and the degree of proximity meets the requirements can be determined, so that the obtained defect detection result is more accurate and more in line with the actual needs of defect detection.

[0039] Optionally, the process of creating a defect feature database is as follows: obtaining at least one defect sample image and one or more sets of preset text templates corresponding to each defect sample image and one or more sets of defect label information corresponding to each defect sample image, the one or more sets of preset text templates corresponding to the same defect sample image and the one or more sets of defect label information correspond one-to-one, and at least some of the preset text templates and the corresponding defect label information express at least partially the same semantic information; inputting each defect sample image in at least one defect sample image into a feature extraction network to obtain one or more sets of preset defect feature vectors corresponding to each defect sample image, the one or more sets of preset defect feature vectors corresponding to each defect sample image correspond one-to-one to the one or more sets of preset text templates corresponding to the defect sample image; and associating and saving the defect label information and the preset defect feature vectors in a one-to-one correspondence.

[0040] Exemplarily, for each defect sample image, the defect sample image may have a corresponding one or more sets of preset text templates. In some embodiments, the preset text template may be defined by the user based on the defect sample image. In other embodiments, it may be generated based on the annotation content of the defect sample image. The defect sample image may have a corresponding image annotation, and the annotation content of the image is set according to a preset annotation rule. The annotation rule may, for example, be that the annotation content at least includes the corresponding defect type and defect area, and the annotation content may also optionally include the object type of the sample object. For example, the image annotation of a defect sample image may be "dust falling on the circuit area", or it may be "dust falling on the circuit area of ​​the wafer". In this case, the defect label information corresponding to each preset text template may include the defect type, defect area, or defect type and defect area contained in the image annotation of the corresponding defect sample image. For any defect sample image, each set of preset text templates corresponding to it may each correspond to a set of defect label information. In some embodiments, the defect label information corresponding to the preset text template for reflecting the preset positioning position of the defect area of ​​the defect area of ​​the sample object may include the defect area and the defect type, and the defect label information corresponding to the preset text template for reflecting the unknown positioning of the defect area in the defect sample image may only include the defect type. For the same defect sample image, at least part of the preset text template corresponding to the defect sample image and the corresponding defect label information may express at least part of the same semantic information. For example, for any defect sample image, the defect sample image may at least include a preset text template for reflecting the preset positioning position of the defect area in the defect sample image, and the defect area expressed by the preset text template is the same as the defect area expressed by the corresponding defect label information. For another example, for any defect sample image, the preset text template included in the defect sample image contains text information describing the object type of the sample object (such as a wafer, a chip, etc.), then each group of defect label information corresponding to the defect sample image may include information describing the object type of the sample object. In this case, the object type of the sample object expressed by the preset text template corresponding to the defect sample image is the same as the object type of the sample object expressed by the defect label information. After the defect sample image is input into the feature extraction network to obtain the corresponding preset defect feature vector, the defect label information can be associated and saved with the preset defect feature vector in a one-to-one correspondence.

[0041] See also Figure 2 As shown in FIG. 1 , it is a schematic flow chart of creating a defect feature base library according to an embodiment of the present invention. Figure 2 In the illustrated embodiment, the feature extraction network includes a frozen pre-trained image encoder, a mapping module f connected in sequence. MAnd a frozen pre-trained text encoder. The input defect sample image is an image obtained by capturing an unknown defect on the stripe line area of ​​the wafer. After the defect sample image is input into the frozen pre-trained image encoder, the corresponding image feature vector v1 can be obtained. The image feature vector can be input into the mapping f M The corresponding text prediction vector f can be obtained from M (v1). The preset text template corresponding to the defect sample image may be “This is a photo of a * on a stripe line area”, and the corresponding defect label information may be, for example, “unknown defect on a stripe line area” or “stripe line area”, or “unknown defect”. M (v1) Inserting this preset text template can get the defect description text: This is a picture of the defect description text on the stripe line area. M (v1) photos. The defect description text can be input into a frozen pre-trained text encoder. The pre-trained text encoder can be a large language model with a tokenized embedding submodule. The tokenized embedding submodule can split the defect description text into multiple tokens. The multiple tokens can include: this is, a piece of, in, stripes, line area, on, f M (v1), photos. The word embedding submodule can map each word to a corresponding vector representation, and the resulting text feature vector can be a continuous vector representation of each word. Figure 2 In the illustrated embodiment, the text feature vector output by the frozen pre-trained text encoder can be recorded as p, and the text feature vector p can be used as a set of preset defect feature vectors to be added to the defect feature base library (shown as "defect prompt feature base library" in the figure). The text feature vector p can be associated with the defect label information corresponding to the preset text template of the defect sample image and stored in the defect feature base library.

[0042] In the above technical scheme, each defect sample image can correspond to one or more sets of preset text templates, so that each defect sample image can obtain one or more sets of preset defect feature vectors, and each set of preset defect feature vectors can correspond to a set of defect label information. This method is conducive to enriching the preset defect feature vectors and defect label information in the defect feature base library. In particular, the same defect sample image can correspond to multiple sets of preset defect feature vectors, which helps to improve the defect detection rate of the object to be tested; on the other hand, each set of preset defect feature vectors corresponds to a set of defect label information, which is conducive to ensuring that the obtained defect detection results can specifically indicate the defects of the object to be tested.

[0043] Optionally, when the preset text template contains text information reflecting a preset positioning position of the defect area, the defect label information contains area information or contains area information and type information; when the preset text template contains text information reflecting that the positioning of the defect area is unknown, the defect label information contains type information.

[0044] Exemplarily, the preset text template may include text information for indicating a preset positioning position of a defective area. The preset positioning position is a clear position of a defective area. For example, a preset text template includes the text information "cutting road area", wherein the cutting road area is the preset positioning position of the defective area represented by the preset text template. In some embodiments, the preset defect feature vector corresponding to the preset text template containing text information for reflecting the preset positioning position of the defective area, and the corresponding defect label information may include region information. For example, the preset text template may be "This is a *photo taken on the cutting road area", and the cutting road area is the preset positioning position of the defective area represented by the preset text template. The region information included in the defect label information of the preset defect feature vector corresponding to the preset text template may be "cutting road area". In other embodiments, the preset defect feature vector corresponding to the preset text template containing text information for reflecting the preset positioning position of the defective area, and the corresponding defect label information may include region information and type information, and the type information may be used to reflect the defect type of the sample object contained in the corresponding defect sample image, and the type information may be, for example, "dust", "scratches", "cracks", etc. In other embodiments, the preset text template may include text information reflecting that the location of the defect area is unknown. In this case, the defect label information may include type information. For example, the preset text template may be "This is a *photo", which does not include text information reflecting the preset location of the defect area. In other words, it includes text information reflecting that the location of the defect area is unknown. The defect label information corresponding to the preset defect feature vector corresponding to the preset text template may include type information reflecting the defect type of the sample object contained in the defect sample image. The type information may be, for example, "dust", "scratches", "cracks", etc.

[0045] In the above technical scheme, for the preset text template containing text information reflecting the unknown location of the defect area, and the preset text template containing text information reflecting the preset location position of the defect area, the corresponding preset defect feature vectors can correspond to a set of defect label information, and the defect label information can provide effective and specific prompts for the defect type and / or defect area of ​​the object to be tested, so that the defect detection result has a strong reference value.

[0046] Optionally, the training process of the feature extraction network is as follows: inputting a first training image into the feature extraction network to obtain a first training image feature vector output by an image feature extraction module and a first training text feature vector output by a text feature extraction module; inputting a second training image into the feature extraction network to obtain a second training image feature vector output by an image feature extraction module and a second training text feature vector output by a text feature extraction module; optimizing a mapping module in the feature extraction network according to a first contrast loss between the first training image feature vector and the first training text feature vector, and a second contrast loss between the second training image feature vector and the second training text feature vector until preset requirements are met; wherein the first training image is an image block containing a training object in the second training image, and when the first training image is input, the corresponding preset text template includes a first preset text template for reflecting that the positioning of a defective area of ​​the training object is unknown, and when the second training image is input, the corresponding preset text template includes a second preset text template for reflecting a preset positioning position of a defective area of ​​the training object.

[0047] Exemplarily, when training the feature extraction network, each set of training images input to the feature extraction network may include a first training image and a second training image. For any set of training images, the set of training images may be two training images obtained by performing image acquisition for the same defect of the same training object. Among them, the first training image may only contain defects but not defect areas, and accordingly, the first preset text template corresponding thereto may be used to reflect that the positioning of the defect area of ​​the training object is unknown. The second training image may include defects and defect areas, and accordingly, the second preset text template corresponding thereto may reflect the preset positioning position of the defect area of ​​the training object. It can be understood that the first training image may be an image block containing the defect of the training object in the second training image. After the first training image is input to the feature extraction network, the image feature extraction module and the text feature extraction module in the feature extraction network may output the first training image feature vector and the first text feature vector, respectively. Similarly, after the second training image is input to the feature extraction network, the image feature extraction module and the text feature extraction module may output the second training image feature vector and the second text feature vector, respectively. The embodiment of the present invention can adjust the parameters of the feature extraction network according to the first contrast loss between the first training image feature vector and the first training text feature vector, and the second contrast loss between the second training image feature vector and the second training text feature vector, until the feature extraction network meets the preset requirements. In some embodiments, a loss threshold can be preset, and the preset requirements may include: the sum of the first contrast loss and the second contrast loss is less than or equal to the loss threshold. In other embodiments, the preset requirements may include: the first contrast loss and the second contrast loss are both less than or equal to the preset loss threshold.

[0048] The above technical solution can use images containing unknown defects that cannot be described in words as training images by training the mapping module in the feature extraction network, and can overcome the problem that the unknown defects cannot be converted into text feature vectors because they cannot be described in words; on the other hand, when training the feature extraction network, both a first training image including an image block with defects and a second training image including an image block with defects and a defective area are input, and the mapping module is trained in combination with multiple contrast losses, which is conducive to making the mapping module pay more attention to the feature information of the defects.

[0049] Optionally, when the second training image is input, the corresponding preset text template also includes a third preset text template for reflecting that the positioning of the defect area of ​​the training object is unknown, and the second training text feature vector includes a first sub-vector corresponding to the second preset text template, and a second sub-vector corresponding to the third preset text template; the second contrast loss includes the contrast loss between the second training image feature vector and the first sub-vector, and the contrast loss between the second training image feature vector and the second sub-vector.

[0050] Exemplarily, the preset text template corresponding to the second training image may further include a third preset text template, and the third preset text template may be used to reflect that the location of the defect area of ​​the training object is unknown. In some embodiments, the third preset text template may be the first preset text template. The second training text feature vector may include a first sub-vector corresponding to the second preset text template and a second sub-vector corresponding to the third preset text template. The second contrast loss may be obtained based on the contrast loss between the second training image feature vector and the first sub-vector and the contrast loss between the second training image feature vector and the second sub-vector. Please refer to Figure 3 As shown, it is a schematic flow chart of training a feature extraction network according to an embodiment of the present invention. Figure 3 In the illustrated embodiment, the training image is an image obtained by capturing an unknown defect on the blue cutting path of the wafer. The first training image is an image obtained by cropping the target (i.e., the unknown defect) in the second training image, and the second training image is the entire training image, which includes the unknown defect and the defect area. The first training image is input into the feature extraction network, and the frozen pre-trained image encoder is used to extract image features to obtain the corresponding image feature vector v1, which can be input into the mapping f M The corresponding text prediction vector f can be obtained from M (v1). The preset text template corresponding to the defect sample image can be "This is a *photo". The text prediction vector f M (v1) Insert the preset text template to get the defect description text: This is a f M(v1) photos. The defect description text can be input into a frozen pre-trained text encoder. The pre-trained text encoder can be a large language model with a tokenized embedding submodule. The tokenized embedding submodule can split the defect description text into multiple tokens. The multiple tokens can include: this is, a, f M (v1), photo. The word embedding submodule can map each word segment to a corresponding vector representation, and the obtained text feature vector can be a continuous vector representation of each word segment. The text feature vector output by the frozen pre-trained text encoder can be denoted as p1. The contrast loss between the text feature vector p1 and the image feature vector v1 is calculated to obtain the first contrast loss Loss1. The second training image is input into the feature extraction network, and the frozen pre-trained image encoder is used to extract image features to obtain the corresponding image feature vector v2. The image feature vector can be input into the mapping f M The corresponding text prediction vector f can be obtained from M (v2). The preset text templates corresponding to the defect sample image may include "This is a *photo" and "This is a *photo on a blue cutting path". The text prediction vector f M (v2) Insert the corresponding two preset text templates to get two sets of defect description texts: This is a f M (v2) Photo, this is a photo of f on the blue cutting road M (v2) photos. The defect description text can be input into a frozen pre-trained text encoder. The pre-trained text encoder can be a large language model with a tokenized embedding submodule. The tokenized embedding submodule can split the defect description text into multiple tokens. M (v2) photo", multiple participles can include: this is, a, f M (v2), photos. For the defect description text "This is a photo of a blue cut line. M (v2) photo", multiple participles can include: this is, a, in, blue, cutting road, on, f M(v2), photo. The word embedding submodule can map each word of each defect description text to a corresponding vector representation. The text feature vectors corresponding to the two defect description texts can be continuous vector representations of each word. The text feature vectors output by the frozen pre-trained text encoder can be denoted as p2 and p3, respectively. The contrast loss between the text feature vector p2 and the image feature vector v2 can be calculated to obtain the contrast loss Loss2. The contrast loss between the text feature vector p3 and the image feature vector v2 can be calculated to obtain the contrast loss Loss3. The second contrast loss can be the sum of Loss2 and Loss3. In the optimization mapping module f M When , the contrast loss function used is Loss=Loss1+Loss2+Loss3.

[0051] In the above technical solution, the second training image input as a whole image corresponds to two sets of preset text templates, and two sets of text feature vectors can be obtained accordingly. When defect detection is performed on the defects contained in the second training image, the detection rate of the defect is relatively large, which is conducive to avoiding missing the defect in the defect detection result.

[0052] Optionally, when the defect sample image is input into the feature extraction network, the preset text template includes a first preset text template for reflecting the unknown positioning of the defect area in the defect sample image and a second preset text template for reflecting the preset positioning position of the defect area in the defect sample image; or the preset text template includes a second preset text template for reflecting the preset positioning position of the defect area in the defect sample image.

[0053] Exemplarily, the defect sample image may include a first preset text template and a second preset text template, the first preset text template may reflect that the location of the defect area in the defect sample image is unknown, and the second preset text template may reflect the preset location of the defect area in the defect sample image. In a specific embodiment, if the defects contained in the defect sample image are located in any area on the sample object, they may be defects to be detected. For example, for a wafer, scratches located in any area may be defects to be detected. Then, the preset text template corresponding to the defect sample image whose defect type of the sample object is scratches and whose defect area falls into any area may include the first preset text template and the second preset text template, or may only include the second preset text template. If the defects contained in the defect sample image may be defects to be detected when they are located in an area at a preset location on the sample object, for example, dust may be a defect to be detected when it is located on a line area, then the preset text template corresponding to the defect sample image whose defect type of the sample object is dust and whose defect area falls into the line area may include the second preset text template.

[0054] In the above technical solution, the preset text template may include a first preset text template for reflecting the preset positioning position of the defect area in the defect sample image and a second preset text template for reflecting the unknown positioning of the defect area in the defect sample image. For certain defects located in any area of ​​the object to be tested, which may be defects to be detected, there may be multiple sets of preset defect feature vectors in the defect feature base library, which is conducive to improving the detection rate of these defects.

[0055] Optionally, when the defect sample image is input into the feature extraction network, when the preset to-be-detected condition corresponding to the defect to be detected in the defect sample image is the first to-be-detected condition, the preset text template includes a first preset text template and a second preset text template; when the preset to-be-detected condition corresponding to the defect to be detected in the defect sample image is the second to-be-detected condition, the preset text template includes a second preset text template and a second preset text template; the first to-be-detected condition refers to that the defect type is the first preset type and the defect area falls within any area on the sample object in the defect sample image, and the second to-be-detected condition refers to that the defect type is the second preset type and the defect area falls within the area at the preset positioning position on the sample object in the defect sample image.

[0056] Exemplarily, the preset text template corresponding to the defect sample image can be determined based on the preset to-be-detected condition corresponding to the defect to be detected in the defect sample image. When the defect to be detected in the defect sample image meets the first to-be-detected condition, the corresponding preset text template may include a first preset text template and a second preset text template. The first to-be-detected condition refers to a defect type of the first preset type and a defect area falling within any area on the sample object in the defect sample image. A defect of the first preset type refers to a defect that can be a defect to be detected when the defect is located in any area on the sample object. For example, when the sample object is a wafer, a scratch can be a defect to be detected when it is in any area on the wafer, and the scratch can be a defect of the first preset type. When the defect to be detected in the defect sample image meets the second to-be-detected condition, the corresponding preset text template may include a second preset text template. A defect of the second preset type refers to a defect that can be a defect to be detected when the defect area of ​​the defect falls within the area at the preset positioning position on the sample object. For example, when the sample object is a wafer, dust in the circuit area on the wafer can be used as a defect to be detected. In this case, the dust can be a defect of the second preset type, and the area at the preset positioning position can be the circuit area.

[0057] In the above technical scheme, when the defect type and defect area contained in the defect sample image meet different preset conditions for detection, the corresponding preset text templates may be different. This is conducive to accurately obtaining the defects to be detected on the object to be tested that meet the corresponding preset conditions for detection, and is conducive to avoiding the misdetection of defects that do not belong to the defects to be detected, thereby facilitating the defect detection results to meet the defect detection requirements.

[0058] See also Figure 4 As shown, it is a schematic block diagram of a defect detection device according to an embodiment of the present invention. According to another aspect of the present invention, a defect detection device 400 is also provided, comprising:

[0059] The acquisition module 410 is used to acquire the image to be tested obtained by performing image acquisition on the object to be tested and at least one set of preset defect feature vectors in the defect feature base library;

[0060] An input module 420, used for inputting the image to be tested into the image feature extraction module to obtain the image feature vector to be tested of the image to be tested;

[0061] A determination module 430 is used to determine a defect detection result of the object to be tested according to the similarity between the feature vector of the image to be tested and each set of preset defect feature vectors in at least one set of preset defect feature vectors, wherein the defect detection result includes information about whether there is a defect to be detected in the object to be tested, and the defect to be detected is a defect whose defect type and defect area both meet the preset conditions to be detected;

[0062] Among them, at least one group of preset defect feature vectors is a text feature vector obtained by inputting a feature extraction network of at least one defect sample image. The feature extraction network includes an image feature extraction module, a mapping module and a text feature extraction module connected in sequence. The image feature extraction module is used to extract the image feature vector of the input image of the input image feature extraction module. The mapping module is used to map the image feature vector into a text prediction vector. The text feature extraction module is used to extract the text feature vector of the defect description text. The defect description text is the text obtained by inserting the text prediction vector into a preset text template corresponding to the input image. The text information contained in the preset text template is used to reflect the positioning of the defect area of ​​the object in the input image. The defect sample image contains a sample object with defects to be detected.

[0063] See also Figure 5 As shown, it is a schematic block diagram of an electronic device according to an embodiment of the present invention. According to another aspect of the present invention, an electronic device is also provided, including: a processor 510 and a memory 520, wherein the memory 510 stores computer program instructions, and the computer program instructions are used by the processor 510 to execute the above-mentioned defect detection method when it is running.

[0064] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned defect detection method of the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned defect detection device according to the embodiment of the present invention or the corresponding modules in the above-mentioned defect detection device. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0065] According to yet another aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions are used to execute the defect detection method as described above when running.

[0066] A person skilled in the art can understand the specific implementation and beneficial effects of the above-mentioned defect detection device by reading the above-mentioned detailed description of the defect detection method, and for the sake of brevity, they will not be repeated here.

[0067] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0068] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0070] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0071] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0072] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0073] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0074] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in a defect detection device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0075] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0076] The above is only a specific embodiment or description of a specific embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A defect detection method, characterized in that: include: Acquire an image to be tested obtained by performing image acquisition on the object to be tested and at least one set of preset defect feature vectors in a defect feature base library; Inputting the image to be tested into an image feature extraction module to obtain an image feature vector of the image to be tested; Determine a defect detection result of the object to be tested according to the similarity between the feature vector of the image to be tested and each set of preset defect feature vectors in the at least one set of preset defect feature vectors, wherein the defect detection result includes information about whether there is a defect to be detected on the object to be tested, and the defect to be detected is a defect whose defect type and defect area both meet preset conditions to be detected; Among them, the at least one group of preset defect feature vectors is a text feature vector obtained by inputting at least one defect sample image into a feature extraction network, and the feature extraction network includes the image feature extraction module, the mapping module and the text feature extraction module connected in sequence, the image feature extraction module is used to extract the image feature vector of the input image input into the image feature extraction module, the mapping module is used to map the image feature vector into a text prediction vector, the text feature extraction module is used to extract the text feature vector of the defect description text, the defect description text is the text obtained by inserting the text prediction vector into a preset text template corresponding to the input image, the text information contained in the preset text template is used to reflect the positioning of the defect area of ​​the object in the input image, and the defect sample image contains a sample object with the defect to be detected.

2. The method according to claim 1, characterized in that The at least one set of preset defect feature vectors corresponds to at least one set of defect label information, the defect label information includes region information of the defect region and / or type information of the defect type, and the region information is used to indicate a preset positioning position of the defect region; The step of determining the defect detection result of the object to be tested according to the similarity between the feature vector of the image to be tested and each set of preset defect feature vectors in the at least one set of preset defect feature vectors comprises: The defect label information corresponding to the preset defect feature vector whose similarity with the feature vector of the image to be tested meets the preset requirement is determined as the defect detection result.

3. The method according to claim 2, characterized in that The process of creating the defect feature base library is as follows: Acquire the at least one defect sample image and one or more groups of preset text templates corresponding to each defect sample image and one or more groups of defect label information corresponding to each defect sample image, wherein the one or more groups of preset text templates corresponding to the same defect sample image correspond to the one or more groups of defect label information one-to-one, and at least part of the preset text templates and the corresponding defect label information express at least part of the same semantic information; Inputting each defect sample image of the at least one defect sample image into the feature extraction network to obtain one or more groups of preset defect feature vectors corresponding to each defect sample image, wherein the one or more groups of preset defect feature vectors corresponding to each defect sample image correspond one to one with the one or more groups of preset text templates corresponding to the defect sample image; The defect label information is associated and stored with the preset defect feature vector in a one-to-one correspondence.

4. The method according to claim 3, characterized in that When the preset text template includes text information for reflecting the preset positioning position of the defect area, the defect label information includes the area information or includes the area information and the type information; When the preset text template includes text information reflecting that the location of the defect area is unknown, the defect label information includes the type information.

5. The method according to any one of claims 1 to 4, characterized in that: The training process of the feature extraction network is as follows: Inputting a first training image into the feature extraction network to obtain a first training image feature vector output by the image feature extraction module and a first training text feature vector output by the text feature extraction module; Inputting a second training image into the feature extraction network to obtain a second training image feature vector output by the image feature extraction module and a second training text feature vector output by the text feature extraction module; According to a first contrast loss between the first training image feature vector and the first training text feature vector, and a second contrast loss between the second training image feature vector and the second training text feature vector, optimizing the mapping module in the feature extraction network until a preset requirement is met; Among them, the first training image is an image block containing a training object in the second training image, and the corresponding preset text template when the first training image is input includes a first preset text template for reflecting that the positioning of the defect area of ​​the training object is unknown, and the corresponding preset text template when the second training image is input includes a second preset text template for reflecting the preset positioning position of the defect area of ​​the training object.

6. The method according to claim 5, characterized in that When the second training image is input, the corresponding preset text template also includes a third preset text template for reflecting that the positioning of the defect area of ​​the training object is unknown, the second training text feature vector includes a first sub-vector corresponding to the second preset text template, and a second sub-vector corresponding to the third preset text template; the second contrast loss includes a contrast loss between the second training image feature vector and the first sub-vector, and a contrast loss between the second training image feature vector and the second sub-vector.

7. The method according to any one of claims 1 to 3, characterized in that: When the defect sample image is input into the feature extraction network, The preset text template includes a first preset text template for reflecting that the position of the defect area in the defect sample image is unknown and a second preset text template for reflecting the preset position of the defect area in the defect sample image; or The preset text template includes a second preset text template for reflecting a preset positioning position of a defect area in the defect sample image.

8. The method according to claim 7, characterized in that When the defect sample image is input into the feature extraction network, When the preset to-be-detected condition corresponding to the to-be-detected defect in the defect sample image is the first to-be-detected condition, the preset text template includes the first preset text template and the second preset text template; When the preset to-be-detected condition corresponding to the to-be-detected defect in the defect sample image is the second to-be-detected condition, the preset text template includes the second preset text template; The first condition to be detected refers to that the defect type is a first preset type and the defect area falls within any area on the sample object in the defect sample image, and the second condition to be detected refers to that the defect type is a second preset type and the defect area falls within the area at a preset positioning position on the sample object in the defect sample image.

9. A defect detection device, characterized in that: include: An acquisition module, used to acquire an image to be tested obtained by performing image acquisition on the object to be tested and at least one set of preset defect feature vectors in a defect feature base library; An input module, used for inputting the image to be tested into an image feature extraction module to obtain an image feature vector of the image to be tested; A determination module, configured to determine a defect detection result of the object to be tested according to the similarity between the feature vector of the image to be tested and each set of preset defect feature vectors in the at least one set of preset defect feature vectors, wherein the defect detection result includes information on whether there is a defect to be detected on the object to be tested, and the defect to be detected is a defect whose defect type and defect area both meet preset conditions to be detected; Among them, the at least one group of preset defect feature vectors is a text feature vector obtained by inputting at least one defect sample image into a feature extraction network, and the feature extraction network includes the image feature extraction module, the mapping module and the text feature extraction module connected in sequence, the image feature extraction module is used to extract the image feature vector of the input image input into the image feature extraction module, the mapping module is used to map the image feature vector into a text prediction vector, the text feature extraction module is used to extract the text feature vector of the defect description text, the defect description text is the text obtained by inserting the text prediction vector into a preset text template corresponding to the input image, the text information contained in the preset text template is used to reflect the positioning of the defect area of ​​the object in the input image, and the defect sample image contains a sample object with the defect to be detected.

10. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the defect detection method according to any one of claims 1 to 8 when executed.

11. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the defect detection method according to any one of claims 1 to 8 when running.

12. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the defect detection method according to any one of claims 1 to 8 when running.

Images