Defect detection method and device, equipment, storage medium and program product

Abstract

The invention discloses a defect detection method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target object image of a to-be-detected object; determining a target defect detection type of the to-be-detected object, and obtaining a reference object image corresponding to the target defect detection type and a reference detection result image of the reference object image; obtaining target object image features of the target object image, reference object image features of the reference object image and result image features of the reference detection result image through a general defect detection model; fusing the target object image features, the reference object image features and the result image features through a general defect detection model to obtain target fused image features; and according to the target fusion image features, defect detection is carried out through a general defect detection model, and a target detection result image conforming to the target defect detection type is obtained. Compared with the prior art, the defect detection cost can be reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a defect detection method, a defect detection device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] To ensure the integrity of a product's practical functionality and appearance, defect inspection is necessary. For example, display panels, as a key feature of intelligence, are widely used in numerous electronic devices, including mobile phones, tablets, TVs, and car computers. These panels require inspection for defects such as brightness defects and dark spots.

[0003] Currently, to detect product defects, specific defect detection models are required for specific defect detection types. For example, for the task of semantic segmentation of defects, a defect detection model suitable for this task is required. Understandably, this current defect detection approach requires a separate defect detection model for each type of defect detection task, resulting in high defect detection costs. Summary of the Invention

[0004] Embodiments of the present application provide a defect detection method, a defect detection device, an electronic device, a computer-readable storage medium, and a computer product, which can reduce the cost of defect detection.

[0005] In a first aspect, the defect detection method provided by the present application includes:

[0006] Acquire a target object image of an object to be detected;

[0007] Determine the target defect detection type of the object to be detected, and obtain a reference object image corresponding to the target defect detection type and a reference detection result image thereof;

[0008] Obtaining target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image through a universal defect detection model;

[0009] Through the universal defect detection model, the target object image features, the reference object image features and the result image features are fused to obtain the target fused image features;

[0010] According to the target fusion image features, defect detection is performed through a general defect detection model to obtain a target detection result image that meets the target defect detection type.

[0011] In a second aspect, the present application provides a defect detection device comprising:

[0012] An object image acquisition module is used to acquire a target object image of the object to be detected;

[0013] A reference image acquisition module is used to determine the target defect detection type of the object to be detected, and obtain a reference object image corresponding to the target defect detection type and a reference detection result image thereof;

[0014] a feature mapping module for obtaining target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image through a universal defect detection model;

[0015] A feature fusion module is used to fuse the target object image features, the reference object image features and the result image features through a general defect detection model to obtain the target fused image features;

[0016] The defect detection module is used to perform defect detection based on the target fusion image features through a general defect detection model to obtain a target detection result image that meets the target defect detection type.

[0017] Optionally, in one embodiment, the feature fusion module is used to obtain preset query features, and based on the preset query features, reference object image features and result image features, the universal defect detection model is used to enhance the attention of the target object image features to obtain a first target fused image feature.

[0018] Optionally, in one embodiment, the feature fusion module is used to perform self-attention enhancement on preset query features and target object image features through a universal defect detection model to obtain a first enhanced query feature and a first enhanced target object image feature; perform self-attention enhancement on reference object image features and result image features through a universal defect detection model to obtain a first enhanced reference object image feature and a first enhanced result image feature; perform self-attention enhancement on the first enhanced target object image feature and the first enhanced reference object image feature through a universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature; and use the second enhanced target object image feature as the first target fusion image feature.

[0019] Optionally, in one embodiment, the feature fusion module is used to splice the preset query feature and the target object image feature through a general defect detection model to obtain a first spliced ​​feature; perform self-attention enhancement on the first spliced ​​feature through the general defect detection model to obtain a first enhanced spliced ​​feature; and split the first enhanced spliced ​​feature through the general defect detection model to obtain a first enhanced query feature and a first enhanced target object image feature.

[0020] Optionally, in one embodiment, the feature fusion module is used to splice the first enhanced target object image feature and the first enhanced reference object image feature through a universal defect detection model to obtain a second spliced ​​feature; perform self-attention enhancement on the second spliced ​​feature through the universal defect detection model to obtain a second enhanced spliced ​​feature; and split the second enhanced spliced ​​feature through the universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature.

[0021] Optionally, in one embodiment, the feature fusion module is used to perform self-attention enhancement on the first enhanced query feature and the first enhanced result image feature through a general defect detection model to obtain a second enhanced query feature and a second enhanced result image feature; and to use the second enhanced query feature as the second target fusion image feature.

[0022] Optionally, in one embodiment, the defect detection module is used to map the first target fusion image features and the second target fusion image features to the same eigenvalue scale through a universal defect detection model to obtain first updated target fusion image features and second updated target fusion image features; and perform defect detection through a universal defect detection model based on the first updated target fusion image features and the second updated target fusion image features to obtain a target detection result image that meets the target defect detection type.

[0023] Optionally, in one embodiment, the reference image acquisition module is used to acquire a reference object image and a reference detection result image thereof, which have the same object type as the object to be detected and correspond to the target defect detection type.

[0024] Optionally, in one embodiment, the reference detection result image includes a color mask image. When the target defect detection type is defect semantic segmentation, the mask area of ​​a preset color in the color mask image is used to indicate the existing defect area. When the target defect detection type is defect instance segmentation, the mask areas of different colors in the color mask image are used to indicate different defect instances. When the target defect detection type is defect target detection, the mask areas of different colors in the color mask image are used to indicate different defect areas.

[0025] Optionally, in one embodiment, the defect detection module is further configured to convert the target detection result image into a bounding box for indicating a defect area in the target object image.

[0026] Optionally, in one embodiment, the object image acquisition module is further configured to acquire a sample target object image of the sample inspection object and label detection result images corresponding to different defect detection types of the sample target object image;

[0027] The reference image acquisition module is further used to acquire sample reference object images corresponding to different defect detection types and sample reference detection result images thereof;

[0028] The feature mapping module is further used to obtain sample target object image features of the sample target object image, sample reference object image features of the sample reference object image, and sample result image features of the sample reference detection result image through the universal defect detection model;

[0029] The feature fusion module is further used to obtain a preset query feature, and to fuse the sample target object image feature, the sample reference object image feature, the result image feature, and the preset query feature through the universal defect detection model to obtain a first sample target fused image feature corresponding to the sample target object image feature, a second sample target fused image feature corresponding to the preset query feature, a third sample target fused image feature corresponding to the sample reference object image feature, and a fourth sample target fused image feature corresponding to the sample result image feature;

[0030] The defect detection module is further configured to map the first sample target fusion image feature, the second sample target fusion image feature, the third sample target fusion image feature, and the fourth sample target fusion image feature to the same feature value scale through a universal defect detection model to obtain a first updated sample target fusion image feature, a second updated sample target fusion image feature, a third updated sample target fusion image feature, and a fourth updated sample target fusion image feature; and perform defect detection through the universal defect detection model based on the first updated sample target fusion image feature and the second updated sample target fusion image feature to obtain a first sample target detection result image; and perform defect detection through the universal defect detection model based on the third updated sample target fusion image feature and the fourth updated sample target fusion image feature to obtain a second sample target detection result image;

[0031] The defect detection device provided in the present application also includes a model training module, which is used to obtain a first loss based on a first sample target detection result image and a label detection result image; obtain a second loss based on a second sample target detection result image and a sample reference detection result image; and update the model parameters of the universal defect detection model based on the first loss and the second loss until a preset update stop condition is met.

[0032] On the third aspect, the electronic device provided in this application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the defect detection method provided in this application.

[0033] In a fourth aspect, the computer-readable storage medium provided in the present application stores a computer program, which is suitable for being run by a processor to implement the steps in the defect detection method provided in the present application.

[0034] In a fifth aspect, the computer program product provided in the present application includes a computer program, which is suitable for being run by a processor to implement the steps in the defect detection method provided in the present application.

[0035] The defect detection solution provided by the present application can realize multiple different types of defect detection through a single universal defect detection model. In particular, unlike related technologies, in addition to obtaining the target object image of the object to be detected, a reference object image corresponding to the target defect detection type of the object to be detected and its reference detection result image are also obtained, and the target object image features of the target object image, the reference object image features of the reference object image, and the result image features of the reference detection result image are respectively obtained through the universal defect detection model, and the aforementioned three image features are fused at the feature level to obtain the target fusion image features, so that the universal defect detection model can perceive the target defect detection type of the object to be detected through the target fusion image features, and accordingly, the universal defect detection model performs defect detection according to the target fusion image features to obtain a target detection result image that meets the target defect detection type. In this way, different types of defect detection are realized through a single universal defect detection model, which reduces the complexity of defect detection and simplifies the deployment and maintenance of the model. Since there is no need to manage multiple models of different defect detection types, the cost of defect detection can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1a This is a schematic diagram of a defect detection system according to an embodiment of the present application;

[0038] Figure 1b This is a schematic diagram illustrating a process of a defect detection method provided by an embodiment of the present application;

[0039] Figure 1c This is an example diagram of the architecture of a general defect detection model provided in an embodiment of the present application;

[0040] Figure 1d is a schematic diagram of obtaining different image features in an embodiment of the present application;

[0041] Figure 1e Schematic diagram of the architecture of the task navigator network in the general defect detection model provided in the embodiment of the present application;

[0042] Figure 1f This is an example of an image of a target detection result corresponding to a hardware accessory in an embodiment of the present application;

[0043] Figure 1g is an example diagram of a sample target object image and its corresponding label detection result image obtained in an embodiment of the present application;

[0044] Figure 2 This is another flowchart of the defect detection method provided by an embodiment of the present application;

[0045] Figure 3 1 is a schematic structural diagram of a defect detection device provided in an embodiment of the present application;

[0046] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] It should be noted that the principles of this application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of this application and should not be considered as limiting other specific embodiments not described in detail herein.

[0048] In the following description of this application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0049] In the following description of this application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0051] It's important to note that artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also encompasses the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0052] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be widely applied to downstream tasks across various AI domains after fine-tuning. AI software technologies primarily encompass machine learning (ML). Deep learning (DL) is a new research direction within ML, introduced to bring ML closer to its original goal: AI. Currently, deep learning is primarily used in fields such as machine vision and natural language processing. Deep learning involves learning the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly useful for interpreting data such as text, images, and audio. Using deep learning techniques and corresponding training sets, network models can be trained to implement diverse functions. For example, taking the generative model as an example, based on different types of training sets, we can train generative models that can generate different types of content, such as generative models that can generate images, generative models that can generate text, and generative models that can generate speech.

[0053] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in object recognition, measurement, and other machine vision tasks. Further image processing is performed to transform the computer's image into an image more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision. Pre-trained models in the field of vision, such as the Swin Transformer, ViT, V-MOE, and MAE, can be fine-tuned to quickly and widely apply to specific downstream tasks. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. Common biometric recognition technologies include facial recognition and fingerprint recognition.

[0054] This application primarily relates to the field of computer vision technology using artificial intelligence technology, and provides a defect detection method, a defect detection device, an electronic device, a computer-readable storage medium, and a computer program product. The defect detection method can be executed by the defect detection device, or by an electronic device incorporating the defect detection device.

[0055] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0056] Please refer to Figure 1aThe present application also provides a defect detection system, which includes an electronic device 100 for executing the defect detection method provided by the present application. The electronic device 100 can be any device equipped with a processor and having processing capabilities, such as a mobile device with a processor such as a smart phone, a tablet computer, a PDA, a laptop computer, a virtual reality device, an augmented reality device, or a mixed reality device, or a fixed device with a processor such as a desktop computer, a television, a server, an industrial device, etc., wherein a target object image of the object to be detected is obtained; the target defect detection type of the object to be detected is determined, and a reference object image corresponding to the target defect detection type and a reference detection result image thereof are obtained; the target object image features of the target object image, the reference object image features of the reference object image, and the result image features of the reference detection result image are obtained through a universal defect detection model; the target object image features, the reference object image features, and the result image features are fused through the universal defect detection model to obtain target fused image features; based on the target fused image features, defect detection is performed through the universal defect detection model to obtain a target detection result image that meets the target defect detection type.

[0057] In addition, if Figure 1a As shown, the defect detection system may further include a memory 200 for storing relevant data in the defect detection process, such as the original data such as the acquired target object image, the determined target defect detection type, the acquired reference object image and its reference detection result image, the intermediate data such as the target object image features, the reference object image features, the result image features, the target fusion image features in the processing process, and the result data such as the target detection result image finally obtained.

[0058] It should be noted that the defect detection system described above is merely an example, which is intended to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. A person of ordinary skill in the art will know that with the evolution of the defect detection system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0059] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0060] Please refer to Figure 1b , Figure 1b FIG. 1 is a flow chart of the defect detection method provided in this embodiment. Figure 1b As shown, the process of the defect detection method can be as follows:

[0061] In 110 , a target object image of an object to be detected is acquired.

[0062] Among them, the object to be inspected is used to refer to the object that needs to be inspected for defects. In actual scenarios, it can be a product obtained by industrial manufacturing, such as a circuit board, a display panel, hardware accessories, etc. It is understandable that due to various reasons such as the production process, the products obtained by industrial manufacturing cannot be perfect, and they may have various defects. For example, for a circuit board, it may have defects such as component displacement, skew, bent legs, and wrong parts; for a display panel, it may have defects such as breakage, scratches, bright spots, dark spots, etc.; for hardware accessories, it may have defects such as pits, cracks, scratches, etc. Whether it is an industrially manufactured product such as a circuit board, a display panel or a hardware accessory, the defects it has may affect its normal use. Therefore, it is necessary to perform defect detection on these products so that the defective products can be repaired or discarded.

[0063] In an embodiment of the present application, an object image of the object to be detected is first obtained, which is recorded as a target object image, so as to use the target object image to implement defect detection of a specific defect detection type on the object to be detected. The object image can be generally understood as an image obtained by photographing the object to be detected without changing the image content. For example, when the electronic device that executes the defect detection method of the present application is configured with a camera, the configured camera can be used to directly photograph the object to be detected, thereby obtaining the target object image of the object to be detected; in addition, the electronic device that executes the defect detection method of the present application can also obtain the target object image obtained by photographing the object to be detected by an external device, and can also obtain the target object image of the object to be detected through other channels such as the Internet.

[0064] In 120 , a target defect detection type of the object to be detected is determined, and a reference object image corresponding to the target defect detection type and a reference detection result image thereof are obtained.

[0065] It should be noted that in the sub-field of defect detection using artificial intelligence technology, there are usually three types of defect detection: defect target detection, defect semantic segmentation, and defect instance segmentation.

[0066] The task of target detection is to find all the targets (objects) of interest in the image and determine their categories and locations. Correspondingly, the task of defect target detection is to find the defects in the image and determine their categories and locations.

[0067] Semantic segmentation is the task of classifying every pixel in an image, assigning a list to each pixel. Once each pixel is labeled with a different category, each corresponding pixel is assigned a new color and reassembled into a new image. Connecting the colored pixels creates an image that shows an object segmented from the entire image and contains all of its semantic information. Defect semantic segmentation, on the other hand, involves segmenting an image into areas with defects and areas without them.

[0068] Semantic segmentation can only classify objects into categories, but not into the same category. Therefore, in addition to semantic segmentation, instance segmentation is also required to distinguish different instances of the same category. Instance segmentation combines object detection with semantic segmentation. As the name suggests, instance segmentation separates specific objects within a category. Defect instance segmentation, on the other hand, separates individual defects in an image and determines their categories.

[0069] In an embodiment of the present application, in addition to obtaining a target object image of the object to be inspected, a target defect detection type of the object to be inspected is further determined. The target defect detection type is used to indicate the type of defect detection that the inspector of the object to be inspected desires to perform on the object to be inspected. For example, an electronic device that executes the defect detection method of the present application provides an input interface for a defect detection type, and receives the target defect detection type of the object to be inspected inputted through the input interface. The input interface can be an input interface based on a graphical user interface, an input interface based on a command prompt, an input interface based on voice recognition, or the like.

[0070] It should be noted that the embodiment of the present application is pre-trained with a general defect detection model. Through training, the general defect detection model is configured to perform defect detection of a specific defect detection type based on the input object image and the task prompt sample for indicating the specific defect detection type, and output the detection result image of the corresponding object image accordingly.

[0071] In an embodiment of the present application, the task prompt example consists of two parts: an object image for reference, and a detection result image for reference obtained by performing defect detection of a specific defect detection type on the object image for reference.

[0072] Correspondingly, in an embodiment of the present application, after determining the target defect detection type of the object to be detected, an object image for reference is further obtained, which is recorded as a reference object image, and a detection result image obtained by completing the defect detection of the target defect detection type of the reference object image is obtained, which is recorded as a reference detection result image. In this way, the reference object image and the reference detection result image can be combined to characterize the target defect detection type.

[0073] It should be noted that the reference object corresponding to the reference object image and the object to be detected can be objects of the same type or different types. In addition, for the reference object image, it can be that only the defect detection of the target defect detection type is completed and the detection result image is obtained, or it can be that the defect detection including the target defect detection type is completed and the detection result images are obtained separately.

[0074] Optionally, in one embodiment, to improve the accuracy of defect detection, obtaining a reference object image corresponding to the target defect detection type and a reference detection result image thereof includes:

[0075] A reference object image and a reference detection result image thereof, which have the same object type as the object to be detected and correspond to the target defect detection type, are obtained.

[0076] In an embodiment of the present application, when obtaining a reference object image and a reference detection result image thereof, the object type of the object to be detected is first determined based on the target object image, and then a reference object image and a reference detection result image thereof are obtained, whose object type is the same as that of the object to be detected and corresponds to the target defect detection type.

[0077] For example, assuming that the object type of the object to be detected is determined to be a display panel, an image of another display panel is obtained as a reference object image, and a detection result image obtained after defect detection of the target defect detection type is performed based on the reference object image is used as a reference detection result image.

[0078] For another example, assuming that the object type of the object to be inspected is determined to be a circuit board, an image of another circuit board is obtained as a reference object image, and a detection result image obtained after defect detection of the target defect detection type is performed based on the reference object image is used as a reference detection result image.

[0079] In 130 , target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image are acquired through the universal defect detection model.

[0080] Please refer to Figure 1cThe general defect detection model provided in this application consists of three parts, namely, an encoder network, a task navigator network and a decoder network, wherein the encoder network is configured to map the input object image, the object image indicating a specific defect detection type and its detection result image to the feature space to obtain its hidden layer representation - image feature; the task navigator network is configured to fuse the image features of the three to obtain a fused image feature, so that the decoder network can perceive the aforementioned specific defect detection type; the decoder network is configured to perform defect detection based on the fused image feature to obtain a detection result image that meets the aforementioned specific defect detection type, and the detection result image indicates the defects in the object image through the mask area.

[0081] It should be noted that, subject to the constraints of realizing the functions of their respective configurations, the network structures of the above encoder network, task navigator network and decoder network can be set by those skilled in the art according to actual needs, and no limitation is imposed here.

[0082] Accordingly, in an embodiment of the present application, after determining the target defect detection type of the object to be detected and obtaining the reference object image corresponding to the target defect detection type and its reference detection result image, the general defect detection model is further used to obtain the image features of the target object image, which are recorded as target object image features, the image features of the reference object image, which are recorded as reference object image features, and the image features of the reference detection result image, which are recorded as result image features.

[0083] Please refer to Figure 1d The target object image, the reference object image, and the reference detection result image can be respectively input into the encoder network of the general defect detection model for feature mapping, and the target object image features of the target object image, the reference object image features of the reference object image, and the result image features of the reference detection result image can be obtained accordingly.

[0084] For example, in an embodiment of the present application, a Transformer-based ViT (Vision Transformer) or Swin-Transformer can be used as an encoder network, where the main idea of ​​ViT is to divide the input image into multiple sampling blocks, and then convert each sampling block into a vector, and finally splice these vectors together to form a sequence; Swin-Transformer is modified based on ViT using a sliding window. Its main idea is to divide the fixed-size sampling blocks in ViT into blocks of different sizes according to the hierarchy. The information between each block is not common and is calculated independently to improve the computing efficiency.

[0085] In 140 , the target object image features, the reference object image features, and the result image features are fused through the universal defect detection model to obtain target fused image features.

[0086] As mentioned above, after obtaining the target object image features of the target object image, the reference object image features of the reference object image, and the result image features of the reference detection result image, the universal defect detection model provided in this application is further used to fuse the target object image features, the reference object image features, and the result image features, and the fused image features are recorded as target fused image features. In this way, the target fused image features fuse the target object image features and the target defect detection type information, so that the universal defect detection model can perceive the target defect detection type expected by the detection party for the object to be detected.

[0087] Among them, the above target object image features, reference object image features and result image features can be input into the task navigator network of the general defect detection model, and the target object image features, reference object image features and result image features are fused through the task navigator network to obtain the corresponding target fused image features.

[0088] Optionally, in one embodiment, a target fused image feature is obtained by fusing the target object image feature, the reference object image feature, and the result image feature through a universal defect detection model, including:

[0089] A preset query feature is obtained, and according to the preset query feature, the reference object image feature and the result image feature, the attention enhancement is performed on the target object image feature through a universal defect detection model to obtain a first target fused image feature.

[0090] The preset query feature can be regarded as a feature container, in which the feature value is initially configured to be 0 or a number close to 0, such as 0.01.

[0091] In an embodiment of the present application, when image features are fused through a general defect detection model, attention is enhanced on the target object image features based on preset query features, reference object image features, and result image features, thereby fusing the preset query features, reference object image features, and result image features into the target object image features, and the target object image features after attention enhancement are correspondingly recorded as the first target fused image features.

[0092] In addition, based on the target object image features, reference object image features and result image features, the preset query features are focused on and enhanced through the general defect detection model, so that the target object image features, reference object image features and result image features are fused into the preset query features, and the preset query features after attention enhancement are recorded as the second target fused image features.

[0093] In addition, based on the target object image features, preset query features and result image features, the reference object image features are attention enhanced through the general defect detection model, so that the target object image features, preset query features and result image features are fused into the reference object image features, and the reference object image features after attention enhancement are correspondingly recorded as the third target fused image features.

[0094] In addition, according to the target object image features, the reference object image features and the preset query features, the result image features are enhanced by the universal defect detection model, so that the target object image features, the reference object image features and the preset query features are fused into the result image features, and the result image features after attention enhancement are recorded as the fourth target fused image features.

[0095] The embodiment of the present application uses the above fusion method to achieve full fusion of input features and task prompt features, so that the general defect detection model can perceive the target defect detection type of the object to be detected.

[0096] Optionally, in one embodiment, according to the preset query features, the reference object image features, and the result image features, attention enhancement is performed on the target object image features by using a universal defect detection model to obtain a first target fused image feature, including:

[0097] By using a universal defect detection model, self-attention enhancement is performed on the preset query feature and the target object image feature to obtain a first enhanced query feature and a first enhanced target object image feature;

[0098] Performing self-attention enhancement on the reference object image features and the result image features through a universal defect detection model to obtain a first enhanced reference object image feature and a first enhanced result image feature;

[0099] Performing self-attention enhancement on the first enhanced target object image feature and the first enhanced reference object image feature through a universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature;

[0100] The second enhanced target object image feature is used as the first target fused image feature.

[0101] In an embodiment of the present application, when a first target fused image feature based on the target object image feature is obtained by fusion, the preset query feature and the target object image feature are first self-attention enhanced through a general defect detection model, and the enhanced preset query feature is recorded as the first enhanced query feature, and the enhanced target object image feature is recorded as the first enhanced target object image feature.

[0102] Among them, the task navigator network of the general defect detection model can be used to splice the preset query features and the target object image features to obtain the first splicing feature, and then self-attention enhancement is performed on the first splicing feature. The enhanced first splicing feature is recorded as the first enhanced splicing feature. This process can be expressed as:

[0103]

[0104] C1=concat(F qi ,F ql );

[0105] Q=K=V=C1;

[0106]

[0107] Among them, C'1 represents the first enhanced splicing feature, C1 represents the first splicing feature, F qi represents the target object image feature, F ql Indicates the preset query feature.

[0108] As above, after performing self-attention enhancement on the first splicing feature to obtain the first enhanced splicing feature, the first enhanced splicing feature is further split through the task navigator network of the general defect detection model to obtain the first enhanced query feature and the first enhanced target object image feature accordingly.

[0109] It can be understood that the first splicing feature is obtained by splicing the preset query feature and the target object image feature in the channel dimension. Accordingly, after the first splicing feature is self-attention enhanced to obtain the first enhanced splicing feature, the first enhanced splicing feature is split according to the channel to obtain the enhanced preset query feature, that is, the first enhanced query feature, and the enhanced target object image feature, that is, the first enhanced target object image feature.

[0110] In addition, the reference object image features and the result image features are self-attention enhanced through the general defect detection model, and the enhanced reference object image features are recorded as the first enhanced reference object image features, and the enhanced result image features are recorded as the first enhanced result image features.

[0111] Among them, the reference object image features and the result image features can be spliced ​​together through the task navigator network of the general defect detection model to obtain a third spliced ​​feature, and then the third spliced ​​feature is enhanced by self-attention. The enhanced third spliced ​​feature is recorded as the third enhanced spliced ​​feature. This process can be expressed as:

[0112]

[0113] C3=concat(F pi ,F pl );

[0114] Q=K=V=C3;

[0115]

[0116] Among them, C'3 represents the third enhanced splicing feature, C3 represents the third splicing feature, F pi represents the reference object image feature, F pl Represents the resulting image features.

[0117] After performing self-attention enhancement on the third splicing feature to obtain the third enhanced splicing feature, the third enhanced splicing feature is further split through the task navigator network of the universal defect detection model to obtain the first enhanced reference object image feature and the first enhanced result image feature accordingly.

[0118] It can be understood that the third splicing feature is obtained by splicing the reference object image feature and the result image feature in the channel dimension. Accordingly, after the third splicing feature is self-attention enhanced to obtain the third enhanced splicing feature, the third enhanced splicing feature is split according to the channel to obtain the enhanced reference object image feature, that is, the first enhanced reference object image feature, and the enhanced result image feature, that is, the first enhanced result image feature.

[0119] As mentioned above, after self-attention enhancement is performed on the preset query feature and the target object image feature through the general defect detection model to obtain the first enhanced query feature and the first enhanced target object image feature, and the reference object image feature and the result image feature are self-attention enhanced to obtain the first enhanced reference object image feature and the first enhanced result image feature, the first enhanced target object image feature and the first enhanced reference object image feature are further self-attention enhanced through the general defect detection model, and the first enhanced target object image feature after further enhancement is recorded as the second enhanced target object image feature, and the first enhanced reference object image feature after further enhancement is recorded as the second enhanced reference object image feature, and the second enhanced target object image feature is used as the first target fusion image feature, and the second enhanced reference object image feature is used as the third target fusion image feature.

[0120] Among them, the first enhanced target object image feature and the first enhanced reference object image feature can be spliced ​​through the task navigator network of the general defect detection model to obtain a second spliced ​​feature, and then the second spliced ​​feature is self-attention enhanced, and the enhanced second spliced ​​feature is recorded as the second enhanced spliced ​​feature. This process can be expressed as:

[0121]

[0122]

[0123] Q=K=V=C2;

[0124]

[0125] Wherein, C'2 represents the second enhanced splicing feature, C2 represents the second splicing feature, represents the first enhanced target object image feature, represents the first enhanced reference object image feature.

[0126] After performing self-attention enhancement on the second splicing feature to obtain a second enhanced splicing feature, the second enhanced splicing feature is further split through the task navigator network of the universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature accordingly.

[0127] It can be understood that the second splicing feature is obtained by splicing the first enhanced target object image feature and the first enhanced reference object image feature in the channel dimension. Accordingly, after the second splicing feature is self-attention enhanced to obtain the second enhanced splicing feature, the second enhanced splicing feature is split according to the channel to obtain the first enhanced target object image feature after further enhancement, that is, the second enhanced target object image feature, and the first enhanced reference object image feature after further enhancement, that is, the second enhanced reference object image feature.

[0128] Optionally, fusing the target object image feature, the reference object image feature, and the result image feature into the preset query feature to obtain a second target fused image feature includes:

[0129] Performing self-attention enhancement on the first enhanced query feature and the first enhanced result image feature through a universal defect detection model to obtain a second enhanced query feature and a second enhanced result image feature;

[0130] The second enhanced query feature is used as the second target fused image feature.

[0131] In an embodiment of the present application, when the second target fused image feature based on the preset query feature is fused, the first enhanced query feature and the first enhanced result image feature are self-attention enhanced through the universal defect detection model to obtain the second enhanced query feature and the second enhanced result image feature, and the second enhanced query feature is used as the second target fused image feature, and the second enhanced result image feature is used as the fourth target fused image feature.

[0132] Among them, the first enhanced query feature and the first enhanced result image feature can be spliced ​​together through the task navigator network of the general defect detection model to obtain a fourth spliced ​​feature, and then the fourth spliced ​​feature is self-attention enhanced. The enhanced fourth spliced ​​feature is recorded as the fourth enhanced spliced ​​feature. This process can be expressed as:

[0133]

[0134]

[0135] Q=K=V=C4;

[0136]

[0137] Wherein, C'4 represents the fourth enhanced splicing feature, C4 represents the fourth splicing feature, represents the first enhanced result image feature, Represents the first enhanced query feature.

[0138] After performing self-attention enhancement on the fourth splicing feature to obtain a fourth enhanced splicing feature, the fourth enhanced splicing feature is further split through a task navigator network of a universal defect detection model to correspondingly obtain a second enhanced query feature and a second enhanced result image feature.

[0139] It can be understood that the fourth splicing feature is obtained by splicing the first enhanced query feature and the first enhanced result image feature in the channel dimension. Accordingly, after the fourth splicing feature is self-attention enhanced to obtain the fourth enhanced splicing feature, the fourth enhanced splicing feature is split according to the channel to obtain the first enhanced query feature after further enhancement, that is, the second enhanced query feature, and the first enhanced result image feature after further enhancement, that is, the second enhanced result image feature.

[0140] As mentioned above, this application first splices the preset query feature and the target object image feature into a first splicing feature, then performs self-attention enhancement to obtain a first enhanced splicing feature, and then splits the first enhanced splicing feature into a first enhanced query feature and a first enhanced target object image feature, so as to achieve horizontal interaction and establish a semantic association between the two.

[0141] Similarly, the reference object image features and the result image features are spliced ​​into a third spliced ​​feature and then self-attention enhancement is performed to obtain a third enhanced spliced ​​feature, which is then split into a first enhanced reference object image feature and a first enhanced result image feature, so as to achieve horizontal interaction and establish a semantic association between the two.

[0142] Furthermore, the first enhanced target object image feature and the first enhanced reference object image feature are spliced ​​into a second spliced ​​feature and then self-attention enhancement is performed to obtain a second enhanced spliced ​​feature, and then the second enhanced spliced ​​feature is split into a second enhanced target object image feature and a second enhanced reference object image feature, so as to achieve vertical interaction and establish a contextual association between the two.

[0143] The first enhanced query feature and the first enhanced result image feature are spliced ​​into a fourth spliced ​​feature, and then self-attention enhancement is performed to obtain the fourth enhanced spliced ​​feature. The fourth enhanced spliced ​​feature is then split into a second enhanced query feature and a second enhanced result image feature, so as to achieve vertical interaction and establish a contextual association between the two.

[0144] In this way, through the above horizontal interaction and vertical interaction, the image features of the target object to be detected, the image features of the reference object that indicate the target defect detection type, and the result image features are fully integrated, so that the general defect detection model can perceive the target defect detection type.

[0145] In 150 , defect detection is performed using a general defect detection model based on the target fusion image features to obtain a target detection result image that meets the target defect detection type.

[0146] As mentioned above, after the target fusion image features are obtained by fusion, defect detection is further performed through a general defect detection model based on the target fusion image features to obtain a target detection result image that meets the target defect detection type. In other words, the target detection result image will indicate the defects in the target object image in a manner that matches the target defect detection type.

[0147] Among them, the target fusion image features obtained by fusion can be input into the decoder network of the general defect detection model. It can be understood that since the target fusion image features carry the information of the target defect detection type, the decoder network can perceive the target defect detection type expected by the detection party for the object to be detected. Accordingly, defect detection is performed according to the target fusion image features through the decoder network, and a target detection result image that meets the target defect detection type will be obtained.

[0148] Exemplarily, in an embodiment of the present application, the decoder network may be composed of 6 Transformer layers and 1 1*1 convolutional layer.

[0149] Optionally, in one embodiment, defect detection is performed using a general defect detection model based on target fusion image features to obtain a target detection result image that meets the target defect detection type, including:

[0150] Mapping the first target fused image feature and the second target fused image feature to the same eigenvalue scale through a universal defect detection model to obtain a first updated target fused image feature and a second updated target fused image feature;

[0151] According to the first updated target fused image features and the second updated target fused image features, defect detection is performed using a universal defect detection model to obtain a target detection result image that meets the target defect detection type.

[0152] In an embodiment of the present application, based on the third target fusion image feature and the fourth target fusion image feature, the first target fusion image feature and the second target fusion image feature are mapped to the same eigenvalue scale through a universal defect detection model to obtain the first updated target fusion image feature and the second updated target fusion image feature.

[0153] Among them, the first target fusion image feature, the second target fusion image feature, the third target fusion image feature and the fourth target fusion image feature can be mapped to the same eigenvalue scale through the task navigator network of the general defect detection model to obtain the first updated target fusion image feature, the second updated target fusion image feature, the third updated target fusion image feature and the fourth updated target fusion image feature.

[0154] For example, please refer to Figure 1e The task navigator network consists of three layers: a horizontal interaction layer, a vertical interaction layer, and a multi-layer perceptron layer. The horizontal interaction layer is configured to perform the horizontal interaction operations described in the above embodiments, the vertical interaction layer is configured to perform the vertical interaction operations described in the above embodiments, and the multi-layer perceptron layer is configured to perform the eigenvalue scaling operations described in the above embodiments. The specific structural configurations of the horizontal interaction layer, the vertical interaction layer, and the multi-layer perceptron layer are subject to the constraints of enabling the aforementioned operations and can be configured by designers skilled in the art based on actual needs.

[0155] As mentioned above, after mapping the first target fusion image features, the second target fusion image features, the third target fusion image features and the fourth target fusion image features to the same eigenvalue scale to obtain the first updated target fusion image features, the second updated target fusion image features, the third updated target fusion image features and the fourth updated target fusion image features, further based on the first updated target fusion image features and the second updated target fusion image features, defect detection is performed through the decoder network of the general defect detection model to obtain a target detection result image that meets the target defect detection type.

[0156] Optionally, in one embodiment, the reference detection result image includes a color mask image, and pixels of the same color constitute a mask area. When the target defect detection type is defect semantic segmentation, the mask area of ​​a preset color in the color mask image is used to indicate the existing defect area (regardless of the defect type). When the target defect detection type is defect instance segmentation, mask areas of different colors in the color mask image are used to indicate different defect instances. When the target defect detection type is defect target detection, rectangular mask areas of different colors in the color mask image are used to indicate different defect areas.

[0157] For example, please refer to Figure 1f , the object to be tested is a hardware accessory with a pit defect, such as Figure 1f As shown, when the target defect detection type is determined to be defect semantic segmentation, the target detection result image obtained by defect detection indicates the area where the hardware accessory has defects through the red mask area (the defect type is not distinguished at this time); when the target defect detection type is determined to be defect instance segmentation, the target detection result image obtained by defect detection indicates the instance area of ​​the pit defect in the hardware accessory through the blue mask area; when the target defect detection type is determined to be defect target detection, the target detection result image obtained by defect detection indicates the rectangular area where the pit defect exists in the hardware accessory through the orange rectangular mask area.

[0158] Optionally, in one embodiment, when the target defect detection type is defect target detection, after performing defect detection using a universal defect detection model based on target fusion image features and obtaining a target detection result image that meets the target defect detection type, the method further includes:

[0159] The object detection result image is converted into a bounding box indicating the defect area in the target object image.

[0160] As mentioned above, when the target defect detection type is defect target detection, different mask areas in the target detection result image indicate different defect areas. In order to increase the readability of the target detection result image, the embodiment of the present application performs defect detection through a general defect detection model based on the target fusion image features. After obtaining the target detection result image that meets the target defect detection type, the target detection result image is further converted into a bounding box for indicating the defect area in the target object image.

[0161] For example, please continue to refer to Figure 1f , the target detection result image corresponding to defect target detection is converted into a bounding box (x, y, w, h) indicating the defect area in the target object image, where x represents the horizontal coordinate of the orange rectangular mask area in the target detection result image, y represents the vertical coordinate of the orange rectangular mask area in the target detection result image, w represents the width of the orange rectangular mask area in the target detection result image, and h represents the height of the orange rectangular mask area in the target detection result image.

[0162] Optionally, in one embodiment, the general defect detection model provided in this application can be trained in the following manner:

[0163] Obtaining a sample target object image of a sample inspection object and label inspection result images corresponding to different defect inspection types of the sample target object image;

[0164] Obtaining sample reference object images and sample reference detection result images corresponding to different defect detection types;

[0165] Obtaining, by means of a universal defect detection model, sample target object image features of a sample target object image, sample reference object image features of a sample reference object image, and sample result image features of a sample reference detection result image;

[0166] Obtaining a preset query feature, and fusing the sample target object image feature, the sample reference object image feature, the result image feature, and the preset query feature through a universal defect detection model to obtain a first sample target fused image feature corresponding to the sample target object image feature, a second sample target fused image feature corresponding to the preset query feature, a third sample target fused image feature corresponding to the sample reference object image feature, and a fourth sample target fused image feature corresponding to the sample result image feature;

[0167] Mapping the first sample target fusion image feature, the second sample target fusion image feature, the third sample target fusion image feature, and the fourth sample target fusion image feature to the same feature value scale through a universal defect detection model to obtain a first updated sample target fusion image feature, a second updated sample target fusion image feature, a third updated sample target fusion image feature, and a fourth updated sample target fusion image feature;

[0168] Performing defect detection using a universal defect detection model based on the first updated sample target fused image features and the second updated sample target fused image features to obtain a first sample target detection result image;

[0169] Performing defect detection using a universal defect detection model based on the third updated sample target fused image feature and the fourth updated sample target fused image feature to obtain a second sample target detection result image;

[0170] Obtaining a first loss according to the first sample target detection result image and the label detection result image;

[0171] Obtaining a second loss according to the second sample target detection result image and the sample reference detection result image;

[0172] The model parameters of the universal defect detection model are updated according to the first loss and the second loss until a preset update stop condition is met.

[0173] The sample inspection object can be any object type requiring defect inspection, such as a circuit board, display panel, or hardware accessory. In this embodiment of the present application, an object image of the sample inspection object is obtained and recorded as a sample target object image. Furthermore, labeled detection result images corresponding to different defect detection types for the sample target object image are also obtained. The labeled detection result images are obtained by pre-labeling.

[0174] In the embodiment of the present application, the label detection result image includes a color mask image, in which pixels of the same color constitute a mask area. The label detection result image corresponding to defect semantic segmentation indicates the existing defect area through a mask area of ​​a preset color; the label detection result image corresponding to defect instance segmentation indicates different defect instances through mask areas of different colors; the label detection result image corresponding to defect target detection indicates different defect areas through rectangular mask areas of different colors. For example, please refer to Figure 1g , shows a sample target object image of a hardware accessory and the label detection result images corresponding to different defect detection types, such as Figure 1gAs shown, in the label detection result image corresponding to defect semantic segmentation, the red mask area is used to indicate the existing defect area (regardless of the defect type); in the label detection result image corresponding to defect instance segmentation, different defect instances are indicated by blue mask areas and green mask areas respectively; in the label detection result image corresponding to defect target detection, different defect areas are indicated by orange rectangular mask areas and yellow rectangular mask areas respectively.

[0175] In addition, sample reference object images corresponding to different defect detection types and their sample reference detection result images are also obtained. For example, for a defect detection type, an object image of a sample object that has actually completed defect detection of that type can be obtained as a sample reference object image, and a result image of the sample reference object image completing defect detection of that type can be obtained as a sample reference detection result image. In addition, for a selected sample target object image, other sample target object images can be used as sample reference object images for the selected sample target object image, and the label detection result image of the other sample target object image corresponding to the defect detection type can be used as the sample reference detection result image.

[0176] As mentioned above, after obtaining the sample target object image and its corresponding label detection result images of different defect detection types, and obtaining the sample reference object image and its sample reference detection result image corresponding to different defect detection types, the sample target object image features of the sample target object image, the sample reference object image features of the sample reference object image, and the sample result image features of the sample reference detection result image are further obtained through the general defect detection model. Specifically, the method of obtaining the target object image features of the target object image, the reference object image features of the reference object image, and the result image features of the reference detection result image through the general defect detection model in the above embodiment can be referred to for implementation, and no further details will be given here.

[0177] Furthermore, the preset query feature is obtained, and through the universal defect detection model, the sample target object image feature, the sample reference object image feature, the result image feature and the preset query feature are fused to obtain the first sample target fused image feature corresponding to the sample target object image feature, the second sample target fused image feature corresponding to the preset query feature, the third sample target fused image feature corresponding to the sample reference object image feature and the fourth sample target fused image feature corresponding to the sample result image feature. Specifically, the method of fusing the first target fused image feature corresponding to the target object image feature, the second target fused image feature corresponding to the preset query feature, the third target fused image feature corresponding to the reference object image feature and the fourth target fused image feature corresponding to the result image feature in the above embodiment can be referred to for implementation, and no further details will be given here.

[0178] In addition, through the general defect detection model, the first sample target fusion image feature, the second sample target fusion image feature, the third sample target fusion image feature and the fourth sample target fusion image feature are mapped to the same feature value scale to obtain the first updated sample target fusion image feature, the second updated sample target fusion image feature, the third updated sample target fusion image feature and the fourth updated sample target fusion image feature. The specific method of obtaining the first updated target fusion image feature, the second updated target fusion image feature, the third updated target fusion image feature and the fourth updated target fusion image feature in the above embodiment can be referred to and implemented accordingly, and will not be repeated here.

[0179] Afterwards, defect detection is performed through a universal defect detection model based on the first updated sample target fusion image features and the second updated sample target fusion image features to obtain a first sample target detection result image; and defect detection is performed through a universal defect detection model based on the third updated sample target fusion image features and the fourth updated sample target fusion image features to obtain a second sample target detection result image. The specific implementation can refer to the defect detection method based on the first updated target fusion image features and the second updated target fusion image features in the above embodiment, which will not be repeated here.

[0180] Next, a first loss is obtained based on the difference between the first sample target detection result image and the labeled detection result image corresponding to the defect detection type of the current sample reference detection result image. A second loss is also obtained based on the difference between the second sample target detection result image and the current sample reference detection result image. The types of the first and second losses obtained are not limited and can be selected by those skilled in the art based on actual needs. For example, both the first and second losses may employ mean squared error (MSE) loss.

[0181] Finally, the model parameters of the universal defect detection model are updated based on the first loss and the second loss until a preset update stop condition is met. The first loss and the second loss can be fused to obtain a fused loss, and the model parameters of the universal defect detection model are updated based on the fused loss. For example, the fused loss can be obtained by fusion of the first loss and the second loss according to their fusion weights, where the fusion weights of the first loss and the second loss can be set by those skilled in the art based on actual needs. For another example, the fused loss can be obtained by adding the first loss and the second loss.

[0182] In addition, the embodiments of the present application do not limit the configuration of the preset update stop conditions, which can be configured by technical personnel in this field according to actual needs. For example, it can be configured as: the number of parameter updates of the general defect detection model reaches a preset number, or it can be configured as: the general defect detection model converges, etc.

[0183] As can be seen from the above, the defect detection solution provided by the present application can realize multiple different types of defect detection through a single universal defect detection model. In particular, unlike related technologies, in addition to obtaining the target object image of the object to be detected, a reference object image corresponding to the target defect detection type of the object to be detected and its reference detection result image are also obtained, and the target object image features of the target object image, the reference object image features of the reference object image, and the result image features of the reference detection result image are respectively obtained through the universal defect detection model, and the above three image features are fused at the feature level to obtain the target fusion image features, so that the universal defect detection model can perceive the target defect detection type of the object to be detected through the target fusion image features, and accordingly, the universal defect detection model performs defect detection according to the target fusion image features to obtain a target detection result image that meets the target defect detection type. In this way, different types of defect detection are realized through a single universal defect detection model, which reduces the complexity of defect detection and simplifies the deployment and maintenance of the model. Since there is no need to manage multiple models of different defect detection types, the cost of defect detection can be effectively reduced. In addition, using a single universal defect detection model can obtain a unified feature representation and learn shared features between different quality inspection tasks, thereby improving the generalization and robustness of the model and better adapting to different defect detection needs. Moreover, when the universal defect detection model learns new knowledge on a defect detection task, this knowledge can be transferred to other defect detection tasks, realizing knowledge transfer and iterative improvement, thereby improving the overall defect detection performance.

[0184] Please refer to Figure 2 , the following describes the defect detection method provided by this application using electronic equipment as the execution subject, such as Figure 2 As shown, the process of the defect detection method can also be as follows:

[0185] In 210 , the electronic device obtains a target object image of an object to be detected.

[0186] Among them, the object to be inspected is used to refer to the object that needs to be inspected for defects. In actual scenarios, it can be a product obtained by industrial manufacturing, such as a circuit board, display panel, hardware accessories, etc.

[0187] In the embodiment of the present application, the electronic device first obtains an object image of the object to be inspected, which is recorded as a target object image, and uses the target object image to implement defect detection of a specific defect detection type on the object to be inspected.

[0188] In 220 , the electronic device determines a target defect detection type of the object to be detected, and obtains a reference object image and a reference detection result image corresponding to the target defect detection type.

[0189] In an embodiment of the present application, in addition to obtaining the target object image of the object to be detected, the electronic device further determines the target defect detection type of the object to be detected. The target defect detection type is used to indicate the type of defect detection that the detector of the object to be detected expects to perform on the object to be detected.

[0190] It should be noted that the embodiments of the present application pre-train a general defect detection model. This general defect detection model is configured through training to perform defect detection of a specific defect detection type based on an input object image and a task prompt example indicating a specific defect detection type, and to output a detection result image corresponding to the object image. The task prompt example consists of two parts: a reference object image and a reference detection result image obtained by performing defect detection of a specific defect detection type on the reference object image.

[0191] Accordingly, after determining the target defect detection type of the object to be inspected, the electronic device further acquires an object image for reference, recorded as the reference object image, and acquires a detection result image obtained by completing defect detection of the target defect detection type for the reference object image, recorded as the reference detection result image. In this way, the reference object image and the reference detection result image can be combined to represent the target defect detection type. The reference object corresponding to the reference object image and the object to be inspected can be of the same type or different types. In addition, the reference object image can be a defect detection image obtained by completing only the defect detection of the target defect detection type, or it can be a defect detection image obtained by completing defect detection including the target defect detection type and obtaining separate detection result images.

[0192] In 230 , the electronic device obtains target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image through the universal defect detection model.

[0193] The general defect detection model provided in this application consists of three parts, namely an encoder network, a task navigator network and a decoder network, wherein the encoder network is configured to map the input object image, the object image indicating a specific defect detection type and its detection result image to a feature space to obtain its hidden layer representation - image features; the task navigator network is configured to fuse the image features of the three to obtain a fused image feature, so that the decoder network can perceive the aforementioned specific defect detection type; the decoder network is configured to perform defect detection based on the fused image feature to obtain a detection result image that meets the aforementioned specific defect detection type, and the detection result image indicates the defects in the object image through the mask area. Subject to the constraints of realizing the functions of each configuration, the network structure of the above encoder network, task navigator network and decoder network can be set by those skilled in the art according to actual needs, and no restrictions are made here.

[0194] In an embodiment of the present application, the electronic device inputs the target object image, the reference object image, and the reference detection result image into the encoder network of the general defect detection model for feature mapping, and correspondingly obtains the target object image features of the target object image, the reference object image features of the reference object image, and the result image features of the reference detection result image.

[0195] In 240, the electronic device obtains a preset query feature, and uses a universal defect detection model to splice the preset query feature and the target object image feature to obtain a first spliced ​​feature, and performs self-attention enhancement on the first spliced ​​feature and then splits it into a first enhanced query feature and a first enhanced target object image feature.

[0196] The preset query feature can be regarded as a feature container, in which the feature value is initially configured to be 0 or a number close to 0, such as 0.01.

[0197] In an embodiment of the present application, the electronic device uses the task navigator network of the universal defect detection model to splice the preset query feature and the target object image feature to obtain a first spliced ​​feature, and then performs self-attention enhancement on the first spliced ​​feature. The enhanced first spliced ​​feature is recorded as the first enhanced spliced ​​feature. This process can be expressed as follows:

[0198]

[0199] C1=concat(F qi ,F ql );

[0200] Q=K=V=C1;

[0201]

[0202] Among them, C'1 represents the first enhanced splicing feature, C1 represents the first splicing feature, F qi represents the target object image feature, F ql Indicates the preset query feature.

[0203] As above, after performing self-attention enhancement on the first splicing feature to obtain the first enhanced splicing feature, the first enhanced splicing feature is further split through the task navigator network of the general defect detection model to obtain the first enhanced query feature and the first enhanced target object image feature accordingly.

[0204] It can be understood that the first splicing feature is obtained by splicing the preset query feature and the target object image feature in the channel dimension. Accordingly, after the first splicing feature is self-attention enhanced to obtain the first enhanced splicing feature, the first enhanced splicing feature is split according to the channel to obtain the enhanced preset query feature, that is, the first enhanced query feature, and the enhanced target object image feature, that is, the first enhanced target object image feature.

[0205] In 250, the electronic device uses a universal defect detection model to splice the reference object image feature and the result image feature to obtain a third spliced ​​feature, and performs self-attention enhancement on the third spliced ​​feature to split it into a first enhanced reference object image feature and a first enhanced result image feature.

[0206] In an embodiment of the present application, the electronic device uses the task navigator network of the universal defect detection model to splice the reference object image features and the result image features to obtain a third spliced ​​feature, and then performs self-attention enhancement on the third spliced ​​feature. The enhanced third spliced ​​feature is recorded as the third enhanced spliced ​​feature. This process can be expressed as follows:

[0207]

[0208] C3=concat(F pi ,F pl );

[0209] Q=K=V=C3;

[0210]

[0211] Among them, C'3 represents the third enhanced splicing feature, C3 represents the third splicing feature, F pi represents the reference object image feature, F pl Represents the resulting image features.

[0212] After performing self-attention enhancement on the third splicing feature to obtain the third enhanced splicing feature, the third enhanced splicing feature is further split through the task navigator network of the universal defect detection model to obtain the first enhanced reference object image feature and the first enhanced result image feature accordingly.

[0213] It can be understood that the third splicing feature is obtained by splicing the reference object image feature and the result image feature in the channel dimension. Accordingly, after the third splicing feature is self-attention enhanced to obtain the third enhanced splicing feature, the third enhanced splicing feature is split according to the channel to obtain the enhanced reference object image feature, that is, the first enhanced reference object image feature, and the enhanced result image feature, that is, the first enhanced result image feature.

[0214] In 260, the electronic device uses a universal defect detection model to splice the first enhanced target object image feature and the first enhanced reference object image feature to obtain a second spliced ​​feature, and performs self-attention enhancement on the second spliced ​​feature and then splits it into a second enhanced target object image feature and a second enhanced reference object image feature, and uses the second enhanced target object image feature as the first target fusion image feature and the second enhanced reference object image feature as the third target fusion image feature.

[0215] In an embodiment of the present application, the electronic device uses the task navigator network of the universal defect detection model to splice the first enhanced target object image feature and the first enhanced reference object image feature to obtain a second spliced ​​feature, and then performs self-attention enhancement on the second spliced ​​feature. The enhanced second spliced ​​feature is recorded as the second enhanced spliced ​​feature. This process can be expressed as follows:

[0216]

[0217]

[0218] Q=K=V=C2;

[0219]

[0220] Wherein, C'2 represents the second enhanced splicing feature, C2 represents the second splicing feature, represents the first enhanced target object image feature, represents the first enhanced reference object image feature.

[0221] After performing self-attention enhancement on the second splicing feature to obtain a second enhanced splicing feature, the second enhanced splicing feature is further split through the task navigator network of the universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature accordingly.

[0222] It can be understood that the second splicing feature is obtained by splicing the first enhanced target object image feature and the first enhanced reference object image feature in the channel dimension. Accordingly, after the second splicing feature is self-attention enhanced to obtain the second enhanced splicing feature, the second enhanced splicing feature is split according to the channel to obtain the first enhanced target object image feature after further enhancement, that is, the second enhanced target object image feature, and the first enhanced reference object image feature after further enhancement, that is, the second enhanced reference object image feature. Accordingly, the second enhanced target object image feature is used as the first target fusion image feature, and the second enhanced reference object image feature is used as the third target fusion image feature.

[0223] In 270, the electronic device uses a universal defect detection model to splice the first enhanced query feature and the first enhanced result image feature to obtain a fourth spliced ​​feature, and performs self-attention enhancement on the fourth spliced ​​feature and splits it into a second enhanced query feature and a second enhanced result image feature, and uses the second enhanced query feature as the second target fused image feature and the second enhanced result image feature as the fourth target fused image feature.

[0224] Through the task navigator network of the general defect detection model, the first enhanced query feature and the first enhanced result image feature are spliced ​​to obtain the fourth spliced ​​feature. Then, self-attention enhancement is performed on the fourth spliced ​​feature, and the enhanced fourth spliced ​​feature is recorded as the fourth enhanced spliced ​​feature. This process can be expressed as:

[0225]

[0226]

[0227] Q=K=V=C4;

[0228]

[0229] Wherein, C'4 represents the fourth enhanced splicing feature, C4 represents the fourth splicing feature, represents the first enhanced result image feature, Represents the first enhanced query feature.

[0230] After performing self-attention enhancement on the fourth splicing feature to obtain a fourth enhanced splicing feature, the fourth enhanced splicing feature is further split through a task navigator network of a universal defect detection model to correspondingly obtain a second enhanced query feature and a second enhanced result image feature.

[0231] It can be understood that the fourth splicing feature is obtained by splicing the first enhanced query feature and the first enhanced result image feature in the channel dimension. Accordingly, after the fourth splicing feature is self-attention enhanced to obtain the fourth enhanced splicing feature, the fourth enhanced splicing feature is split according to the channel to obtain the first enhanced query feature after further enhancement, that is, the second enhanced query feature, and the first enhanced result image feature after further enhancement, that is, the second enhanced result image feature. Accordingly, the second enhanced query feature is used as the second target fusion image feature, and the second enhanced result image feature is used as the fourth target fusion image feature.

[0232] In 280 , the electronic device maps the first target fused image feature and the second target fused image feature to the same feature value scale according to the third target fused image feature and the fourth target fused image feature to obtain a first updated target fused image feature and a second updated target fused image feature.

[0233] Among them, the electronic device maps the first target fusion image feature, the second target fusion image feature, the third target fusion image feature and the fourth target fusion image feature to the same eigenvalue scale through the task navigator network of the general defect detection model, and obtains the first updated target fusion image feature, the second updated target fusion image feature, the third updated target fusion image feature and the fourth updated target fusion image feature.

[0234] In 290 , the electronic device performs defect detection using a universal defect detection model based on the first updated target fused image features and the second updated target fused image features to obtain a target detection result image that meets the target defect detection type.

[0235] It is understandable that the third updated target fused image features and the fourth updated target fused image features primarily reflect the features of the reference object image and its reference detection result image, while the first updated target fused image features and the second updated target fused image features are fully fused as described above, enabling the model to perceive the target defect detection type of the object to be detected. Therefore, in the embodiments of the present application, the electronic device performs defect detection using the decoder network of the universal defect detection model based on the first updated target fused image features and the second updated target fused image features, obtaining a target detection result image that conforms to the target defect detection type.

[0236] To facilitate better implementation of the above defect detection method, the present application also provides a corresponding defect detection device. The meanings of the terms are the same as those in the above defect detection method. For specific implementation details, please refer to the description in the above method embodiment.

[0237] Please refer to Figure 3 , Figure 3This is a schematic diagram of the structure of a defect detection device provided in an embodiment of the present application. The defect detection device may include an object image acquisition module 310, a reference image acquisition module 320, a feature mapping module 330, and a feature fusion module 340, wherein:

[0238] The object image acquisition module 310 is used to acquire a target object image of the object to be detected;

[0239] The reference image acquisition module 320 is used to determine the target defect detection type of the object to be detected, and obtain a reference object image corresponding to the target defect detection type and a reference detection result image thereof;

[0240] A feature mapping module 330 is configured to obtain target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image through a universal defect detection model;

[0241] A feature fusion module 340 is configured to fuse target object image features, reference object image features, and result image features using a universal defect detection model to obtain target fused image features;

[0242] The defect detection module 350 is used to perform defect detection using a general defect detection model based on target fusion image features to obtain a target detection result image that meets the target defect detection type.

[0243] Optionally, in one embodiment, the feature fusion module 340 is used to obtain preset query features, and based on the preset query features, reference object image features and result image features, perform attention enhancement on the target object image features through a universal defect detection model to obtain a first target fused image feature.

[0244] Optionally, in one embodiment, the feature fusion module 340 is used to perform self-attention enhancement on preset query features and target object image features through a universal defect detection model to obtain a first enhanced query feature and a first enhanced target object image feature; perform self-attention enhancement on reference object image features and result image features through a universal defect detection model to obtain a first enhanced reference object image feature and a first enhanced result image feature; perform self-attention enhancement on the first enhanced target object image feature and the first enhanced reference object image feature through a universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature; and use the second enhanced target object image feature as the first target fusion image feature.

[0245] Optionally, in one embodiment, the feature fusion module 340 is used to splice the preset query feature and the target object image feature through a general defect detection model to obtain a first spliced ​​feature; perform self-attention enhancement on the first spliced ​​feature through the general defect detection model to obtain a first enhanced spliced ​​feature; and split the first enhanced spliced ​​feature through the general defect detection model to obtain a first enhanced query feature and a first enhanced target object image feature.

[0246] Optionally, in one embodiment, the feature fusion module 340 is used to splice the first enhanced target object image feature and the first enhanced reference object image feature through a universal defect detection model to obtain a second spliced ​​feature; perform self-attention enhancement on the second spliced ​​feature through the universal defect detection model to obtain a second enhanced spliced ​​feature; and split the second enhanced spliced ​​feature through the universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature.

[0247] Optionally, in one embodiment, the feature fusion module 340 is used to perform self-attention enhancement on the first enhanced query feature and the first enhanced result image feature through a general defect detection model to obtain a second enhanced query feature and a second enhanced result image feature; and to use the second enhanced query feature as the second target fusion image feature.

[0248] Optionally, in one embodiment, the defect detection module 350 is used to map the first target fusion image features and the second target fusion image features to the same eigenvalue scale through a universal defect detection model to obtain first updated target fusion image features and second updated target fusion image features; and perform defect detection through a universal defect detection model based on the first updated target fusion image features and the second updated target fusion image features to obtain a target detection result image that meets the target defect detection type.

[0249] Optionally, in one embodiment, the reference image acquisition module 320 is used to acquire a reference object image and a reference detection result image thereof, which have the same object type as the object to be detected and correspond to the target defect detection type.

[0250] Optionally, in one embodiment, the reference detection result image includes a color mask image. When the target defect detection type is defect semantic segmentation, the mask area of ​​a preset color in the color mask image is used to indicate the existing defect area. When the target defect detection type is defect instance segmentation, the mask areas of different colors in the color mask image are used to indicate different defect instances. When the target defect detection type is defect target detection, the mask areas of different colors in the color mask image are used to indicate different defect areas.

[0251] Optionally, in one embodiment, the defect detection module 350 is further configured to convert the target detection result image into a bounding box for indicating a defect area in the target object image.

[0252] Optionally, in one embodiment, the object image acquisition module 310 is further configured to acquire a sample target object image of the sample detection object and label detection result images corresponding to different defect detection types of the sample target object image;

[0253] The reference image acquisition module 320 is further used to acquire sample reference object images corresponding to different defect detection types and sample reference detection result images thereof;

[0254] The feature mapping module 330 is further configured to obtain, through the universal defect detection model, sample target object image features of the sample target object image, sample reference object image features of the sample reference object image, and sample result image features of the sample reference detection result image;

[0255] The feature fusion module 340 is further configured to obtain a preset query feature, and fuse the sample target object image feature, the sample reference object image feature, the result image feature, and the preset query feature through a universal defect detection model to obtain a first sample target fused image feature corresponding to the sample target object image feature, a second sample target fused image feature corresponding to the preset query feature, a third sample target fused image feature corresponding to the sample reference object image feature, and a fourth sample target fused image feature corresponding to the sample result image feature;

[0256] The defect detection module 350 is further configured to map the first sample target fusion image feature, the second sample target fusion image feature, the third sample target fusion image feature, and the fourth sample target fusion image feature to the same feature value scale through a universal defect detection model to obtain first updated sample target fusion image features, second updated sample target fusion image features, third updated sample target fusion image features, and fourth updated sample target fusion image features; and perform defect detection through the universal defect detection model based on the first updated sample target fusion image features and the second updated sample target fusion image features to obtain a first sample target detection result image; and perform defect detection through the universal defect detection model based on the third updated sample target fusion image features and the fourth updated sample target fusion image features to obtain a second sample target detection result image.

[0257] The defect detection device provided in the present application also includes a model training module, which is used to obtain a first loss based on a first sample target detection result image and a label detection result image; obtain a second loss based on a second sample target detection result image and a sample reference detection result image; and update the model parameters of the universal defect detection model based on the first loss and the second loss until a preset update stop condition is met.

[0258] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described again here.

[0259] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the processor is configured to execute the steps of the defect detection method provided in the above embodiment by calling a computer program stored in the memory.

[0260] Please refer to Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0261] The electronic device may include one or more processors 101, one or more computer-readable storage media memories 102, a power supply 103, an input unit 104, and other components. It will be understood by those skilled in the art that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0262] Processor 101 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and circuits. It executes software programs and / or modules stored in memory 102 and accesses data stored in memory 102 to perform various functions of the electronic device and process data. Optionally, processor 101 may include one or more processing cores. Alternatively, processor 101 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 101.

[0263] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0264] The electronic device also includes a power supply 103 for supplying power to various components. Optionally, the power supply 103 can be logically connected to the processor 101 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 103 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0265] The electronic device may further include an input unit 104, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0266] Although not shown, the electronic device may further include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 101 loads the executable code corresponding to one or more computer programs into the memory 102, and the processor 101 executes the steps of the defect detection method provided in this application, such as:

[0267] Acquire a target object image of an object to be detected;

[0268] Determine the target defect detection type of the object to be detected, and obtain a reference object image corresponding to the target defect detection type and a reference detection result image thereof;

[0269] Obtaining target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image through a universal defect detection model;

[0270] Through the universal defect detection model, the target object image features, the reference object image features and the result image features are fused to obtain the target fused image features;

[0271] According to the target fusion image features, defect detection is performed through a general defect detection model to obtain a target detection result image that meets the target defect detection type.

[0272] It should be noted that the electronic device provided in the embodiment of the present application and the defect detection method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0273] This application also provides a computer-readable storage medium having a computer program stored thereon. When the stored computer program is executed on a processor of an electronic device provided in an embodiment of this application, the processor of the electronic device implements the steps of the defect detection method provided in this application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0274] The present application also provides a computer program product, which includes a computer program. When the computer program is executed on the processor of the electronic device provided in the embodiment of the present application, the processor of the electronic device implements the steps in the defect detection method provided in the present application.

[0275] The above is a detailed introduction to the defect detection method, defect detection device, electronic device, computer-readable storage medium and computer program product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core ideas. At the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0276] It should be noted that when the above embodiments of this application are applied to specific products or technologies, the relevant user data is involved, and the user's permission or consent must be obtained, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A defect detection method, characterized in that: include: Acquire a target object image of an object to be detected; Determining a target defect detection type of the object to be detected, and obtaining a reference object image and a reference detection result image corresponding to the target defect detection type; Obtaining, by a universal defect detection model, target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image; By using the universal defect detection model, the target object image features, the reference object image features and the result image features are fused to obtain target fused image features; According to the target fusion image features, defect detection is performed using the universal defect detection model to obtain a target detection result image that meets the target defect detection type.

2. The defect detection method according to claim 1, characterized in that: The method of fusing the target object image features, the reference object image features, and the result image features through the universal defect detection model to obtain target fused image features includes: A preset query feature is obtained, and according to the preset query feature, the reference object image feature and the result image feature, attention enhancement is performed on the target object image feature through the universal defect detection model to obtain a first target fused image feature.

3. The defect detection method according to claim 2, characterized in that: The step of performing attention enhancement on the target object image feature by using the universal defect detection model according to the preset query feature, the reference object image feature, and the result image feature to obtain a first target fused image feature includes: Performing self-attention enhancement on the preset query feature and the target object image feature using the universal defect detection model to obtain a first enhanced query feature and a first enhanced target object image feature; Performing self-attention enhancement on the reference object image feature and the result image feature using the universal defect detection model to obtain a first enhanced reference object image feature and a first enhanced result image feature; performing self-attention enhancement on the first enhanced target object image feature and the first enhanced reference object image feature using the universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature; The second enhanced target object image feature is used as the first target fused image feature.

4. The defect detection method according to claim 3, characterized in that: The self-attention enhancement is performed on the preset query feature and the target object image feature by the universal defect detection model to obtain a first enhanced query feature and a first enhanced target object image feature, including: Using the universal defect detection model, the preset query feature and the target object image feature are spliced ​​to obtain a first spliced ​​feature; Performing self-attention enhancement on the first splicing feature using the universal defect detection model to obtain a first enhanced splicing feature; The first enhanced splicing feature is split using the universal defect detection model to obtain a first enhanced query feature and a first enhanced target object image feature.

5. The defect detection method according to claim 3, characterized in that: The step of performing self-attention enhancement on the first enhanced target object image feature and the first enhanced reference object image feature using the universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature comprises: splicing the first enhanced target object image feature and the first enhanced reference object image feature using the universal defect detection model to obtain a second splicing feature; Performing self-attention enhancement on the second splicing feature using the universal defect detection model to obtain a second enhanced splicing feature; The second enhanced splicing feature is split by the universal defect detection model to obtain a second enhanced target object image feature and a second enhanced reference object image feature.

6. The defect detection method according to claim 3, characterized in that: The method further comprises: fusing the target object image features, the reference object image features, and the result image features through the universal defect detection model to obtain target fused image features. Performing self-attention enhancement on the first enhanced query feature and the first enhanced result image feature using the universal defect detection model to obtain a second enhanced query feature and a second enhanced result image feature; The second enhanced query feature is used as a second target fused image feature.

7. The defect detection method according to claim 6, characterized in that: The method of performing defect detection by using the universal defect detection model based on the target fusion image features to obtain a target detection result image that meets the target defect detection type includes: Mapping the first target fused image feature and the second target fused image feature to the same feature value scale using the universal defect detection model to obtain a first updated target fused image feature and a second updated target fused image feature; Defect detection is performed using the universal defect detection model according to the first updated target fused image features and the second updated target fused image features to obtain a target detection result image that meets the target defect detection type.

8. The defect detection method according to any one of claims 1 to 7, characterized in that: The obtaining of a reference object image corresponding to the target defect detection type and a reference detection result image thereof includes: A reference object image and a reference detection result image thereof, which have the same object type as the object to be detected and correspond to the target defect detection type, are acquired.

9. The defect detection method according to any one of claims 1 to 7, characterized in that: The reference detection result image includes a color mask image. When the target defect detection type is defect semantic segmentation, the mask area of ​​preset color in the color mask image is used to indicate the existing defect area. When the target defect detection type is defect instance segmentation, the mask areas of different colors in the color mask image are used to indicate different defect instances. When the target defect detection type is defect target detection, the mask areas of different colors in the color mask image are used to indicate different defect areas.

10. The defect detection method according to claim 9, characterized in that: When the target defect detection type is defect target detection, after performing defect detection using the universal defect detection model based on the target fusion image features and obtaining a target detection result image that meets the target defect detection type, the method further includes: The target detection result image is converted into a bounding box for indicating a defect area in the target object image.

11. The defect detection method according to claim 6, characterized in that: Before acquiring the target object image of the object to be detected, the method further includes: Acquire a sample target object image of a sample detection object and label detection result images corresponding to different defect detection types of the sample target object image; Obtaining sample reference object images and sample reference detection result images corresponding to different defect detection types; Obtaining, by means of the universal defect detection model, sample target object image features of the sample target object image, sample reference object image features of the sample reference object image, and sample result image features of the sample reference detection result image; Obtaining a preset query feature, and fusing the sample target object image feature, the sample reference object image feature, the result image feature, and the preset query feature through the universal defect detection model to obtain a first sample target fused image feature corresponding to the sample target object image feature, a second sample target fused image feature corresponding to the preset query feature, a third sample target fused image feature corresponding to the sample reference object image feature, and a fourth sample target fused image feature corresponding to the sample result image feature; Mapping the first sample target fusion image feature, the second sample target fusion image feature, the third sample target fusion image feature, and the fourth sample target fusion image feature to the same feature value scale using the universal defect detection model to obtain a first updated sample target fusion image feature, a second updated sample target fusion image feature, a third updated sample target fusion image feature, and a fourth updated sample target fusion image feature; Performing defect detection using the universal defect detection model based on the first updated sample target fused image features and the second updated sample target fused image features to obtain a first sample target detection result image; performing defect detection using the universal defect detection model based on the third updated sample target fused image feature and the fourth updated sample target fused image feature to obtain a second sample target detection result image; Obtaining a first loss according to the first sample target detection result image and the label detection result image; Obtaining a second loss according to the second sample target detection result image and the sample reference detection result image; According to the first loss and the second loss, the model parameters of the universal defect detection model are updated until a preset update stop condition is met.

12. A defect detection device, characterized in that: include: An object image acquisition module is used to acquire a target object image of the object to be detected; A reference image acquisition module is used to determine the target defect detection type of the object to be detected, and to obtain a reference object image and a reference detection result image corresponding to the target defect detection type; a feature mapping module, configured to obtain target object image features of the target object image, reference object image features of the reference object image, and result image features of the reference detection result image through a universal defect detection model; a feature fusion module, configured to fuse the target object image features, the reference object image features, and the result image features through the universal defect detection model to obtain target fused image features; The defect detection module is used to perform defect detection through the universal defect detection model according to the target fusion image features to obtain a target detection result image that meets the target defect detection type.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory to implement the steps of the defect detection method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being executed by a processor to implement the steps in the defect detection method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the defect detection method according to any one of claims 1 to 11 are implemented.

Images