Defect detection method, device, equipment and storage medium

By using the position enhancement network in connector defect detection to process local features and comparing similarity with reference features, the problem of insufficient sample number in the prior art is solved, the accuracy and applicability of the detection are improved, and the High-precision defect detection.

CN117197134BActive Publication Date: 2024-11-01TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202311463360.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-06
Publication Date
2024-11-01
Estimated Expiration
2043-11-06

AI Technical Summary

Technical Problem

The existing deep learning-based object detection method is difficult to apply to actual scenarios due to insufficient sample size in connector defect detection, and the model accuracy is not high, so it cannot meet the requirements of high-precision quality inspection.

Method used

The position enhancement network is used to perform position enhancement processing on the local features in the image to be detected, and the similarity comparison is performed in combination with the pre-configured reference features to determine whether there are defects in the image block, and the detection accuracy is improved through feature comparison.

Benefits of technology

It improves the accuracy and scope of application of defect detection, reduces the difficulty of use in actual scenarios, and can generate high-precision defect detection results more quickly.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117197134B_ABST
    Figure CN117197134B_ABST
Patent Text Reader

Abstract

The present application discloses a defect detection method, device, equipment and storage medium, which can be applied to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios. The method uses a framework based on anomaly detection to realize defect detection. In the process of defect detection, location information is used for feature enhancement to improve the accuracy of defect detection, including the use of a location-sensitive feature extraction layer to enhance the ability to extract location information, and the use of distance based on location information as a basis for feature comparison when establishing a reference feature set, so that the feature comparison process is more focused on local areas. In addition, the method also generates pseudo-labels based on the prediction results of the initial anomaly detection model, and then uses the pseudo-labels to train the anomaly segmentation model, thereby achieving domain adaptability fine-tuning of the backbone network features, thereby improving the backbone network's feature extraction capabilities in the target device image field, and thus improving the accuracy of defect detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to the field of industrial quality inspection technology, and provides a defect detection method, device, equipment and storage medium. Background Art

[0002] The camera module is the basic visual component for realizing environmental perception in intelligent machines. It is widely used in common medical, automated teller machine (ATM), roadside camera module, home camera module and mobile phone camera module. Common camera modules are assembled from multiple different discrete components. The external light source generates an electrical signal on the complementary metal oxide semiconductor (CMOS) photosensitive element through the lens, and transmits it to the image processing center through the circuit with a specific transmission protocol. The connector in the circuit is the medium connecting the CMOS photosensitive element and the image processing center. Its quality directly affects the transmission of the electrical signal. However, since the connector needs to be soldered during the assembly process, it is easy to cause various defects such as tin overflow and pin bending. Such defects will directly affect the use of the camera module. Therefore, it is particularly important to inspect the connector before leaving the factory.

[0003] At present, in order to meet the large-scale quality inspection demand for camera modules, manual quality inspection is gradually being replaced by artificial intelligence (AI) quality inspection systems. However, when common AI quality inspection systems perform defect detection on connectors, they often use target detection methods based on deep learning. This method classifies all defects according to the differences in appearance characteristics, classifies defects with similar appearance characteristics into the same category, and then collects corresponding image samples for each category of defects to train the deep learning model. However, in order to train a relatively robust deep learning model, it is necessary to collect enough image samples for each category, usually more than 1,000.

[0004] However, in actual scenarios, due to the limitations of the manufacturer's yield control and the optimization and upgrading of production conditions, the number of image samples of relevant defects is usually very small, and because there are usually multiple defect categories distributed on the connector, there are even fewer image samples for a specific category. It is impossible to collect enough training data for various categories in a short period of time for large-scale deep learning model training. Therefore, the target detection method based on deep learning is difficult to apply to actual scenarios and is not very practical. If training is directly based on a small number of image samples, the accuracy of the obtained deep learning model is not high and cannot meet the high-precision requirements of connector quality inspection. Summary of the invention

[0005] The embodiments of the present application provide a defect detection method, apparatus, device and storage medium for improving quality inspection accuracy.

[0006] In one aspect, a defect detection method is provided, the method comprising:

[0007] Receiving an image to be inspected containing a target device, and performing feature extraction on each image block contained in the image to be inspected to obtain corresponding local features;

[0008] Through the trained position enhancement network, based on the position information of each image block in the image to be detected, the corresponding local features are respectively subjected to position enhancement processing to obtain corresponding enhanced features containing position information;

[0009] Based on the similarity between each enhanced feature and a pre-configured reference feature, respectively determine the abnormality assessment value corresponding to each of the image blocks; wherein each reference feature is extracted based on a defect-free image, and each reference feature includes position information of a corresponding image block, and the defect-free image includes a defect-free target device;

[0010] Based on the obtained abnormality evaluation values, a defect detection result of the image to be inspected is determined.

[0011] In one aspect, a defect detection device is provided, the device comprising:

[0012] A feature extraction unit is used to receive an image to be detected containing a target device, and perform feature extraction on each image block contained in the image to be detected to obtain corresponding local features;

[0013] A position enhancement unit is used to perform position enhancement processing on corresponding local features based on the position information of each image block in the image to be detected through a trained position enhancement network, so as to obtain corresponding enhanced features containing position information;

[0014] an abnormality detection unit, configured to determine an abnormality assessment value corresponding to each image block based on a similarity between each enhanced feature and a pre-configured reference feature; wherein each reference feature is extracted based on a defect-free image, and each reference feature includes position information of a corresponding image block, and the defect-free image includes a defect-free target device;

[0015] The result output unit is used to determine the defect detection result of the image to be detected based on the obtained abnormality evaluation values.

[0016] On the one hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0017] In one aspect, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0018] In one aspect, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the steps of any of the above methods.

[0019] In the embodiment of the present application, after feature extraction is performed on each image block in the image to be inspected containing the target device, the trained position enhancement network is used to perform position enhancement on the corresponding local features according to the position information of each image block to obtain the corresponding enhanced features, and then determine whether each image block has defects according to the similarity between the enhanced features and the reference features, and finally generate the defect detection results of the image to be inspected. Since each reference feature is extracted from a defect-free image of a device of the same type and is a normal defect-free feature, by comparing the similarity with the normal feature, it is possible to accurately and conveniently determine whether the image block in the image to be inspected has defects. At the same time, since the acquisition difficulty of defect-free images in actual scenes is relatively low, a large number of defect-free images can be easily acquired to extract reference features, which reduces the difficulty of use in actual scenes and improves the scope of application and practicality of defect detection.

[0020] In addition, in an embodiment of the present application, after extracting the local features of the image block, the position enhancement will be performed on the image block based on the position information to obtain enhanced features. Similarly, each reference feature is also a feature containing the position after position enhancement processing. When comparing with the reference feature, it will be targeted at a specific image block area based on the position information, and the feature comparison will be limited to a certain image space range, thereby improving the accuracy of the feature comparison and thereby improving the accuracy of the defect detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0022] Figure 1 It is a structural diagram of a camera module;

[0023] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0024] Figure 3 A flowchart of the training process provided in the embodiment of the present application;

[0025] Figure 4A to Figure 4C A schematic diagram of the process of initial feature extraction provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the marking process provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the domain adaptation principle provided in an embodiment of the present application;

[0028] Fig. 7A and Figure 7B A schematic diagram of the training process of the abnormal segmentation model provided in the embodiment of the present application;

[0029] Fig. 8A and Figure 8B A schematic diagram of another training process of the abnormal segmentation model provided in an embodiment of the present application;

[0030] Figure 9A to Figure 9C A schematic diagram of a feature set reduction process provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of a defect detection method according to an embodiment of the present invention;

[0032] Fig.11 A schematic diagram of the image registration and cutout process provided in an embodiment of the present application;

[0033] Fig. 12A and Fig. 12B A schematic diagram of an image feature extraction process provided in an embodiment of the present application;

[0034] Fig.13 A schematic diagram of the structure of a first feature extraction network provided in an embodiment of the present application;

[0035] Figure 14A to Figure 14C A schematic diagram of a position enhancement process provided in an embodiment of the present application;

[0036] Fig.15 A schematic diagram of a defect detection process provided by an embodiment of the present application;

[0037] Fig.16A schematic diagram of the structure of a defect detection device provided in an embodiment of the present application;

[0038] Fig.17 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0039] Fig.18 A schematic diagram of the composition structure of another computer device to which an embodiment of the present application is applied. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0041] It is understandable that when the embodiments of the present application are applied to specific products or technologies, relevant licenses or consents are required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0042] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0043] Camera module: Generally speaking, a camera module is assembled from multiple different discrete components, see Figure 1 As shown in the figure, the camera module includes a connector, a flexible printed circuit (FPC) and lens-related components, such as a lens barrel, a lens, a lens mount, etc. The external light source generates an electrical signal on the CMOS photosensitive element through the lens, and transmits it to the image processing center through the circuit. The quality of the connector, as a medium, directly affects the transmission of the electrical signal.

[0044] Defect detection: refers to the process of detecting whether an object in an image has defects and the location of the defects given an image. The defect detection object of the embodiment of the present application is the target device, which can be any device with a fixed structure and shape, such as a connector or circuit board of a camera module, etc. Of course, it can also be other possible devices. The embodiment of the present application does not limit the target device. Taking the connector of the camera module as an example, for the defect detection of the connector, it usually refers to the fixed-point image shooting of the connector on the generation line, and then detecting whether the connector has defects, such as tin overflow, pin bending, etc., based on the captured image, and locating the location of these defects.

[0045] Anomaly Detection Algorithm: Anomaly detection is a technique for identifying abnormal situations and mining non-logical data. In computer vision, it is usually used to identify abnormal situations in images. For defect detection scenarios, it can be understood as detecting defects in connectors in images. Anomaly detection algorithms can usually be performed by comparing with reference data sets. In the embodiment of the present application, a feature data set containing defect-free devices of the same type as the target device can be created for feature comparison in actual applications.

[0046] Anomaly segmentation algorithm: The anomaly segmentation algorithm can also be used to identify abnormal situations, but the difference from anomaly segmentation is that the anomaly detection algorithm only has normal samples during the training process and is ultimately used to detect abnormal samples, while the anomaly segmentation algorithm can have both normal and abnormal samples during the training process and is used to detect abnormal pixels contained in the input samples.

[0047] The embodiments of the present application relate to artificial intelligence and machine learning (ML) technology, and are mainly designed based on machine learning and computer vision (CV) technology in artificial intelligence.

[0048] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0049] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0050] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0051] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0052] Artificial Neural Network (ANN) abstracts the human brain neural network from the perspective of information processing, establishes a simple model, and forms different networks according to different connection methods. A neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called an activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight, which is equivalent to the memory of the artificial neural network. The output of the network varies depending on the network connection method, weight value and activation function. The network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0053] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0054] The solution provided in the embodiment of the present application mainly involves the application of artificial intelligence in industrial quality inspection, and specifically involves defect detection scenarios for devices with fixed connection structures and fixed shapes, such as connectors of camera modules, circuit boards and other devices. When performing defect detection on devices, it is necessary to adopt an artificial neural network model based on deep learning, that is, the embodiment of the present application combines machine learning and methods related to the anomaly detection architecture to obtain an anomaly detection model for defect detection based on images actually collected. The feature extraction network in the anomaly detection model is obtained through domain adaptive training based on machine learning methods, and the reference features of the anomaly detection model are extracted based on defect-free images. Then, in actual scenarios, the features of the actual image can be extracted through the feature extraction network of the anomaly detection model, and compared with the reference features to achieve the purpose of defect detection on the target device.

[0055] Specifically, the defect detection process in the embodiment of the present application can be divided into two parts, including a training part and an application part. Among them, the training part involves the technical field of machine learning. In the training part, the embodiment of the present application adopts a staged training method. First, in the first stage, a normal feature set is obtained by extracting a defect-free image, and the corresponding sample features are extracted in the same way for the sample image, and then the normal feature set is compared with the sample features to annotate the sample image as a training label. In the second stage, the training label is used to train the abnormal segmentation model, and the model parameters are continuously adjusted through the optimization algorithm until the model converges and enters the third stage, and the feature extraction is performed again for the defect-free image to obtain the final reference feature set; the application part is used to use the final reference feature set for defect detection and output the defect detection results of the target device. In addition, it should be noted that the artificial neural network model in the embodiment of the present application can be either online training or offline training, which is not specifically limited here. In this article, offline training is used as an example for illustration.

[0056] The technical ideas of the embodiments of the present application are briefly described below.

[0057] In the related art, the defect detection method of the connector is mainly a target detection method based on deep learning. This method needs to first group all defects according to the difference in appearance features, group defects with similar appearance features into one group, and then collect relevant sample images for each group of defects. However, in order to train a relatively robust deep learning model, it is necessary to collect enough image samples for each category, usually more than 1,000. In addition, in the selection of deep models, considering that some defects have limitations such as size and area, a detection model based on instance segmentation is usually used, which can output a mask for each type of defect. By analyzing the size, shape and area of ​​the mask, some minor acceptable defects can be filtered to avoid misidentification. However, in actual scenarios, the number of defect samples related to connectors is usually small, and it is impossible to collect enough data for large-scale deep learning model training in a short time. At the same time, there are usually many defects distributed on the connector, and there are even fewer samples for a specific defect type. Therefore, the target detection method based on deep learning is difficult to apply to actual scenarios and is not very practical. If the training is directly based on a small number of image samples, the accuracy of the obtained deep learning model is not high, and it cannot meet the high-precision requirements of connector quality inspection.

[0058] Based on this, the embodiment of the present application provides a defect detection method, which extracts features from each image block in the image to be detected that contains the target device, and then performs position enhancement on the corresponding local features according to the position information of each image block to obtain the corresponding enhanced features, and then determines whether each image block has defects according to the similarity between the enhanced features and the reference features, and finally generates the defect detection results of the image to be detected. Since each reference feature is extracted from a defect-free image of a device of the same type and is a normal feature without defects, by comparing the similarity with the normal features, it is possible to accurately and conveniently determine whether the image block in the image to be detected has defects. At the same time, since it is relatively easy to collect defect-free images in actual scenes, it is possible to easily collect a large number of defect-free images to extract reference features, which reduces the difficulty of use in actual scenes and improves the scope of application and practicality of defect detection.

[0059] In addition, in an embodiment of the present application, after extracting the local features of the image block, the position enhancement will be performed on the image block based on the position information to obtain enhanced features. Similarly, each reference feature is also a feature containing the position after position enhancement processing. When comparing with the reference feature, it will be targeted at a specific image block area based on the position information, and the feature comparison will be limited to a certain image space range, thereby improving the accuracy of the feature comparison and thereby improving the accuracy of the defect detection results.

[0060] Moreover, in the embodiment of the present application, during the training phase, the feature representations of all image blocks on the defect-free image are stored in a feature memory bank, and then a coreset selection method is used to construct a compact normal feature set. For any given test image, the features are extracted in the same way and compared with the normal feature set. The areas with large differences will be determined as abnormal areas. Then, a pseudo-label is constructed based on the results of this abnormality detection and a supervised abnormality segmentation model is trained. The coreset-based abnormality detection model is reconstructed with the network parameters of the model as the final detection model. Since the original backbone network is pre-trained on a common data set, the above-mentioned supervised training method can reduce the field differences in the target device image field, thereby enhancing the adaptability of the features and improving the ability to discover abnormal features, so as to improve the accuracy of defect detection. At the same time, considering that in an actual industrial environment, the images taken of different parts have only a small offset in spatial position, and the types and forms of defects appearing in different positions are also different, because in the training stage of the embodiment of the present application, a position enhancement network is introduced for the coreset-based anomaly detection method to inject position information into the backbone network, so that the network is more sensitive to position differences. For example, a position-sensitive feature extraction layer can be introduced in the last layer of the backbone network, so that the extracted features are sensitive to position. At the same time, the spatial coordinate distance is also introduced as a weight in the feature comparison link of the coreset reduction and the final reasoning stage, so that the model pays more attention to anomalies in similar positions without being affected by features in distant irrelevant areas. Then, the position information in the training stage and the application stage can be coordinated to limit the feature comparison to a specific image area, thereby improving the accuracy of the feature comparison, thereby improving the accuracy of the defect detection results.

[0061] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0062] The solution provided in the embodiments of the present application can be applied to device defect detection based on images, such as defect detection of connectors or other circuit boards inside camera modules. The solution can be applied as a basic technology in various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Figure 2 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, an image acquisition device 101 and a defect detection device 102 may be included.

[0063] The image acquisition device 101 is used as a front-end device to acquire images of the target device to be inspected, which may include but is not limited to mobile phones, tablet computers (PADs), laptop computers, desktop computers, smart vehicle-mounted devices, smart voice interaction devices, smart home appliances, smart wearable devices, aircraft, etc. It may also be a camera, camcorder or other device set on the inspection line specifically for shooting images to be inspected.

[0064] The defect detection device 102 is used to perform a defect detection process based on the image to be detected acquired by the image acquisition device 101. It can be any device with the processing capabilities required for the defect detection process, such as a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, an intelligent vehicle-mounted device, an intelligent wearable device, etc., or a server. When it is a server, it can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDNs), as well as big data and artificial intelligence platforms, but is not limited to these.

[0065] In actual application, the image acquisition device 101 can send the acquired image to be inspected containing the target device to the defect detection device 102, and the defect detection device 102 uses the defect detection method of the embodiment of the present application to perform defect detection on the image to be inspected to determine whether the target device in the image to be inspected has defects. Among them, the defect detection device 102 can include one or more processors 1021, a memory 1022, and an interactive I / O interface 1023, etc. In addition, the defect detection device 102 can also be configured with a database 1024, which can be used to store reference feature sets and model parameters related to the model. Among them, the memory of the defect detection device 102 and the image acquisition device 101 can also store the program instructions required to be executed in the defect detection method provided in the embodiment of the present application, and these program instructions can be used to implement the defect detection process provided in the embodiment of the present application when executed by the processor.

[0066] It should be noted that, in actual applications, when the processing capability of the image acquisition device is sufficient, the defect detection device 102 and the image acquisition device 101 can be implemented by the same device, and the embodiments of the present application do not impose any restrictions on this.

[0067] In the embodiment of the present application, the image acquisition device 101 and the defect detection device 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network, for example, the wireless network can be a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course, it can be other possible networks, which are not limited in the embodiment of the present application.

[0068] Below, in combination with the application scenarios described above, the method provided by the exemplary embodiment of the present application is described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0069] Since the defect detection method provided in the present application is based on a reference feature set and an anomaly detection model after training, before introducing the actual defect detection process, the model training process is first introduced here.

[0070] See also Figure 3 As shown, it is a flow chart of the training process provided by the embodiment of the present application. The method can be executed by a computer device, which can be, for example, Figure 2 The image acquisition device or defect detection device shown in the figure, the specific implementation process of the method is as follows:

[0071] Step 301: In the first stage, the anomaly detection model is trained, that is, the feature extraction is performed on each image block contained in each defect-free image through the second feature extraction network to obtain an initial feature set, which includes the initial features corresponding to each image block of each defect-free image.

[0072] In the embodiment of the present application, a defect detection method based on anomaly detection algorithm is adopted in the actual application stage of the model. The anomaly detection algorithm is a method for determining anomalies by comparing normal data sets. The anomaly detection algorithm is an unsupervised algorithm that does not require the collection of defect data sets, only normal data sets are required, and in some cases, the anomaly detection algorithm only requires a small amount of normal data sets, avoiding the problem of low model accuracy caused by the difficulty in collecting low-frequency defect data sets, and can locate anomalies well when the defect data set is insufficient. At the same time, this type of method does not need to characterize defects of each category and form, avoiding the problem of new categories or new features not being able to be detected, and also avoiding the identification of difficult-to-distinguish defect categories.

[0073] Therefore, the training process of the anomaly detection model can be understood as extracting normal features from the normal data set through the feature extraction network of the anomaly detection model. For example, for the defect detection of the connector of the camera module, the normal data set is the defect-free image of the connector, that is, the collected image of the connector without defects is used as the normal sample image for feature extraction.

[0074] See also Figure 4A As shown, it is a schematic diagram of the process of initial feature extraction provided by the embodiment of the present application. Taking the target device as a connector as an example, multiple defect-free images corresponding to it can be collected, each defect-free image is an image of the connector that has been verified to have no defects, or an image in which the defects are within the qualified range of the connector, and these defect-free images are input into the backbone network of the anomaly detection model, such as Figure 4A As shown, the backbone network is the second feature extraction network, which is used to extract corresponding image features from each defect-free image. Since each image is a defect-free image, the features extracted thereby can be used as the initial features of the anomaly detection algorithm to form an initial feature set.

[0075] Among them, for any defect-free image, it may be composed of multiple image blocks, and the second feature extraction network can extract the initial features corresponding to each image block respectively, that is, for a defect-free image, it may correspond to multiple initial features in the initial feature set, and each initial feature may be a feature corresponding to an image block included in the defect-free image. Figure 4A As shown in , the defect-free image can be divided into image blocks 1~N, corresponding to the initial features f1~fN respectively.

[0076] In practical applications, the second feature extraction network can adopt any feature extraction network based on the convolutional feature extraction module, such as ResNet50 or a derivative network of the same type, or a basic network based on transformer (a model architecture applied to natural language processing), such as ViT, etc., so as to better extract contextual information in the image.

[0077] In addition, considering that a device may be further subdivided into different types, and different types may have different morphologies or connection structures, there may be more or less differences between them. Therefore, targeted defect detection can be performed for these different types. For example, there are many types of connectors, and targeted defect detection can be performed for different types of connectors. Below, taking the connector as an example, the following method can be used to perform targeted defect detection.

[0078] In one possible implementation, different defect detection scenarios can be divided according to the type of connector. In each defect detection scenario, defect detection can be performed on only one type of connector. Then, corresponding defect-free images can be collected for different types of connectors and feature extraction can be performed to obtain a corresponding initial feature set.

[0079] See also Figure 4B As shown, it is a schematic diagram of extracting initial features for different types of connectors. For different connector types 1~M, a set of related defect-free images is collected for type 1, and features are extracted for each defect-free image to obtain the initial feature set. Similarly, a set of related defect-free images is collected for type 2, and features are extracted for each defect-free image to obtain the initial feature set, and so on.

[0080] In one possible implementation, the type of connector may not be distinguished during feature extraction, but type-related information may be added to the extracted features to distinguish the features of different types of connectors. In this way, when storing features, there is no need to distinguish between different types, and all initial features can be stored in a unified manner, reducing the difficulty of feature management.

[0081] See also Figure 4C As shown in FIG. 1 , it is a schematic diagram of extracting initial features for different types of connectors. For each type of defect-free image, the corresponding initial features can be extracted by the second feature extraction network, and then the corresponding type identification features can be added to the initial features as the final initial features. Figure 4C As shown, the type identification feature can be spliced ​​after the initial feature, and of course it can also be added at other positions, such as adding the type identification feature before the initial feature. The embodiment of the present application does not limit this.

[0082] Step 302: In the second stage, the anomaly segmentation model is trained, that is, the anomaly segmentation model is iteratively trained using sample images and training labels annotated on the sample images based on the initial feature set until an iteration termination condition is met.

[0083] In an embodiment of the present application, the network parameters in the above-mentioned second feature extraction network are unadjusted parameters, which are equivalent to common parameters that can be applied to all image fields, not just the image defect detection field where the target device is located. The effect of feature extraction in the image defect detection field where the target device is located is also equal to that in other fields. Therefore, it is impossible to extract more useful features for the target device image defect detection field in a targeted manner. Therefore, in order to narrow the domain differences of the network parameters of the feature extraction network in specific image defect scenarios, in the second stage, the parameters of the feature extraction network can be domain adapted in the target device image field.

[0084] Specifically, the domain adaptation in the second stage is achieved by training the anomaly segmentation model. The anomaly segmentation model is trained in a supervised manner. Therefore, before training the anomaly segmentation model, it is necessary to collect enough sample images and label the sample images to obtain the training labels of each sample image and participate in the subsequent training process. The training in the second stage may include the following steps:

[0085] Step 3021: Obtain multiple sample images.

[0086] The sample image can be any image containing the target device, and can be a sample image without defects or a sample image with defects, and there is no restriction in the second stage of training. In practical applications, the sample image can be acquired by setting an image acquisition device on the production line, or by acquiring the sample image from the network, and the embodiment of the present application does not limit this.

[0087] Each sample image corresponds to a training label, which is used to indicate the probability that each image block in the corresponding sample image is a defective image block. The training label can be obtained by annotating in the following ways:

[0088] In a possible implementation, the training labels can be obtained by manual annotation. Considering that there may be errors and omissions in human annotation, the manual annotation results of multiple people can be obtained, and then the final training labels can be obtained by combining the annotation results of multiple people.

[0089] In a possible implementation, the training label can be obtained based on the qualification label for the target device on the production line. When collecting sample images of the production line, generally speaking, a sample image can be an image of a target device, and often a target device will be labeled whether it is qualified or not. The sample image can be annotated according to the qualification label to obtain the corresponding training label. For example, taking the connector as an example, when the connector is labeled as unqualified, it can be annotated with a training label containing a defective image block. However, it should be noted that if the qualification label only marks the existence of a defect but does not indicate the location of the defect, then the training label is only that the entire sample image is abnormal, but the location of the abnormality cannot be specified; and if the qualification label not only marks the existence of a defect but also marks the location of the defect, then the annotated training label can also be only the specific image block with the defect.

[0090] In a possible implementation, the initial feature set obtained above is a normal feature without defects. By comparing it with the normal feature set, it can be determined whether the sample image has defects. Therefore, the initial feature set can also be used to annotate the sample image to obtain corresponding training labels. Figure 3 This is specifically illustrated in the following, namely:

[0091] Step 3022: Based on the initial feature set, label multiple sample images respectively to obtain corresponding training labels.

[0092] Specifically, for each sample image, the aforementioned second feature extraction network can be used to extract features from each image block contained therein, and the sample features of each sample image can be correspondingly extracted, and the sample features correspond to the initial features one by one. Figure 5 As shown, it is a schematic diagram of the labeling process provided in an embodiment of the present application, wherein, for a defect-free image, feature extraction is performed on each image block 1~N to obtain the corresponding initial features f1~fN, and for a sample image, the same method can be used to obtain the sample features p1~pN corresponding to each image block 1~N in the sample image, and then the probability that the image block corresponding to the sample feature is a defective image block can be determined by comparing the initial feature with the sample feature.

[0093] Exemplarily, for a sample image P, the second feature extraction network is used to extract features from each image block in a sample image, and the corresponding sample features p1~pN are obtained. Then, based on the similarity between each sample feature obtained and the initial feature set, the sub-label of the corresponding image block can be determined, and each sub-label is used to indicate the probability that the corresponding image block is a defective image block. For example, for the sample feature p1, the similarity between p1 and all the initial features in the initial feature set can be calculated, and then based on the obtained similarities, the defect probability of the image block corresponding to p1 can be obtained, and then the image block can be marked with a corresponding sub-label. For example, when the calculated defect probability is high, it can be regarded as a defect, and a sub-label representing that the image block corresponding to p1 is a defective image block can be marked. Otherwise, when the calculated defect probability is low, a sub-label representing that the image block corresponding to p1 is a non-defective image block can be marked. After all image blocks are labeled, the training label of the sample image P can be obtained based on the obtained sub-labels.

[0094] In a possible implementation, 0 or 1 can be used as the value of the sub-label. When it is 0, it indicates that the corresponding image block is a defective image block, and when it is 1, it indicates that the corresponding image block is a non-defective image block; or, when it is 1, it indicates that the corresponding image block is a defective image block, and when it is 0, it indicates that the corresponding image block is a non-defective image block. Finally, the following can be obtained: Figure 5 The training label shown is a schematic diagram of the training label, which converts the sample image into the corresponding binary feature map through the above process.

[0095] In a possible implementation manner, the defect probability obtained above may be directly used as a sub-label, and the training label finally formed includes the defect probability corresponding to each image block.

[0096] Step 3023: Based on multiple sample images and their corresponding training labels, iteratively train the anomaly segmentation model until an iteration termination condition is met; wherein the first feature extraction network and the second feature extraction network included in the anomaly segmentation model share network parameters.

[0097] In order to improve the domain adaptation of the backbone network in the image field where the target device is located, the embodiment of the present application uses an adaptive threshold segmentation method to extract the area with a higher anomaly score in the sample image, and obtain a binary anomaly map as a pseudo label, that is, the result obtained by comparing the above initial feature set with the sample feature is used as the initial anomaly segmentation map, and the anomaly segmentation model is trained based on this, such as Figure 6As shown in the figure, the entire backbone network is fine-tuned end-to-end using pseudo labels, thereby achieving domain adaptation from ImageNet (the dataset used for backbone network feature extraction) to the image data in the field of the target device (such as camera module connector image data). Finally, based on the fine-tuned backbone network, the anomaly detection model is retrained, and the anomaly detection model generated in this step is used as the final inference model and applied to the actual defect detection process.

[0098] It should be noted that in addition to the training labels mentioned above, in actual applications, soft labels can also be used as training labels. For example, the abnormal score map can be directly normalized and then used as a training label. The embodiment of the present application does not limit this, as long as it can provide a good optimization target for the domain adaptation stage.

[0099] Next, the training process of the anomaly segmentation model when the backbone network is in different situations is introduced.

[0100] In one possible implementation, see Fig. 7A and Figure 7B As shown, Fig. 7A A schematic diagram of the training process of the abnormal segmentation model provided in the embodiment of the present application, Figure 7B A structural schematic diagram of an abnormal segmentation model provided for an embodiment of the present application, the abnormal segmentation model includes a first feature extraction network and a defect prediction network, the first feature extraction network is used to implement the feature extraction process, and it shares network parameters with the second feature extraction network, and the meaning of sharing is that the network structure of the first feature extraction network and the second feature extraction network are the same, and the network parameters contained in the network are also the same; the defect prediction network performs forward prediction based on the extracted features to obtain a final defect prediction result, and the defect prediction result is used to characterize whether each image block of the sample image is a defective image block.

[0101] Since each training process of the anomaly segmentation model is similar, we will take one of them as an example here. Fig. 7A As shown, the following steps may be included:

[0102] Step 30231A: For the multiple sample images input this time, the first feature extraction network in the abnormal segmentation model used this time is used to extract features from each image block of each sample image to obtain corresponding sample features.

[0103] In practical applications, the multiple sample images input in each iteration may be all sample images, or may be some sample images extracted from all sample images, and may be set according to specific requirements in practical applications.

[0104] Specifically, similar to the above-mentioned second feature extraction network, the first feature extraction network can also perform feature extraction on each image block in each sample image to obtain sample features of each image block in each sample image.

[0105] Step 30232A: Based on the sample features corresponding to each image block in each sample image, obtain the defect detection result of each sample image.

[0106] See also Figure 7B As shown, the defect prediction network can perform forward prediction based on the sample features corresponding to each image block in each sample image to obtain the defect detection result of each sample image. Among them, the defect prediction network can adopt any prediction layer structure that can map the sample features to a defect probability, such as a fully connected layer, a softmax layer, etc., and the embodiment of the present application does not limit this.

[0107] In a possible implementation, the defect detection result may be characterized in a manner similar to that of the training label, that is, for a sample image, it may be characterized by the defect probability values ​​of the various image blocks it includes, and the obtained defect detection result may include the defect probability values ​​of the various image blocks.

[0108] In a possible implementation manner, the defect detection result may be characterized by the defect quantity and defect position, that is, image blocks with defects in the sample image are marked, or it may also be called a defect mask of the sample image.

[0109] Step 30233A: Based on the defect detection results of each sample image and the corresponding training labels, obtain the model loss value of the anomaly segmentation model.

[0110] See also Figure 7B As shown, the training label can be obtained by using a second feature extraction network to annotate the sample features extracted from the sample image with the initial features extracted from the defect-free image, that is, the label obtained by comparing with the defect-free image can characterize whether the image block in the sample image has defects, and the defect detection result is the actual prediction result of the sample image through the abnormal segmentation model. The difference between the two can characterize the accuracy of the abnormal segmentation model to a certain extent.

[0111] Specifically, the model loss value of the anomaly segmentation model can be obtained according to the difference between the defect detection result of each sample image and the corresponding training label. The model loss value represents the prediction accuracy of the current anomaly segmentation model. Generally speaking, the smaller the model loss value, the higher the accuracy of the model. During the training process, it is necessary to continuously optimize the model loss value so that the model reaches the optimal state.

[0112] In the embodiment of the present application, the model loss value can be calculated using any possible loss function, such as the cross entropy loss function (Cross Entropy Loss Function), the mean squared error (MSE) loss function, the squared absolute error loss function, the maximum likelihood loss (Likelihood Loss, LHL) function, etc. Of course, it can also be other possible loss functions, and the embodiment of the present application does not limit this.

[0113] Step 30234A: Determine whether the model has reached the training termination condition. If yes, the model training ends.

[0114] In the embodiment of the present application, the training termination condition may include at least one of the following conditions:

[0115] (1) The number of iterations reaches the set threshold.

[0116] (2) The model loss value is less than the set loss threshold.

[0117] Of course, there may be other possible conditions, which are not limited in the embodiments of the present application.

[0118] Step 30235A: If the result of step 30234A is no, the parameters of the first feature extraction network are adjusted based on the model loss value.

[0119] In the embodiment of the present application, when the number of iterations does not exceed the preset number threshold, and the model loss value is not less than the set loss threshold, the determination process of step 30234A is no, that is, it is considered that the current abnormal segmentation model does not meet the iteration termination condition, and the model parameters need to be adjusted to continue training. After the parameter adjustment, the next iterative training process is entered, that is, jumping to step 30231A.

[0120] In one possible implementation, when the model still does not meet the convergence conditions, the model weight parameters can be updated by optimization algorithms such as gradient descent method and stochastic gradient descent algorithm to minimize the above objective function, and training can be continued with the updated model weight parameters so that the model can better predict defects in sample images.

[0121] When the number of iterations has exceeded the pre-set threshold, or the model loss value is less than the set loss threshold, the determination process of step 30234A is yes, that is, it is considered that the current anomaly segmentation model has met the convergence conditions, the model training is completed, and the trained anomaly segmentation model is obtained.

[0122] In one possible implementation, see Fig. 8A and Figure 8BAs shown, Fig. 8A Another schematic diagram of the training process of the abnormal segmentation model provided in the embodiment of the present application is as follows: Figure 8B Another structural schematic diagram of the anomaly segmentation model provided in an embodiment of the present application, the anomaly segmentation model includes a first feature extraction network, a position enhancement network and a defect prediction network. The functions of the first feature extraction network and the defect prediction network are the same as those in the first embodiment described above. The position enhancement network is used to enhance the corresponding sample features according to the position information of each image block, so that subsequent predictions can be made based on the position information, limiting the feature prediction within a certain image space range, thereby improving the accuracy of defect prediction.

[0123] Similarly, since each training process of the anomaly segmentation model is similar, we will take one of them as an example here. Fig. 8A As shown, the following steps may be included:

[0124] Step 30231B: For the multiple sample images input this time, feature extraction is performed on each image block of each sample image through the first feature extraction network in the abnormal segmentation model used this time to obtain corresponding sample features.

[0125] Step 30232B: Through the position enhancement network used this time, based on the position information of each image block in the corresponding sample image, the corresponding sample features are respectively subjected to position enhancement processing to obtain enhanced features containing position information.

[0126] See also Figure 8B As shown, considering that the convolution operation commonly used in feature extraction is translationally equivariant, a unified convolution kernel parameter can be shared at different positions of the image, but this results in the convolution operation only being able to sense local information and being unable to sense the position information of the image block. Therefore, an embodiment of the present application introduces a position enhancement network CoordConv, which is used to add corresponding position information to the sample features after feature extraction. CoordConv represents the coordinates of the pixel points of the feature map by adding corresponding channels to the input feature map of the convolution, so that the coordinates can be perceived to a certain extent during the convolution learning process to improve the subsequent detection accuracy.

[0127] The process of position enhancement network will be explained in detail in the actual defect detection stage, so it will not be elaborated here. In addition, it should be noted that the position enhancement network will not be added when training the anomaly detection model in the first stage, but will only be added when fine-tuning the supervised model. After the training is completed, it will play a major role in the training of the anomaly detection model in the third stage, and will play a role in the final reasoning stage, which is the actual application stage of the model.

[0128] Step 30233B: Based on the enhanced features corresponding to each image block, a defect detection result of a sample image is obtained.

[0129] Based on the enhanced features, the defect detection results of the sample image can be obtained. By comparing the difference between the defect detection case result and the training label of the sample image, the accuracy of the current network, or the accuracy of the position enhancement network used this time, can be characterized. Therefore, the parameters of the position enhancement network used this time can be adjusted according to the difference, so as to continuously tune the parameters of the position enhancement network through an iterative process. For example, the parameters of the position enhancement network can be adjusted according to the model loss value determined by the difference between the defect detection case result and the training label of the sample image.

[0130] Step 30234B: Based on the defect detection results of each sample image and the corresponding training labels, obtain the model loss value of the anomaly segmentation model.

[0131] Step 30235B: Determine whether the model has reached the training termination condition. If yes, the model training ends.

[0132] It should be noted that the process of step 30231B and steps 30233B to 30235B is the same as Fig. 7A The processing methods of the corresponding steps in are similar, so please refer to the relevant introduction of the aforementioned process and will not be repeated here.

[0133] Step 30236B: If the result of step 30235B is no, the parameters of the first feature extraction network are adjusted based on the model loss value, and the parameters of the position enhancement network are adjusted based on the model loss value.

[0134] In the embodiment of the present application, when the number of iterations does not exceed the preset number threshold, and the model loss value is not less than the set loss threshold, the determination process of step 30235B is no, that is, it is considered that the current abnormal segmentation model does not meet the iteration termination condition, then it is necessary to adjust the model parameters to continue training, and Fig. 7A The difference is that, in addition to the first feature extraction network, the number of adoptions of the position enhancement network needs to be adjusted. After the parameter adjustment, the next iterative training process is entered, that is, jumping to step 30231B.

[0135] In the embodiment of the present application, after the training of the anomaly segmentation model is completed, the model parameters of the anomaly segmentation model can be saved and the third stage of training can be entered. However, it should be noted that in actual application, the training process of the above-mentioned anomaly segmentation model can also be trained separately on the basis of the above-mentioned three stages, that is, directly according to the collected sample images and training labels through the above-mentioned training process to obtain the trained anomaly segmentation model, and the trained anomaly segmentation model can be used for feature extraction and position enhancement in the actual prediction stage.

[0136] It should be noted that Figure 8B The process shown can also be regarded as training for the position enhancement network. That is, in the training process of the abnormal segmentation model, the parameters of the first feature extraction network and the position enhancement network can be adjusted at the same time. Figure 8B It is specifically shown in this that the parameters of the first feature extraction network can also be fixed, and only the parameters of the position enhancement network can be adjusted, or the first feature extraction network can be adjusted first, and after a certain stage, only the parameters of the position enhancement network can be adjusted. In actual scenarios, it can be configured according to needs, and the embodiments of the present application do not limit this.

[0137] Step 303: In the third stage, the anomaly detection model is retrained, that is, based on the backbone network of the trained anomaly segmentation model, feature extraction is performed again on each defect-free image to obtain a reference feature set.

[0138] In the embodiment of the present application, through the second stage of domain adaptability training, the backbone network in the anomaly segmentation model can be adapted to the feature extraction of image defect detection containing the target device, and the extracted image features are more adapted to the field of image defect detection containing the target device. Therefore, in the third stage, the anomaly detection model can be retrained based on the backbone network of the second stage, that is, it is necessary to re-extract features of the defect-free image through the backbone network to obtain a reference feature set. Each reference feature in this reference feature set can be used in the actual defect detection scenario. Among them, when the backbone network is Figure 7B When the network structure is shown, the first feature extraction network in the trained abnormal segmentation model can be used to extract features from each defect-free image to obtain a reference feature set; or, when the backbone network is Figure 8B When the network structure is shown, the features of each defect-free image can be extracted based on the first feature extraction network in the trained abnormal segmentation model, and the extracted local features can be position enhanced by the position enhancement network in the trained abnormal segmentation model to obtain a reference feature set.

[0139] In an embodiment of the present application, an adaptive threshold segmentation method is used to extract areas with higher anomaly scores, and a binary anomaly map is obtained as a pseudo-label, which is used to train an anomaly segmentation model. In the training process of the anomaly segmentation model, a CorrdConv layer is introduced to enhance the sensitivity of the final feature to the position and improve the accuracy of defect detection.

[0140] In the embodiment of the present application, the deep features extracted by the above backbone network are used as the feature representation of the corresponding image block, and the reference feature set memory bank is constructed to serve as the normal feature library of the anomaly detection model and participate in the actual defect detection process. In addition, when constructing the memory bank, in order to estimate the anomaly score of the image to be detected only for a specific area, the image block position information is also introduced for the image feature.

[0141] Specifically, for a given image L, for the image block a at position (i, j), here we take the feature map after 8 times downsampling as an example, assuming that its feature representation is f a , let p a The encoding of its position information can be expressed as:

[0142]

[0143] Where W and H are the width and height of the image respectively.

[0144] Then the enhanced features after position enhancement processing can be expressed as:

[0145]

[0146] in, Represents the feature after position enhancement.

[0147] Then, based on The acquisition method can traverse the image blocks in all normal sample images, extract the feature representations of all image blocks, and finally generate a memory bank.

[0148] However, the size of the generated memory bank directly affects the memory consumption and the efficiency of the abnormal reasoning process. Therefore, in order to reduce the memory consumption caused by saving the memory bank and the time consumption during abnormal reasoning, and to facilitate feature comparison in the reasoning stage, the size of the memory bank needs to be reduced. Of course, for the initial feature set, the following process can also be used to reduce the feature set.

[0149] Specifically, after the training of the abnormality segmentation model is completed, feature extraction can be performed on each image block of each defect-free image based on the first feature extraction network, or feature extraction can be performed on each image block of each defect-free image based on the first feature extraction network and the position enhancement network to obtain an updated feature set, wherein the updated feature set includes updated features corresponding to each image block of each defect-free image, and then some updated features can be selected from the updated feature set according to the similarity between the two updated features in the updated feature set to obtain the final reference feature set.

[0150] In a possible implementation, one of the reference features in the updated feature set whose similarity exceeds a certain threshold can be directly reduced. For example, if the similarity between two reference features exceeds 95%, the effect of feature comparison with any of the reference features is substantially similar, so one of them can be deleted.

[0151] In a possible implementation, a coreset selection algorithm may be used to select features from the initial reference feature set to obtain a final reduced reference feature set.

[0152] See also Fig.9A FIG. 1 is a flow chart of a reduction process provided by an embodiment of the present application. Fig. 9B The result diagram of each iteration is shown. Fig.9A and Fig. 9B Specifically, taking the greedy approximation algorithm as an example, according to the similarity between the updated features in the updated feature set, some updated features are selected from the updated feature set to obtain the reference feature set, which may include the following steps. However, it should be noted that in actual applications, other possible algorithms may also be used, and the embodiments of the present application do not limit this.

[0153] Step 901: Select at least one update feature from the update feature set and initialize the reference feature set.

[0154] Specifically, the reference feature set can be initialized by randomly selecting features from the update feature set. The number of selected features can be one or more. Then, the initialized reference feature set is iteratively updated until the iteration termination condition is met. Since the process of each iteration is similar, this is specifically introduced as an example. In each iteration update, the following operations are performed:

[0155] Step 902: for each update feature in the update feature set, determine a corresponding feature pair, each feature pair including an update feature and a reference feature in the reference feature set with the highest similarity to the update feature.

[0156] In an embodiment of the present application, the similarity can be characterized by the distance in the same feature space. Therefore, for each updated feature in the updated feature set, the distance between it and each reference feature in the current reference feature set is calculated, and then the reference feature closest to the updated feature is selected to obtain a feature pair.

[0157] It should be noted that, in practical applications, the updated features that have been added to the reference feature set do not need to be calculated in the next iteration, that is, the updated feature set in each iteration is the updated features remaining after the last selection.

[0158] Step 903: For each reference feature included in each feature pair, add the updated feature with the minimum similarity to each reference feature to the reference feature set.

[0159] For each feature pair obtained, the updated feature in the feature pair with the smallest similarity can be added to the reference feature set, so as to avoid too many similar features being added to the reference feature set, so that each feature in the reference feature set is representative.

[0160] In a possible implementation, by traversing each update feature in the update feature set , calculate its difference with each reference feature in the reference feature set Q The distance between them, assuming that it is closest to the reference feature, after the traversal is completed, a series of ( , ), select the pair with the largest distance among these feature pairs, and convert the corresponding feature Add to Q, and one iteration is completed. Then one iteration process can be expressed as follows:

[0161]

[0162]

[0163] in, represents the updated features added to the reference feature set at each iteration, Characterize each updated feature in the updated feature set , the reference feature closest to the reference feature set Q The distance between Characterize from all feature pairs ( , ) Select the updated feature from the feature pair with the largest distance As , Characterizing Update Features Belongs to the difference set between the updated feature set P and the reference feature set Q, the reference feature belongs to the reference feature set Q, The reference feature set Q after the iteration is the union of the newly selected features and the reference feature set Q before the iteration.

[0164] See also Fig. 9C As shown in the figure, it is a schematic diagram of feature selection in one iteration. The diamond represents the update feature, the circle represents the reference feature, and d1~d4 represent the distance between an update feature and a reference feature. For each update feature, the reference feature closest to it is selected to form a feature pair, and then the pair with the largest distance is selected from all feature pairs, and the corresponding update feature is added to Q.

[0165] It should be noted here that the selected pair with the largest distance may refer to the pair with the largest distance among all feature pairs, or may refer to the pair with the largest distance among these feature pairs that contain the reference feature for each reference feature.

[0166] The above process is then repeated until a termination condition is met. The termination condition may be, for example, that Q meets a predetermined size or that the number of iterations reaches a certain threshold, etc. This embodiment of the present application does not impose any limitation on this.

[0167] See also Fig. 9B As shown, when only one update feature is initially selected to be added to the reference feature set, in the first iteration, the similarities between each update feature in the update feature set and the reference feature are calculated respectively, and then the update feature with the smallest similarity is selected and added to the reference feature set, so as to obtain a reference feature set containing two reference features; in the second iteration, similarly, the similarities between each update feature in the update feature set and the reference feature are calculated respectively, and then the update feature with the smallest similarity is selected and added to the reference feature set, so as to obtain a reference feature set containing four reference features. By adding them one by one, a reference feature set that meets the prediction size can be finally obtained. At this time, the reference features contained therein are all representative reference features, so as to facilitate the memory usage and efficiency when searching the reference feature set in the subsequent practical application stage.

[0168] Next, the defect detection process in the actual application stage is introduced. Fig.10 The figure is a flow chart of the defect detection method provided in the embodiment of the present application.

[0169] Step 1001: receiving an image to be inspected containing a target device.

[0170] Among them, the target device can be any device with a fixed connection structure and form, such as a connector or circuit board of a camera module. In practical applications, the image to be detected can be collected and uploaded in real time by an image acquisition device. For example, the image acquisition device can be set on a production line or an assembly line to capture the image to be detected containing the target device, and detect in real time whether each target device has defects. Alternatively, the image to be detected can also be uploaded by any terminal device. For example, when it is necessary to assist in determining whether an image containing a target device has defects, the image to be detected can be uploaded through the terminal device.

[0171] In an embodiment of the present application, in order to improve the accuracy of defect detection of the target device, the image to be inspected can include the full picture of the target device as much as possible. For planar devices, such as connectors or circuit boards, images taken from one angle can be used as the image to be inspected. For three-dimensional devices, images can be taken from multiple angles and spliced ​​together as the image to be inspected.

[0172] Step 1002: extract features from each image block contained in the image to be detected to obtain corresponding local features.

[0173] In practical applications, considering that the image acquisition device will inevitably be restricted by various factors such as the sample placement angle and the stability of the robot arm when taking pictures, the imaging results of the same point will have inconsistent angles. For example, for connector points, since the neatness of the pins needs to be analyzed, the inconsistency of the image angle will bring additional errors, resulting in missed detection. Therefore, after obtaining the captured image to be tested, image registration and image extraction operations are required, and then the feature extraction process is performed based on the image to be tested after the operation. The image registration and image extraction operations can be performed based on pre-configured templates.

[0174] Specifically, template images of various types of target devices can be pre-configured, and then after receiving the image to be detected, the corresponding target template image is obtained from the pre-configured multiple template images based on the target type of the target device, and the image to be detected is registered based on the preset feature points in the target template image to obtain a registered image. Among them, the image registration can use the accelerated robust feature point detection (Speeded-Up Robust Features, SURF) algorithm to perform fast and robust feature point detection, and perform image registration based on the detected feature points.

[0175] Then, based on the preset image area in the target template image, the local image area where the target device is located is obtained from the registered image, and then the features of each image block contained in the local image area are extracted to obtain the corresponding local features. Specifically, the preset image area in each template image can be pre-configured, and then after registration, the registered image can be cut out according to the configured preset image area.

[0176] See also Fig.11 As shown in the figure, for connector defect detection, the input image to be inspected includes not only the connector but also the FPC and other parts. After image registration and clipping, the main area containing only the connector distribution can be extracted, which is convenient for subsequent analysis and avoids the introduction of foreign objects in the surrounding background to improve the accuracy of the features.

[0177] In a possible implementation, the feature extraction process of step 1002 may be an image preprocessing process. Fig. 12A As shown, the output of this method can be the pixel-level image features of the image to be detected, that is, each local feature can be the image feature of each pixel, which is equivalent to the image block at this time being understood as a pixel. The feature map before and after feature extraction has the same size as the image to be detected, and one feature value represents the feature of a pixel.

[0178] For example, the feature extraction process may be to perform normalization processing on the input image to be detected.

[0179] In a possible implementation, the feature extraction process of step 1002 may also be performed based on a deep learning feature extraction network, that is, the deep feature extraction of the image to be detected may be performed through the feature extraction network. Fig. 12B As shown, the output of this method can be the image features after downsampling of the image to be detected, then each local feature can be the image feature of each image block, and the image block at this time contains multiple pixels. The feature map before and after feature extraction is reduced in size compared to the image to be detected, and one feature value represents the feature of an image area. For example, the feature extraction network here can be the first feature extraction network after the aforementioned training. The first feature extraction network can adopt a network structure based on a convolutional neural network, such as a residual network such as ResNet50 or a related derivative network.

[0180] For example, see Fig.13 As shown, a possible schematic diagram of the structure of the first feature extraction network is shown. Fig.13The first feature extraction network shown is composed of M cascaded residual units (Blocks), i.e., residual unit 1 to residual unit M. The value of M can be configured according to the feature extraction depth required by the actual scene. For example, it can be configured to 5, and the size of the feature map finally extracted is 1 / 16 of the image to be detected. Of course, the size of the feature map extracted by the first two residual units can also be configured not to change, and the size of the feature map finally extracted is 1 / 8 of the image to be detected. Finally, an iterative method is adopted to extract features of the image to be detected, and in each iteration, based on the preset convolution kernel, the input features of this iteration are convolutionally extracted to obtain convolution features, wherein in the first iteration, the input features are the image to be detected, and in the non-first iteration, the input features are the output features of the previous iteration, and then, residual connection processing is performed based on the convolution features and the input features to obtain the splicing features, which are the output features of the current residual unit, and if this iteration is the last iteration, the splicing features are used as the output features of this iteration, that is, the image features finally extracted, and if this iteration is not the last iteration, it is necessary to perform downsampling processing on the splicing features based on the feature scale, and use the obtained downsampled features as the output features of this iteration, and output them to the next residual unit for further processing.

[0181] Since the structure of each residual unit is similar, Fig.13 Specifically, the structure of residual unit 2 is used to illustrate. For each residual unit, it includes a side network and a backbone network. The backbone network is represented as F(x), which includes two network layers (weight layer). First, the channel is downsampled through weight layer1, and a 3x3 convolution kernel is used for feature transformation. Then, the channel is upsampled through weight layer2 to restore the original channel size. The layers are activated by a linear rectification function (Relu). The side network directly short-circuits the input x, and finally obtains the output feature x' through a residual connection with the input. It can be expressed as:

[0182]

[0183] Every two residual units are connected through a spatial downsampling unit, that is, after the residual unit outputs the feature, it is necessary to perform spatial downsampling, and output the image features obtained by downsampling to the next residual unit for further processing, so as to increase the network receptive field and the local translation invariant effect. At the same time, the number of reference channels of each residual unit also increases with the deepening of the network layer, so that richer semantic information can be extracted. Among them, spatial downsampling refers to downsampling at the image scale level. After spatial downsampling, the size of the image will be reduced by half. Spatial downsampling can be achieved by using the maximum pooling (Maxpool) or the convolution layer with a step size of 2.

[0184] See also Fig.13 As shown in the figure, the final result of feature extraction is the feature map of the image to be detected, and each feature vector in the feature map represents the local features corresponding to each image block in the image to be detected. For example, for a feature map with a size of 1 / 8 of the image to be detected, each feature vector represents the local features of a local image area of ​​16x16 in the image to be detected.

[0185] Step 1003: Using the trained position enhancement network, based on the position information of each image block in the image to be detected, position enhancement processing is performed on the corresponding local features to obtain corresponding enhanced features containing position information.

[0186] In the embodiment of the present application, considering that the convolution operation is equivariant to translation, a uniform convolution kernel parameter can be shared at different positions of the image. However, this results in the traditional convolution operation being able to sense only local information and not position information. However, when performing defect detection, the difference between different positions is large, so it is necessary to consider the position information of the image block. Therefore, the embodiment of the present application introduces a position enhancement network to add the position information of each image block, so that the position information can be involved in the subsequent defect detection, thereby improving the accuracy of defect detection.

[0187] In a possible implementation, the position enhancement network may directly add corresponding position features after the local features extracted in step 1002 to obtain enhanced features.

[0188] For details, see Fig.14AAs shown, it is a schematic diagram of position enhancement provided by an embodiment of the present application. Among them, when the position information is represented by coordinates, the position feature map at the horizontal coordinate (i coordinate) level and the position feature map at the vertical coordinate (j coordinate) level can be obtained according to the position information of each image block in the image to be detected, and then these two position feature maps are concatenated with the overall feature map of the image to be detected, and then the corresponding enhanced feature map can be obtained. For one of the image blocks, see Fig.14A As shown, the corresponding longitudinal position feature can be obtained based on the longitudinal position information of the image block in the image to be detected, and the corresponding transverse position feature can be obtained based on the transverse position information of the image block in the image to be detected. Both the longitudinal position feature and the transverse position feature belong to the local position feature corresponding to the image block in the position feature map, and then the local features, transverse position features and longitudinal position features of the image block are spliced ​​to obtain the corresponding local enhanced features, that is, the enhanced features of the image block.

[0189] Of course, in practical applications, other forms may be used to represent the position information. For example, the image blocks may be numbered, and the position information is then represented according to the numbers of the image blocks. One number uniquely corresponds to an image region in the image to be detected.

[0190] In a possible implementation, the position enhancement network can directly add the corresponding position features after the local features extracted in step 1002, and further perform feature extraction through the convolutional feature extraction layer to obtain the corresponding enhanced features. That is, after adding two channels to the sample features, two layers of CorrdConv can be added at the end of the backbone network to extract position-sensitive feature representations.

[0191] For details, see Fig. 14BAs shown, another position enhancement schematic diagram provided by an embodiment of the present application. According to the position information of each image block in the image to be detected, the position feature map on the horizontal axis level and the position feature map on the vertical axis level can be obtained respectively, and then the two position feature maps are spliced ​​on the spliced ​​feature map of the image to be detected. After the convolution operation is performed through the convolution feature extraction layer, the corresponding enhanced feature map can be obtained. It can be seen that when the size of the overall feature map is H×W×C, the size of the spliced ​​feature map obtained after splicing is H×W×(C+2), which is equivalent to adding two layers on the feature channel level. After feature extraction is performed through the convolution feature extraction layer, a feature representation containing position information can be extracted, and the feature representation is sensitive to position information, so that the subsequent feature comparison process will have the participation of position information, and the size of the extracted enhanced feature map is H'×W'×C'. In one possible case, H'×W'×C' can be H×W×C, which is equivalent to the final enhanced feature map being consistent with the size before enhancement, thereby facilitating the structural setting of the network and reducing the complexity of the network.

[0192] The above-mentioned position-sensitive CoordConv (short for position enhancement network) represents the coordinates of feature map pixels or image blocks by adding corresponding channels to the input feature map of the convolution, so that the coordinates can be perceived to a certain extent during the convolution learning process to improve the detection accuracy.

[0193] It should be noted that although Fig. 14B What is shown is the feature map of the entire image. The local feature is the feature vector corresponding to a feature point in the feature map. Similarly, the enhanced feature of an image block is also the feature vector corresponding to a feature point in the enhanced feature map.

[0194] In a possible implementation, when the local features extracted in step 1002 are obtained only by preprocessing the image to be detected, the size of the feature map extracted in step 1002 may be consistent with the image to be detected, and then pixel-level position information may be added thereto and input into the first feature extraction network for feature extraction, thereby obtaining corresponding enhanced features.

[0195] See also Fig. 14CAs shown, it is another position enhancement schematic diagram provided by an embodiment of the present application. According to the position information of each pixel point in the image to be detected, the position feature map on the horizontal axis level and the position feature map on the vertical axis level can be obtained respectively, and then these two position feature maps are spliced ​​after the image to be detected, and input into the first feature extraction network for feature extraction, and then the corresponding enhanced feature map can be obtained. It can be seen that when the size of the image to be detected is H×W×C, the size of the spliced ​​feature map obtained after splicing is H×W×(C+2), which is equivalent to adding two layers to the feature channel level. After feature extraction, the size of the enhanced feature map extracted is H'×W'×C'. Generally speaking, as the feature extraction goes deeper, H'×W' is usually greater than H×W, and C is usually less than C', for example, H'×W' is 1 / 8 or 1 / 16 of H×W, C is 3, and C' is 256 or 512, etc.

[0196] It should be noted that the feature enhancement process here can also be applied to the feature enhancement in the above-mentioned training stage, and the enhancement method used in the training stage should be the same as the enhancement method used in the application stage.

[0197] Step 1004: Based on the similarity between each enhanced feature and the pre-configured reference feature, the abnormality assessment value corresponding to each image block is determined respectively.

[0198] Among them, each reference feature is the feature in the reference feature set extracted above, and is extracted based on the defect-free image of the target device. Each reference feature contains the location information of the corresponding image block; and each enhanced feature is also extracted in the same way. The idea of ​​the anomaly detection algorithm is to realize abnormal reasoning by comparing the features between the enhanced features and the reference features, and obtain the abnormal evaluation value corresponding to the image block. That is to say, based on the constructed reference feature set, for any image block on a given image to be detected, it is compared with the reference features in the reference feature set, and the abnormal evaluation value is calculated according to the similarity between the features. The area with a high abnormal evaluation value will be regarded as a defect.

[0199] In a possible implementation manner, the abnormality assessment value of the image block may be determined by comparing with a reference feature with a high similarity.

[0200] Specifically, for each enhanced feature, the reference feature with the highest similarity to the enhanced feature is determined from the reference feature set, and then the abnormality evaluation value of the image block corresponding to the enhanced feature can be determined based on the difference between the reference feature and the enhanced feature. Generally speaking, if the similarity between an image block and the reference feature that is most similar to it is still less than a certain threshold, then its abnormality evaluation value should be high, and the probability that it is a defective image block is obviously high.

[0201] In some cases, for each image to be detected, it is only necessary to know whether the image is a defective image, and then it can be transferred to subsequent manual review. Therefore, taking distance as an example here to measure similarity, for N image blocks included in an image to be detected, when detecting defects in actual scenes, for each image block, a reference feature with a distance d closest to it can be found from a reference feature set. In this way, N image blocks can find N reference features to form N feature pairs, so that the pair with the largest distance d can be found from these N feature pairs as the potential most abnormal image block, and the distance d between the feature pairs can be used to determine whether the image to be detected is abnormal, that is, whether it is a defective image block, so that the defect detection process of the image to be detected can be quickly performed.

[0202] In a possible implementation, when it is determined that an image block may have a defect, the abnormality assessment value of the image block may be comprehensively determined by combining reference features surrounding the reference feature with a high degree of similarity. The surrounding reference features refer to reference features within a specified distance range of the reference feature, or may refer to a specified number of reference features closest to the reference feature.

[0203] Specifically, for each enhanced feature, the reference feature A with the highest similarity to the enhanced feature is determined from the reference feature set. Then, for reference feature A, multiple reference features B whose similarity to reference feature A is not greater than a preset similarity threshold can be determined from the reference feature set, that is, reference features B located around reference feature A in the reference feature set. It can be understood that only the similarity of reference features extracted from the same image block in different defect-free images will be relatively high. Therefore, in essence, the intention here is to perform feature comparison with most image blocks at the same position to determine whether an image block has defects. Therefore, the abnormality assessment value of the image block corresponding to the enhanced feature can be determined based on the difference between the reference feature A and multiple reference features B and the enhanced feature.

[0204] For example, for each image block , find the closest reference feature in the reference feature set, calculate the distance between the two as d, and then for all image blocks, find the pair with the largest d as the potential most abnormal area, and its corresponding reference feature is recorded as , the abnormal evaluation value between the two can be expressed as:

[0205]

[0206] This distance can be used as a measure of whether it is an anomaly, and in order to comprehensively consider the situation of adjacent reference features, The m closest instances in the reference feature set are calculated If the distances are large, then the probability that the image block is abnormal is lower. The abnormal evaluation value can be expressed as follows:

[0207]

[0208] Step 1005: Determine the defect detection result of the image to be detected based on the obtained abnormality evaluation values.

[0209] Specifically, after calculating the abnormality evaluation value of each image block, it can be determined whether each image block has defects. By combining the defect results of each image block, the defect detection result of the target device can be obtained. For example, the defect detection result can indicate the number of defects and the defect location of the target device. Alternatively, when only the abnormality evaluation value of the image block with the greatest possibility of abnormality is calculated, it can be determined whether the current image to be detected has defects, that is, whether the target device has defects. When the image block with the greatest possibility of abnormality has an abnormality, it can be known that the target device has defects and the number of defects is at least one.

[0210] The defect detection method described above in the embodiment of the present application can be used as part of 3C quality inspection capabilities and applied in AI quality inspection scenarios. For example, it can analyze and determine the abnormalities of the connector in the camera module, and give a sample-level defect judgment together with the analysis results of other points.

[0211] See also Fig.15 The figure is a schematic diagram of the defect detection process of the embodiment of the present application.

[0212] In the training phase, the reference feature set required by the anomaly detection algorithm is obtained in advance through the training process. Specifically, the embodiment of the present application inputs all normal sample images into the backbone network to extract feature representations, and these feature representations will be reduced using the coreset construction method, so that only representative normal area features are left. Then, based on the constructed coreset, for any feature point on a given image to be inferred, it is compared with the normal feature in the coreset, and the anomaly score is calculated based on the distance between the features. The area with a high anomaly score will be regarded as a defect. Finally, based on the anomaly detection model preliminarily established above, a pseudo-label is marked on the abnormal area, and an anomaly segmentation model is trained in a supervised manner based on the pseudo-label. The segmentation model shares the backbone network with the above-mentioned anomaly detection model to achieve domain adaption of the backbone network on the target device image through a supervised training method, and then based on the fine-tuned backbone network, the anomaly detection model is reconstructed using the method of the first step to obtain the final reference feature set.

[0213] In the actual application stage, after the image to be detected of the camera module is obtained by camera shooting, the image to be detected is uploaded to the defect detection equipment. The defect detection equipment first performs image registration and cutout through the pre-configured image template to obtain the local area image of the target device area, which is conducive to the abnormal detection method to perform efficient abnormal feature comparison. Then. The enhanced features containing position information are extracted through the above-mentioned position-sensitive feature extraction process, and then the abnormal detection algorithm is used to compare the features with the above-mentioned reference feature set to perform NG judgment, that is, to determine whether there is a defect. Fig.15 As shown, it is finally possible to determine whether there is an abnormality in the image to be detected and which image blocks have abnormalities, such as outputting a corresponding defect mask. In some scenarios, the defect type to which the defect mask belongs can also be determined in combination with the content of each area in the image to be detected, for example Fig.15 The pin bending and solder overflow shown can be combined with other judgment methods to determine the corresponding defect type.

[0214] In summary, the embodiment of the present application proposes a technical solution for device quality inspection based on anomaly detection. This solution is different from the traditional target detection-based method. It is proposed based on the framework of anomaly detection. In the model training stage, no defect samples are required. Only normal sample images are used to complete the modeling. In the application stage, potential abnormal areas can be located. In addition, the solution is based on position information to assist feature comparison. On the basis of the original features, the image block position coordinates are introduced as an additional measure of the distance between features, so that the feature comparison can be limited to a certain spatial range, reducing the feature comparison space, so that the detection ability and the overkill level can be better balanced. In addition, in order to reduce the domain difference of the backbone network parameters in the industrial quality inspection scenario, the initial abnormal segmentation map is also used as a pseudo-label to supervise the backbone network. At the same time, this link also introduces CoordConv to directly inject position information into the network, so that the network is more sensitive to the difference in position. Finally, based on the backbone network after domain adaptation, using normal sample images, the final anomaly detection model is constructed to realize the defect detection process in the actual application stage.

[0215] See also Fig.16 Based on the same inventive concept, the embodiment of the present application further provides a defect detection device 160, which includes:

[0216] The feature extraction unit 1601 is used to receive an image to be detected containing a target device, and perform feature extraction on each image block contained in the image to be detected to obtain corresponding local features;

[0217] The position enhancement unit 1602 is used to perform position enhancement processing on the corresponding local features based on the position information of each image block in the image to be detected through the trained position enhancement network, so as to obtain corresponding enhanced features containing position information;

[0218] The abnormality detection unit 1603 is used to determine the abnormality evaluation value corresponding to each image block based on the similarity between each enhanced feature and a pre-configured reference feature; wherein each reference feature is extracted based on a defect-free image, and each reference feature includes position information of a corresponding image block, and the defect-free image includes a defect-free target device;

[0219] The result output unit 1604 is used to determine the defect detection result of the image to be detected based on the obtained abnormality evaluation values.

[0220] In a possible implementation, the device further includes a training unit 1605, configured to:

[0221] Based on multiple sample images and their corresponding training labels, the initial position enhancement network is iteratively trained until the iteration termination condition is met to obtain a trained position enhancement network; each training label is used to indicate: the probability that each image block in the corresponding sample image is a defective image block; in one training process, the following steps are performed:

[0222] For a sample image, feature extraction is performed on each image block of the sample image to obtain corresponding sample features;

[0223] Through the position enhancement network used this time, based on the position information of each image block in a sample image, the corresponding sample features are respectively subjected to position enhancement processing to obtain corresponding enhanced features containing position information;

[0224] Obtaining a defect detection result of a sample image based on the enhanced features corresponding to each image block of the sample image;

[0225] Based on the differences between the obtained multiple defect detection results and the corresponding training labels, the parameters of the position enhancement network used this time are adjusted.

[0226] In a possible implementation, the training unit 1605 is further configured to:

[0227] Through the second feature extraction network, feature extraction is performed on each image block contained in each defect-free image to obtain an initial feature set, where the initial feature set includes initial features corresponding to each image block of each defect-free image;

[0228] Based on the initial feature set, multiple sample images are respectively labeled to obtain corresponding training labels, and based on the multiple sample images and the corresponding training labels, the anomaly segmentation model is iteratively trained until an iteration termination condition is met; wherein the anomaly segmentation model includes a first feature extraction network and a position enhancement network, and the first feature extraction network and the second feature extraction network share network parameters;

[0229] Based on the first feature extraction network in the trained anomaly segmentation model, feature extraction is performed on each defect-free image respectively, and the extracted local features are position enhanced by the position enhancement network in the trained anomaly segmentation model to obtain a reference feature set, which includes preconfigured reference features.

[0230] In a possible implementation, the training unit 1605 is specifically configured to:

[0231] For multiple sample images, perform the following operations respectively:

[0232] For a sample image, extract features from each image block in the sample image through a second feature extraction network to obtain corresponding sample features;

[0233] Based on the similarity between each sample feature obtained and the initial feature set, a sub-label of the corresponding image block is determined, and each sub-label is used to indicate: the probability that the corresponding image block is a defective image block;

[0234] Based on the obtained sub-labels, a training label of a sample image is obtained.

[0235] In a possible implementation, the training unit 1605 is specifically configured to:

[0236] Based on the first feature extraction network, feature extraction is performed on each image block of each defect-free image respectively, and position enhancement processing is performed on the extracted local features through the position enhancement network to obtain an updated feature set, wherein the updated feature set includes updated features corresponding to each image block of each defect-free image;

[0237] According to the similarity between each two updated features in the updated feature set, some updated features are selected from the updated feature set to obtain a reference feature set.

[0238] In a possible implementation, the training unit 1605 is specifically configured to:

[0239] Select at least one update feature from the update feature set to initialize the reference feature set;

[0240] Iteratively update the reference feature set; wherein, in each iterative update, the following operations are performed:

[0241] For each update feature in the update feature set, a corresponding feature pair is determined respectively, each feature pair comprising an update feature and a reference feature in the reference feature set that has the highest similarity to the update feature;

[0242] For each reference feature included in each feature pair, the updated feature with the smallest similarity to each reference feature is added to the reference feature set.

[0243] In a possible implementation manner, the position enhancement unit 1602 is specifically configured to:

[0244] Through the trained position enhancement network, the following operations are performed for each image block:

[0245] For an image block, based on the lateral position information of the image block in the image to be detected, a corresponding lateral position feature is obtained;

[0246] Based on the longitudinal position information of an image block in the image to be detected, a corresponding longitudinal position feature is obtained;

[0247] Based on the local features, horizontal position features and vertical position features of an image block, corresponding enhanced features are obtained.

[0248] In a possible implementation, the feature extraction unit 1601 is specifically configured to:

[0249] The feature extraction of the image to be detected is performed in an iterative manner; in each iteration, the following operations are performed:

[0250] Based on the preset convolution kernel, the input features of this iteration are subjected to convolution feature extraction processing to obtain convolution features; wherein, in the first iteration, the input features are the image to be detected, and in the non-first iteration, the input features are the output features of the previous iteration;

[0251] Perform residual connection processing based on convolution features and input features to obtain splicing features;

[0252] If this iteration is the last iteration, the concatenated features are used as the output features of this iteration;

[0253] If this iteration is not the last iteration, the concatenated features are downsampled based on the feature scale, and the obtained downsampled features are used as the output features of this iteration.

[0254] In a possible implementation, the anomaly detection unit 1603 is specifically configured to:

[0255] For each enhancement feature, perform the following operations:

[0256] For an enhanced feature, determining a reference feature having the highest similarity to the enhanced feature;

[0257] Based on the difference between the reference feature and the enhanced feature, an abnormality assessment value of the image block corresponding to the enhanced feature is determined.

[0258] In a possible implementation, the anomaly detection unit 1603 is specifically configured to:

[0259] For each enhancement feature, perform the following operations:

[0260] For an enhanced feature, determining a reference feature having the highest similarity to the enhanced feature;

[0261] Determine, from the preconfigured reference features, a plurality of reference features whose similarity to the reference feature with the highest similarity is not greater than a preset similarity threshold;

[0262] Based on the reference feature with the highest similarity and the difference between each of the multiple reference features and an enhanced feature, an abnormality assessment value of an image block corresponding to an enhanced feature is determined.

[0263] In a possible implementation, the feature extraction unit 1601 is specifically configured to:

[0264] Based on the target type of the target device, obtaining a corresponding target template image from a plurality of preconfigured template images;

[0265] Based on the preset feature points in the target template image, image registration is performed on the image to be detected to obtain a registered image;

[0266] Based on a preset image area in the target template image, a local image area where the target device is located is obtained from the registered image;

[0267] Feature extraction is performed on each image block contained in the local image area to obtain corresponding local features.

[0268] Through the above-mentioned device, a framework based on anomaly detection is used to implement defect detection of the target device. In the defect detection process, location information is used to enhance features and improve the accuracy of defect detection. Specifically, on the one hand, a feature extraction layer based on position sensitivity is used to enhance the ability to extract location information. On the other hand, when establishing a reference feature set, a distance based on location information is used as a basis for feature comparison, so that the feature comparison process is more focused on local areas. In addition, in addition to the enhancement of location perception capabilities, the embodiment of the present application also generates pseudo-labels based on the prediction results of the initial anomaly detection model, and then uses the pseudo-labels to train the anomaly segmentation model, thereby achieving domain adaptation fine-tuning of the backbone network features, thereby improving the feature extraction capabilities of the backbone network in the field of target device images, and thus improving the accuracy of defect detection.

[0269] The device can be used to execute the methods shown in the various embodiments of the present application. Therefore, for the functions that can be implemented by the various functional modules of the device, reference can be made to the description of the aforementioned embodiments and no further details will be given.

[0270] See also Fig.17 Based on the same technical concept, the present application also provides a computer device. In one embodiment, the computer device can be, for example, Figure 1 The server shown in FIG. Fig.17 As shown, it includes a memory 1701 , a communication module 1703 and one or more processors 1702 .

[0271] The memory 1701 is used to store computer programs executed by the processor 1702. The memory 1701 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required to run the functions of the embodiment of the present application, etc.; the data storage area may store various functional information and operation instruction sets, etc.

[0272] The memory 1701 may be a volatile memory, such as a random-access memory (RAM); the memory 1701 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1701 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1701 may be a combination of the above memories.

[0273] The processor 1702 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1702 is configured to implement the above-mentioned defect detection method when calling the computer program stored in the memory 1701 .

[0274] The communication module 1703 is used to communicate with terminal devices and other servers.

[0275] The specific connection medium between the memory 1701, the communication module 1703 and the processor 1702 is not limited in the embodiment of the present application. Fig.17 In the embodiment, the memory 1701 and the processor 1702 are connected via a bus 1704. The bus 1704 is connected to the processor 1702 via a bus 1704. Fig.17 The connections between the other components are described with bold lines, which are only for illustration and are not intended to be limiting. The bus 1704 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.17 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0276] The memory 1701 stores a computer storage medium, and the computer storage medium stores computer executable instructions. The computer executable instructions are used to implement the defect detection method of the embodiment of the present application. The processor 1702 is used to execute the defect detection method of the above-mentioned embodiments.

[0277] In another embodiment, the computer device may also be a terminal device, such as Figure 1 In this embodiment, the structure of the computer device can be as follows: Fig.18 As shown, it includes: a communication component 1810, a memory 1820, a display unit 1830, a camera 1840, a sensor 1850, an audio circuit 1860, a Bluetooth module 1870, a processor 1880 and other components.

[0278] The communication component 1810 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The computer device can help the user to send and receive information through the WiFi module.

[0279] The memory 1820 can be used to store software programs and data. The processor 1880 executes various functions and data processing of the terminal device by running the software programs or data stored in the memory 1820. The memory 1820 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1820 stores an operating system that enables the terminal device to run. In the present application, the memory 1820 can store an operating system and various application programs, and can also store code for executing the defect detection method of the embodiment of the present application.

[0280] The display unit 1830 can also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device. Specifically, the display unit 1830 may include a display screen 1832 disposed on the front of the terminal device. The display screen 1832 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 1830 may be used to display the defect detection related page in the embodiment of the present application.

[0281] The display unit 1830 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device. Specifically, the display unit 1830 may include a touch screen 1831 arranged on the front of the terminal device, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0282] The touch screen 1831 can be covered on the display screen 1832, or the touch screen 1831 and the display screen 1832 can be integrated to realize the input and output functions of the terminal device, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 1830 can display the application program and the corresponding operation steps.

[0283] The camera 1840 can be used to capture still images, and users can post comments on the images taken by the camera 1840 through the application. The camera 1840 can be one or more. The object is projected onto the photosensitive element through the lens to determine the optical image. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 1880 to convert it into a digital image signal.

[0284] The terminal device may further include at least one sensor 1850, such as an acceleration sensor 1851, a distance sensor 1852, a fingerprint sensor 1853, and a temperature sensor 1854. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0285] The audio circuit 1860, the speaker 1861, and the microphone 1862 can provide an audio interface between the user and the terminal device. The audio circuit 1860 can transmit the electrical signal converted from the received audio data to the speaker 1861, which is converted into a sound signal for output. The terminal device can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1862 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1860 and converted into audio data, and then the audio data is output to the communication component 1810 to be sent to, for example, another terminal device, or the audio data is output to the memory 1820 for further processing.

[0286] The Bluetooth module 1870 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable computer device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1870 to exchange data.

[0287] The processor 1880 is the control center of the terminal device. It uses various interfaces and lines to connect the various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1820 and calling data stored in the memory 1820. In some embodiments, the processor 1880 may include one or more processing units; the processor 1880 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1880. In the present application, the processor 1880 can run the operating system, application programs, user interface display and touch response, as well as the defect detection method of the embodiment of the present application. In addition, the processor 1880 is coupled to the display unit 1830.

[0288] Based on the same inventive concept, an embodiment of the present application also provides a computer storage medium, which stores a computer program. When the computer program is run on a computer device, the computer device executes the steps of the defect detection method according to various exemplary embodiments of the present application described above in this specification.

[0289] In some possible implementations, various aspects of the defect detection method provided by the present application may also be implemented in the form of a computer program product, which includes a computer program. When the computer program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the defect detection method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device can execute the steps of each embodiment.

[0290] The computer program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0291] The computer program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on a computer device. However, the computer program product of the present application is not limited thereto. In the present application, the readable storage medium may be any tangible medium containing or storing a program, and the computer program included therein may be used by or in combination with a command execution system, apparatus or device.

[0292] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0293] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0294] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages.

[0295] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0296] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0297] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing a computer program.

[0298] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0299] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A defect detection method, characterized in that: The method comprises: Obtaining a reference feature set, the reference feature set comprising some updated features selected from the update feature set according to similarities between any two updated features in the update feature set, the update feature set being obtained by extracting features from image blocks respectively included in a plurality of defect-free images, each reference feature in the reference feature set corresponding to an image block in a defect-free image, and each reference feature comprising position information of a corresponding image block, the defect-free image comprising a defect-free target device; Receiving an image to be inspected containing a target device; Extracting features from each image block contained in the image to be detected to obtain corresponding local features; Through the trained position enhancement network, based on the position information of each image block in the image to be detected, the corresponding local features are respectively subjected to position enhancement processing to obtain corresponding enhanced features containing position information; For each enhanced feature obtained, respectively determine the reference feature with the highest similarity to each enhanced feature from the reference feature set to obtain a plurality of feature pairs, each feature pair including an enhanced feature and the reference feature with the highest similarity to the enhanced feature; From the plurality of feature pairs, determining a feature pair with the smallest similarity between the enhanced feature and the reference feature; If the similarity between the enhanced feature in the one feature pair and the reference feature is less than the abnormal similarity measure, determining a plurality of reference features having a similarity with the reference feature in the one feature pair greater than a preset similarity threshold; Based on the determined differences between each of the reference features and the enhanced features, determining an abnormality assessment value of the image block corresponding to the enhanced feature in the feature pair; wherein, if the similarities between each of the reference features and the enhanced feature in the feature pair are less than the abnormality similarity measure, then the abnormality assessment value indicates that the probability that the image block corresponding to the enhanced feature in the feature pair is a defective image block is less than a preset probability threshold; Based on the obtained abnormality evaluation value, a defect detection result of the image to be inspected is determined.

2. The method according to claim 1, characterized in that Before performing position enhancement processing on the corresponding local features based on the position information of each image block in the image to be detected by the trained position enhancement network to obtain the corresponding enhanced features containing the position information, the method further includes: Based on multiple sample images and their corresponding training labels, the initial position enhancement network is iteratively trained until an iteration termination condition is met to obtain the trained position enhancement network; each training label is used to indicate: the probability that each image block in the corresponding sample image is a defective image block; in one training process, the following steps are performed: For a sample image, extract features from each image block of the sample image to obtain corresponding sample features; Through the position enhancement network used this time, based on the position information of each image block in the sample image, position enhancement processing is performed on the corresponding sample features respectively to obtain corresponding enhanced features containing position information; Obtaining a defect detection result of the sample image based on the enhanced features corresponding to each image block of the sample image; Based on the differences between the obtained multiple defect detection results and the corresponding training labels, the parameters of the position enhancement network used this time are adjusted.

3. The method according to claim 1, characterized in that Before respectively determining the abnormality assessment values ​​corresponding to each of the image blocks based on the similarity between each of the enhanced features and the preconfigured reference features, the method further includes: Through a second feature extraction network, feature extraction is performed on each image block contained in each defect-free image to obtain an initial feature set, wherein the initial feature set includes initial features corresponding to each image block of each defect-free image; Based on the initial feature set, a plurality of sample images are respectively labeled to obtain corresponding training labels, and based on the plurality of sample images and the respective corresponding training labels, an abnormal segmentation model is iteratively trained until an iteration termination condition is met; wherein the abnormal segmentation model includes a first feature extraction network and the position enhancement network, and the first feature extraction network and the second feature extraction network share network parameters; Based on the first feature extraction network in the trained abnormal segmentation model, feature extraction is performed on each defect-free image respectively, and the extracted local features are position enhanced by the position enhancement network in the trained abnormal segmentation model to obtain a reference feature set, which includes the preconfigured reference features.

4. The method according to claim 3, characterized in that Based on the initial feature set, labeling a plurality of sample images respectively to obtain corresponding training labels includes: For the multiple sample images, perform the following operations respectively: For a sample image, extract features from each image block in the sample image through the second feature extraction network to obtain corresponding sample features; Based on the similarity between each sample feature obtained and the initial feature set, a sub-label of the corresponding image block is determined, each sub-label is used to indicate: the probability that the corresponding image block is a defective image block; Based on the obtained sub-labels, a training label of the one sample image is obtained.

5. The method according to claim 3, characterized in that Based on the first feature extraction network in the trained abnormal segmentation model, feature extraction is performed on each defect-free image respectively, and position enhancement processing is performed on the extracted local features through the position enhancement network in the trained abnormal segmentation model to obtain a reference feature set, including: Based on the first feature extraction network, feature extraction is performed on each image block of each defect-free image respectively, and position enhancement processing is performed on the extracted local features through the trained position enhancement network in the abnormal segmentation model to obtain an updated feature set, wherein the updated feature set includes the updated features corresponding to each image block of each defect-free image; According to the similarity between any two update features in the update feature set, some update features are selected from the update feature set to obtain the reference feature set.

6. The method according to claim 5, characterized in that The selecting part of the update features from the update feature set according to the similarity between the update features in each pair to obtain the reference feature set comprises: Selecting at least one update feature from the update feature set to initialize the reference feature set; Iteratively update the reference feature set; wherein, during each iterative update, the following operations are performed: For each update feature in the update feature set, a corresponding feature pair is determined respectively, each feature pair comprising an update feature and a reference feature in the reference feature set that has the highest similarity to the update feature; For each reference feature included in each feature pair, the updated feature having the smallest similarity with each reference feature is added to the reference feature set.

7. The method according to any one of claims 1 to 6, characterized in that: Through the trained position enhancement network, based on the position information of each image block in the image to be detected, the corresponding local features are respectively subjected to position enhancement processing to obtain corresponding enhanced features containing position information, including: The trained position enhancement network is used to perform the following operations on each image block: For an image block, based on the lateral position information of the image block in the image to be detected, obtaining a corresponding lateral position feature; Based on the longitudinal position information of the one image block in the image to be detected, obtaining a corresponding longitudinal position feature; Based on the local features, the horizontal position features and the vertical position features of the one image block, corresponding enhanced features are obtained.

8. The method according to any one of claims 1 to 6, characterized in that: Extracting features from each image block contained in the image to be detected to obtain corresponding local features includes: The feature extraction of the image to be detected is performed in an iterative manner; in each iteration, the following operations are performed: Based on a preset convolution kernel, a convolution feature extraction process is performed on the input feature of this iteration to obtain a convolution feature; wherein, in the first iteration, the input feature is the image to be detected, and in the non-first iteration, the input feature is the output feature of the previous iteration; Performing residual connection processing based on the convolution feature and the input feature to obtain a splicing feature; If this iteration is the last iteration, the splicing feature is used as the output feature of this iteration; If this iteration is not the last iteration, downsampling processing is performed on the spliced ​​features based on the feature scale, and the obtained downsampled features are used as output features of this iteration.

9. The method according to any one of claims 1 to 6, characterized in that: Extracting features from each image block contained in the image to be detected to obtain corresponding local features includes: Based on the target type of the target device, obtaining a corresponding target template image from a plurality of preconfigured template images; Based on the preset feature points in the target template image, performing image registration on the image to be detected to obtain a registered image; Based on a preset image area in the target template image, obtaining a local image area where the target device is located from the registered image; Feature extraction is performed on each image block included in the local image area to obtain corresponding local features.

10. A defect detection device, characterized in that: The device comprises: A training unit is used to obtain a reference feature set, wherein the reference feature set includes some updated features selected from the update feature set according to similarities between two updated features in the update feature set, wherein the update feature set is obtained by extracting features from image blocks respectively included in a plurality of defect-free images, wherein each reference feature in the reference feature set corresponds to an image block in a defect-free image, and each reference feature includes position information of the corresponding image block, and the defect-free image includes a defect-free target device; A feature extraction unit, used for receiving an image to be detected containing a target device, and performing feature extraction on each image block contained in the image to be detected to obtain corresponding local features; A position enhancement unit, configured to perform position enhancement processing on corresponding local features based on the position information of each image block in the image to be detected through a trained position enhancement network, so as to obtain corresponding enhanced features containing position information; An abnormality detection unit is used to obtain, for each enhanced feature obtained, a plurality of feature pairs from the reference features with the highest similarity to each enhanced feature in the reference feature set, each feature pair including a reference feature with the highest similarity to the enhanced feature; determine, from the plurality of feature pairs, a feature pair with the smallest similarity between the enhanced feature and the reference feature; if the similarity between the enhanced feature in the feature pair and the reference feature is less than the abnormal similarity measure, determine a plurality of reference features whose similarity to the reference feature in the feature pair is not greater than a preset similarity threshold; and, based on the differences between each of the determined reference features and the enhanced feature, determine an abnormality evaluation value of an image block corresponding to the enhanced feature in the feature pair; wherein, if the similarity between each of the reference features and the enhanced feature in the feature pair is less than the abnormal similarity measure, the probability that the abnormality evaluation value indicates that the image block corresponding to the enhanced feature in the feature pair is a defective image block is less than a preset probability threshold; The result output unit is used to determine the defect detection result of the image to be detected based on the obtained abnormality evaluation value.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Citation Information

Patent Citations

  • Image defect detection method and system based on deep twin network

    CN111445459A

  • Sand mold defect detection method and system based on machine vision

    CN114387233A

  • Depth feature memory bank-based unsupervised defect detection method and terminal

    CN115619784A

  • Image anomaly identification method based on depth vision model

    CN115760695A

  • Image recognition method and device, electronic equipment and storage medium

    CN116385745A