A defect detection method, apparatus, electronic device and storage medium

Abstract

This application relates to the field of data processing technology, and in particular to a defect detection method, apparatus, electronic device, and storage medium. The method involves: acquiring point images of an object to be detected at preset target points, and obtaining a defect detection result corresponding to the object based on the sub-detection results obtained from each point image. Specifically, for each acquired point image, the following operations are performed: acquiring a point template image associated with the corresponding target point, and determining a target region corresponding to the detection indication area in the point image; and determining the sub-detection result of the object at the target point based on the classification category information, severity category information, and defect pixel location information obtained from detecting each candidate defect within the target region. This improves the robustness of defect detection, reduces the implementation difficulty of defect detection, and increases the accuracy of defect detection.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a defect detection method, apparatus, electronic device and storage medium. Background Technology

[0002] With the modernization of technology, automated testing methods can now be used to detect defects in various components, such as automated quality inspection of 3C structural parts.

[0003] With the help of related technologies, a trained defect detection model can be used to detect defects based on a point map collected for the object to be detected. The detection results include the defect category, the confidence value corresponding to the defect category, and the defect location box. Alternatively, a trained defect detection algorithm can be used to detect defects based on a point map collected for the object to be detected. The detection results include the defect category, the confidence value corresponding to the defect category, the defect location box, and the defect pixel position. Based on the obtained detection results, it can be determined whether the object to be detected has defects.

[0004] Thus, when determining whether an object to be detected has a defect based on the defect category and the location box, only a general judgment on different types of defects can be made based on the defect category. Moreover, since it is impossible to avoid the problem of the object to be detected deviating from the specified target point when collecting point images, the detection results obtained from point images at different target points are prone to missed detections and false detections, resulting in misjudgments on whether defects exist and greatly reducing the efficiency of defect detection. Summary of the Invention

[0005] This application provides a defect detection method, apparatus, electronic device, and storage medium to solve the problems of low efficiency and accuracy of defect detection in the prior art, and the tendency to make false judgments.

[0006] Firstly, a defect detection method is proposed, including:

[0007] Acquire point images of the object to be detected at each preset target point;

[0008] Based on the sub-detection results obtained from detecting each point image, the defect detection result corresponding to the object to be detected is obtained. Specifically, for each point image acquired, the following operations are performed:

[0009] A point template image associated with the corresponding target point is obtained, and the point image and the point template image are image aligned. The target area corresponding to the detection indication area is determined in the point image, wherein the point template image is configured with a corresponding detection indication area.

[0010] Using a trained defect classification model, based on the point images, the classification category information, severity category information and location box corresponding to each identified candidate defect are obtained. Using a trained defect segmentation model, based on the images in each location box, the defect pixel location information set is identified respectively.

[0011] Based on the classification category information, severity category information, and defect pixel location information set of each candidate defect within the target area, the sub-detection result of the object to be detected at the target point is determined.

[0012] Secondly, a defect detection device is proposed, comprising:

[0013] The acquisition unit is used to acquire point images of the object to be detected at each preset target point.

[0014] The detection unit is used to obtain the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from the detection of each point image. Specifically, for each point image acquired, the following operations are performed:

[0015] A point template image associated with the corresponding target point is obtained, and the point image and the point template image are image aligned. The target area corresponding to the detection indication area is determined in the point image, wherein the point template image is configured with a corresponding detection indication area.

[0016] Using a trained defect classification model, based on the point images, the classification category information, severity category information and location box corresponding to each identified candidate defect are obtained. Using a trained defect segmentation model, based on the images in each location box, the defect pixel location information set is identified respectively.

[0017] Based on the classification category information, severity category information, and defect pixel location information set of each candidate defect within the target area, the sub-detection result of the object to be detected at the target point is determined.

[0018] Optionally, when training the defect classification model, the device further includes a training unit, which is specifically used for:

[0019] A defect classification model to be trained is constructed based on a preset target detection algorithm, and a set of sample point images is obtained. The defect classification model includes a sub-network that realizes the defect severity prediction function. The set of sample point images includes sample point images collected by different types of acquisition devices and corresponding to different target points. Each sample point image is labeled with a defect classification category label, a defect severity category label, and a location box label.

[0020] The defect classification model to be trained is trained through multiple rounds of iterative training using the sample point image set until the preset convergence condition is met, thus obtaining the trained defect classification model.

[0021] Optionally, when determining the sub-detection result of the object to be detected at the target point based on the classification category information, severity category information, and defect pixel location information set of each candidate defect within the target area, the detection unit is used to:

[0022] Among the candidate defects within the target area, target defects whose confidence value associated with the classification category information is higher than a set threshold, or whose degree category information does not belong to the preset normal degree information, are selected.

[0023] For each target defect, perform the following operations: Based on the set of defect pixel location information corresponding to the target defect, determine the area information corresponding to the target defect, and based on the area information and preset area detection conditions, obtain the corresponding area detection result;

[0024] Based on the area detection results corresponding to each target defect, the sub-detection results of the object to be detected at the target point are determined.

[0025] Optionally, after obtaining the corresponding area detection result based on the area information and preset area detection conditions, and before determining the sub-detection result of the object to be detected at the target point based on the area detection result corresponding to each target defect, the detection unit is further configured to:

[0026] Determine the defect region bounding box corresponding to the target defect, and based on the number of pixels constructing the edge of the defect region bounding box and the mapping relationship between pixels and size, determine the length information corresponding to the target defect;

[0027] Based on the length information and the preset length detection conditions, the corresponding length detection result is obtained.

[0028] Optionally, when obtaining the corresponding area detection result based on preset area detection conditions, the detection unit is used to:

[0029] Based on the region information to which the positioning box corresponding to the target defect belongs in the point image, and the classification category information corresponding to the target defect, obtain the area threshold configured for the area information of the target defect;

[0030] When the area information of the target defect is determined to reach a preset area threshold, the corresponding area detection result is judged as an area undetermined anomaly; when the area information of the target defect is determined not to reach the preset area threshold, the area detection result is judged as an area detection normal.

[0031] Optionally, when determining the sub-detection result of the object to be detected at the target point based on the area detection result corresponding to each target defect, the detection unit is used to:

[0032] Obtain the threshold for the total number of defects set for each category information, and count the total number of target defects whose area detection results are undetermined anomalies among the target defects belonging to the same category information.

[0033] If the total number of target defects corresponding to a certain category of classification information is higher than the corresponding total defect threshold, the sub-detection result of the object to be detected at the target point is determined to be abnormal; and if the total number of target defects corresponding to each category of classification information is not higher than the corresponding total defect threshold, the sub-detection result of the object to be detected at the target point is determined to be normal.

[0034] Optionally, when determining the target region corresponding to the detection indication region in the point image by performing image alignment processing between the point image and the point template image, the detection unit is used to:

[0035] Calculate the parameter transformation matrix configured for the point image when converting it to be aligned with the point template image;

[0036] Based on the parameter transformation matrix and the location information corresponding to the detection indication area in the point template image, a target area corresponding to the detection indication area is determined in the point image.

[0037] Optionally, when using a trained defect segmentation model to identify the defect pixel location information set based on the images in each localization box, the detection unit is used for:

[0038] Obtain a trained defect segmentation model, wherein the defect localization model is constructed based on a preset segmentation algorithm and is used to locate the set of defect pixel locations covered by defects.

[0039] The image content selected by each defect region is input into the defect localization model to obtain the set of defect pixel positions determined by the defect localization model for each image content.

[0040] Optionally, when obtaining the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from detecting each point image, the detection unit is used to:

[0041] If, among the sub-detection results obtained from detecting each of the image points, there is a sub-detection result that is judged as an abnormality, then the defect detection result corresponding to the object to be detected is judged as unqualified; and,

[0042] If all sub-detection results obtained from the detection of each point image are normal, the defect detection result corresponding to the object to be detected is judged as qualified.

[0043] Thirdly, an electronic device is proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned defect detection method when executing the program.

[0044] Fourthly, a computer-readable storage medium is proposed, on which a computer program is stored, which, when executed by a processor, implements the above-mentioned defect detection method.

[0045] Fifthly, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the aforementioned defect detection method.

[0046] The beneficial effects of this application are as follows:

[0047] This application proposes a defect detection method, apparatus, electronic device, and storage medium. The method acquires point images of the object to be detected at preset target locations. Based on the sub-detection results obtained from each point image, a defect detection result corresponding to the object to be detected is obtained. For each point image acquired, the following operations are performed: A point template image associated with the corresponding target location is acquired, and the point image and the point template image are aligned to obtain a target region corresponding to the detection indication region. The point template image contains a corresponding detection indication region. A trained defect classification model is used to obtain the classification category information, severity category information, and location box for each identified candidate defect based on the point image. A trained defect segmentation model is used to identify the defect pixel location information set based on the images within each location box. Based on the classification category information, severity category information, and defect pixel location information set for each candidate defect within the target region, the sub-detection result of the object to be detected at the target location is determined.

[0048] In this way, by aligning the point images and point template images corresponding to the same target points, the consistency of image formation for point images with the same target points is ensured, reducing the processing difficulty of automated defect detection and improving the robustness of defect detection. Simultaneously, by effectively locating the target region corresponding to the detection indication region in the point image, subsequent defect detection based on the point image can focus on the target region, avoiding misjudgments caused by defects in other areas of the point image. Furthermore, considering that the defect detection model and the defect localization model are based on different sample sizes, different models are used for the recognition of localization boxes and the identification of defect pixel location information. Instead of using a single model to simultaneously identify bounding boxes and defect pixel locations, this approach reduces the difficulty of sample generation and defect detection. Furthermore, the defect detection results include information on the degree of defect detection, effectively classifying the degree of defect anomaly and providing more reliable data for determining the results. Finally, the determination of the corresponding defect detection result for the target object is based on a comprehensive assessment of the detection information for each candidate defect in the image, taking into account the combined impact of all candidate defects and improving the accuracy of defect detection. Attached Figure Description

[0049] Figure 1 This is a schematic diagram illustrating possible application scenarios in the embodiments of this application;

[0050] Figure 2A This is a schematic diagram of the defect detection process in an embodiment of this application;

[0051] Figure 2B This is a schematic diagram of point images collected at different target locations in the embodiments of this application;

[0052] Figure 2C This is a schematic diagram of the process for obtaining sub-detection results from point images in an embodiment of this application;

[0053] Figure 2D This is a schematic diagram illustrating the composition of the defect classification model constructed in the embodiments of this application;

[0054] Figure 3 This is a schematic diagram of a general process for defect detection in the embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the overall process involved in defect detection in the embodiments of this application;

[0056] Figure 5 This is a schematic diagram of the logic structure of the defect detection device in the embodiments of this application;

[0057] Figure 6This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the structure of a computing device in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0060] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0061] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0062] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. The defect detection method proposed in this application can be applied to the processing of AI technologies.

[0063] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0064] A defect refers to a situation where the detected object fails to provide the security that the user has the right to expect, or where the detected object poses an unreasonable danger. In this embodiment of the application, the detected defects may have different classification categories and severity categories. After defect detection is performed on the object, the defect detection results can be used to determine whether the object is qualified. In the case where the object is a part, a part with a defect detection result of "unqualified" refers to a part that cannot meet the normal factory requirements and needs to be removed from all the tested parts, or needs to be repaired. A part with a defect detection result of "qualified" refers to a part where all detected defects are negligible, or a part where no defects were detected.

[0065] Over-detection rate: refers to the percentage of qualified (OK) parts out of the total number of parts detected after defect inspection.

[0066] Missed detection rate: refers to the percentage of defective parts that are not detected after defect inspection out of the total number of parts inspected.

[0067] Return rate: This refers to the percentage of returned batches out of the total number of delivered batches, representing the probability that a batch of products is defective and returned.

[0068] Target location: In this embodiment of the application, it refers to the shooting position configured for an object to be inspected; in this application, in order to realize the defect detection of the object to be inspected, each detection area configured for the object to be inspected can be predetermined, and the target locations that can be collected from different detection areas can be determined. The target locations are photographed and generated as point template maps, and the content on the object to be inspected that is of interest at the corresponding target location is marked by configuring the detection indication area.

[0069] Point image: In this embodiment of the application, a point image refers to an image taken at a target point for the object to be detected.

[0070] Image alignment: In this embodiment of the application, the two images after image alignment processing have the same shooting angle, and the product structures on the images can be completely aligned without obvious offset, rotation, or size difference.

[0071] The design concept of the embodiments of this application is briefly introduced below:

[0072] Under relevant technologies, artificial intelligence algorithms can be used to perform defect detection. Among the possible implementation methods under existing technologies, a combination of deep learning algorithms and rule-based judgment can be used for defect detection.

[0073] Specifically, in some possible implementations, deep learning algorithms can be used to train a defect detection model. Then, the defect detection model can be used to output defect detection information based on the point map of the object to be detected. The defect detection information includes the defect classification information, the confidence value corresponding to the classification information, and the defect location box. The training samples used to train the defect detection model are obtained by collecting images of actual objects with defects on the production line, and then classifying the defects and labeling the location boxes.

[0074] In some other possible embodiments, a deep learning algorithm can be used to train a defect detection model that achieves accurate defect detection. During the processing, the trained defect detection model is used to output defect detection results based on the imaging point bitmap of the object to be detected. The defect detection results include defect category, defect confidence, defect bounding box, and fine pixel position of defect.

[0075] Furthermore, based on the detection results obtained by processing using the above different implementation methods, rule judgment is performed. In the specific judgment process, a comprehensive judgment can be made by using the preset defect confidence threshold and defect area threshold, and finally, it is determined whether the object to be detected has a defect.

[0076] However, with existing processing methods, when acquiring point images, there is an inevitable issue of camera or object offset when different parts are captured at the same target point. This results in poor image consistency when images are taken of different objects at the same target point, leading to a high probability of false detections under the current technology framework, easily misjudging the object as defective. Furthermore, current technology trains models only based on point images taken on the current production line, making it difficult to detect defects when there are significant differences in imaging due to variations in process parameters at different target points within the object.

[0077] Furthermore, in scenarios requiring consideration of defect pixel locations, it is necessary to simultaneously label the defect category, defect location, and defect pixel location in the sample point images. This incurs significant manpower and material costs, and the training results are highly dependent on the sample data, leading to low model detection efficiency due to the low efficiency of sample data generation. Moreover, since point images may overlap with the content of other target points, interference in defect detection can occur between point images of different target points, resulting in over-detection due to false positives. In this scenario, existing methods cannot effectively address these issues.

[0078] In view of this, this application proposes a defect detection method, device, electronic device, and storage medium. The method acquires point images of the object to be detected at preset target points, and obtains the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from each point image. Specifically, for each point image acquired, the following operations are performed: a point template image associated with the corresponding target point is acquired, and the point image and the point template image are aligned to obtain a target region corresponding to the detection indication region. The point template image contains a corresponding detection indication region. A trained defect classification model is used to obtain the classification category information, severity category information, and location box corresponding to each identified candidate defect based on the point image. A trained defect segmentation model is used to identify the defect pixel location information set based on the images within each location box. Based on the classification category information, severity category information, and defect pixel location information set of each candidate defect within the target region, the sub-detection result of the object to be detected at the target point is determined.

[0079] In this way, by aligning the point images and point template images corresponding to the same target points, the consistency of image formation for point images with the same target points is ensured, reducing the processing difficulty of automated defect detection and improving the robustness of defect detection. Simultaneously, by effectively locating the target region corresponding to the detection indication region in the point image, subsequent defect detection based on the point image can focus on the target region, avoiding misjudgments caused by defects in other areas of the point image. Furthermore, considering that the defect detection model and the defect localization model are based on different sample sizes, different models are used for the recognition of localization boxes and the identification of defect pixel location information. Instead of using a single model to simultaneously identify bounding boxes and defect pixel locations, this approach reduces the difficulty of sample generation and defect detection. Furthermore, the defect detection results include information on the degree of defect detection, effectively classifying the degree of defect anomaly and providing more reliable data for determining the results. Finally, the determination of the corresponding defect detection result for the target object is based on a comprehensive assessment of the detection information for each candidate defect in the image, taking into account the combined impact of all candidate defects and improving the accuracy of defect detection.

[0080] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0081] See Figure 1The diagram shown illustrates a possible application scenario in an embodiment of this application. This application scenario diagram includes an image acquisition device 110 and a processing device 120.

[0082] It should be noted that when the image acquisition device 110 is an image acquisition device with information transmission function, the image acquisition device 110 and the processing device 120 can establish a communication connection through wired or wireless communication. When the image acquisition device 110 is an image acquisition device without information transmission function, the acquired images can be directly exported from the memory card of the image acquisition device 110 and provided to the processing device 120 for processing.

[0083] In this embodiment of the application, the image acquisition device 110 is used to acquire two-dimensional images. Specifically, it can be a terminal device camera, a two-dimensional line scan camera, an ordinary camera capable of capturing two-dimensional images, or a grayscale camera commonly used in industry.

[0084] The processing device 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It can also be a computer device with analytical processing capabilities, such as a personal computer or laptop.

[0085] The technical solution proposed in this application can detect defects in objects under inspection based on point images collected from different target locations in various application scenarios. Several possible application scenarios are illustrated below:

[0086] Scenario 1: Applied to automated inspection of parts.

[0087] In the automated defect detection of structural parts such as 3C structural components, the technical solution proposed in this application can be applied to factory workshops. After the part to be inspected is placed in a part mold, it is transported by a conveyor belt to the automated inspection workshop where defect detection is performed. A robotic arm then picks up the part from the part mold and rotates it to a preset position, enabling an image acquisition device to capture point images at different target locations on the part. The rotation operation of the robotic arm corresponds to the image acquisition device's shooting operation; that is, when the robotic arm rotates the part to reach a target location, the image acquisition device captures the corresponding point image, and at least one point image is captured at each target location.

[0088] Furthermore, for each point image acquired by the image acquisition device, the image is uploaded to the processing device. In some possible implementations, the image acquisition device can be integrated into the processing device. The processing device performs defect detection on the acquired point images and obtains sub-detection results for each point image. Finally, based on the point images acquired at each target point, the processing device determines the corresponding sub-detection results and obtains the final defect detection result for the object to be detected based on the obtained sub-detection results.

[0089] Scenario 2: Applied to the detection of defects in product appearance.

[0090] Specifically, the objects to be inspected can be assembled products that require inspection for appearance defects, such as home appliances (televisions, refrigerators, etc.), furniture (cabinets, sofas, etc.), office equipment (desks, computers, etc.), and vehicles (various types of cars, etc.).

[0091] For each assembled product, the processing device can be configured with corresponding shooting points (or target points) and point template images. Then, according to the actual processing needs, a defect classification model and a defect segmentation model can be trained for each assembled product. Based on this, when performing automated defect detection, the processing device acquires point images taken at different target points, and then performs defect detection on an assembled product based on the point images to obtain the defect detection results.

[0092] In this embodiment, the trained defect classification model and defect segmentation model used by the processing device for defect detection can be trained independently based on training samples, or they can be obtained from other devices. This application does not impose specific restrictions on this. In the following description, this application will only use the example of the processing device training its own defect classification model and defect segmentation model to explain the defect detection process in detail:

[0093] The defect detection process involved in the embodiments of this application will be described in detail below from the perspective of the processing equipment, with reference to the accompanying drawings:

[0094] See Figure 2A As shown, it is a schematic diagram of the defect detection process in an embodiment of this application. The following is in conjunction with the attached diagram. Figure 2A The defect detection process in the embodiments of this application will be described as follows:

[0095] Step 201: The processing device acquires point images of the object to be detected at each preset target point.

[0096] In this embodiment of the application, in order to meet the defect detection needs of the object to be detected, each target point can be preset for the object to be detected. The total number of target points set for the object to be detected is set according to the actual processing needs, and this application does not impose a specific limit on this. Moreover, for each target point, a corresponding point template diagram is generated for the object to be detected, and each point template diagram is marked with a corresponding detection indication area.

[0097] The processing equipment acquires point images of the object to be inspected at each preset target point, so that the defect detection of the object to be inspected can be realized based on the acquired point images.

[0098] For example, see Figure 2B As shown, it is a schematic diagram of point images collected at different target locations in an embodiment of this application. Figure 2B As shown in the diagram, assuming the object to be detected is a hexagonal prism, in order to detect defects on the surface of the hexagonal prism, target points can be set for each of the eight faces of the hexagonal prism, so that the point images collected at different target points can contain the content to be identified; combined with Figure 2B As can be seen, assuming that the detection content targeted by target point 1 is the rectangular area indicated by the thick line, then the point image collected by target point 1 will look like the content shown on its right; assuming that the detection content targeted by target point 7 is the hexagonal area indicated by the thick line, then the point image collected by target point 7 will look like the content shown on its right.

[0099] Step 202: The processing device obtains the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from the detection of each point image.

[0100] In this embodiment of the application, the processing device detects the corresponding sub-detection results for each obtained point image, and determines the defect detection result corresponding to the object to be detected based on the obtained sub-detection results.

[0101] When the processing device determines the defect detection result corresponding to the object to be detected based on each sub-detection result, if the processing device determines that there is a sub-detection result that is judged as abnormal among the sub-detection results obtained for each point image, the defect detection result corresponding to the object to be detected is judged as unqualified; and if the processing device determines that all the sub-detection results obtained for each point image are normal, the defect detection result corresponding to the object to be detected is judged as qualified.

[0102] Specifically, if the processing equipment determines that all sub-detection results are normal, that is, if the processing equipment does not obtain any abnormal sub-detection results when detecting each point image of the object to be detected, the object to be detected can be judged as qualified. Conversely, if the processing equipment determines that there are abnormal sub-detection results, that is, if the processing equipment determines that there is at least one abnormal sub-detection result when detecting each point image of the object to be detected, the object to be detected can be judged as unqualified. Among them, sub-detection results include two types: normal and abnormal, and defect detection results include two types: qualified and unqualified.

[0103] In this way, when determining the defect detection result of the object to be detected, the influence of the sub-detection results of the point images at different target points is taken into account. After comprehensively considering the sub-detection results corresponding to each point image, the defect result of the object to be detected can be effectively determined.

[0104] In this embodiment of the application, when the processing device obtains the corresponding sub-detection result for each point image, refer to... Figure 2C As shown, this is a flowchart illustrating the process of obtaining sub-detection results from a point image in an embodiment of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 2C The following describes the detection process performed by the processing device for each point image acquired in the embodiments of this application:

[0105] Step 2021: The processing device acquires the point template image associated with the corresponding target point, and obtains the target area corresponding to the detection indication area by performing image alignment processing between the point image and the point template image. The point template image is configured with the corresponding detection indication area.

[0106] In this embodiment of the application, when the processing device aligns the point image and the corresponding point template image to obtain the target area corresponding to the detection indication area, the processing device calculates the parameter transformation matrix configured for the point image when converting the point image to be aligned with the point template image; then, based on the parameter transformation matrix and the position information corresponding to the detection indication area in the point template image, the processing device determines the target area corresponding to the detection indication area in the point image.

[0107] Specifically, when calculating the parametric transformation matrix that converts the point image into a point template image, the processing device may employ any of the following implementation methods, including but not limited to:

[0108] Implementation Method 1: Calculate the parameter transformation matrix by comparing the correlation of regions.

[0109] Specifically, the processing device can mark the positioning area and its corresponding neighboring areas in the point template map, and determine the matching area corresponding to the position of the positioning area in the point image; then, using the scale-invariant feature transform (SIFT) algorithm, after determining the matching area with the highest correlation between the positioning area and the matching area, the parameter transformation matrix is ​​obtained by establishing the feature mapping relationship between the matching area and the matching area.

[0110] Method 2: Calculate the parameter transformation matrix by extracting and matching feature points.

[0111] In this embodiment of the application, the processing device may use a preset feature extraction algorithm to extract features from the point template image to obtain a first feature image, and to extract features from the point image to obtain a second feature image. After using a preset feature comparison algorithm to establish a matching relationship between each feature point in the first feature image and the second feature image, a parameter transformation matrix is ​​established based on the obtained matching relationship.

[0112] Specifically, the processing device can employ a combination of supercomputing power (SuperPoint) and superglue (SuperGlue) to achieve feature point extraction and matching. The device can use a pre-trained feature point extraction model built on the SuperPoint algorithm to extract features from a point template image, obtaining a feature map of HxWxC dimensions. Similarly, it can extract features from the point image, where H represents the height of the feature map, W represents the width, and C represents the number of channels. Then, the processing device inputs the two extracted feature maps into a pre-trained feature matching model built on the SuperGlue algorithm. After graph convolution processing, point pair matching relationships are obtained. Geometric verification is then used to filter out point pair matching relationships that do not meet geometric constraints. Based on the remaining point pair matching relationships, the parameter transformation matrix is ​​estimated.

[0113] Implementation Method 3: Calculate the parameter transformation matrix using the nearest point iteration method.

[0114] In this embodiment of the application, the processing device can use a preset edge feature extraction algorithm to obtain contour maps composed of extracted contour points for the point template map and the point image, and use a nearest neighbor search algorithm to determine the matching point pairs between the two contour maps when the pixel value error between each matching point pair is the smallest, and establish a parameter transformation matrix according to the positional relationship of each matching point pair.

[0115] Specifically, the processing device can perform edge extraction on the point image and the point template image to obtain a contour map. After constructing the error using an iterator, it establishes matching point pairs between the contour maps through a nearest neighbor search algorithm. Then, it updates the error constructed in the iterator based on the pixel value error between the matching point pairs, thereby optimizing the error towards a smaller value. Finally, it determines each matching point pair with the smallest error and then establishes a parameter transformation matrix based on the positional relationship of each matching point pair. The method of establishing the parameter transformation matrix based on the known positional relationship is a conventional technique in this field, and this application will not describe it in detail.

[0116] In this way, by using any of the three methods of calculating the parameter transformation matrix, the point image can be transformed to be aligned with the point template image.

[0117] Furthermore, after processing the parameter transformation matrix obtained by the device, the target area corresponding to the detection indication area is determined in the point image based on the parameter transformation matrix and the position information corresponding to the detection indication area in the point template image.

[0118] Specifically, after determining the parameter transformation matrix for transforming the point image to align with the point template image, the processing device can, based on the parameter transformation matrix and the position information of the detection indication area in the point template image, reversely determine the target area corresponding to the detection indication area in the point image.

[0119] In this way, the target area for detection can be determined in the point image, providing a basis for subsequent detection processes to focus only on defects within the target area and not on defects outside the target area. This can avoid misjudging the defect situation at the current target point due to defects in other areas.

[0120] Step 2022: The processing device uses a trained defect classification model to obtain the classification category information, severity category information and location box information corresponding to each candidate defect based on the point image. It then uses a trained defect segmentation model to identify the defect pixel location information set based on the image in each location box.

[0121] In this embodiment, when the processing device trains the defect classification model to be trained and obtains the trained defect classification model, the processing device can construct the defect classification model to be trained based on a preset target detection algorithm and acquire a set of sample point images. The defect classification model includes a sub-network that implements the defect severity prediction function. The set of sample point images includes sample point images collected by different types of acquisition devices, corresponding to different target points. Each sample point image is labeled with a defect classification category label, a defect severity category label, and a location box label. Then, the set of sample point images is used to perform multiple rounds of iterative training on the defect classification model to be trained until the preset convergence condition is met, thus obtaining the trained defect classification model.

[0122] It should be noted that, in the embodiments of this application, the target detection algorithm used when constructing the defect classification model to be trained can be Faster R-CNN, Cascade R-CNN, or YOLOv5. However, the deterministic detection model under related technologies cannot output defect severity category information. In this application, in order to enable the constructed defect classification model to output defect severity category information, a sub-network implementing defect severity prediction function needs to be added to the original algorithm to enable the prediction of defect severity category information. In the Faster R-CNN algorithm, the sub-network implementing defect severity prediction function is a branch added to the head structure, which generally consists of 4 convolutional layers, 1 average pooling layer, and 1 activation layer in sequence. In the YOLOv5 algorithm, the sub-network implementing defect severity prediction function is a degree branch added to the head structure, which generally consists of 4 convolutional layers.

[0123] See Figure 2D As shown, this is a schematic diagram of the composition of the defect classification model constructed in the embodiments of this application. Figure 2D As shown in the diagram, the constructed defect classification model includes the backbone network of Faster R-CNN, the neck network of Faster R-CNN, and the head network of Faster R-CNN. The backbone and neck parts adopt the network structure of related technologies. The head network adds a branch (or sub-network) to realize the defect degree prediction function. The branch to realize the defect degree prediction function is composed of 4 convolutional layers, 1 average pooling layer, and 1 activation layer in sequence.

[0124] During the training of the defect classification model, the processing device first constructs a set of sample point images for model training. This set includes sample point images acquired by different types of image acquisition devices at different target locations for the object to be detected. Each sample point image is labeled with a defect classification category label, a severity category label, and a location box label. The defect classification category may include categories such as dents and scratches, and the defect severity category may include categories such as severe and minor. Depending on the actual processing needs, the location box can be a rectangle used to select a small area where the defect is located.

[0125] In the specific training process, the processing device can use a set of sample point images to perform multiple rounds of iterative training on the defect classification model to be trained until the preset convergence condition is met, thus obtaining the trained defect classification model. The preset convergence condition can be the convergence condition for determining the convergence of the model under relevant technologies, such as the number of times the loss value is continuously lower than the first preset value reaches the second preset value, or the number of training rounds reaches the third preset value. The first preset value, the second preset value, and the third preset value are set according to the actual processing needs.

[0126] In one round of iterative training, the processing device inputs sample point images into the defect classification model to be trained, obtaining defect classification information, bounding boxes, and severity category information. Then, the cross-entropy loss function is used to calculate the loss value based on the category difference between the defect classification information and the corresponding classification label, the positional difference between the bounding box and the corresponding bounding box label, and the category difference between the severity category information and the corresponding severity category label. The model parameters are then adjusted in reverse based on the obtained loss value. The number of sample point images input in parallel during one round of training is set according to the actual processing needs, and this application does not impose specific restrictions on this.

[0127] In this embodiment of the application, when the processing device uses the defect classification model to be trained for processing, it can first extract the localization box of the sample point image in the first stage, and then perform regression analysis on the localization box to determine the defect classification category information and degree category information.

[0128] In this way, the trained defect classification model can output defect classification information, severity information, and bounding boxes, which is equivalent to simultaneously defining the degree of defect abnormality and providing more reference for defect detection. Moreover, since the generated sample point image set includes images collected by various acquisition devices at different target points, the trained defect classification model has the ability to process images collected by different acquisition devices. Furthermore, it can take into account the image content at different target points, so that the classification results are not affected by the differences in process parameters of the object to be detected at different target points, reducing the impact of imaging differences on the detection results and ensuring the detection effect of defect classification, severity, and bounding boxes.

[0129] Meanwhile, when the processing device uses the trained defect segmentation model to identify the defect pixel location information set based on the images in each location box, the processing device can obtain the trained defect location model. The defect segmentation model is constructed based on a preset segmentation algorithm and is used to locate the defect pixel location set covered by the defect. Then, the image content selected by each defect region box is input into the defect location model to obtain the defect pixel location set determined by the defect location model for each image content.

[0130] Specifically, the processing device can construct a defect segmentation model to be trained based on a preset segmentation algorithm. The segmentation algorithm used includes, but is not limited to, the object-contextual representation network (OCR Net) algorithm and the Unet semantic segmentation network algorithm.

[0131] For the Unet semantic segmentation algorithm, the image is first convolved and pooled. Assuming the image size is 224×224, it will be processed into four feature maps of different sizes: 112×112, 56×56, 28×28, and 14×14. Then, the 14×14 feature map is upsampled or deconvolved to obtain a 28×28 feature map. This 28×28 feature map is then concatenated with the previously obtained 28×28 feature map. Next, the concatenated feature map is convolved and upsampled to obtain a 56×56 feature map, which is then concatenated with the previously obtained 56×56 feature map. Similarly, the concatenated feature map is convolved and upsampled to obtain a 112×112 feature map. This process, repeated four times, yields a prediction result of 224x224, the same size as the input image.

[0132] When training the defect segmentation model, the processing device can label the positions of each defect pixel in the image content containing defects for each object to be detected, obtaining a set of defect pixel position labels for each image content. Then, based on the sample image content set constructed by similar labeling, multiple rounds of iterative training are carried out until the preset convergence condition is met, resulting in the trained defect segmentation model. In addition, the obtained trained defect segmentation model can be applied to defect segmentation scenarios for other products (or other objects to be detected) without retraining. In the actual processing, the trained defect segmentation model can directly infer the refined defect pixels, i.e., the set of defect pixel position information, based on the input defect interest region (i.e., the image content selected by the localization box).

[0133] For training the defect segmentation model, the training method under relevant technologies can be followed. The defect segmentation model to be trained is subjected to multiple rounds of iterative training using a sample image content set until the preset convergence condition is met, and the trained defect segmentation model is obtained. The obtained defect segmentation model can be used as a pre-trained model and applied to defect segmentation scenarios of other objects to be detected. The preset convergence condition can be the convergence condition for determining the convergence of the model under relevant technologies. For example, the number of times the loss value is continuously lower than the fourth preset value reaches the fifth preset value, or the number of training rounds reaches the sixth preset value. The fourth, fifth, and sixth preset values ​​are set according to the actual processing needs.

[0134] In one round of iterative training, the processing device inputs sample image content into the defect segmentation model to be trained to obtain a set of predicted defect pixel locations; then, the cross-entropy loss function is used to calculate the loss value based on the positional difference between the set of predicted defect pixel locations and the corresponding defect pixel location label; then, the model parameters are adjusted in reverse according to the obtained loss value. The number of sample image contents input in parallel during one round of training is set according to the actual processing needs, and this application does not impose specific restrictions on this.

[0135] In this way, by using the defect segmentation model, the defective part of the image content is regarded as the foreground content, and the other parts of the image content other than the defect are regarded as the background content. Foreground and background segmentation processing is performed, which enables accurate localization of defects in the image content at the pixel level.

[0136] In addition, considering that the sample size used to train the defect detection model and the defect localization model is different, using different models to identify the defect region bounding box and the defect pixel location information set can greatly reduce the difficulty of sample generation, and thus reduce the difficulty of implementing defect detection.

[0137] Step 2023: The processing device determines the sub-detection result of the object to be detected at the target point based on the classification category information, severity category information, and defect pixel location information set of each candidate defect in the target area.

[0138] In this embodiment of the application, after the processing device determines the target area corresponding to the detection indication area in the point template image in the point image, it can determine the defects that are at least partially located in the target area in the content of the corresponding positioning box, or the defects that are at least partially located in the target area in the set of corresponding defect pixel positions, as candidate defects; then, it further filters the candidate defects to select the target defects that need to be further judged, and combines the classification results and segmentation results corresponding to the target defects to realize the specific judgment of the target defects.

[0139] Specifically, the processing device can filter out target defects from among the candidate defects in the target area whose confidence value associated with the classification category information is higher than a set threshold, or whose degree category information does not belong to the preset normal degree information; and then perform the following operations for each target defect: determine the area information corresponding to the target defect based on the defect pixel location information set corresponding to the target defect, and obtain the corresponding area detection result based on the area information and the preset area detection conditions; and then determine the sub-detection result of the object to be detected at the target point based on the area detection result corresponding to each target defect.

[0140] It should be noted that, in the embodiments of this application, for the trained defect classification model, when the defect classification model outputs the defect classification category information, severity category information, and localization box, a corresponding confidence value is identified for each possible classification category information, and a corresponding confidence value is identified for each possible severity category information. Typically, the defect classification category information with the highest confidence value is determined as the defect detection classification category information, and the severity category information with the highest confidence value is determined as the defect detection severity category information.

[0141] Based on this, the processing device selects target defects from candidate defects by measuring the relationship between the confidence value associated with the defect classification information and a set threshold, as well as measuring the matching between the defect severity classification information and normal severity information. Specifically, the severity classification information can be broadly divided into normal severity information and abnormal severity information according to actual processing needs, with each of the normal severity information and abnormal severity information corresponding to at least one severity classification category.

[0142] In this embodiment of the application, if the confidence value associated with the classification category information finally determined for a certain candidate defect (let's say candidate defect A) is less than a set threshold, and the degree category information determined for candidate defect A belongs to the preset normal degree information, then candidate defect A is not the target defect; otherwise, candidate defect A can be determined as the target defect.

[0143] For example, assuming the severity category information includes four types: mild, moderate, severe, and extreme, then, based on the actual processing needs, the severity category information of "mild" can be determined as normal severity information, and the severity categories of "moderate", "severe", and "extreme" can be determined as abnormal severity information.

[0144] For example, suppose the threshold for the classification information is set to 85%, and the classification information includes scratches, dents, and missing parts; suppose the classification information obtained for candidate defect 1 is: scratches 60% (i.e., 60% of the time it is a scratch); dents 45%; missing parts 10%, then the classification information obtained for candidate defect 1 can be: scratches; suppose the severity classification information obtained for candidate defect 1 is moderate, then although the classification information of candidate defect 1 is not higher than the set threshold, because the severity classification information corresponding to candidate defect 1 is "moderate", it does not belong to the preset normal severity information, so candidate defect 1 is selected as the target defect.

[0145] When the processing device performs area detection on a target defect, it calculates the actual physical area of ​​the current target defect based on the segmented defect pixel location information set and the conversion ratio between pixel position and area, thus obtaining the corresponding area information. Then, based on the obtained area information and the preset area detection conditions, the corresponding area detection result is obtained. Since the image acquisition device usually provides a pixel precision value when acquiring point images, that is, the physical quantity length corresponding to a pixel, the square of the physical quantity length corresponding to a pixel can be used as the area corresponding to a pixel.

[0146] Specifically, the processing device can obtain an area threshold configured for the area information of the target defect based on the region information to which the positioning box corresponding to the target defect belongs in the point image and the classification information corresponding to the target defect. Then, when it is determined that the area information of the target defect reaches the preset area threshold, the corresponding area detection result is judged as an area undetermined anomaly, and when it is determined that the area information of the target defect does not reach the preset area threshold, the area detection result is judged as an area detection normal.

[0147] It should be noted that, in this embodiment of the application, different area thresholds can be set for defects in different regions of the point image and defects of different classification categories, according to actual processing needs. The processing device can mark high-incidence areas of defects on the object to be detected based on historical detection results, and / or mark over-detection areas caused by texture differences between different batches of objects to be detected; then, the areas corresponding to high-incidence areas of defects can be set as sensitive areas and the areas corresponding to over-detection areas can be set as shielded areas in the point template image in advance, and the confidence threshold and area threshold within the preset areas can be dynamically adjusted to solve the problems of over-detection and missed detection of defects.

[0148] When determining the area detection result corresponding to the target defect, the processing device determines the area threshold corresponding to the target defect, and then determines the area detection result of the target defect based on the size relationship between the area information of the target object and the corresponding area threshold. The area detection result may be either the positive area detection length or the area undetermined anomaly.

[0149] In this way, by specifically obtaining the area threshold corresponding to the target defect, it is possible to effectively define the defect area while taking into account the classification information of the target defect and the region to which the target defect belongs in the point image. This is equivalent to realizing the judgment of the target defect from the perspective of the defect area.

[0150] When considering the area detection results for target defects, the processing device determines the sub-detection results of the object to be detected at the target location based on the area detection results corresponding to each target defect. The processing device can obtain the total defect count threshold set for each category of information and count the total number of target defects with undetermined area detection results among the target defects belonging to the same category of information. If it is determined that the total number of target defects corresponding to a category of information is higher than the corresponding total defect count threshold, the sub-detection results of the object to be detected at the target location are judged as detection anomalies. If it is determined that the total number of target defects corresponding to each category of information is not higher than the corresponding total defect count threshold, the sub-detection results of the object to be detected at the target location are judged as detection normal.

[0151] Specifically, when determining the sub-detection results of the object to be detected at the target location, the processing equipment can comprehensively judge the results by combining the area detection results of each target defect within the target area. In the specific judgment process, firstly, the classification information corresponding to each target defect within the target area is determined, and the total number of defects thresholds pre-configured for each classification information is obtained. Then, the total number of target defects belonging to each classification information with area detection results of undetermined abnormality is counted. When it is determined that the total number of target defects under a certain classification information reaches the corresponding total number of defects threshold, the sub-detection result of the object to be detected at the target location can be judged as a detection abnormality. And when it is determined that the total number of target defects corresponding to each classification information is not higher than the corresponding total number of defects threshold, the sub-detection result of the object to be detected at the target location can be judged as a detection normality.

[0152] In this way, when determining the sub-detection results of the object to be detected at the target point, a comprehensive judgment is made based on the area detection results of multiple target defects. This is equivalent to introducing a multi-instance level judgment in the defect detection process, taking into account the superimposed influence of the same type of target defects, and ensuring the defect detection effect.

[0153] In addition, in the embodiments of this application, when making a comprehensive judgment on multiple instances to determine the sub-detection result, other considerations such as defect length can also be taken into account.

[0154] Specifically, after the processing device obtains the corresponding area detection results based on the area information and preset area detection conditions, before determining the sub-detection results of the object to be detected at the target point based on the area detection results corresponding to each target defect, it can also determine the positioning box corresponding to the target defect, and determine the length information corresponding to the target defect based on the number of pixels that construct the edge of the positioning box and the mapping relationship between pixels and size; then, based on the length information and preset length detection conditions, the corresponding length detection results are obtained.

[0155] Similar to the method for determining the area detection results described above, the processing device can pre-set different length thresholds for defects in different areas of the point image and defects corresponding to different classification categories, based on actual processing needs.

[0156] When determining the length detection result corresponding to the target defect, the physical length (or length information) of the target defect is calculated based on the ratio of pixel points to millimeters and the total number of edge pixels of the target defect's positioning box. If the length information is determined to be greater than the corresponding set length threshold, the length detection result of the target defect is determined to be an abnormal length undetermined. If the length information is determined to be less than the corresponding set length threshold, the length detection result of the target defect is determined to be a normal length detection.

[0157] Furthermore, the processing device can obtain the total defect count threshold set for each category of information, and separately count the total number of target defects whose length detection results are of undetermined length anomalies among the target defects belonging to the same category of information; then, if it is determined that the total number of target defects of undetermined length anomalies corresponding to a certain category of information exceeds the corresponding total defect count threshold, the sub-detection result of the object to be detected at the target point is judged as a detection anomaly; and if it is determined that the total number of target defects of undetermined length anomalies corresponding to each category of information does not exceed the corresponding total defect count threshold, the sub-detection result of the object to be detected at the target point is judged as a detection normal.

[0158] In this way, by comprehensively considering the length information of different target defects, the superposition effect of the length information of different target defects is taken into account, which helps to improve the defect detection effect.

[0159] Based on this, when finally determining the sub-detection result of the object to be detected at the target point, if the processing device determines that the sub-detection result is a detection anomaly based on the area detection result, or if the processing device determines that the sub-detection result is a detection anomaly based on the length detection result, the sub-detection result of the object to be detected at the target point can be determined to be a detection anomaly.

[0160] The following describes the defect detection process in this application embodiment using the quality inspection of 3C structural components as an example. Specifically, 3C structural components can be mobile phone camera brackets, mobile phone mid-frames, SIM card slots, computer casings, smartwatch casings, etc.

[0161] See Figure 3 As shown, it is a schematic diagram of the general process of defect detection in the embodiments of this application. The following is in conjunction with the attached diagram. Figure 3 The general process for defect detection in the embodiments of this application is described below:

[0162] Combined with appendix Figure 3 As can be seen from the content shown, the defect detection process implemented in this application embodiment can be specifically implemented by four functional modules, namely: image alignment module, defect classification module, defect foreground and background segmentation module, and rule determination module.

[0163] In the specific processing process, the processing device first uses the image alignment module to perform image alignment processing between the point image at the target point and the corresponding point template image, and calculates the parameter transformation matrix when the point image is transformed to be aligned with the point template image;

[0164] Then, the processing device uses the defect classification module to call the trained defect classification model to infer defect information, which includes defect classification information, severity information, and location bounding boxes.

[0165] It should be noted that in some possible embodiments of this application, the processing device can train a defect classification model for each target point separately; then, after obtaining the point image corresponding to a certain target point, the defect classification model corresponding to the target location can be used to process the point image. In the training process of the defect classification model, sample images collected at the corresponding target points can be used for training. The related processing is the same as the method of training the defect classification model mentioned above, and will not be described in detail here.

[0166] For example, if the casing and keyboard of a product are made of different materials, the same algorithm can be used to build different defect classification models, and different data can be used to train the different defect classification models.

[0167] In this way, by training different defect classification models for different target locations, it is possible to effectively detect point images at different target locations when the process parameters vary greatly, thereby improving the detection effect of defects at different target locations.

[0168] Furthermore, the processing device, with the help of the defect foreground and background segmentation module, calls the trained defect segmentation model and obtains a set of fine pixel position information based on the image content in the localization box.

[0169] Finally, the processing device, with the help of the rule determination module, makes judgments based on the obtained defect information and pixel location information set, using single-instance level rules and multi-instance level rules. The single-instance level rules make judgments on a single piece of detection information, mainly by judging the target area, shielded area and sensitive area, classification category information, confidence value, degree category, length information, area information, etc. The multi-instance level rules make comprehensive judgments on multiple pieces of detection information, for example, judging the number of defects that meet the length detection condition and the area detection condition among multiple defects. Here, one instance represents one defect.

[0170] See Figure 4 As shown, this is a schematic diagram of the overall process involved in defect detection in an embodiment of this application. The following is a detailed explanation in conjunction with the attached diagram. Figure 4 The following describes the processing steps involved in implementing the defect detection process in the embodiments of this application:

[0171] Combined with appendix Figure 4 As shown, when performing defect detection, the processing device first inputs four pieces of information into the image alignment module: the current point image, the point number of the target point corresponding to the current point image, the point template image corresponding to the current target point, and the detection area indicator area in the point template image. The output is the parameter transformation matrix that needs to be used when the point image is converted to be aligned with the point template image.

[0172] Secondly, the processing device sends the target location number and the current location image to the defect classification module to obtain defect information output by the trained defect classification model. The defect information includes the defect classification category information, the local bounding box, the confidence level, and the degree category information.

[0173] Next, the processing device determines the region of interest based on the current point image and the bounding box, and then inputs the region of interest into the trained defect segmentation model to calculate the fine pixels of the current defect, that is, to obtain a set of pixel location information, where... Figure 4 The N represents the total number of identified defects. For each defect, there is corresponding defect information. M represents the total number of remaining defects after filtering out negligible defects according to rules.

[0174] Finally, the processing device summarizes the defect information of the N defects and sends it to the rule determination module for judgment. The rule determination module makes a judgment, and the following seven rules can be used to achieve parallel judgment in the specific execution process.

[0175] Rule 1 is the focus area rule, used to determine whether a defect is located within the target area. Specifically, based on the detection indicator area in the point template image and the alignment transformation parameters (or parameter transformation matrix) corresponding to the current point image, the focus area (or target area) in the current point image is calculated. Then, it is determined whether the current defect is within the focus area of ​​the current point image. If so, the defect is determined to be a candidate defect; otherwise, the defect is determined to be a negligible defect.

[0176] Rule 2 is the confidence rule, used to determine whether the confidence value of a defect reaches a confidence threshold. After obtaining the corresponding confidence threshold based on the target location and the defect classification information, it is determined whether the confidence value of the defect classification meets the corresponding confidence threshold. If so, the current defect is identified as a target defect; otherwise, the current defect is identified as an negligible defect.

[0177] Rule three is the area rule, used to determine whether the area information reaches the corresponding area threshold. Specifically, based on the ratio between the pixel points and the area input by the system, the true physical area of ​​the current defect is calculated, and whether it is an area-pending anomaly is determined according to the area threshold.

[0178] Rule four is the length rule, used to determine whether the length information meets the corresponding length threshold. Specifically, based on the ratio of pixel points to millimeters input by the system and the length of the defect's location box, the physical length (or length information) of the current defect is calculated; then, based on the relationship between the length information and the corresponding length threshold, it is determined whether it is a length-undetermined anomaly.

[0179] Rule 5 uses the severity category information in the defect information to determine whether the severity category information matches the severity range. Specifically, the severity of the defect can be defined according to the customer's quality requirements for 3C parts, and then the acceptable severity range (or normal severity information) can be determined. When the severity category information is determined to be within the severity range, it can be regarded as a negligible defect, and when the severity category information is determined to be outside the severity range, it is determined to be a non-negligible defect, that is, a target defect.

[0180] Rule 6 is based on the shielded area and sensitive area to determine whether a defect is located in a sensitive area or a shielded area. Through the system's preset shielded area and sensitive area, high-incidence areas of defects on the product (3C structural components) or over-detected areas caused by defect batch textures can be marked. Based on this, shielded areas and sensitive areas can be set in a targeted manner, and the problems of over-detection and missed detection of defects can be solved by dynamically adjusting the confidence threshold, area threshold, length threshold, etc. of defects in the shielded area and sensitive area.

[0181] Rule 7 is a multi-instance level judgment rule, used to achieve comprehensive judgment of multiple instances. It determines whether a defect of a certain type can be ignored based on whether the number of defects reaching a certain length threshold or the number of defects reaching a certain area threshold.

[0182] Furthermore, based on the overall rule determination, it is possible to detect negligible and non-negligible defects in the image of the target location. If a non-negligible defect is found in a certain image of the target location, that is, if the sub-detection result of the part at a certain target location is determined to be abnormal, then the part can be determined to be unqualified and recorded as NG; otherwise, if it is determined that the defects detected at each target location are negligible defects, then the part can be determined to be qualified and recorded as OK.

[0183] Thus, by employing the automated defect detection method for 3C structural components proposed in this application, general artificial intelligence algorithms can be used. Through sequential image alignment, defect classification, defect foreground / background segmentation, and rule-based processing, the consistency of image quality is ensured for point images acquired at the same target location for the same type of object to be inspected. Furthermore, because the defect classification model is trained based on point images from different partitions of the structural component (i.e., different target locations), the impact of differences in the component's partitioning processes on detection is reduced, thereby improving defect detection accuracy and robustness.

[0184] Furthermore, based on Tables 1-5, a comparative analysis is conducted on the performance differences between the defect detection method proposed in this application and existing defect detection methods. The manual missed detection rate is derived from production line observation statistics, while the manual over-detection rate is not included in the statistics.

[0185] Table 1

[0186] This invention technology 0.0015％ 16.4％ 0.35％ 4s Previous inventions 0.01％ 22％ - - artificial 0.1％ - 2.48％ 30s

[0187] Table 1 is a comparison table of the differences in detection effects when performing defect detection on mobile phone camera brackets in the embodiments of this application. According to the contents shown in Table 1, it can be seen that the defect detection method proposed in this application can greatly reduce the missed detection rate of key defects, the batch rejection rate, and the single-piece quality inspection time.

[0188] Table 2

[0189] The invention technology 0.5％ 20％ 5s artificial 5％ - 120s

[0190] Table 2 is a comparison table of the differences in detection effects when performing defect detection on the mobile phone frame in the embodiments of this application. According to the contents shown in Table 2, it can be seen that the defect detection method proposed in this application can reduce the missed detection rate and the single-piece quality inspection time.

[0191] Table 3

[0192] The invention technology 0.5％ 5％ 3s artificial 2％ - 15s

[0193] Table 3 is a comparison table of the differences in detection effects when performing defect detection on the SIM card slot in the embodiments of this application. According to the contents shown in Table 3, the defect detection method proposed in this application can greatly reduce the missed detection rate and the single-piece quality inspection time.

[0194] Table 4

[0195] The invention technology 2.3％ 20％ 15s artificial 15％ - 60s

[0196] Table 4 is a comparison table of the differences in detection effects when performing defect detection on computer casings in the embodiments of this application. As can be seen from the content shown in Table 4, the defect detection method proposed in this application can greatly reduce the missed detection rate and the single-piece quality inspection time.

[0197] Table 5

[0198] The invention technology 0.001％ 2％ 3s artificial 0.3％ - 10s

[0199] Table 5 is a comparison table of the differences in detection effects when performing defect detection on the smartwatch casing in the embodiments of this application. As can be seen from the content shown in Table 5, the defect detection method proposed in this application can greatly reduce the missed detection rate and the single-piece quality inspection time.

[0200] Based on the same inventive concept, see [reference] Figure 5 As shown, this is a schematic diagram of the logical structure of a defect detection device in an embodiment of this application. The defect detection device 500 includes, wherein,

[0201] The acquisition unit 501 is used to acquire point images of the object to be detected at each preset target point.

[0202] Detection unit 502 is used to obtain the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from detecting each point image separately. Specifically, for each point image acquired, the following operations are performed:

[0203] Obtain a point template image associated with the corresponding target point, and determine the target area corresponding to the detection indication area in the point image by performing image alignment processing between the point image and the point template image. The point template image is configured with the corresponding detection indication area.

[0204] Using a trained defect classification model, based on point images, the classification category information, severity category information and localization box of each candidate defect are obtained. Using a trained defect segmentation model, based on the images in each localization box, the defect pixel location information set is obtained respectively.

[0205] Based on the classification category information, severity category information, and defect pixel location information of each candidate defect within the target area, the sub-detection result of the object to be detected at the target point is determined.

[0206] Optionally, when training the defect classification model, the device further includes a training unit 503, which is specifically used for:

[0207] A defect classification model to be trained is constructed based on a preset target detection algorithm, and a set of sample point images is obtained. The defect classification model includes a sub-network that implements the defect severity prediction function. The set of sample point images includes sample point images collected by different types of acquisition devices and corresponding to different target points. Each sample point image is labeled with a defect classification category label, a defect severity category label, and a localization box label.

[0208] The defect classification model to be trained is trained through multiple rounds of iterative training using a set of sample point images until the preset convergence condition is met, thus obtaining the trained defect classification model.

[0209] Optionally, when determining the sub-detection result of the object to be detected at the target point based on the classification category information, severity category information, and defect pixel location information set of each candidate defect within the target area, the detection unit 502 is used for:

[0210] Among the candidate defects in the target area, select the target defects whose confidence value associated with the classification category information is higher than the set threshold, or whose degree category information does not belong to the preset normal degree information.

[0211] For each target defect, perform the following operations: Based on the set of defect pixel location information corresponding to the target defect, determine the area information corresponding to the target defect, and based on the area information and preset area detection conditions, obtain the corresponding area detection result;

[0212] Based on the area detection results corresponding to each target defect, the sub-detection results of the object to be detected at the target point are determined.

[0213] Optionally, after obtaining the corresponding area detection results based on area information and preset area detection conditions, and before determining the sub-detection results of the object to be detected at the target point based on the area detection results corresponding to each target defect, the detection unit 502 is further configured to:

[0214] Determine the defect region bounding box corresponding to the target defect, and based on the number of pixels at the edge of the defect region bounding box and the mapping relationship between pixels and size, determine the length information corresponding to the target defect;

[0215] Based on the length information and preset length detection conditions, the corresponding length detection result is obtained.

[0216] Optionally, when obtaining the corresponding area detection result based on preset area detection conditions, the detection unit 502 is used to:

[0217] Based on the region information to which the location box corresponding to the target defect belongs in the point image, and the classification information corresponding to the target defect, obtain the area threshold configured for the area information of the target defect.

[0218] When the area information of the target defect is determined to reach the preset area threshold, the corresponding area detection result is judged as an area undetermined anomaly. When the area information of the target defect is determined to not reach the preset area threshold, the area detection result is judged as an area detection normal.

[0219] Optionally, when determining the sub-detection result of the object to be detected at the target point based on the area detection result corresponding to each target defect, the detection unit 502 is used for:

[0220] Obtain the threshold for the total number of defects set for each category information, and count the total number of target defects whose area detection results are undetermined anomalies among the target defects belonging to the same category information.

[0221] If the total number of target defects corresponding to a certain category of classification information is higher than the corresponding total defect threshold, the sub-detection result of the object to be detected at the target location is judged as abnormal; and if the total number of target defects corresponding to each category of classification information is not higher than the corresponding total defect threshold, the sub-detection result of the object to be detected at the target location is judged as normal.

[0222] Optionally, when determining the target region corresponding to the detection indication region in the point image by performing image alignment processing between the point image and the point template image, the detection unit 502 is used to:

[0223] Calculate the parameter transformation matrix configured for the point image when converting it to be aligned with the point template image;

[0224] Based on the parameter transformation matrix and the location information corresponding to the detection indication area in the point template image, the target area corresponding to the detection indication area is determined in the point image.

[0225] Optionally, when using a trained defect segmentation model to identify the defect pixel location information set based on the images in each localization box, the detection unit 502 is used for:

[0226] Obtain the trained defect segmentation model, wherein the defect localization model is constructed based on a preset segmentation algorithm and is used to locate the set of defect pixel locations covered by defects;

[0227] The image content selected by each defect region is input into the defect localization model to obtain the set of defect pixel positions determined by the defect localization model for each image content.

[0228] Optionally, when obtaining the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from detecting each point image, the detection unit 502 is used for:

[0229] If, among the sub-detection results obtained from detecting images at each point, there is a sub-detection result that is judged as an abnormality, then the defect detection result corresponding to the object to be detected is judged as unqualified; and,

[0230] When the sub-detection results obtained from the detection of each point image are all normal, the defect detection result corresponding to the object to be detected is judged as qualified.

[0231] Having introduced the defect detection method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0232] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0233] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device, see reference. Figure 6 As shown, this is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 600 may include at least a processor 601 and a memory 602. The memory 602 stores program code, which, when executed by the processor 601, causes the processor 601 to perform any of the aforementioned defect detection steps.

[0234] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps of rendering map data according to the various exemplary embodiments of this application described above. For example, the processor may perform actions such as... Figure 2A , Figure 2C The steps are shown in the figure.

[0235] The following reference Figure 7 To describe a computing device 700 according to this embodiment of the present application. For example... Figure 7 As shown, the computing device 700 is presented in the form of a general-purpose computing device. The components of the computing device 700 may include, but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including storage unit 702 and processing unit 701).

[0236] Bus 703 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the various bus architectures.

[0237] Storage unit 702 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 7021 and / or cache memory 7022, and may further include read-only memory (ROM) 7023.

[0238] Storage unit 702 may also include a program / utility 7025 having a set (at least one) program module 7024, such program module 7024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0239] The computing device 700 can also communicate with one or more external devices 704 (e.g., keyboard, pointing device, etc.), one or more devices that enable objects to interact with the computing device 700, and / or any device that enables the computing device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 705. Furthermore, the computing device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 706. As shown, network adapter 706 communicates with other modules used in the computing device 700 via bus 703. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0240] Based on the same inventive concept as the above-described method embodiments, various aspects of defect detection provided in this application can also be implemented as a program product, which includes program code. When the program product is run on an electronic device, the program code causes the electronic device to perform the steps in the defect detection methods according to various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 2A , 2C The steps are shown in the figure.

[0241] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0242] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0243] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A defect detection method, characterized in that, include: Acquire point images of the object to be detected at each preset target point; Based on the sub-detection results obtained from detecting each point image, the defect detection result corresponding to the object to be detected is obtained. Specifically, for each point image acquired, the following operations are performed: A point template image associated with the corresponding target point is obtained, and the point image and the point template image are image aligned. The target area corresponding to the detection indication area is determined in the point image. The point template image is configured with the corresponding detection indication area. When adjusting the point template image, based on historical detection results, the over-kill area caused by defect batch texture is set as a shielded area and the high-incidence defect area is set as a sensitive area. For the shielded area and the sensitive area, the confidence threshold and area threshold of each category information of the defect are dynamically adjusted. Using a trained defect classification model, based on the point images, the classification category information, severity category information and location box corresponding to each identified candidate defect are obtained. Using a trained defect segmentation model, based on the images in each location box, the defect pixel location information set is identified respectively. Among the candidate defects within the target area, target defects whose classification category information is associated with a confidence value higher than the corresponding confidence threshold, or whose severity category information does not belong to the preset normal severity information, are selected. For each target defect, the following operations are performed: based on the defect pixel location information set corresponding to the target defect, the area information corresponding to the target defect is determined, and based on the size relationship between the area information and the corresponding area threshold, the corresponding area detection result is obtained; based on the area detection result corresponding to each target defect, the sub-detection result of the object to be detected at the target point is determined.

2. The method as described in claim 1, characterized in that, The defect classification model obtained through training includes: A defect classification model to be trained is constructed based on a preset target detection algorithm, and a set of sample point images is obtained. The defect classification model includes a sub-network that realizes the defect severity prediction function. The set of sample point images includes sample point images collected by different types of acquisition devices and corresponding to different target points. Each sample point image is labeled with a defect classification category label, a defect severity category label, and a location box label. The defect classification model to be trained is trained through multiple rounds of iterative training using the sample point image set until the preset convergence condition is met, thus obtaining the trained defect classification model.

3. The method as described in claim 1, characterized in that, After obtaining the corresponding area detection result based on the area information and preset area detection conditions, before determining the sub-detection result of the object to be detected at the target point based on the area detection result corresponding to each target defect, the method further includes: Determine the defect region bounding box corresponding to the target defect, and based on the number of pixels constructing the edge of the defect region bounding box and the mapping relationship between pixels and size, determine the length information corresponding to the target defect; Based on the length information and the preset length detection conditions, the corresponding length detection result is obtained.

4. The method as described in claim 1, characterized in that, The process of obtaining the corresponding area detection result based on the relationship between the area information and the corresponding area threshold includes: Based on the region information to which the positioning box corresponding to the target defect belongs in the point image, and the classification category information corresponding to the target defect, obtain the area threshold configured for the area information of the target defect; When the area information of the target defect is determined to reach a preset area threshold, the corresponding area detection result is judged as an area undetermined anomaly; when the area information of the target defect is determined not to reach the preset area threshold, the area detection result is judged as an area detection normal.

5. The method as described in claim 4, characterized in that, The step of determining the sub-detection result of the object to be detected at the target point based on the area detection result corresponding to each target defect includes: Obtain the threshold for the total number of defects set for each category information, and count the total number of target defects whose area detection results are undetermined anomalies among the target defects belonging to the same category information. If the total number of target defects corresponding to a certain category of classification information is higher than the corresponding total defect threshold, the sub-detection result of the object to be detected at the target point is determined to be abnormal; and if the total number of target defects corresponding to each category of classification information is not higher than the corresponding total defect threshold, the sub-detection result of the object to be detected at the target point is determined to be normal.

6. The method according to any one of claims 1-5, characterized in that, The step of aligning the point image with the point template image to determine the target region corresponding to the detection indication region in the point image includes: Calculate the parameter transformation matrix configured for the point image when converting it to be aligned with the point template image; Based on the parameter transformation matrix and the location information corresponding to the detection indication area in the point template image, a target area corresponding to the detection indication area is determined in the point image.

7. The method according to any one of claims 1-5, characterized in that, The trained defect segmentation model is used to identify a set of defect pixel location information based on the images in each localization box, including: Obtain a trained defect segmentation model, wherein the defect segmentation model is constructed based on a preset segmentation algorithm and is used to locate the set of defect pixel locations covered by defects; The image content selected by each defect region is input into the defect segmentation model to obtain the set of defect pixel positions determined by the defect segmentation model for each image content.

8. The method according to any one of claims 1-5, characterized in that, The method of obtaining the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from the image at each point includes: If, among the sub-detection results obtained from detecting each of the image points, there is a sub-detection result that is judged as an abnormality, then the defect detection result corresponding to the object to be detected is judged as unqualified; and, If all sub-detection results obtained from the detection of each point image are normal, the defect detection result corresponding to the object to be detected is judged as qualified.

9. A defect detection device, characterized in that, include: The acquisition unit is used to acquire point images of the object to be detected at each preset target point. The detection unit is used to obtain the defect detection result corresponding to the object to be detected based on the sub-detection results obtained from the detection of each point image. Specifically, for each point image acquired, the following operations are performed: A point template image associated with the corresponding target point is obtained, and the point image and the point template image are image aligned. The target area corresponding to the detection indication area is determined in the point image. The point template image is configured with the corresponding detection indication area. When adjusting the point template image, based on historical detection results, the over-kill area caused by defect batch texture is set as a shielded area and the high-incidence defect area is set as a sensitive area. For the shielded area and the sensitive area, the confidence threshold and area threshold of each category information of the defect are dynamically adjusted. Using a trained defect classification model, based on the point images, the classification category information, severity category information and location box corresponding to each identified candidate defect are obtained. Using a trained defect segmentation model, based on the images in each location box, the defect pixel location information set is identified respectively. Among the candidate defects within the target area, target defects whose classification category information is associated with a confidence value higher than the corresponding confidence threshold, or whose severity category information does not belong to the preset normal severity information, are selected. For each target defect, the following operations are performed: based on the defect pixel location information set corresponding to the target defect, the area information corresponding to the target defect is determined, and based on the size relationship between the area information and the corresponding area threshold, the corresponding area detection result is obtained; based on the area detection result corresponding to each target defect, the sub-detection result of the object to be detected at the target point is determined.

10. The apparatus as claimed in claim 9, characterized in that, When training to obtain the defect classification model, the device further includes a training unit, which is specifically used for: A defect classification model to be trained is constructed based on a preset target detection algorithm, and a set of sample point images is obtained. The defect classification model includes a sub-network that realizes the defect severity prediction function. The set of sample point images includes sample point images collected by different types of acquisition devices and corresponding to different target points. Each sample point image is labeled with a defect classification category label, a defect severity category label, and a location box label. The defect classification model to be trained is trained through multiple rounds of iterative training using the sample point image set until the preset convergence condition is met, thus obtaining the trained defect classification model.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the defect detection method as described in any one of claims 1-8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the defect detection method as described in any one of claims 1-8.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the defect detection method as described in any one of claims 1-8.

Images