Artificial intelligence-based product defect identification method, related apparatus, and media

Abstract

The present disclosure provides an artificial intelligence-based product defect identification method, related devices and media. The product defect identification method comprises: obtaining a to-be-detected image and a reference image of a product; generating a first feature map of the to-be-detected image and a second feature map of the reference image; determining a calibration matrix based on the first feature map and the second feature map; calibrating the first feature map using the calibration matrix to obtain a third feature map; and identifying defects of the product from the third feature map. The present embodiment reduces the detection time of product defects and improves detection efficiency. The present embodiment can be applied to various scenarios such as industrial automation, artificial intelligence, and Internet of Things.

Description

Technical Field

[0001] This disclosure relates to the field of industrial automation, and in particular to a product defect identification method, related apparatus and medium based on artificial intelligence. Background Technology

[0002] In industrial automated quality inspection, due to the complexity of industrial parts, multiple angles are typically used to photograph them, and defects are then detected from these images. However, the robotic arm of the camera can experience some shaking, which can cause image shakiness. Furthermore, industrial cameras generally use fixed-focus lenses, meaning that only a small portion of the Region of Interest (ROI) may be sharp in the captured image, while the rest is blurry, allowing for defect identification only within the ROI. Therefore, after capturing part images, it is generally necessary to calibrate them to reduce the aforementioned shaking and ROI mismatch issues, and then identify defects from the calibrated images.

[0003] In existing technologies, image calibration and defect identification from calibrated images are two separate processes. The calibration process must be completed before the defect identification process from the calibrated image can be performed, which is time-consuming. Summary of the Invention

[0004] This disclosure provides an artificial intelligence-based product defect identification method, related apparatus, and medium, which can reduce the detection time of product defects and improve detection efficiency.

[0005] According to one aspect of this disclosure, an artificial intelligence-based product defect identification method is provided, comprising:

[0006] Acquire the product's image to be inspected and a reference image;

[0007] Generate a first feature map of the image to be detected and a second feature map of the reference image;

[0008] Based on the first feature map and the second feature map, determine the calibration matrix;

[0009] The first feature map is calibrated using the calibration matrix to obtain the third feature map;

[0010] Defects in the product are identified from the third feature map.

[0011] According to one aspect of this disclosure, an artificial intelligence-based product defect identification device is provided, comprising:

[0012] The acquisition unit is used to acquire the product's image to be inspected and a reference image;

[0013] The generation unit is used to generate a first feature map of the image to be detected and a second feature map of the reference image;

[0014] The determining unit is configured to determine a calibration matrix based on the first feature map and the second feature map;

[0015] A calibration unit is used to calibrate the first feature map using the calibration matrix to obtain a third feature map;

[0016] The first identification unit is used to identify defects in the product from the third feature map.

[0017] Optionally, the determining unit is specifically used for:

[0018] Obtain a set of matching feature point pairs, wherein the set of matching feature point pairs includes multiple matching feature point pairs, each of the matching feature point pairs includes a first feature point and a second feature point, the first feature point comes from the first feature map, and the second feature point comes from the second feature map and corresponds to the first feature point;

[0019] The calibration matrix is ​​determined based on the set of matching feature point pairs.

[0020] Optionally, the determining unit is specifically used for:

[0021] Based on the first feature map and the second feature map, a correlation score matrix is ​​obtained, wherein each feature point in the first feature map corresponds to a column of the correlation score matrix, and each feature point in the second feature map corresponds to a row of the correlation score matrix. The elements in the correlation score matrix are equal to the correlation score of the feature point in the first feature map corresponding to the column of the element and the correlation score of the feature point in the second feature map corresponding to the row of the element.

[0022] Select multiple first feature points in the first feature map;

[0023] For each selected first feature point, based on the correlation score matrix, obtain the second feature point corresponding to the first feature point in the second feature map.

[0024] Optionally, the determining unit is specifically used for:

[0025] In the column corresponding to the first feature point selected in the correlation score matrix, a sliding window is set, wherein the sliding window includes a first sliding window element, a second sliding window element, a third sliding window element, and a fourth sliding window element. Each of the first, second, third, and fourth sliding window elements covers one correlation score in the column. The second sliding window element is below and adjacent to the first sliding window element. The number of rows between the third sliding window element and the second sliding window element is the number of columns of the second feature map minus 2. The fourth sliding window element is below and adjacent to the third sliding window element.

[0026] The sliding window is made to slide in the column corresponding to the first feature point, and the second feature point is obtained based on the four elements that fall in the sliding window during the sliding process.

[0027] Optionally, the determining unit is specifically used for:

[0028] For each sliding position during the sliding process of the sliding window, determine the sum of the four correlation scores falling within the sliding window;

[0029] The second feature point is obtained based on the four feature points corresponding to the four largest correlation scores in the second feature map.

[0030] Optionally, the determining unit is specifically used for:

[0031] Take any one of the four feature points as the first reference feature point, the feature point in the same row as the first reference feature point is the second reference feature point, the feature point in the same column as the first reference feature point is the third reference feature point, and the feature point in the four feature points that is neither in the same row nor in the same column as the first reference feature point is the fourth reference feature point.

[0032] A coordinate system is established with the center of the first reference feature point as the origin, the direction from the origin to the center of the second reference feature point as the positive direction of the horizontal axis, the direction from the origin to the center of the third reference feature point as the positive direction of the vertical axis, and the standard side length of the feature point as the unit of the horizontal and vertical axes.

[0033] Based on the correlation scores of the first reference feature point, the second reference feature point, the third reference feature point, and the fourth reference feature point, the center coordinates of the second feature point are determined;

[0034] The second feature point is obtained based on the center coordinates of the second feature point and the standard side length of the feature point.

[0035] Optionally, the determining unit is specifically used for:

[0036] The abscissa of the center is determined based on the sum of the correlation scores between the second reference feature point and the fourth reference feature point, and the sum of the correlation scores between the first reference feature point and the third reference feature point.

[0037] The ordinate of the center is determined based on the sum of the correlation scores between the third and fourth reference feature points and the sum of the correlation scores between the first and second reference feature points.

[0038] Optionally, the determining unit is specifically used for:

[0039] After obtaining the correlation score matrix based on the first feature map and the second feature map, the first feature map and the second feature map are input into the self-attention model to generate the first correlation score adjustment value;

[0040] The first feature map and the second feature map are input into the interaction attention model to generate a second correlation score adjustment value.

[0041] The correlation score matrix is ​​adjusted using the first correlation score adjustment value and the second correlation score adjustment value.

[0042] Optionally, the determining unit is specifically used for:

[0043] Perform a first number of first processes, the first process comprising: extracting a second number of matching feature point pairs from the set of matching feature point pairs; calculating a candidate calibration matrix based on each of the matching feature point pairs; determining a first error of the matching feature point pairs in the set of matching feature point pairs on the candidate calibration matrix; and determining a third number of matching feature point pairs whose first error is less than a first threshold.

[0044] The candidate calibration matrix calculated by the first process with the largest third number in the first process is used as the determined calibration matrix.

[0045] Optionally, the first feature map includes multiple first feature maps with different feature extraction ratios, the second feature map includes multiple second feature maps with different feature extraction ratios, and correspondingly, the third feature map includes multiple third feature maps with different feature extraction ratios.

[0046] The first identification unit is specifically used for:

[0047] Multiple third feature maps are input into the defect recognition model, and the defect recognition model identifies defects based on the multiple third feature maps.

[0048] Optionally, the AI-based product defect identification device further includes:

[0049] The second identification unit is used to input multiple third feature maps into the defect classification model, and the defect classification model identifies the type of the defect based on the identified defects.

[0050] The third identification unit is used to input multiple third feature maps into the defect contour recognition model, and the defect contour recognition model identifies the contour of the identified defect.

[0051] Optionally, the AI-based product defect identification device further includes:

[0052] The training unit is used to jointly train the defect identification model and the defect classification model in the following manner:

[0053] Construct a sample image set, which has multiple sample images. Each sample image has a first label and a second label. The first label indicates whether the sample image has a defect, and the second label indicates the type of the defect.

[0054] Generate the fourth feature map of the sample images in the sample image set;

[0055] The defect identification model is used to identify defects based on the fourth feature map, and the defect classification model is used to classify the identified defects based on the fourth feature map.

[0056] An error function is constructed based on the comparison between the identification result of the defect identification model and the first label, and the comparison between the classification result of the defect classification model and the second label.

[0057] The defect identification model and the defect classification model are trained using the error function.

[0058] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the artificial intelligence-based product defect identification method as described above.

[0059] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the artificial intelligence-based product defect identification method described above.

[0060] According to one aspect of this disclosure, a computer program product is provided, the computer program product including a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the artificial intelligence-based product defect identification method as described above.

[0061] In this embodiment of the disclosure, since the two processes involved in product defect identification—image calibration and defect identification from the calibrated image—generally both include a step of generating a feature map from the image to be inspected, the feature map generation step in both processes is merged. The generated feature map is used for both calibration and final defect identification. The calibration process directly calibrates the feature map, and product defect identification is performed directly on the calibrated feature map. Compared to calibrating the image to be inspected and then generating a feature map from the calibrated image, this reduces the number of times the feature map generation step is executed, thereby reducing defect detection time and improving defect detection efficiency.

[0062] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0063] The accompanying drawings are provided to further understand the technical solutions of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0064] Figure 1 This is a system architecture diagram of an artificial intelligence-based product defect identification method applied according to an embodiment of the present disclosure;

[0065] Figure 2A This is a schematic diagram illustrating a scenario in which an artificial intelligence-based product defect identification method according to an embodiment of the present disclosure is applied in industrial automated quality inspection.

[0066] Figure 2B This is a schematic diagram illustrating the application of an artificial intelligence-based product defect identification method according to embodiments of this disclosure in industrial automated quality inspection.

[0067] Figure 3 This is a flowchart of an artificial intelligence-based product defect identification method according to an embodiment of the present disclosure;

[0068] Figure 4 This is an execution architecture diagram of an artificial intelligence-based product defect identification method according to an embodiment of the present disclosure;

[0069] Figure 5 yes Figure 4 A structural diagram of a specific implementation of a feature extraction model in China;

[0070] Figure 6 yes Figure 4 A structural diagram of a specific implementation of a feature pyramid network;

[0071] Figure 7 yes Figure 3 A flowchart of a specific implementation of step 330;

[0072] Figure 8 yes Figure 7 A schematic diagram of matching the set of feature point pairs in step 710;

[0073] Figure 9 yes Figure 7 A flowchart of a specific implementation of step 710;

[0074] Figure 10 yes Figure 9 A schematic diagram of obtaining the correlation score matrix based on the first feature map and the second feature map in step 910;

[0075] Figure 11 yes Figure 9 A flowchart of a specific implementation of step 930;

[0076] Figure 12A -B shows Figure 11 Steps 1110-1120 illustrate the principle of obtaining the second feature point through four elements in a sliding window;

[0077] Figure 13A -C shows Figure 11 A schematic diagram of the various sliding positions of the sliding window in step 1120;

[0078] Figure 14 It shows Figure 11 A flowchart of a specific implementation of step 1120;

[0079] Figure 15A -C shows Figure 14 In step 1410, the sum of the four correlation scores at different sliding positions of the sliding window;

[0080] Figure 16 It shows Figure 14 A flowchart of a specific implementation of step 1420;

[0081] Figure 17A -B shows Figure 16 A schematic diagram illustrating steps 1610-1640 for obtaining the second feature point;

[0082] Figure 18 yes Figure 7 A flowchart of another specific implementation of step 710;

[0083] Figure 19A yes Figure 18 A schematic diagram of the self-attention model in step 911;

[0084] Figure 19B yes Figure 18 A schematic diagram of the interactive attention model in step 912;

[0085] Figure 20A -E shows Figure 18 A schematic diagram illustrating the process of adjusting the correlation score matrix using the first correlation score adjustment value and the second correlation score adjustment value in step 913;

[0086] Figure 21A -B shows Figure 7 A specific implementation of step 720 includes a data list.

[0087] Figure 22 This is a flowchart of an artificial intelligence-based product defect identification method according to another embodiment of the present disclosure;

[0088] Figure 23 This is a specific flowchart for jointly training a defect identification model and a defect classification model;

[0089] Figure 24 This is a block diagram of a product defect identification device according to an embodiment of the present disclosure;

[0090] Figure 25 This is a terminal structure diagram of an artificial intelligence-based product defect identification method according to an embodiment of the present disclosure;

[0091] Figure 26 This is a server architecture diagram for implementing an artificial intelligence-based product defect identification method according to an embodiment of the present disclosure. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0093] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0094] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve desired results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning. With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0095] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0096] Homography transformation refers to the mapping relationship from one plane to another, usually represented by a homography transformation matrix. Once the homography transformation matrix is ​​determined, transforming the coordinates of any point on one plane using the homography transformation matrix will yield the coordinates of the point mapped to that point on the other plane. A homography transformation matrix is ​​typically a 3x3 matrix with 9 elements, but one element changes with the values ​​of the other 8 elements, so only 8 elements can be arbitrarily changed, resulting in 8 degrees of freedom. Since the 8 elements are unknown, with the coordinates of 4 known point pairs (each pair includes a point in one plane and its corresponding point in the other plane), and each known point having 2 coordinate values ​​(x-coordinate and y-coordinate), a system of 8 linear equations can be established and solved to obtain the 8 elements. Therefore, a 3x3 homography transformation matrix can usually be uniquely determined by 4 known point pairs.

[0097] Industrial automated quality inspection refers to the process of using computer algorithms to automate quality checks during the production of industrial products. Compared to traditional manual quality inspection, industrial automated quality inspection is characterized by intelligence, efficiency, and stability, and represents the future direction of industrial quality inspection.

[0098] In industrial automated quality inspection, deep learning-based detection algorithms are commonly used to detect defects in parts. Due to the complexity of parts, images are typically taken from multiple angles, and defects are identified from these images. Two problems arise during multi-angle shooting: first, the robotic arm of the camera experiences some shaking, resulting in shaky images; second, industrial cameras generally use fixed-focus lenses, and at some oblique angles, only a small area of ​​the image is sharp, while the rest is blurry. Therefore, a region of interest (ROI) needs to be drawn for each angle, and the detection algorithm's result is only valid if it falls within this ROI. Based on these two issues, in industrial automated quality inspection, the captured part images are typically calibrated first to eliminate the effects of shaking and ensure the ROI in the part image matches the ideal ROI before defect identification. Currently, image calibration and defect identification from calibrated images are isolated processes; all steps of image calibration must be completed before defect identification from calibrated images can be performed, which is time-consuming and inefficient.

[0099] Therefore, a product defect identification solution that can reduce detection time and improve detection efficiency is needed.

[0100] System architecture and scenario description of the embodiments disclosed herein

[0101] Figure 1This is a system architecture diagram of an artificial intelligence-based product defect identification method according to an embodiment of the present disclosure. The system 100 includes a workbench 110, an assembly line 120, a product 130, multiple cameras 140 at different angles, a defect identification device 150, and a reference image library 160.

[0102] Workbench 110 is a work platform used for producing products. The equipment for producing products is arranged on workbench 110, workers operate on workbench 110, and product defects are inspected on workbench 110. For producing parts, workbench 110 is the processing table for those parts.

[0103] Assembly line 120, also known as a production line, is an industrial production method where each production unit focuses on handling only one segment of the work to improve efficiency. For example, in parts processing, workers responsible for casting work on one section of the assembly line 120, workers responsible for cutting work on another section, and inspection workers or equipment on yet another section for inspection. Because each production unit often relies on conveyor belts or other transport equipment to transfer products between units, the transport equipment itself is sometimes referred to as assembly line 120.

[0104] When a product is transferred from the assembly line 120 to the inspection location, multiple cameras 140 are installed at the inspection location to capture images of the product from multiple angles. For example... Figure 1 As shown, cameras 140 are installed at various angles above the production line 120. The reason for taking images of the product from different angles is that some defects may only be visible in an image from a certain angle and not in other angles. Taking images of the product from multiple angles helps to improve the comprehensiveness of defect detection.

[0105] The camera 140 sends images of the product from different angles to the defect identification device 150, which then identifies defects from these images. The AI-based product defect identification method of this embodiment is primarily executed by the defect identification device 150. This device can be a terminal (e.g., a single device installed at a specific inspection location in a workshop, responsible only for defect detection at that location) or a server (e.g., uniformly identifying defects from product images taken at different inspection locations in multiple workshops or even multiple factories). Terminals can include desktop computers, laptops, PDAs (personal digital assistants), mobile phones, dedicated terminals, and other forms. A server can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0106] The reference image library 160 stores reference images of the product from various angles. When identifying defects from product images taken from different angles, the defect identification device 150 refers to the reference images of the product from different angles in the reference image library 160.

[0107] The embodiments disclosed herein can be applied to industrial automated quality inspection scenarios. Figure 2A A schematic diagram of an industrial automated quality inspection scenario is shown.

[0108] like Figure 2A As shown, product 130 is conveyed on assembly line 120, which is mounted on workbench 110. The end point of the conveying is receiving bin 190. Before entering receiving bin 190, product 130 is inspected to identify whether it has defects. Defective products may need to be reproduced. At the end of assembly line 120, just before product 130 enters receiving bin 190, an inspection bracket 138 is installed above assembly line 120 to support necessary equipment for defect detection, such as camera 140.

[0109] As product 130 is conveyed on assembly line 120, it passes under inspection bracket 138. Cameras 140 are positioned at multiple angles under inspection bracket 138 to capture images of product 130 from different angles, which are then sent to defect identification device 150 for defect detection. After passing under inspection bracket 138, product 130 enters receiving bin 190 at the end of assembly line 120.

[0110] Figure 2B This document describes the principle of product defect detection in an industrial automated quality inspection scenario. Multiple cameras 140 at different angles capture images of the product. These images become the images to be inspected. As mentioned above, these images suffer from jitter and only clearly defined Regions of Interest (ROIs). To mitigate these issues, a reference image library 160 stores jitter-free reference images of the product with clearly defined ROI locations. The defect identification device 150 uses these reference images to calibrate the images to be inspected, reducing jitter and performing ROI matching to obtain a calibrated image. From the calibrated image, product defects can be identified more accurately.

[0111] General Description of Embodiments in this Disclosure

[0112] According to one embodiment of this disclosure, an artificial intelligence-based product defect identification method is provided.

[0113] A product refers to any tangible, manufactured item. It can be a complete item (such as a car) or a part of a complete item (such as a component). It can be a consumer good or the manufacturing tool itself (for example, a production line that manufactures cutting machines; the cutting machine is a product, but it can also be used to cut other products).

[0114] Product defects refer to externally visible parts of a product that do not conform to production specifications. These can include stains, bumps, dents, or defects, as well as incorrect shapes resulting from improper manufacturing. For example, if a part's standard shape is hexagonal but it is mistakenly manufactured as an octagon, the entire shape is considered a product defect.

[0115] Product defect identification methods refer to methods for identifying defects in products. In a narrow sense, it is only a method for identifying whether a product has defects. In a broad sense, it also includes identifying the type of defect and the outline of the defect after it has been identified (described in detail in the embodiments below).

[0116] like Figure 3 As shown, an artificial intelligence-based product defect identification method according to an embodiment of this disclosure includes:

[0117] Step 310: Obtain the product's image to be inspected and the reference image;

[0118] Step 320: Generate the first feature map of the image to be detected and the second feature map of the reference image;

[0119] Step 330: Determine the calibration matrix based on the first feature map and the second feature map;

[0120] Step 340: Use the calibration matrix to calibrate the first feature map to obtain the third feature map;

[0121] Step 350: Identify product defects from the third feature map.

[0122] Steps 310-350 are described in detail below.

[0123] The product image to be inspected in step 310 refers to the original image of the product used for defect detection. It is acquired from camera 140. As described above, cameras 140 at different angles capture product images to be inspected from different angles and send them to the defect identification device 150. The defect identification device 150 then receives the product images to be inspected from different angles.

[0124] In step 310, the reference image of the product refers to the product image used as a reference for defect detection. It is a product image free from jitter and possessing a standard ROI region at a specific camera angle. Comparing it with the image to be inspected can eliminate jitter and correct the ROI region in the image to be inspected. It is obtained from the reference image library 160. The reference image library 160 can index reference images according to product and camera angle. When an image to be inspected is received from a camera 140 at a specific angle from a certain inspection position, the product to be inspected at that inspection position can be determined based on the inspection position, and the reference image corresponding to that product and camera angle can be obtained from the reference image library 160 based on the product index and camera angle index.

[0125] In step 320, a feature map refers to an image reconstructed from features extracted from the original image. It differs from the original image in that each pixel in the original image is a true mapping of the real world; for example, the color of a pixel in the original image corresponds to the color of a corresponding location on a photographed object in the real world. However, a pixel in the feature map is not necessarily a true mapping of the real world; it is a feature value extracted from a specific aspect of the original image. For example, colors are divided into RGB (red, green, blue). A feature value in the feature map might only represent the red value of the corresponding location. After convolution of the feature map with nodes in a convolutional neural network, the resulting feature map is even less likely to correspond one-to-one with points on real-world objects; it is more abstract and is a generalization based on the pixels of the original image.

[0126] The first feature map is the feature map extracted from the image to be detected, and the second feature map is the feature map extracted from the reference image. In step 320, the generation of the first feature map of the image to be detected and the second feature map of the reference image can be achieved through a feature extraction model. The feature extraction model can employ deep learning models such as convolutional neural networks (CNNs).

[0127] Figure 5 The diagram shows the architecture of a CNN as an example of a feature extraction model. Figure 5 As shown, a CNN consists of an input layer, multiple hidden layers 1-n, and an output layer. The input layer has multiple processing nodes, each with a weight matrix (convolutional kernel). Each hidden layer also has multiple processing nodes, each with a weight matrix (convolutional kernel). The output layer also has multiple processing nodes, each with a weight matrix (convolutional kernel).

[0128] Each processing node in the input layer receives features extracted from the input image (the image to be detected or a reference image). For example, Figure 5The three processing nodes in the input layer receive the pixel values ​​of R (red), G (green), and B (blue) from the input image, respectively. A certain feature extraction ratio can be set. The feature extraction ratio refers to the proportion of extracted features to the total number of pixels in the input image. For example, a feature extraction ratio of 1 / 8 means extracting one feature for every 8 pixels in the input image. If the feature is the R value of a pixel, extracting one feature for every 8 pixels means extracting one R value for every 8 pixels. Assuming... Figure 5 The CNN is a feature extraction model with a 1 / 8 feature extraction ratio. The first processing node of the input layer receives the R value of one pixel extracted from every 8 pixels in the input image; the second processing node of the input layer receives the G value of one pixel extracted from every 8 pixels in the input image; and the third processing node of the input layer receives the B value of one pixel extracted from every 8 pixels in the input image.

[0129] Each processing node in the input layer convolves the CNN input with its own convolutional kernel to obtain its output, which is then used as the input to each processing node in hidden layer 1. Each processing node in hidden layer 1 convolves the outputs of each processing node in the input layer with its own convolutional kernel to obtain its output, which is then used as the input to each processing node in hidden layer 2. This process continues until each processing node in the output layer convolves the outputs of each processing node in hidden layer n with its own convolutional kernel, resulting in the output feature map. This output feature map has the same feature extraction ratio as the input.

[0130] When the input image is the image to be detected, the output feature map is the first feature map. When the input image is a reference image, the output feature map is the second feature map. Since the image to be detected may include product images taken from multiple different angles, there are also multiple corresponding first feature maps. Since the reference image may include multiple reference images taken from multiple different angles, there are also multiple corresponding second feature maps.

[0131] Furthermore, the first feature map may also include multiple first feature maps with different feature extraction ratios, and the second feature map may also include multiple second feature maps with different feature extraction ratios. For example... Figure 4 As shown, three feature extraction models can be set with feature extraction ratios (1 / 8, 1 / 16, and 1 / 32). All of them can employ... Figure 5The CNN structure shown is as follows. Since the feature extraction ratio of 1 / 16 is less than that of 1 / 8, and the feature extraction ratio of 1 / 32 is less than that of 1 / 16, the output of the feature extraction model with a feature extraction ratio of 1 / 8 can be used as the input of the feature extraction model with a feature extraction ratio of 1 / 16, and vice versa. The feature extraction models with feature extraction ratios of 1 / 8, 1 / 16, and 1 / 32 are combined to form the backbone network.

[0132] It is important to note that Figure 4 The number of feature extraction models shown is exemplary. Those skilled in the art can make embodiments with more or fewer feature extraction models after benefiting from the teachings of the embodiments of this disclosure. Furthermore, the aforementioned feature extraction ratios of 1 / 8, 1 / 16, and 1 / 32 are also exemplary, and those skilled in the art can replace them with other feature extraction ratios as needed.

[0133] The advantage of setting multiple feature extraction models with different feature extraction ratios is that they simultaneously generate multiple feature maps with different feature extraction ratios for defect identification. Feature maps with different feature extraction ratios can reflect the global features identified at different feature extraction ratios, significantly improving identification accuracy compared to schemes that identify defects from feature maps with only one feature extraction ratio.

[0134] Next, in step 330, a calibration matrix is ​​determined based on the first feature map and the second feature map.

[0135] A calibration matrix is ​​a matrix used to calibrate an input image to reduce jitter and ROI mismatch. An example is the homography transformation matrix, as described above. The homography transformation matrix reflects the mapping relationship from one image to another. Transforming the coordinates of any point in one image using the homography transformation matrix yields the coordinates of that point mapped to another image. If the other image is an ideal image (without jitter and ROI mismatch) (the reference image), then the mapped coordinates are the coordinates where that point is expected to be located. Thus, transforming the entire input image using the homography transformation matrix yields the calibrated image, which reduces jitter and ROI mismatch.

[0136] Since the second feature map is obtained based on the reference image, which reflects a state without jitter and ROI mismatch, the calibration matrix can be determined based on the first and second feature maps. The specific determination method will be described in detail later.

[0137] Next, in step 340, the first feature map is calibrated using a calibration matrix to obtain the third feature map.

[0138] In one embodiment, calibration can be performed using, for example... Figure 4 The calibration matrix application function is shown below. A calibration matrix application function is a function that applies a calibration matrix to an input pixel or feature point matrix to obtain a calibrated pixel or feature point matrix. In one embodiment, the calibration matrix application function includes a feature point offset matrix generation function and a feature point offset function implementation.

[0139] The offset matrix generation function generates a feature point offset matrix corresponding to the calibration matrix. The feature point offset matrix records the offset values ​​required for the position of each feature point in the input feature map. Offsetting the coordinates of each feature point in the input feature point matrix according to the corresponding offset value in the feature point offset matrix yields the offset coordinates of that feature point. An example of this is the `affine_grid` function in PyTorch. The feature point offset function performs the above offset; it offsets the coordinates of each feature point in the input feature point matrix according to the feature point offset matrix, obtaining the calibrated feature point coordinates, thus producing the third feature map. An example of this is the `grid_sample` function in PyTorch.

[0140] In addition to the above embodiments, calibration can also be performed through other implementation methods, such as manually estimating the amount that each feature point in the first feature map needs to change based on the calibration matrix. However, the above method, which uses the feature point offset matrix generation function and the feature point offset function, improves the efficiency of calibration and reduces processing overhead.

[0141] It should be noted that in one embodiment, such as Figure 4 As shown, the calibration matrix may not be used as the first feature map generated by each feature extraction model, but only applied to the lowest-level feature extraction models of the backbone network (the feature extraction models with the largest feature extraction ratio, such as...). Figure 4 The first feature map generated by the feature extraction model (which extracts 1 / 8 of the features) produces a corresponding third feature map. However, higher-level feature extraction models in the backbone network may not need to generate a first feature map; instead, they perform further feature extraction based on the third feature map generated by the lower-level model, thus producing the corresponding third feature maps for those layers. For example... Figure 4As shown, the feature extraction model with a 1 / 16 feature extraction ratio performs further feature extraction based on the third feature map generated by the feature extraction model with a 1 / 8 feature extraction ratio, producing a third feature map with a 1 / 16 feature extraction ratio. Similarly, the feature extraction model with a 1 / 32 feature extraction ratio performs further feature extraction based on the third feature map generated by the 1 / 16 feature extraction model, producing a third feature map with a 1 / 32 feature extraction ratio. Since the first feature map is only an intermediate result generated by each feature extraction model, therefore... Figure 4 Only the third feature map generated by these feature extraction models is shown; the first feature map is not shown.

[0142] In the above embodiments, the advantage of applying the calibration matrix only to the lowest feature extraction model of the backbone network is that, since the higher-level feature extraction models do not need to generate the first feature map, they can directly generate a third feature map with a higher feature extraction ratio based on the third feature map extracted by the lower-level feature extraction model, thereby improving feature extraction efficiency.

[0143] Additionally, in one embodiment, such as Figure 4 As shown, the third feature maps proposed by each feature extraction model can be further processed by the feature pyramid network to generate third feature maps with a higher feature extraction ratio.

[0144] Generally, backbone networks have inherent overhead limitations and should not have too many layers of feature extraction models; 3-4 layers are typically recommended. However, for subsequent defect identification, the more layers of feature maps with different feature extraction ratios used as input, the more high-level image features the defect identification model can discover from these feature maps, thereby improving the accuracy of defect identification. Therefore, in one embodiment, based on the 3-4 layers of feature maps with different feature extraction ratios generated by the backbone network, more feature maps with other feature extraction ratios are generated. In this embodiment, this is implemented using a feature pyramid network.

[0145] Figure 6An exemplary structure of a feature pyramid network is shown. The feature pyramid network has multiple processing layers corresponding one-to-one with multiple feature extraction models in the backbone network. Each processing layer includes an input convolutional module and an output convolutional module. The input convolutional module receives the third feature map output by the corresponding feature extraction model as input. The convolution result of the input convolutional module is fed to an upsampler for upsampling, obtaining an upsampled result consistent with the feature extraction ratio of the next processing layer, which is then fed to the next processing layer. Except for the topmost processing layer, other processing layers receive the upsampled result sampled by the upsampler from the previous processing layer, add it to the convolution result of the input convolutional module of the current processing layer, and use it as the input of the output convolutional module of the current processing layer. In this way, the third feature map output by the current processing layer not only reflects the features from the third feature map of the corresponding feature extraction ratio of the current processing layer, but also reflects the higher-level features summarized by higher-level processing layers, which helps improve the accuracy of subsequent defect identification.

[0146] like Figure 6 As shown, the third feature map output by the feature extraction model with a 1 / 8 feature extraction ratio is C3, the third feature map output by the feature extraction model with a 1 / 16 feature extraction ratio is C4, and the third feature map output by the feature extraction model with a 1 / 32 feature extraction ratio is C5. The processed third feature maps output by the processing layers corresponding to C3, C4, and C5 in the feature pyramid network are P3, P4, and P5, respectively. P4 not only reflects the features of the third feature map C4 from this processing layer, but also reflects the high-level features from the third feature map C5 passed down from higher layers, greatly improving the semantics of the processed third feature map P4. P3 not only reflects the features of the third feature map C3 from this processing layer, but also reflects the high-level features from the third feature maps C4 and C5 passed down from higher layers, greatly improving the semantics of the processed third feature map P3.

[0147] In addition, the convolution result of the input convolution module of the highest processing layer of the backbone network is downsampled by the downsampler to the desired lower feature extraction ratio. After being convolved by the corresponding output convolution module, it is output as the third feature map with a lower feature extraction ratio. In this way, based on the feature maps with different feature extraction ratios in the 3-4 layers generated by the backbone network, more feature maps with other feature extraction ratios are generated, so as to improve the accuracy of defect recognition.

[0148] like Figure 6As shown, the convolution result of the input convolutional module corresponding to the third feature map C5 with a 1 / 32 feature extraction ratio in the feature pyramid network is passed through a 2:1 downsampler to produce features with a 1 / 64 feature extraction ratio, and then convolved by the corresponding output convolutional module to obtain the third feature map P6 with a 1 / 64 feature extraction ratio. The features with a 1 / 64 feature extraction ratio are then passed through a 2:1 downsampler to produce features with a 1 / 128 feature extraction ratio, and then convolved by the corresponding output convolutional module to obtain the third feature map P7 with a 1 / 128 feature extraction ratio, thus expanding the third feature map from P3-5 to P6-7.

[0149] Next, in step 350, defects in the product are identified from the third feature map.

[0150] As mentioned above, product defect identification can be divided into narrow and broad definitions. In a narrow sense, it only identifies whether a product has a defect. In this case, the third feature map can be input into the trained defect identification model. If the third feature map only includes a third feature map with one feature extraction ratio, then only this third feature map with that ratio can be input into the defect identification model, which will then identify whether the product has a defect based on this third feature map. In the case of a third feature map such as... Figure 4 In the case of a third feature map with multiple different feature extraction ratios, all of these third feature maps with multiple feature extraction ratios can be input into the defect identification model, which will then identify whether the product is defective based on the third feature maps with these multiple feature extraction ratios.

[0151] This defect identification model can employ structures such as Convolutional Neural Networks (CNNs), decision trees, and Recurrent Neural Networks (RNNs). When using CNNs, one can refer to... Figure 5 The structural diagram shown will not be repeated.

[0152] When training the defect recognition model, a sample image set can be pre-constructed. This sample image set contains a large number of sample images of the product used for training. Whether the product has a defect or not is known in each sample image. Based on whether the product has a defect, a first label is assigned to each sample image. The first label indicates whether the sample image has a defect. Next, using a method similar to steps 310-340 of the embodiments of this disclosure, a fourth feature map, similar to the third feature map of the embodiments of this disclosure, is generated for the sample images. The fourth feature map is input into the defect recognition model, which generates a recognition result indicating whether a defect exists. The recognition result is compared with the first label. If the proportion of recognition results in the sample image set that match the first label is higher than a first proportion (e.g., 95%), the defect recognition model is successfully trained. If it is not higher than the first proportion, the parameters in the defect recognition model are adjusted until the proportion of recognition results in the sample image set that match the first label is higher than the first proportion.

[0153] In a broad sense, identifying product defects involves more than just confirming their presence; it also includes identifying their type and outlining their characteristics. This can be achieved through methods such as... Figure 4 The defect classification model and defect contour recognition model are shown. At this point, as... Figure 22 As shown, the defect identification method in this embodiment of the disclosure further includes:

[0154] Step 360: Input multiple third feature maps into the defect classification model, and the defect classification model will identify the type of defect based on the identified defects;

[0155] Step 370: Input multiple third feature maps into the defect contour recognition model, and the defect contour recognition model will identify the contour of the identified defects.

[0156] Defect classification models and defect contour recognition models are similar to defect recognition models, and can employ structures such as Convolutional Neural Networks (CNNs), decision trees, and Recurrent Neural Networks (RNNs). For brevity, these will not be elaborated upon further. Note that both defect classification and defect contour recognition models only begin operating if the defect recognition model identifies a defect; therefore, if... Figure 4 As shown, the defect recognition model also has an output that directly connects to the defect classification model and the defect contour recognition model as trigger inputs.

[0157] like Figure 4 As shown, the defect identification model, defect classification model, and defect contour identification model can all be located within the identification head. The identification head is a device for detecting defects in a product.

[0158] The advantage of steps 360-370 is that, after identifying whether a product is defective in step 350, steps 360-370 further identify the defect type and defect contour, which helps to further analyze the defect and generate countermeasures. In addition, by inputting multiple third feature maps with different feature extraction ratios, the defect classification and contour recognition can take into account information at different levels in the third feature maps with different feature extraction ratios, thereby improving the quality of defect classification and contour recognition.

[0159] Furthermore, considering that identifying the existence of defects is inseparable from identifying the types of defects, therefore, in one embodiment, such as Figure 23 As shown, the defect identification model and the defect classification model can be jointly trained through the following steps:

[0160] Step 2310: Construct a sample image set;

[0161] Step 2320: Generate the fourth feature map of the sample images in the sample image set;

[0162] Step 2330: Use the defect identification model to identify defects based on the fourth feature map, and use the defect classification model to classify the identified defects based on the fourth feature map;

[0163] Step 2340: Based on the comparison of the identification results of the defect identification model with the first label, and the comparison of the classification results of the defect classification model with the second label, construct the error function;

[0164] Step 2350: Use the error function to train the defect identification model and the defect classification model.

[0165] Steps 2310-2350 are described in detail below.

[0166] The sample image set in step 2310 is a collection of sample images. These sample images are product images used as samples for jointly training the defect identification model and the defect classification model. A certain percentage of these product images must represent defective products. For example, more than 20% should represent defective products to ensure effective model training. Each sample image is pre-labeled with a first label indicating whether it has a defect. For sample images whose first label indicates a defect, a second label is assigned. The second label indicates the type of defect. In one example, a sample image set of 1000 images can be used. The more sample images in the set, the better the joint training effect.

[0167] Next, in step 2320, a fourth feature map of the sample images in the sample image set is generated. In one embodiment, the fourth feature map can be extracted from the sample images according to a predetermined feature extraction ratio using a deep learning model such as CNN. In another embodiment, a method similar to steps 310-340 of the embodiments of this disclosure can be used to obtain corresponding reference images for these sample images, and then feature maps are obtained from the sample images and reference images respectively. Based on the feature maps obtained from the sample images and reference images, a calibration matrix is ​​determined, and then the feature maps obtained from the sample images are calibrated using the calibration matrix to obtain the fourth feature map. The latter embodiment can reduce the impact of jitter and ROI region mismatch in the sample images and improve the model training effect.

[0168] Then, in step 2330, the fourth feature map is input into the defect identification model, which identifies whether a product defect exists based on the fourth feature map. Simultaneously, the fourth feature map identifying the defect is input into the defect classification model, which identifies the type of product defect.

[0169] Next, in step 2340, an error function is constructed based on the comparison between the identification result of the defect identification model and the first label, and the comparison between the classification result of the defect classification model and the second label. In one embodiment, the error function is constructed as follows:

[0170] E = (B1 / C1 + B2 / C2) / 2 (Formula 3)

[0171] Where E represents the joint training error of the defect identification model and the defect classification model, C1 represents the total number of sample images in the sample image set, B1 represents the number of sample images whose identification results from the defect identification model are inconsistent with the first label, C2 represents the number of sample images in the sample image set that have a second label and are also identified as defective by the defect identification model, and B2 represents the number of sample images in the sample image set whose classification results from the defect classification model are consistent with the second label. The reason for setting C2 as the number of sample images in the sample image set that have a second label and are also identified as defective by the defect identification model is that for sample images where the defect identification model does not identify defects, the defect classification model cannot be triggered to classify defects. Furthermore, for sample images where the defect identification model identifies defects but are actually not defective, they do not have a second label and cannot be compared with the classification results of the defect classification model.

[0172] In Formula 3, B1 / C1 represents the misclassification rate of the defect identification model, and B2 / C2 represents the misclassification rate of the defect classification model. Averaging these two ratios yields the joint training error. This joint training error reflects the errors of both the defect identification and classification models. Using it to jointly train the defect identification and classification models improves the overall model training performance.

[0173] In another embodiment, the error function can also be constructed as follows:

[0174] E=α·B1 / C1+β·B2 / C2 Formula 4

[0175] In this equation, α represents the weight of the misidentification rate of the defect identification model, and β represents the weight of the misclassification rate of the defect classification model. Generally, α + β = 1. Compared to Equation 3, Equation 4 replaces the average of the misidentification rates of the defect identification model and the misclassification rates of the defect classification model with a weighted sum. Since the weighted sum method allows for flexible adjustment of the weights α and β according to actual needs, and α and β actually represent the importance of defect identification and defect classification in specific applications, it improves the flexibility of model training. Simultaneously, it enables defect identification and defect classification to better adapt to the requirements of application scenarios.

[0176] Then, in step 2350, the defect identification model and the defect classification model are trained using an error function. Specifically, an error threshold (e.g., 5%) can be set. If the joint training error E is less than this threshold, training is successful, and training of the defect identification model and the defect classification model stops. If it is not less than the threshold, the parameters in the defect identification model and the defect classification model are adjusted, and the sample images from the sample image set are re-input into the two models for training until the joint training error E is less than the threshold.

[0177] The advantage of the embodiment in steps 2310-2350 is that it does not train the defect identification model and the defect classification model in isolation. Instead, it considers that the existence of a defect is inseparable from the types of defects, and uses the errors generated by the two models to jointly construct an error function, thereby jointly adjusting the parameters of the two models. Therefore, it greatly improves the training effect of the defect identification model and the defect classification model, and improves the accuracy of defect identification and classification.

[0178] The above content describes the steps other than step 330 in detail, except for step 330, which requires further discussion. In steps 310-370, since product defect identification generally involves two processes—image calibration and defect identification from the calibrated image—both generally include the step of generating a feature map from the image to be inspected, the feature map generation steps in both processes are combined. The feature map is used for calibration and also serves as the basis for subsequent defect identification. The product defect identification process in step 350 is performed directly on the third feature map generated by calibration. Compared to the existing technology where product defect identification is performed on the image to be inspected (first calibrating the image to be inspected, then generating a feature map from the calibrated image, and finally identifying the defect on the feature map), it is clear that the process of generating a feature map from the calibrated image is omitted, reducing defect detection time and improving defect detection efficiency.

[0179] The detailed implementation process of step 330 is described below.

[0180] Further detailed description of step 330

[0181] like Figure 7 As shown, in one embodiment, step 330 includes:

[0182] Step 710: Obtain the set of matching feature point pairs;

[0183] Step 720: Determine the calibration matrix based on the set of matching feature point pairs.

[0184] Step 710 can be performed by Figure 4 The matching feature point pair acquisition module performs the operation, and step 720 can be performed by... Figure 4 The calibration matrix generation algorithm module is executed. Steps 710-720 are described in detail below.

[0185] The calibration matrix is ​​a matrix used to calibrate the first feature map formed from the image to be detected. Calibrating the first feature map eliminates jitter and ROI mismatch issues. The reference image, however, is free of jitter and ROI mismatch. Therefore, the calibration of the first feature map relies on comparison with a second feature map generated from the reference image. A matching feature point pair is a pair of feature points found in both the first and second feature maps to achieve this comparison. This matching feature point pair includes a first feature point from the first feature map and a second feature point from the second feature map that corresponds to the first feature point. The set of matching feature point pairs is a collection of multiple such matching feature point pairs.

[0186] The number of matching feature point pairs in the matching feature point pair set is related to the size of the calibration matrix. Take an n×n homography transformation matrix as an example. Because the n of this homography transformation matrix... 2 One of the transformation coefficients needs to change along with the other elements; therefore, (n) 2 -1) degrees of freedom. Since a matching feature point pair has x-coordinates and y-coordinates, two equations about the transformation coefficients can be written based on the x-coordinates and y-coordinates of a matching feature point pair. It is necessary to write (n) 2 -1) / 2 or more equations are needed to solve for the homography transformation matrix. Therefore, the number of matching feature point pairs in the set of matching feature point pairs should be at least greater than or equal to (n 2 The smallest integer of (-1) / 2.

[0187] It should be noted that the second feature point corresponding to the first feature point in the second feature map is not necessarily an intrinsic feature point extracted from the reference image. It can be a new feature point formed by combining parts of several intrinsic feature points extracted from the reference image. The second feature map is formed by extracting features from the reference image according to a predetermined feature extraction ratio. For example, for a 1 / 8 feature extraction ratio, one pixel is selected from every eight pixels in the reference image, and an intrinsic feature point (e.g., the R value of that pixel) is formed accordingly. The second feature point corresponding to the first feature point in the second feature map may not necessarily coincide exactly with one of the intrinsic feature points; it is very likely that it partially overlaps with several of the intrinsic feature points (e.g., the part corresponding to the middle of multiple extracted pixels in the reference image).

[0188] like Figure 8As shown, each small square in the first or second feature map corresponds to an intrinsic feature point extracted from the image to be detected or the reference image. Assuming the center of the small square in the upper left corner of the first feature map is the origin of the coordinate system, with the horizontal direction to the right as the positive x-axis and the vertical direction downwards as the positive y-axis, a Cartesian coordinate system is established. The unit of the Cartesian coordinate system is the side length of a feature point. Three first feature points P1-3 are selected in the first feature map, all of which are intrinsic feature points in the first feature map, meaning they coincide with a small square in the first feature map, with center coordinates of (0,0), (1,1), and (1,2) respectively. The corresponding second feature points in the second feature map may not necessarily coincide with a small square. Figure 8 In the second feature map, the second feature point P1' corresponding to the first feature point P11 coincides with a small square, with its center coordinates being (2,1). The second feature point P2' corresponding to the first feature point P2 in the second feature map does not coincide with a single small square, but occupies a portion of each of three small squares with center coordinates (2,2), (3,2), and (2,3). The second feature point P3' corresponding to the first feature point P3 in the second feature map does not coincide with a single small square, but occupies a portion of each of two small squares with center coordinates (1,3) and (2,3). Thus, the first feature point P1 and the second feature point P1', the first feature point P2 and the second feature point P2', and the first feature point P3 and the second feature point P3' are three matching feature point pairs, forming a set of matching feature point pairs.

[0189] The specific implementation process of step 710 will be described in detail later.

[0190] In step 720, a calibration matrix is ​​determined based on the set of matching feature point pairs.

[0191] In one embodiment, step 720 includes:

[0192] Perform the first number of times the first process, the first process including: extracting a second number of matching feature point pairs from the set of matching feature point pairs; calculating a candidate calibration matrix based on each matching feature point pair; determining a first error of the matching feature point pairs in the set of matching feature point pairs on the candidate calibration matrix; determining a third number of matching feature point pairs whose first error is less than a first threshold.

[0193] The candidate calibration matrix calculated by the first process with the largest third number in the first process is taken as the determined calibration matrix.

[0194] The above embodiment is based on the idea that the set of matching feature point pairs formed in step 710 contains multiple matching feature point pairs. These matching feature point pairs may play different roles in the calibration matrix determination process. For example, some matching feature point pairs are relatively close and exhibit similar characteristics, so using them all to determine the calibration matrix may result in a low-quality calibration matrix. Therefore, it is necessary to select some representative matching feature point pairs to determine the calibration matrix. In the above embodiment, multiple (first number of times) matching feature point pairs can be taken, and multiple (second number of times) matching feature point pairs can be taken each time. For these second number of matching feature point pairs, the corresponding calibration matrix can be obtained as a candidate calibration matrix. Then, the number of matching feature point pairs in the entire set of matching feature point pairs that conform to this candidate calibration matrix is ​​measured (using the first error method mentioned above). The more matching feature point pairs that conform to this candidate calibration matrix, the higher the quality of this candidate calibration matrix. For each candidate calibration matrix obtained by taking multiple matching feature point pairs, the number of matching feature point pairs in the entire set that conform to the candidate calibration matrix can be obtained (third number). The largest number of the third option means that the quality of selecting multiple matching feature point pairs is the highest, and the candidate calibration matrix generated in this selection can be used as the determined calibration matrix.

[0195] The calculation of the candidate calibration matrix based on each matching feature point pair in the first process described above can be achieved by listing and solving the equations for the change coefficients of the candidate calibration matrix based on the x-coordinate and y-coordinate of each matching feature point pair.

[0196] In one embodiment, the first error of the matching feature point pair in the candidate calibration matrix of the set of matching feature point pairs in the first process described above can be achieved by the following process: mapping the first feature point in the matching feature point pair to the third feature point in the second feature map according to the candidate calibration matrix; determining the distance between the third feature point and the second feature point in the matching feature point pair as the first error.

[0197] The process of mapping the candidate calibration matrix to the second feature map is the reverse process of calculating the candidate calibration matrix based on the matching feature point pairs. For the second number of matching feature point pairs extracted from the set of matching feature point pairs, since the candidate calibration matrix is ​​calculated based on them, the third feature point obtained after mapping the first feature point to the second feature map according to the candidate calibration matrix is ​​the second feature point. At this time, the distance between the third feature point and the second feature point is 0, and the first error is 0. For those matching feature point pairs outside the second number of matching feature point pairs in the set of matching feature point pairs, the third feature point obtained after mapping them to the second feature map according to the candidate calibration matrix may not be the second feature point. However, if the third feature point deviates very little from the second feature point, that is, the distance is not large, it can also be considered that the matching feature point pair roughly conforms to the candidate calibration matrix. Only when the distance between the third feature point and the second feature point is large enough, for example, not less than the first threshold, is the matching feature point pair considered not to conform to the candidate calibration matrix. The distance between the third feature point and the second feature point is equal to the first error. Therefore, from the set of matching feature point pairs, there are a second number of matching feature point pairs with a first error of 0, and the first errors of the remaining matching feature point pairs vary in size. The number of these first errors that is less than the first threshold determines the size of the third number.

[0198] The aforementioned distance can be either Euclidean or cosine distance, and the first error can be calculated using the methods for calculating Euclidean or cosine distance.

[0199] The advantage of the above embodiment, which uses the distance between the third feature point mapped to the second feature map and the second feature point as the first error, is that for each first feature point, it finds the actual third feature point corresponding to it in the second feature map through the candidate calibration matrix, and obtains the first error through the distance between the third feature point and the second feature point. The first error calculated in this way reflects the real deviation brought to all matching feature point pairs in the matching feature point pair set after mapping using the candidate calibration matrix. Using this as the first error has higher accuracy and improves the quality of the determined calibration matrix.

[0200] Suppose there are 10 matching feature point pairs in the set. The first set has 5 pairs, the second set has 4 pairs. The following section combines... Figure 21A -B describes an example of the above embodiment. That is, 5 rounds will be drawn from 10 matching feature point pairs, with 4 feature point pairs randomly drawn in each round.

[0201] like Figure 21A As shown, in the first round of extraction, 4 feature point pairs were extracted from 10 matching feature point pairs. These 4 feature point pairs were used to calculate the candidate calibration matrix, resulting in candidate calibration matrix A. At this point, as... Figure 21BAs shown, the first error of each of the 10 matching feature point pairs on the candidate calibration matrix A is calculated, and the values ​​are 0.19, 0.17, 0.25, 0, 0, 0, 0, 0.23, 0.19, 0, and 0.16, respectively. The four pairs with a first error of 0 correspond to the four feature point pairs extracted for calculating the candidate calibration matrix. The first threshold is 0.2. The first error of matching feature point pair 3 is 0.25, which is not less than the first threshold of 0.2; the first error of matching feature point pair 7 is 0.23, which is not less than the first threshold of 0.2; and the first error of matching feature point pair 10 is 0.26, which is not less than the first threshold of 0.2. Removing these three matching feature point pairs from the 10 pairs leaves 7 matching feature point pairs. The third number is 7. Return. Figure 21A In the second round, when four feature point pairs are extracted, candidate calibration matrix B is obtained, with a third number of 6. In the third round, four feature point pairs are extracted, candidate calibration matrix C is obtained, with a third number of 5. In the fourth round, four feature point pairs are extracted, candidate calibration matrix D is obtained, with a third number of 8. In the fifth round, four feature point pairs are extracted, candidate calibration matrix E is obtained, with a third number of 6. Since the largest third number among the five rounds of extraction is 8, corresponding to the fourth round, the corresponding calibration matrix is ​​D. At this point, calibration matrix D is the determined calibration matrix.

[0202] In specific implementations, the above embodiments can also utilize an interior point set. The interior point set is a collection of matched feature point pairs whose first error is less than a first threshold. This interior point set can be updated with each round of the first process. After each round of the first process, the third number of matched feature point pairs with a first error less than the first threshold generated in that round is compared with the number of existing matched feature point pairs in the interior point set. If the third number is greater than the number of existing matched feature point pairs in the interior point set, then all matched feature points in the interior point set are replaced with matched feature point pairs with a first error less than the first threshold generated in that round. After all rounds have been executed, the candidate calibration matrix corresponding to the matched feature points in the interior point set is the determined calibration matrix.

[0203] like Figure 21AAs shown, after completing the first process of the first round, the determined calibration matrix is ​​candidate calibration matrix A. The third number of matching feature point pairs with a first error less than the first threshold generated in this round is 7. At this time, the inlier set is empty, and the 7 matching feature point pairs with a first error less than the first threshold are placed into the inlier set. After completing the first process of the second round, the determined calibration matrix is ​​candidate calibration matrix B. The third number generated in this round is 6, which is not greater than 7, so no processing is done. After completing the first process of the third round, the determined calibration matrix is ​​candidate calibration matrix C. The third number generated in this round is 5, which is not greater than 7, so no processing is done. After completing the first process of the fourth round, the determined calibration matrix is ​​candidate calibration matrix D. The third number generated in this round is 8, which is greater than 7. The 8 matching feature point pairs with a first error less than the first threshold generated in the fourth round replace the 7 matching feature points in the inlier set. After completing the first process of the fifth round, the determined calibration matrix is ​​candidate calibration matrix E. The third number generated in this round is 6, which is not greater than 8, so no processing is done. At this point, the matching feature points in the inlier set are generated in the fourth round, and the corresponding candidate calibration matrix is ​​D. The candidate calibration matrix D is then used as the determined calibration matrix.

[0204] The above implementation of the interior point set simplifies the process of determining the calibration matrix and improves the efficiency of determining the calibration matrix.

[0205] The advantage of the above embodiment of performing the first process a first number of times and determining the calibration matrix based on the candidate calibration matrix corresponding to the largest third number is that it selects multiple rounds of matching feature point pairs, generates multiple candidate calibration matrices, and compares the actual effects of these multiple candidate calibration matrices based on the error of the entire set of matching feature point pairs on each candidate calibration matrix. By taking the actual error as a starting point, the accuracy of determining the calibration matrix is ​​improved.

[0206] The advantage of steps 710-720 is that the calibration matrix is ​​determined from the actual matching feature point pairs obtained from the first feature map and the second feature map. Since these feature point pairs are actually extracted from the first feature map and the second feature map, the calibration matrix determined in this way reflects the actual objective situation and has high calibration accuracy.

[0207] The detailed process of step 710 will be discussed below.

[0208] Further detailed description of step 710

[0209] In one embodiment, such as Figure 9 As shown, step 710 includes:

[0210] Step 910: Based on the first feature map and the second feature map, obtain the correlation score matrix;

[0211] Step 920: Select multiple first feature points in the first feature map;

[0212] Step 930: For each selected first feature point, based on the correlation score matrix, obtain the second feature point corresponding to the first feature point in the second feature map.

[0213] Steps 910-930 are described below.

[0214] The correlation score in step 910 represents the degree of correlation between feature points in the first feature map and feature points in the second feature map. Its value is generally between 0% and 100%. If feature point A in the first feature map maps to feature point B in the second feature map, then the correlation score between feature point A and feature point B is 100%. Feature points in the second feature map that are closer to feature point B will have a higher correlation score with feature point A. Conversely, feature points in the second feature map that are farther from feature point B will have a lower correlation score with feature point A.

[0215] Any intrinsic feature point in the first feature map (a feature point generated from pixels directly extracted from the image to be detected), Figure 8 Each small square in the first feature map and any intrinsic feature point in the second feature map will have a correlation score. The matrix formed by all these correlation scores is called the correlation score matrix.

[0216] When forming the correlation score matrix, all feature points of the first feature map are stretched into a row vector, with each element of the row vector corresponding to a column of the correlation score matrix. Similarly, all feature points of the second feature map are stretched into a column vector, with each element of the column vector corresponding to a row of the correlation score matrix. Thus, if the first feature map is a (P1×P2) matrix and the second feature map is a (Q1×Q2) matrix, the correlation score matrix will have (P1×P2) columns and (Q1×Q2) rows, forming a (P1×P2)×(Q1×Q2) matrix. Figure 10 As shown, the first feature map P has 3 rows and 2 columns. First, we sort the feature points P in the first row. 00 and P 01 Extract it. Then extract the feature point P from the second row. 10 and P 11 Take it out and place it in P. 00 and P 01 After that. Then, take the feature point P in the third row. 20 and P 21 Take it out and place it in P. 10 and P 11 After that. Thus, we obtain the row vector (P) 00 P 01 P 10P 11 P 20 P 21 The first feature map Q has 2 rows and 4 columns. First, the feature points Q in the first row are... 00 Q 01 Q 02 Q 03 Extract it. Then extract the feature points Q from the second row. 10 Q 11 Q 12 Q 13 Take it out and place it in Q. 00 Q 01 Q 02 , Following this, it becomes a column vector (Q). 00 Q 01 Q 02 Q 03 Q 10 Q 11 Q 12 Q 13 ), where the 8 elements correspond to the 8 rows of the correlation score matrix.

[0217] Each element in the resulting correlation score matrix represents the correlation score of the feature point in the first feature map corresponding to the column containing that element, and the feature point in the second feature map corresponding to the row containing that element. For example... Figure 10 The element 0.3 in the 4th row and 3rd column of the correlation score matrix is ​​the feature point P corresponding to the 3rd column in the first feature map. 10 and the feature point Q corresponding to row 4 in the second feature map 02 The correlation score. Figure 10 The correlation score between each feature point in the first feature map P and each feature point in the second feature map Q is shown.

[0218] This correlation score matrix can be formed manually, one by one. That is, for each first feature point in the first feature map, a person manually checks in the second feature map which feature point it maps to. The correlation score between this feature point and the first feature point is 100%. The correlation scores between other feature points in the second feature map and the first feature point are manually given based on the proximity of these other feature points to the feature points mapped to the first feature point. This embodiment requires a lot of manual time and is inaccurate.

[0219] In one embodiment, the correlation score matrix can be generated automatically in the following way:

[0220] Using the feature point rows in the first feature map as the first dimension, the feature point columns in the first feature map as the second dimension, and adding a predetermined third dimension, a first three-dimensional matrix is ​​constructed using the feature point values ​​in the first feature map as vector elements.

[0221] The second three-dimensional matrix is ​​constructed by taking the feature point rows in the second feature map as the fourth dimension, the feature point columns in the second feature map as the fifth dimension, and adding a predetermined sixth dimension, and using the feature point values ​​in the second feature map as vector elements.

[0222] Multiplying the transpose of the first three-dimensional matrix by the second three-dimensional matrix yields a four-dimensional matrix with the third and sixth dimensions eliminated.

[0223] Obtain the correlation score matrix from the four-dimensional matrix.

[0224] The first feature map is composed of the feature values ​​(such as R values) of pixels extracted from the image to be detected according to a certain feature extraction ratio. The feature value of each pixel constitutes an intrinsic feature point value. These intrinsic feature point values ​​are arranged into a two-dimensional matrix. The rows of the two-dimensional matrix are the first dimension, and the columns of the two-dimensional matrix are the second dimension. To obtain the correlation score matrix, a one-dimensional thickness can be added to the two-dimensional matrix. That is, each pixel is regarded as having thickness, and the corresponding feature point also has thickness coordinates. Since the thickness coordinates are canceled in the calculation below, arbitrary thickness coordinates can be set (for example, the thickness coordinates of all elements of the two-dimensional matrix are 1). In this way, the two-dimensional matrix becomes a three-dimensional matrix, namely the first three-dimensional matrix, and the feature point values ​​of the two-dimensional matrix become the feature point values ​​at the corresponding coordinates of the first three-dimensional matrix.

[0225] Similarly, the intrinsic feature point values ​​in the second feature map can be considered as arranged into a two-dimensional matrix. The rows of the two-dimensional matrix are the fourth dimension, and the columns are the fifth dimension. To obtain the correlation score matrix, a sixth dimension, the thickness dimension, can be added to the two-dimensional matrix. Arbitrary thickness coordinates can be set. In this way, the two-dimensional matrix becomes a three-dimensional matrix, namely the second three-dimensional matrix, and the feature point values ​​of the two-dimensional matrix become the feature point values ​​at the corresponding coordinates in the second three-dimensional matrix.

[0226] By adding a thickness dimension, transforming the first and second feature maps into a first 3D matrix and a second 3D matrix, and then multiplying the second 3D matrix by the transpose of the first 3D matrix, the added thickness dimension is completely eliminated, resulting in a four-dimensional matrix with the third and sixth dimensions eliminated. Assuming the first 3D matrix is ​​(M1×M2×M3) and the second 3D matrix is ​​(N1×N2×N3), where M3 and N3 are the number of elements in the thickness dimension, the resulting four-dimensional matrix is ​​an (M1×M2×N1×N2) matrix.

[0227] Two-dimensional or three-dimensional matrices have shapes that are discernible to the human eye, while four-dimensional matrices do not. However, if we combine multiple dimensions of a four-dimensional matrix into one, arranging the element values ​​of those dimensions into a single dimension, we can reduce the four-dimensional matrix to a two- or three-dimensional matrix with a definite shape. In this case, a certain dimension of the two- or three-dimensional matrix is ​​the result of combining multiple dimensions of the four-dimensional matrix; it stretches the possible combinations of values ​​from multiple dimensions into a single dimension. For example, in the aforementioned (M1×M2×N1×N2) matrix, the first dimension of the four-dimensional matrix has M1 elements, and the second dimension has M2 elements. Stretching the first and second dimensions into one dimension results in (M1×M2) elements. Similarly, in a four-dimensional matrix, the fourth dimension has N1 elements, and the fifth dimension has N2 elements. Stretching the fourth and fifth dimensions into one dimension results in (N1×N2) elements. In this way, the four-dimensional matrix becomes a two-dimensional matrix, with (M1×M2) elements in one dimension and (N1×N2) elements in the other dimension. Normalizing this two-dimensional matrix (making the largest element value in the two-dimensional matrix 1 and dividing all other element values ​​in the two-dimensional matrix by the largest element value) yields the correlation score matrix.

[0228] The advantage of the above embodiment, which uses the first feature map and the transpose of the second feature map to obtain the correlation score matrix, is that the correlation score matrix can be obtained through simple matrix operations, which reduces processing overhead and improves processing efficiency.

[0229] The above operation of generating the correlation score matrix can be performed by Figure 4 The relevance score matrix generation module is executed. After the relevance score matrix generation module generates the relevance score matrix, in one embodiment, the relevance scores in the relevance score matrix generated by the relevance score matrix generation module can be adjusted by a self-attention model and an interactive attention model.

[0230] like Figure 18 As shown, in this embodiment, after step 910, the product defect identification method includes:

[0231] Step 911: Input the first feature map and the second feature map into the self-attention model to generate the first correlation score adjustment value;

[0232] Step 912: Input the first feature map and the second feature map into the interaction attention model to generate the second correlation score adjustment value;

[0233] Step 913: Adjust the correlation score matrix using the first correlation score adjustment value and the second correlation score adjustment value.

[0234] A self-attention model is a model that measures the interaction between multiple elements of an input to determine the degree of attention each element receives, thereby generating relevant adjustment values. In step 911, the first feature map is input into the self-attention model. Based on the interaction relationships between feature points in the first feature map, the model measures the degree of attention received by each feature point and generates a first correlation score adjustment value. This first correlation score adjustment value is applied to the feature points in the first feature map. Each feature point in the first feature map corresponds to a column in the correlation score matrix; therefore, each column in the correlation score matrix is ​​adjusted according to this first correlation score adjustment value.

[0235] Similarly, the second feature map is input into the self-attention model. The model measures the degree of attention received by each feature point in the second feature map based on their interaction relationships, and generates a first correlation score adjustment value accordingly. This first correlation score adjustment value is specific to the feature points in the second feature map and corresponds to a row in the correlation score matrix. Each row in the correlation score matrix is ​​adjusted according to this first correlation score adjustment value.

[0236] Figure 19A An exemplary structure of a self-attention model is shown. The self-attention model includes multiple self-attention model nodes, and the number of nodes enabled is determined by the number of received feature point values. Assume the first feature map includes U feature point values ​​P0, P1, ..., P2. U-1 Correspondingly, the self-attention model enables U self-attention model nodes L0, L1, ... L U-1 Where L is a positive integer greater than or equal to 2. The first self-attention model node L0, in addition to generating a first correlation score adjustment value a0 based on the input feature point P0, also generates its own state information S0 based on the input feature point P0, and passes it to the next self-attention model node L1. The (i+1)th (i = 1, 2, ..., U-1)th self-attention model node L... i On the one hand, based on the input feature point P i And the previous self-attention model node L i-1 The transmitted status information S i-1 This generates the corresponding first correlation score adjustment value a. i On the other hand, based on the input feature point P i And the previous self-attention model node L i-1 The transmitted status information S i-1 Generate its own state information S i To the next self-attention model node L i+1The state information here reflects the values ​​of U feature points P0, P1, ... P. U-1 The interaction between them. Thus, the adjusted values ​​of the first correlation score a0, a1, ... a U-1 This reflects the values ​​of U feature points P0, P1, ... P U-1 The mutual influence. Figure 19A The working process of the self-attention model is illustrated using only multiple feature point values ​​from the first feature map as input. The working process is similar when using multiple feature point values ​​from the second feature map as input.

[0237] The interactive attention model measures the interaction between multiple inputs with multiple elements, thereby generating relevant adjustment values. In step 912, a first feature map and a second feature map are input into the interactive attention model. Feature points in the first feature map and feature points in the second feature map interact with each other. The interactive attention model can measure the interaction between feature points in the first and second feature maps, and generate a second correlation score adjustment value accordingly. This second correlation score adjustment value is for a point pair consisting of one feature point in the first feature map and one feature point in the second feature map. When adjusting according to this second correlation score adjustment value, only one element value in the correlation score matrix is ​​adjusted, that is, the element value at the intersection of the column corresponding to the feature point in the first feature map and the row corresponding to the feature point in the second feature map.

[0238] Figure 19B An exemplary structure of an interactive attention model is shown. The interactive attention model includes multiple interactive attention model nodes, and the number of nodes activated is determined by the number of feature point values ​​received in the first feature map. Let's assume the first feature map includes U feature point values ​​P0, P1, ..., P2. U-1 For example, the interactive attention model enables U interactive attention model nodes G0, G1, ... G U-1 Additionally, the first interactive attention model node G0 also receives all V feature point values ​​Q0, Q1, ... Q from the second feature map. V-1 As the initial state information of the V-path, the first interactive attention model node G0, in addition to using the first feature point value P0 in the first feature map, also uses the initial state information Q0, Q1, ... Q of the V-path. V-1 This generates the first feature point value P0 in the first feature map and the second fractional adjustment value b for each feature point value in the second feature map. 0，0 b 0，1 ...b 0，V-1 In addition, based on the first feature point value P0 in the first feature map and the initial state information Q0, Q1, ... Q of the V-path, V-1This generates the state information S of the V-path itself. 0，0 S 0，1 ...S 0，V-1 The information is then passed on to the next interactive attention model node G1. The (i+1)th (i = 1, 2, ..., U-1)th interactive attention model node G... i In addition to the value P of the (i+1)th feature point in the first feature map i and the received V-channel status information S i-1，0 S i-1，1 ...S i-1，V-1 , producing P i The second fractional adjustment value b for each feature point value in the second feature map i，0 b i，1 ...b j，V-1 In addition, based on the value P of the (i+1)th feature point in the first feature map i and the received V-channel status information S i-1，0 S i-1，1 ...S i-1，V-1 This generates the state information S of the V-path itself. i，0 S i，1 ...S i，V-1 It is passed to the next interactive attention model node G. i+1 The state information transmitted by each node in the interaction attention model during the above process reflects the interaction between each feature point in the second feature map and the feature points in the first feature map. Therefore, the obtained second correlation score adjustment value can measure the mutual influence between the feature points in the first feature map and the feature points in the second feature map.

[0239] After obtaining the first and second correlation score adjustment values, the correlation score matrix can be adjusted in step 913 using these values. Specifically, since the first correlation score adjustment value corresponds to a column or row in the correlation score matrix, the element values ​​in that column or row corresponding to the first correlation score adjustment value can be adjusted using this value. Similarly, since the second correlation score adjustment value corresponds to a single element in the correlation score matrix, the element value corresponding to that second correlation score adjustment value can be adjusted using this value.

[0240] Figure 20A A schematic diagram of a correlation score matrix is ​​shown. The first feature map has 4×2 feature points, and the second feature map has 4×2 feature points. Therefore, the correlation score matrix is ​​a (4×2)×(4×2) matrix, which is an 8×8 matrix. Figure 20BThe diagram shows the eight adjusted first correlation scores obtained for eight feature points in the first feature map. The adjusted first correlation score for the third feature point P2 is 0.1, corresponding to the third column of the correlation score matrix; therefore, all elements in the third column are incremented by 0.1. The adjusted first correlation score for the eighth feature point P7 is 0.2, corresponding to the eighth column of the correlation score matrix; therefore, all elements in the eighth column are incremented by 0.2. The resulting correlation score matrix is ​​shown below. Figure 20C As shown.

[0241] Figure 20D The diagram shows the adjusted second correlation score values ​​corresponding to 64 point pairs formed by 8 feature points from the first feature map and 8 feature points from the second feature map. It is presented in matrix form, with a structure similar to the correlation score matrix. Figure 20D The element in the i-th row and j-th column of the matrix represents exactly the adjusted value of the second correlation score in the i-th row and j-th column of the correlation score matrix. Therefore, it can be... Figure 20C The matrix and Figure 20D Adding the matrices together, we get Figure 20E The matrix, i.e., the adjusted correlation score matrix.

[0242] The advantage of the embodiment of steps 911-913 above is that it uses a self-attention model to measure the interaction relationship between each feature point within the first feature map or the second feature map, and uses an interactive attention model to measure the interaction relationship between feature points between the first feature map and the second feature map, thereby generating corresponding score adjustment values ​​and adjusting the correlation score matrix, thereby improving the quality of the correlation score matrix and improving the accuracy of defect identification.

[0243] After completing step 913, as follows Figure 18 As shown, in step 920, multiple first feature points are selected in the first feature map. For example... Figure 8 As shown, the selected first feature points should be as many as possible to be inherent feature points of the first feature map, such as P1, P2 and P3.

[0244] Next, in step 930, for each selected first feature point, based on the correlation score matrix, the second feature point corresponding to the first feature point in the second feature map is obtained. Note that the second feature point here is not necessarily an inherent feature point of the second feature map. For example... Figure 8 As shown, it can be a part of each of multiple inherent feature points.

[0245] Since the correlation score matrix lists the correlation scores between each intrinsic feature point in the first feature map and each intrinsic feature point in the second feature map, the closer the result of mapping the feature point in the second feature map to the first feature point is, the higher the correlation score. Therefore, steps 910-930 can use the correlation score matrix to obtain the second feature point corresponding to the first feature point. The second feature point found in this way helps to improve the quality of the obtained set of matching feature point pairs and improve the accuracy of defect identification.

[0246] The implementation process of step 930 is described in detail below.

[0247] Further details of step 930

[0248] For each feature point selected in the first feature map, it corresponds to a column in the correlation score matrix. The correlation score represents how closely the feature point in the second feature map is mapped to the result of the first feature point in the second feature map. If, for a first feature point in the first feature map, there is a column in the correlation score matrix with a correlation score of 100%, it means that this is the feature point mapped to by the first feature point, and it can be selected as the second feature point. In one embodiment, if there is no correlation score of 100% in the above column, the feature point corresponding to the highest correlation score can be roughly selected as the second feature point.

[0249] However, as Figure 8 As shown, the second feature point is not necessarily an intrinsic feature point in the second feature map; it may cover a portion of each of multiple intrinsic feature points. Therefore, if the second feature point covers a portion of each of multiple intrinsic feature points, the covered intrinsic feature points can be found first, and then the second feature point can be located. Specifically, since the second feature point covers a maximum of four intrinsic feature points, in order to accurately locate the second feature point, the four associated intrinsic feature points must first be found. For example... Figure 8 As shown, for the second feature point P3', the four associated inherent feature points can be feature points with center point coordinates (2,2), (3,2), (2,3), and (3,3).

[0250] The four associated intrinsic feature points mentioned above are actually four feature points obtained by intersecting two adjacent rows and two adjacent columns on the second feature map. For example... Figure 12A As shown, a Cartesian coordinate system is established with the center of the intrinsic feature point at the top left corner of the first or second feature map as the origin, the horizontal direction to the right as the positive x-axis, and the vertical direction downwards as the positive y-axis. The center coordinates of the first feature point X on the first feature map are (i... a ,j aThe first feature point X maps to the second feature map, and the second feature point Y is not exactly an intrinsic feature point of the second feature map. The center coordinates of the four associated intrinsic feature points Y1-Y4 are respectively (i b ,j b ), (i b +1,j b ), (i b +1, j b ), (i b +1, j b +1).

[0251] The coordinates of the first feature point X are (i a ,j a This indicates that there is j above the first feature point X in the first feature map. a Rows, each row has W feature points. a To the left of the row containing the first feature point X, there is also i a There are 10 feature points, therefore, as Figure 12B As shown, the first feature point corresponds to column (j) of the correlation score matrix. a ·W a +i a The following will discuss the... Figure 12A The distribution of the correlation scores corresponding to the four inherent feature points Y1-Y4 in this column.

[0252] For the inherent feature point Y1(i) b j b ), in the second feature map, the inherent feature point Y1(i b j b Above ) is j b Rows, each row has W feature points. b The inherent feature point Y1(i b j b There is an i to the left of the row containing ) b There are 1 feature point, therefore, the inherent feature point Y1(i) b j b ) corresponds to row (j) of the correlation score matrix b ·W b +i b The corresponding correlation score is 0.1; the inherent feature point Y2(i b +1, j b ) corresponds to row (j) of the correlation score matrix b ·W b +i b +1), the corresponding correlation score is 0.3; inherent feature point Y3(i b j b+1) corresponds to row ((j) of the correlation score matrix. b +1)·W b +i b The corresponding correlation score is 0.2; the inherent feature point Y4(i b +1, j b +1) corresponds to row ((j) of the correlation score matrix. b +1)·W b +i b +1), the corresponding correlation score is 0.5.

[0253] Therefore, it can be seen that in column (j) a ·W a +i a In ), the inherent feature point Y1(i b j b ) and inherent feature point Y2(i b +1, j b The correlation scores corresponding to ) are adjacent, and the inherent feature points Y3(i) are adjacent. b j b +1) and inherent feature point Y4(i b +1, j b +1) The correlation scores corresponding to the adjacent features are also adjacent, and the inherent feature points Y2(i b +1, j b ) and inherent feature point Y3(i b j b The correlation score of +1 was separated by (W) b -2) correlation scores. The correlation scores corresponding to the four inherent feature points Y1-4 always exhibit the above distribution pattern.

[0254] In the above regarding Figure 12A The description of -B assumes that the second feature point is known and discusses the distribution of the four inherent feature points associated with the second feature point in the column corresponding to the first feature point in the correlation score matrix. However, the second feature point is actually unknown. Step 930 aims to find the second feature point by identifying the four inherent feature points associated with it. As mentioned above, the relative positions of the correlation scores corresponding to the four inherent feature points Y1-4 are fixed, therefore, it can be determined by... Figure 13A -C is shown as a sliding window to find the four associated intrinsic feature points.

[0255] In this embodiment, such as Figure 11 As shown, step 930 includes:

[0256] Step 1110: Set a sliding window in the column corresponding to the first feature point selected in the correlation score matrix;

[0257] Step 1120: Slide the sliding window in the column corresponding to the first feature point, and obtain the second feature point based on the four elements that fall in the sliding window during the sliding process.

[0258] A sliding window is a window that slides down. Generally, as the window slides, specific actions can be performed based on the content that falls into the window. For example... Figure 13A As shown in -C, the sliding window can include a first sliding window element W1, a second sliding window element W2, a third sliding window element W3, and a fourth sliding window element W4. A sliding window element is a unit within the sliding window, and its size exactly covers a correlation score in the correlation score matrix. Since W1-W4 can only be used in the column (j) corresponding to the first feature point... a ·W a +i a The elements slide within the first sliding window (W1), each covering a correlation score in a column. The second sliding window element W2 is below and immediately adjacent to the first sliding window element W1. The third sliding window element W3 is below the second sliding window element W2 and is separated from it by a row number equal to the number of columns W in the second feature map. b Subtract 2. The fourth sliding window element W4 is below and adjacent to the third sliding window element W3. This is consistent with the relative positional relationship of the correlation scores corresponding to the four inherent feature points Y1-4 as described above. Thus, the four sliding window elements W1-4 can cover the four inherent feature points Y1-4.

[0259] This allows the sliding window to slide within the column corresponding to the first feature point. Specifically, the first sliding window element W1 can be made to cover column (j) first. a ·W a +i a The topmost relevance score is determined, and the other three sliding window elements W2-4 change accordingly according to the above rules. This yields four relevance scores falling within the four sliding window elements W1-4. Next, the first sliding window element W1-4 is slid down one sliding window element's side length, again yielding four relevance scores falling within the four sliding window elements W1-4, until the fourth sliding window element W4 covers column (j). a ·W a +i a The last correlation score can only be lowered further.

[0260] like Figure 13A As shown, first cover column (j) from the first sliding window element W1. a ·W a +i a Starting with the topmost correlation score, the four correlation scores obtained are 0.1, 0.3, 0.2, and 0.5. Then, as... Figure 13BAs shown, each sliding window element W1-4 slides down by the length of one sliding window element's side, resulting in four correlation degrees of 0.3, 0.1, 0.5, and 0.1, respectively. Then, as... Figure 13C As shown, each sliding window element W1-4 slides down by the length of one more sliding window element, resulting in four relevance scores of 0.1, 0.2, 0.1, and 0.3 respectively. At this point, the fourth sliding window element W4 has covered the bottom relevance score of the column. The sliding ends.

[0261] Next, the second feature point is obtained based on the four elements that fall within the sliding window during its sliding process. In one embodiment, the second feature point is obtained based on the sum of the four correlation scores that fall within the sliding window. This is because if the center of the second feature point is exactly located between the centers of the four intrinsic feature points Y1-4 of the second feature point, or in other words, if the second feature point covers a portion of each of these four intrinsic feature points, the sum of the correlation scores corresponding to these four intrinsic feature points must be greater than the sum of the correlation scores of the other four intrinsic feature points with the same positional relationship on the second feature map. Therefore, the second feature point can be obtained based on the sum of the four correlation scores that fall within the sliding window at each position during the sliding process.

[0262] Specifically, in this embodiment, such as Figure 14 As shown, step 1120 includes:

[0263] Step 1410: For each sliding position during the sliding process of the sliding window, determine the sum of the four correlation scores falling within the sliding window;

[0264] Step 1420: Based on the four feature points corresponding to the four largest correlation scores in the second feature map, obtain the second feature points.

[0265] like Figure 15A As shown, when the first sliding window element W1 covers column (j a ·W a +i a When the topmost relevance score is reached, the sum of the four relevance scores falling within the sliding window is 0.1 + 0.3 + 0.2 + 0.5 = 1.1. When each sliding window element W1-4 slides down by the length of one sliding window element's side, as... Figure 15B As shown, the sum of the four correlation scores falling within the sliding window is 0.3 + 0.1 + 0.5 + 0.1 = 1.0. When each sliding window element W1-4 slides down by the length of one more sliding window element, as shown... Figure 15C As shown, the sum of the four correlation scores falling within the sliding window is 0.1 + 0.2 + 0.1 + 0.3 = 0.7. It can be seen that in... Figure 15AThe sliding position falls within the window where the sum of the four correlation scores is maximized. Therefore, in step 1420, the second feature point is obtained based on the four feature points corresponding to these four correlation scores in the second feature map. The specific method for obtaining this feature point will be described in detail below.

[0266] The advantage of steps 1410-1420 is that it obtains the second feature point based on the sum of the four correlation scores falling within the sliding window at each sliding position. If the center of the second feature point lies between the centers of the four feature points corresponding to the four correlation scores falling within the sliding window, the overall correlation score of these four feature points will be larger than the correlation scores of four other feature points with similar positional relationships. Obtaining the second feature point by comparing and maximizing the sum is simple and easy, greatly reducing the overhead of obtaining the second feature point.

[0267] The advantage of steps 1110-1120 is that, by using a sliding window, while maintaining the fixed positional relationship of the four inherent feature points associated with the second feature point, all possible four inherent feature points that satisfy the fixed positional relationship in the second feature map are traversed, and four associated inherent feature points used to obtain the second feature point are found. This improves the efficiency of obtaining the second feature point while maintaining the accuracy of obtaining the second feature point.

[0268] The following is a further explanation of the specific implementation of step 1420.

[0269] Further details of step 1420

[0270] Generally, feature points are extracted from pixels and have the same side length as the pixels. Therefore, the key to obtaining the second feature point in step 1420 is determining the position of the center of the second feature point. If the feature point is considered as a square, the center of the feature point refers to the center of the inscribed circle of the square.

[0271] In this embodiment, such as Figure 16 As shown, step 1420 includes:

[0272] Step 1610: Take any one of the four feature points as the first reference feature point, the feature point in the same row as the first reference feature point as the second reference feature point, the feature point in the same column as the first reference feature point as the third reference feature point, and the feature point in the four feature points that is neither in the same row nor in the same column as the first reference feature point as the fourth reference feature point.

[0273] Step 1620: Establish a coordinate system with the center of the first reference feature point as the origin, the direction from the origin to the center of the second reference feature point as the positive direction of the horizontal axis, the direction from the origin to the center of the third reference feature point as the positive direction of the vertical axis, and the standard side length of the feature point as the unit of the horizontal and vertical axes.

[0274] Step 1630: Determine the center coordinates of the second feature point based on the correlation scores of the first reference feature point, the second reference feature point, the third reference feature point, and the fourth reference feature point;

[0275] Step 1640: Obtain the second feature point based on the center coordinates of the second feature point and the standard side length of the feature point.

[0276] To determine the center of the second feature point, a coordinate system must first be established. Steps 1610-1620 describe the process of establishing a Cartesian coordinate system. Step 1420 involves obtaining the second feature based on the four feature points corresponding to the four feature points with the highest correlation scores in the second feature map. For ease of description in establishing the coordinate system, these four feature points are first named. For example... Figure 17A As shown, among the four feature points Y1-4, assuming Y1 is selected as the first reference feature point, then Y2, located in the same row as Y1, becomes the second reference feature point; Y3, located in the same column as Y1, becomes the third reference feature point; and Y4, located neither in the same row nor in the same column as Y1, becomes the fourth reference feature point. Note that the second, third, and fourth reference feature points change depending on the selected first reference feature point. For example, if Y2 is selected as the first reference feature point, then Y1 becomes the second reference feature point, Y4 becomes the third reference feature point, and Y3 becomes the fourth reference feature point.

[0277] like Figure 17A As shown, a Cartesian coordinate system is established with the center of the first reference feature point Y1 as the origin, the direction from the origin to the center of the second reference feature point Y2 as the positive x-axis, and the direction from the origin to the center of the third reference feature point Y3 as the positive y-axis. The standard side length of the feature points is used as the unit for the horizontal and vertical axes. Thus, the coordinates of the centers of Y1, Y2, Y3, and Y4 are (0,0), (1,0), (0,1), and (1,1), respectively. The center coordinates of the second feature point Y lie within the square formed by these four points.

[0278] Next, in step 1630, the center coordinates of the second feature point are calculated based on the correlation scores of the first to fourth reference feature points.

[0279] In one embodiment, step 1630 includes:

[0280] The x-coordinate of the center of the second feature point is determined by the sum of the correlation scores between the second and fourth reference feature points and the sum of the correlation scores between the first and third reference feature points.

[0281] The ordinate of the center of the second feature point is determined by the sum of the correlation scores of the third and fourth reference feature points and the sum of the correlation scores of the first and second reference feature points.

[0282] The center coordinates of the second feature point include its x-coordinate and y-coordinate, both between 0 and 1. The center coordinates of Y1, Y2, Y3, and Y4 are (0,0), (1,0), (0,1), and (1,1) respectively, forming a square centered at (0.5,0.5). The second feature point is located within this square. The x-coordinate of the second and fourth reference feature points is 1, while the x-coordinate of the first and third reference feature points is 0. If the correlation scores of the first to fourth reference feature points are all the same, then the left and right sides of the second feature point are equally matched with the first feature point, and the first feature point is mapped to the same position in the square; therefore, the x-coordinate of the second feature point is considered to be 0.5. If the correlation score between the second and fourth reference feature points (x-coordinate 1) is generally greater than the correlation score between the first and third reference feature points (x-coordinate 0), it indicates that the second feature point should be shifted to the right from the center, because the right side is closer to the position mapped to the first feature point. The rightward shift of the second feature point can be determined by summing the correlation scores between the second and fourth reference feature points, and summing the correlation scores between the first and third reference feature points. The greater the sum of the correlation scores between the second and fourth reference feature points is compared to the sum of the correlation scores between the first and third reference feature points, the greater the rightward shift of the second feature point's x-coordinate. The formula is set as follows:

[0283] x T =0.5+(S 11 +S 10 -S 00 -S 01 ) / 2 Formula 1

[0284] Where, x T 0.5 is the x-coordinate of the center of the second feature point, and 0.5 is the x-coordinate of the center of the square formed by the centers of the first to fourth reference feature points. 00 S is the correlation score of the first reference feature point. 10 S is the correlation score of the second reference feature point. 01 It is the correlation score of the third reference feature point, S 11 It is the correlation score of the fourth reference feature point, (S 11+S 10 -S 00 -S 01 ) / 2 is the distance that the x-coordinate of the second feature point should shift from 0.5 in the positive x-axis direction (if it is negative, it means shifting in the negative x-axis direction).

[0285] Similarly, the formula for calculating the ordinate of the second feature point can be set as follows:

[0286] y T =0.5+(S 11 +S 01 -S 00 -S 10 ) / 2 Formula 2

[0287] Among them, y T 0.5 is the ordinate of the center of the second feature point, and 0.5 is the ordinate of the center of the square formed by the centers of the first to fourth reference feature points. 00 S 10 S 01 and S 11 The definition is the same as in Formula 1, (S 11 +S 01 -S 00 -S 10 ) / 2 is the distance by which the ordinate of the second feature point should shift from 0.5 towards the positive y-axis (if it is negative, it means shifting towards the negative y-axis).

[0288] For example, from Figure 15A S was obtained from 00 =0.1, S 10 =0.3, S 01 =0.2, S 11 =0.5, substitute them into Formula 1, and we get x T =0.5 + (0.5 + 0.3 - 0.1 - 0.2) / 2 = 0.75. Substituting these values ​​into formula 2, we get y. T =0.5 + (0.5 + 0.2 - 0.1 - 0.3) / 2 = 0.65, therefore, if Figure 17B As shown, the coordinates of the center of the second feature point are (0.75, 0.65).

[0289] The advantage of the above embodiment lies in that it considers that within the square formed by the centers of the four reference feature points, the second and fourth reference feature points have the largest x-coordinates, while the first and third reference feature points have the smallest x-coordinates. The comparison between the sum of the correlation scores of the two reference feature points with the largest x-coordinates and the sum of the correlation scores of the two reference feature points with the smallest x-coordinates determines the degree of left or right offset of the second feature point from its center position. Therefore, it uses the sum of the correlation scores of the second and fourth reference feature points, and the sum of the correlation scores of the first and third reference feature points, to determine the x-coordinate of the center of the second feature point. Similarly, it uses the sum of the correlation scores of the third and fourth reference feature points, and the sum of the correlation scores of the first and second reference feature points, to determine the y-coordinate of the center of the second feature point. This method improves the efficiency of obtaining the second feature point.

[0290] Next, in step 1640, the second feature point is obtained based on the center coordinates of the second feature point and the standard side length of the feature point. For example... Figure 17B As shown, with a standard side length of 1, it can be determined that the second feature point is located within the square formed by (0.25, 0.15), (1.25, 0.15), (0.25, 1.15), and (1.25, 1.15).

[0291] The advantage of steps 1610-1640 is that it takes into account that the position of the center of the second feature point is related to the correlation scores of the first, second, third, and fourth reference feature points. The correlation scores of the first, second, third, and fourth reference feature points can determine the orientation of the center of the second feature point in the up, down, left, and right directions. Therefore, it determines the center coordinates of the second feature point based on the correlation scores of the first, second, third, and fourth reference feature points, which greatly improves the efficiency of obtaining the second feature point.

[0292] Description of apparatus and devices according to embodiments of this disclosure

[0293] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0294] It should be noted that in various specific embodiments of this application, when processing data related to the characteristics of the target object, such as target object attribute information or attribute information sets, is required, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application need to obtain target object attribute information, separate permission or consent from the target object will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the separate permission or consent of the target object will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0295] Figure 24 A schematic diagram of the structure of a product defect identification device 2400 provided in an embodiment of this disclosure. The product defect identification device 2400 includes:

[0296] The acquisition unit 2410 is used to acquire the product's image to be inspected and a reference image;

[0297] The generation unit 2420 is used to generate a first feature map of the image to be detected and a second feature map of the reference image;

[0298] The determining unit 2430 is used to determine the calibration matrix based on the first feature map and the second feature map;

[0299] The calibration unit 2440 is used to calibrate the first feature map using a calibration matrix to obtain the third feature map;

[0300] The first identification unit 2450 is used to identify defects in the product from the third feature map.

[0301] Optionally, the determination unit 2430 is specifically used for:

[0302] Obtain a set of matching feature point pairs, wherein the set of matching feature point pairs includes multiple matching feature point pairs, each matching feature point pair includes a first feature point and a second feature point, the first feature point comes from a first feature map, and the second feature point comes from a second feature map and corresponds to the first feature point;

[0303] The calibration matrix is ​​determined based on the set of matching feature point pairs.

[0304] Optionally, the determination unit 2430 is specifically used for:

[0305] Based on the first feature map and the second feature map, an association score matrix is ​​obtained, wherein each feature point in the first feature map corresponds to a column of the association score matrix, and each feature point in the second feature map corresponds to a row of the association score matrix. The elements in the association score matrix are equal to the association score of the feature point in the first feature map corresponding to the column in which the element is located, and the association score of the feature point in the second feature map corresponding to the row in which the element is located.

[0306] Select multiple first feature points in the first feature map;

[0307] For each selected first feature point, based on the correlation score matrix, the second feature point corresponding to the first feature point is obtained in the second feature map.

[0308] Optionally, the determination unit 2430 is specifically used for:

[0309] In the column corresponding to the first feature point selected in the correlation score matrix, a sliding window is set, wherein the sliding window includes a first sliding window element, a second sliding window element, a third sliding window element, and a fourth sliding window element. Each of the first, second, third, and fourth sliding window elements covers one correlation score in that column. The second sliding window element is below and adjacent to the first sliding window element. The third sliding window element is below the second sliding window element and the number of rows between it and the second sliding window element is the number of columns of the second feature map minus 2. The fourth sliding window element is below and adjacent to the third sliding window element.

[0310] The sliding window is made to slide in the column corresponding to the first feature point, and the second feature point is obtained based on the four elements that fall in the sliding window during the sliding process.

[0311] Optionally, the determination unit 2430 is specifically used for:

[0312] For each sliding position during the sliding process of the sliding window, determine the sum of the four correlation scores that fall within the sliding window;

[0313] The second feature point is obtained based on the four feature points corresponding to the four highest correlation scores in the second feature map.

[0314] Optionally, the determination unit 2430 is specifically used for:

[0315] Take any one of the four feature points as the first reference feature point, the feature point in the same row as the first reference feature point is the second reference feature point, the feature point in the same column as the first reference feature point is the third reference feature point, and the feature point in the four feature points that is neither in the same row nor in the same column as the first reference feature point is the fourth reference feature point.

[0316] A coordinate system is established with the center of the first reference feature point as the origin, the direction from the origin to the center of the second reference feature point as the positive direction of the horizontal axis, the direction from the origin to the center of the third reference feature point as the positive direction of the vertical axis, and the standard side length of the feature point as the unit of the horizontal and vertical axes.

[0317] Based on the correlation scores of the first reference feature point, the second reference feature point, the third reference feature point, and the fourth reference feature point, the center coordinates of the second feature point are determined.

[0318] The second feature point is obtained based on the center coordinates of the second feature point and the standard side length of the feature point.

[0319] Optionally, the determination unit 2430 is specifically used for:

[0320] The x-coordinate of the center of the second feature point is determined by the sum of the correlation scores between the second and fourth reference feature points and the sum of the correlation scores between the first and third reference feature points.

[0321] The ordinate of the center of the second feature point is determined by the sum of the correlation scores of the third and fourth reference feature points and the sum of the correlation scores of the first and second reference feature points.

[0322] Optionally, the determination unit 2430 is specifically used for:

[0323] After obtaining the correlation score matrix based on the first feature map and the second feature map, the first feature map and the second feature map are input into the self-attention model to generate the first correlation score adjustment value.

[0324] The first and second feature maps are input into the interaction attention model to generate a second correlation score adjustment value.

[0325] Adjust the correlation score matrix using the first correlation score adjustment value and the second correlation score adjustment value.

[0326] Optionally, the determination unit 2430 is specifically used for:

[0327] Perform the first number of times the first process, the first process including: extracting a second number of matching feature point pairs from the set of matching feature point pairs; calculating a candidate calibration matrix based on each matching feature point pair; determining a first error of the matching feature point pairs in the set of matching feature point pairs on the candidate calibration matrix; determining a third number of matching feature point pairs whose first error is less than a first threshold.

[0328] The candidate calibration matrix calculated by the first process with the largest third number in the first process is taken as the determined calibration matrix.

[0329] Optionally, the first feature map includes multiple first feature maps with different feature extraction ratios, the second feature map includes multiple second feature maps with different feature extraction ratios, and correspondingly, the third feature map includes multiple third feature maps with different feature extraction ratios.

[0330] The first identification unit 2450 is specifically used for:

[0331] Multiple third feature maps are input into the defect recognition model, which then identifies the defect based on these third feature maps.

[0332] Optionally, the product defect identification device 2400 also includes:

[0333] The second identification unit (not shown) is used to input multiple third feature maps into the defect classification model, and the defect classification model identifies the type of defect based on the identified defects.

[0334] The third identification unit (not shown) is used to input multiple third feature maps into the defect contour recognition model, which then identifies the contour of the identified defect.

[0335] Optionally, the product defect identification device 2400 further includes a training unit (not shown) for jointly training the defect identification model and the defect classification model in the following manner:

[0336] Construct a sample image set, which has multiple sample images. Each sample image has a first label and a second label. The first label indicates whether the sample image has a defect, and the second label indicates the type of defect.

[0337] The fourth feature map of the sample images in the generated sample image set;

[0338] The defect identification model is used to identify defects based on the fourth feature map, and the defect classification model is used to classify the identified defects based on the fourth feature map.

[0339] An error function is constructed based on the comparison between the identification results of the defect identification model and the first label, and the comparison between the classification results of the defect classification model and the second label.

[0340] The defect identification model and the defect classification model are trained using the error function.

[0341] As mentioned above, the defect identification device 150 can be implemented as a terminal or a server. When implemented as a terminal, refer to... Figure 25 , Figure 25 The structural block diagram of a terminal for implementing the product defect identification method of this embodiment includes: a radio frequency (RF) circuit 2510, a memory 2515, an input unit 2530, a display unit 2540, a sensor 2550, an audio circuit 2560, a wireless fidelity (WiFi) module 2570, a processor 2580, and a power supply 2590, etc. Those skilled in the art will understand that... Figure 25 The terminal structure shown does not constitute a limitation on mobile phones or computers and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0342] The RF circuit 2510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 2580; in addition, it transmits uplink data to the base station.

[0343] The memory 2515 can be used to store software programs and modules. The processor 2580 executes various terminal functions and data processing by running the software programs and modules stored in the memory 2515.

[0344] The input unit 2530 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 2530 may include a touch panel 2531 and other input devices 2532.

[0345] The display unit 2540 can be used to display input or provided information, as well as various menus of the terminal. The display unit 2540 may include a display panel 2541.

[0346] Audio circuitry 2560, speaker 2561, and microphone 2562 provide an audio interface.

[0347] In this embodiment, the processor 2580 included in the terminal can execute the product defect identification method of the previous embodiment.

[0348] The terminals disclosed in this embodiment include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0349] Figure 26 This is a partial structural block diagram of a server for implementing the product defect identification method of this disclosure. The server can vary significantly due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 2622 (e.g., one or more processors) and memory 2632, and one or more storage media 2630 (e.g., one or more mass storage devices) for storing application programs 2642 or data 2644. The memory 2632 and storage media 2630 may be temporary or persistent storage. The program stored in the storage media 2630 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 2622 may be configured to communicate with the storage media 2630 and execute the series of instruction operations in the storage media 2630 on the server.

[0350] The server may also include one or more power supplies 2626, one or more wired or wireless network interfaces 2650, one or more input / output interfaces 2658, and / or one or more operating systems 2641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0351] The processor 2622 in the server can be used to execute the product defect identification method of the present disclosure embodiments.

[0352] This disclosure also provides a computer-readable storage medium for storing program code for executing the product defect identification methods of the foregoing embodiments.

[0353] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the product defect identification method described above.

[0354] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0355] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0356] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0357] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0358] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0359] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0360] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0361] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0362] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A product defect identification method based on artificial intelligence, characterized in that, include: Acquire the product's image to be inspected and a reference image; Generate a first feature map of the image to be detected and a second feature map of the reference image; Obtain a set of matching feature point pairs, wherein the set of matching feature point pairs includes multiple matching feature point pairs, each of the matching feature point pairs includes a first feature point and a second feature point, the first feature point comes from the first feature map, and the second feature point comes from the second feature map and corresponds to the first feature point; Based on the set of matching feature point pairs, determine the calibration matrix; The first feature map is calibrated using the calibration matrix to obtain the third feature map; Identify defects in the product from the third feature map; The step of obtaining the set of matching feature point pairs includes: Based on the first feature map and the second feature map, a correlation score matrix is ​​obtained, wherein each feature point in the first feature map corresponds to a column of the correlation score matrix, and each feature point in the second feature map corresponds to a row of the correlation score matrix. The elements in the correlation score matrix are equal to the correlation score of the feature point in the first feature map corresponding to the column of the element and the correlation score of the feature point in the second feature map corresponding to the row of the element. The first feature map and the second feature map are input into the self-attention model to generate a first correlation score adjustment value; The first feature map and the second feature map are input into the interaction attention model to generate a second correlation score adjustment value; The correlation score matrix is ​​adjusted using the first correlation score adjustment value and the second correlation score adjustment value; Select multiple first feature points in the first feature map; For each selected first feature point, based on the adjusted correlation score matrix, a second feature point corresponding to the first feature point in the second feature map is obtained.

2. The method according to claim 1, characterized in that, For each selected first feature point, based on the correlation score matrix, the step of obtaining the second feature point corresponding to the first feature point in the second feature map includes: In the column corresponding to the first feature point selected in the correlation score matrix, a sliding window is set, wherein the sliding window includes a first sliding window element, a second sliding window element, a third sliding window element, and a fourth sliding window element. Each of the first, second, third, and fourth sliding window elements covers one correlation score in the column. The second sliding window element is below and adjacent to the first sliding window element. The number of rows between the third sliding window element and the second sliding window element is the number of columns of the second feature map minus 2. The fourth sliding window element is below and adjacent to the third sliding window element. The sliding window is made to slide in the column corresponding to the first feature point, and the second feature point is obtained based on the four elements that fall in the sliding window during the sliding process.

3. The method according to claim 2, characterized in that, The step of obtaining the second feature point based on the four elements falling within the sliding window during its sliding process includes: For each sliding position during the sliding process of the sliding window, determine the sum of the four correlation scores falling within the sliding window; The second feature point is obtained based on the four feature points corresponding to the four largest correlation scores in the second feature map.

4. The method according to claim 3, characterized in that, The process of obtaining the second feature points based on the four feature points corresponding to the four highest correlation scores in the second feature map includes: Take any one of the four feature points as the first reference feature point, the feature point in the same row as the first reference feature point is the second reference feature point, the feature point in the same column as the first reference feature point is the third reference feature point, and the feature point in the four feature points that is neither in the same row nor in the same column as the first reference feature point is the fourth reference feature point. A coordinate system is established with the center of the first reference feature point as the origin, the direction from the origin to the center of the second reference feature point as the positive direction of the horizontal axis, the direction from the origin to the center of the third reference feature point as the positive direction of the vertical axis, and the standard side length of the feature point as the unit of the horizontal and vertical axes. Based on the correlation scores of the first reference feature point, the second reference feature point, the third reference feature point, and the fourth reference feature point, the center coordinates of the second feature point are determined; The second feature point is obtained based on the center coordinates of the second feature point and the standard side length of the feature point.

5. The method according to claim 4, characterized in that, The step of determining the center coordinates of the second feature point based on the correlation scores of the first reference feature point, the second reference feature point, the third reference feature point, and the fourth reference feature point includes: The abscissa of the center is determined based on the sum of the correlation scores between the second reference feature point and the fourth reference feature point, and the sum of the correlation scores between the first reference feature point and the third reference feature point. The ordinate of the center is determined based on the sum of the correlation scores between the third and fourth reference feature points and the sum of the correlation scores between the first and second reference feature points.

6. The method according to claim 1, characterized in that, Determining the calibration matrix based on the set of matched feature point pairs includes: Perform a first number of first processes, the first process comprising: extracting a second number of matching feature point pairs from the set of matching feature point pairs; calculating a candidate calibration matrix based on the second number of matching feature point pairs; determining a first error of the matching feature point pairs in the set of matching feature point pairs on the candidate calibration matrix; and determining a third number of matching feature point pairs whose first error is less than a first threshold. The candidate calibration matrix calculated by the first process with the largest third number in the first process is used as the determined calibration matrix.

7. The method according to claim 1, characterized in that, The first feature map includes multiple first feature maps with different feature extraction ratios, the second feature map includes multiple second feature maps with different feature extraction ratios, and correspondingly, the third feature map includes multiple third feature maps with different feature extraction ratios. The step of identifying defects in the product from the third feature map includes: Multiple third feature maps are input into the defect recognition model, and the defect recognition model identifies defects based on the multiple third feature maps.

8. The method according to claim 7, characterized in that, After inputting multiple third feature maps into a defect recognition model, and having the defect recognition model identify defects based on the multiple third feature maps, the method further includes: Multiple third feature maps are input into a defect classification model, which then identifies the type of defect based on the identified defects. Multiple third feature maps are input into the defect contour recognition model, which then identifies the contour of the identified defect.

9. The method according to claim 8, characterized in that, The defect identification model and the defect classification model are jointly trained in the following manner: Construct a sample image set, which has multiple sample images. Each sample image has a first label and a second label. The first label indicates whether the sample image has a defect, and the second label indicates the type of the defect. Generate the fourth feature map of the sample images in the sample image set; The defect identification model is used to identify defects based on the fourth feature map, and the defect classification model is used to classify the identified defects based on the fourth feature map. An error function is constructed based on the comparison between the identification result of the defect identification model and the first label, and the comparison between the classification result of the defect classification model and the second label. The defect identification model and the defect classification model are trained using the error function.

10. A product defect identification device based on artificial intelligence, characterized in that, include: The acquisition unit is used to acquire the product's image to be inspected and a reference image; The generation unit is used to generate a first feature map of the image to be detected and a second feature map of the reference image; The determining unit is configured to determine a calibration matrix based on the first feature map and the second feature map; A calibration unit is used to calibrate the first feature map using the calibration matrix to obtain a third feature map; The first identification unit is used to identify defects in the product from the third feature map. Specifically, the determining unit is used for: Obtain a set of matching feature point pairs, wherein the set of matching feature point pairs includes multiple matching feature point pairs, each of the matching feature point pairs includes a first feature point and a second feature point, the first feature point comes from the first feature map, and the second feature point comes from the second feature map and corresponds to the first feature point; The calibration matrix is ​​determined based on the set of matching feature point pairs; When the determining unit performs the process of obtaining a set of matching feature point pairs, it is specifically used for: Based on the first feature map and the second feature map, a correlation score matrix is ​​obtained, wherein each feature point in the first feature map corresponds to a column of the correlation score matrix, and each feature point in the second feature map corresponds to a row of the correlation score matrix. The elements in the correlation score matrix are equal to the correlation score of the feature point in the first feature map corresponding to the column of the element and the correlation score of the feature point in the second feature map corresponding to the row of the element. After obtaining the correlation score matrix based on the first feature map and the second feature map, the first feature map and the second feature map are input into the self-attention model to generate the first correlation score adjustment value; The first feature map and the second feature map are input into the interaction attention model to generate a second correlation score adjustment value; The correlation score matrix is ​​adjusted using the first correlation score adjustment value and the second correlation score adjustment value; Select multiple first feature points in the first feature map; For each selected first feature point, based on the adjusted correlation score matrix, a second feature point corresponding to the first feature point in the second feature map is obtained.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the artificial intelligence-based product defect identification method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the artificial intelligence-based product defect identification method according to any one of claims 1 to 9.

Images