Method and system for identifying product defects on production line

By using the encoder and decoder structure of multi-layer, multi-scale deformable attention unit on the production line, redetermine the reference points and identify product defects, the problems of low efficiency and poor accuracy of traditional detection methods are solved, and efficient and accurate Product defect identification improves the overall efficiency of the production line.

CN119090834AInactive Publication Date: 2024-12-06GUANGZHOU HOUHUI TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411163493.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When facing large-scale and high-speed production lines, the traditional artificial quality inspection method has low detection efficiency, high error rate and high labor cost. The existing automated inspection system has low detection accuracy and poor adaptability, making it difficult to meet the growing product quality. need.

Method used

Using a multi-layer encoder and decoder structure with multi-scale deformable attention units, by taking production line product images, extracting feature maps and redefining reference points, focusing on sampling points that contribute greatly to detection, and identifying them in combination with prediction heads. Product defect type and location.

Benefits of technology

It improves the accuracy and efficiency of product defect identification, realizes all-round quality monitoring of diversified products, reduces the emergence of defective products, and improves the overall efficiency of the production line.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090834A_ABST
    Figure CN119090834A_ABST
Patent Text Reader

Abstract

The invention relates to the field of machine vision, in particular to a product defect identification method and system on a production line, and the method specifically comprises the steps: shooting a product image of the production line, and extracting the features of the product image through a backbone network to obtain at least one feature map; the at least one feature map serves as input of an encoder, the encoder comprises multiple layers of structures containing multi-scale deformable attention units, and each deformable attention unit in the multi-scale deformable attention units re-determines a reference point according to offset and weight of a sampling point of the reference point; taking the output of the encoder as the input of a decoder to obtain the output of the decoder; wherein the reference point of the last layer of the decoder is obtained according to the reference point of each layer in front of the decoder; and the output of the encoder passes through the prediction head to obtain the type and position of the product defect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a method and system for identifying product defects on a production line. Background Art

[0002] In the manufacturing industry, production efficiency continues to improve, and product quality control has become one of the core competitiveness of enterprises. Machine failures, unstable raw material quality, and unreasonable operators or production processes during the production process may all lead to product defects. However, traditional manual quality inspection methods have gradually shown problems such as low detection efficiency, high error rate, and high labor costs when facing large-scale, high-speed production lines, making it difficult to meet the growing demand for product quality. Although the existing automated inspection systems have improved the inspection speed to a certain extent, they are often limited by the complexity of the algorithm and hardware configuration, resulting in low detection accuracy and poor adaptability. Therefore, there is an urgent need for a more efficient and accurate method and system for identifying product defects on production lines to achieve all-round quality monitoring of diversified products, reduce the occurrence of defective products, and improve the overall efficiency of the production line. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a method for identifying product defects on a production line, the method comprising the following steps: Taking an image of a product on a production line, extracting features of the product image through a backbone network to obtain at least one feature map; using the at least one feature map as an input of an encoder, wherein the encoder includes a multi-layer structure including multi-scale deformable attention units, each deformable attention unit in the multi-scale deformable attention unit re-determines a reference point according to an offset and a weight of a sampling point of the reference point; The output of the encoder is used as the input of the decoder to obtain the output of the decoder; wherein the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder; The output of the encoder is passed through the prediction head to obtain the type and location of product defects.

[0004] Preferably, the re-determining the reference point according to the offset and weight of the sampling point of the reference point is specifically: Get the offset and weight of the reference point sampling point in each attention head, and determine the sampling point that contributes the most to the attention for each attention head; Each sampling point with the greatest contribution is input into the linear regression layer of the reference point to obtain multiple sampling points of each sampling point with the greatest contribution, and the average distances between the multiple sampling points of each sampling point with the greatest contribution and other sampling points with the greatest contribution are calculated, and the sampling point with the greatest contribution with the smallest average distance is used as a new reference point.

[0005] Preferably, after re-determining the reference point, the method further comprises: For the re-determined reference point, the features of the re-determined reference point and the features of all reference points that contribute most to each attention head are fused as the query features of the re-determined reference point.

[0006] Preferably, the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder, specifically: The reference point of the first layer in the decoder is used as the reference point, and the adjusted position and prediction score of each reference point in each layer are obtained according to the offset of the reference point of each previous layer; The adjusted position with the largest prediction score is used as the reference point position of the last layer.

[0007] Preferably, the decoder comprises multiple layers, each layer comprising a self-attention unit and a multi-scale deformable cross-attention unit.

[0008] In another aspect, the present invention further provides a product defect identification system on a production line, the system comprising the following modules: A feature extraction and encoding module is used to capture product images of the production line, extract features of the product images through a backbone network to obtain at least one feature map; the at least one feature map is used as an input of an encoder, the encoder includes a multi-layer structure containing multi-scale deformable attention units, each deformable attention unit in the multi-scale deformable attention unit re-determines the reference point according to the offset and weight of the sampling point of the reference point; A decoding module, used for taking the output of the encoder as the input of a decoder to obtain the output of the decoder; wherein the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder; The prediction module is used to obtain the type and location of product defects through the prediction head of the encoder output.

[0009] Preferably, the re-determining the reference point according to the offset and weight of the sampling point of the reference point is specifically: Get the offset and weight of the reference point sampling point in each attention head, and determine the sampling point that contributes the most to the attention for each attention head; Each sampling point with the greatest contribution is input into the linear regression layer of the reference point to obtain multiple sampling points of each sampling point with the greatest contribution, and the average distances between the multiple sampling points of each sampling point with the greatest contribution and other sampling points with the greatest contribution are calculated, and the sampling point with the greatest contribution with the smallest average distance is used as a new reference point.

[0010] Preferably, after re-determining the reference point, the method further comprises: For the re-determined reference point, the features of the re-determined reference point and the features of all reference points that contribute most to each attention head are fused as the query features of the re-determined reference point.

[0011] Preferably, the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder, specifically: The reference point of the first layer in the decoder is used as the reference point, and the adjusted position and prediction score of each reference point in each layer are obtained according to the offset of the reference point of each previous layer; The adjusted position with the largest prediction score is used as the reference point position of the last layer.

[0012] Preferably, the decoder comprises multiple layers, each layer comprising a self-attention unit and a multi-scale deformable cross-attention unit.

[0013] Finally, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented.

[0014] When calculating the attention of the reference point, the present invention redefines the reference point according to the offset and weight of the sampling point of the reference point, so that the attention is focused on the sampling point that contributes most to the attention, and determines the sampling point that contributes most to the attention in each attention head, and then redefines the new reference point; and the reference point of the last layer of the decoder is also redetermined, and the reference point of the last layer is determined according to the reference point of each layer before the decoder. The present invention improves the accuracy and efficiency of recognition by changing the position of the deformable attention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of Embodiment 1; Figure 2 Schematic diagram of deformable attention; Figure 3 This is a structural diagram of the second embodiment. DETAILED DESCRIPTION

[0016] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0017] The first embodiment, as Figure 1 As shown, this embodiment provides a method for identifying product defects on a production line, the method comprising the following steps: S1, taking an image of a product on a production line, extracting features of the product image through a backbone network to obtain at least one feature map; using the at least one feature map as an input of an encoder, the encoder comprising a multi-layer structure containing multi-scale deformable attention units, each deformable attention unit in the multi-scale deformable attention unit re-determines a reference point according to an offset and a weight of a sampling point of the reference point; A camera is installed on a production line, such as an assembly line, to capture images of products on the production line, and then at least one feature map is extracted through a backbone network, such as ResNet. Preferably, C3-C5 feature maps are extracted through ResNet, and C6 feature maps are obtained through convolution, and then each feature map channel is unified to 256. The production line includes, but is not limited to, production lines for automobiles, computers, bicycles, electric vehicles, mobile phones, electronic products, clothes, beverage filling, food, household appliances, etc.

[0018] After obtaining at least one feature map, at least one feature map is used as the input of the encoder. The encoder includes multiple layers, each layer includes a multi-scale deformable attention unit. Each deformable attention unit of the multi-scale deformable attention unit calculates the offset and weight of each sampling point according to the characteristics of the reference point. A reference point has multiple sampling points in an attention head. For example, if the number of attention heads is 5 and the number of sampling points is set to 4, then each attention head for a known reference point has 4 sampling points. If there are 5 attention heads, there will be 20 sampling points. Figure 2 The variable attention offset and weight of the three heads and four sampling points are shown. The sampling point is the part that the reference point focuses on. For the above example, one reference point focuses on four sampling points. If each attention head of the same reference point focuses on the same sampling point, it means that the sampling point is a key point. In order to play the role of the reference point, the present invention redefines the reference point according to the offset and weight of the sampling point. In one embodiment, the reference point is redetermined according to the offset and weight of the sampling point of the reference point, specifically: Get the offset and weight of the reference point in each attention head, and determine the sampling point that contributes most to attention for each attention head; Each sampling point corresponds to an offset relative to the reference point, and there is also an attention weight of the sampling point. The result of each attention head is determined according to the sampling point and the weight. Specifically, for an attention head, the attention weight of the sampling point is multiplied by the feature of the sampling point and then added. However, some sampling points contribute more to the result of each attention head, that is, they play a decisive role in the attention head, and this sampling point is taken as the sampling point with the greatest contribution to attention. The specific method of determining the sampling point with the greatest contribution to attention is: calculate the product result of the feature of the sampling point corresponding to the attention head and the weight of the sampling point, and take the sampling point with the largest product result as the sampling point with the greatest contribution to the attention head. For example, an attention head has 4 sampling points, and the values ​​of these four sampling points in the feature map where the reference point is located are v1-v4, and the weights are w1-w4, respectively. The product results are v1×w1, v2×w2, v3×w3, and v4×w4, respectively. The four largest ones are the sampling points with the greatest contribution to attention. If there are 5 attention heads, there will be 5 sampling points with the greatest contribution.

[0019] Each sampling point with the greatest contribution is input into the linear regression layer of the reference point to obtain multiple sampling points of each sampling point with the greatest contribution, and the average distances between the multiple sampling points of each sampling point with the greatest contribution and other sampling points with the greatest contribution are calculated, and the sampling point with the greatest contribution with the smallest average distance is used as a new reference point.

[0020] After obtaining all the sampling points with the largest contribution, each sampling point with the largest contribution is input into the linear regression layer (Linear) of the reference point to obtain a new sampling point. The linear regression layer is a linear regression layer that calculates the offset according to the query feature, such as Figure 2 Linear1. For each sampling point with the greatest contribution, multiple new sampling points can be obtained, and the average distance between the multiple new sampling points and other sampling points with the greatest contribution is calculated, and the sampling point with the smallest average distance is used as the new reference point. For example, for Head1, the sampling point with the greatest contribution is sp3, so sp3 is input into the linear regression layer of the reference point to obtain the sampling point of sp3, and the distance between these reference points and the sampling points with the greatest contribution in other attention heads is calculated. The smallest average distance indicates that sp3 also pays more attention to these areas. Selecting the sampling point with the smallest average distance as the new reference point can allow the new reference point to pay attention to the hot spot area.

[0021] In Deformable DETR, the feature of the reference point (also called the query feature z q ) is only derived from the reference point, and features of other parts are often ignored. In another embodiment, after the reference point is re-determined, the method further includes: For the re-determined reference point, the features of the re-determined reference point and the features of all reference points that contribute most to each attention head are fused as the query features of the re-determined reference point. For example, for the reference point rp1, the reference point is re-determined as rp1'. After determining the query feature z of rp1' q When the features of all the reference points with the largest contribution are merged into the features of rp1', the fusion method includes but is not limited to splicing, average pooling, etc. In another embodiment, the features of the re-determined reference points and the features of the original reference points are merged into new reference point features.

[0022] S2, using the output of the encoder as the input of the decoder to obtain the output of the decoder; wherein the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder; After obtaining the output of the encoder, the output of the encoder is input to the decoder. The decoder uses the DeformableDETR decoder. In order to improve the recognition accuracy, in the last layer of the decoder, the reference point is obtained according to the reference point of each previous layer of the decoder. For example, if the decoder has 6 layers, the reference point of the last layer, that is, the 6th layer, is determined based on the reference points of the first 5 layers. Specifically, the reference point of the first layer in the decoder is used as the reference point, and the adjusted position and prediction score of each reference point in each layer are obtained according to the offset of the reference point of each previous layer; the adjusted position with the largest prediction score is used as the reference point position of the last layer. For example, a reference point of the first layer of the decoder is rp5, and rp5 will participate in the calculation of the first layer of the decoder. For the output of the first layer, a prediction head will be used to predict the offset relative to the reference point, and the position of the reference point of the second layer of the decoder will be determined according to the offset; the output of the second layer will be predicted by a prediction head to predict the position offset relative to the reference point of the second layer, and so on. The reference point position will be re-determined for each layer.

[0023] If the decoder has 6 layers, then from the first layer to the fifth layer, a reference point will correspond to 4 re-determined reference points. Assuming that a reference point rp5 of the first layer is the reference point, the second layer corresponds to a new reference point rp5' based on the reference point rp5, where rp5'=rp5+offset1, and the third layer corresponds to a new reference point rp5'' based on the reference point rp5', where rp5''=rp5'+offset1. Such a reference point will have 5 reference points including the reference point, and a reference point is selected from these 5 reference points as the reference point of the last layer. Specifically, the reference point with the largest prediction score is selected as the reference point of the last layer, where the prediction score is a classification prediction of the reference point, and the higher the prediction score, the more likely this reference point is to be a target detection object. The decoder preferably uses the decoder in Deformable DETR, where the decoder includes multiple layers, each layer including a self-attention unit (TransformerSelf-Attention) and a multi-scale deformable cross-attention unit (Multi-scale Deformable Cross-Attention).

[0024] S3, the output of the encoder is passed through the prediction head to obtain the type and location of the product defect.

[0025] After obtaining the output of the encoder, the prediction head is used to further obtain the type and location information of the product defect, and the defect type and location are identified in the image. The second embodiment provides a product defect recognition system on a production line. Figure 3 As shown, the system includes the following modules: A feature extraction and encoding module is used to capture product images of the production line, extract features of the product images through a backbone network to obtain at least one feature map; the at least one feature map is used as an input of an encoder, the encoder includes a multi-layer structure containing multi-scale deformable attention units, each deformable attention unit in the multi-scale deformable attention unit re-determines the reference point according to the offset and weight of the sampling point of the reference point; A decoding module, used for taking the output of the encoder as the input of a decoder to obtain the output of the decoder; wherein the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder; The prediction module is used to obtain the type and location of product defects through the prediction head of the encoder output.

[0026] Preferably, the re-determining the reference point according to the offset and weight of the sampling point of the reference point is specifically: Get the offset and weight of the reference point sampling point in each attention head, and determine the sampling point that contributes the most to the attention for each attention head; Each sampling point with the greatest contribution is input into the linear regression layer of the reference point to obtain multiple sampling points of each sampling point with the greatest contribution, and the average distances between the multiple sampling points of each sampling point with the greatest contribution and other sampling points with the greatest contribution are calculated, and the sampling point with the greatest contribution with the smallest average distance is used as a new reference point.

[0027] Preferably, after re-determining the reference point, the method further comprises: For the re-determined reference point, the features of the re-determined reference point and the features of all reference points that contribute most to each attention head are fused as the query features of the re-determined reference point.

[0028] Preferably, the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder, specifically: The reference point of the first layer in the decoder is used as the reference point, and the adjusted position and prediction score of each reference point in each layer are obtained according to the offset of the reference point of each previous layer; The adjusted position with the largest prediction score is used as the reference point position of the last layer.

[0029] Preferably, the decoder comprises multiple layers, each layer comprising a self-attention unit and a multi-scale deformable cross-attention unit.

[0030] A third embodiment, this embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first embodiment.

[0031] A fourth embodiment, this embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described in the first embodiment can be implemented.

[0032] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0033] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0034] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0035] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0036] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. Among them, the server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

Claims

1. A method for identifying product defects on a production line, characterized in that: The method comprises the following steps: Taking an image of a product on a production line, extracting features of the product image through a backbone network to obtain at least one feature map; using the at least one feature map as an input of an encoder, wherein the encoder includes a multi-layer structure including multi-scale deformable attention units, each deformable attention unit in the multi-scale deformable attention unit re-determines a reference point according to an offset and a weight of a sampling point of the reference point; The output of the encoder is used as the input of the decoder to obtain the output of the decoder; wherein the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder; The output of the encoder is passed through the prediction head to obtain the type and location of product defects.

2. The method according to claim 1, characterized in that The reference point is re-determined according to the offset and weight of the sampling point of the reference point, specifically: Get the offset and weight of the reference point sampling point in each attention head, and determine the sampling point that contributes the most to the attention for each attention head; Each sampling point with the greatest contribution is input into the linear regression layer of the reference point to obtain multiple sampling points of each sampling point with the greatest contribution, and the average distances between the multiple sampling points of each sampling point with the greatest contribution and other sampling points with the greatest contribution are calculated, and the sampling point with the greatest contribution with the smallest average distance is used as a new reference point.

3. The method according to claim 2, characterized in that After re-determining the reference point, the method further includes: For the re-determined reference point, the features of the re-determined reference point and the features of all reference points that contribute most to each attention head are fused as the query features of the re-determined reference point.

4. The method according to claim 1, characterized in that The reference point of the last layer of the decoder is obtained according to the reference point of each layer before the decoder, specifically: The reference point of the first layer in the decoder is used as the reference point, and the adjusted position and prediction score of each reference point in each layer are obtained according to the offset of the reference point of each previous layer; The adjusted position with the largest prediction score is used as the reference point position of the last layer.

5. The method according to claim 1, characterized in that The decoder includes multiple layers, each of which includes self-attention units and multi-scale deformable cross-attention units.

6. A product defect recognition system on a production line, characterized in that: The system includes the following modules: A feature extraction and encoding module is used to capture product images of the production line, extract features of the product images through a backbone network to obtain at least one feature map; the at least one feature map is used as an input of an encoder, the encoder includes a multi-layer structure containing multi-scale deformable attention units, each deformable attention unit in the multi-scale deformable attention unit re-determines the reference point according to the offset and weight of the sampling point of the reference point; A decoding module, used for taking the output of the encoder as the input of a decoder to obtain the output of the decoder; wherein the reference point of the last layer of the decoder is obtained according to the reference point of each previous layer of the decoder; The prediction module is used to obtain the type and location of product defects through the prediction head of the encoder output.

7. The system according to claim 6, characterized in that The reference point is re-determined according to the offset and weight of the sampling point of the reference point, specifically: Get the offset and weight of the reference point sampling point in each attention head, and determine the sampling point that contributes the most to attention for each attention head; Each sampling point with the greatest contribution is input into the linear regression layer of the reference point to obtain multiple sampling points of each sampling point with the greatest contribution, and the average distances between the multiple sampling points of each sampling point with the greatest contribution and other sampling points with the greatest contribution are calculated, and the sampling point with the greatest contribution with the smallest average distance is used as a new reference point.

8. The system according to claim 7, characterized in that After re-determining the reference point, the method further includes: For the re-determined reference point, the features of the re-determined reference point and the features of all reference points that contribute most to each attention head are fused as the query features of the re-determined reference point.

9. The system according to claim 6, characterized in that The reference point of the last layer of the decoder is obtained according to the reference point of each layer before the decoder, specifically: The reference point of the first layer in the decoder is used as the reference point, and the adjusted position and prediction score of each reference point in each layer are obtained according to the offset of the reference point of each previous layer; The adjusted position with the largest prediction score is used as the reference point position of the last layer.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.