Product inspection method, device, system, and computer storage medium

Abstract

The application provides a product detection method, device and system and a computer storage medium, relates to the field of data processing, and the method comprises the following steps: image acquisition is performed on a product assembly production line to obtain a production line image; a product image including a product to be detected in the production line image is extracted; a detection point image of a component detection area in the product image is extracted; the detection point image is input into a detection model to obtain a detection result; and in the case that the detection result meets any one of the following conditions, it is determined that the product to be detected in the product image has a defect. The present scheme reduces the labor cost required for product inspection in the product assembly production line, avoids the missed detection and misdetected phenomenon caused by human negligence, and improves the accuracy of product detection on the product assembly production line.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a product testing method, apparatus, system, and computer storage medium. Background Technology

[0002] A product assembly line is a continuous production line that connects a series of component handling devices. Multiple workstations are set up on the production line, and various components are assembled sequentially by manual or automated mechanical conveyor belts to obtain the corresponding products.

[0003] Existing assembly lines for consumer products such as televisions mainly rely on manual product inspection, which not only consumes a lot of manpower but also may result in missed or false inspections due to human negligence, thus affecting the accuracy of product inspection. Summary of the Invention

[0004] The embodiments of this application provide a product testing method, apparatus, system, and computer storage medium to solve the problems in the prior art where the method of manually testing products on the product assembly line not only requires a lot of manpower, but also results in low accuracy of product inspection due to human negligence.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] On the one hand, a product testing method is provided, the method comprising:

[0007] Images of the product assembly line are acquired to obtain production line images;

[0008] Extract the product image that includes the product to be inspected from the production line image;

[0009] Extract the detection point image of the component detection area in the product image;

[0010] The detection point image is input into the detection model to obtain the detection result;

[0011] If the test results meet any of the following conditions, it is determined that the product under inspection in the product image has a defect:

[0012] The detection type of the detection point image in the detection result is a defect type;

[0013] The detection result shows that the edge box position of the detection point image exceeds the first edge box position range or the edge box position of the detection point image does not exist;

[0014] In the detection results, the feature similarity between the detection point image and the reference component image is less than the similarity threshold.

[0015] Optionally, inputting the detection point image into the detection model to obtain the detection result includes:

[0016] Determine the target type of the detection model corresponding to the component detection area;

[0017] The detection point image is input into the detection model of the target type to obtain the detection result;

[0018] The target type of the detection model includes any one of the following: classifier type, segmenter type, and image template type.

[0019] Optionally, if the detection model is a classifier type, the step of inputting the detection point image into the target type detection model to obtain the detection result includes:

[0020] Extract the first image feature from the detection point image;

[0021] The first image feature is input into a detection model of the classifier type to obtain the confidence level of the first image feature for each detection type.

[0022] The detection type with the highest confidence level is the detection type of the detection point image, wherein the detection type of the detection point image includes at least a compliance type or a defect type.

[0023] Optionally, when the detection model is of the segmenter type, the step of inputting the detection point image into the target type detection model to obtain the detection result includes:

[0024] Extract second image features from the detection point image;

[0025] The second image feature is input into a segmenter-type detection model to obtain edge feature points of the second image feature that meet the color threshold.

[0026] The positions of the edge feature points are used as the bounding box positions of the detection point image.

[0027] Optionally, when the detection model is of image template type, the step of inputting the detection point image into the target type detection model to obtain the detection result includes:

[0028] Extract third image features from the detection point image, and extract reference image features from the reference component image;

[0029] The third image features are matched with the reference image features to obtain the feature similarity between the detection point image and the reference component image.

[0030] Optionally, extracting the product image including the product to be inspected from the production line image includes:

[0031] Edge detection is performed on the production line image to obtain the edge box position of the product to be inspected in the production line image;

[0032] The production line image within the range of the second edge frame position of the product to be inspected is used as the product image.

[0033] Optionally, extracting the product image including the product to be inspected from the production line image includes:

[0034] Perform color segmentation on the production line image to obtain the image proportion of the target color or target texture in the production line image;

[0035] Production line images with an image proportion greater than the target color or target texture threshold are used as product images.

[0036] Optionally, before extracting the detection point image of the component detection area in the product image, the method further includes:

[0037] Extract the first feature points from the product image;

[0038] The first feature point is compared with the second feature point of the reference product image to obtain the matching feature point. The reference product image is a product image that has been pre-annotated with compliance inspection points and meets the requirements of rotation and translation.

[0039] Based on the matched feature points, the coordinate transformation parameters are obtained;

[0040] The product image is registered according to the coordinate transformation parameters.

[0041] Optionally, before extracting the product image that includes the product to be inspected from the production line image, the method further includes:

[0042] The brightness of the production line image is optimized using a targeted approach.

[0043] Optionally, after determining that the product to be inspected in the product image has a defect, the method further includes:

[0044] The system controls the product assembly line to stop operating and outputs alarm messages for defective products.

[0045] On the other hand, a product testing device is provided, the device comprising:

[0046] The imaging module is used to acquire images of the product assembly line and obtain images of the production line.

[0047] The first extraction module is used to extract product images that include the products to be inspected from the production line images;

[0048] The second extraction module is used to extract the detection point image of the component detection area in the product image;

[0049] The processing module is used to input the detection point image into the detection model to obtain the detection result;

[0050] The determination module is configured to determine that the product under inspection in the product image has a defect if the detection result meets any of the following conditions:

[0051] The detection type of the detection point image in the detection result is a defect type;

[0052] The detection result shows that the edge box position of the detection point image exceeds the first edge box position range or the edge box position of the detection point image does not exist;

[0053] In the detection results, the feature similarity between the detection point image and the reference component image is less than the similarity threshold.

[0054] Optionally, the processing module is further configured to:

[0055] Determine the target type of the detection model corresponding to the component detection area;

[0056] The detection point image is input into the detection model of the target type to obtain the detection result;

[0057] The target type of the detection model includes any one of the following: classifier type, segmenter type, and image template type.

[0058] Optionally, if the detection model is a classifier type, the processing module is further configured to:

[0059] Extract the first image feature from the detection point image;

[0060] The first image feature is input into a detection model of the classifier type to obtain the confidence level of the first image feature for each detection type.

[0061] The detection type with the highest confidence level is the detection type of the detection point image, wherein the detection type of the detection point image includes at least a compliance type or a defect type.

[0062] Optionally, if the detection model is a segmenter type, the processing module is further configured to:

[0063] Extract second image features from the detection point image;

[0064] The second image feature is input into a segmenter-type detection model to obtain edge feature points of the second image feature that meet the color threshold.

[0065] The positions of the edge feature points are used as the bounding box positions of the detection point image.

[0066] Optionally, if the detection model is of image template type, the processing module is further configured to:

[0067] Extract third image features from the detection point image, and extract reference image features from the reference component image;

[0068] The third image features are matched with the reference image features to obtain the feature similarity between the detection point image and the reference component image.

[0069] Optionally, the first extraction module is further configured to:

[0070] Edge detection is performed on the production line image to obtain the edge box position of the product to be inspected in the production line image;

[0071] The production line image within the range of the second edge frame position of the product to be inspected is used as the product image.

[0072] Optionally, the first extraction module is further configured to:

[0073] Perform color segmentation on the production line image to obtain the image proportion of the target color or target texture in the production line image;

[0074] Production line images with an image proportion greater than the target color or target texture threshold are used as product images.

[0075] Optionally, the device further includes:

[0076] Registration module, used for:

[0077] Extract the first feature points from the product image;

[0078] The first feature point is compared with the second feature point of the reference product image to obtain the matching feature point. The reference product image is a product image that has been pre-annotated with compliance inspection points and meets the requirements of rotation and translation.

[0079] Based on the matched feature points, the coordinate transformation parameters are obtained;

[0080] The product image is registered according to the coordinate transformation parameters.

[0081] Optionally, the device further includes:

[0082] The preprocessing module is also used to optimize the brightness of the production line image using a target method;

[0083] Optionally, the device further includes:

[0084] The alarm module is used to control the product assembly line to stop operating and output alarm prompts for defective products.

[0085] In another aspect, a product inspection system is provided, the system comprising: a product assembly line, a camera disposed toward the product assembly line, and an industrial control computer, the industrial control computer being used to execute the above-described product inspection method.

[0086] In another aspect, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, enables the above-described product testing method.

[0087] The embodiments of this application provide a product inspection method, apparatus, system, and computer storage medium. This solution extracts product images containing the products to be inspected from production line images captured by the product assembly line, and extracts inspection point images of the components to be inspected from the product images and inputs them into the inspection model for product inspection to identify defective products to be inspected. This reduces the labor costs required for product inspection in the product assembly line, avoids missed inspections and false inspections due to human error, and improves the accuracy of product inspection in the product assembly line. Attached Figure Description

[0088] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0089] Figure 1 A schematic flowchart of a product testing method provided in an embodiment of this application;

[0090] Figure 2 A schematic flowchart of another product testing method provided in an embodiment of this application;

[0091] Figure 3 A schematic flowchart illustrating a product image extraction method provided in an embodiment of this application;

[0092] Figure 4 A schematic flowchart of another product image extraction method provided in an embodiment of this application;

[0093] Figure 5 A flowchart illustrating the steps of a detection result acquisition method provided in this application embodiment;

[0094] Figure 6 A schematic diagram of the screw inspection results provided in an embodiment of this application;

[0095] Figure 7 A flowchart illustrating the steps of another detection result acquisition method provided in this application embodiment;

[0096] Figure 8 A schematic diagram illustrating the inspection results of the adhesive tape and plug provided in the embodiments of this application;

[0097] Figure 9 A flowchart illustrating the steps of another method for obtaining detection results provided in this application embodiment;

[0098] Figure 10 A structural block diagram of a product testing device provided in an embodiment of this application;

[0099] Figure 11 A schematic diagram of the structure of a product testing system provided in this application embodiment. Detailed Implementation

[0100] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0101] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity or execution order.

[0102] Example 1

[0103] Reference Figure 1 This application provides a product testing method, the method comprising:

[0104] Step 101: Acquire images of the product assembly line to obtain production line images.

[0105] In this embodiment, the product assembly line refers to a production line that transports various components of a product via conveyor belts to assembly workstations to assemble a complete product. Each workstation can be used for manual assembly or automated mechanical assembly. In the prior art, dedicated personnel are typically assigned to inspect the assembled products on the product assembly line. This method not only requires a large amount of manpower but also suffers from errors and omissions due to human factors, reducing the accuracy of product inspection. The production line image is obtained by capturing images of the conveyor belt of the product assembly line using cameras installed on the production line.

[0106] This application's embodiments abandon the existing method of manually inspecting products on the product assembly line. Instead, they use images of the product assembly line obtained through image acquisition to provide data for subsequent product inspection steps. It should be noted that the image acquisition frequency of the product assembly line can be determined based on the conveyor belt's efficiency in transporting products, or it can be a preset fixed shooting frequency. The specific frequency can be determined according to actual needs and is not limited here.

[0107] Step 102: Extract the product image that includes the product to be inspected from the production line image.

[0108] In this embodiment, a product image refers to an image of the product to be inspected that is displayed. The product to be inspected is a product that has not undergone product inspection, obtained by assembling various components on a product assembly line.

[0109] Because products can be transported at certain intervals on a conveyor belt on a product assembly line, meaning there is some empty space between products on the conveyor belt, the production line images obtained by taking pictures of the product assembly line may not contain all products, or may only contain some products. Before product inspection, it is necessary to extract product images containing the products to be inspected from the production line images, thereby avoiding inspection of production line images that do not contain the products to be inspected, which would affect the effectiveness of product inspection.

[0110] Step 103: Extract the detection point image of the component detection area in the product image.

[0111] In this embodiment, the component detection area refers to the image display area in the product image where the component to be detected is located. This component detection area is preset by the user according to the actual detection requirements of the product to be inspected. The detection point image refers to the image in the product image located within the component detection area.

[0112] It should be noted that the detection point image can be a partial image or the entire image of the product. The specific determination can be based on the relative size of the component to be inspected in the product image. That is, if the component to be inspected occupies most of the image content of the product image, the detection point image can be the entire image of the product image. If the component to be inspected occupies only a small part of the image content of the product image, the detection point image can be a partial image of the product image. Of course, the relative size of the detection point image and the product image can be set according to actual needs, subject to the feasibility of this solution, and is not limited here.

[0113] Step 104: Input the detection point image into the detection model to obtain the detection result.

[0114] In this embodiment, the detection model is used to detect the component to be inspected in the detection point image. This detection model can be a machine learning model or a data model. It is understood that whether the component to be inspected in the product is assembled correctly can be determined by the detection point image showing the required assembly of the component. For example, if a screw is missing, the image of the screw will not appear in the detection point image; or if a plug is inserted incorrectly, the position of the plug in the detection point image will clearly be outside the correct plug position, and so on. In the prior art, manual inspection of products also involves identifying the patterns of each detection point in the product. Therefore, this embodiment uses a detection model to replace manual inspection, which can reduce the manpower required for product inspection and avoid the impact of human error on the accuracy of product inspection.

[0115] Different detection models yield different results. For example, a machine learning model based on deep learning can predict the product under inspection, i.e., determine whether the product has defects; while a data model may only provide parameters such as brightness, chromaticity, and contrast of the inspection point image, requiring further analysis to determine whether the product under inspection in that inspection point image has defects. Of course, the specific detection model can be determined according to actual needs, as long as it can determine whether the component under inspection in the inspection point image has defects; no limitation is made here.

[0116] Step 105: If the detection result meets any of the following conditions, it is determined that the product under inspection in the product image has a defect:

[0117] The detection type of the detection point image in the detection result is a defect type;

[0118] The detection result shows that the edge box position of the detection point image exceeds the first edge box position range or the edge box position of the detection point image does not exist;

[0119] In the detection results, the feature similarity between the detection point image and the reference component image is less than the similarity threshold.

[0120] In this embodiment, the detection type of the detection point image is the type of detection point image predicted by the detection model, while the defect type is the type of defect present in the product to be inspected in the detection point image. It can be understood that if the detection model identifies the detection point image as a defect type, it can be determined that the product to be inspected in the detection point image has a defect.

[0121] The bounding box position of the inspection point image is the position of the bounding box of the product to be inspected extracted by the inspection model from the inspection point image. The first bounding box position range is the range of bounding box positions that the product to be inspected in the inspection point image should fall within when the product to be inspected is a compliant product. It can be understood that if the bounding box position of the target inspection component in the product to be inspected in the inspection point image exceeds a certain range, it can be considered that the target inspection component is incorrectly assembled in the product to be inspected, or that the bounding box position of the target inspection component in the product to be inspected is not detected, and it can be considered that the target inspection component is not assembled. In either case, the product to be inspected is considered to have a defect.

[0122] The feature similarity between the inspection point image and the reference component image refers to the similarity of image features between the two images. The similarity threshold is the minimum feature similarity between the product under inspection in the inspection point image and the reference component image required for the product to be compliant. In other words, if the feature similarity between the inspection point image and the reference component image is too low, it indicates a significant difference between the product under inspection in the inspection point image and a compliant product, thus confirming a defect in the product under inspection in that inspection point image.

[0123] Furthermore, if the detection result is the number of non-compliant parts among multiple parts to be inspected, it could mean that the number of defective parts is below a certain threshold. Of course, the above is merely an example; the specific determination depends on actual needs and is not limited here.

[0124] This application provides a product inspection method that extracts product images containing the products to be inspected from production line images captured by the product assembly line, and extracts inspection point images of the components to be inspected from the product images and inputs them into an inspection model to identify defective products. This reduces the labor costs required for product inspection in the product assembly line, avoids missed and false inspections due to human error, and improves the accuracy of product inspection in the product assembly line.

[0125] Example 2

[0126] Reference Figure 2 This application provides another product testing method, the method comprising:

[0127] Step 201: Image acquisition is performed on the product assembly line to obtain production line images.

[0128] This step can be referred to in the detailed description of step 101, and will not be repeated here.

[0129] Step 202: Optimize the brightness of the production line image using a target method.

[0130] The target method includes at least one of histogram equalization and gamma calibration.

[0131] In this embodiment, the uneven lighting distribution in the product assembly line environment can cause some images in the obtained production line images to be too bright or too dark, affecting the accuracy of subsequent product inspection. Therefore, it is necessary to optimize the brightness of the acquired production line images. Specifically, image brightness optimization methods such as histogram equalization and gamma correction can be used to process the production line images. Histogram equalization and gamma correction can both be implemented using existing technologies, which will not be elaborated here. Of course, other image brightness optimization methods can also be used to process the production line images, as long as they can improve the image brightness quality of the production line images; these will not be elaborated here either.

[0132] This application embodiment optimizes the brightness of the acquired production line images, thus avoiding the situation where the image quality of the production line images is low due to the uneven light distribution in the environment where the product assembly production line is located.

[0133] Step 203: Extract the product image that includes the product to be inspected from the production line image.

[0134] This step can be referred to in the detailed description of step 102, and will not be repeated here.

[0135] Optional, refer to Figure 3 Step 203 may include:

[0136] Sub-step 2031: Perform edge detection on the production line image to obtain the edge box position of the product to be inspected in the production line image.

[0137] In this embodiment, edge detection is a conventional technique for detecting sets of pixels in an image whose grayscale values ​​change drastically. Since there is a significant difference in grayscale between the conveyor belt and the product in a product assembly line, edge detection is performed on the production line image to obtain the location of the set of pixels in the production line whose grayscale values ​​change drastically, which is the edge box location of the product to be inspected in the production line image.

[0138] Sub-step 2032: The production line image within the second edge frame position range of the edge frame position of the product to be inspected is used as the product image.

[0139] In this embodiment, the second edge box position range refers to the required edge box range within the product image where the edge box of the product to be inspected should be located. It is understood that, in order for the product image participating in subsequent inspection to contain the component to be inspected within the product to be inspected, it is necessary to ensure that the edge box position of the product to be inspected in the product image is within a certain range. That is, if the edge box of the product to be inspected is within the second edge box position range, it can be determined that the production line image contains the component to be inspected within the product to be inspected. The specific second edge box position range can be predetermined according to the actual needs of product inspection, as long as the obtained product image contains the component to be inspected within the product to be inspected; no limitation is made here.

[0140] This application embodiment extracts product images containing the products to be inspected from production line images based on edge detection, thus ensuring the validity of the product images participating in product inspection.

[0141] Optional, refer to Figure 4 Step 203 may include:

[0142] Sub-step 2033: Perform color segmentation on the production line image to obtain the image proportion of the target color or target texture in the production line image.

[0143] In the embodiments of this application, color segmentation refers to an algorithm that segments an image based on a specific color or texture in the image. The color threshold is determined in advance based on the target color or target texture of the product to be inspected in the product image. The target color and target texture are the color and texture of the product to be inspected.

[0144] By performing color segmentation on the production line image, the target pixels containing the target color or target texture in the production line image are obtained. The ratio of the number of these target pixels to the total number of pixels in the production line image is then used as the image proportion of the target color or target texture in the production line image.

[0145] Sub-step 2034: The production line image with an image proportion greater than the image proportion threshold of the target color or target texture is used as the product image.

[0146] In this embodiment, the image proportion threshold refers to the required percentage of pixels representing the target color or texture in the product image. Considering that the image of the target color or texture in the product image should not be too small, this proportion threshold needs to be at least greater than 50%, such as 60% or 70%, to ensure the product to be inspected in the obtained product image. The specific threshold can also be set according to the actual needs of product inspection and is not limited here. If the image proportion of the target color or texture in the production line image is greater than this image proportion threshold, then the production line image can be used as the product image for subsequent image inspection.

[0147] This application embodiment extracts product images containing the products to be inspected from production line images based on color segmentation, thus ensuring the validity of the product images used for product inspection.

[0148] It should be noted that extracting product images through color segmentation works well for products with obvious colors or textures, such as circuit boards that are green overall. For products with indistinct overall colors or textures, the edge segmentation method in steps 2031 to 2032 works better.

[0149] Step 204: Extract the first feature point of the product image.

[0150] In this embodiment of the application, the first feature point refers to the image feature point in the product image, which can be obtained based on feature extraction algorithms such as LBP (Local Binary Pattern), HOG (Histogram of Oriented Gradient), and SIFT (Scale Invariant Feature Transform).

[0151] Step 205: Compare the similarity between the first feature point and the second feature point of the reference product image to obtain the matching feature point. The reference product image is a product image that has been pre-annotated with compliance inspection points and meets the requirements for rotation and translation.

[0152] In this embodiment, the reference product image is a pre-selected product image with good image quality, small rotation and translation, and where the components at each detection point are properly assembled. The reference product image also includes the detection point images of each component to be inspected. The second feature point refers to an image feature point in the reference product image. The extraction method for the second feature point is similar to that for the first feature point; please refer to the description of the first feature point extraction method in step 204 for details. Furthermore, when selecting the reference product image, the rotation and translation parameters of the reference product image can be obtained using RANSAC (Random Sample Consensus).

[0153] By comparing the similarity of the first feature point and the second feature point, the first feature point and the second feature point with higher similarity are used as the matching feature points.

[0154] Step 206: Obtain coordinate transformation parameters based on the matched feature points.

[0155] In this embodiment, the coordinate transformation parameter is the transformation parameter used to transform the first feature point into the second feature point.

[0156] Step 207: Perform image registration on the product image according to the coordinate transformation parameters.

[0157] In this embodiment, the coordinates of each pixel in the product image are transformed based on the coordinate transformation parameters, thereby registering the product image and aligning the product detection areas with those in the reference product image.

[0158] This application embodiment registers the obtained product image with a reference product image, thereby standardizing the position of each product detection area in the product image, reducing the amount of processing required for subsequent product detection, and improving the efficiency of product detection.

[0159] Step 208: Extract the detection point image of the component detection area in the product image.

[0160] This step can be referred to in the detailed description of step 103, and will not be repeated here.

[0161] Step 209: Determine the target type of the detection model corresponding to the component detection area.

[0162] In this embodiment, the target type of the detection model refers to the type of detection model required for different component detection areas. The association between different component detection areas and the target type of the detection model is pre-configured, specifically determined according to the type of component to be detected in the component detection area, or it can be pre-configured by the user according to their own needs. The specific configuration can be determined based on actual requirements, and will not be elaborated further here. The association between component detection areas and the target type of the detection model can be pre-stored as a configuration file. When performing file detection, this configuration file is read to determine the target type of the detection model corresponding to each component detection area.

[0163] Step 210: Input the detection point image into the detection model of the target type to obtain the detection result.

[0164] The target type of the detection model includes any one of the following: classifier type, segmenter type, and image template type.

[0165] In this embodiment, the classifier-type detection model can be a machine learning model trained by pre-extracting features from sample product images containing both defective and compliant products and then inputting these features into a classifier to be trained. This classifier-type detection model can be implemented based on classifier algorithms such as SVM (Support Vector Machine) or Random Forest, and the feature extraction can be based on feature extraction algorithms such as LBP, HOG, or SIFT.

[0166] Segmenter-type detection models can be machine learning models trained by pre-extracting features from sample product images containing both defective and compliant products and then feeding these features into a segmenter to be trained. This segmenter can be implemented using algorithms such as color-based pre-set segmentation set connectivity analysis. Feature extraction is similar to that of classifier-type detection models and will not be elaborated upon here.

[0167] Image template-type detection models can be image similarity detection data models created in advance using sample product images of compliant products, or they can be image similarity detection data models created using sample product images of defective products.

[0168] This application embodiment performs product detection by inputting the detection point image into a detection model of the target type corresponding to its component detection area. This allows the product detection method to be configured by configuring the association between the component detection area and the target type of the detection model, thereby improving the flexibility of product detection.

[0169] Optional, refer to Figure 5 When the detection model is of type classifier, step 210 includes:

[0170] Sub-step 2101: Extract the first image feature from the detection point image.

[0171] Sub-step 2102 involves inputting the first image features into a detection model of the classifier type to obtain the confidence level of the first image features for each detection type.

[0172] Sub-step 2103: The detection type with the highest confidence is the detection type of the detection point image, wherein the detection type of the detection point image includes at least a compliance type or a defect type.

[0173] In this embodiment, the first image feature can be obtained using feature extraction algorithms such as LBP, HOG, and SIFT. A classifier-type detection model can predict the detection type of the component to be detected in the detection point image, specifically classifying it into defect types and compliance types. The prediction result of the classifier-type detection model includes the confidence level of the detection point image for each detection type. Typically, the detection type with the highest confidence level is output as the detection type of the detection point image.

[0174] In practical applications, classifier-type detection models are suitable for detecting products with fixed shapes, such as screws and plugs. This is because the image features of fixed-shape components are relatively obvious, allowing classifier-type detection models to effectively distinguish them. For example, refer to... Figure 6 This shows the test results for different screws in a certain electronic product.

[0175] In this model, A1, B1, C1, and D2 are the detected defective screws, while A2, B2, C2, and D1 are the detected compliant screws. It is evident that screws in A1, B1, C1, and D2 are not installed at their designated positions, while screws in A2, B2, C2, and D1 are compliantly installed. For screws, which are components with a fixed shape, classifier-type detection models can effectively perform product inspection.

[0176] This application embodiment improves the accuracy of product detection for parts with fixed shapes by using a detection model of classifier type to perform product detection on product images.

[0177] Optional, refer to Figure 7 When the detection model is of the segmenter type, step 210 includes:

[0178] Sub-step 2104: Extract the second image features from the detection point image.

[0179] Sub-step 2105: Input the second image feature into a segmenter-type detection model to obtain edge feature points of the second image feature that meet the color threshold.

[0180] Sub-step 2106: The position of the edge feature point is used as the edge box position of the detection point image.

[0181] In this embodiment, the second image feature can be obtained using feature extraction algorithms such as LBP, HOG, and SIFT. The target detection component refers to the component to be detected contained in the detection point image. The first bounding box position range refers to the range within which the bounding box position of the target detection component needs to be located in the detection point image.

[0182] Specifically, edge feature points can be extracted from the second image features using the following formula (1):

[0183] (1)

[0184] Where g(i,j) are the coordinates of the edge feature points, f(i,j) are the coordinates of the second image feature, and T is the coordinate of the first bounding box.

[0185] Segmenter-type detection models can effectively extract the bounding boxes of target components from the detection point image based on a color threshold. The bounding box of the target component is compared to a second bounding box; if it falls within the second bounding box, it is considered a compliant product; otherwise, it is considered a defective product. Of course, if the target component is not present in the detection point image, such as when tape is not pasted, its bounding box will not be detected, and the product to which it belongs is also determined to be defective. Therefore, classifier-type detection models are suitable for product inspection of components with readily visible colors and variable shapes, such as connecting wires and tape.

[0186] For example, refer to Figure 8 This diagram shows the inspection results for various types of adhesive tape and plugs in a certain electronic product. E1 and F2 represent defective adhesive tape, E2 and F1 represent compliant adhesive tape, G1 and H2 represent defective plugs, and G2 and H1 represent compliant plugs. It is evident that the shapes of adhesive tape and plugs may vary, and the classifier-type inspection model can effectively inspect components with non-fixed shapes, such as adhesive tape and plugs.

[0187] This application embodiment improves the accuracy of product detection for parts with irregular shapes by using a segmenter-type detection model to perform product detection on product images.

[0188] Optional, refer to Figure 9 When the detection model is of the image template type, step 210 includes:

[0189] Sub-step 2107: Extract third image features from the detection point image and extract reference image features from the reference component image.

[0190] Sub-step 2108: Perform similarity matching between the third image features and the reference image features to obtain the feature similarity between the detection point image and the reference component image.

[0191] In this embodiment, the third image feature can be obtained using feature extraction algorithms such as LBP, HOG, and SIFT. The reference component image is extracted beforehand from the reference product image, that is, the images of each compliant component in the reference product image. By comparing the similarity between the inspection point image of the product to be inspected and the reference component image of the compliant product, if the similarity is less than a similarity threshold, it indicates that the difference between the reference component image and the inspection point image is large, and it can be determined that the product to be inspected in the inspection point image has a defect.

[0192] Specifically, the similarity between the features of the third image and the features of the reference image can be obtained using the following formula (2):

[0193] (2)

[0195] Where R(x, y) represents the feature similarity, and (x, y) represents the coordinate values ​​of the third image feature. The coordinate values ​​of the reference image features, and The correlation coefficient.

[0196] Furthermore, when multiple inspection point images exist, the number of inspection point images whose similarity to the reference component image is less than a similarity threshold can be greater than or less than a data threshold. In other words, if the number of non-compliant and compliant components falls below a certain threshold, the product under inspection in that inspection point image is confirmed to be defective. It should be noted that different inspection point images correspond to different reference component images because different inspection point images contain different components, and the corresponding reference component images for compliant components are also different. Furthermore, the similarity threshold can be the same for different inspection point images or can be set individually, depending on actual needs; no specific limitation is made here.

[0197] This application embodiment improves the accuracy of product detection by using an image template-type detection model to perform product detection on product images.

[0198] Step 211: If the detection result meets any of the following conditions, determine that the product under inspection in the product image has a defect:

[0199] The detection type of the detection point image in the detection result is a defect type;

[0200] The detection result shows that the edge box position of the detection point image exceeds the first edge box position range or the edge box position of the detection point image does not exist;

[0201] In the detection results, the feature similarity between the detection point image and the reference component image is less than the similarity threshold.

[0202] This step can be referred to in the detailed description of step 105, and will not be repeated here.

[0203] Step 212: Control the product assembly line to stop operating and output alarm information for defective products.

[0204] In this embodiment of the application, when a defect is found in a product to be inspected during the inspection of the product assembly line using this solution, the product assembly line can be stopped by the industrial control computer, and the staff of the product assembly line can be notified by means of ringing, flashing warning lights, voice warnings, and text warning lights on the screen, so that the staff can carry out manual inspection, rework and repair of the defective product.

[0205] In this embodiment, after a defective product is detected, the product assembly line is automatically stopped and an alarm message is output to inform the staff to handle the defective product. This allows the staff to promptly detect defective products without having to constantly monitor the product inspection results, saving the labor costs required for product inspection.

[0206] This application provides another product inspection method. This method extracts product images containing the products to be inspected from production line images captured by the product assembly line. It then extracts inspection point images of the components to be inspected from the product images and inputs them into a detection model for product inspection to identify defective products. This reduces the labor costs required for product inspection in the product assembly line, avoids missed or false detections due to human error, and improves the accuracy of product inspection on the product assembly line. Furthermore, it employs detection models of classifier type, segmenter type, and image template type to inspect different types of components, further improving the accuracy of product inspection. Before processing the acquired production line images, preprocessing methods such as brightness optimization and registration not only improve image quality but also reduce the amount of data processing required subsequently. Finally, it promptly outputs alarm information to notify personnel upon discovering defective products, further reducing the labor costs required for product inspection.

[0207] Example 3

[0208] Reference Figure 10 This application provides a product testing device 30, the device comprising:

[0209] The imaging module 301 is used to acquire images of the product assembly line and obtain images of the production line.

[0210] The first extraction module 302 is used to extract product images including the products to be inspected from the production line images;

[0211] The second extraction module 303 is used to extract the detection point image of the component detection area in the product image;

[0212] Processing module 304 is used to input the detection point image into the detection model to obtain the detection result;

[0213] The determining module 305 is configured to determine that the product under inspection in the product image has a defect if the detection result meets any of the following conditions:

[0214] The detection type of the detection point image in the detection result is a defect type;

[0215] The detection result shows that the edge box position of the detection point image exceeds the first edge box position range or the edge box position of the detection point image does not exist;

[0216] In the detection results, the feature similarity between the detection point image and the reference component image is less than the similarity threshold.

[0217] Optionally, the processing module 304 is further configured to:

[0218] Determine the target type of the detection model corresponding to the component detection area;

[0219] The detection point image is input into the detection model of the target type to obtain the detection result;

[0220] The target type of the detection model includes any one of the following: classifier type, segmenter type, and image template type.

[0221] Optionally, if the detection model is a classifier type, the processing module 304 is further configured to:

[0222] Extract the first image feature from the detection point image;

[0223] The first image feature is input into a detection model of the classifier type to obtain the confidence level of the first image feature for each detection type.

[0224] The detection type with the highest confidence level is the detection type of the detection point image, wherein the detection type of the detection point image includes at least a compliance type or a defect type.

[0225] Optionally, if the detection model is a segmenter type, the processing module 304 is further configured to:

[0226] Extract second image features from the detection point image;

[0227] The second image feature is input into a segmenter-type detection model to obtain edge feature points of the second image feature that meet the color threshold.

[0228] The positions of the edge feature points are used as the bounding box positions of the detection point image.

[0229] Optionally, when the detection model is of the image template type, the processing module 304 is further configured to:

[0230] Extract third image features from the detection point image, and extract reference image features from the reference component image;

[0231] The third image features are matched with the reference image features to obtain the feature similarity between the detection point image and the reference component image.

[0232] Optionally, the first extraction module 30 is further configured to:

[0233] Edge detection is performed on the production line image to obtain the edge box position of the product to be inspected in the production line image;

[0234] The production line image within the range of the second edge frame position of the product to be inspected is used as the product image.

[0235] Optionally, the first extraction module 302 is further configured to:

[0236] Perform color segmentation on the production line image to obtain the image proportion of the target color or target texture in the production line image;

[0237] Production line images with an image proportion greater than the target color or target texture threshold are used as product images.

[0238] Optionally, the device further includes:

[0239] Registration module 306 is used for:

[0240] Extract the first feature points from the product image;

[0241] The first feature point is compared with the second feature point of the reference product image to obtain the matching feature point. The reference product image is a product image that has been pre-annotated with compliance inspection points and meets the requirements of rotation and translation.

[0242] Based on the matched feature points, the coordinate transformation parameters are obtained;

[0243] The product image is registered according to the coordinate transformation parameters.

[0244] Optionally, the device further includes:

[0245] The preprocessing module 307 is also used to optimize the brightness of the production line image using a target method;

[0246] Optionally, the device further includes:

[0247] The alarm module 308 is used to control the product assembly line to stop operating and output alarm prompts for defective products.

[0248] The embodiments of the above-described apparatus are basically similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the description of the method embodiments.

[0249] The embodiments of this application provide a product inspection device, which extracts product images containing products to be inspected from production line images obtained by taking pictures of the product assembly line, and extracts inspection point images of the components to be inspected from the product images and inputs them into the inspection model to identify defective products to be inspected. This reduces the labor cost required for product inspection in the product assembly line, avoids missed inspections and false inspections due to human error, and improves the accuracy of product inspection in the product assembly line.

[0250] Example 4

[0251] This application provides a product inspection system, which includes a product assembly line, a camera facing the product assembly line, and an industrial control computer. The industrial control computer is used to execute the product inspection method described in either embodiment one or embodiment two.

[0252] Reference Figure 11 This diagram illustrates the structure of a product inspection system according to an embodiment of this application. An industrial control computer 1 controls a camera 3 to capture images of the product to be inspected 5 on the product assembly line 2. The industrial control computer 1 then executes the product inspection method described in either Embodiment 1 or Embodiment 2 to perform product inspection based on the captured production line images. Furthermore, a light source 4 can be installed above the product assembly line 2 to illuminate it, thereby ensuring the quality of the production line images captured by the camera 3.

[0253] This application provides a product inspection system that extracts product images containing the products to be inspected from production line images captured by the product assembly line, and extracts inspection point images of the components to be inspected from the product images and inputs them into the inspection model to identify defective products. This reduces the labor costs required for product inspection in the product assembly line, avoids missed and false inspections due to human error, and improves the accuracy of product inspection in the product assembly line.

[0254] Example 5

[0255] This application provides a computer storage medium storing a computer program, which, when executed by a processor, can implement the above-described product testing method.

[0256] The embodiments of this application provide a computer storage medium. By executing the computer program on the computer storage medium, the above-mentioned product testing method can be implemented, thereby reducing the labor cost required for product inspection in the product assembly line, avoiding missed and false inspections due to human error, and improving the accuracy of product testing in the product assembly line.

[0257] Those skilled in the art will understand that this application includes devices for performing one or more of the operations described in this application. These devices may be specifically designed and manufactured for the desired purpose, or may include known devices found in general-purpose computers. These devices have computer programs stored therein that can be selectively activated or reconfigured. Such computer programs may be stored in a storage medium of the device (e.g., a computer) or in any type of medium suitable for storing electronic instructions and coupled to a bus, including but not limited to any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. That is, the storage medium includes any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0258] Those skilled in the art will understand that each block in these structural diagrams and / or block diagrams and / or flowcharts, as well as combinations of blocks in these structural diagrams and / or block diagrams and / or flowcharts, can be implemented using computer program instructions. Those skilled in the art will understand that these computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing method for implementation, thereby executing the schemes specified in the blocks or plurality of blocks of the structural diagrams and / or block diagrams and / or flowcharts disclosed in this application by the processor of the computer or other programmable data processing method.

[0259] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A product testing method, characterized in that, The method includes: Images are captured from the product assembly line to obtain production line images; the production line images are obtained by taking pictures of the conveyor belt of the product assembly line using cameras installed on the product assembly line. Extracting product images including the product to be inspected from the production line images includes: Edge detection is performed on the production line image to obtain the edge box position of the product to be inspected in the production line image; The production line image within the range of the second edge frame position of the product to be inspected is used as the product image. Extract the first feature points from the product image; The first feature point is compared with the second feature point of the reference product image to obtain the matching feature point. The reference product image is a product image that has been pre-annotated with compliance inspection points and meets the requirements of rotation and translation. Based on the matched feature points, the coordinate transformation parameters are obtained; The product image is registered according to the coordinate transformation parameters. Extract the detection point image of the component detection area in the product image; The detection point image is input into the detection model to obtain the detection result; the detection result is that the number of non-compliant parts among multiple parts to be detected is lower than the number threshold. If the test results meet any of the following conditions, it is determined that the product under inspection in the product image has a defect: The detection type of the detection point image in the detection result is a defect type; The detection result shows that the edge box position of the detection point image exceeds the first edge box position range or the edge box position of the detection point image does not exist; In the detection results, the feature similarity between the detection point image and the reference component image is less than the similarity threshold.

2. The method according to claim 1, characterized in that, The step of inputting the detection point image into the detection model to obtain the detection result includes: Determine the target type of the detection model corresponding to the component detection area; The detection point image is input into the detection model of the target type to obtain the detection result; The target type of the detection model includes any one of the following: classifier type, segmenter type, and image template type.

3. The method according to claim 2, characterized in that, When the detection model is of classifier type, the step of inputting the detection point image into the target type detection model to obtain the detection result includes: Extract the first image feature from the detection point image; The first image feature is input into a detection model of the classifier type to obtain the confidence level of the first image feature for each detection type. The detection type with the highest confidence level is the detection type of the detection point image, wherein the detection type of the detection point image includes at least a compliance type or a defect type.

4. The method according to claim 2, characterized in that, When the detection model is of the segmenter type, the step of inputting the detection point image into the target type detection model to obtain the detection result includes: Extract second image features from the detection point image; The second image feature is input into a segmenter-type detection model to obtain edge feature points of the second image feature that meet the color threshold. The positions of the edge feature points are used as the bounding box positions of the detection point image.

5. The method according to claim 2, characterized in that, When the detection model is of the image template type, the step of inputting the detection point image into the target type detection model to obtain the detection result includes: Extract third image features from the detection point image, and extract reference image features from the reference component image; The third image features are matched with the reference image features to obtain the feature similarity between the detection point image and the reference component image.

6. The method according to claim 1, characterized in that, The step of extracting the product image including the product to be inspected from the production line image also includes: Perform color segmentation on the production line image to obtain the image proportion of the target color or target texture in the production line image; Production line images with an image proportion greater than the target color or target texture threshold are used as product images.

7. The method according to claim 1, characterized in that, Before extracting the product image that includes the product to be inspected from the production line image, the process further includes: The brightness of the production line image is optimized using a targeted approach.

8. The method according to claim 1, characterized in that, After determining that the product to be inspected in the product image has a defect, the method further includes: The system controls the product assembly line to stop operating and outputs alarm messages for defective products.

9. A product testing device, characterized in that, include: One or more processors; A memory for storing one or more programs that, when executed by one or more processors, enable the one or more processors to implement the product testing method according to any one of claims 1-8.

10. A product testing system, characterized in that, The system includes: a product assembly line, a camera positioned toward the product assembly line, and an industrial control computer, the industrial control computer being used to execute the product inspection method according to any one of claims 1 to 8.

11. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it can implement the product testing method according to any one of claims 1-8.

Images