Screen hole area defect detection method, system, device and storage medium

Abstract

The application relates to the technical field of screen detection, and provides a screen hole area defect detection method, a system, a device and a storage medium. The application provides a partition detection method for screen hole area defects. In the method, the positions of the defects existing in a screen and the positions of at least one area corresponding to a screen hole area are determined, the specific area of the screen hole area into which the defects existing in the screen fall is determined, the types of the defects falling into each area are compared with the types of preset defective defects corresponding to each area, and then it is determined whether the defects falling into each area are defective defects, so that the accuracy of screen hole area defect detection is improved.

Description

Technical Field

[0001] This application relates to the field of screen inspection technology, and in particular to a method, system, device and storage medium for detecting defects in the hole area of ​​a screen. Background Technology

[0002] With the widespread use of electronic devices, people's demands for the user experience are also increasing. In pursuit of a better user experience, a hole is often punched in the screen of electronic devices to place the camera below the hole. Because the hole is small and the screen-to-body ratio is high, this does not affect the user experience.

[0003] However, during the manufacturing process of electronic devices, defects may occur in the screen hole area due to factors such as process technology or environment, leading to screen malfunctions. Therefore, defect detection in the screen hole area is essential. Detecting these defects before shipment can improve the yield rate of electronic device screens, thereby bringing direct economic benefits. However, the accuracy of current screen hole area defect detection methods is not high. Summary of the Invention

[0004] This application provides a method, system, apparatus, and storage medium for detecting defects in screen hole areas, which can improve the accuracy of screen hole area defect detection.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, a method for detecting defects in a screen hole area is provided. The screen includes a hole area and a support plate covering area surrounding the hole area. The hole area includes a device light-transmitting area, an ink-covered area surrounding the device light-transmitting area, and a screen wiring area surrounding the ink-covered area. The method for detecting defects in a screen hole area includes: firstly, acquiring a test image of the screen, which includes the device light-transmitting area, the ink-covered area, the screen wiring area, and part of the support plate covering area; secondly, determining the location and type of defects existing in the screen, and the location of at least one area among the device light-transmitting area, the ink-covered area, the screen wiring area, and the support plate covering area, based on the test image; and finally, determining whether the defects existing in the screen are defects based on the location and type of defects existing in the screen, the location of at least one area, and the correspondence between at least one area and preset defects.

[0007] This application provides a method for detecting defects in the screen hole area. In this method, the specific area in which the defect falls into the screen hole area can be determined based on the location of the defect and the location of at least one area corresponding to the screen hole area. The type of defect falling into each area is compared with the type of preset defect corresponding to each area to determine whether the defect falling into each area is a defect, thereby improving the accuracy of screen hole area defect detection.

[0008] In one possible implementation of the first aspect, the at least one region includes: the device light-transmitting region and the screen trace region; determining whether a defect at the defect location is a defective defect based on the location and type of the defect existing in the screen, the location of the at least one region, and the correspondence between the at least one region and a preset defect includes: determining the defect as a defective defect when the location of the defect on the screen falls within the device light-transmitting region and the type of the defect is the same as the type of a preset defective defect corresponding to the device light-transmitting region; and determining the defect as a defective defect when the location of the defect on the screen falls within the screen trace region and the type of the defect is the same as the type of a preset defective defect corresponding to the screen trace region.

[0009] Secondly, a screen hole area defect detection system is provided. The screen includes a hole area and a support plate covering area surrounding the hole area. The hole area includes a device light-transmitting area, an ink-covered area surrounding the device light-transmitting area, and a screen wiring area surrounding the ink-covered area. The system includes an image acquisition device and a processing device connected to the image acquisition device. The image acquisition device is used to acquire a test image of the screen, the test image including the device light-transmitting area, the ink-covered area, the screen wiring area, and a portion of the support plate covering area. The processing device is used to determine the location and type of defects existing in the screen, and the location of at least one area among the device light-transmitting area, the ink-covered area, the screen wiring area, and the support plate covering area, based on the test image. It is also used to determine whether the defects existing in the screen are defective based on the location and type of the defects, the location of the at least one area, and the correspondence between the at least one area and preset defects.

[0010] Thirdly, a screen hole area defect detection device is provided, the device including a memory and a processor, the memory being used to store instructions, which, when executed by the processor, cause the screen hole area defect detection device to perform the method of the first aspect or any possible implementation thereof.

[0011] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed, implement the method of the first aspect or any possible implementation of the first aspect.

[0012] Fifthly, a computer program product is provided, the computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method of the first aspect or any possible implementation thereof.

[0013] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0014] Figure 1 This is an exemplary flowchart of the screen defect detection method provided in the first aspect of this application.

[0015] Figure 2 This is a schematic diagram of the back of the screen provided in an embodiment of this application;

[0016] Figure 3 This is a schematic diagram of the image to be tested provided in an embodiment of this application;

[0017] Figure 4 This is a schematic diagram of the feature extraction layer, feature fusion layer, and output layer of the YOLOv5 network model provided in the embodiments of this application;

[0018] Figure 5 This is a schematic diagram of an output result provided by an embodiment of this application after inputting the image to be tested into a defect classification model;

[0019] Figure 6 This is a schematic diagram of the structure of the screen defect detection system provided in the second aspect of this application embodiment;

[0020] Figure 7 The embodiments provided in this application Figure 6 A partial structural diagram;

[0021] Figure 8 This is provided by the embodiments of this application. Figure 7 Exploded view of the structure shown;

[0022] Figure 9 This is an exploded view of the fixture provided in the embodiments of this application;

[0023] Figure 10 This is a schematic diagram of the program interface of the screen defect detection system provided in the embodiments of this application;

[0024] Figure 11 This is a schematic diagram of the structure of the screen defect detection device provided in the third aspect of this application. Detailed Implementation

[0025] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0026] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0027] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0028] With the widespread use of electronic devices, people's demands for the user experience are also increasing. In pursuit of a better user experience, a hole is often punched in the screen of electronic devices to place the camera below the hole. Because the hole is small and the screen-to-body ratio is high, this does not affect the user experience.

[0029] However, during the manufacturing process of electronic devices, defects may occur in the screen's perforation area due to factors such as manufacturing processes or the environment, leading to screen malfunctions. For example, cracks may appear on the sidewalls of the perforation area during the drilling process, allowing moisture to easily enter the display layer during use, causing display device failure and resulting in black spots on the screen, causing display problems. Similarly, dirt or lint may exist on the screen cover during production. If this dirt or lint is located in the light-transmitting area of ​​the screen components, it will affect the camera's shooting performance.

[0030] Therefore, defect detection in the screen hole area is essential. Detecting defects in the screen hole area before shipment can improve the yield rate of electronic device screens, thus bringing direct economic benefits. However, the accuracy of current screen hole area defect detection is not high.

[0031] In view of this, embodiments of this application provide a method for detecting defects in a screen hole area. The screen includes a hole area and a support plate covering area surrounding the hole area. The hole area includes a device light-transmitting area, an ink-covered area surrounding the device light-transmitting area, and a screen trace area surrounding the ink-covered area. The method for detecting defects in a screen hole area includes: first, acquiring a test image of the screen, which includes a device light-transmitting area, an ink-covered area, a screen trace area, and a portion of the support plate covering area; second, determining the location and type of defects existing in the screen, and the location of at least one area among the device light-transmitting area, the ink-covered area, the screen trace area, and the support plate covering area, based on the test image; and finally, determining whether the defects existing in the screen are defects based on the location and type of defects existing in the screen, the location of at least one area, and the correspondence between at least one area and a preset defect.

[0032] This application provides a method for detecting defects in the screen hole area. In this method, the specific area in which the defect falls into the screen hole area can be determined based on the location of the defect and the location of at least one area corresponding to the screen hole area. The type of defect falling into each area is compared with the type of preset defect corresponding to each area to determine whether the defect falling into each area is a defect, thereby improving the accuracy of screen hole area defect detection.

[0033] The following is a detailed description of a screen hole area defect detection method provided by an embodiment of this application, with reference to the accompanying drawings. Figure 1 This is an exemplary flowchart of a method for detecting defects in the hole area of ​​a screen.

[0034] Step 101: Obtain the image to be tested in the screen hole area. The image to be tested includes the device light-transmitting area, ink-covered area, screen wiring area, and part of the support plate covered area in the screen hole area.

[0035] An electronic device screen typically comprises, from top to bottom, a protective cover, a display layer, and a support plate. The protective cover, the outermost component, is usually made of tempered glass or plastic and protects the internal display layer from scratches, impacts, and dust. The display layer, the core of the screen, is responsible for generating and displaying images; it can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other display technologies. The support plate provides a robust backing, enhancing the screen's overall strength and stability.

[0036] In some cases, the screen of an electronic device may also include a touch layer, which (also known as a touchscreen sensor layer) is typically located above the display layer because it requires direct contact with the user to detect touches and gestures. In some cases, the touch layer may also be integrated into the display layer, forming an embedded touch technology.

[0037] It is worth noting that the screen of an electronic device may also include other film layer structures, which will not be listed in this embodiment.

[0038] To increase the screen-to-body ratio, holes are currently made in the protective cover layer beneath the screen (such as the display layer and support plate) to place the camera below these holes. Simultaneously, to avoid light interference and improve the overall aesthetics of the screen, ink is applied to the side of the protective cover layer closest to the display layer, directly opposite the holes, forming an ink-covered area. This ink-covered area is ring-shaped, and the light-transmitting area within the ring is called the device light-transmitting area. External light enters the camera beneath the screen through this area.

[0039] To improve the alignment accuracy between the light-transmitting area of ​​the device and the under-display camera and reduce light leakage, the diameter of the through-holes formed in the film layer (such as the display layer and the support plate) under the cover plate is often larger than the diameter of the light-transmitting area of ​​the device, and the diameter of the through-holes located on the support plate is larger than the diameter of the through-holes located on the display layer. Therefore, the screen schematic diagram seen from the support plate side is as follows. Figure 2 As shown, the screen 100 includes a hole area 11 and a support plate covering area 12 surrounding the hole area 11. See also Figure 3 The aperture area 11 includes a device light-transmitting area 111, an ink-covered area 112 surrounding the device light-transmitting area 111, a screen wiring area 113 surrounding the ink-covered area 112, and a support plate covering area 12 surrounding the ink-covered area 112. The device light-transmitting area 111 and the ink-covered area 112 are portions of the protective cover plate as seen from the support plate side, the screen wiring area 113 is a portion of the display layer as seen from the support plate side, and the support plate covering area 12 is a portion of the support plate as seen from the support plate side.

[0040] The light-transmitting area 111 of the device is circular, and the ink-covered area 112 is annular, with the inner diameter of the ink-covered area 112 being the same as the diameter of the light-transmitting area 111. The screen trace area 113 is annular, with the inner diameter of the screen trace area 113 being the same as the outer diameter of the ink-covered area 112.

[0041] In this embodiment, defect detection is performed on the screen hole area. First, an image of the screen hole area to be tested needs to be acquired, such as... Figure 3 As shown, the image to be tested needs to include the screen's hole area 11, namely the screen's device light-transmitting area 111, ink-covered area 112, and screen wiring area 113, as well as part of the support plate covered area 12.

[0042] It is worth noting that, in this embodiment, when performing defect detection on the screen hole area, the image to be tested is a partial image of the back of the screen taken from the support plate side of the screen, rather than an image of the front of the screen taken from the protective cover side.

[0043] Step 102: Determine the location and type of defects in the screen based on the image to be tested, as well as the location of at least one area in the device light-transmitting area, ink-covered area, screen trace area, and support plate covered area.

[0044] First, we will explain how to determine the location and type of defects on the screen based on the image to be tested.

[0045] For example, the image to be tested can be input into a pre-trained defect classification model to obtain the location and type of defects present on the screen in the image to be tested.

[0046] The pre-trained defect classification model is obtained through the following steps: collecting sample images, including defect-free and defective sample images, where each defective sample image includes a defect bounding box and the defect type corresponding to the defect bounding box; training the neural network model using a subset of the sample images until the model converges; evaluating the performance of the converged neural network model using another subset of untrained sample images; and optimizing the converged neural network model based on the performance evaluation results until its performance meets a performance threshold.

[0047] Specifically, sample images of the screen's perforated area can be captured using a camera, including defect-free and defective sample images. In this embodiment, defects include cracks, dirt, and fuzz. The sample images are labeled according to four categories: 1. Cracks, 2. Dirt, 3. Fuzz, and 4. No Defects, to obtain a sample defect dataset. When labeling defects of categories 1, 2, and 3, the location of the defect needs to be outlined in the sample image to form a sample defect bounding box. Each sample defect bounding box corresponds to a sample defect type. For example, if the defect in the sample defect bounding box is a fuzz, then the corresponding sample defect type is 3.

[0048] The sample defect dataset was divided into a training set, a validation set, and a test set. Sample images from the training set were input into the neural network model to be trained until the model converged. Then, the performance of the converged neural network model was evaluated using the validation set, and based on the evaluation results, the model was optimized until its performance met a performance threshold. Finally, the trained neural network model was tested using the test set to assess its performance on unknown data.

[0049] In this embodiment, the neural network model to be trained is the YOLOv5 network model. The YOLOv5 network model includes an input layer, a feature extraction layer, a feature fusion layer, and an output layer. See [link to documentation]. Figure 4 The feature extraction layer is also known as the Backbone network layer, the feature fusion layer is also known as the Neck network, and the output layer is also known as the Head network.

[0050] The above-mentioned method of training a neural network model using a subset of sample images until the model converges includes: inputting a subset of sample images into the input layer for preprocessing; inputting the preprocessed sample images into the feature extraction layer to extract features from each sample image, resulting in multiple feature maps of different resolutions for each sample image; inputting the multiple feature maps of different resolutions for each sample image into the feature fusion layer for feature fusion, resulting in multiple fused feature maps of different resolutions for each sample image; inputting the multiple fused feature maps of different resolutions for each sample image into the output layer to obtain the predicted bounding box position, the predicted defect type at the predicted bounding box position, and the confidence level for each sample image; determining whether the neural network model has converged based on the predicted bounding box position, the predicted defect type at the predicted bounding box position, the position of the defect box in the sample image, and the defect type of the defect box; if the neural network model has not converged, returning to the step of inputting a subset of sample images into the input layer for preprocessing until the neural network model converges.

[0051] First, the sample images of the training set are input into the input layer of the model, and then the sample images are preprocessed in the input layer. The preprocessing includes: (1) Data augmentation, which can be achieved by stitching together multiple sample images through at least one of random scaling, random cropping, and random arrangement to obtain a stitched image. For example, after stitching together multiple sample images through at least one of random scaling, random cropping, and random arrangement, 8 or even 10 stitched images can be obtained, thereby enriching the sample images of the training set. (2) Adaptive anchor boxes, which can be adaptively drawn based on the width and height of the sample defect boxes in the stitched image. Specifically, first, the width and height information of all sample defect boxes in the stitched image is collected, and then a clustering algorithm is used to cluster these widths and heights to obtain the initial outlines. The purpose of adaptive anchor boxes is to predict the bounding boxes of defects in the sample images. If the size of the anchor box is close to the actual size of the defect, the model can more easily learn how to accurately predict the bounding boxes. (3) Adaptive image scaling: The resolution of the image after the bounding box is scaled to a preset resolution, which is the same as the resolution of the image that can be input into the feature extraction layer. For example, assuming the preset resolution is 640*640 and the resolution of the sample image is 1270*960, the resolution of the sample image needs to be scaled from 1270*960 to 640*640 before the scaled image is input into the feature extraction layer.

[0052] The feature extraction layer (i.e., the backbone network layer) extracts features from the input image, obtaining multiple feature maps of different resolutions for each image. For example, ... Figure 4 As shown, for each input image, feature extraction yields three feature maps with different resolutions. These feature maps capture features at different scales, which helps detect defects of varying sizes. For example, feature maps with resolutions of 256*80*80, 512*40*40, and 1024*20*20 are shown. 256*80*80 represents a feature map with a resolution of 80*80 pixels, containing 80*80 pixels, each with 256 feature information. 512*40*40 represents a feature map with a resolution of 40*40 pixels, containing 40*40 pixels, each with 512 feature information. 1024*20*20 represents a feature map with a resolution of 20*20 pixels, containing 20*20 pixels, each with 1024 feature information. The feature information extracted by the feature extraction layer can include edges, corners, textures, color distribution, shape, and other information.

[0053] For each sample image, multiple feature maps at different resolutions are input into a feature fusion layer (i.e., the neck network) for feature fusion, resulting in multiple fused feature maps at different resolutions for each sample image. For example, feature maps at different resolutions can be fused first, and then the fused feature map can be fused with feature maps of other resolutions. For example, ... Figure 4 As shown, after fusing the feature maps with resolutions of 256*80*80, 512*40*40, and 1024*20*20, the resulting feature map also has resolutions of 256*80*80, 512*40*40, and 1024*20*20.

[0054] Multiple fused feature maps at different resolutions for each sample image are input into the output layer (i.e., the Head network) to obtain the predicted bounding box location, the predicted defect type at the corresponding bounding box location, and the confidence level for each sample image. Specifically, such as... Figure 4 As shown, each fused feature map is passed through a set of convolutional layers to generate a series of prediction vectors. Each prediction vector contains the location (x, y, w, h), class, and confidence score of the target bounding box. The location of the prediction vector on the feature map corresponds to its spatial location in the original image. For example, a point on the fused feature map might correspond to a specific region in the original image. Since each defect in the image may be detected at multiple locations, forming multiple prediction boxes, non-maximum suppression (NMS) is applied to filter out the most likely prediction boxes. NMS retains the prediction box with the highest confidence score and suppresses other prediction boxes that have high overlap with it. Finally, the prediction boxes processed by NMS are output, each accompanied by a class label and confidence score.

[0055] Next, based on the predicted bounding box position and the predicted defect type at the corresponding predicted bounding box position for each sample image, as well as the position of the sample defect box in the sample image and the sample defect type at the corresponding sample defect box, it is determined whether the neural network model has converged. If the neural network model has not converged, the process returns to the step of inputting a portion of the sample images into the input layer for preprocessing, until the neural network model converges.

[0056] Specifically, the convergence of a neural network model can be determined by calculating the loss function.

[0057] Loss function G loss The following formula (1) is used to calculate:

[0058]

[0059] Where A represents the position of the sample defect box, B represents the position of the predicted box, and C represents the position of the minimum bounding rectangle of the sample defect box and the predicted box, which includes the area of ​​the sample defect box and the predicted box. A, B and C above all refer to the same defect.

[0060] During the input of sample images, repeat the above steps and calculate the loss function. If the loss function value tends to stabilize and no longer decreases significantly as sample images are input, then the neural network model can be determined to have converged.

[0061] Next, the performance of the converged neural network model is evaluated using a validation set. Based on the evaluation results, the converged neural network model is optimized until its performance meets a performance threshold. For example, sample images from the validation set can be input into the trained model, and performance parameters such as mean precision, accuracy, or recall can be calculated to see if they meet the performance threshold. For instance, mean precision should be above 90%. If the performance parameters calculated based on the output do not meet the performance threshold, the weight parameters of the neural network model are optimized, and sample images are re-inputted for validation until the neural network model's performance meets the performance threshold. At this point, model training is complete.

[0062] Finally, the images in the prediction set can be input into the trained model to obtain the location, category, and confidence level of the predicted bounding box in each image.

[0063] The following example illustrates the model training process.

[0064] Suppose we use the YOLOv5 model to detect cracks in the hole area of ​​a mobile phone screen.

[0065] (1) First, prepare the data.

[0066] Data collection: 1,000 images of the hole area of ​​the mobile phone screen were taken using the camera, which included cracks, dirt and fuzz of various sizes.

[0067] Data annotation: Annotation tools were used to mark cracks, dirt, and fuzz in the images. Images without defects were also marked.

[0068] Dataset split: 700 images as the training set, 200 images as the validation set, and 100 images as the test set.

[0069] (2) Model Training

[0070] Initialization: Load the pre-trained weights of YOLOv5 to provide a good starting point for model training.

[0071] Iterative Training: First Training Cycle: The model makes random predictions, resulting in a high loss value; for example, the training set loss is 10.0, and the validation set loss is 9.5. Tenth Training Cycle: The model begins to learn some features, and the loss value decreases; the training set loss drops to 3.0, and the validation set loss drops to 3.5. Fifty Training Cycle: The model becomes more accurate; the training set loss drops to 0.8, and the validation set loss drops to 1.0.

[0072] Feature extraction and fusion: In each training cycle, the model extracts features through the backbone network and fuses them through the neck network to generate a fused feature map for prediction.

[0073] Prediction Output and Loss Calculation: The Head network outputs predicted bounding boxes and calculates the difference between the predicted boxes and the ground truth boxes using a loss function, continuously adjusting the weights to reduce the loss.

[0074] (3) Verification process

[0075] Validation set evaluation: After each training epoch, the model's performance is evaluated using the validation set. For example, at the 50th training epoch: the average accuracy on the validation set is calculated to be 90%.

[0076] (4) Testing process

[0077] Using the optimal weights obtained from the 50th training epoch, defect detection was performed on 100 test set images. The model's output was recorded, including the location, category, and confidence score of the predicted bounding boxes. Results included: Crack detection: The model successfully identified cracks in 95 images with 95% accuracy. Dirt detection: The model identified dirt in 90 images with 90% accuracy. Hair detection: The model identified hair in 85 images with 85% accuracy. Defect-free images: The model correctly identified all defect-free images.

[0078] Through this process, not only was a YOLOv5 model capable of detecting defects in the screen hole area trained, but the generalization ability and accuracy of the model were also ensured through the verification and testing process.

[0079] Secondly, it explains how to determine the location of at least one area among the device's light-transmitting area, ink-covered area, screen trace area, and support plate-covered area based on the image to be tested.

[0080] For example, at least one region includes: a device light-transmitting region and a screen trace region; determining the position of at least one region among the device light-transmitting region, the ink-covered region, the screen trace region, and the support plate covered region based on the image to be tested includes: determining the second radius of the device light-transmitting region, the second radius of the screen trace region, and the second radius of the ink-covered region based on a first radius of the device light-transmitting region, a first radius of the ink-covered region, a first radius of the screen trace region, and preset camera parameters, wherein the first radius is the radius of each region in the screen, and the second radius is the radius of each region in the image to be tested; determining the position of the device light-transmitting region in the image to be tested based on the second radius of the device light-transmitting region; and determining the position of the screen trace region in the image to be tested based on the second radius of the screen trace region and the second radius of the ink-covered region.

[0081] Specifically, see Figure 3 The preset camera parameters include the actual physical size represented by each pixel. Dividing the radius of the ink-covered area 112 on the screen by the actual physical size represented by each pixel yields the number of pixels corresponding to the radius of the ink-covered area 112 in the image under test, thus obtaining the radius of the ink-covered area 112 in the image under test. For example, if the screen area captured by the camera is 12.7mm*12.7mm in size, with a resolution of 1270*1270 and a magnification of 2x, then the actual physical size represented by each pixel is 0.005mm. If the radius of the light-transmitting area 111 of the device is 2.72mm, then the number of pixels occupied by the radius of the light-transmitting area 111 is 544.

[0082] Thus, edge coordinates are detected in the image under test using an edge grayscale algorithm. Then, combining the second radius of the device's light-transmitting region 111, the Hough circle detection origin algorithm is used to find the center (X, Y) and the position of the circle. The detected circle is marked in the image under test; this circle represents the edge position of the device's light-transmitting region 111. Therefore, the position of the device's light-transmitting region 111 in the image under test can be determined based on its edge position. Similarly, based on the second radius of the screen trace region 113, the edge position of the screen trace region 113 can be found in the image under test. Thus, based on the edge positions of the device's light-transmitting region 111 and the screen trace region 113, the position of the screen trace region 113 in the image under test can be determined.

[0083] It is worth noting that the above example uses the determination of the device light-transmitting area 111 and the screen trace area 113 as examples. The methods for determining other areas can be referred to the above example, and will not be described one by one in this embodiment.

[0084] Step 103: Based on the location and type of defects on the screen, the location of at least one area, and the correspondence between at least one area and preset defects, determine whether the defect at the defect location is a defective defect.

[0085] In actual testing, the requirements for defects vary in different areas. See also... Figure 3 For example, in the light-transmitting area 111, any defect such as cracks, fuzz, or dirt will affect the screen's light transmission and consequently the camera's shooting performance below the screen. Therefore, cracks, fuzz, or dirt falling into the light-transmitting area 111 can be considered defects. In the ink-covered area 112, cracks may cause light to enter the area below the screen through the cracks, interfering with camera shooting. Fuzz or dirt, however, will not affect camera shooting; therefore, cracks falling into the ink-covered area 112 can be considered defects. In the screen trace area 113, cracks are believed to be caused by the breakage of inorganic material / metal traces inside the screen hole area, affecting the display of the screen 100. Fuzz or dirt, however, will not affect the screen's display. Therefore, cracks falling into the screen trace area 113 can be considered defects. For the support plate covered area 12, since the support plate only serves a supporting function, defects such as cracks, fuzz, or dirt in the support plate covered area 12 will not affect the function of the support plate and are not considered defects.

[0086] Therefore, this application provides a method for detecting defects in the screen hole area. In this method, the specific area in which the defect falls into the screen hole area can be determined based on the location of the defect and the location of at least one area corresponding to the screen hole area. The type of defect falling into each area is compared with the type of preset defect corresponding to each area to determine whether the defect falling into each area is a defect, thereby improving the accuracy of screen hole area defect detection.

[0087] For example, at least one region includes: a device light-transmitting region and a screen trace region; determining whether a defect at a defect location is a defective defect based on the location and type of a defect existing on the screen, the location of at least one region, and the correspondence between at least one region and a preset defect includes: determining the defect as a defective defect if the location of the defect on the screen falls within the device light-transmitting region and the type of the defect is the same as the type of a preset defective defect corresponding to the device light-transmitting region; and determining the defect as a defective defect if the location of the defect on the screen falls within the screen trace region and the type of the defect is the same as the type of a preset defective defect corresponding to the screen trace region.

[0088] Assuming the image to be tested is input into a pre-trained defect classification model, the result is as follows: Figure 5 The results show that there are defects of type 1 and type 3 on the screen. Type 1 defects are cracks, which appear in the screen trace area 113; type 3 defects are fuzz, which appear in the device light-transmitting area 111. If the preset defect corresponding to the screen trace area 113 is a crack, then the defect of type 1 can be determined as a defect. If the preset defect corresponding to the device light-transmitting area 111 is a crack, fuzz, or dirt, since the fuzz appears in the device light-transmitting area 111, the defect of type 3 can be determined as a defect.

[0089] The above examples are for illustrative purposes only. The specific preset defects corresponding to each area can be set as needed, and are not listed one by one in this embodiment. Furthermore, for the ink-covered area 112 and the support plate-covered area 12, the same method can be used to determine whether the defects falling into their respective areas are defects.

[0090] This application also provides a screen hole area defect detection system, hereinafter referred to as the detection system, such as... Figure 6 As shown, this is used to implement the screen hole area defect detection method in the embodiments of this application. See also Figure 2 and Figure 3 The screen 100 being tested includes a hole area 11 and a support plate covering area 12 surrounding the hole area 11. The hole area 11 includes a device light-transmitting area 111, an ink covering area 112 surrounding the device light-transmitting area 111, and a screen wiring area 113 surrounding the ink covering area 112.

[0091] like Figure 6 and Figure 7 As shown, the detection system includes an image acquisition device 20 and a processing device 21 connected to the image acquisition device 20.

[0092] Image acquisition device 20 is used to acquire the image to be tested on the screen. (See image to be tested). Figure 3 The image under test includes the device light-transmitting area 111, the ink-covered area 112, the screen wiring area 113, and part of the support plate covered area 12.

[0093] The processing device 21 is used to determine the location and type of defects existing in the screen, and the location of at least one of the device light-transmitting area 111, ink-covered area 112, screen wiring area 113 and support plate covered area 12, based on the image to be tested; it is also used to determine whether the defects existing in the screen are defects based on the location and type of defects existing in the screen, the location of at least one area, and the correspondence between at least one area and preset defects.

[0094] In this embodiment, the processing device 21 can determine which area of ​​the screen hole area the defect falls into based on the location of the defect and the location of at least one area. Then, it can perform partitioned detection for each area and compare the type of defect falling into each area with the preset defect type corresponding to each area to determine whether the defect falling into each area is a defect, thereby improving the accuracy of screen hole area defect detection.

[0095] In one example, the system further includes a displacement device 22, on which the image acquisition device 20 is mounted; the displacement device 22 can move in both the horizontal and vertical directions, and thus cause the image acquisition device 20 to move.

[0096] For example, see Figure 7 and Figure 8 The displacement device 22 includes a horizontal motion axis 222 that can move in the horizontal direction, and a vertical motion axis 221 that is mounted on the horizontal motion axis 222 and can move in the vertical direction. The image acquisition device 20 is mounted on the vertical motion axis 221. The system also includes a support plate 23, on which the displacement device 22 is mounted.

[0097] In one example, see Figure 7 and Figure 8 The system also includes: a fixed frame 24 mounted on the displacement device 22, and an image acquisition device 20 mounted on the fixed frame 24.

[0098] In one example, see Figure 7 and Figure 8 The image acquisition device 20 includes a camera 201 and a lens 202 disposed on the light-incident surface of the camera 201. The camera 201 is used for image acquisition; an electrically coupled camera can be used, or other cameras can be employed as needed, with a resolution of 20 megapixels or higher. The lens 202 collects reflected light from the object being illuminated and focuses it onto the camera 201. Suitable lens parameters for the lens 202 include: magnification of 2, working distance of 65 mm, depth of field of 0.35 mm, resolution of 4.5 micrometers, telecentricity of 0.11°, and optical distortion of 0.041.

[0099] When performing screen hole area detection, the screen is placed below the image acquisition device 20. The horizontal position of the image acquisition device 20 is changed by adjusting the horizontal motion axis 222, and the vertical height of the image acquisition device 20 is changed by adjusting the vertical motion axis 221.

[0100] In one example, see Figure 7 and Figure 8The detection system also includes a light source 25, which can be mounted on a vertical motion axis 221 and moved by the vertical motion axis 221. The light source 25 is located below the image acquisition device 20 and provides light to the screen under test. In this embodiment, a ring-shaped white light source can be used, and its brightness and position can be adjusted according to requirements.

[0101] In one example, see Figure 6 The detection system also includes a movable stage 27, which provides support for the screen to be tested and can be moved horizontally according to commands to ensure that the position of the sample to be tested meets the testing requirements.

[0102] In one example, see Figure 6 The detection system also includes a housing 28. A processing device 21 is disposed on the housing 28. The processing device 21 can be a computer or the like, and may include a display screen for displaying the test environment and test parameters. The image acquisition device 20 and the displacement device 22 can be placed inside the housing 28.

[0103] It is worth noting that, see Figure 9 To facilitate the inspection of screen 100, a clamp 26 can be used to hold screen 100 for monitoring. Specifically, clamp 26 includes an upper clamp 261 and a lower clamp 262. The upper clamp 261 and the lower clamp 262 clamp screen 100 in the middle. The middle area of ​​the upper clamp 261 and the lower clamp 262 is hollow, and the hole area of ​​screen 100 can be observed through the hollow area, which facilitates the image acquisition device 20 to take pictures of the hole area of ​​screen.

[0104] In one example, the processing device 21 is used to determine that a defect is a defective defect if the location of a defect on the screen falls into the light-transmitting area of ​​the device and the type of the defect is the same as the type of a preset defective defect corresponding to the light-transmitting area of ​​the device; and to determine that a defect is a defective defect if the location of a defect on the screen falls into the screen trace area and the type of the defect is the same as the type of a preset defective defect corresponding to the screen trace area.

[0105] In one example, the processing device 21 is used to determine the second radius of the device's light-transmitting area, the second radius of the screen's trace area, and the second radius of the ink-covered area based on the first radius of the device's light-transmitting area, the first radius of the ink-covered area, the first radius of the screen trace area, and preset camera parameters, wherein the first radius is the radius of each area in the screen, and the second radius is the radius of each area in the image to be tested; determine the position of the device's light-transmitting area in the image to be tested based on the second radius of the device's light-transmitting area; and determine the position of the screen trace area in the image to be tested based on the second radius of the screen trace area and the second radius of the ink-covered area.

[0106] In one example, the processing device 21 is used to input the image to be tested into a pre-trained defect classification model to obtain the location and type of defects present on the screen in the image to be tested.

[0107] In one example, the processing device 21 is used to collect sample images, including defect-free sample images and defective sample images. The defective sample images include: sample defect boxes located in the sample images and sample defect types corresponding to the sample defect boxes; train the neural network model to be trained using a portion of the sample images until the neural network model converges; evaluate the performance of the converged neural network model using another portion of the sample images that were not used for training; and optimize the converged neural network model based on the evaluation results of the performance of the converged neural network model until the performance of the neural network model meets the performance threshold.

[0108] In one example, the neural network model to be trained includes an input layer, a feature extraction layer, a feature fusion layer, and an output layer. The processing device 21 is used to input a portion of the sample images into the input layer for preprocessing; input the preprocessed sample images into the feature extraction layer to extract features from each sample image, obtaining multiple feature maps of different resolutions corresponding to each sample image; input the multiple feature maps of different resolutions corresponding to each sample image into the feature fusion layer for feature fusion, obtaining multiple fused feature maps of different resolutions corresponding to each sample image; input the multiple fused feature maps of different resolutions corresponding to each sample image into the output layer to obtain the predicted bounding box position, the predicted defect type at the predicted bounding box position, and the confidence level for each sample image; determine whether the neural network model has converged based on the predicted bounding box position, the predicted defect type at the predicted bounding box position, the position of the defective bounding box in the sample image, and the defect type of the defective bounding box; if the neural network model has not converged, return to the step of inputting a portion of the sample images into the input layer for preprocessing, until the neural network model converges.

[0109] In one example, the processing device 21 is used to stitch together multiple sample images by at least one of random scaling, random cropping, and random arrangement to obtain a stitched image; adaptively draw bounding boxes based on the width and height of the sample defect boxes in the stitched image; and scale the resolution of the drawn image to a preset resolution, which is the same as the resolution of the image that can be input to the feature extraction layer.

[0110] The following describes the testing process of the testing system. Some details are not shown, and the steps can be modified, added, or removed as needed.

[0111] S1, Test preparation: including equipment startup, sample pretreatment, fixture preparation, etc.

[0112] S2, Parameter Adjustment: This mainly includes adjusting the position of the stage to ensure the accuracy of the shooting position and adjusting the brightness of the light source. You can adjust it manually based on experience, or you can choose the automatic mode to automate the equipment parameters.

[0113] S3, Image Acquisition: The computer sends a photo capture command to acquire one or more images.

[0114] S4, Defect Calculation: The program calculates a series of data based on the obtained image using the screen hole area defect detection method described above.

[0115] S5, Output: The computer outputs the data calculated in S4.

[0116] The program interface of the detection system is as follows Figure 10 As shown, the interface mainly includes an operation bar, image display, data display, and engineering information. The operation bar, located on the left side of the interface, includes options for user name, test mode, vision debugging, vision calibration, and system settings. The test mode includes both online and offline testing. The image display is located in the center, showing the image captured by the image acquisition device on the screen aperture area. Data display is located in the lower left and lower right corners of the interface. The lower left corner displays the total number of tests, output, and number of defects. The lower right corner displays the type, quantity, and size of defects in the current image under test, as well as the percentage of defects in historical tests. Engineering information is displayed in the upper right corner of the interface.

[0117] Figure 11 This is a schematic structural block diagram of a screen defect detection device 300 provided in an embodiment of this application. The screen 100 defect detection device 300 includes: a processor 310, a memory 320, a communication interface 330, and a bus 340.

[0118] The processor 310 can be connected to the memory 320. The memory 320 can be used to store the program code and data. Therefore, the memory 320 can be a storage unit inside the processor 310, an external storage unit independent of the processor 310, or a component that includes both the storage unit inside the processor 310 and the external storage unit independent of the processor 310.

[0119] Optionally, the screen defect detection device 300 may also include a bus 340. The memory 320 and communication interface 330 can be connected to the processor 310 via the bus 340. The bus 340 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 340 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0120] It should be understood that in the embodiments of this application, the processor 310 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 310 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0121] The memory 320 may include read-only memory and random access memory, and provides instructions and data to the processor 310. A portion of the processor 310 may also include non-volatile random access memory. For example, the processor 310 may also store device type information.

[0122] When the screen defect detection device is running, the processor 310 executes computer execution instructions in the memory 320 to perform the operation steps of the screen defect detection method described above using the hardware resources in the screen defect detection device.

[0123] It should be understood that the screen defect detection device 300 according to the embodiments of this application may correspond to the processing device 21 in the screen defect detection device system of the embodiments of this application, and may correspond to the execution of the embodiments of this application. Figure 1 For the sake of brevity, the corresponding entities in the method shown will not be described in detail here.

[0124] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed, implement the method provided in the embodiments of this application.

[0125] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to execute the method provided in the embodiments of this application.

[0126] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0127] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0128] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0129] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0131] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0132] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a memory (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A method for detecting defects in the hole area of ​​a screen, characterized in that, The screen includes an aperture area and a support plate covering area surrounding the aperture area. The aperture area includes a device light-transmitting area, an ink-covered area surrounding the device light-transmitting area, and a screen wiring area surrounding the ink-covered area. The method includes: Acquire a test image of the screen, the test image including the light-transmitting area of ​​the device, the ink-covered area, the screen wiring area, and part of the support plate covered area; The location and type of defects in the screen, as well as the locations of the light-transmitting area of ​​the device, the ink-covered area, the screen wiring area, and the support plate-covered area are determined based on the image to be tested. Based on the location and type of defects in the screen, the location of each area, and the correspondence between each area and preset defects, determine whether the defects in the screen are defects. The step of determining whether a defect in the screen is a defect based on the location and type of the defect, the location of each area, and the correspondence between each area and a preset defect includes: Based on the location of the defects in the screen and the location of the corresponding area in the screen hole area, determine the specific area in which the defects fall within the screen hole area. The types of defects falling into each area are compared with the types of pre-defined defective defects corresponding to each area to determine whether the defects falling into each area are defective.

2. The method according to claim 1, characterized in that, The area includes: the light-transmitting area of ​​the device and the screen wiring area; The step of determining whether a defect at a given location is a defective defect based on the location and type of defects on the screen, the location of each area, and the correspondence between each area and preset defects includes: If the location of the defect on the screen falls within the light-transmitting area of ​​the device, and the type of the defect is the same as the type of a preset defect corresponding to the light-transmitting area of ​​the device, then the defect is determined to be a defect. If the location of a defect on the screen falls within the screen trace area, and the type of the defect is the same as the type of a preset defect corresponding to the screen trace area, then the defect is determined to be a defective defect.

3. The method according to claim 1 or 2, characterized in that, The area includes: the light-transmitting area of ​​the device and the screen wiring area; Determining the positions of the light-transmitting area of ​​the device, the ink-covered area, the screen trace area, and the support plate covered area based on the image to be tested includes: Based on the first radius of the light-transmitting area of ​​the device, the first radius of the ink-covered area, the first radius of the screen trace area, and the preset camera parameters, the second radius of the light-transmitting area of ​​the device, the second radius of the screen trace area, and the second radius of the ink-covered area are determined, wherein the first radius is the radius of each area in the screen, and the second radius is the radius of each area in the image to be tested; The position of the light-transmitting area of ​​the device in the image under test is determined based on the second radius of the light-transmitting area of ​​the device. The position of the screen trace area in the image under test is determined based on the second radius of the screen trace area and the second radius of the ink-covered area.

4. The method according to any one of claims 1 to 3, characterized in that, Determining the location and type of defects in the screen based on the image to be tested includes: The image to be tested is input into a pre-trained defect classification model to obtain the location and type of defects present on the screen in the image to be tested.

5. The method according to claim 4, characterized in that, The pre-trained defect classification model is obtained in the following way: Collect sample images, including defect-free sample images and defective sample images. The defective sample images include: a sample defect box located in the sample image and a sample defect type corresponding to the sample defect box. The neural network model to be trained is trained using a portion of the sample images until the neural network model converges; The performance of the converged neural network model is evaluated using another portion of the sample images that were not used in training. Based on the performance evaluation results of the converged neural network model, the converged neural network model is optimized until its performance meets the performance threshold.

6. The method according to claim 5, characterized in that, The neural network model to be trained includes: an input layer, a feature extraction layer, a feature fusion layer, and an output layer; The step of training the neural network model to be trained using a portion of the sample images until the neural network model converges includes: A portion of the sample images is input into the input layer for preprocessing. The preprocessed sample images are input into the feature extraction layer to extract features from each sample image, resulting in multiple feature maps of different resolutions for each sample image. Multiple feature maps of different resolutions corresponding to each sample image are input into the feature fusion layer for feature fusion to obtain multiple fused feature maps of different resolutions corresponding to each sample image; Multiple fused feature maps of different resolutions corresponding to each sample image are input into the output layer to obtain the predicted bounding box position, the predicted defect type, and the confidence level corresponding to each sample image. Based on the predicted bounding box position of each sample image, the predicted defect type corresponding to the predicted bounding box position, and the position of the sample defect box in the sample image and the sample defect type corresponding to the sample defect box, it is determined whether the neural network model has converged. If the neural network model fails to converge, the process returns to the step of inputting a portion of the sample images into the input layer to preprocess the sample images, until the neural network model converges.

7. The method according to claim 6, characterized in that, The preprocessing of the sample image includes: Multiple sample images are stitched together using at least one of random scaling, random cropping, and random arrangement to obtain a stitched image. Adaptive outlining is performed based on the width and height of the sample defect box in the stitched image; The resolution of the image after the outline is scaled to a preset resolution, which is the same as the resolution of the image that can be input into the feature extraction layer.

8. A screen hole area defect detection system, characterized in that, The screen includes a hole area and a support plate covering area surrounding the hole area. The hole area includes a device light-transmitting area, an ink covering area surrounding the device light-transmitting area, and a screen wiring area surrounding the ink covering area. The system includes: an image acquisition device and a processing device connected to the image acquisition device; The image acquisition device is used to acquire a test image of the screen, the test image including the light-transmitting area of ​​the device, the ink-covered area, the screen wiring area, and part of the support plate covered area; The processing device is used to determine the location and type of defects existing in the screen, as well as the location of the light-transmitting area of ​​the device, the ink-covered area, the screen wiring area, and the support plate covered area, based on the image to be tested. It is also used to determine whether the defects existing on the screen are defects based on the location and type of the defects, the location of each area, and the correspondence between each area and the preset defects. The step of determining whether a defect in the screen is a defect based on the location and type of the defect, the location of each area, and the correspondence between each area and a preset defect includes: Based on the location of the defects in the screen and the corresponding location of the screen hole area, determine the specific area in which the defects fall within the screen hole area. The types of defects falling into each area are compared with the types of pre-defined defective defects corresponding to each area to determine whether the defects falling into each area are defective.

9. The system according to claim 8, characterized in that, The processing device is used to determine that the defect is a defective defect when the location of the defect on the screen falls into the light-transmitting area of ​​the device and the type of the defect is the same as the type of a preset defect corresponding to the light-transmitting area of ​​the device. If the location of a defect on the screen falls within the screen trace area, and the type of the defect is the same as the type of a preset defect corresponding to the screen trace area, then the defect is determined to be a defective defect.

10. A device for detecting defects in the hole area of ​​a screen, characterized in that, The device includes a memory and a processor, the memory being used to store instructions that, when executed by the processor, cause the screen hole area defect detection device to perform the method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed, implement the method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, The computer program product includes: computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 7.

Images