Method and system for detecting foreign matters, electronic equipment and storage medium

Abstract

The invention discloses a method and a system for detecting foreign matters, electronic equipment and a storage medium, which are used for reducing the loss of an OLED (Organic Light Emitting Diode) screen. The method for detecting the foreign matter comprises the following steps: acquiring a plurality of target detection images acquired by a camera module; extracting a plurality of defect imaging features from the plurality of target detection images; analyzing the plurality of defect imaging features to obtain an analysis result; respectively correcting the effective display areas of the plurality of target detection images to obtain a plurality of corrected images; performing intersection operation on the defect area of the second correction image and the defect area of the third correction image to obtain an operation result; calculating the ratio of the area of the operation result to the area of the defect area of the third corrected image and the ratio of the area of the operation result to the area of the defect area of the second corrected image to obtain two ratio results; respectively comparing the two ratio results with a preset threshold value to obtain two comparison results; and determining the position of the foreign matter according to the analysis result and the two comparison results.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image processing detection, and in particular, to a method, a system, an electronic device, and a storage medium for detecting foreign objects. Background Art

[0002] In the field of OLED display manufacturing, due to its complex manufacturing process and high process difficulty, the cost of the screen body is relatively high. Therefore, in the entire mobile phone screen manufacturing process, it is necessary to reduce the loss of OLED screens as much as possible. In the bonding process of the CG cover plate and the OLED display, the characteristics of the OCA glue being colorless, transparent, and having strong adhesion are usually used to achieve the bonding of the two. During the bonding process of the CG and the OCA glue, poor bonding often occurs due to factors such as dust falling and poor incoming material conditions, which may lead to the scrapping of the final product.

[0003] In the prior art, polarization light detection technology, laser scanning technology, and image processing methods are commonly used to detect foreign objects. However, the bonding foreign objects are usually black, and some surface foreign objects and foreign objects under the CG film also have characteristics similar to those of the bonding foreign objects, which causes interference to the detection of which layer of bonding has foreign objects. Summary of the Invention

[0004] The present application discloses a method, a system, an electronic device, and a storage medium for detecting foreign objects, which are used to reduce the loss of OLED screens.

[0005] The first aspect of the present application discloses a method for detecting foreign objects, including: Obtaining a plurality of target detection images collected by a camera module, where the camera module is a combination of multiple cameras and multiple light sources, and the plurality of target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image; Respectively extracting corresponding multiple defect imaging features from the plurality of target detection images; Analyzing the plurality of defect imaging features to obtain an analysis result; Respectively correcting the effective display areas of the plurality of target detection images to obtain a plurality of corrected images, where the plurality of corrected images include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image; Performing an intersection operation on the defect areas of the second corrected image and the defect areas of the third corrected image to obtain an operation result; Calculating the ratio of the area of the operation result to the area of the defect area of the third corrected image, and the ratio of the area of the operation result to the area of the defect area of the second corrected image, to obtain two ratio results; Respectively comparing the two ratio results with a preset threshold to obtain two comparison results; Confirm the position of the foreign object according to the analysis result and the two comparison results.

[0006] Optionally, the multiple defect imaging features include a first defect imaging feature, a second defect imaging feature, a third defect imaging feature, and a fourth defect imaging feature. The first defect imaging feature corresponds to the first detection image, the second defect imaging feature corresponds to the second detection image, the third defect imaging feature corresponds to the third detection image, and the fourth defect imaging feature corresponds to the fourth detection image. Analyzing the multiple defect imaging features to obtain an analysis result includes: When it is analyzed that the first defect imaging feature and the fourth defect imaging feature are white, and the second defect imaging feature and the third defect imaging feature are black, the analysis result indicates that the foreign object is a sub-membrane foreign object; When it is analyzed that the second defect imaging feature and the third defect imaging feature are black, and the first defect imaging feature or the fourth defect imaging feature is black, the analysis result indicates that the foreign object is a surface foreign object.

[0007] Optionally, before respectively correcting the effective display areas of the multiple target detection images to obtain multiple corrected images, it further includes: Map the third detection image and the fourth detection image into the second detection image through a preset transformation matrix; Extract the second effective display area of the second detection image according to the gray gradient method; According to the second effective display area, respectively intercept the corresponding first effective display area, third effective display area, and fourth effective display area of the first detection image, the third detection image, and the fourth detection image.

[0008] Optionally, the effective display areas of the multiple target detection images include a first effective display area, a second effective display area, a third effective display area, and a fourth effective display area. Respectively correcting the effective display areas of the multiple target detection images to obtain multiple corrected images, the multiple corrected images include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image, including: Obtain a circumscribed rectangle according to the contour of the second effective display area; Calculate a perspective transformation matrix according to the physical size and the four corner coordinates and length and width of the circumscribed rectangle; According to the perspective transformation matrix, respectively correct the first effective display area, the second effective display area, the third effective display area, and the fourth effective display area to respectively obtain corresponding first corrected images, second corrected images, third corrected images, and fourth corrected images.

[0009] Optionally, after performing an intersection operation on the defective areas of the second corrected image and the defective areas of the third corrected image to obtain an operation result, the following steps are further included: When a protective film is covered on the CG cover plate, perform convolution filtering on the effective display areas of the multiple target detection images respectively to obtain corresponding multiple filtering results; Extract the gray feature values of the multiple target detection images; Extract the gray feature values of the multiple filtering results.

[0010] Optionally, after extracting the gray feature values of the multiple filtering results, the following steps are further included: Extract the gray feature value of the operation result; Confirm the position of the foreign object according to the gray feature values of the multiple target detection images, the gray feature values of the multiple filtering results, and the gray feature value of the operation result.

[0011] Optionally, the camera module includes an upper camera, a lower camera, an upper light source, and a lower light source. Obtaining the multiple target detection images collected by the camera module, where the multiple target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image includes: Obtain the first detection image collected by the combination of the upper camera and the upper light source; Obtain the second detection image collected by the combination of the upper camera and the lower light source; Obtain the third detection image collected by the combination of the lower camera and the upper light source; Obtain the fourth detection image collected by the combination of the lower camera and the lower light source.

[0012] A second aspect of the present application discloses a system for detecting foreign objects, including: An acquisition unit, configured to acquire multiple target detection images collected by a camera module, where the camera module is a combination of multiple cameras and multiple light sources, and the multiple target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image; A defective imaging feature unit, configured to extract corresponding multiple defective imaging features from the multiple target detection images respectively; An analysis unit, configured to analyze the multiple defective imaging features to obtain an analysis result; A correction unit, configured to correct the effective display areas of the multiple target detection images respectively to obtain multiple corrected images, where the multiple corrected images include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image; An intersection operation unit is configured to perform an intersection operation on the defective regions of the second corrected image and the third corrected image to obtain an operation result; A ratio unit is configured to calculate the ratio of the area of the operation result to the area of the defective region of the third corrected image, and the ratio of the area of the operation result to the area of the defective region of the second corrected image, to obtain two ratio results; A comparison unit is configured to respectively compare the two ratio results with a preset threshold to obtain two comparison results; A first confirmation unit is configured to confirm the position of the foreign object according to the analysis result and the two comparison results.

[0013] A third aspect of the present application provides an electronic device, including: A processor, a memory, an input / output unit, and a bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method for detecting a foreign object as described in the first aspect and any optional method for detecting a foreign object in the first aspect.

[0014] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored, and when the program is executed on a computer, it executes the method for detecting a foreign object as described in the first aspect and any optional method for detecting a foreign object in the first aspect.

[0015] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: The present application provides a method for detecting foreign objects, which includes obtaining a plurality of target detection images collected by an imaging module. The imaging module is a combination of multiple cameras and multiple light sources. The plurality of target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image; respectively extracting corresponding multiple defect imaging features from the plurality of target detection images; analyzing the multiple defect imaging features to obtain an analysis result; respectively correcting the effective display areas of the plurality of target detection images to obtain a plurality of corrected images, where the plurality of corrected images include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image; performing an intersection operation on the defect areas of the second corrected image and the defect areas of the third corrected image to obtain an operation result; calculating the ratio of the area of the operation result to the area of the defect area of the third corrected image, and the ratio of the area of the operation result to the area of the defect area of the second corrected image to obtain two ratio results; respectively comparing the two ratio results with a preset threshold to obtain two comparison results; and confirming the position of the foreign object based on the analysis result and the two comparison results. The method of the present application collects a plurality of target detection images through multiple cameras and multiple light sources, uses the principle of optical imaging characteristics and feature differences under multiple light sources to preliminarily analyze the defect imaging features, and then performs geometric correction on the effective display areas of each image to ensure that each image is strictly aligned for subsequent intersection operations; then calculates the ratio of the result of the intersection operation to the areas of the defect areas on two of the corrected images, and determines the layer where the defect (foreign object) is located by comparing the ratio with the threshold and combining the previous preliminary analysis, realizing high-precision detection and positioning of adhered foreign objects and surface foreign objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0017] Figure 1 Schematic diagram of an embodiment of the method for detecting foreign objects in the present application; Figure 2 Schematic diagram of another embodiment of the method for detecting foreign objects in the present application; Figure 3 Schematic diagram of another embodiment of the method for detecting foreign objects in the present application; Figure 4 Schematic diagram of an embodiment of the system for detecting foreign objects in the present application; Figure 5 Schematic diagram of another embodiment of the electronic device in the present application; Figure 6Schematic diagram of the target detection image for the upper camera and lower light source combination to photograph foreign objects under the film or the lower camera and upper light source combination to photograph foreign objects under the film; Figure 7 Schematic diagram of the target detection image for the upper camera and upper light source combination to photograph foreign objects under the film or the lower camera and lower light source combination to photograph foreign objects under the film; Figure 8 Schematic diagram of the target detection image for the upper camera and upper light source combination to photograph foreign objects on the surface or the lower camera and lower light source combination to photograph foreign objects on the surface; Figure 9 Schematic diagram of the original image without correction; Figure 10 For Figure 9 Schematic diagram of the image obtained after correction; Figure 11 Schematic diagram of the effective display area of the target detection image without convolution filtering; Figure 12 For Figure 11 Schematic diagram of the image obtained after convolution filtering; Figure 13 Schematic diagram of the operation result of the intersection operation when the foreign object is a foreign object under the film: Figure 14 Schematic diagram of the operation result of the intersection operation when the foreign object is a foreign object on the surface. Detailed implementation mode

[0018] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0019] It should be understood that when used in the specification of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0020] It should also be understood that the term "and / or" used in the specification of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0021] As used in the specification of this application, the term "if" can be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0022] In addition, in the description of this application's specification, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0023] The reference to "one embodiment" or "some embodiments" or the like described in this application's specification means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but rather mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0024] Due to the complex manufacturing process and high process difficulty of OLED display screens, the cost of the screen body is relatively high. During the manufacturing process of the entire mobile phone screen, it is necessary to minimize the loss of OLED screens as much as possible. During the bonding process of the CG cover plate and the OLED display screen, the two are bonded together by utilizing the characteristics of OCA glue such as being colorless, transparent, and having strong adhesion. However, during the bonding process of CG and OCA, it is very easy to cause poor bonding due to reasons such as dust falling and poor incoming material conditions, resulting in the scrapping of the final product. If it is possible to check whether foreign objects are generated during the bonding of CG and OCA before bonding with OLED, the loss caused by bonding foreign objects can be minimized.

[0025] Based on this, this application discloses a method, system, electronic device, and storage medium for detecting foreign objects to reduce the loss of OLED screens.

[0026] Next, the technical solutions in this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without making creative efforts belong to the scope of protection of this application.

[0027] The method of the present application can be applied to servers, devices, terminals, or other devices with logical processing capabilities, and the present application does not limit this. For the convenience of description, the following takes the execution entity as a terminal as an example for description.

[0028] Please refer to Figure 1 , an embodiment of a method for detecting foreign objects provided by the present application includes: 101. Obtain a plurality of target detection images collected by a camera module, where the camera module is a combination of multiple cameras and multiple light sources, and the plurality of target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image; In this embodiment, it should be noted that the camera module includes an upper camera, a lower camera, an upper light source, and a lower light source. When detecting foreign objects in a target detection object (the target detection object is composed of a CG cover plate, OCA glue, and a double film of an OLED display screen in sequence), the camera above the CG cover plate is the upper camera, and the light source located between the upper camera and the CG cover plate is called the upper light source; similarly, the camera below the double film is the lower camera, and the light source located between the lower camera and the double film is called the lower light source. Among them, the upper light source and the lower light source are coaxial light sources, and the optical path is perpendicular to the target detection object; the upper camera and the lower camera are pre-calibrated for their basic positions to initially reduce the position deviation between the captured images. Then, 4 target detection images are collected by the camera module, respectively: obtain the first detection image collected by the combination of the upper camera and the upper light source; obtain the second detection image collected by the combination of the upper camera and the lower light source; obtain the third detection image collected by the combination of the lower camera and the upper light source; and obtain the fourth detection image collected by the combination of the lower camera and the lower light source. During the collection process, the position of the detection object is kept fixed to ensure comparability between different images.

[0029] 102. Extract corresponding multiple defect imaging features from the multiple target detection images respectively; In this embodiment, each target detection image can be grayscale processed to convert the color image into a grayscale image, simplifying the subsequent feature extraction process. Then, for different images, analyze the gray value distribution, texture information, etc. of the pixels in the image. For areas where foreign objects may exist, extract features such as their gray values, contrast ratios, and edge contours as defect imaging features. For example, if the gray value of a certain area in the image is significantly different from the surrounding area and the edge is clear, these features can be recorded as defect imaging features.

[0030] 103. Analyze the multiple defect imaging features to obtain an analysis result; In this embodiment, the multiple defect imaging features include a first defect imaging feature, a second defect imaging feature, a third defect imaging feature, and a fourth defect imaging feature. The first defect imaging feature corresponds to the first detection image, the second defect imaging feature corresponds to the second detection image, the third defect imaging feature corresponds to the third detection image, and the fourth defect imaging feature corresponds to the fourth detection image.

[0031] It should be noted that taking pictures using the upper and lower cameras and light sources utilizes the principle of optical imaging, specifically the transmission and reflection of light. For example, if a foreign object is between the CG cover plate and the heavy film (referred to as a foreign object under the film), that is, mixed in the medium of OCA glue, when the camera and the light source are on the same side, then the light transmits out from the position without the foreign object and is basically vertically reflected back at the position with the foreign object. The final image shows that the area with defects (foreign objects) is brighter than other parts, that is, the defect imaging feature is white. Please refer to Figure 7 , Figure 7 FIG. is a schematic diagram of the target detection image of the foreign object under the film taken by the combination of the upper camera and the upper light source or the combination of the lower camera and the lower light source. Figure 7 The defect imaging feature shown is white; when the camera and the light source are on different sides, the light transmits into the camera from the position without the foreign object, and at the position with the foreign object, the light can hardly enter the camera. The final image shows that the area with defects is darker than other parts, that is, the defect imaging feature is black. Please refer to Figure 6 , Figure 6 FIG. is a schematic diagram of the target detection image of the foreign object under the film taken by the combination of the upper camera and the lower light source or the combination of the lower camera and the upper light source. Figure 6 The defect imaging feature shown is black. If the foreign object is on the CG cover plate or under the heavy film (referred to as a surface foreign object), when the camera and the light source are on the same side, then the light transmits out from the position without the foreign object, and at the defective place, the light will be scattered due to the rough surface of the foreign object. The final image shows that the defective area is darker than other parts, that is, the defect imaging feature is black. Please refer to Figure 8 , Figure 8 FIG. is a schematic diagram of the target detection image of the surface foreign object taken by the combination of the upper camera and the upper light source or the combination of the lower camera and the lower light source. Figure 8 The defect imaging feature shown is black; when the camera and the light source are on different sides, the light transmits into the camera from the position without the foreign object, and at the position with the foreign object, the light can hardly enter the camera. The final image shows that the area with defects is darker than other parts, that is, the defect imaging feature is black.

[0032] In summary, the above rules are summarized as follows: when it is analyzed that the imaging features of the first defect and the fourth defect are white, and the imaging features of the second defect and the third defect are black, the analysis result indicates that the foreign object is a submembrane foreign object; when it is analyzed that the imaging features of the second defect and the third defect are black, and the imaging feature of the first defect or the fourth defect is black, the analysis result indicates that the foreign object is a surface foreign object.

[0033] 104. Correct the effective display areas of multiple target detection images respectively to obtain multiple corrected images, including a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image. In this embodiment, an image segmentation algorithm, such as methods based on threshold segmentation, region growing, etc., is used to identify the areas in the image that contain the detected object and are valuable for foreign object detection, remove the irrelevant background parts, and determine the effective display areas. Among them, the cut effective display area is a rectangle that can basically enclose the defect. There will inevitably be perspective distortion in the cut effective display area. For images with perspective distortion, a perspective transformation algorithm is used. By selecting multiple feature points in the image, such as the corner points or specific marker points of the detected object, the perspective transformation matrix is calculated. The effective display area is transformed using this matrix to correct the image to an image under a standard perspective, making the shape and size of the object more accurate for subsequent precise analysis. The effective display area of the first target detection image is corrected to obtain the first corrected image, the effective display area of the second target detection image is corrected to obtain the second corrected image, the effective display area of the third target detection image is corrected to obtain the third corrected image, and the effective display area of the fourth target detection image is corrected to obtain the fourth corrected image.

[0034] 105. Perform an intersection operation on the defect areas of the second corrected image and the third corrected image to obtain the operation result. In this embodiment, although step 103 can initially determine the location of the foreign object, some foreign objects are tightly attached to the surface of the CG cover plate or the heavy film. At this time, the captured image is also close to the situation of the foreign object under the film, so it is difficult to distinguish. Therefore, further quantitative judgment is required. From the descriptions of steps 101 and 104, it can be seen that the second corrected image corresponds to the second target detection image collected by the upper camera and the lower light source, and the third corrected image corresponds to the third target detection image collected by the lower camera and the upper light source. Both belong to the situation where the camera and the light source are on different sides. The reason for choosing such an orientation is that the transmitted light clearly presents all foreign objects on the target detection object, and at the same time, the contour of the defect is relatively clear, which is convenient for subsequent defect feature analysis; while in the orientation where the camera and the light source are on the same side, when the foreign object is on the CG cover plate and both the camera and the light source are under the heavy film, due to the shielding of the screen body, the imaging effect of the foreign object is not good. When the foreign object is under the heavy film and both the camera and the light source are above the CG cover plate. Also due to the shielding of the screen body, the imaging effect of the foreign object is not good. Therefore, in this embodiment, the second corrected image and the third corrected image are selected for quantitative detection.

[0035] There will be an obvious position difference of the surface foreign object in the second corrected image and the third corrected image. This is mainly because the working distance of the surface foreign object to one of the cameras is closer than that to the other camera. At the same time, the thickness of the CG cover plate is relatively small (0.5 mm), and the depth of field of the camera is greater than 1 mm. And the focal length f basically does not change with respect to the surface foreign object and the adhered foreign object. Therefore, from the formula: 1 / f = 1 / w + 1 / u (where f is the focal length, w is the working distance of the camera, and u is the image distance), it can be known that when the working distances of the two cameras are different, there will be a position deviation in the imaging of the target defect in the two cameras. If the intersection of the second corrected image and the third corrected image is taken to see whether the defect areas in the two images are completely or basically overlapped, the position deviation of the defect area can be initially seen. Refer to Figure 13 , Figure 13 is a schematic diagram of the operation result of the intersection operation when the foreign object is a foreign object under the film; refer to Figure 14 , Figure 14 is a schematic diagram of the operation result of the intersection operation when the foreign object is a surface foreign object.

[0036] 106. Calculate the ratio of the area of the operation result to the area of the defect area of the third corrected image, and the ratio of the area of the operation result to the area of the defect area of the second corrected image, to obtain two ratio results; In this embodiment, after taking the intersection of the defective regions of the two images in step 105, sometimes it is not possible to accurately determine whether a foreign object is a surface foreign object only by visually observing the position deviation. Therefore, numerical calculations are required, and more accurate judgments can be made through numerical comparisons, that is, quantifying the detection process. The first numerical values to be obtained are the area S of the defective region of the operation result of the intersection operation in step 105, the area S1 of the defective region of the second corrected image, and the area S2 of the defective region of the third corrected image. The area of the intersection region, the defective region of the second corrected image, and the defective region of the third corrected image can be calculated by using image analysis software or algorithms to count the number of pixels in the region. For irregular regions, pixel filling methods or grid-based area estimation methods can be used to ensure the accuracy of area calculation. Then, the area of the intersection region is divided by the area of the defective region of the second corrected image and the area of the defective region of the third corrected image respectively to obtain two ratio results for subsequent comparison and analysis with a preset threshold.

[0037] 107. Compare the two ratio results with the preset threshold respectively to obtain two comparison results; In this embodiment, a reasonable preset threshold is determined based on the experimental results obtained from a large number of pre-production experiments. This threshold should be able to accurately distinguish foreign objects at different positions, and at the same time, considering the errors and uncertainties in the detection process, ensure the robustness of the threshold. In this embodiment, the preset threshold can be set to 75%. That is, if the comparison results show that both ratio results are greater than 75%, it means that the intersection of the defective regions of the two images meets the standard of "completely or basically coincident", and it can be determined that the foreign object is a sub-membrane foreign object; if the comparison results show that at least one of the two ratio results is less than 75%, it means that the intersection of the defective regions of the two images does not meet the standard of "completely or basically coincident", and it can be determined that the foreign object is a surface foreign object.

[0038] 108. Confirm the position of the foreign object according to the analysis result and the two comparison results.

[0039] In this embodiment, the analysis result obtained in step 103 (judging whether the foreign object is a sub-membrane foreign object or a surface foreign object) is combined with the two comparison results obtained in step 107. If the analysis result is a sub-membrane foreign object and both comparison results meet the ratio range corresponding to the sub-membrane foreign object, it is confirmed that the foreign object is located under the membrane; if the analysis result is a surface foreign object and the comparison results also meet the characteristics of the surface foreign object, it is confirmed that the foreign object is a surface foreign object, thereby accurately determining the position of the foreign object.

[0040] In this embodiment, first, a plurality of target detection images collected by the imaging module are obtained. The imaging module is a combination of multiple cameras and multiple light sources. The plurality of target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image. Corresponding multiple defect imaging features are extracted from the plurality of target detection images respectively. The multiple defect imaging features are analyzed to obtain an analysis result. The effective display areas of the plurality of target detection images are corrected respectively to obtain a plurality of corrected images, which include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image. The intersection operation is performed on the defect area of the second corrected image and the defect area of the third corrected image to obtain an operation result. The ratio of the area of the operation result to the area of the defect area of the third corrected image and the ratio of the area of the operation result to the area of the defect area of the second corrected image are calculated to obtain two ratio results. The two ratio results are compared with a preset threshold respectively to obtain two comparison results. The position of the foreign object is confirmed according to the analysis result and the two comparison results. The method of the present application collects a plurality of target detection images through multiple cameras and multiple light sources, uses the principle of optical imaging characteristics and feature differences under multiple light sources to preliminarily analyze the defect imaging features, and then geometrically corrects the effective display area of each image to ensure that each image is strictly aligned for subsequent intersection operation. Then, the ratio of the result of the intersection operation to the area of the defect areas on two of the corrected images is calculated. By comparing the ratio with the threshold and combining the previous preliminary analysis, the layer where the defect (foreign object) is located is determined, and high-precision detection and positioning of the adhered foreign object and the surface foreign object are realized.

[0041] Please refer to Figure 2 , before performing step 104, the effective display areas of the plurality of target detection images should be obtained first, and then these effective display areas are corrected. Specifically, it may include, but is not limited to, the following: 201. Map the third detection image and the fourth detection image into the second detection image through a preset transformation matrix; In this embodiment, a preset transformation matrix is obtained by calibrating the imaging module and studying the imaging law of the detected object. This preset transformation matrix is usually calculated based on the internal parameters, external parameters of the camera, and the relative position relationship between the detected object and the camera. For example, the Zhang Zhengyou calibration method can be used to obtain the internal parameters and distortion parameters of the camera, and the relative rotation and translation relationships between the cameras are determined in combination with the mechanical structure, and then the preset transformation matrix for image mapping is calculated. Specifically, for the third detection image, each pixel point in the image is traversed, and according to the preset transformation matrix, the coordinate transformation calculation is performed on the coordinates of each pixel point. Assuming that the original pixel point coordinates are (x, y), after the action of the transformation matrix M, the new coordinates (x', y') are obtained. In the same way, the coordinate transformation is performed on all pixel points in the fourth detection image. The pixel points of the transformed third detection image and fourth detection image are placed at the corresponding positions of the second detection image according to the new coordinates. During the placement process, if the new coordinates exceed the range of the second detection image, corresponding boundary processing is performed, such as truncation or filling.

[0042] 202. Extract the second effective display area of the second detection image according to the gray gradient method; In this embodiment, if the second detection image is a color image, it is first converted into a gray image. The conversion method can adopt the weighted average method, and the red, green, and blue channels of the color image are weighted and summed according to certain weights to obtain the gray image. For the gray-scaled second detection image, the gray gradient is calculated for each pixel point in the image. The gray gradient represents the speed and direction of the change of the gray value in the image. Then a suitable gradient amplitude threshold T is set, and all pixel points in the image are traversed, and the pixel points with a gradient amplitude greater than the threshold T are marked. These marked pixel points are often located at the edges of the objects in the image or in the areas where the gray value changes greatly, and are related to the boundaries of the detected object and the areas where foreign objects are located. Connected component analysis is performed on the marked pixel points, and the mutually connected pixel points are grouped into regions. The regions that meet the characteristics of the detected object are selected from these regions, and the set of these regions is the second effective display area. For example, if the detected object is a rectangular screen body, then the selected regions should have a shape feature similar to a rectangle and a size within a reasonable range.

[0043] 203. Respectively intercept the corresponding first effective display area, third effective display area, and fourth effective display area of the first detection image, third detection image, and fourth detection image according to the second effective display area; In this embodiment, since the third detection image and the fourth detection image have been mapped to the second detection image in step 201, there is a certain corresponding relationship between them. By recording the transformation information of pixel points during the mapping process, the positions of the pixel points corresponding to each pixel point in the second effective display area in the first detection image, the third detection image, and the fourth detection image can be determined. Then, according to the boundary of the second effective display area, the corresponding third effective display area and fourth effective display area of the third detection image and the fourth detection image are respectively intercepted. For the first detection image, since it is an image taken by the same camera as the second detection image, the corresponding boundary coordinate range can be directly found in the first detection image according to the boundary coordinates of the second effective display area.

[0044] 204. Obtain the circumscribed rectangle according to the contour of the second effective display area; In this embodiment, not only can a closed rectangle, that is, the circumscribed rectangle, be formed by fitting four straight lines around the effective display area, but also an edge detection algorithm, such as the Canny edge detection algorithm, can be used to extract the edges of the second effective display area. The Canny edge detection algorithm can accurately extract the edge contours of objects in the image through steps such as Gaussian filtering to smooth the image, calculating the gradient magnitude and direction, non-maximum suppression, and double-threshold detection and connecting edges. After being processed by the Canny algorithm, the edge contours of the second effective display area are obtained, and these edge contours are composed of a series of pixel points. Then, the minimum circumscribed rectangle algorithm is used to process the extracted edge contours. The goal of this algorithm is to find a minimum rectangle that can completely contain these edge contour points. An algorithm based on rotating calipers can be used. By continuously rotating a virtual rectangle frame to closely fit the edge contour points, the minimum circumscribed rectangle is finally obtained. Calculate the coordinates of the four corner points of this circumscribed rectangle, as well as the length and width of the rectangle. These four corner point coordinates and the length and width information will be used for subsequent calculation of the perspective transformation matrix.

[0045] 205. Calculate the perspective transformation matrix according to the physical size, the four corner coordinates, and the length and width of the circumscribed rectangle; In this embodiment, the actual physical size of the detected object is known. For example, the actual length of the target detection object is L and the width is W. At the same time, the four corner point coordinates, length l, and width w of the circumscribed rectangle of the contour of the second effective display area can be obtained from step 204. The actual physical size represented by each pixel in the image is calculated through a proportional relationship. Then, according to the principle of perspective transformation, a perspective transformation matrix M1 is found such that the points in the image can correspond to the actual physical size after transformation. Using the known corner point coordinates of the circumscribed rectangle and the corresponding actual physical coordinates, this perspective transformation matrix M1 is solved by methods such as the least squares method. During the solution process, multiple equations need to be constructed. Substitute the corner point coordinates into the perspective transformation formula to form a system of equations, and then solve the system of equations to obtain the values of each element of the perspective transformation matrix M1.

[0046] 206. According to the perspective transformation matrix, correct the first effective display area, the second effective display area, the third effective display area, and the fourth effective display area respectively, and obtain the corresponding first corrected image, second corrected image, third corrected image, and fourth corrected image.

[0047] In this embodiment, for each pixel point in the first effective display area, according to the perspective transformation matrix M1 calculated in step 205, its coordinates are transformed. The coordinates of the pixel point are extended to homogeneous coordinates, and then multiplied by the perspective transformation matrix M1 to obtain the transformed homogeneous coordinates. Then, the homogeneous coordinates are converted to ordinary coordinates, and this new coordinate is the position of the corrected pixel point. According to the above method, the coordinate transformation is performed on all pixel points in the first effective display area. The transformed pixel points are placed on a new image canvas according to the new coordinate positions to form the first corrected image. During the placement process, if the new coordinate is not an integer, an interpolation algorithm, such as the bilinear interpolation algorithm, is used to calculate the pixel value at this position according to the values of the surrounding pixel points. In the same way, the second effective display area, the third effective display area, and the fourth effective display area are processed to obtain the second corrected image, the third corrected image, and the fourth corrected image respectively. These corrected images eliminate the distortion caused by factors such as the camera perspective and the object placement, making the shape and size of the objects in the image more accurate and facilitating the subsequent precise analysis and detection of foreign objects in the image. Refer to Figure 9 and Figure 10 , Figure 9 is a schematic diagram of the original image without correction, Figure 10 is for Figure 9 the schematic diagram of the image obtained after correction.

[0048] Please refer to Figure 3, in practice, during the process of bonding the CG cover plate and the OCA adhesive for the OLED display screen, a protective film is often already bonded to the CG cover plate, and there will inevitably be foreign objects between the CG cover plate and the protective film as well as on the protective film. The steps for detecting foreign objects in these two places can specifically include, but are not limited to, the following: 301. When a protective film covers the CG cover plate, perform convolution filtering on the effective display areas of multiple target detection images to obtain corresponding multiple filtering results; In this embodiment, a suitable convolution kernel is selected according to the detection requirements and image characteristics. Taking a 3×3 convolution kernel as an example, for the effective display area of each target detection image, starting from the first pixel in the upper left corner of the image, align the center of the convolution kernel with this pixel. Multiply each element of the convolution kernel by the corresponding image pixel value, and then add the products. The result obtained is used as the pixel value at this position after convolution. Then the convolution kernel moves one pixel to the right at a certain step size (such as a step size of 1), and the above operation is repeated until the end of this row. Then the convolution kernel moves down one row and continues to process the pixels in the next row until the convolution operation of the entire effective display area is completed, thereby obtaining the corresponding filtered result image. Refer to Figure 11 and Figure 12 , Figure 11 is a schematic diagram of the effective display area of the target detection image before convolution filtering, Figure 12 is for Figure 11 is a schematic diagram of the image obtained after convolution filtering of

[0049] 302. Extract the gray feature values of multiple target detection images; In this embodiment, if the target detection image is a color image, it needs to be first converted into a gray image. The weighted average method can be used to sum the pixel values of the red, green, and blue channels in the color image with certain weights to obtain the corresponding gray value. Then calculate the average gray value, maximum gray value, and minimum gray value of the image.

[0050] 303. Extract the gray feature values of multiple filtering results; In this embodiment, similar to step 302, calculate the average gray value, maximum gray value, and minimum gray value of the filtered image. Since the filtering changes the pixel value distribution of the image, these gray feature values will reflect the characteristics of the filtered image, such as the gray change situation after edge enhancement or noise suppression, which helps to further analyze the characteristics of foreign objects in the filtered image.

[0051] 304. Extract the gray feature values of the operation result; In this embodiment, the operation result is the result obtained after the intersection operation in step 105 when a protective film covers the CG cover plate. Similar to step 302, calculate the average gray value, maximum gray value, and minimum gray value of the operation result.

[0052] 305. Confirm the position of the foreign object based on the gray feature values of multiple target detection images, the gray feature values of multiple filtering results, and the gray feature value of the operation result.

[0053] In this embodiment, according to Figure 1 and Figure 2 As described in the illustrated embodiment, after determining that the foreign object is neither the foreign object between the CG cover plate and the heavy film nor the foreign object under the heavy film, it is necessary to consider whether it is a foreign object on the protective film or a foreign object between the protective film and the CG cover plate. Based on a large amount of experimental data and actual detection experience, a new set of judgment rules is established. For example, if the average gray value of the target detection image is low, the gray change at the edge of the filtered image is obvious (the gray standard deviation is large), and the gray feature of a certain area in the operation result image matches the foreign object feature (such as the average gray value is within a specific range), it can be preliminarily judged that there may be a foreign object in this area. Combining with the imaging characteristics of foreign objects at different positions in the previous optical imaging principle, if it conforms to the gray feature combination of the foreign object under the film (such as the foreign object area is brighter in some images and darker in other images), it is confirmed that the foreign object is the foreign object between the protective film and the CG cover plate; if it conforms to the gray feature combination of the surface foreign object (such as the foreign object area is darker in multiple images), it is judged as the foreign object on the protective film. By comprehensively analyzing these gray feature values from different sources, the position of the foreign object can be accurately determined.

[0054] The above embodiment has described the method for detecting foreign objects provided by the present application. Next, a method, a system, an electronic device, and a storage medium for detecting foreign objects provided by the present application will be described: Please refer to Figure 4 , an embodiment of a system for detecting foreign objects provided by the present application includes: An acquisition unit 401, configured to acquire multiple target detection images collected by a camera module, where the camera module is a combination of multiple cameras and multiple light sources, and the multiple target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image; A defect imaging feature unit 402, configured to extract corresponding multiple defect imaging features from the multiple target detection images respectively; An analysis unit 403, configured to analyze the multiple defect imaging features to obtain an analysis result; A correction unit 404, configured to correct the effective display areas of the multiple target detection images respectively to obtain multiple corrected images, where the multiple corrected images include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image; An intersection operation unit 405, configured to perform an intersection operation on the defect areas of the second corrected image and the third corrected image to obtain an operation result; A ratio unit 406 is used to calculate the ratio of the area of the operation result to the area of the defective region of the third corrected image and the ratio of the area of the operation result to the area of the defective region of the second corrected image, and obtain two ratio results; A comparison unit 407 is used to compare the two ratio results with a preset threshold respectively, and obtain two comparison results; A first confirmation unit 408 is used to confirm the position of the foreign object according to the analysis result and the two comparison results.

[0055] Optionally, the multiple defect imaging features include a first defect imaging feature, a second defect imaging feature, a third defect imaging feature, and a fourth defect imaging feature. The first defect imaging feature corresponds to the first detection image, the second defect imaging feature corresponds to the second detection image, the third defect imaging feature corresponds to the third detection image, and the fourth defect imaging feature corresponds to the fourth detection image. The analysis unit 403 is specifically used for: When it is analyzed that the first defect imaging feature and the fourth defect imaging feature are white, and the second defect imaging feature and the third defect imaging feature are black, the analysis result indicates that the foreign object is a submembrane foreign object; When it is analyzed that the second defect imaging feature and the third defect imaging feature are black, and the first defect imaging feature or the fourth defect imaging feature is black, the analysis result indicates that the foreign object is a surface foreign object.

[0056] Optionally, before the correction unit 404, it further includes: A mapping unit 409 is used to map the third detection image and the fourth detection image into the second detection image through a preset transformation matrix; An extraction unit 410 is used to extract the second effective display area of the second detection image according to the gray gradient method; An interception unit 411 is used to respectively intercept the corresponding first effective display area, third effective display area, and fourth effective display area of the first detection image, the third detection image, and the fourth detection image according to the second effective display area.

[0057] Optionally, the effective display areas of the multiple target detection images include a first effective display area, a second effective display area, a third effective display area, and a fourth effective display area. The correction unit 404 is specifically used for: Obtain a circumscribed rectangle according to the contour of the second effective display area; Calculate a perspective transformation matrix according to the physical size, the four corner coordinates and the length and width of the circumscribed rectangle; According to the perspective transformation matrix, respectively correct the first effective display area, the second effective display area, the third effective display area, and the fourth effective display area, and respectively obtain the corresponding first corrected image, second corrected image, third corrected image, and fourth corrected image.

[0058] Optionally, after the intersection operation unit 405, the following are further included: A filtering unit 412, configured to perform convolution filtering on the effective display areas of multiple target detection images respectively when a protective film is covered on the CG cover plate, so as to obtain corresponding multiple filtering results; A first grayscale eigenvalue unit 413, configured to extract the grayscale eigenvalues of multiple target detection images; A second grayscale eigenvalue unit 414, configured to extract the grayscale eigenvalues of multiple filtering results.

[0059] Optionally, after the second grayscale eigenvalue unit 414, the following are further included: A third grayscale eigenvalue unit 415, configured to extract the grayscale eigenvalue of the operation result; A second confirmation unit 416, configured to confirm the position of the foreign object according to the grayscale eigenvalues of multiple target detection images, the grayscale eigenvalues of multiple filtering results, and the grayscale eigenvalue of the operation result.

[0060] Optionally, the imaging module includes an upper camera, a lower camera, an upper light source, and a lower light source. The obtaining unit 401 is specifically configured to: Obtain a first detection image collected by the combination of the upper camera and the upper light source; Obtain a second detection image collected by the combination of the upper camera and the lower light source; Obtain a third detection image collected by the combination of the lower camera and the upper light source; Obtain a fourth detection image collected by the combination of the lower camera and the lower light source.

[0061] Please refer to Figure 5 , this application provides an electronic device, including: A processor 501, a memory 502, an input / output unit 503, and a bus 504.

[0062] The processor 501 is connected to the memory 502, the input / output unit 503, and the bus 504.

[0063] The memory 502 stores a program, and the processor 501 calls the program to execute the method for detecting foreign objects as described in Figure 1 , Figure 2 and Figure 3 .

[0064] This application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, it executes the method for detecting foreign objects as described in Figure 1 , Figure 2 and Figure 3 .

[0065] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0066] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0067] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0068] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0069] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, read-only memory), random access memories (RAM, random access memory), magnetic disks, or optical discs that can store program codes.

Claims

1. A method for detecting foreign objects, characterized in that, Including: Obtain a plurality of target detection images collected by a camera module, where the camera module is a combination of multiple cameras and multiple light sources, and the plurality of target detection images include a first detection image, a second detection image, a third detection image, and a fourth detection image; Extract corresponding multiple defect imaging features from the plurality of target detection images respectively; Analyze the multiple defect imaging features to obtain an analysis result; Correct the effective display areas of the plurality of target detection images respectively to obtain a plurality of corrected images, where the plurality of corrected images include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image; Perform an intersection operation on the defect areas of the second corrected image and the defect areas of the third corrected image to obtain an operation result; Calculate the ratio of the area of the operation result to the area of the defect area of the third corrected image, and the ratio of the area of the operation result to the area of the defect area of the second corrected image to obtain two ratio results; Compare the two ratio results with a preset threshold respectively to obtain two comparison results; Confirm the position of the foreign object according to the analysis result and the two comparison results.

2. The method according to claim 1, wherein The multiple defect imaging features include a first defect imaging feature, a second defect imaging feature, a third defect imaging feature, and a fourth defect imaging feature. The first defect imaging feature corresponds to the first detection image, the second defect imaging feature corresponds to the second detection image, the third defect imaging feature corresponds to the third detection image, and the fourth defect imaging feature corresponds to the fourth detection image. The analyzing the multiple defect imaging features to obtain an analysis result includes: When it is analyzed that the first defect imaging feature and the fourth defect imaging feature are white, and the second defect imaging feature and the third defect imaging feature are black, the analysis result indicates that the foreign object is a submembrane foreign object; When it is analyzed that the second defect imaging feature and the third defect imaging feature are black, and the first defect imaging feature or the fourth defect imaging feature is black, the analysis result indicates that the foreign object is a surface foreign object.

3. The method according to claim 1, wherein Before the correcting the effective display areas of the plurality of target detection images respectively to obtain a plurality of corrected images, it further includes: Map the third detection image and the fourth detection image into the second detection image through a preset transformation matrix; Extract the second effective display area of the second detection image according to the gray gradient method; Intercept the corresponding first effective display area, third effective display area, and fourth effective display area of the first detection image, the third detection image, and the fourth detection image respectively according to the second effective display area.

4. The method according to claim 3, characterized in that The effective display areas of the plurality of target detection images include a first effective display area, a second effective display area, a third effective display area, and a fourth effective display area. The correcting the effective display areas of the plurality of target detection images respectively to obtain a plurality of corrected images, where the plurality of corrected images include a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image includes: Obtain a circumscribed rectangle according to the contour of the second effective display area; Calculate a perspective transformation matrix according to the physical dimensions, the four corner coordinates, length, and width of the circumscribed rectangle; According to the perspective transformation matrix, correct the first effective display area, the second effective display area, the third effective display area, and the fourth effective display area respectively, and obtain corresponding first corrected images, second corrected images, third corrected images, and fourth corrected images.

5. The method according to claim 1, wherein After performing an intersection operation on the defective areas of the second corrected image and the defective areas of the third corrected image to obtain an operation result, it further includes: When a protective film is covered on the CG cover plate, perform convolution filtering on the effective display areas of the multiple target detection images respectively to obtain corresponding multiple filtering results; Extract the gray feature values of the multiple target detection images; Extract the gray feature values of the multiple filtering results.

6. The method according to claim 5, characterized in that After extracting the gray feature values of the multiple filtering results, it further includes: Extract the gray feature value of the operation result; Confirm the position of the foreign object according to the gray feature values of the multiple target detection images, the gray feature values of the multiple filtering results, and the gray feature value of the operation result.

7. The method according to any one of claims 1-6, characterized in that, The imaging module includes an upper camera, a lower camera, an upper light source, and a lower light source. Obtaining the multiple target detection images collected by the imaging module, the multiple target detection images including a first detection image, a second detection image, a third detection image, and a fourth detection image includes: Obtain a first detection image collected by the combination of the upper camera and the upper light source; Obtain a second detection image collected by the combination of the upper camera and the lower light source; Obtain a third detection image collected by the combination of the lower camera and the upper light source; Obtain a fourth detection image collected by the combination of the lower camera and the lower light source.

8. A system for detecting foreign objects, characterized in that, It includes: An acquisition unit, configured to acquire multiple target detection images collected by an imaging module, the imaging module being a combination of multiple cameras and multiple light sources, and the multiple target detection images including a first detection image, a second detection image, a third detection image, and a fourth detection image; A defective imaging feature unit, configured to extract corresponding multiple defective imaging features from the multiple target detection images respectively; An analysis unit, configured to analyze the multiple defective imaging features to obtain an analysis result; A correction unit, configured to correct the effective display areas of the multiple target detection images respectively to obtain multiple corrected images, the multiple corrected images including a first corrected image, a second corrected image, a third corrected image, and a fourth corrected image; An intersection operation unit, configured to perform an intersection operation on the defective areas of the second corrected image and the defective areas of the third corrected image to obtain an operation result; A ratio unit, configured to calculate the ratio of the area of the operation result to the area of the defective area of the third corrected image, and the ratio of the area of the operation result to the area of the defective area of the second corrected image, to obtain two ratio results; A comparison unit, configured to compare the two ratio results with a preset threshold respectively to obtain two comparison results; A first confirmation unit, configured to confirm the position of the foreign object according to the analysis result and the two comparison results.

9. An electronic device, characterized in that, Comprising: a processor, a memory, an input / output unit, and a bus; the processor is connected to the memory, the input / output unit, and the bus; the memory stores a program, and the processor calls the program to execute the method for detecting a foreign object according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A program is stored on the computer-readable storage medium, and when the program is executed on a computer, it executes the method for detecting a foreign object according to any one of claims 1 to 7.

Images