Screen hole area defect detection method, system and device and storage medium

Abstract

The invention relates to the technical field of screen detection, and provides a screen hole area defect detection method, system and device and a storage medium, the invention provides a partition detection method for the screen hole area defect, and in the method, according to the position of a defect existing in a screen and the position of at least one area corresponding to a screen hole area, the screen hole area defect can be detected according to the position of the defect existing in the screen and the position of at least one area corresponding to the screen hole area. According to the method, the specific areas where the defects of the screen fall into the screen hole area are determined, and the types of the defects falling into the areas are compared with the types of the preset bad defects corresponding to the areas, so that whether the defects falling into the areas are bad defects or not is determined, and the accuracy of defect detection of the screen hole area is improved.

Description

Technical Field

[0001] The present application relates to the field of screen detection technology, and in particular to a method, system, device and storage medium for detecting defects in a screen hole area. Background Art

[0002] With the popularity of electronic devices, people's demand for a better user experience is also increasing. In pursuit of a better user experience, a hole is usually dug in the screen of an electronic device, and the camera is placed below the hole in the screen. Because the hole in the screen is small, the screen-to-body ratio is high, which does not affect the user experience.

[0003] However, during the production of electronic devices, defects may develop in the screen hole area due to process or environmental factors, leading to screen failure. Therefore, defect detection in the screen hole area is essential. Testing for defects in the screen hole area before shipment can improve the yield rate of electronic device screens, thereby generating direct economic benefits. However, current methods for detecting defects in the screen hole area are not very accurate. Summary of the Invention

[0004] The present application provides a screen hole area defect detection method, system, device and storage medium, which can improve the accuracy of screen hole area defect detection.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a method for detecting defects in a screen hole area is provided, wherein the screen includes a hole area and a support plate covering area outside the hole area, the hole area includes a device light-transmitting area, an ink covering area located outside the device light-transmitting area, and a screen routing area located outside the ink covering area; the method for detecting defects in a screen hole area includes: first, obtaining an image to be tested of the screen, the image to be tested including the device light-transmitting area, the ink covering area, the screen routing area, and part of the support plate covering area; secondly, determining the position and type of defects existing on the screen, as well as the position of at least one area among the device light-transmitting area, the ink covering area, the screen routing area, and the support plate covering area according to the image to be tested; finally, determining whether the defect existing on the screen is a bad defect based on the position and type of the defect existing on the screen, the position of at least one area, and the correspondence between at least one area and a preset bad defect.

[0007] The embodiment of the present application provides a partition detection method for screen hole area defects, in which, according to the position of the defect existing in the screen and the position of at least one area corresponding to the screen hole area, the specific area of the screen hole area where the defect existing in the screen falls is determined, the type of the defect falling into each area is compared with the type of the preset defective defect corresponding to each area, and then whether the defect falling into each area is a defective defect is determined, so that the accuracy of screen hole area defect detection is improved.

[0008] In a possible implementation manner of the first aspect, the at least one area includes the device light transmission area and the screen trace area; and the determining, according to the position and type of the defect existing in the screen, the position of the at least one area, and the correspondence between the at least one area and the preset defective defect, whether the defect at the defect position is a defective defect includes: in a case where the position of the defect existing in the screen falls into the device light transmission area and the type of the defect is the same as the type of the preset defective defect corresponding to the device light transmission area, determining that the defect is a defective defect; and in a case where the position of the defect existing in the screen falls into the screen trace area and the type of the defect is the same as the type of the preset defective defect corresponding to the screen trace area, determining that the defect is a defective defect.

[0009] In a second aspect, a screen hole area defect detection system is provided, the screen includes a hole area and a support plate coverage area in the periphery of the hole area, the hole area includes a device light transmission area, an ink coverage area in the periphery of the device light transmission area, and a screen trace area in the periphery of the ink coverage area; the system includes an image acquisition device and a processing device connected with the image acquisition device; the image acquisition device is used to acquire a to-be-tested image of the screen, the to-be-tested image includes the device light transmission area, the ink coverage area, the screen trace area, and part of the support plate coverage area; the processing device is used to determine the position and type of the defect existing in the screen, and the position of at least one area of the device light transmission area, the ink coverage area, the screen trace area and the support plate coverage area; and is further used to determine whether the defect existing in the screen is a defective defect according to the position and type of the defect existing in the screen, the position of the at least one area, and the correspondence between the at least one area and the preset defective defect.

[0010] In a third aspect, a screen hole area defect detection device is provided, the device includes a memory and a processor, the memory is used to store instructions, when the instructions are executed by the processor, the screen hole area defect detection device executes the method in the first aspect or any possible implementation manner in the first aspect.

[0011] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed, the method in the first aspect or any possible implementation manner of the first aspect is implemented.

[0012] In a fifth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0013] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is an exemplary flow chart of the first aspect of the screen defect detection method provided by an embodiment of the present application;

[0015] Figure 2 This is a schematic diagram of the back of the screen provided in an embodiment of the present application;

[0016] Figure 3 is a schematic diagram of an image to be tested provided in an embodiment of the present application;

[0017] Figure 4 Schematic diagram of the feature extraction layer, feature fusion layer, and output layer of the YOLOv5 network model provided in an embodiment of the present application;

[0018] Figure 5 This is a schematic diagram of an output result after inputting the image to be tested into the defect classification model provided by an embodiment of the present application;

[0019] Figure 6 1 is a schematic structural diagram of a screen defect detection system according to a second aspect of an embodiment of the present application;

[0020] Figure 7 The embodiments of this application provide Figure 6 Schematic diagram of some structures in ;

[0021] Figure 8 This embodiment of the present application provides Figure 7 Exploded view of the structure shown;

[0022] Figure 9 is an exploded view of the clamp provided in an embodiment of the present application;

[0023] Figure 10 This is a schematic diagram of the program interface of the screen defect detection system provided in an embodiment of the present application;

[0024] Figure 11 It is a structural diagram of the screen defect detection device according to the third aspect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solution in this application will be described below with reference to the accompanying drawings.

[0026] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in this article is merely a way to describe the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0027] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0028] With the popularity of electronic devices, people's demand for a better user experience is also increasing. In pursuit of a better user experience, a hole is usually dug in the screen of an electronic device, and the camera is placed below the hole in the screen. Because the hole in the screen is small, the screen-to-body ratio is high, which does not affect the user experience.

[0029] However, during the production of electronic devices, due to factors such as process or environment, defects may occur in the screen hole area, resulting in poor screen performance. For example, cracks may appear on the sidewalls of the screen hole area during the punching process. During use, moisture can easily enter the display layer of the screen, causing the display device to fail, resulting in black spots on the screen during display, and causing poor display problems. For another example, during the production process, dirt or lint may appear on the screen cover. If the dirt or lint is located in the light-transmitting area of ​​the screen, it will affect the camera's shooting performance.

[0030] Therefore, defect detection in the screen hole area is essential. Before shipment, defect detection in the screen hole area can improve the yield rate of electronic device screens, thereby bringing direct economic benefits. However, the accuracy of current screen hole area defect detection is not high.

[0031] In view of this, an embodiment of the present application provides a method for detecting defects in a screen hole area, wherein the screen includes a hole area and a support plate covering area outside the hole area, the hole area includes a device light-transmitting area, an ink covering area located outside the device light-transmitting area, and a screen routing area located outside the ink covering area; the method for detecting defects in a screen hole area includes: first, obtaining an image to be tested of the screen, the image to be tested including the device light-transmitting area, the ink covering area, the screen routing area, and part of the support plate covering area; secondly, determining the position and type of defects existing on the screen, as well as the position of at least one area among the device light-transmitting area, the ink covering area, the screen routing area, and the support plate covering area according to the image to be tested; finally, determining whether the defect existing on the screen is a bad defect based on the position and type of the defect existing on the screen, the position of at least one area, and the correspondence between at least one area and a preset bad defect.

[0032] An embodiment of the present application provides a method for zoning detection of defects in the screen hole area. In this method, the specific area of ​​the screen hole area where the defects on the screen fall can be determined based on the location of the defects on the screen and the location of at least one area corresponding to the screen hole area, and the type of defects falling into each area can be compared with the type of preset bad defects corresponding to each area to determine whether the defects falling into each area are bad defects, thereby improving the accuracy of screen hole area defect detection.

[0033] The following describes in detail a screen hole area defect detection method provided by an embodiment of the present application with reference to the accompanying drawings. Figure 1 The figure is an exemplary flow chart of a method for detecting defects in a screen hole area.

[0034] Step 101: Acquire an image to be tested of the screen hole area, where the image to be tested includes the device light-transmitting area, the ink-covered area, the screen trace area, and a portion of the support plate covered area of ​​the screen hole area.

[0035] The screen of an electronic device consists of at least a protective cover, a display layer, and a support plate stacked from top to bottom. The protective cover is the outermost component of the screen, usually made of tempered glass or plastic, and is used to protect the internal display layer from scratches, impacts, and dust. The display layer is the core part of the screen, responsible for generating and displaying images. The display layer can be a liquid crystal display (LCD), organic light-emitting diode (OLED), or other types of display technology. The support plate provides a solid backplane for the screen, enhancing its overall strength and stability.

[0036] In some cases, the screen of an electronic device may also include a touch layer. The touch layer (also called a touch screen sensor layer) is typically located above the display layer because it requires direct contact with the user to detect touches and gestures. In some cases, the touch layer can also be integrated into the display layer to form in-cell touch technology.

[0037] It is worth noting that the screen of the electronic device may also include other film layer structures, which are not listed one by one in this embodiment.

[0038] In order to increase the screen-to-body ratio, holes are currently drilled in the film layers (such as the display layer and support plate) under the protective cover of the screen, and the camera is placed under the through-hole of the screen. At the same time, in order to avoid light interference and improve the overall aesthetics of the screen, ink is applied to the side of the protective cover close to the display layer and facing the through-hole to form an ink-covered area. The ink-covered area is ring-shaped, and the light-transmitting area within the ring is also called the device light-transmitting area. External light enters the camera under the screen from the device light-transmitting area.

[0039] In order to improve the alignment accuracy between the device's light-transmitting area and the under-screen camera and reduce light leakage, the diameter of the through-hole formed on the film layer below the protective cover (such as the display layer and the support plate) is often larger than the diameter of the device's light-transmitting area, and the diameter of the through-hole on the support plate is larger than the diameter of the through-hole on the display layer. Therefore, the screen diagram seen from the support plate side is as follows Figure 2 As shown, the screen 100 includes a hole area 11 and a support plate covering area 12 outside the hole area 11. Figure 3 The hole area 11 includes a device light-transmitting area 111, an ink-covered area 112 located outside the device light-transmitting area 111, a screen wiring area 113 located outside the ink-covered area 112, and a support plate covering area 12 located outside the ink-covered area 112. The device light-transmitting area 111 and the ink-covered area 112 are the portions of the protective cover plate as seen from the support plate side, the screen wiring area 113 is the portion of the display layer as seen from the support plate side, and the support plate covering area 12 is the portion of the support plate as seen from the support plate side.

[0040] The device light-transmitting area 111 is circular, and the ink-covered area 112 is annular, with the inner diameter of the ink-covered area 112 being the same as the diameter of the device light-transmitting area 111. The screen trace area 113 is annular, with the inner diameter of the screen trace area 113 being the same as the outer diameter of the ink-covered area 112.

[0041] In this embodiment, the screen hole area is inspected for defects. First, the image of the screen hole area to be inspected needs to be obtained, such as Figure 3 As shown, the image to be tested needs to include the hole area 11 of the screen, that is, the device light-transmitting area 111 of the screen, the ink-covered area 112 and the screen wiring area 113 , as well as part of the support plate covered area 12 .

[0042] It is worth noting that when performing defect detection on the screen hole area in this embodiment, the image to be tested is a partial image of the back of the screen taken from the support plate side of the screen, not an image of the front of the screen taken from the protective cover side.

[0043] Step 102: Determine the location and type of defects on the screen and the location of at least one of the device light-transmitting area, ink-covered area, screen trace area, and support plate covered area based on the image to be tested.

[0044] First, how to determine the location and type of defects on the screen based on the image to be tested is described.

[0045] For example, the image to be tested may be input into a pre-trained defect classification model to obtain the location and type of defects existing on the screen in the image to be tested.

[0046] Among them, the pre-trained defect classification model is obtained in the following way: collecting sample images, the sample images include defect-free sample images and defective sample images, and the defective sample images include: sample defect frames located in the sample images, and sample defect types corresponding to the sample defect frames; using a part of the sample images to train the neural network model until the neural network model converges; using another part of the sample images that did not participate in the training to evaluate the performance of the converged neural network model; according to the evaluation results of the performance of the converged neural network model, optimizing the converged neural network model until the performance of the neural network model meets the performance threshold.

[0047] Specifically, a camera can be used to capture sample images of the screen hole area, including both defect-free and defective images. In this embodiment, the defects include cracks, dirt, and lint. The sample images are annotated according to four categories: 1: crack, 2: dirt, 3: lint, and 4: no defects, to obtain a sample defect dataset. When annotating defects of categories 1, 2, and 3, the locations of the defects must be framed in the sample image to form sample defect frames. Each sample defect frame corresponds to a sample defect type. For example, if the defect in the sample defect frame is lint, the corresponding sample defect type is 3.

[0048] The sample defect dataset is divided into a training set, a validation set, and a test set. Sample images from the training set are fed into the neural network model to be trained until the model converges. The validation set is then used to evaluate the performance of the converged neural network model. Based on the performance evaluation results, the converged neural network model is optimized until its performance meets the performance threshold. Finally, the trained neural network model is tested on the test set to verify its performance on unknown data.

[0049] In this embodiment, the neural network model to be trained adopts the YOLOv5 network model. The YOLOv5 network model includes an input layer, a feature extraction layer, a feature fusion layer, and an output layer. Figure 4 The feature extraction layer is also called the Backbone network layer, the feature fusion layer is also called the Neck network, and the output layer is also called the Head network.

[0050] The above-mentioned method of using a part of the sample images to train the neural network model to be trained until the neural network model converges includes: inputting a part of the sample images into the input layer to preprocess the sample images; inputting the preprocessed sample images into the feature extraction layer to extract the features in each sample image to obtain multiple feature maps of different resolutions corresponding to each sample image; inputting the multiple feature maps of different resolutions corresponding to each sample image into the feature fusion layer for feature fusion to obtain multiple fused feature maps of different resolutions corresponding to each sample image; inputting the multiple fused feature maps of different resolutions corresponding to each sample image into the output layer to obtain the predicted box position corresponding to each sample image, the predicted defect type corresponding to the predicted box position and the confidence; determining whether the neural network model converges according to the predicted box position of each sample image, the predicted defect type corresponding to the predicted box position, the position of the sample defect box located in the sample image, and the sample defect type corresponding to the sample defect box; if the neural network model does not converge, returning to the step of inputting a part of the sample images into the input layer to preprocess the sample images until the neural network model converges.

[0051] First, after the sample images of the training set are input into the input layer of the model, the sample images are preprocessed at the input layer. The preprocessing includes: (1) data augmentation, which can be used to stitch together multiple sample images by at least one of random scaling, random cropping, and random arrangement to obtain a stitched image. For example, after stitching together four sample images by at least one of random scaling, random cropping, and random arrangement, eight or even ten stitched images can be obtained, thereby enriching the sample images of the training set. (2) adaptive anchor frame, which can be adaptively framed according to the width and height of the sample defect frame in the stitched image. Specifically, first, the width and height information of all sample defect frames in the stitched image are collected, and then these widths and heights are clustered using a clustering algorithm to obtain an initial frame. The purpose of the adaptive anchor frame is to predict the bounding box of the defect in the sample image. If the size of the anchor frame is close to the actual size of the defect, it is easier for the model to learn how to accurately predict the bounding box. (3) Adaptive image scaling: scaling the resolution of the framed image to the preset resolution, which is the same as the resolution of the image that can be input to the feature extraction layer. For example, if the preset resolution is 640*640 and the resolution of the sample image is 1270*960, the resolution of the sample image needs to be scaled from 1270*960 to 640*640 before being input to the feature extraction layer.

[0052] The feature extraction layer (i.e., the backbone network layer) extracts features from the input image and obtains multiple feature maps of different resolutions corresponding to each image. Figure 4 As shown in the figure, for each input image, feature extraction produces three feature maps of different resolutions. These feature maps capture features of varying scales, helping to detect defects of varying sizes. For example, there are three feature maps with resolutions: 256*80*80, 512*40*40, and 1024*20*20. The resolution of 256*80*80 indicates a feature map with an 80*80 resolution, consisting of 80*80 pixels, with each pixel containing 256 features; the resolution of 512*40*40 indicates a feature map with a 40*40 resolution, consisting of 40*40 pixels, with each pixel containing 512 features; and the resolution of 1024*20*20 indicates a feature map with a 20*20 resolution, consisting of 20*20 pixels, with each pixel containing 1024 features. The feature information extracted by the feature extraction layer can include edges, corners, textures, color distribution, shapes, and other information.

[0053] The feature maps of different resolutions corresponding to each sample image are input into the feature fusion layer (i.e., Neck network) for feature fusion, and the fused feature maps of different resolutions corresponding to each sample image are obtained. For example, the feature maps of different resolutions can be fused, and then the fused feature maps can be fused with feature maps of other resolutions. Figure 4 As shown in the figure, after fusing the feature maps of three resolutions of 256*80*80, 512*40*40, and 1024*20*20, the resulting feature fusion maps are also 256*80*80, 512*40*40, and 1024*20*20.

[0054] The fused feature maps of multiple different resolutions corresponding to each sample image are input into the output layer (i.e., the Head network) to obtain the predicted box position, the predicted defect type and the confidence level of each sample image. Figure 4 As shown in the figure, each fused feature map passes through a set of convolutional layers to generate a series of prediction vectors. Each prediction vector contains the location (x, y, w, h), category and confidence of the target bounding box. Among them, the location of the prediction vector on the feature map corresponds to the spatial location in the original image. For example, a point on the fused feature map may correspond to a specific area in the original image. Since each defect in the image may be detected at multiple locations, forming multiple prediction boxes, non-maximum suppression is required to filter out the most likely prediction box. Non-maximum suppression retains the prediction box with the highest confidence and suppresses other prediction boxes that have a high overlap with it. Finally, the prediction box processed by non-maximum suppression will be output, and each prediction box will be accompanied by a category label and confidence score.

[0055] Next, the neural network model is determined to have converged based on the predicted box position of each sample image, the predicted defect type corresponding to the predicted box position, and the position of the sample defect box in the sample image and the sample defect type corresponding to the sample defect box. If the neural network model has not converged, the process returns to the step of feeding a portion of the sample image into the input layer and preprocessing the sample image until the neural network model converges.

[0056] Specifically, the loss function can be calculated to determine whether the neural network model has converged.

[0057] Loss function G loss The following formula (1) is used to calculate:

[0058]

[0059] Wherein, A represents the position of the sample defect frame, B represents the predicted frame position, and C represents the position of the minimum circumscribed rectangle of the sample defect frame and the predicted frame, which contains the area of the sample defect frame and the predicted frame. The above A, B and C are all for the same defect.

[0060] During the input of the sample image, the above steps are repeatedly performed, and the loss function is calculated. If the loss function value tends to be stable and no longer decreases significantly as the sample image is input, it can be determined that the neural network model converges.

[0061] After that, the performance of the converged neural network model is evaluated using the validation set, and the converged neural network model is optimized according to the evaluation result of the performance of the converged neural network model until the performance of the neural network model meets the performance threshold. For example, the sample images of the validation set can be input into the trained model, and whether the performance parameters such as the average precision, accuracy or recall rate of the output result meet the performance threshold, for example, the average precision needs to be more than 90%. If the performance parameters calculated according to the output result do not meet the performance threshold, the weight parameters of the neural network model are optimized, and the sample images are input again for verification until the performance of the neural network model meets the performance threshold. At this time, the model training is completed.

[0062] Finally, the images in the prediction set can also be input into the trained model to obtain the position, class and confidence of the predicted frame in each image.

[0063] The process of model training will be described below with an actual example.

[0064] Suppose the YOLOv5 model is used to detect cracks on the hole area of a mobile phone screen.

[0065] (1) First, data preparation

[0066] Data collection: 1000 pictures of mobile phone screen hole area are taken by camera, which contains various sizes of cracks, dirt and lint.

[0067] Data labeling: use labeling tools to label cracks, dirt, lint in the pictures, and mark the pictures without defects.

[0068] Data set division: 700 pictures as training set, 200 pictures as validation set, 100 pictures as test set.

[0069] (2) Model training

[0070] Initialization: load the pre-trained weights of YOLOv5 to provide a good starting point for model training.

[0071] Iterative training: Cycle 1: The model makes random predictions, and the loss is high, for example, 10.0 on the training set and 9.5 on the validation set. Cycle 10: The model begins to learn some features, and the loss decreases, dropping to 3.0 on the training set and 3.5 on the validation set. Cycle 50: The model becomes more accurate, with the loss dropping to 0.8 on the training set and 1.0 on the validation set.

[0072] Feature extraction and fusion: In each training cycle, the model extracts features through the Backbone network and performs feature fusion through the Neck network to generate a fused feature map for prediction.

[0073] Prediction output and loss calculation: The head network outputs the predicted box and calculates the gap between the predicted box and the true box through the loss function, continuously adjusting the weight to reduce the loss.

[0074] (3) Verification process

[0075] Validation set evaluation: After each training cycle, the performance of the model is evaluated using the validation set. For example, in the 50th training cycle, the average precision on the validation set is calculated to be 90%.

[0076] (4) Testing process

[0077] Using the optimal model weights from the 50th training epoch, perform defect detection on 100 test set images. Record the model output, including the location, category, and confidence level of the predicted box. Example results: Crack detection: The model successfully identified cracks in 95 images with 95% accuracy. Dirt detection: The model identified dirt in 90 images with 90% accuracy. Hair detection: The model identified hair in 85 images with 85% accuracy. Defect-free images: The model correctly identified all images as free of defects.

[0078] Through this process, not only was a YOLOv5 model capable of detecting screen hole defects trained, but the generalization ability and accuracy of the model were also ensured through the verification and testing process.

[0079] Secondly, it is described how to determine the position of at least one area among the device light-transmitting area, the ink-covered area, the screen wiring area and the support plate covered area according to the image to be tested.

[0080] Exemplarily, at least one area includes: a device light-transmitting area and a screen routing area; determining the position of at least one area among the device light-transmitting area, the ink-covered area, the screen routing area and the support plate coverage area according to the image to be tested, including: determining the second radius of the device light-transmitting area, the second radius of the screen routing area, and the second radius of the ink-covered area according to the first radius of the device light-transmitting area, the first radius of the ink-covered area, and the preset camera parameters, wherein the first radius is the radius of each area in the screen, and the second radius is the radius of each area in the image to be tested; determining the position of the device light-transmitting area in the image to be tested according to the second radius of the device light-transmitting area; determining the position of the screen routing area in the image to be tested according to the second radius of the screen routing area and the second radius of the ink-covered area.

[0081] Specifically, see Figure 3 The preset camera parameters include the actual physical size represented by each pixel. Dividing the radius of the ink-covered area 112 on the screen by the actual physical size represented by each pixel yields the number of pixels occupied by the radius of the ink-covered area 112 in the image to be measured, thereby obtaining the radius of the ink-covered area 112 in the image to be measured. For example, if the screen area captured by the camera is 12.7mm*12.7mm, the resolution is 1270*1270, and the magnification is 2x, then the actual physical size represented by each pixel is 0.005mm. If the radius of the device's light-transmitting area 111 is 2.72mm, then the number of pixels occupied by the radius of the device's light-transmitting area 111 is 544.

[0082] In this way, edge coordinate detection is performed in the image to be tested using an edge grayscale algorithm to obtain edge coordinates. Subsequently, the center (X, Y) and the position of the circle are found using the Hough circle detection center origin algorithm in combination with the second radius of the device light-transmitting area 111. The detected circle is marked in the image to be tested. This circle is the edge position of the device light-transmitting area 111. Thus, the position of the device light-transmitting area 111 in the image to be tested can be determined based on the edge position of the device light-transmitting area 111. Similarly, the edge position of the screen trace area 113 can be found in the image to be tested based on the second radius of the screen trace area 113. Thus, the position of the screen trace area 113 in the image to be tested can be determined based on the edge position of the device light-transmitting area 111 and the edge position of the screen trace area 113.

[0083] It is worth noting that the above examples are described by taking the determination of the device light-transmitting area 111 and the screen wiring area 113 as examples. The determination methods of other areas can refer to the above examples and are not described one by one in this embodiment.

[0084] Step 103: Determine whether the defect at the defect location is a bad defect based on the location and type of the defect on the screen, the location of at least one area, and the correspondence between the at least one area and a preset bad defect.

[0085] In actual testing, due to different requirements for defects in different areas. Figure 3 For example, any defects such as cracks, lint, or dirt in the device's light-transmitting area 111 will affect the screen's light transmission, which in turn affects the camera's performance below the screen. Therefore, cracks, lint, or dirt falling into the device's light-transmitting area 111 are considered defects. Cracks in the ink-covered area 112 can cause light to enter the screen below through the cracks in the ink-covered area 112, thereby interfering with camera performance. However, lint or dirt will not affect camera performance, so cracks in the ink-covered area 112 are considered defects. Cracks in the screen trace area 113 are believed to be caused by a break in the inorganic material / metal traces within the screen's aperture area, thus affecting the display of the screen 100. However, lint or dirt will not affect the display. Therefore, cracks in the screen trace area 113 are considered defects. As for the support plate covering area 12 , since the support plate only plays a supporting role, defects such as cracks, lint or dirt in the support plate covering area 12 will not affect the function of the support plate and are not considered as bad defects.

[0086] Therefore, an embodiment of the present application provides a method for zoning detection of defects in the screen hole area. In this method, the specific area of ​​the screen hole area where the defects on the screen fall can be determined based on the location of the defects on the screen and the location of at least one area corresponding to the screen hole area, and the type of defects falling into each area can be compared with the type of preset bad defects corresponding to each area to determine whether the defects falling into each area are bad defects, thereby improving the accuracy of screen hole area defect detection.

[0087] Exemplarily, at least one area includes: a device light-transmitting area and a screen wiring area; based on the location and type of the defect on the screen, the location of at least one area, and the correspondence between at least one area and a preset bad defect, determining whether the defect at the defect location is a bad defect includes: when the location of the defect on the screen falls into the device light-transmitting area, and the type of the defect is the same as the type of the preset bad defect corresponding to the device light-transmitting area, determining the defect to be a bad defect; when the location of the defect on the screen falls into the screen wiring area, and the type of the defect is the same as the type of the preset bad defect corresponding to the screen wiring area, determining the defect to be a bad defect.

[0088] Assume that the image to be tested is input into the pre-trained defect classification model, and the following is obtained: Figure 5 The results shown show that the screen has Type 1 and Type 3 defects. Type 1 is a crack, appearing in screen trace area 113; Type 3 is a hair, appearing in device light-transmitting area 111. If the default defect corresponding to screen trace area 113 is a crack, then Type 1 is considered a bad defect. If the default defect corresponding to device light-transmitting area 111 is a crack, hair, or dirt, then Type 3 is considered a bad defect because hair appears in device light-transmitting area 111.

[0089] The above examples are merely illustrative. Specific preset defects corresponding to each area can be set as needed and are not listed one by one in this embodiment. Furthermore, the ink-covered area 112 and the support plate-covered area 12 can be determined in the same manner to determine whether a defect falling within the area is a defect.

[0090] The present application also provides a screen hole area defect detection system, hereinafter referred to as the detection system. Figure 6 As shown, it is used to implement the screen hole area defect detection method in the embodiment of this application. Figure 2 and Figure 3 The screen 100 under inspection includes a hole area 11 and a support plate covering area 12 outside the hole area 11. The hole area 11 includes a device light-transmitting area 111, an ink covering area 112 located outside the device light-transmitting area 111, and a screen routing area 113 located outside the ink covering area 112.

[0091] like Figure 6 and Figure 7 As shown, the detection system includes: an image acquisition device 20 and a processing device 21 connected to the image acquisition device 20.

[0092] The image acquisition device 20 is used to obtain the image to be tested of the screen. Figure 3 The image to be tested includes the device light-transmitting area 111, the ink-covered area 112, the screen wiring area 113, and a portion of the support plate covered area 12.

[0093] The processing device 21 is used to determine the location and type of defects existing on the screen, as well as the location of at least one area among the device light-transmitting area 111, the ink-covered area 112, the screen wiring area 113 and the support plate covered area 12 based on the image to be tested; and is also used to determine whether the defect existing on the screen is a bad defect based on the location and type of defects existing on the screen, the location of at least one area, and the correspondence between at least one area and a preset bad defect.

[0094] In the embodiment of the present application, the processing device 21 can determine which area of ​​the screen hole area the defect on the screen falls into based on the location of the defect on the screen and the location of at least one area, thereby performing partition detection on each area, and comparing the type of defects falling into each area with the type of preset defect corresponding to each area, so as to determine whether the defects falling into each area are defective defects, thereby improving the accuracy of defect detection in the screen hole area.

[0095] In one example, the system further includes a displacement device 22 , on which the image acquisition device 20 is disposed; the displacement device 22 can move in the horizontal direction and the vertical direction, and drive the image acquisition device 20 to move.

[0096] For example, see Figure 7 and Figure 8 The displacement device 22 includes a horizontal motion shaft 222 that can move horizontally, and a vertical motion shaft 221 that is disposed on the horizontal motion shaft 222 and can move vertically. The image acquisition device 20 is disposed on the vertical motion shaft 221. The system also includes a support plate 23 on which the displacement device 22 is mounted.

[0097] In one example, see Figure 7 and Figure 8 The system further includes: a fixing frame 24 provided on the displacement device 22 , and the image acquisition device 20 is provided on the fixing frame 24 .

[0098] In one example, see Figure 7 and Figure 8 Image acquisition device 20 includes a camera 201 and a lens 202 located on the light incident surface of camera 201. Camera 201 is used for image acquisition and can be an electrically coupled camera. Other cameras 201 can also be used as needed, with a resolution of 20 megapixels or greater. Lens 202 collects light reflected from the object being illuminated and focuses it onto camera 201. Parameters for lens 202 include: magnification: 2, working distance: 65 mm, depth of field: 0.35 mm, resolution: 4.5 μm, telecentricity: 0.11°, and optical distortion: 0.041.

[0099] When performing screen hole area detection, the screen is placed under the image acquisition device 20, and the horizontal position of the image acquisition device 20 is changed by adjusting the horizontal motion axis 222, and the vertical height of the image acquisition device 20 is changed by adjusting the up and down motion axis 221.

[0100] In one example, see Figure 7 and Figure 8The detection system further comprises a light source 25, which is arranged on the up-down movement shaft 221 and is driven to move by the up-down movement shaft 221. The light source 25 is located below the image acquisition device 20 and provides light source for the to-be-tested screen. In this embodiment, a ring-shaped white light source can be used, and the brightness and position of the light source can be adjusted according to requirements.

[0101] In one example, referring to Figure 6 The detection system further comprises a movable stage 27, which provides support for the to-be-tested screen and can be moved in a horizontal direction according to a command to ensure that the position of the to-be-tested sample meets the test requirements.

[0102] In one example, referring to Figure 6 The detection system further comprises a housing 28. The processing device 21 is arranged on the housing 28 and can be a computer or the like and can comprise a display screen for displaying test environment and test parameters and the like. The image acquisition device 20 and the displacement device 22 can be arranged inside the housing 28.

[0103] It is worth noting that, referring to Figure 9 In order to facilitate the detection of the screen 100, the screen 100 can be clamped by the clamp 26 for monitoring. Specifically, the clamp 26 comprises an upper clamp 261 and a lower clamp 262, which clamp the screen 100 therebetween. The middle region of the upper clamp 261 and the lower clamp 262 is hollow, and the hole region of the screen 100 can be observed through the hollow region, so that the image acquisition device 20 can shoot the hole region of the screen.

[0104] In one example, the processing device 21 is configured to determine that the defect is a defective defect in a case where a position of the defect existing on the screen falls into the device light transmission region, and a type of the defect is same as a type of a preset defective defect corresponding to the device light transmission region; and determine that the defect is a defective defect in a case where the position of the defect existing on the screen falls into the screen wiring region, and the type of the defect is same as a type of a preset defective defect corresponding to the screen wiring region.

[0105] In one example, the processing device 21 is configured to determine a second radius of the device light transmission region, a second radius of the screen wiring region, and a second radius of the ink coverage region according to a first radius of the device light transmission region, a first radius of the ink coverage region, a first radius of the screen wiring region, and a preset camera parameter, wherein the first radius is a radius of each region in the screen, and the second radius is a radius of each region in the to-be-tested image; determine a position of the device light transmission region in the to-be-tested image according to the second radius of the device light transmission region; and determine a position of the screen wiring region in the to-be-tested image according to the second radius of the screen wiring region and the second radius of the ink coverage region.

[0106] In one example, the processing device 21 is configured to input the to-be-tested image into the pre-trained defect classification model to obtain the position and type of the defect existing on the screen in the to-be-tested image.

[0107] In one example, the processing device 21 is configured to collect sample images, the sample images including sample images without defects and sample images with defects, the sample images with defects including sample defect frames located in the sample images and sample defect types corresponding to the sample defect frames; train the neural network model to be trained using a part of the sample images until the neural network model converges; evaluate the performance of the converged neural network model using another part of the sample images not involved in the training; and optimize the converged neural network model according to the evaluation result of the performance of the converged neural network model until the performance of the neural network model meets a performance threshold.

[0108] In one example, the neural network model to be trained includes an input layer, a feature extraction layer, a feature fusion layer and an output layer; the processing device 21 is configured to input a part of the sample images into the input layer to pre-process the sample images; input the pre-processed sample images into the feature extraction layer to extract features in each sample image to obtain a plurality of feature maps of different resolutions corresponding to each sample image; input the plurality of feature maps of different resolutions corresponding to each sample image into the feature fusion layer to perform feature fusion to obtain a plurality of fused feature maps of different resolutions corresponding to each sample image; input the plurality of fused feature maps of different resolutions corresponding to each sample image into the output layer to obtain a predicted frame position, a predicted defect type corresponding to the predicted frame position and a confidence degree corresponding to each sample image; and determine whether the neural network model converges according to the predicted frame position of each sample image, the predicted defect type corresponding to the predicted frame position, the position of the sample defect frame located in the sample image and the sample defect type corresponding to the sample defect frame; if the neural network model does not converge, return to perform the step of inputting a part of the sample images into the input layer to pre-process the sample images until the neural network model converges.

[0109] In one example, the processing device 21 is configured to splice a plurality of sample images together by at least one of random scaling, random cropping and random arrangement to obtain a spliced image; perform adaptive frame drawing according to the width and height of the sample defect frame in the spliced image; and scale the resolution of the frame-drawn image to a preset resolution, the preset resolution being the same as the resolution of the image that can be input into the feature extraction layer.

[0110] The detection process of the detection system is described below, part of the details are not shown, and the steps can be modified, added or reduced according to requirements.

[0111] S1, test preparation: including equipment startup, sample pretreatment, fixture preparation, etc.

[0112] S2, parameter adjustment: mainly includes adjusting the position of the stage to ensure the accuracy of the shooting position and the brightness of the light source, etc., which can be adjusted according to experience in manual mode, or automatic mode can be selected to automate the equipment parameters.

[0113] S3, image acquisition: The computer sends a photo-taking command to acquire one or more pictures.

[0114] S4, defect calculation: The program uses the above-mentioned screen hole area defect detection method to perform calculations based on the acquired image to obtain a series of data.

[0115] S5, result output: The computer outputs the data calculated in S4.

[0116] The program interface of the detection system is as follows Figure 10 As shown, it mainly includes the operation bar, image display, data display, project information and other contents. Among them, the operation bar is displayed on the left side of the interface, and the operation bar includes the user name, test mode, visual debugging, visual calibration, system settings and other contents. Among them, the test mode includes two test modes: online test and offline test. The image display is in the middle of the interface, which is used to display the image of the screen hole area captured by the image acquisition device. The data display is located in the lower left and lower right of the interface respectively. The lower left interface shows the total amount of test, output and number of defects. The lower right interface shows the type, quantity and size of defects in the current image to be tested, as well as the proportion of defective defects in historical tests. Project information is displayed in the upper right of the interface.

[0117] Figure 11 3 is a schematic structural block diagram of a screen defect detection device 300 provided in an embodiment of the present application. The screen 100 defect detection device 300 includes: a processor 310, a memory 320, a communication interface 330, and a bus 340.

[0118] The processor 310 may be connected to a memory 320. The memory 320 may be used to store the program code and data. Therefore, the memory 320 may be a storage unit within the processor 310, an external storage unit independent of the processor 310, or a component including both a storage unit within the processor 310 and an external storage unit independent of the processor 310.

[0119] Optionally, the screen defect detection device 300 may further include a bus 340. The memory 320 and the communication interface 330 may be connected to the processor 310 via the bus 340. The bus 340 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus 340 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The fact that only one line is used does not mean that there is only one bus or one type of bus.

[0120] It should be understood that in the embodiment of the present application, the processor 310 may adopt a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Alternatively, the processor 310 may adopt one or more integrated circuits for executing relevant programs to implement the technical solutions provided in the embodiment of the present application.

[0121] The memory 320 may include a read-only memory and a random access memory, and provides instructions and data to the processor 310. A portion of the processor 310 may also include a non-volatile random access memory. For example, the processor 310 may also store information about the device type.

[0122] When the screen defect detection device is running, the processor 310 executes the computer-executable instructions in the memory 320 to use the hardware resources in the screen defect detection device to perform the operating steps of the above-mentioned screen defect detection method.

[0123] It should be understood that the screen defect detection device 300 according to the embodiment of the present application may correspond to the processing device 21 in the screen defect detection device system in the embodiment of the present application, and may correspond to the execution of the screen defect detection device 300 according to the embodiment of the present application. Figure 1 For the sake of brevity, the corresponding subjects in the shown method are not repeated here.

[0124] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed, the method provided in the embodiment of the present application is implemented.

[0125] The present application also provides a computer program product, which includes: computer program code, which, when executed on a computer, enables the computer to execute the method provided in the embodiments of the present application.

[0126] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0127] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0128] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment as needed.

[0131] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a memory (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] The above description is merely a specific implementation of the embodiments of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present application should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A method for detecting defects in a screen hole area, characterized in that: The screen includes a hole area and a support plate covering area outside the hole area, the hole area includes a device light-transmitting area, an ink covering area located outside the device light-transmitting area, and a screen routing area located outside the ink covering area; the method includes: Acquire an image to be tested of the screen, wherein the image to be tested includes the light-transmitting area of ​​the device, the ink-covered area, the screen trace area, and a portion of the support plate covered area; Determining the location and type of defects present on the screen, and the location of at least one of the device light-transmitting area, the ink-covered area, the screen trace area, and the support plate-covered area, based on the image to be tested; Whether the defect on the screen is a bad defect is determined according to the location and type of the defect on the screen, the location of the at least one area, and the correspondence between the at least one area and a preset bad defect.

2. The method according to claim 1, characterized in that The at least one area includes: the device light transmission area and the screen wiring area; The determining whether the defect at the defect location is a bad defect based on the location and type of the defect on the screen, the location of the at least one area, and the correspondence between the at least one area and a preset bad defect includes: If the position of the defect on the screen falls within the light-transmitting area of ​​the device, and the type of the defect is the same as the type of the preset defect corresponding to the light-transmitting area of ​​the device, determining that the defect is a defect; When the position of the defect existing on the screen falls within the screen wiring area, and the type of the defect is the same as the type of the preset bad defect corresponding to the screen wiring area, the defect is determined to be a bad defect.

3. The method according to claim 1 or 2, characterized in that The at least one area includes: the device light transmission area and the screen wiring area; The determining, according to the image to be tested, the position of at least one of the device light-transmitting area, the ink-covered area, the screen wiring area, and the support plate covered area comprises: Determine, based on the first radius of the device light-transmitting area, the first radius of the ink-covered area, the first radius of the screen trace area, and preset camera parameters, the second radius of the device light-transmitting area, the second radius of the screen trace area, and the second radius of the ink-covered area, wherein the first radius is the radius of each area on the screen, and the second radius is the radius of each area in the image to be measured; Determining a position of the light-transmitting area of ​​the device in the image to be measured according to a second radius of the light-transmitting area of ​​the device; The position of the screen wiring area in the image to be measured is determined according to the second radius of the screen wiring area and the second radius of the ink coverage area.

4. The method according to any one of claims 1 to 3, characterized in that The determining the position and type of the defect on the screen according to the image to be tested includes: The image to be tested is input into a pre-trained defect classification model to obtain the location and type of defects existing on the screen in the image to be tested.

5. The method according to claim 4, characterized in that The pre-trained defect classification model is obtained in the following way: Collecting sample images, wherein the sample images include a sample image without defects and a sample image with defects, wherein the sample image with defects includes: a sample defect frame located in the sample image and a sample defect type corresponding to the sample defect frame; Using a portion of the sample images to train a neural network model until the neural network model converges; Using another portion of the sample images that have not participated in the training to evaluate the performance of the neural network model after convergence; According to the evaluation result of the performance of the converged neural network model, the converged neural network model is optimized until the performance of the neural network model meets the performance threshold.

6. The method according to claim 5, characterized in that The neural network model to be trained includes: an input layer, a feature extraction layer, a feature fusion layer and an output layer; The step of training the neural network model to be trained using a portion of the sample images until the neural network model converges includes: Inputting a portion of the sample image into the input layer to preprocess the sample image; Inputting the preprocessed sample images into the feature extraction layer, extracting features from each sample image to obtain a plurality of feature maps of different resolutions corresponding to each sample image; Inputting the multiple feature maps of different resolutions corresponding to each sample image into the feature fusion layer for feature fusion, thereby obtaining the multiple fused feature maps of different resolutions corresponding to each sample image; Inputting multiple fused feature maps of different resolutions corresponding to each sample image into the output layer, obtaining a prediction box position corresponding to each sample image, a predicted defect type corresponding to the prediction box position, and a confidence level; Determining whether the neural network model has converged based on the predicted frame position of each sample image, the predicted defect type corresponding to the predicted frame position, and the position of the sample defect frame in the sample image and the sample defect type corresponding to the sample defect frame; If the neural network model has not converged, the process returns to the step of inputting a portion of the sample image into the input layer and preprocessing the sample image until the neural network model converges.

7. The method according to claim 6, characterized in that The preprocessing of the sample image comprises: splicing the plurality of sample images together by at least one of random scaling, random cropping, and random arrangement to obtain a spliced ​​image; Adaptively drawing a frame according to the width and height of the sample defect frame in the spliced ​​image; The resolution of the framed image is scaled to a preset resolution, which is the same as the resolution of the image that can be input to the feature extraction layer.

8. A screen hole area defect detection system, characterized in that: The screen includes a hole area and a support plate covering area outside the hole area. The hole area includes a device light-transmitting area, an ink covering area outside the device light-transmitting area, and a screen wiring area outside the ink covering area. The system includes: an image acquisition device and a processing device connected to the image acquisition device; The image acquisition device is used to obtain the image to be tested of the screen, wherein the image to be tested includes the light-transmitting area of ​​the device, the ink-covered area, the screen wiring area, and part of the support plate covered area; The processing device is used to determine the position and type of defects existing on the screen, and the position of at least one area among the device light-transmitting area, the ink-covered area, the screen wiring area, and the support plate covered area according to the image to be tested; It is also used to determine whether the defect on the screen is a bad defect based on the location and type of the defect on the screen, the location of the at least one area, and the correspondence between the at least one area and the preset bad defect.

9. The system according to claim 8, characterized in that The processing device is configured to determine that a defect existing on the screen is a bad defect if the position of the defect falls within the light-transmitting area of ​​the device and the type of the defect is the same as the type of the preset bad defect corresponding to the light-transmitting area of ​​the device; When the position of the defect existing on the screen falls within the screen wiring area, and the type of the defect is the same as the type of the preset bad defect corresponding to the screen wiring area, the defect is determined to be a bad defect.

10. A screen hole area defect detection device, characterized in that: The device includes a memory and a processor, wherein the memory is used to store instructions. When the instructions are executed by the processor, the screen hole area defect detection device performs the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed, the method according to any one of claims 1 to 7 is implemented.

12. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

Images