Display screen defect detection method, model training method, device, equipment and medium

By acquiring and synchronizing the images of three display screens, using feature extraction and alignment fusion modules, the problems of low detection accuracy and high time consumption in the prior art are solved, and more efficient display screen defect detection is achieved.

CN118229632BActive Publication Date: 2024-11-29厦门国科安芯科技有限公司
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410298288.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-15
Publication Date
2024-11-29
Estimated Expiration
2044-03-15

AI Technical Summary

Technical Problem

In the detection of display defects, multiple images need to be taken through different optical schemes and divided into two stages for detection, resulting in low detection accuracy, large time consumption and high algorithm complexity, and failure to fully utilize the same defect in different Feature information in the image.

Method used

Three screen images (two strabismus images and one main image) are obtained, and input them synchronously to the trained display screen defect detection model for detection. The image feature extraction module, feature alignment fusion module and detection head module are used to generate high Dimension of feature information space for defect detection.

Benefits of technology

It improves the accuracy of display defect detection, reduces error and time consumption, reduces algorithm complexity, and can fully include defect information from multiple angles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118229632B_ABST
    Figure CN118229632B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a display screen defect detection method, model training method, device, equipment and medium. By acquiring three screen images of a display screen to be detected, the three screen images include two oblique images and one main view image, and using the two oblique images and one main view image, the display screen to be detected is simultaneously subjected to defect detection to obtain a defect detection result of the display screen to be detected. Compared with the prior art, the present disclosure uses three screen images to synchronously perform defect detection on the display screen to be detected to obtain a defect detection result of the display screen to be detected, which does not need to be split into two stages, can fully include different information in the three screen images, can obtain more comprehensive and complete defect features, and can fully utilize the rich defect information from multiple angles in the three images during detection, thereby greatly reducing the error of display screen defect detection, improving the accuracy of display screen defect detection, reducing time consumption, and reducing algorithm complexity.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of display screen detection, and in particular to a display screen defect detection method, model training method, device, equipment and medium. Background Art

[0002] Screen defect detection of display screens is an important step to ensure the quality of display screens and plays an important role in the display screen production process. Screen defects include dot and line defects (such as dark dots and dark lines), Mura defects, appearance defects, etc.

[0003] In related technologies, display screen defects need to be detected by algorithms after high-definition industrial cameras capture images. Different defects often require different optical solutions to obtain clearer images. Different optical solutions have different parameters such as camera shooting angle, light source intensity and light source angle. Therefore, the detection of the same defect type or different screen defect types requires multiple images captured by different optical solutions, and then screen defect detection is performed on the multiple images through algorithms.

[0004] In the existing technical solution, multiple images are tested separately using traditional algorithms, and then the test results of each image are integrated to obtain the final defect detection result. This solution does not use the feature information of the same defect in different images. In the subsequent processing, the pixel positions of the same defect in different images need to be aligned, so that the whole process is divided into two stages, and both stages will produce errors, resulting in low detection accuracy, and increasing time consumption and algorithm complexity. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a display screen defect detection method, a model training method, an apparatus, a device and a medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a display screen defect detection method, comprising:

[0007] Acquire three screen images of the display screen to be detected, wherein the three screen images include two oblique view images and one front view image;

[0008] The defect detection of the display screen to be detected is performed synchronously by using two oblique view images and one main view image to obtain the defect detection result of the display screen to be detected.

[0009] In some embodiments, the method of synchronously performing defect detection on the display screen to be detected by using two oblique images and one main view image to obtain a defect detection result of the display screen to be detected includes:

[0010] The two oblique view images and one main view image are input into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected.

[0011] In some embodiments, after acquiring two oblique view images and one front view image of the display screen to be detected, the method further includes:

[0012] Preprocessing the two oblique view images and one main view image respectively to obtain two target oblique view images and one target main view image;

[0013] The two oblique images and one main view image are input into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected, including:

[0014] The two target oblique images and the one target main view image are input into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs a defect detection result of the display screen to be detected.

[0015] In some embodiments, the display screen defect detection model includes an image feature extraction module, a feature alignment and fusion module, and a detection head module;

[0016] The two oblique images and one main view image are input into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected, including:

[0017] Inputting two oblique view images and one main view image into the image feature extraction module to obtain feature information corresponding to the three screen images respectively;

[0018] Inputting the feature information corresponding to the three screen images respectively into the feature alignment and fusion module to obtain a high-dimensional feature information space of the screen image;

[0019] The high-dimensional feature information space is input into the detection head module to obtain the defect detection result of the display screen to be detected.

[0020] In a second aspect, the present disclosure provides a model training method, including:

[0021] Build a display defect detection model;

[0022] Acquire multiple sets of training sample data and actual defect results corresponding to each set of training sample data, wherein each set of training sample data includes two sample oblique view images and one sample front view image;

[0023] The display screen defect detection model to be trained is trained based on each set of training sample data and the actual defect results corresponding to each set of training sample data to obtain the defect detection results of each set of training sample data;

[0024] Performing loss calculation on the defect detection results of each set of training sample data based on the actual defect results corresponding to each set of training sample data, to obtain the loss calculation result of the display screen defect detection model;

[0025] Based on the loss calculation result of the display screen defect detection model, the model parameters of the display screen defect detection model are updated.

[0026] In some embodiments, after updating the model parameters of the display screen defect detection model based on the loss calculation result of the display screen defect detection model, the method further includes:

[0027] Recording the number of training cycles of the display screen defect detection model;

[0028] Determine whether the defect detection result of each set of training sample data converges to the actual defect result corresponding to each set of training sample data;

[0029] If the defect detection results of each set of training sample data have converged to the actual defect results corresponding to each set of training sample data, or the number of training cycles of the display screen defect detection model is greater than a preset number threshold, it is determined that the training of the display screen defect detection model is completed.

[0030] In a third aspect, an embodiment of the present disclosure provides a display screen defect detection device, comprising:

[0031] An acquisition module, used for acquiring three screen images of the display screen to be detected, wherein the three screen images include two oblique view images and one main view image;

[0032] The detection module is used to use two oblique images and one main view image to synchronously perform defect detection on the display screen to be detected, and obtain a defect detection result of the display screen to be detected.

[0033] In a fourth aspect, an embodiment of the present disclosure provides a model training device, including:

[0034] A building module for building a display screen defect detection model;

[0035] A sample acquisition module, used to acquire multiple sets of training sample data and actual defect results corresponding to each set of training sample data, wherein each set of training sample data includes two sample oblique view images and one sample front view image;

[0036] An obtaining module is used to train the display screen defect detection model to be trained based on each set of training sample data and the actual defect results corresponding to each set of training sample data, and obtain the defect detection results of each set of training sample data;

[0037] A loss calculation module, used to perform loss calculation on the defect detection results of each set of training sample data based on the actual defect results corresponding to each set of training sample data, to obtain the loss calculation results of the display screen defect detection model;

[0038] An updating module is used to update the model parameters of the display screen defect detection model based on the loss calculation result of the display screen defect detection model.

[0039] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0040] Memory;

[0041] Processor; and

[0042] Computer programs;

[0043] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect or the second aspect.

[0044] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method as described in the first aspect or the second aspect.

[0045] In a seventh aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the display screen defect detection method as described above is implemented.

[0046] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0047] In the display screen defect detection method, model training method, device, equipment and medium provided by the embodiments of the present disclosure, three screen images of the display screen to be detected are obtained, and the three screen images include two oblique images and one main view image. The two oblique images and one main view image are used to synchronously perform defect detection on the display screen to be detected, and the defect detection result of the display screen to be detected is obtained. Compared with the solution in the prior art that multiple images are first detected separately using traditional algorithms, and then the detection results of each image are integrated to obtain the final defect detection result, the present disclosure uses two oblique images and one main view image to synchronously perform defect detection on the display screen to be detected, and obtains the defect detection result of the display screen to be detected. It does not need to be split into two stages. The three screen images are used for synchronous detection, which can fully include different information in the three screen images, and can obtain more comprehensive and complete defect features. During detection, the rich defect information from multiple angles in the three images can be fully utilized, thereby greatly reducing the error of display screen defect detection, improving the accuracy of display screen defect detection, reducing time consumption, and reducing algorithm complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 is a flow chart of a display screen defect detection method provided by an embodiment of the present disclosure;

[0051] Figure 2 is a schematic diagram of a display screen defect detection scenario provided by an embodiment of the present disclosure;

[0052] Figure 3 is a flow chart of a display screen defect detection method provided by another embodiment of the present disclosure;

[0053] Figure 4 is a flow chart of a model training method provided by an embodiment of the present disclosure;

[0054] Figure 5 is a schematic diagram of a framework of a display screen defect detection model provided by an embodiment of the present disclosure;

[0055] Figure 6A schematic diagram of the structure of a display screen defect detection device provided in an embodiment of the present disclosure;

[0056] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure;

[0057] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0060] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0061] As an emerging technology, deep learning has flourished in the field of image processing. Its ability to autonomously learn image data features avoids the tediousness of manually designing algorithms. Its precise detection performance, high detection efficiency, and good generalization performance for various types of image tasks have made deep learning technology widely used in the field of image processing, including image detection, image classification, image reconstruction, etc.

[0062] Screen defect detection of display screens is an important step to ensure the quality of display screens and plays an important role in the display screen production process. Screen defects include dot and line defects (such as dark dots and dark lines), Mura defects, appearance defects, etc.

[0063] In related technologies, display screen defects need to be detected by algorithms after high-definition industrial cameras capture images. Different defects often require different optical solutions to obtain clearer images. Different optical solutions have different parameters such as camera shooting angle, light source intensity and light source angle. Therefore, the detection of the same defect type or different screen defect types requires multiple images captured by different optical solutions, and then screen defect detection is performed on the multiple images through algorithms.

[0064] In the existing technical solution, multiple images are tested separately using traditional algorithms, and then the test results of each image are integrated to obtain the final defect detection result. This solution does not use the feature information of the same defect in different images. In the subsequent processing, the pixel positions of the same defect in different images need to be aligned, so that the whole process is divided into two stages, and both stages will produce errors, resulting in low detection accuracy, and increasing time consumption and algorithm complexity.

[0065] Convolutional Neural Networks (CNN) are a type of feedforward neural network that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure. Therefore, they are also called "Shift-Invariant Artificial Neural Networks (SIANN)".

[0066] Time delay network and LeNet-5 are the earliest convolutional neural networks. With the introduction of deep learning theory and the improvement of numerical computing equipment, convolutional neural networks have developed rapidly and have been applied to computer vision, natural language processing and other fields. Convolutional neural networks are constructed by imitating the visual perception mechanism of organisms. They can perform supervised learning and unsupervised learning. The convolution kernel parameter sharing in the hidden layer and the sparsity of the inter-layer connection enable convolutional neural networks to learn grid-like topology features such as pixels and audio with a small amount of computation, with stable results and no additional feature engineering requirements for the data.

[0067] The convolutional neural network mainly includes the input layer and the hidden layer, which are described as follows:

[0068] 1.1 Convolutional Neural Network Input Layer

[0069] The input layer of a convolutional neural network can process multi-dimensional data. The input layer of a one-dimensional convolutional neural network receives a one-dimensional or two-dimensional array, where the one-dimensional array is usually time or spectrum sampling; the two-dimensional array may contain multiple channels; the input layer of a two-dimensional convolutional neural network receives a two-dimensional or three-dimensional array; the input layer of a three-dimensional convolutional neural network receives a four-dimensional array. Since convolutional neural networks are widely used in the field of computer vision, many studies presuppose three-dimensional input data when introducing their structure, that is, two-dimensional pixels and RGB channels on a plane. Similar to other neural network algorithms, since the gradient descent algorithm is used for learning, the input features of convolutional neural networks need to be standardized. Specifically, before inputting the learning data into the convolutional neural network, the input data needs to be normalized in the channel or time / frequency dimension. If the input data is pixels, the original pixel values ​​distributed in [0,255] can also be normalized to the [0,1] interval. Standardization of input features is conducive to improving the learning efficiency and performance of convolutional neural networks.

[0070] 1.2 Convolutional Neural Network Hidden Layer

[0071] The hidden layer of a convolutional neural network includes three common structures: convolutional layer, pooling layer, and fully connected layer. In some more modern algorithms, there may be complex structures such as Inception modules and residual blocks. Among the common structures, convolutional layers and pooling layers are unique to convolutional neural networks. The convolution kernel in the convolution layer contains weight coefficients, while the pooling layer does not. Taking LeNet-5 as an example, the order of the three common structures in the hidden layer is usually: input-convolutional layer-pooling layer-fully connected layer-output. Convolutional kernel: The function of the convolutional layer is to extract features from the input data. It contains multiple convolution kernels. Each element of the convolution kernel corresponds to a weight coefficient and a bias vector, which is similar to a neuron in a feedforward neural network. Each neuron in the convolutional layer is connected to multiple neurons in a similar area in the previous layer. The size of the area depends on the size of the convolution kernel, which is also called the "receptive field", which is similar to the receptive field of visual cortical cells. When the convolution kernel is working, it will regularly scan the input features, perform matrix element multiplication on the input features within the receptive field, and superimpose the deviation.

[0072] This disclosure also introduces several attention mechanisms, which are described as follows:

[0073] 1. Self-Attention

[0074] The self-attention mechanism was originally applied to the field of speech and text to process the input character encoding vectors. The first step in calculating self-attention is to create 3 vectors from each input vector. For each word, we create a Query vector, a Key vector, and a Value vector. These vectors are generated by multiplying the embedded word by the 3 training matrices we created during the training process.

[0075] The second step in calculating self attention is to calculate the score. For example, in calculating the self attention of the first word "thinking". We need to score each word of the input sentence based on this word. When we encode a word at a certain position, the score determines the degree of attention to other words of the input sentence. The corresponding word is scored by clicking the query vector and the key vector. So, if we process the self attention at the starting position, the first score is the dot product of and, and the second score is the dot product of and.

[0076] The calculation for the third and fourth steps is to divide the score of the second step by (The dimension of the key vector is usually 64, and its square root = 8, which can make the gradient more stable during training. It is not a unique value and is not limited to a specific value. The output is then normalized by the softmax function so that the final list sums to 1. This softmax score determines the degree of representation of each word position in each sentence. Obviously, the softmax score of the current word corresponding to the position of this word in the sentence is the highest, but sometimes the attention mechanism can also pay attention to other words besides the current word.

[0077] The fifth step is to multiply each Value vector by the softmax score. The actual meaning here is to reduce the attention to irrelevant words while keeping the attention to the current word unchanged.

[0078] The sixth step is to accumulate the vector of weighted values. In this step, the output of the self-attention layer for the first word is generated.

[0079] 2. Cross-Attention

[0080] The computation of Cross-Attention is the same as Self-Attention except that the input to the self-attention mechanism is different. Cross-Attention combines two independent embedding sequences of the same dimension asymmetrically, while the input to Self-Attention is a single embedding sequence. One of the sequences is used as the query input, while the other sequence is used as the key and value input.

[0081] 3. Deformable Attention

[0082] The core problem of applying Transformer attention to image feature maps is that it will look at all possible spatial locations. To solve this problem, a deformable attention module is introduced. The deformable attention module only cares about a small set of key sampling points around the reference point, regardless of the spatial size of the feature map. By assigning a small fixed number of keys to each query, the convergence and feature space resolution issues can be alleviated.

[0083] Given an input feature map x∈R C×H×W , let q be the context feature Z q The query element, P q is a two-dimensional reference point. Input feature map x (C x H x W), each pixel in the feature map is a C-channel vector Z q , the index of each pixel point (reference point) is the two-dimensional position coordinate P q M represents the number of heads in the multi-head attention mechanism. Each head only considers Z q Nearby K points (K is much smaller than H x W). Δ p m q k Represents the position offset of the sampling, which is a two-dimensional coordinate (the initial sampling point is fixed, but the coordinates of more noteworthy points will be predicted through the fully connected layer calculation later). m q k Represents the attention weight (∑ k=1 A m q k =1). m and W' m Represents the weight parameters of two sets of fully connected layers. Among them, the two values ​​A that need to be learned and predicted m q k and Δ p m q kAll through Z q Calculated. Specifically, the position offset Δ p m q k It is learnable and is obtained by passing the query through the fully connected layer. m q k It is directly obtained by passing the query through the fully connected layer. Each query samples K positions in each head and only needs to interact with the features of these positions (x(P q +Δ p m q k ) represents the value interpolated based on the sampling point position).

[0084] Deformable-Attention greatly reduces the amount of calculation by only calculating a limited number of other feature points near a feature point instead of calculating all the feature points in the entire feature map, and makes the feature point ignore many other feature points that are not related to itself, making the attention calculation more accurate. It is especially suitable for images with higher resolution and fewer pixels occupied by defects.

[0085] Figure 1 A flow chart of a display screen defect detection method provided in an embodiment of the present disclosure. The method can be applied to electronic devices, which may be portable mobile devices such as smart phones, tablet computers, laptop computers, in-vehicle navigation devices, and smart sports equipment; or fixed devices such as personal computers and smart home appliances. The method can be applied to scenarios where display screen defects are detected, which can reduce the error of display screen defect detection, improve the accuracy of display screen defect detection, reduce time consumption, and reduce algorithm complexity. It is understandable that the display screen defect detection method provided in an embodiment of the present disclosure can also be applied in other scenarios.

[0086] Below Figure 1 The display screen defect detection method shown in the figure is introduced, and the specific steps of the method are as follows:

[0087] S101, acquiring three screen images of a display screen to be detected, wherein the three screen images include two oblique view images and one front view image.

[0088] like Figure 2 As shown, a main camera perpendicular to the screen to be detected and two oblique cameras with side light sources are used to obtain three screen images of the display screen to be detected. Optionally, the three screen images include two oblique images and one main view image.

[0089] S102, using two oblique view images and one main view image, synchronously perform defect detection on the display screen to be detected to obtain a defect detection result of the display screen to be detected.

[0090] In this step, after acquiring two oblique images and one front view image, the electronic device uses the two oblique images and one front view image to synchronously perform defect detection on the display screen to be detected, and obtains a defect detection result of the display screen to be detected. Specifically, the display screen to be detected can be defect-detected based on a defect detection algorithm, or can be defect-detected by a preset display screen defect detection model, which is not specifically limited in this embodiment.

[0091] The disclosed embodiment obtains three screen images of the display screen to be detected, wherein the three screen images include two oblique images and one main view image, and uses the two oblique images and one main view image to synchronously perform defect detection on the display screen to be detected, thereby obtaining a defect detection result of the display screen to be detected. Compared with the solution in the prior art that first uses a traditional algorithm to detect multiple images separately, and then integrates the detection results of each image to obtain the final defect detection result, the disclosed embodiment uses two oblique images and one main view image to synchronously perform defect detection on the display screen to be detected, thereby obtaining a defect detection result of the display screen to be detected, without having to split it into two stages. Using three screen images to synchronously detect can fully include different information in the three screen images, and can obtain more comprehensive and complete defect features. During detection, the rich defect information from multiple angles in the three images can be fully utilized, thereby greatly reducing the error of display screen defect detection, improving the accuracy of display screen defect detection, reducing time consumption, and reducing algorithm complexity.

[0092] Figure 3 This is a flow chart of a display screen defect detection method provided by another embodiment of the present disclosure, and the method includes the following specific steps:

[0093] S201, acquiring three screen images of a display screen to be detected, wherein the three screen images include two oblique view images and one front view image.

[0094] like Figure 2 As shown, a main camera perpendicular to the screen to be detected and two oblique cameras with side light sources are used to obtain three screen images of the display screen to be detected. Optionally, the three screen images include two oblique images and one main view image.

[0095] In some embodiments, after obtaining two oblique images and one main view image of the display screen to be inspected, the method further includes: preprocessing the two oblique images and one main view image respectively to obtain two target oblique images and one target main view image. Optionally, the preprocessing of the image includes but is not limited to image cropping. The defective image captured by the camera includes some surrounding background parts in addition to the screen part of the display screen. This part of the image is unnecessary. Useless or redundant images will also increase the time cost and GPU memory consumption during detection. Therefore, it is necessary to remove these useless images by cropping and only retain the display screen image content.

[0096] S202, inputting two oblique view images and one main view image into a trained display screen defect detection model, and having the display screen defect detection model perform defect detection on the display screen to be detected, and outputting a defect detection result of the display screen to be detected.

[0097] After acquiring two oblique images and one front view image, the electronic device inputs the two oblique images and one front view image into a trained display defect detection model, and the display defect detection model performs defect detection on the display to be detected and outputs the defect detection result of the display to be detected.

[0098] In some embodiments, the step of inputting the two oblique views and the one front view image into a trained display screen defect detection model, performing defect detection on the display screen to be detected by the display screen defect detection model, and outputting the defect detection result of the display screen to be detected comprises: inputting the two target oblique views and the one target front view image into a trained display screen defect detection model, performing defect detection on the display screen to be detected by the display screen defect detection model, and outputting the defect detection result of the display screen to be detected.

[0099] In some embodiments, the display screen defect detection model includes an image feature extraction module, a feature alignment and fusion module, and a detection head module.

[0100] Specifically, S202 includes but is not limited to S2021, S2022, and S2023:

[0101] S2021, inputting two oblique view images and one main view image into the image feature extraction module to obtain feature information corresponding to the three screen images respectively;

[0102] S2022, inputting the feature information corresponding to the three screen images respectively into the feature alignment and fusion module to obtain a high-dimensional feature information space of the screen image;

[0103] S2023: Input the high-dimensional feature information space into the detection head module to obtain a defect detection result of the display screen to be detected.

[0104] like Figure 5 As shown in the figure, the display screen defect detection model is divided into the Backbone of the image feature extraction module (left part), the feature alignment and fusion module (middle part) for feature alignment and fusion between different images, and the detection head module (right part) for defect detection. The entire display screen defect detection model is divided into three parts. The left side is the image feature extraction module. Three screen images are obtained by taking pictures with three cameras, one main view image and two oblique views. The image feature information of each of the three screen images is extracted through the Backbone; further, the image feature information of each of the three screen images is sent to the feature alignment and fusion module in the middle, and the feature information of each of the three screen images is fused and aligned using a multi-head hybrid attention mechanism. Then, a unified high-dimensional feature information space is generated through channel superposition and convolution operations, and sent to the detection head module; the detection head module on the right performs subsequent defect detection based on the high-dimensional feature information space.

[0105] The disclosed embodiment obtains three screen images of the display screen to be detected, wherein the three screen images include two oblique images and one main view image, and inputs the two oblique images and the one main view image into a well-trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected. Compared with the solution in the prior art that multiple images are first detected separately using traditional algorithms, and then the detection results of each image are integrated to obtain the final defect detection result, the disclosed embodiment inputs two oblique images and one main view image into a well-trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected, without the need to split into two stages, and uses three screen images for synchronous detection, which can fully include different information in the three screen images, and can obtain more comprehensive and complete defect features. During detection, the rich defect information from multiple angles in the three images can be fully utilized, thereby greatly reducing the error of display screen defect detection, improving the accuracy of display screen defect detection, reducing time consumption, and reducing algorithm complexity.

[0106] Figure 4 A flow chart of the model training method provided in the embodiment of the present disclosure, such as Figure 4 As shown, the method includes the following steps:

[0107] S301, constructing a display screen defect detection model.

[0108] In this step, the electronic device constructs a display screen defect detection model. The display screen defect detection model can be a neural network model. Specifically, a neural network MIFT-Net based on deep learning multi-image fusion is designed. In the image feature information extraction stage, a hierarchical convolution method based on a pyramid structure is used to fuse feature information of different levels in the same image, which can better obtain comprehensive and rich feature information of the entire image. In the feature alignment and fusion stage, a multi-head hybrid attention mechanism based on Self-Attention and Cross-Attention is used to align the features of multiple different feature maps of the same defective screen, and fuse the multi-angle information of each image to obtain a high-dimensional feature information space. In the defect detection stage, the detection head module is used to perform subsequent defect detection of various types based on the high-dimensional feature information space.

[0109] like Figure 5 As shown in the figure, the display screen defect detection model is divided into the Backbone of the image feature extraction module (left part), the feature alignment and fusion module (middle part) for feature alignment and fusion between different images, and the detection head module (right part) for defect detection. The entire display screen defect detection model is divided into three parts. The left side is the image feature extraction module. Three screen images are obtained by taking pictures with three cameras, one main view image and two oblique views. The image feature information of each of the three screen images is extracted through the Backbone; further, the image feature information of each of the three screen images is sent to the feature alignment and fusion module in the middle, and the feature information of each of the three screen images is fused and aligned using a multi-head hybrid attention mechanism. Then, a unified high-dimensional feature information space is generated through channel superposition and convolution operations, and sent to the detection head module; the detection head module on the right performs subsequent defect detection based on the high-dimensional feature information space.

[0110] The following is a detailed introduction to the image feature extraction module, feature alignment and fusion module, and detection head module:

[0111] 1) Image feature extraction module

[0112] Optionally, the image feature extraction and fusion module includes a pyramid feature extraction submodule, a block pixel attention submodule, a feature compression submodule, and an attention random inactivation submodule.

[0113] 1.1) Pyramid feature extraction submodule

[0114] This module extracts and fuses features from different input layers multiple times, utilizes the multi-angle information of feature maps at each layer, and better extracts the global feature information of the input image. It also directly adds shallow feature information to deep features through Res connections, thus avoiding the disadvantage of feature information dissipation.

[0115] The input feature map F1 passes through 4 convolution blocks to generate 4 hierarchical features (F2-F5). The size of each hierarchical feature is half of the previous layer, and the channel is twice that of the previous layer. The F5 feature layer obtains the intermediate result feature layer D5.

[0116] D5 is bilinearly interpolated to obtain R5, making the feature size the same as F4. R5 is then superimposed with F4 according to the feature channel, and then a 1×1 convolution block is used to fuse the features between channels to obtain the intermediate result feature layer D4.

[0117] D4 and R5 are added, and R4 is obtained through bilinear interpolation, so that the feature size is the same as F3. R4 is then superimposed with F3 according to the feature channel, and then a 1×1 convolution block is used to fuse the features between channels to obtain the intermediate result feature layer D3.

[0118] D3 and R4 are added and bilinearly interpolated to obtain R3, so that the feature size is the same as F2. R3 is then superimposed with F2 according to the feature channel, and then a 1×1 convolution block is used to fuse the features between channels to obtain the intermediate result feature layer D2.

[0119] D2 and R3 are added, and R2 is obtained through bilinear interpolation, so that the feature size is the same as F1. R2 is then superimposed with F1 according to the feature channel, and then a 1×1 convolution block is used to fuse the features between channels to obtain the intermediate result feature layer D1.

[0120] D1 and R2 are added to obtain the final result R1. The feature size of R1 is the same as that of F1, but the number of channels is more than that of F1.

[0121] 1.2) Block Pixel Attention Submodule (RPA)

[0122] The block pixel attention submodule assigns a weight to each pixel in the input feature area, so that the neural network pays more attention to the areas with obvious image features. The input feature (B, C, H, W) is first compressed into (B, C*r, H / 2, W / 2) by a BatchNorm-DefConv-ReLU, where r<1; then restored to (B, C, H / 4, W / 4) by a BatchNorm-DefConv, and the weight of each pixel value is generated by the SigMoid function. Finally, bilinear interpolation is used to restore it to (B, C, H, W), which is multiplied one-to-one with the original input feature.

[0123] 1.3) Feature Squeeze

[0124] This module uses convolution to further extract defect feature information from the feature map output by the previous layer and compresses the length and width of the feature map.

[0125] 1.4) Attention Dropout Module

[0126] Based on the attention random inactivation (Dropout) method, different from the random method used in general Dropout, it uses attention to retain more important feature information, making the performance and generalization of the neural network better.

[0127] The input features are subjected to two batch normalization + variable convolution + ReLU / SigMiod to generate an attention matrix of the same size as the original features. According to the value of the attention matrix, the neurons at the corresponding positions of the original feature matrix whose attention is less than the threshold are set to zero.

[0128] 2) Feature alignment and fusion module

[0129] By using a multi-head (N) hybrid attention mechanism, each hybrid attention head uses the self-attention mechanism based on Deformable-Attention + the mutual attention module based on Deformable-Attention + the sum / normalization operation multiple times (k times) to align and fuse the feature information of three different images, and then the output of each hybrid attention head is superimposed through a channel and convolution operation to obtain a unified feature information space.

[0130] The multi-head attention mechanism can obtain more comprehensive information about the image from different angles. The CrossAttention module can align and fuse the feature information of multiple images. The Deformable Attention module calculates the information of all feature points by focusing only on the information of a limited number of feature points near a certain feature point, and pays more attention to other feature points that are closely related to the feature point, effectively solving the problem of a surge in computational complexity caused by the high resolution of defective images and the difficulty in detecting small-size defects.

[0131] Optionally, the feature alignment fusion module includes an attention mechanism sub-module, a hybrid attention mechanism sub-module, and a high-dimensional feature information space generation sub-module.

[0132] 2.1) Attention mechanism submodule

[0133] Input three sets of feature codes to get three sets of Query and Value values, then traverse the groups in pairs, use the Query values ​​to perform mutual attention calculation based on Deformable-Attention, superimpose the results channels, and use 1×1 convolution to fuse channel features to get three sets of Z values.

[0134] 2.2) Hybrid Attention Mechanism Submodule

[0135] Input three image feature codes to get three sets of Query and Value values. The three sets (Q, V) first perform self-attention calculations based on Deformable-Attention to get their own Z values, and then encode to get three new sets of Query and Value values. Then, they are traversed and grouped in pairs, and mutual attention calculations are performed to get three new sets of Z values, which are summed and normalized with the previous Z values. The results are then sent to the next attention module, and the results of each summing and normalization are superimposed by channel, and the output results of each attention mechanism are summed.

[0136] 2.3) High-dimensional feature information space generation submodule

[0137] The output feature results of the attention module are channel-superimposed and convolved to obtain a unified high-dimensional feature information space. The defect feature information of the three screen images is aligned and fused to obtain the overall information of the entire display screen for subsequent detection and processing.

[0138] 3) Detection head module (Detect HEAD)

[0139] After obtaining a unified high-dimensional feature information space of multiple images of the entire display screen, display screen defect detection can be performed. Optionally, the detection head module can be used for different defect detection tasks. The detection head module includes a classification network, a target detection network, a segmentation network, etc. A variety of different detection heads can also be used for the same detection task, which is not specifically limited here.

[0140] S302, obtaining multiple sets of training sample data and actual defect results corresponding to each set of training sample data, wherein each set of training sample data includes two sample oblique view images and one sample front view image.

[0141] In this step, the electronic device obtains multiple sets of training sample data and the actual defect results corresponding to each set of training sample data, wherein each set of training sample data includes two sample oblique images and one sample front view image. Optionally, the multiple sets of training sample data and the actual defect results corresponding to each set of training sample data may be data stored locally in the electronic device or may be data stored in a server.

[0142] In some embodiments, the multiple sets of training sample data are defect images obtained by taking photos. Before being sent to the display screen defect detection model for training, these images need to be preprocessed, including image cropping and image data set enhancement.

[0143] Image cropping (or scaling): In addition to the display screen, the defect image captured by the camera also includes some surrounding background parts. This part of the image is unnecessary and may affect the training and detection of the display defect detection model. The extra images will also increase the time cost of display defect detection model training and detection and GPU memory consumption. Therefore, it is necessary to remove these useless images by cropping and only retain the display image content. The length and width of the display image and the photo image are made to be powers of 2 for ease of later calculations.

[0144] Dataset enhancement: A large number of sample images are required for display defect detection model training. Modeling is performed using data features learned from a large number of samples. Sometimes the data set is not sufficient, and taking photos to increase the data set requires additional time and labor costs. Sample data needs to be artificially increased through data enhancement. Data enhancement includes image operations such as rotating, offsetting, mirroring, cropping, stretching, and grayscale transformation, so that new images have certain differences from the previous image. In a sense, new images are generated and the data set is expanded. Training the model based on sufficient training sample data can improve the accuracy of display defect detection model display defect detection.

[0145] S303 , training the display screen defect detection model to be trained based on each set of training sample data and the actual defect result corresponding to each set of training sample data, to obtain the defect detection result of each set of training sample data.

[0146] In this step, the electronic device trains the display screen defect detection model to be trained based on each set of training sample data and the actual defect results corresponding to each set of training sample data, and obtains the defect detection results of each set of training sample data. Since the parameters of the model are not optimal, the defect detection results during the training process may not be close to the actual defect results. Training the model based on sufficient training sample data can improve the accuracy of the display screen defect detection model in detecting display screen defects.

[0147] S304: Perform loss calculation on the defect detection results of each set of training sample data based on the actual defect results corresponding to each set of training sample data to obtain the loss calculation result of the display screen defect detection model.

[0148] After obtaining the defect detection results of each set of training sample data, the defect detection results of each set of training sample data are subjected to loss calculation based on the actual defect results corresponding to each set of training sample data, and the loss calculation results of the display screen defect detection model are obtained. The smaller the loss calculation result, the closer the defect detection result is to the actual defect result, the more accurate the foreign matter detection is, and the higher the current accuracy of the model is. Specifically, the loss calculation can be performed through a loss function.

[0149] S305 . Based on the loss calculation result of the display screen defect detection model, update the model parameters of the display screen defect detection model.

[0150] In this step, the model parameters of the display screen defect detection model are updated according to the loss calculation result of the display screen defect detection model, thereby improving the detection accuracy of the display screen defect detection model. As the number of training times increases, the parameters of the model are continuously adjusted so that the defect detection results are closer and closer to the real defect results.

[0151] In some embodiments, S305 may also include but is not limited to S306, S307, and S308:

[0152] S306: Record the number of training cycles of the display screen defect detection model.

[0153] S307 , judging whether the defect detection result of each set of training sample data converges to the real defect result corresponding to each set of training sample data.

[0154] The electronic device determines whether the defect detection result of each set of training sample data converges to the real defect result corresponding to each set of training sample data. If the defect detection result is close to the real defect result, it means that the defect detection result converges to the real defect result.

[0155] S308: If the defect detection result of each set of training sample data has converged to the actual defect result corresponding to each set of training sample data, or the number of training cycles of the display screen defect detection model is greater than a preset number threshold, it is determined that the training of the display screen defect detection model is completed.

[0156] When the defect detection results of each group of training sample data have converged to the actual defect results corresponding to each group of training sample data, or the number of training cycles of the display screen defect detection model is greater than a preset threshold, the training of the display screen defect detection model is completed, and the display screen defect detection model can be used to detect display screen defects.

[0157] The embodiment of the present disclosure constructs a display screen defect detection model, obtains multiple sets of training sample data and the real defect results corresponding to each set of training sample data, wherein each set of training sample data includes two sample oblique images and one sample front view image. Then, the display screen defect detection model to be trained is trained based on each set of training sample data and the real defect results corresponding to each set of training sample data to obtain the defect detection results of each set of training sample data, and the loss calculation of the defect detection results of each set of training sample data is performed based on the real defect results corresponding to each set of training sample data to obtain the loss calculation results of the display screen defect detection model. Further, based on the loss calculation results of the display screen defect detection model, the model parameters of the display screen defect detection model are updated, and the number of training cycles of the display screen defect detection model is recorded. It is determined whether the defect detection results of each set of training sample data converge to the real defect results corresponding to each set of training sample data. Furthermore, if the defect detection results of each set of training sample data have converged to the real defect results corresponding to each set of training sample data, or the number of training cycles of the display screen defect detection model is greater than the preset number threshold, it is determined that the training of the display screen defect detection model is completed. Since the display screen defect detection model is trained, loss calculation is performed on the defect detection results of each group of training sample data based on the actual defect results corresponding to each group of training sample data, and the loss calculation results of the display screen defect detection model are obtained. Through the loss calculation results, the model parameters are continuously updated to improve the accuracy of the display screen defect detection model, thereby improving the display screen defect detection performance.

[0158] Figure 6 The structure diagram of the display screen defect detection device provided in the embodiment of the present disclosure is shown in FIG. The display screen defect detection device may be the electronic device described in the above embodiment, or the display screen defect detection device may be a component or assembly in the electronic device. The display screen defect detection device provided in the embodiment of the present disclosure may execute the processing flow provided in the display screen defect detection method embodiment, such as Figure 6 As shown, the display screen defect detection device 50 includes: an acquisition module 51 and a detection module 52; wherein the acquisition module 51 is used to acquire three screen images of the display screen to be detected, and the three screen images include two oblique images and one main view image; the detection module 52 is used to use the two oblique images and one main view image to synchronously perform defect detection on the display screen to be detected, and obtain the defect detection result of the display screen to be detected.

[0159] Optionally, the detection module 52 uses two oblique views and one front view image to simultaneously perform defect detection on the display screen to be detected, and when obtaining the defect detection result of the display screen to be detected, it is specifically used to: input the two oblique views and one front view image into a trained display screen defect detection model, and use the display screen defect detection model to perform defect detection on the display screen to be detected, and output the defect detection result of the display screen to be detected.

[0160] Optionally, after obtaining the two oblique images and one front view image of the display screen to be inspected, the display screen defect detection device 50 further includes: a processing module 53; the processing module 53 is used to pre-process the two oblique images and the one front view image respectively to obtain two target oblique images and one target front view image;

[0161] The detection module 52 inputs the two oblique images and one main view image into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected. Specifically, it is used to: input the two target oblique images and the one target main view image into the trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected.

[0162] Optionally, the display screen defect detection model includes an image feature extraction module, a feature alignment and fusion module, and a detection head module;

[0163] The detection module 52 inputs two oblique images and one main view image into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected. When outputting the defect detection result of the display screen to be detected, it is specifically used to: input two oblique images and one main view image into the image feature extraction module to obtain feature information corresponding to the three screen images respectively; input the feature information corresponding to the three screen images respectively into the feature alignment and fusion module to obtain a high-dimensional feature information space of the screen image; and input the high-dimensional feature information space into the detection head module to obtain the defect detection result of the display screen to be detected.

[0164] Figure 6 The display screen defect detection device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar and will not be described in detail here.

[0165] Figure 7The schematic diagram of the structure of the model training device provided in the embodiment of the present disclosure. The model training device may be the electronic device as described in the above embodiment, or the model training device may be a component or assembly in the electronic device. The model training device provided in the embodiment of the present disclosure may execute the processing flow provided in the embodiment of the model training method, such as Figure 7 As shown, the model training device 60 includes: a construction module 61, a sample acquisition module 62, a obtaining module 63, a loss calculation module 64, and an updating module 65; wherein the construction module 61 is used to construct a display screen defect detection model; the sample acquisition module 62 is used to obtain multiple groups of training sample data and the actual defect results corresponding to each group of training sample data, wherein each group of training sample data includes two sample oblique images and one sample front view image; the obtaining module 63 is used to train the display screen defect detection model to be trained based on each group of training sample data and the actual defect results corresponding to each group of training sample data, and obtain the defect detection results of each group of training sample data; the loss calculation module 64 is used to perform loss calculation on the defect detection results of each group of training sample data based on the actual defect results corresponding to each group of training sample data, and obtain the loss calculation results of the display screen defect detection model; the updating module 65 is used to update the model parameters of the display screen defect detection model based on the loss calculation results of the display screen defect detection model.

[0166] Optionally, the model training device 60 also includes: a recording module 66, a judgment module 67, and a determination module 68; the recording module 66 is used to record the number of training cycles of the display screen defect detection model; the judgment module 67 is used to judge whether the defect detection results of each group of training sample data converge to the actual defect results corresponding to each group of training sample data; the determination module 68 is used to determine that the training of the display screen defect detection model is completed if the defect detection results of each group of training sample data have converged to the actual defect results corresponding to each group of training sample data, or the number of training cycles of the display screen defect detection model is greater than a preset threshold value.

[0167] Figure 7 The display screen defect detection device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar and will not be described in detail here.

[0168] Figure 8 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 8 , which shows a structural schematic diagram of an electronic device 500 suitable for implementing the embodiments of the present disclosure. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0169] like Figure 8 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503 to implement the display screen defect detection method of the embodiment described in the present disclosure. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0170] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0171] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart, thereby implementing the display screen defect detection method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above functions defined in the method of the embodiment of the present disclosure are executed.

[0172] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0173] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0174] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0175] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0176] Acquire three screen images of the display screen to be detected, wherein the three screen images include two oblique view images and one front view image;

[0177] The defect detection of the display screen to be detected is performed synchronously by using two oblique view images and one main view image to obtain the defect detection result of the display screen to be detected.

[0178] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0179] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0182] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0183] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0184] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0185] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A display screen defect detection method, characterized in that: The method comprises: Acquire three screen images of the display screen to be detected, wherein the three screen images include two oblique view images and one front view image; Using two oblique images and one main view image, the defect detection of the display screen to be detected is synchronously performed to obtain a defect detection result of the display screen to be detected; The method of synchronously performing defect detection on the display screen to be detected by using two oblique images and one main view image to obtain a defect detection result of the display screen to be detected includes: Input the two oblique view images and one main view image into a trained display screen defect detection model, and use the display screen defect detection model to perform defect detection on the display screen to be detected, and output the defect detection result of the display screen to be detected; The display screen defect detection model includes an image feature extraction module, a feature alignment and fusion module, and a detection head module; In the image feature information extraction stage, a hierarchical convolution method based on a pyramid structure is used to fuse feature information at different levels in the same image, which can better obtain comprehensive and rich feature information of the entire image; in the feature alignment and fusion stage, a multi-head hybrid attention mechanism based on Self-Attention and Cross-Attention is used to align multiple different feature maps of the same defective screen, fuse the multi-angle information of each image, and obtain a high-dimensional feature information space; in the defect detection stage, the detection head module is used to perform subsequent defect detection based on the high-dimensional feature information space.

2. The method according to claim 1, characterized in that After obtaining two oblique view images and one main view image of the display screen to be detected, the method further includes: Preprocessing the two oblique view images and one main view image respectively to obtain two target oblique view images and one target main view image; The two oblique images and one main view image are input into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected, including: The two target oblique images and the one target main view image are input into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs a defect detection result of the display screen to be detected.

3. The method according to claim 1, characterized in that The two oblique images and one main view image are input into a trained display screen defect detection model, and the display screen defect detection model performs defect detection on the display screen to be detected, and outputs the defect detection result of the display screen to be detected, including: Inputting two oblique view images and one main view image into the image feature extraction module to obtain feature information corresponding to the three screen images respectively; Inputting the feature information corresponding to the three screen images respectively into the feature alignment and fusion module to obtain a high-dimensional feature information space of the screen image; The high-dimensional feature information space is input into the detection head module to obtain the defect detection result of the display screen to be detected.

4. A model training method, characterized in that: The method comprises: Build a display defect detection model; Acquire multiple sets of training sample data and actual defect results corresponding to each set of training sample data, wherein each set of training sample data includes two sample oblique view images and one sample front view image; The display screen defect detection model to be trained is trained based on each set of training sample data and the actual defect results corresponding to each set of training sample data to obtain the defect detection results of each set of training sample data; Performing loss calculation on the defect detection results of each set of training sample data based on the actual defect results corresponding to each set of training sample data, to obtain the loss calculation result of the display screen defect detection model; Based on the loss calculation result of the display screen defect detection model, updating the model parameters of the display screen defect detection model; In the image feature information extraction stage, a hierarchical convolution method based on a pyramid structure is used to fuse feature information at different levels in the same image, which can better obtain comprehensive and rich feature information of the entire image; in the feature alignment and fusion stage, a multi-head hybrid attention mechanism based on Self-Attention and Cross-Attention is used to align multiple different feature maps of the same defective screen, fuse the multi-angle information of each image, and obtain a high-dimensional feature information space; in the defect detection stage, the detection head module is used to perform subsequent defect detection based on the high-dimensional feature information space.

5. The method according to claim 4, characterized in that After updating the model parameters of the display screen defect detection model based on the loss calculation result of the display screen defect detection model, the method further includes: Recording the number of training cycles of the display screen defect detection model; Determine whether the defect detection result of each set of training sample data converges to the actual defect result corresponding to each set of training sample data; If the defect detection results of each set of training sample data have converged to the actual defect results corresponding to each set of training sample data, or the number of training cycles of the display screen defect detection model is greater than a preset number threshold, it is determined that the training of the display screen defect detection model is completed.

6. A display screen defect detection device, characterized in that: include: An acquisition module, used for acquiring three screen images of the display screen to be detected, wherein the three screen images include two oblique view images and one main view image; A detection module, used to use two oblique images and one main view image to synchronously perform defect detection on the display screen to be detected, and obtain a defect detection result of the display screen to be detected; The detection module uses two oblique images and one main view image to synchronously perform defect detection on the display screen to be detected, and when obtaining the defect detection result of the display screen to be detected, it is specifically used to: Input the two oblique view images and one main view image into a trained display screen defect detection model, and use the display screen defect detection model to perform defect detection on the display screen to be detected, and output the defect detection result of the display screen to be detected; The display screen defect detection model includes an image feature extraction module, a feature alignment and fusion module, and a detection head module; In the image feature information extraction stage, a hierarchical convolution method based on a pyramid structure is used to fuse feature information of different levels in the same image, which can better obtain comprehensive and rich feature information of the entire image; In the feature alignment and fusion stage, a multi-head hybrid attention mechanism based on Self-Attention and Cross-Attention is used to align multiple feature maps of the same defective screen, fuse the multi-angle information of each image, and obtain a high-dimensional feature information space; In the defect detection stage, the detection head module is used to perform subsequent defect detection based on the high-dimensional feature information space.

7. A model training device, characterized in that: include: A building module for building a display screen defect detection model; A sample acquisition module, used to acquire multiple sets of training sample data and actual defect results corresponding to each set of training sample data, wherein each set of training sample data includes two sample oblique view images and one sample front view image; An obtaining module is used to train the display screen defect detection model to be trained based on each set of training sample data and the actual defect results corresponding to each set of training sample data, and obtain the defect detection results of each set of training sample data; A loss calculation module, used to perform loss calculation on the defect detection results of each set of training sample data based on the actual defect results corresponding to each set of training sample data, to obtain the loss calculation results of the display screen defect detection model; An updating module, configured to update a model parameter of the display screen defect detection model based on a loss calculation result of the display screen defect detection model; In the image feature information extraction stage, a hierarchical convolution method based on a pyramid structure is used to fuse feature information of different levels in the same image, which can better obtain comprehensive and rich feature information of the entire image; In the feature alignment and fusion stage, a multi-head hybrid attention mechanism based on Self-Attention and Cross-Attention is used to align multiple feature maps of the same defective screen, fuse the multi-angle information of each image, and obtain a high-dimensional feature information space; In the defect detection stage, the detection head module is used to perform subsequent defect detection based on the high-dimensional feature information space.

8. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Part defect detection method and system

    CN109934821A

  • Defect detection method, device and equipment for flat display panel and medium

    CN112884694A