Real-time segmentation method for display screen defects in multiple backgrounds based on improved Mask R-CNN

Abstract

The application discloses a real-time display screen defect segmentation method based on an improved MaskR-CNN in a multi-background. The application combines a traditional MaskR-CNN model feature extraction network and an attention mechanism module, and in the process of extracting the features of display screen defect images in different backgrounds, the weight proportion of output features is adaptively adjusted by adding an attention module, the interference of different backgrounds on defect detection is reduced, the features learned by the deep learning network model in the pixel-level segmentation are more concentrated, and the defect features can be more targetedly learned; the attention mechanism module is combined in the multi-scale feature fusion network of the traditional MaskR-CNN model, more attention is paid to the display screen defect feature information in the acquisition of feature information; on the basis of the whole framework, the training parameters are adjusted, the whole model can be used for the optimization of multi-target detection persons in different backgrounds, and the detection precision and feature segmentation are better than those of the traditional model.

Description

Technical Field

[0001] This invention relates to a real-time segmentation method for display screen defects under multiple backgrounds based on an improved Mask R-CNN, belonging to the fields of display screen defect detection and computer vision. Background Technology

[0002] Liquid crystal display (LCD) technology has been widely applied in the electronics and information industry, playing a vital role in promoting the transformation of economic development patterns, optimizing and upgrading industrial structures, and driving the development of high-tech industries. With the proliferation of various monitor brands, competition in the monitor market has intensified. To enhance market competitiveness, manufacturers have focused on improving product quality and production efficiency.

[0003] During the manufacturing process of monitors, particles adhering to the raw materials or entering the monitor during assembly can cause stains on the screen. To improve the quality of monitors, manufacturers add a defect detection process to the monitor production line to select out unqualified monitors and increase the pass rate.

[0004] For display screen inspection on production lines, most LCD manufacturers still rely on traditional human visual inspection (HVI) methods to detect defects. However, traditional human eye defect detection methods struggle to meet the requirements of production efficiency, inspection quality, and low cost. Therefore, researching rapid, objective, and human-compliant automated machine vision defect detection methods has become both an urgent requirement and a challenge in the development of LCD technology. Summary of the Invention

[0005] This invention provides a real-time segmentation method for display screen defects under multiple backgrounds based on an improved Mask R-CNN, for the purpose of detecting and segmenting display screen defects under multiple backgrounds.

[0006] The technical solution of this invention is: a real-time segmentation method for display screen defects under multiple backgrounds based on an improved Mask R-CNN, comprising:

[0007] Acquire defective images of multi-background displays, construct a sample dataset, and divide the sample dataset into a training dataset and a test dataset;

[0008] By combining the feature extraction network and / or feature fusion network in the traditional Mask R-CNN model with the attention mechanism module, several improved Mask R-CNN models were created; ablation experiments were conducted on several improved Mask R-CNN models to select the improved Mask R-CNN model with the best performance.

[0009] The optimal performance improved Mask R-CNN model is initialized with training parameters, and then trained to obtain the optimal weights.

[0010] The system acquires images from the display screen, loads the obtained optimal weights into the improved Mask R-CNN model with the best performance, and makes predictions on the acquired images to obtain the prediction results.

[0011] The method of combining the feature extraction network and / or feature fusion network in the traditional Mask R-CNN model with the attention mechanism module includes:

[0012] An improved feature extraction network in the Mask R-CNN model is constructed by combining the feature extraction network and attention mechanism module from the traditional Mask R-CNN model; and / or

[0013] An improved feature fusion network in the Mask R-CNN model is constructed by combining the feature fusion network in the traditional Mask R-CNN model with the attention mechanism module.

[0014] The improved Mask R-CNN model includes a traditional Mask R-CNN feature extraction network and m attention mechanism modules in its feature extraction network, and a traditional Mask R-CNN feature fusion network and n attention mechanism modules in its feature fusion network. Multiple improved Mask R-CNN models are obtained by changing the number of attention mechanism modules in the feature extraction network and the number of attention mechanism modules in the feature fusion network. Where m, n = 0, 1, 2, 3, 4; m and n are not both 0.

[0015] The specific method of combining the feature extraction network and attention mechanism module in the traditional Mask R-CNN model is as follows: the feature layer output by the residual network module of the feature extraction network in the traditional Mask R-CNN model is passed through the attention mechanism module to obtain the output feature layer.

[0016] The specific method of combining the feature fusion network and attention mechanism module in the traditional Mask R-CNN model is as follows: the input of the feature fusion network in the traditional Mask R-CNN model is added to the attention mechanism module after passing through a two-dimensional convolution operation.

[0017] The process of training the optimal improved Mask R-CNN model is as follows: If it is the first training iteration, the training epoch of the training parameters is increased, and the optimal improved Mask R-CNN model is trained for the first time using the increased training epoch to obtain the pre-trained weight file; if it is a retraining iteration, the training epoch in the training parameters is adjusted to the initial setting value, and the pre-trained weights are loaded into the optimal improved Mask R-CNN model for retraining to obtain the optimal weights.

[0018] The training epochs for the first training of the best-performing improved Mask R-CNN model are longer than the initial setting value. The training epochs for selecting the best-performing improved Mask R-CNN model from multiple improved Mask R-CNN models through ablation experiments are shorter than the initial setting value.

[0019] According to another aspect of the present invention, a real-time segmentation method for display defects in multiple backgrounds based on an improved Mask R-CNN is also provided, comprising:

[0020] The acquisition module is used to acquire defective images of multi-background display screens, construct a sample dataset, and divide the sample dataset into a training dataset and a test dataset.

[0021] The selection module is used to combine the feature extraction network and / or feature fusion network in the traditional Mask R-CNN model with the attention mechanism module to create multiple improved Mask R-CNN models; ablation experiments are performed on multiple improved Mask R-CNN models to select the improved Mask R-CNN model with the best performance;

[0022] The training module is used to initialize the training parameters of the improved Mask R-CNN model with the best performance, and then train the improved Mask R-CNN model with the best performance to obtain the optimal weights.

[0023] The execution module is used to acquire images from the display screen, load the obtained optimal weights into the improved Mask R-CNN model with the best performance, and make predictions on the acquired images from the display screen to obtain prediction results.

[0024] According to another aspect of the present invention, a terminal device is provided, including a memory, a processor, and a program stored in the memory and executable by the processor, wherein the processor executes the program to implement the real-time segmentation method for display defects in multiple backgrounds based on the improved Mask R-CNN described in any one of the above.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the real-time segmentation method for display defects in multiple backgrounds based on the improved Mask R-CNN described in any one of the above.

[0026] The beneficial effects of this invention are:

[0027] This invention combines the traditional Mask R-CNN model's feature extraction network with an attention mechanism module. During feature extraction from display defect images under different backgrounds, the attention module adaptively adjusts the weight ratio of output features, reducing interference from different backgrounds on defect detection. This makes the features learned by the deep learning network model in pixel-level segmentation more focused and allows for more targeted learning of defect features. The combination of the attention mechanism module with the traditional Mask R-CNN model's multi-scale feature fusion network further emphasizes display defect features during feature acquisition. Based on this framework, and with adjustments to training parameters, the model can be optimized for multi-target detection of people under different backgrounds, achieving superior detection accuracy and feature segmentation compared to traditional models. This provides a solution and method for intelligent detection and segmentation of display defects. Attached Figure Description

[0028] Figure 1 This is a flowchart of the present invention;

[0029] Figure 2 Classification diagram of defects in different backgrounds;

[0030] Figure 3 A diagram of the attention mechanism structure;

[0031] Figure 4 To improve the Mask R-CNN model;

[0032] Figure 5 To improve the feature extraction network structure diagram;

[0033] Figure 6 This is a diagram of the Conv block structure;

[0034] Figure 7 This is a diagram of the Indetity block structure.

[0035] Figure 8 To improve the feature fusion network structure diagram;

[0036] Figure 9 A flowchart for training the model;

[0037] Figure 10This is a manually annotated image; the upper part of the image is the original annotated image, and the lower part is an enlarged version of the original image.

[0038] Figure 11 The image shows the prediction results; the top of the image is the original image, the middle image is the detection result image, and the bottom image is a magnified view of the detection result. Detailed Implementation

[0039] The invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited to the description.

[0040] Example 1: As Figure 1-11 As shown, a real-time segmentation method for display screen defects in multiple backgrounds based on an improved Mask R-CNN includes: acquiring images of display screen defects in multiple backgrounds, constructing a sample dataset, and dividing the sample dataset into a training dataset and a test dataset; combining the feature extraction network and / or feature fusion network in the traditional Mask R-CNN model with an attention mechanism module to create multiple improved Mask R-CNN models; performing ablation experiments on the multiple improved Mask R-CNN models to select the best-performing improved Mask R-CNN model; initializing the training parameters of the best-performing improved Mask R-CNN model, and then training the best-performing improved Mask R-CNN model to obtain the optimal weights; acquiring images of the display screen, loading the obtained optimal weights into the best-performing improved Mask R-CNN model, and predicting the acquired images of the display screen to obtain the prediction results.

[0041] Optionally, the combination of the feature extraction network and / or feature fusion network in the traditional Mask R-CNN model with the attention mechanism module includes:

[0042] An improved feature extraction network in the Mask R-CNN model is constructed by combining the feature extraction network and attention mechanism module from the traditional Mask R-CNN model; and / or

[0043] An improved feature fusion network in the Mask R-CNN model is constructed by combining the feature fusion network in the traditional Mask R-CNN model with the attention mechanism module.

[0044] Optionally, the improved Mask R-CNN model includes a traditional Mask R-CNN feature extraction network and m attention mechanism modules in its feature extraction network, and a traditional Mask R-CNN feature fusion network and n attention mechanism modules in its feature fusion network. Multiple improved Mask R-CNN models can be obtained by changing the number of attention mechanism modules in the feature extraction network and the number of attention mechanism modules in the feature fusion network; where m, n = 0, 1, 2, 3, 4; m and n cannot both be 0. The feature layers L1, L2, L3, and L4 output by the residual network module in the traditional feature extraction network are also included. It should be noted that when m takes the value of 1, 2, or 3, the attention mechanism module can be added after any m feature layers L1, L2, L3, or L4 (for example, when m takes the value of 1, L1 can be selected as the feature layer to obtain the output through the attention mechanism module, or L2, L3, or L4 can be selected. If L2 is selected, then L1, L2, L3, and L4 are used as the input of the feature fusion network); the same applies when n takes the value of 1, 2, or 3.

[0045] Optionally, the combination of the feature extraction network and attention mechanism module in the traditional Mask R-CNN model specifically involves passing the feature layer output by the residual network module of the feature extraction network in the traditional Mask R-CNN model through the attention mechanism module to obtain the output feature layer.

[0046] Optionally, combining the feature fusion network and attention mechanism module in the traditional Mask R-CNN model specifically involves adding the input of the feature fusion network in the traditional Mask R-CNN model to the attention mechanism module after performing a two-dimensional convolution operation.

[0047] Optionally, training the optimal improved Mask R-CNN model specifically involves: if it is the first training iteration, increasing the training epoch of the training parameters, and using the increased training epoch to train the optimal improved Mask R-CNN model for the first time to obtain a pre-trained weight file; if it is a retraining iteration, adjusting the training epoch in the training parameters to the initial setting value, loading the pre-trained weights into the optimal improved Mask R-CNN model for retraining to obtain the optimal weights.

[0048] Optionally, the training epochs for the initial training of the optimal improved Mask R-CNN model are longer than the initial setting, while the training epochs for selecting the optimal improved Mask R-CNN model from multiple improved Mask R-CNN models through ablation experiments are shorter than the initial setting. Based on this design, the time required to obtain the optimal improved Mask R-CNN model can be effectively saved without affecting the acquisition of the optimal improved Mask R-CNN model.

[0049] Furthermore, based on experimental data, the present invention provides the following optional specific implementation methods:

[0050] Step 1: Collect images of display screen defects, create a sample dataset, and divide the dataset into training and testing datasets;

[0051] Step 2: Combine the feature extraction network and feature fusion network in the traditional Mask R-CNN model with the attention mechanism to create multiple improved Mask R-CNN models. Perform ablation experiments on the multiple improved Mask R-CNN models to select the Mask R-CNN model with the best performance.

[0052] Step 3: Initialize the training parameters of the best-performing Mask R-CNN model, and then train the best-performing improved Mask R-CNN model to obtain the optimal weights.

[0053] Step 4: Turn on the camera and set the initial parameters of the real-time monitoring module;

[0054] Step 5: Acquire images from the display screen, load the obtained optimal weights into the best-performing Mask R-CNN model, and perform detection and segmentation on the display screen images to obtain detection and segmentation results.

[0055] Specifically:

[0056] In Step 1, the display defect images are collected in real-world scenarios using an industrial camera, capturing images of defects found on displays manufactured by the display screen manufacturer. The sample dataset includes six types of display defects against the same background. In this embodiment, the collected display defects are categorized into six types based on different backgrounds: white background, red background, green background, gray background, gradient background, and blue background. The white, red, green, gray, and blue backgrounds are solid color backgrounds. Figure 2The example demonstrates a gradient background. The sample dataset is divided into a training dataset and a test dataset. In this example, the training dataset contains 900 images, and the test dataset contains 100 images. The training dataset is used to train the model; the test dataset is used to test the model's performance after the entire model has been trained; subsequently, images of the required detection displays can be collected for detection. Furthermore, to ensure the model's generalization ability, this embodiment designs the proportion of defects under different backgrounds in the training and test datasets to be similar, specifically: the number of different colors differs by 1-2 images. For example, in the 100 test dataset images, the proportions of the six backgrounds are similar: 17 white background images, 16 red background images, 17 green background images, 16 gray background images, 17 gradient background images, and 17 blue background images.

[0057] The attention mechanism module in Step 2 is as follows: Figure 3 As shown, the attention mechanism module performs global average pooling on the obtained input feature layer, performs 1D convolution with a kernel size of K, and then passes the input feature layer through a sigmoid activation function to obtain the weights for each channel. These weights are then multiplied by the corresponding elements of the original input feature layer to obtain the final output feature map. The original input feature layer is the input to the attention mechanism module.

[0058] In Step 2, the value of the convolutional kernel K in the attention mechanism module is determined by the number of channels C. In this example, the number of channels C of the four feature layers passing through the attention mechanism module in the feature extraction network are 128, 256, 512, and 1024, respectively. (The formula is missing from the original text.) The K value is 5; the four feature layer channels in the feature fusion network that pass through the attention mechanism module are all 256, according to the formula. The value of K is 5; where odd is the odd number returned after rounding the specified value upwards.

[0059] See Figure 4 The following is a description of the optional specific structures for improving the Mask R-CNN model:

[0060] The traditional Mask R-CNN model consists of a head network, a neck network, and a backbone network. The backbone network includes a feature extraction network and a feature fusion network. The image passes through the feature extraction network and the feature fusion network, then through the neck network (i.e., the RPN network), and finally into the head network for prediction and segmentation. The improved Mask R-CNN model modifies the feature extraction network and the feature fusion network of the traditional Mask R-CNN model, while keeping the rest unchanged.

[0061] The improved Mask R-CNN model's feature extraction network is obtained by combining the original network's feature extraction network with an attention mechanism module. For example... Figure 5 As shown, the feature extraction network employs four attention mechanism modules. The feature layers L1, L2, L3, and L4 output from the residual network module are passed through these attention mechanism modules to obtain the output feature layers S1, S2, S3, and S4, respectively. Specifically: the input image undergoes zero-padding (width and height both 3), followed by a 2D convolution with a stride of 2, a 7x7 kernel, and 64 channels. Then, normalization, activation operations, and max pooling are performed. Next, the L1 feature layer is extracted through one Conv block and two Identity blocks, and then processed through the attention mechanism to obtain S1. S1 undergoes one Conv block and three Identity blocks to extract the L2 feature layer, and then through the attention mechanism to obtain S2. S2 undergoes one Conv block and 22 Identity blocks to extract the L3 feature layer, and then through the attention mechanism to obtain S3. S3 undergoes one Conv block and two Identity blocks to extract the L4 feature layer, and then through the attention mechanism to obtain S4. The Conv block is as follows: Figure 6 As shown, its structure is divided into two branches. The first branch is the input, which undergoes a 2D convolution (Conv2d 2x2) with a stride of 2, a kernel of 1x1, and 64 channels. After normalization and activation, it undergoes another 2D convolution with a stride of 2, a kernel of 3x3, and 64 channels. After normalization and activation, it undergoes another 2D convolution with a stride of 2, a kernel of 1x1, and 256 channels. After normalization, the output is obtained. The second branch is the input, which undergoes a 2D convolution (Conv2d 2x2) with a stride of 2, a kernel of 2x2, and 256 channels. After normalization, the output is obtained. The outputs of the two branches are merged, and after activation, the final output is obtained. The Identity block is shown below. Figure 7 As shown, the input is passed through a 2D convolution (Conv2d 1x1) with a stride of 1, a kernel size of 1x1, and 64 channels, then normalized and activated. It is then passed through a 2D convolution with a stride of 1, a kernel size of 3x3, and 64 channels, then normalized and activated. It is then passed through a 2D convolution with a stride of 1, a kernel size of 1x1, and 256 channels, then activated. Finally, it is merged with the original input and activated again to obtain the final output.

[0062] The improved Mask R-CNN model's feature fusion network is obtained by combining the original network's feature fusion network with an attention mechanism module, such as... Figure 8As shown, the feature fusion network employs four attention mechanism modules. The feature layers S1, S2, S3, and S4 output from the combination of the feature extraction network and the attention mechanism modules are used as inputs to the feature fusion network. Similarly, the inputs to the feature fusion network in the traditional Mask R-CNN model are all added to the attention mechanism modules after a 2D convolution operation. Specifically: The S4 obtained from the image after the feature extraction network is subjected to a 2D convolution with a stride of 1, a kernel of 1x1, and 256 channels, followed by the attention mechanism to obtain feature layer N4. N4 is then expanded by a factor of two using the Upsampling2d operation, and added to the output of S3 after a 2D convolution with a stride of 1, a kernel of 1x1, and 256 channels, followed by the attention mechanism module, to obtain feature layer N3. N3 is then expanded by a factor of two using the Upsampling2d operation, and added to S2 after a 2D convolution with a stride of 1 and a kernel of 1x1. The outputs of a 2D convolution with 256 channels and the attention mechanism module are summed to obtain N2. N2 is then doubled in size by the Upsampling2d operation and summed with the outputs of S1 after a 2D convolution with a stride of 1, a kernel of 1x1, and 256 channels, followed by the attention mechanism module to obtain N1. Then, N4, N3, N2, and N1 are subjected to 2D convolutions with a stride of 1, a kernel of 3x3, and 256 channels to obtain P4, P3, P2, and P1, respectively. P4 is then subjected to max pooling to obtain P5. Finally, P1, P2, P3, P4, and P5 are sequentially fed into the neck network and head network to predict the target.

[0063] The ablation experiment in Step 2 refers to exploring the performance of a network model by modifying a small part of its structure when studying complex deep neural networks (using the training dataset for 30 rounds of training to obtain the weight file; and using the test dataset to evaluate the optimal weight file of the model). In this example, the modified small parts of the structure mainly include the number of attention mechanism modules in the feature extraction network and the number of attention mechanisms in the feature fusion network. As shown in Table 1 below, Model 2 indicates one attention mechanism module in the feature extraction network and one attention mechanism module in the feature fusion network. AP50 refers to the average accuracy with IOU above 0.5, AP75 is the average accuracy with IOU above 0.75, and AP50:95 represents the average accuracy with IOU from 0.5 to 0.95. In this example, by comparing the accuracy of five models under different IOUs, it was found that MASK-7 had an AP50 of 94.6%, AP75 of 80.3%, and AP50:90 of 68.3%, exhibiting the best overall detection performance. IOU is the intersection-union ratio between the predicted and actual locations; a higher IOU value indicates a closer match between the predicted and actual targets. It should be noted that experiments showed the best model performance when the attention mechanism module in the feature extraction and feature fusion networks had a value of 4. Therefore, only a few examples of other attention mechanism module values ​​were shown. In Table 1, when the attention mechanism value in the feature extraction network and feature fusion network is 1, the L1 layer is passed through the attention mechanism module to obtain the output feature layer S1, and S1 is then subjected to a two-dimensional convolution operation before being added to the attention mechanism module; when the attention mechanism value in the feature extraction network and feature fusion network is 2, the L1 and L2 layers are passed through the attention mechanism module to obtain the output feature layers S1 and S2, and S1 and S2 are then subjected to a two-dimensional convolution operation before being added to the attention mechanism module; when the attention mechanism value in the feature extraction network and feature fusion network is 3, the L1, L2, and L3 layers are passed through the attention mechanism module to obtain the output feature layers S1, S2, and S3, and S1, S2, and S3 are then subjected to a two-dimensional convolution operation before being added to the attention mechanism module.

[0064] Table 1 Ablation Experiment Results

[0065] Network Model Feature extraction network Feature Fusion Network AP50 AP75 AP50: 95 MASK-1 0 0 85.6 72.2 59.6 MASK-2 1 1 91.3 78.1 63.6 MASK-3 0 4 89.2 76.2 62.4 MASK-4 4 0 92.4 78.4 64.8 MASK-5 2 2 92.6 79.3 65.4 MASK-6 3 3 93.2 79.8 67.2 MASK-7 4 4 94.6 80.3 68.3

[0066] The training parameters in Step 3 are located in the `train.py` configuration file of the mask R-CNN toolkit. The number of images extracted by the BA (TCHS_SIZE) is set to 2, indicating that two images will be randomly selected as a group and fed into the model for each training iteration. The training epoch is set to 100, meaning that all images in the training dataset will be used in the model for 100 training iterations. The learning rate is set to 0.0001; step is set to 30; and the weight decay coefficient (scales) is set to 0.1, indicating that the initial learning rate is 0.0001, and the learning rate decays to 0.1 times the current learning rate after 30 training iterations.

[0067] like Figure 9 As shown, the training model described in Step 3 mainly involves the following steps:

[0068] Step 3.1: Select the MASK-7 network model with the best performance in the ablation experiment as the training model, and set the training parameters for Step 3.

[0069] Step 3.2: Train the improved Mask R-CNN model with the best performance using the Mask R-CNN toolkit; if this is the first training iteration, proceed to Step 3.3; otherwise, proceed to Step 3.4. Figure 10 As shown, the object being trained is the manually labeled bounding box parameters, and the training result is represented by the weight parameters obtained after training; the bounding box parameters are the labeled targets surrounded by continuous labeled points, and their shape is an irregular closed bounding circle.

[0070] Step 3.3: The specific training steps for the initial training of the Mask-7 network model are as follows:

[0071] Step 3.3.1: Increase the training epoch in the training parameters of Step 3, that is, adjust the epoch to 200 epochs (or more), and keep the other parameters unchanged. The purpose is to use a larger training epoch to expand the optimization space of the model and prevent training to a local optimum that would affect the performance.

[0072] Step 3.3.2: No pre-trained weight file needs to be loaded; the training dataset will be used as the training sample for this training.

[0073] Step 3.3.3: Each image from the training samples described in Step 3.3.2 is sequentially fed into the Mask-7 network model, and the weight parameters are updated. Specifically, the forward propagation calculation of the Mask-7 network model is performed using randomly initialized weight parameters to obtain a set of intermediate parameters. Then, the intermediate parameters are used to perform backpropagation calculation to obtain a new set of weight parameters. The new weight parameters will iterate over the old weight parameters used to calculate the forward propagation. The iteration of the old weight parameters with the new weight parameters is the form of the training model.

[0074] Step 3.3.4: The process of completing one forward and backward propagation for all images in the training dataset described in Step 3.3.3 is recorded as one round of training the best-performing MASK-7 network model. Repeat Step 3.3.2 to Step 3.3.3 until the network model has been trained for 200 rounds.

[0075] Step 3.3.5: Using the test dataset, select the MASK-7_best.weights weight file with the best training effect throughout the entire process in Step 3.3.4, and generate the MASK-7.BEST pre-trained weight file from this file.

[0076] Step 3.4: When retraining the MASK-7 network model, the specific training steps are as follows:

[0077] Step 3.4.1: Adjust the training parameters according to Step 3 (when training again, the pre-trained weights are loaded, which is equivalent to continuing the optimization based on the previous optimization. Therefore, the training period set in Step 3 is less than that of the first training to save training time).

[0078] Step 3.4.2: Load the MASK-7.BEST pre-trained weight file and use the training dataset as the sample for this training.

[0079] Step 3.4.3: Each image in the training dataset described in Step 3.4.2 is fed into the MASK-7 network model one by one to update the weight parameters. The weight parameter update is specifically as follows: the MASK-7.BEST pre-trained weight file described in Step 3.4.2 is used as the initial weight parameters to perform forward propagation calculation of the MASK-7 network model and obtain a set of intermediate parameters. Then, the intermediate parameters are used to perform backpropagation calculation to obtain a new set of weight parameters. The new weight parameters will iterate over the old weight parameters used to calculate the forward propagation.

[0080] Step 3.4.4: The process of completing one forward and backward propagation for all images in the training dataset described in Step 3.4.2 is recorded as one round of training the best-performing MASK-7 network model. Repeat Step 3.4.2 to Step 3.4.3 until the network model has been trained for 100 training epochs.

[0081] Step 3.4.5: After the entire training is completed in Step 3.4.4, use the test dataset to select the MASK-7_best.weight weight file with the best performance.

[0082] In this embodiment, the initial parameters of the real-time monitoring module described in Step 4 include the camera's on-to-off time, image acquisition speed, pixel size, and memory space. Specifically, the camera's on-to-off time is 1000 seconds, one 2590×1942 image is acquired per second, and the memory space is set to 500M.

[0083] Step 5 uses the MASK-7_best.weight file from Step 3.4.5 as its weights. This file is loaded into the MASK-7 network model for image detection and segmentation, yielding detection and segmentation results. The detection results are used to determine whether the display screen to be detected has defects and to display the defect detection results; the segmentation results are used to segment the defect contours from the target prediction bounding box.

[0084] like Figure 11 As shown, the MASK-7 loaded with the MASK-7_best.weight weight file performs the detection of display defects in real-time video captured by the camera. The detection result indicates that a defect exists. The detection result includes the target prediction box (box) of the defect, the type of defect is spot, the confidence level is 0.958, and the segmentation result is the defect segmentation contour (irregular closed lines).

[0085] According to another aspect of the present invention, a real-time segmentation method for display screen defects under multiple backgrounds based on an improved Mask R-CNN is also provided, comprising:

[0086] The acquisition module is used to acquire defective images of multi-background display screens, construct a sample dataset, and divide the sample dataset into a training dataset and a test dataset.

[0087] The selection module is used to combine the feature extraction network and / or feature fusion network in the traditional Mask R-CNN model with the attention mechanism module to create multiple improved Mask R-CNN models; ablation experiments are performed on multiple improved Mask R-CNN models to select the improved Mask R-CNN model with the best performance;

[0088] The training module is used to initialize the training parameters of the improved Mask R-CNN model with the best performance, and then train the improved Mask R-CNN model with the best performance to obtain the optimal weights.

[0089] The execution module is used to acquire images from the display screen, load the obtained optimal weights into the improved Mask R-CNN model with the best performance, and make predictions on the acquired images from the display screen to obtain prediction results.

[0090] According to another aspect of the present invention, a terminal device is provided, including a memory, a processor, and a program stored in the memory and executable by the processor, wherein the processor executes the program to implement the real-time segmentation method for display defects in multiple backgrounds based on the improved Mask R-CNN described in any one of the above.

[0091] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the real-time segmentation method for display defects in multiple backgrounds based on the improved Mask R-CNN described in any one of the above.

[0092] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A real-time segmentation method for display screen defects under multiple backgrounds based on an improved Mask R-CNN, characterized in that: include: Acquire defective images of multi-background displays, construct a sample dataset, and divide the sample dataset into a training dataset and a test dataset; By combining the feature extraction network and feature fusion network in the traditional Mask R-CNN model with the attention mechanism module, several improved Mask R-CNN models were created. Ablation experiments were conducted on these improved Mask R-CNN models to select the one with the best performance. The optimal performance improved Mask R-CNN model is initialized with training parameters, and then trained to obtain the optimal weights. The system captures images from the display screen, loads the obtained optimal weights into the improved Mask R-CNN model with the best performance, and makes predictions on the captured images to obtain prediction results. The method of combining the feature extraction network and feature fusion network in the traditional Mask R-CNN model with the attention mechanism module includes: An improved feature extraction network in the Mask R-CNN model is constructed by combining the feature extraction network in the traditional Mask R-CNN model with the attention mechanism module. An improved feature fusion network in the Mask R-CNN model is constructed by combining the feature fusion network in the traditional Mask R-CNN model with the attention mechanism module. The improved Mask R-CNN model includes a traditional Mask R-CNN feature extraction network and m attention mechanism modules in its feature extraction network, and a traditional Mask R-CNN feature fusion network and n attention mechanism modules in its feature fusion network. Multiple improved Mask R-CNN models are obtained by changing the number of attention mechanism modules in the feature extraction network and the number of attention mechanism modules in the feature fusion network. Where m, n = 1, 2, 3, 4. The specific method of combining the feature extraction network and attention mechanism module in the traditional Mask R-CNN model is as follows: the feature layer output by the residual network module of the feature extraction network in the traditional Mask R-CNN model is passed through the attention mechanism module to obtain the output feature layer; The specific method of combining the feature fusion network and attention mechanism module in the traditional Mask R-CNN model is as follows: the input of the feature fusion network in the traditional Mask R-CNN model is added to the attention mechanism module after passing through a two-dimensional convolution operation.

2. The real-time segmentation method for display screen defects under multiple backgrounds based on improved Mask R-CNN according to claim 1, characterized in that: The process of training the optimal improved Mask R-CNN model is as follows: if it is the first training, the training period of the training parameters is increased, and the optimal improved Mask R-CNN model is trained for the first time using the increased training period to obtain the pre-trained weight file. If retraining is required, the training epoch in the training parameters will be adjusted to the initial settings, and the pre-trained weights will be loaded into the improved Mask R-CNN model with the best performance for retraining to obtain the optimal weights.

3. The real-time segmentation method for display screen defects under multiple backgrounds based on improved Mask R-CNN according to claim 2, characterized in that: The training epochs for the first training of the optimal-performing improved Mask R-CNN model are longer than the initial setting value. The training epochs for selecting the optimal-performing improved Mask R-CNN model from multiple improved Mask R-CNN models through ablation experiments are shorter than the initial setting value.

4. A real-time segmentation system for display screen defects under multiple backgrounds based on an improved Mask R-CNN, characterized in that: include: The acquisition module is used to acquire defective images of multi-background display screens, construct a sample dataset, and divide the sample dataset into a training dataset and a test dataset. The selection module is used to combine the feature extraction network and / or feature fusion network in the traditional Mask R-CNN model with the attention mechanism module to create multiple improved Mask R-CNN models; ablation experiments are performed on multiple improved Mask R-CNN models to select the improved Mask R-CNN model with the best performance; The training module is used to initialize the training parameters of the improved Mask R-CNN model with the best performance, and then train the improved Mask R-CNN model with the best performance to obtain the optimal weights. The execution module is used to acquire images from the display screen, load the obtained optimal weights into the improved Mask R-CNN model with the best performance, predict the images from the acquired display screen, and obtain the prediction results. The method of combining the feature extraction network and feature fusion network in the traditional Mask R-CNN model with the attention mechanism module includes: An improved feature extraction network in the Mask R-CNN model is constructed by combining the feature extraction network in the traditional Mask R-CNN model with the attention mechanism module. An improved feature fusion network in the Mask R-CNN model is constructed by combining the feature fusion network in the traditional Mask R-CNN model with the attention mechanism module. The improved Mask R-CNN model includes a traditional Mask R-CNN feature extraction network and m attention mechanism modules in its feature extraction network, and a traditional Mask R-CNN feature fusion network and n attention mechanism modules in its feature fusion network. Multiple improved Mask R-CNN models are obtained by changing the number of attention mechanism modules in the feature extraction network and the number of attention mechanism modules in the feature fusion network. Where m, n = 1, 2, 3, 4. The specific method of combining the feature extraction network and attention mechanism module in the traditional Mask R-CNN model is as follows: the feature layer output by the residual network module of the feature extraction network in the traditional Mask R-CNN model is passed through the attention mechanism module to obtain the output feature layer; The specific method of combining the feature fusion network and attention mechanism module in the traditional Mask R-CNN model is as follows: the input of the feature fusion network in the traditional Mask R-CNN model is added to the attention mechanism module after passing through a two-dimensional convolution operation.

5. A terminal device, characterized in that: The method includes a memory, a processor, and a program stored in the memory and executable by the processor, wherein the processor executes the program to implement the real-time segmentation method for display defects in multiple backgrounds based on an improved Mask R-CNN as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the real-time segmentation method for display defects in multiple backgrounds based on an improved Mask R-CNN, as described in any one of claims 1-3.

Images