Intelligent lock panel display abnormity detection method and system based on AI visual technology

Abstract

The invention provides an intelligent lock panel display abnormity detection method and system based on an AI visual technology, and the method comprises the following steps: 1, classifying the defect types of an intelligent lock panel, including the logo color display error of the intelligent lock panel, the digital color display error, the black spot defect of digital display, and incomplete display; 2, acquiring an intelligent lock panel image to form a data set; step 3, constructing and training a model, wherein the model comprises a YOLOv3 network and 11 ResNet18 network classification models; the YOLOv3 network is composed of a Darknet-53 network and three branch network locks, the Darknet-53 network is a basic network, and the three branch networks are used for carrying out prediction of three scales respectively; after the images pass through the YOLOv3 network, the YOLOv3 model cuts the identified category images and sends the identified category images to the ResNet18 network; and 4, evaluating the model performance through the model evaluation index so as to obtain an accurate detection result. By applying the technical scheme, the defect accuracy of the intelligent lock panel can be improved.

Description

Technical Field

[0001] The present invention relates to the field of smart lock panel detection technology, and in particular to a method and system for detecting smart lock panel display anomalies based on AI vision technology. Background Art

[0002] In the era of human-machine integration and intelligence, smart locks are the product of the fusion of multiple technologies, including electronics, the Internet of Things, and artificial intelligence. They are a transformation and development of traditional mechanical locks, which only require a single key to unlock. This key is not only inconvenient to carry around but also easily copied, posing a potential safety hazard to household property. As a necessity for future smart homes, smart locks have enormous future market potential.

[0003] The most important difference between smart locks and traditional locks is that traditional locks lack human-computer interaction. The reason why smart locks are smart is that they can provide human-computer interaction. If you want to achieve interaction between locks and people, the most indispensable thing must be the display panel.

[0004] The display panel is an essential part of every smart lock. During manufacturing, defective or faulty display panels are unavoidable. To prevent these defective or faulty products from entering the market, they must be inspected before shipment. Traditionally, this involves manual spot checks, which are costly and suffer from inconsistent inspection quality. The smart lock industry holds promising prospects. Developing a system that replaces manual inspection of smart lock panels with a machine would undoubtedly improve production and quality for companies. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method and system for detecting display anomalies of smart lock panels based on AI vision technology to improve the accuracy of smart lock panel defects.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for detecting display anomalies of smart lock panels based on AI vision technology, comprising the following steps:

[0007] Step 1: Classify the types of smart lock panel defects, including incorrect logo color display, incorrect digital color display, black dot defects in digital display, and incomplete display;

[0008] Step 2: Collect smart lock panel images to form a data set;

[0009] Step 3: Build and train a model consisting of a YOLOv3 network and 11 ResNet18 network classification models; the YOLOv3 network consists of a Darknet-53 network and three branch networks. The Darknet-53 is the base network, and the three branch networks perform predictions at three different scales. After the image passes through the YOLOv3 network, the YOLOv3 model will crop the identified category image and feed it into the ResNet18 network. The ResNet18 network consists of seven parts: an input layer, a convolutional layer, four residual blocks, and a fully connected layer output layer.

[0010] Step 4: Evaluate the model performance through model evaluation indicators to obtain accurate detection results.

[0011] In a preferred embodiment, the data set is collected by K210, and the smart lock panel is photographed from multiple angles. The collected image format is a three-channel RGB jpg file. A total of 1,377 pictures are taken, and 217 blurred pictures are filtered out to remove them. Finally, 1,160 training pictures are obtained, 125 pictures are used for verification, and 171 pictures are used for testing; the 2,160 pictures are renamed by writing a python script, and their labels are 0-2159.

[0012] In a preferred embodiment, a data set in the VOC2007 format is used. Under the directory structure of the VOC2007 format, there are three subdirectories, namely JPEGImages, Annotations, and ImageSets; the JPEGImages directory stores all images used for training, and the format does not have to be jpg format files; the Annotations directory stores all XML format annotations corresponding to the image names in the first directory; the ImageSets directory stores txt files for dividing the data set, and a new Main subdirectory needs to be created in the third directory. Two txt files are stored in the Main subdirectory, namely train.txt and val.txt; each line in train.txt writes the name of a training image, and each line in val.txt writes the name of a verification image.

[0013] In a preferred embodiment, the YOLOv3 target detection model has a total of 11 category labels, namely "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", and "logo"; each category label corresponds to a ResNet18 network classification model.

[0014] In a preferred embodiment, the final results are divided into four types according to the model prediction results and the actual results: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN); TP means predicting a positive class as a positive class, TN means predicting a negative class as a negative class, FP means predicting a negative class as a positive class, and FN means predicting a positive class as a negative class. The definition formulas of precision and recall are as follows:

[0015]

[0016] In a preferred embodiment, the average precision (mAP) is used to represent the accuracy of the prediction results of all categories, and AP represents the accuracy of the prediction results of each category. mAP is the average value of all APs added together. Before using the mAP evaluation model, the intersection over union (IoU) needs to be set. The IoU is the degree of overlap between the predicted box and the actual marked box. The larger the IoU, the better. The formulas for AP and mAP are as follows:

[0017]

[0018]

[0019] In a preferred embodiment, the model loss function is as follows:

[0020]

[0021] H(p,q) is the cross entropy function used to measure the similarity between P and q, where p represents the distribution of true labels and q is the distribution of predicted labels of the trained model. i is the training sample, and n is the number of samples.

[0022] The present invention also provides a smart lock panel display anomaly detection system based on AI vision technology, which adopts the smart lock panel display anomaly detection method based on AI vision technology, including: a visual processor, an image acquisition lens, an LCD display panel and a light source; the visual processor is connected to the image acquisition lens and the LCD display panel; the image acquisition lens is used to capture the image of the smart lock panel, the visual processor is used to process image information, the LCD display panel is used to display the detection results, and the light source is used to fill the smart lock panel with light.

[0023] Compared with existing technologies, this invention has the following advantages: it uses machine learning to perform real-time model inference on camera-captured images to generate detection results. It also uses a custom-built smart lock panel dataset for model training and analyzes the model's performance. This technical solution can improve the accuracy of smart lock panel defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The overall structural framework of the system of the preferred embodiment of the present invention;

[0025] Figure 2 Schematic diagram of the defect types of the smart lock panel in the preferred embodiment of the present invention, where (a) is a logo color display error, (b) is a logo color display error, (c) is a logo color display error and the numbers 0, 2, and 7 are displayed incorrectly, (d) is a logo color display error and the numbers 0 and 9 are displayed incorrectly, (e) is the number 0 not displayed, and (f) is the number 8 not displayed;

[0026] Figure 3 A partial image of a data set according to a preferred embodiment of the present invention;

[0027] Figure 4 The data set directory structure of the preferred embodiment of the present invention;

[0028] Figure 5 A model structure diagram of a preferred embodiment of the present invention;

[0029] Figure 6 Flow chart of the experimental process of the preferred embodiment of the present invention;

[0030] Figure 7 This is the YOLOv3 model loss function image of the preferred embodiment of the present invention;

[0031] Figure 8 This is the model test result on PC of the preferred embodiment of the present invention;

[0032] Figure 9 This is a test diagram of the ResNet18 network classification model in the preferred embodiment of the present invention;

[0033] Figure 10 The following is a comparison of PC test results and actual test results of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0037] Smart lock panel display anomaly detection method based on AI vision technology, reference Figure 1-10 , including the following steps:

[0038] Step 1: Classify the types of smart lock panel defects, including incorrect logo color display, incorrect digital color display, black dot defects in digital display, and incomplete display;

[0039] Step 2: Collect smart lock panel images to form a data set;

[0040] Step 3: Build and train a model consisting of a YOLOv3 network and 11 ResNet18 network classification models; the YOLOv3 network consists of a Darknet-53 network and three branch networks. The Darknet-53 is the base network, and the three branch networks perform predictions at three different scales. After the image passes through the YOLOv3 network, the YOLOv3 model will crop the identified category image and feed it into the ResNet18 network. The ResNet18 network consists of seven parts: an input layer, a convolutional layer, four residual blocks, and a fully connected layer output layer.

[0041] Step 4: Evaluate the model performance through model evaluation indicators to obtain accurate detection results.

[0042] Specifically, in order to obtain a deep learning target detection model, the collection of data sets is essential. To train a target detection model, it is necessary to collect a large number of pictures of the target to be detected before training. The data set must be taken in sufficient quantities and in multiple angles and background environments. This can prevent the data from being overfitted and causing insufficient generalization of the model, and improve the robustness of the model. This design uses K210 to collect data sets, and shoots the smart lock panel from multiple angles. The collected image format is a three-channel RGB jpg file. A total of 1,377 pictures were taken, and 217 blurred pictures were filtered out. Finally, 1,160 training pictures were obtained, 125 pictures were used for verification, and 171 pictures were used for testing. The 2,160 pictures were renamed by writing a python script, and their labels are 0-2159. Some of the pictures in the data set are as follows: Figure 3 shown.

[0043] This design uses a dataset in the VOC2007 format. Under the directory structure of the VOC2007 format, there are three subdirectories, namely JPEGImages, Annotations, and ImageSets. The first directory stores all images used for training, and the format does not have to be jpg files. The second directory stores all XML-formatted annotations corresponding to the image names in the first directory. The third directory stores the txt files that divide the dataset. A new Main subdirectory needs to be created in the third directory. Two txt files are stored in the Main subdirectory, namely train.txt and val.txt. Each line in train.txt writes the name of a training image, and each line in val.txt writes the name of a verification image. The directory structure of the dataset is as follows: Figure 4 shown.

[0044] The dataset was annotated using the target detection annotation tool labelImg. The labelImg software has a visualization page that allows for easy generation of XML files with label categories, locations, and target box sizes

[38] . This design intends to use a binary classification method, so a total of 12 models need to be trained to annotate samples using labelImg.

[0045] The model of the present invention uses a YOLOv3 target detection model and 11 ResNet18 network classification models. The target detection model first obtains the various targets to be detected in the input image, and then crops them and sends them to the corresponding category ResNet18 network classification model to complete the detection. The YOLOv3 network consists of a Darknet-53 network and three branch network locks. Darknet-53 is the basic network, and the three branch networks perform predictions at three scales respectively. After the image passes through the YOLOv3 network, the YOLOv3 model will crop the identified category image and send it to the ResNet18 network respectively. The ResNet18 network consists of seven parts, namely the input layer, convolutional layer, four residual blocks and a fully connected layer output layer, and finally outputs the detection results.

[0046] YOLOv3 target detection model has a total of 11 class labels, which are "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "logo". Each class label corresponds to a ResNet18 network classification model. The YOLOv3 network training parameters are initialized as follows: the initial learning rate is 0.0001, the batch size is set to 6, and the epoch training cycle is 260. The ResNet18 classification network model training parameters are set as follows: the learning rate is 0.001, the batch size is 6, and the epoch training cycle is 100.

[0047] Two evaluation criteria are used in this paper to evaluate the performance of the model, namely precision and recall. According to the model prediction results and actual results, the final results can be divided into four types: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). TP is to predict positive class as positive class, TN is to predict negative class as negative class, FP is to predict negative class as positive class, and FN is to predict positive class as negative class

[39] . The definition formulas of precision and recall are as follows:

[0048]

[0049] The mean average precision (mAP) is used to represent the prediction accuracy of all classes, and AP represents the prediction accuracy of each class, so mAP is the average of all AP. Before using mAP to evaluate the model, the Intersection over Union (IoU) needs to be set. The Intersection over Union is the degree of overlap between the predicted box and the actual labeled box, and the larger the Intersection over Union, the better the prediction result. In actual situations, an Intersection over Union greater than or equal to 0.5 is considered a good prediction result. The AP, mAP formulas are as follows:

[0050]

[0051] The loss function is also an indicator that describes the performance of the model. Simply put, each input will get a predicted value after model inference, and the difference between the predicted value and the actual value is called loss. The smaller the loss, the better the model. There are many types of loss functions. In essence, they are all the same type of functions that calculate the difference between the predicted value and the actual value. The CrossEntropyLoss function (cross entropy function) is used as the loss function in the classification error. The concept of cross entropy comes from information theory, which is used to measure the difference between two probability distributions. It is implemented in Pytorch through the two lines of code entroy = nn.CrossEntropyLoss()output = entroy(input, target). The formula is as follows:

[0052]

[0053] H(p,q) is the cross entropy function used to measure the similarity between P and q, where p represents the distribution of true labels and q is the distribution of predicted labels of the trained model. i is the training sample, and n is the number of samples.

[0054] The present invention also provides a smart lock panel display anomaly detection system based on AI vision technology, which adopts the smart lock panel display anomaly detection method based on AI vision technology, including: a visual processor, an image acquisition lens, an LCD display panel and a light source; the visual processor is connected to the image acquisition lens and the LCD display panel; the image acquisition lens is used to capture the image of the smart lock panel, the visual processor is used to process image information, the LCD display panel is used to display the detection results, and the light source is used to fill the smart lock panel with light.

[0055] The core visual processor used in this invention is: the Allwinner v831 chip is the core visual processor, and the processor peripheral circuits also include a CPU reset circuit, a microSD drive circuit, a USB OTG circuit with a Type-C interface, and a serial port circuit, etc. These peripheral interface circuits and the visual processor together constitute the core of the system. The system's software development software uses the MaixPy3 open source software, which is a Python3 software development kit (SDK) based on linux cpython, and uses the Python programming language for embedded software development and programming. The labelImg software is used for labeling and production of data sets. LabelImg is a graphic image annotation tool written in Python. By using the QT graphical interface to annotate the target image, an XML file in the PASCAL VOC format can be obtained. The model is trained under the Pytorch framework, and finally the model is deployed on the hardware platform. According to the two schemes, this design trained a total of 2 YOLOv3 target detection models and 11 ResNet18 network classification models. Self-made data sets were used for model training and testing. The first solution uses a YOLOv3 target detection model to identify the smart lock panel, and the performance of the model is measured by the average accuracy and loss function. The second solution uses a YOLOv3 target detection model to crop the target to be detected in the image and then send it to the corresponding classification network for classification. The YOLOv3 target detection model is measured by the average accuracy and loss function, and the performance of the 11 ResNet18 network classification models is measured by the accuracy of the test set. Finally, according to the test results, the accuracy of Solution 2 is higher than that of Solution 1, and Solution 2 is finally selected for deployment. The experimental process flow chart is as follows Figure 6 shown.

[0056] The detection results of each category model are shown in Table 1. It can be clearly seen from the table that the detection accuracy of each category is above 88%.

[0057] Table 1 Model test results

[0058]

[0059] After deploying the model to the hardware platform, the accuracy of each category was tested. A total of 60 smart lock panels were tested, and the detection accuracy of each category was statistically analyzed and compared with the results of the computer test. The results are as follows: Figure 10 shown.

[0060] Finally, the system was tested on a total of 180 smart lock panels, including 120 panels that displayed normal function and 60 panels that displayed defects. Six rounds of testing were conducted, with 30 panels that displayed normal function and 10 panels that displayed defects randomly sampled for testing in each round. The test results are shown in Table 2.

[0061] Table 2:

[0062]

[0063]

Claims

1. A method for detecting abnormal display on smart lock panels based on AI vision technology, characterized in that The following steps are involved: Step 1: Classify the types of smart lock panel defects, including incorrect logo color display, incorrect digital color display, black dot defects in digital display, and incomplete display; Step 2: Collect smart lock panel images to form a data set; Step 3: Build and train a model consisting of a YOLOv3 network and 11 ResNet18 network classification models; the YOLOv3 network consists of a Darknet-53 network and three branch networks. The Darknet-53 is the base network, and the three branch networks perform predictions at three different scales. After the image passes through the YOLOv3 network, the YOLOv3 model will crop the identified category images and send them to the ResNet18 network respectively; The ResNet18 network consists of seven parts: the input layer, the convolutional layer, four residual blocks, and the fully connected layer output layer; Step 4: Evaluate the model performance through model evaluation indicators to obtain accurate detection results.

2. The method for detecting abnormal display of smart lock panel based on AI vision technology according to claim 1 is characterized in that: The dataset was collected using K210, and the smart lock panel was photographed from multiple angles. The captured images were in three-channel RGB jpg format. A total of 1,377 images were taken, and 217 blurred images were removed. Finally, 1,160 training images were obtained, 125 of which were used for verification and 171 for testing. A Python script was written to rename the 2,160 images, with labels ranging from 0 to 2,159.

3. The method for detecting abnormal display of smart lock panel based on AI vision technology according to claim 2 is characterized in that: The dataset uses the VOC2007 format. The directory structure of the VOC2007 format has three subdirectories: JPEGImages, Annotations, and ImageSets. The JPEGImages directory stores all images used for training, and the format does not have to be jpg files. The Annotations directory stores all XML-formatted annotations corresponding to the image names in the first directory. The ImageSets directory stores txt files that divide the dataset. A new Main subdirectory needs to be created in the third directory, and two txt files, train.txt and val.txt, need to be stored in the Main subdirectory. Each line in train.txt contains the name of a training image, and each line in val.txt contains the name of a validation image.

4. The method for detecting display anomalies of smart lock panels based on AI vision technology according to claim 1, characterized in that: The YOLOv3 target detection model has a total of 11 category labels, namely "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", and "logo". Each category label corresponds to a ResNet18 network classification model.

5. The method for detecting abnormal display of smart lock panel based on AI vision technology according to claim 1, characterized in that: The final results are divided into four types according to the model prediction results and the actual results: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). TP means predicting a positive class as a positive class, TN means predicting a negative class as a negative class, FP means predicting a negative class as a positive class, and FN means predicting a positive class as a negative class. The definition formulas of precision and recall are as follows:

6. The method for detecting abnormal display of smart lock panel based on AI vision technology according to claim 1, characterized in that: The average accuracy mAP is used to represent the accuracy of the prediction results of all categories. AP represents the accuracy of the prediction results of each category. mAP is the average value of all APs. Before using the mAP evaluation model, you need to set the intersection over union (IoU). The IoU is the degree of overlap between the predicted box and the actual marked box. The larger the IoU, the better. The AP and mAP formulas are as follows:

7. The method for detecting abnormal display of smart lock panel based on AI vision technology according to claim 1, characterized in that: The model loss function is as follows: H(p,q) is the cross entropy function used to measure the similarity between P and q, where p represents the distribution of true labels and q is the distribution of predicted labels of the trained model. i is the training sample, and n is the number of samples.

8. The smart lock panel display anomaly detection system based on AI vision technology is characterized by: The method for detecting display anomalies of a smart lock panel based on AI vision technology as described in any one of claims 1 to 7 is adopted, comprising: a visual processor, an image acquisition lens, an LCD display panel and a light source; the visual processor is connected to the image acquisition lens and the LCD display panel; the image acquisition lens is used to capture images of the smart lock panel, the visual processor is used to process image information, the LCD display panel is used to display detection results, and the light source is used to fill light for the smart lock panel.

Images