A general keyboard anomaly detection method

Abstract

The application provides a general keyboard anomaly detection method, comprising the following steps: a model training stage, through picture collection, correction module and data labeling processing, a classification model and a detection model are obtained; a model application stage, each key of the image keyboard is defined with a corresponding type, and a None type representing an abnormal key position type is added; a detection picture is read, and a correction module, a classification module and a detection module are sequentially operated on the detection picture, and detection information of each key is output; according to target coordinate information, category information, keyboard row inference labels and vertical coordinate offset threshold values output by a detection algorithm, the target coordinate information, the category information, the keyboard row inference labels and the vertical coordinate offset threshold values are compared with real label values of the type of product, and result data is output, the application supports self-defined multiple types of keyboards, and the compatibility is greatly improved. The detection efficiency is greatly improved, and the detection is efficiently completed. The influence of the external environment is reduced, and the stability of keyboard detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of keyboard anomaly detection, in particular to a general keyboard anomaly detection method. BACKGROUND

[0002] In the process of keyboard production, keyboard surface defect detection is a complex, repetitive, and highly concentrated work, which brings great pressure to the detection personnel; at the same time, the detection personnel are inevitably affected by fatigue, mood, feeling and technical level, and it is difficult to achieve accuracy and quantification; in order to reduce the labor intensity of the detection personnel in the detection process and improve the production efficiency, the keyboard detection algorithm is correspondingly generated.

[0003] At present, many domestic and foreign enterprises that research machine vision have developed corresponding traditional detection algorithms, which can automatically identify and detect keyboard keys after simple setting, but there is still room for improvement in compatibility and detection stability. The current keyboard key detection is mainly detected by using traditional algorithms, which may have problems such as unstable detection, detection limitations, external exposure influence, low detection efficiency, etc. in the detection process of traditional algorithms, which will cause many inconveniences to the users.

[0004] Therefore, based on this, it is of great practical significance to propose a general and efficient and stable keyboard detection algorithm. SUMMARY

[0005] In the traditional keyboard key detection process, there are certain limitations, instability and other technical problems, and the purpose of the present application is to solve the problem by using an algorithm based on deep learning. The algorithm can better detect and determine in terms of efficiency and stability. Through the algorithm, most of the needs of keyboard detection can be met, and due to the use of GPU, efficient reasoning and application can be accelerated.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] A general keyboard anomaly detection method comprises the following steps:

[0008] In the model training stage, the classification model and the detection model are obtained through image acquisition, correction module and data labeling processing;

[0009] In the model application stage, the corresponding type of each key of the image keyboard is defined, and the None type is added to represent the abnormal key type;

[0010] Read the detection picture, and sequentially perform the correction module, classification module and detection module on it, and output the detection information of each key;

[0011] According to the target coordinate information, the category information, the keyboard each line inference label, and the vertical coordinate offset threshold output by the detection algorithm, the result data is output by comparing with the real label value of the type of product.

[0012] As a further scheme of the present application, in the detection process, the picture is corrected by the correction module to make the picture face up, and then the classification module is used to judge the keyboard type, so that the keyboard label information list corresponding to the product is found, and then the detection module is used to detect the keyboard label information list to obtain the key related data.

[0013] As a further scheme of the present application, the target information after the detection information output is saved, including the left upper and right lower horizontal and vertical coordinates of each target point and the category of the target point, and screening is performed, the left upper horizontal and vertical coordinate information of each target, that is, t1(x1, y1) at this time, wherein x1 is the left upper horizontal coordinate, y1 is the left upper vertical coordinate, and the category information of each target are divided, and are defined as two tables.

[0014] As a further scheme of the present application, the outer layer simultaneously traverses the two tables, saves the first element traversed by the current two tables to a temporary table, and pops / removes the current element from the original table.

[0015] As a further scheme of the present application, the inner layer continues to simultaneously traverse the two tables, and the inner layer traverses and judges: the vertical coordinates of each target are subtracted from the vertical coordinates of the first element in the outer layer traversal and the absolute value is taken, if the difference is less than the threshold value, it is determined that they are in the same row, and they are added to the temporary table, and the format is [[[x1, y1, cl], [x2, y2, c2]], [], [],..], that is, the elements belonging to the same row are added to the same list, and after being added to the table, they are deleted from the original table.

[0016] As a further scheme of the present application, the above-mentioned inner layer traversal operation is continued until the traversal ends, and the values in the same row are combined into the same list.

[0017] As a further scheme of the present application, the second round of outer layer traversal is performed, at this time, the two tables traversed have filtered out all the data of the first round of outer layer traversal, and the above-mentioned steps of outer layer simultaneous traversal to inner layer traversal operation are repeated until the traversal ends.

[0018] As a further scheme of the present application, until all the data have ended the traversal, the left upper horizontal and vertical coordinate information and the category information of each row of data are stored in the nested list in the final temporary table, and the format is [[x1, y1, c1], [x2, y2, c2],...], and the length is the total number of rows of the inference result keyboard.

[0019] As a further scheme of the present application, the data of each group in the temporary table is sorted according to the size of the horizontal coordinate and the vertical coordinate, the vertical coordinate sorting is used to arrange each row of data in the order from top to bottom, and the horizontal coordinate sorting is used to arrange each key position in the same row in the order from left to right; the category value in each group of sorted data is taken out and arranged at intervals.

[0020] As a further scheme of the present application, the real value and the inference value of the category of each row of the keyboard after merging are compared, and according to the comparison result, it is judged whether they are all True, and if so, the keyboard is normal, otherwise the keyboard is abnormal.

[0021] The present application has the following beneficial effects:

[0022] The present application supports customizing multiple types of keyboards, greatly improves compatibility, greatly improves detection efficiency, efficiently completes detection, reduces the influence of external environment, and improves the stability of keyboard detection.

[0023] The present application proposes a set of efficient and stable scheme and successfully applies it to the keyboard detection project, especially in the post-processing stage, a new idea of data segmentation and recombination is proposed, and a post-processing scheme controlled by horizontal coordinate and vertical coordinate offset threshold is also proposed, which greatly improves the stability of detection, and through the module, the accuracy of the model can be improved, and some over-detection behaviors caused by the model can be filtered.

[0024] To make the structure characteristics and effects of the present application clearer, the present application will be described in detail below in combination with the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The keyboard label information list provided by the present application.

[0026] Figure 2 The single key annotation information graph provided by the present application.

[0027] Figure 3 The product sample graph provided by the present application.

[0028] Figure 4 The annotation graph provided by the present application.

[0029] Figure 5 The key row graph provided by the present application.

[0030] Figure 6 The detection result graph provided by the present application.

[0031] Figure 7 The model training flowchart provided by the present application.

[0032] Figure 8 is the algorithm detection flowchart provided by the present application. DETAILED DESCRIPTION

[0033] The present application will be further described below in conjunction with the drawings and relevant knowledge, and a clear and complete description will be made. Obviously, the described application is only a part of the embodiments of the present application, not all embodiments.

[0034] Embodiment 1

[0035] The present application provides a general keyboard anomaly detection method, comprising the following steps:

[0036] In the model training stage, by collecting pictures, through the correction module and data labeling processing, the classification model and the detection model are obtained. Specifically, in the detection process, the picture is corrected by the correction module to make the picture face up, and then the classification module is used to judge the keyboard category, so as to find the corresponding keyboard label information list of the product, and then the detection module is used to detect it to obtain the key related data.

[0037] In the model application stage, the corresponding type of each key of the image keyboard is defined, and the None type representing the abnormal key type is added.

[0038] The detection picture is read, and the correction module, the classification module and the detection module are operated in sequence, and the detection information of each key is output. Specifically, the target information after saving the detection information output is included, including the left upper, right lower horizontal and vertical coordinates of each target point and the category of the target point, and the selection is performed. The left upper horizontal and vertical coordinate information of each target, that is, t1(x1,y1) at this time, is split, wherein x1 is the left upper horizontal coordinate, y1 is the left upper vertical coordinate, and the category information of each target is defined as two tables.

[0039] According to the target coordinate information, category information, keyboard each line inference label and vertical coordinate offset threshold output by the detection algorithm, the result data is output by comparing with the real label value of the type product.

[0040] In the present application, the outer layer simultaneously traverses two tables, saves the first element traversed by the current two tables to a temporary table, and pops / removes the current element from the original table.

[0041] In the present application, the inner layer continues to simultaneously loop through the two tables, the inner loop, and the judgment: the ordinate of each target is subtracted from the ordinate of the first element in the outer loop and the absolute value is taken, if the difference is less than the threshold value, it is determined to be the same row, and it is determined to be the same row, and it is added to the temporary table, which is in the format of [[ [x1, y1, cl], [x2, y2, c2] ], [], [],.. ], that is, elements belonging to the same row are added to the same list, and after being added to the table, it is deleted from the original table.

[0042] In the present application, the above-mentioned inner loop operation is continued until the end of the loop, the values of the same row are combined into the same list, and the second round of outer loop is performed, at this time, the two tables traversed in the first round of outer loop are filtered out, and the above-mentioned steps of outer loop and inner loop are repeated until the end of the loop.

[0043] In the present application, until all the data has been traversed, the upper left corner horizontal and vertical coordinate information and category information of each row of data are stored in the nested list in the final temporary table, and the format is [[x1, y1, c1], [x2, y2, c2],...], and the length is the total number of rows of the keyboard.

[0044] In the present application, the data in each group in the temporary table is sorted according to the size of the horizontal coordinate and the vertical coordinate, the vertical coordinate is sorted to arrange each row of data in order from top to bottom, and the horizontal coordinate is sorted to arrange each key in the same row in order from left to right; the category value in each sorted group of data is taken out and arranged at intervals.

[0045] In the present application, the real value and the inference value of the category of each row of the keyboard after being combined are compared, and according to the comparison result, it is judged whether they are all True, and the keyboard is normal, otherwise the keyboard is abnormal.

[0046] The present application proposes a set of efficient and stable scheme and successfully applies it to the keyboard detection project, especially in the post-processing stage, a new idea of data segmentation and recombination is proposed, and a post-processing scheme controlled by horizontal coordinate and vertical coordinate offset threshold is also proposed, which greatly improves the stability of detection, through the module, the accuracy of the model can be improved, and some over-detection behaviors caused by the model can be filtered.

[0047] Embodiment 2

[0048] A general keyboard abnormality detection method, comprising: a model training stage, collecting a certain number of pictures, first passing through a correction module and data labeling processing, product original picture and labeled picture are respectively as Figure 3 and Figure 4As shown, two high-quality models, a classification model (cls) and a detection model (det), are obtained.

[0049] In the model application stage, the type of each key of the image keyboard is defined, and a None type is added to represent an abnormal key type. The defined key type map is shown in the accompanying Figure 1 As shown, the relevant annotation information of the keys is shown in the accompanying Figure 2 As shown, each key corresponds to a type, and the order is from left to right and from top to bottom. After the preparation work is completed, the detection is started. First, the picture is read, and the correction module, the classification module, and the detection module are operated in sequence, and the detection information of each key is output. Then, the post-processing operation is performed. According to the target coordinate information, the category information, the keyboard line inference label, and the vertical coordinate offset threshold value output by the detection algorithm, the real label value of the product of this type is compared, and finally, the result data is returned. The related core flowchart can be seen in the accompanying Figure 8 As shown, the product detection result is shown in the accompanying Figure 6 As shown.

[0050] In this embodiment, the pre-processing algorithm specifically includes: in the detection process, first, the picture is corrected by the correction module to make the picture face up, and then the classification module (cls) is used to judge the keyboard category, so as to find the keyboard label information list corresponding to the product, and then the detection module (det) is used to detect it to obtain the key related data, and the pre-processing ends.

[0051] In this embodiment, the post-processing algorithm specifically includes:

[0052] Preconditions:

[0053] input_data=[t1*,t2*,t3*,t4*,....tn*]#save the target information after model detection, including the left upper, right lower horizontal and vertical coordinates of each target point and the category of the target point.

[0054] i_data_list=[t1,t2,t2,t3,t4,...tn]#filter input_data, only save the left upper horizontal and vertical coordinate information of each target, that is, t1(x1,y1) at this time, where x1 is the left upper horizontal coordinate and y1 is the left upper vertical coordinate.

[0055] i_class_list=[i1,i2,i3,i4,...in]#filter input_data, only save the category information of each target.

[0056] label_row=[L1, L2, L3, L4,..Lm] # represents the result of the keyboard each row of the category after merging (true value), for example, the second row of the keyboard is “~_1_2_3_4_..._-” represents the keyboard of all categories after merging the results of the second row (part of the keyboard category can be re-normalized, such as ~ can be replaced by “num”), where the subscript m represents the total number of rows of the keyboard

[0057] inference_row=[I1, I2, I3, I4,..Im] # represents the result of the keyboard each row of the category after merging (inference value), the list length is consistent with label_row. (The last abnormal result will be judged by comparing the list with label_row)

[0058] offset_thresh_y # offset threshold of the vertical coordinate, less than the threshold is determined as the same row

[0059] result_tmp_list # temporary table for temporary results

[0060] result_final_list # represents the result of comparing label_row with inference_row, the length is consistent with the two tables,

[0061] The post-processing algorithm is as follows:

[0062] 1. Split the input_data data into i_data_list and i_class_list two tables

[0063] 2. Outer loop simultaneously traverses i_data_list and i_class_list two tables

[0064] 3. Save the first element traversed by the current two tables to the result_tmp_list table (its format is [x1, y1, c1], where x1, y1 is the horizontal and vertical coordinates, and c1 is the category), and pop / remove the current element from the original table (i_data_list, i_class_list).

[0065] 4. Continue to simultaneously loop through i_data_list and i_class_list two tables (at this time, the two tables have removed the first element of the outer loop)

[0066] 5. Inner layer traversal, judgment: the vertical coordinates in i_data_list are subtracted from the vertical coordinates of the first element in the outer layer traversal and the absolute value is taken, if the difference is less than offset_thresh_y, it is determined to be the same row, and is added to the result_tmp_list table, the format is [[[x1, y1, cl], [x2, y2, c2]], [], [],..], that is, the elements belonging to the same row are added to the same list, and after being added to the table, they are deleted from the original table.

[0067] 6. Continue the inner layer traversal operation of step 5, until the end of the traversal, the values of the same row have been combined into the same list.

[0068] 7. Second round of outer layer traversal, at this time the i_data_list and i_class_list tables have filtered out all the data of the first round of outer loop.

[0069] 8. Repeat steps 2 to 6

[0070] 9. Until all data has been traversed, the final result_tmp_list format is: result_tmp_list = [[], [], [], [],..], the nested list inside stores the top-left horizontal and vertical coordinate information and class information of each row of data, the format is [[x1, y1, c1], [x2, y2, c2],...], the length is the total number of rows of the inference result keyboard

[0071] 10. Sort the data in each group in result_tmp_list according to the horizontal coordinate (x value) and the vertical coordinate (y), the vertical coordinate sorting arranges each row of data in order from top to bottom (from small to large), the horizontal coordinate sorting arranges each key in the same row in order from left to right (from small to large), note that since the keys in the keyboard are not uniform in format, there are different sizes, which can be judged according to the coordinates of the top-left corner of each key to divide the rows, for example, the three keys ←, ↓, → can be treated as a row, as shown in Figure 5 , which is set to the 7th row.

[0072] 11. Take out the class values in each group of data in result_tmp_list that has been sorted, and arrange them according to the "_" interval, for example, the sorted class in result_tmp_list is [1, 2, 3], then it is combined into the string form 1_2_3.

[0073] 12、Repeat the merging operation of step 11, at this time, each row of the merged string represents "I1, I2, I3" in the inference_row table and so on

[0074] 13、By default, set the value of result_final_list to True, for example, if the current keyboard has 3 rows, then result_final_list = [True, True, True]

[0075] 14、Compare the inference_row obtained in step 12 with each row of the label_row, if they are consistent, do not change the value of the corresponding position of result_final_list,

[0076] Suppose: the current keyboard has three rows, and it is known through the comparison of the data of inference_row and label_row tables that they are consistent, then the final output result_final_list is [True, True, True]

[0077] Still take three rows as an example, if the second row is inconsistent after comparison, and the other rows are consistent, then change the value of the second row of result_final_list to False, then the final output result_final_list is [True, False, True]

[0078] 15、Finally, according to result_final_list, judge whether they are all True, then output the keyboard is normal (OK), otherwise, the keyboard is abnormal (NG).

[0079] Since the arrangement is performed through "_", the related position information can be located at the same time when the abnormality is detected.

[0080] Finally, call the model on the client side and receive the returned result.

[0081] The technical principles of the present application are described above in combination with specific embodiments, which are only preferred embodiments of the present application. The protection scope of the present application is not limited to the above-mentioned embodiments only, any technical solution falling within the idea of the present application belongs to the protection scope of the present application. Other specific embodiments of the present application can be thought of by those skilled in the art without creative labor, and these embodiments will fall within the protection scope of the present application.

Claims

1. A general-purpose keyboard anomaly detection method, characterized by, The method comprises the following steps: A model training stage, by collecting pictures, through a correction module and data labeling processing, obtaining a classification model and a detection model, wherein the classification model is used to judge the keyboard category to find the corresponding product keyboard label information list, and the detection model is used to detect and obtain the key related data; In the model application stage, the corresponding type of each key of the image keyboard is defined, and the None type is added to represent the abnormal key position type; Read the detection picture, and sequentially perform the correction module, classification module and detection module on the detection picture, and output the detection information of each key, wherein the detection information includes the upper left horizontal and vertical coordinates, the lower right horizontal and vertical coordinates and the category of each key; According to the target coordinate information, the category information, the keyboard each row inference label and the vertical coordinate offset threshold output by the detection algorithm, the keyboard each row inference label obtained by processing is compared with the real label value of the product corresponding to the keyboard category, and the result data is output.

2. The method of claim 1, wherein the key exception is detected by a key exception detection program. In the detection process, the picture is corrected by the correction module to make the picture face up, and then the classification module is used to judge the keyboard category, so that the corresponding keyboard label information list of the product is found, and then the detection module is used for detection to obtain the key related data.

3. The method of claim 2, wherein the key press is detected by a key press exception method comprising: Save each target information after the detection information output, including the upper left horizontal and vertical coordinates of each target point, the lower right horizontal and vertical coordinates and the category of the target point, and perform screening; the target information is data split to obtain two tables: the first table is i_data_list, which only saves the upper left horizontal and vertical coordinate information of each target, that is, t1 (x1, y1), wherein x1 is the horizontal coordinate of the upper left corner, and y1 is the vertical coordinate of the upper left corner; the second table is i_class_list, which only saves the category information of each target.

4. The method of claim 3, wherein the key press is detected by the processor. The outer layer simultaneously traverses the two tables, saves the first element of the current two tables to a temporary table, and pops / removes the current element from the original table.

5. The method of claim 4, wherein the key press is detected by the processor. The inner layer continues to simultaneously traverse the two tables, and the inner layer traverses and judges: the vertical coordinates of each target are subtracted from the vertical coordinates of the first element in the outer layer traversal and the absolute value is taken, if the difference is less than the threshold value, it is determined that they are in the same row, and they are added to the temporary table, and the format is [[ [x1, y1, cl], [x2, y2, c2] ], [], [],.. ], that is, the elements belonging to the same row are added to the same list, and after being added to the table, they are deleted from the original table.

6. The method of claim 5, wherein the key press is detected by the processor. Continue the above-mentioned inner layer traversal operation until the end of the traversal, and the values in the same row are combined into the same list.

7. The method of claim 6, wherein the key press is detected by the processor. The second round of outer layer traversal, at this time, the two tables traversed have filtered out all the data of the first round of outer layer loop, and the above-mentioned outer layer simultaneous traversal to inner layer traversal operation steps are repeated until the end of the traversal.

8. The method of claim 7, wherein the key press is detected by the processor. Until all the data has been traversed, the last temporary table contains the upper left corner coordinates and category information of each row of data in the nested list, in the format of [[x1, y1, c1], [x2, y2, c2],... ], and the length is the total number of rows of the keyboard.

9. The method of claim 8, wherein the key press is detected by a key press detection module. Sort the data in each group in the temporary table according to the size of the horizontal and vertical coordinates. The vertical coordinate sorting arranges each row of data in the order from top to bottom, and the horizontal coordinate sorting arranges each key in the same row in the order from left to right. The category value in each sorted group of data is extracted and arranged at intervals.

10. The method of claim 9, wherein the key press is detected by a key press exception method comprising: Compare the real value and the inference value of the category of each row of the keyboard after merging. According to the comparison result, it is judged whether all are True. If so, the keyboard is normal, otherwise the keyboard is abnormal. ​

Images