Machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition

Abstract

The invention discloses a machine-readable card image recognition method based on YOLOv7 model positioning and multi-clustering recognition, which comprises the following steps of: firstly, recognizing a user-defined mark through a YOLOv7 model, determining four corners of a quadrilateral of a selection question region, then extracting an image of a machine-readable card part according to the four corners, then firstly performing trapezoidal correction on the machine-readable card image, and finally performing multi-clustering recognition on the machine-readable card image. Performing denoising, closed operation and graying preprocessing on the image of the region, converting the original image of the machine-readable card to be identified into a black and white binary image, searching rectangles of various scales by adopting the contour in the CV2, then determining each rectangle according to a clustering method, finding out a central point on an X axis and a Y axis, and finally, obtaining the original image of the machine-readable card. And according to the rectangular area ratio of the white points, determining whether filling is carried out, determining a filling position, obtaining an answer, and carrying out comparison. According to the method, verification of multiple examples is carried out, experiments prove that the method is high in automation degree, accurate in recognition and high in speed, the test paper cost is greatly reduced, the paper marking efficiency is improved, and the method has wide popularization value.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a machine-readable card image recognition method based on YOLOv7 model positioning and DBSCAN clustering. Background Art

[0002] Existing card reader technology mainly uses optical scanning to identify printed or written marks in a certain format and converts them into electrical signals that can be accepted by computers, such as cursor readers. Special equipment and personnel maintenance are required. In recent years, image recognition methods for machine-readable cards have also appeared on the market. Overall, there are the following disadvantages: 1) There is no clear standard on the threshold of image binarization, and there are high requirements for image shooting. 2) The machine-readable card area is simply positioned at the four corners of the large rectangular frame when cutting, which cannot avoid the situation where there is ambiguity caused by the table in the title. 3) The positioning of the machine-readable card option box is simply based on the area. It cannot adapt to various shooting scenes. Summary of the invention

[0003] In order to solve the problems in the prior art, the present invention provides a machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering. The steps of the present invention are greatly simplified, and the data can be collected using a mobile phone, which is suitable for self-operation by a large number of teachers. It is especially suitable for test papers with more multiple-choice questions, such as: College English Test Band 4 and Band 6 (multiple-choice questions have 30 choices).

[0004] The present invention provides a method for machine-readable card image recognition based on YOLOv7 model positioning and multiple clustering recognition, comprising the following steps:

[0005] Step 1, designing clearly distinguishable positioning mark blocks, including the four corner positioning mark blocks of the machine-readable card border area, the basic title number positioning mark block, and the barcode extension positioning mark block;

[0006] Step 2: Take a photo of the test paper containing the positioning mark block, identify the barcode extension positioning mark block information and the four-corner positioning mark block information in the photo, use the YOLOv7 model to identify the positioning mark block, and record the position of each positioning mark block in a txt document with the corresponding file name;

[0007] Step 3, extract the image containing the machine-readable card area from the photographed image according to the coordinates of the four-corner positioning mark blocks in step 2 and perform trapezoidal correction. Suppose the read image is a three-channel color image img_source, which is a three-dimensional array;

[0008] Step 4, convert the image img_source into a grayscale image img_gray, which is a two-dimensional array;

[0009] Step 5, adaptively adjust the binarization threshold threshold according to the img_gray image, convert all the filled areas and rectangular boxes to white, and the background to black, to obtain the binarized image img_bw;

[0010] Step 6, according to the optimal binarization threshold determined in step 5, a center point matrix of the filled rectangular frame is formed according to the coordinates of the center points of the rows and columns, and the coordinates of the filled rectangular frame are calculated;

[0011] Step 7, according to the coordinates of the fill frame obtained in step 6, the proportion of the white point value extended by the center coordinates in the binarized image img_bw is obtained to determine whether the fill frame is filled;

[0012] Step 8, obtain the corresponding fill-in box result for each question based on the pre-set number of questions, and determine whether the question is scored by comparing it with the standard answer.

[0013] Furthermore, the specific implementation of step 5 is as follows;

[0014] Step 5.1, set the threshold range to [120, 180, 130, 170, 140, 160, 150, 190], a total of 8 values, set the value brought in for the i-th time to be threshold i , get the binary image img_bw i , a total of 8: {img_bw1,img_bw2,img_bw3,img_bw4,img_bw5,img_bw6,img_bw7,img_bw8};

[0015] Step 5.2, using the rectangular convolution kernel kernel as a parameter, i Perform a closing operation to obtain the corresponding images img_close i , by looping through the img_close i Collect all the inflection points of the contour, collect all the inflection points that can form the contour, for img_close i , get contours of various shapes {contours1, contours2,…contours n};

[0016] Step 5.3, all contours {contours1, contours2, ... contours n}Using the rectangular outer frame method, we can find the rectangular area sequence, rectangular width sequence, and rectangular length sequence respectively. We can perform kmeans clustering on these three sequences respectively. We can find the top two clusters with the largest number of items in each sequence, find the mean of each cluster, and then average the means of the top two clusters again to predict the optimal size of the filled rectangular frame in the machine-readable card area. w and fillrec h , set the area range of the filled rectangle to:

[0017] fillrec_max=1.2*max(maxarea1,maxarea2)

[0018] fillrec_min=0.8*min(maxarea1,maxarea2)

[0019] Among them, maxarea1 and maxarea2 are the top two clusters with the largest number of rectangular area sequences;

[0020] Step 5.4, the area of ​​the filled rectangle changes from {contours1, contours2, …contours n}, and the number of rectangles remaining after this step is k, and we get {areacontours1,areacontours2,…,areacontours k};

[0021] Step 5.5: further perform the calculation on the set {areacontours1, areacontours2, …, areacontours k}Through shape (fillrec w ,fillrec h ) to filter and add the rectangular boxes that meet the conditions to the list Here m <k;

[0022] Step 5.6, find the center point of each filled rectangle selected in step 5.5 to form

[0023] Step 5.7, perform DBSCAN clustering on the x and y sequences of the filled rectangular boxes obtained in step 5.6 to obtain the classification labels labels_x and labels_y of the center points of all rectangular boxes, and obtain the center point coordinates {x_center1, x_center2…x_center c},{y_center1,y_center2…y_center r} and the number of categories is set to rows out , cols out , where rows out = r, cols out =c;

[0024] Step 5.8, to avoid the situation of no solution, if the number of DBSCAN clusters of the horizontal and vertical coordinate sequences is not equal to the preset fix_rows and fix_cols respectively but close, then the expansion distances epx_inx and epx_iny are slightly adjusted according to the close method, and the final adjusted row and column clustering result is rows out , cols out ;

[0025] If step 5 still cannot match the preset row and column values, it means that the image is severely tilted and cannot be matched, and manual correction is required;

[0026] Step 5.9, calculate the number of clusters and the coordinates of the center point of each cluster based on the results of cluster fitting. If the number of clusters in the horizontal and vertical coordinate sequences is equal to the preset fix_rows and fix_cols, respectively, where fix_rows and fix_cols represent the number of rows and columns preset in the machine-readable card, indicating that the binarization threshold threshold meets the requirements, then retain the cluster center point and terminate the loop. Otherwise, continue to repeat steps 5.1-5.9 until all 8 thresholds are traversed.

[0027] Furthermore, in step 5.3, the optimal size of the filled rectangle in the machine readable card area is fillrec w and fillrec h The calculation formula is as follows:

[0028]

[0029]

[0030] Among them, maxw1 and maxw2 are the top two clusters with the largest number of rectangles in the rectangle width sequence; maxh1 and maxh2 are the top two clusters with the largest number of rectangles in the rectangle length sequence.

[0031] Furthermore, in step 5.6, the center point of the rectangular frame rec centeri The coordinates of are obtained by taking the average of the coordinates of the upper left corner and the lower right corner of each rectangular box.

[0032]

[0033] in, and The center points of the rectangular boxes are The horizontal and vertical coordinates of and Rectangular box contours i The horizontal coordinates of the upper left corner and the lower right corner, and Rectangular box contours i The vertical coordinates of the upper left corner and lower right corner.

[0034] Furthermore, in step 5.7, the clustering X-axis extension distance key parameters and Y-axis extension distance key parameters are set as follows:

[0035] eps_inx = int(fillrec w / 4+4)

[0036] eps_iny=int(fillrec h / 4+4)

[0037] fillrec w , fillrec h is the result from step 5.3, int means rounding;

[0038] dbscanx=DBSCAN(eps=eps_inx,min_samples=fix_cols / 2)

[0039] dbscany=DBSCAN(eps=eps_iny,min_samples=fix_rows / 2)

[0040] labels_x=dbscanx.fit_predict(x_list_in)

[0041] labels_y=dbscany.fit_predict(y_list_in)

[0042] Among them, min_samples determines the minimum number of points that can form row and column center point clusters, and the extended distances eps_inx and eps_iny affect whether neighboring points within the extended distance range are planned into the same cluster; fix_rows and fix_cols are the number of rows and columns pre-set in the machine-readable card; dbscanx and dbscany are the clustering information lists after row and column clustering respectively, and eps is the clustering extension distance parameter.

[0043] Furthermore, in step 5.8, for the set values ​​of fix_rows and fix_cols, the way to make fine adjustments to eps_inx and eps_iny is as follows;

[0044] If cols out <fix_cols eps_inx=eps_inx-2

[0045] Otherwise eps_inx = eps_inx + 2

[0046] If rows out <fix_rowseps_iny=eps_iny-2

[0047] Otherwise eps_iny=eps_iny+2.

[0048] Furthermore, the method of determining whether the fill box is filled in step 7 is as follows:

[0049] is the coordinate of the center point of the (i, j)th column. Taking the center point as the coordinate, it is expanded into a rectangular box. The coordinates of the upper left corner and the lower right corner are: Then find the proportion s of the area occupied by white dots in the coordinate range of the rectangular box in the img_bw two-dimensional matrix. If s is greater than the set threshold, the fill box is considered to be filled and the fill mark is 1. Otherwise, the fill box is not filled and the fill mark is 0, thus forming a fill matrix, which also has fix_rows rows and fix_cols columns.

[0050] Furthermore, in step 8, the positions of the four filling boxes corresponding to question j in the matrix are set to:

[0051] A_answer=int(j / 20)*80+j%20

[0052] B_answer=int(j / 20)*80+j%20+20

[0053] C_answer=int(j / 20)*80+j%20+40

[0054] D_answer=int(j / 20)*80+j%20+60

[0055] Among them, int means rounding. Assuming the corresponding fill mark is [0,0,1,0], the corresponding answer is C.

[0056] The present invention also provides a machine-readable card image recognition system based on YOLOv7 model positioning and multiple clustering recognition, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as described in the above scheme.

[0057] The present invention also provides a computer-readable storage medium, including a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as described in the above scheme.

[0058] The invention discloses a machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering. The algorithm first recognizes a custom mark through a YOLOv7 model, determines the four corners of a quadrilateral in a multiple-choice question area, then extracts an image of a machine-readable card portion according to four corner points, then first performs trapezoidal correction on the machine-readable card image, performs denoising, closed operation, and grayscale preprocessing on the regional image, converts the original machine-readable card image to be recognized into a black and white binary image, uses the contour in CV2 to find rectangles of various scales, then determines the center point of each rectangle in the X-axis and Y-axis according to a clustering method, and finally determines whether it is filled or not according to the rectangular area ratio occupied by the white point, determines the filling position to obtain the answer and compares it. The method has been verified by multiple examples, and experiments have shown that the method has a high degree of automation, accurate recognition, and fast speed, and greatly reduces the test paper cost, improves the efficiency of marking, and has broad promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The positioning mark blocks customized for the embodiment of the present invention include 43 categories in total.

[0060] Figure 2 This is an example diagram of corresponding marked folders in a customized data set according to an embodiment of the present invention.

[0061] Figure 3 This is an example diagram of the 22nd type of sub-dataset in the customized data set according to an embodiment of the present invention.

[0062] Figure 4 This is the positioning mark result recognized by the embodiment of the present invention. The left picture is the recognized custom mark box, and the right picture is the corresponding mark box coordinates.

[0063] Figure 5 This is a picture after the positioning mark screenshot is automatically corrected in an embodiment of the present invention.

[0064] Figure 6 1 is a binarized image with different binarization threshold parameters in the embodiment of the present invention.

[0065] Figure 7 It is a schematic diagram of a rectangular frame after screening and identification in an embodiment of the present invention.

[0066] Figure 8 Schematic diagram of row and column coordinate points after multiple clustering of the center points of the rectangular frame in an embodiment of the present invention.

[0067] Fig. 9 It is a schematic diagram of a rectangular frame result of a certain question in an embodiment of the present invention.

[0068] Fig.10 It is a selection box after comparing with the standard answer in the embodiment of the present invention, in which the correct answer is green and the error is not marked.

[0069] Fig.11 The figure is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0071] like Fig.11 As shown, the present invention provides a machine-readable card image recognition method based on YOLOv7 model positioning and DBSCAN clustering. In image preprocessing, the method uses the OpenCV library, which is a cross-platform computer vision and machine learning software library released based on the Apache2.0 license (open source), and contains many general algorithms in computer vision, hereinafter referred to as the OpenCV library.

[0072] The present invention comprises the following steps:

[0073] Step 1: Design clearly distinguishable positioning marker blocks, totaling 43 categories ( Figure 1 ), including 21 categories of basic question number positioning marks. 10 categories of machine-readable card border area 4-corner positioning marks. 12 categories of barcode extended positioning marks. Among them, the 4-corner positioning marks of the machine-readable card border area are used to represent the four corners of the machine-readable card area, and any 4 of them can be selected and used in combination. The 20 categories of basic question number positioning marks represent the positioning marks of each question in the test paper, with a maximum of 21 major questions. Barcode extended positioning is used for barcode recognition area positioning, which can be used for subsequent expansions such as subsequent student number recognition. It is not used in this version. The positioning mark block must have obvious different characteristics in shape.

[0074] Step 2: Take a photo of the test paper containing the positioning mark block, identify the student ID barcode information and the four-corner positioning mark block information in the photo, use the YOLOv7 model to identify the positioning mark block, and record the position of each positioning mark block in a txt document with the corresponding file name.

[0075] Step 3, extract the image containing the machine-readable card area from the photographed image according to the coordinates of the four corner positioning mark blocks in step 2 and perform trapezoidal correction. Suppose the read image is a three-channel color image img_source, which is a three-dimensional array.

[0076] Step 4, convert the image img_source into a grayscale image img_gray, which is a two-dimensional array.

[0077] Step 5, adaptively adjust the binarization threshold threshold based on the img_gray image. The part greater than the threshold is set to 0, and the part less than the threshold is set to 255 to achieve black and white binary conversion. The grayscale image presents a clear black and white visual effect. All filled areas and rectangular boxes will be converted to white, and the background is black. However, the setting of the threshold greatly affects the displayed outline, thereby affecting the positioning of subsequent steps. Through experiments, relying solely on a single OpenCV library image binarization threshold processing function adaptiveThreshold has not achieved good results. It is impossible to accurately locate the position of the fill selection box. There are always various omissions or erroneous additions of rectangular outline boxes. In fact, each image must be adjusted according to the actual influencing factors. This method uses a specified range loop to obtain the optimal threshold. The specific implementation method is as follows:

[0078] 5.1, considering that the black border color is affected by the paper background, light brightness, and printing color, we set the threshold range to [120, 180, 130, 170, 140, 160, 150, 190], a total of 8 values, which are sorted according to the highest probability order based on historical experience, with small and large values ​​alternately inserted. Let the value introduced for the i-th time be threshold i , get the binary image img_bw i , a total of 8: {img_bw1,img_bw2,img_bw3,img_bw4,img_bw5,img_bw6,img_bw7,img_bw8};

[0079] 5.2, use the cv2.getStructuringElement function in the OpenCV library to construct a rectangular convolution kernel kernel, and use the MORPH_RECT parameter. Its purpose is to make the boundaries of irregular shapes (rectangles, polygons, etc.) in the image more obvious. Using the rectangular convolution kernel kernel as a parameter, iPerform a closing operation to obtain the image img_close, and use the findContours contour search function in the OpenCV library to find all the contour inflection points in img_close, and collect all the inflection point sets that can form contours, including various shapes {contours1, contours2, ... contours n}.

[0080] 5.3, for all contours {contours1, contours2, ... contours n}Using the rectangular frame method, we can find the rectangular area sequence, rectangular width sequence, and rectangular length sequence, which are expressed as:

[0081] rec_area_list={rec1,rec2,…rec n}

[0082] rec_w_list={w1,w2,…w n}

[0083] rec_h_list={h1,h2,…h n}

[0084] Use kmeans clustering with a preset category parameter of 6 for these three sets respectively, find the top two clusters with the largest number of sequences in each sequence, and find the mean of each cluster. Set the means of the top two clusters to maxarea1, maxarea2, maxw1, maxw2, maxh1, maxh2, and average the means of the top two of each indicator again. In this way, the optimal size of the fill rectangle in the machine-readable card area can be predicted. w and fillrec h The optimal size of the filled rectangle is:

[0085]

[0086] This size will be used in subsequent comparisons.

[0087] Because of the tilt of the camera angle and the fullness of the filling, the rectangle found by the outline will be extremely irregular, so the area variation range of the filled rectangle is set to:

[0088] fillrec_max=1.2*max(maxarea1,maxarea2)

[0089] fillrec_min=0.8*min(maxarea1,maxarea2)

[0090] Because there are many contour rectangles found by CV2.findContours, including rectangles of various sizes, it is necessary to locate each filled rectangle frame according to the optimal values ​​of area, length and width, and this type of filled rectangle frame in the machine-readable card area should have the largest number. Due to the tilt of the shooting angle, the filled rectangle frame rectangles will be divided into multiple classes. Regardless of the area, length and width sequence, the class with the largest number of members does not necessarily represent the actual size of the filled rectangle frame. By taking the average of the top two clusters of each indicator, the influence of the tilt angle is neutralized and close to the actual value.

[0091] Step 5.4, filter {contours1, contours2, … contours by the best area mean in step 5.3 n}Draw a rectangle that meets the conditions.

[0092] fillrec_min=<{contours1,contours2,…contoursk}<=fillrec_max

[0093] Let the number of rectangles remaining after this screening step be k.

[0094] Step 5.5: further perform the calculation on the set {contours1, contoursk2,…, contours k}Through shape (fillrec w ,fillrec h ) to screen. Because the rectangle that meets the area may be greatly deformed due to the irregularity of filling, it is also necessary to screen the aspect ratio from the appearance. The specific method is to use the predicted filling rectangle shape {fillrec w ,fillrec h}Filter each contour {contours1, contoursk2,…, contours k}, let the width of the fitted rectangle in this set be w_contours i , height is h_contours i , if it satisfies:

[0095] 1.2*fillrec w >w_contours i >0.5*fillrec w

[0096] 1.2*fillrec h >h_contours i >0.5*fillrec h

[0097] This means that the contour is a filled rectangle that meets the conditions, then contours i Add to List Here m <k。

[0098] Step 5.6, find the center point of each filled rectangle selected in step 5.5 to form Rectangle center point The coordinates are obtained by taking the average of the coordinates of the upper left corner and the lower right corner of each rectangular box.

[0099]

[0100] i=1,2…m

[0101] Thus, the horizontal and vertical coordinate sequences of the center points of the rectangular frame are formed.

[0102] Step 5.7, perform DBSCAN clustering on the x-coordinate and y-coordinate sequences of the center point of the filled rectangular box obtained in step 5.6, and set the key parameters of the clustering X-axis (row) expansion distance and Y-axis (column) expansion distance as follows:

[0103] eps_inx = int(fillrec w / 4+4)

[0104] eps_iny=int(fillrec h / 4+4)

[0105] fillrec w , fillrec h is the result from step 5.3, int means rounding, fix_rows, fix_cols are the number of rows and columns pre-set in the machine-readable card. Among them, min_samples determines the minimum number of points that can form a cluster of row and column center points. For example, in a fill box with 20 rows and 12 columns, the number of rows should not be less than 10 points to form a cluster, and the number of columns should not be less than 6 points to form a cluster. The extended distance eps_inx, eps_iny affects whether the adjacent points within the distance extension range are planned into the same cluster. dbscanx, dbscany are the clustering information lists after row and column clustering, respectively, and eps is the clustering extension distance parameter.

[0106] dbscanx=DBSCAN(eps=eps_inx,min_samples=fix_cols / 2)

[0107] dbscany=DBSCAN(eps=eps_iny,min_samples=fix_rows / 2)

[0108] labels_x=dbscanx.fit_predict(x_list_in)

[0109] labels_y=dbscany.fit_predict(y_list_in)

[0110] Through this step, we get the classification labels labels_x and labels_y of the center points of all rectangular boxes. For the same type of clusters, we can get the center point coordinates of the clusters {x_center1, x_center2…x_center c},{y_center1,y_center2…y_center r} and the number of categories is set to rows out , cols out , where rows out = r, cols out =c.

[0111] Step 5.8, to avoid the situation of no solution, if the number of DBSCAN clusters of the horizontal and vertical coordinate sequences is not equal to the preset fix_rows and fix_cols respectively but close, then make a slight adjustment to eps_inx and eps_iny according to the close method. Take cluster detection of 20 rows, 12 columns, and a total of 60 multiple-choice questions as an example, fix_rows = 20, fix_cols = 12 columns, then:

[0112] if cols out <fix_cols eps_inx=eps_inx-2

[0113] else eps_inx=eps_inx+2

[0114] ifrows out <fix_rowseps_iny=eps_iny-2

[0115] elseps_iny=eps_iny+2

[0116] The final adjusted row and column clustering result is rows out , cols out .

[0117] Step 5.9, calculate the number of clusters and the coordinates of the center point of each cluster based on the results of cluster fitting. If the number of clusters in the horizontal and vertical coordinate sequences is equal to the preset fix_rows and fix_cols (the number of rows and columns preset in the machine-readable card), it means that the binarization threshold threshold meets the requirements, and the cluster center point is retained. The loop is terminated, otherwise, steps 5.1-5.9 are repeated until all 8 thresholds are traversed.

[0118] Step 6: If the optimal binarization threshold is found in step 5, the center point matrix of the filled rectangular box is formed according to the coordinates of the center points of the rows and columns. Set row = rows out , col = cols out , set it in the general way of the image, with the upper left corner as point o, the x-axis to the right, and the y-axis downward.

[0119] Then the coordinates of the filled rectangle in row i and column j are: (x_center j ,y_center i ).

[0120]

[0121] If step 5 still cannot match the preset row and column values, it means that the image is severely tilted and cannot be matched, and manual correction is required.

[0122] Step 7, according to the coordinates of the filled frame obtained in step 6, the proportion of the white point value extended by the central coordinates in the binarized image img_bw is calculated to determine whether the frame is filled.

[0123] The specific method is as follows: is the coordinate of the center point of the (i, j)th column, and the center point is used as the coordinate to expand into a rectangular box. The coordinates of the upper left corner and the lower right corner are: Then find the proportion s of the area occupied by white dots in the coordinate range of the rectangular frame in the img_bw two-dimensional matrix. If s>0.5, the filled frame is considered to be filled and the filled mark is 1. Otherwise, the filled frame is not filled and the filled mark is 0. Thus, a filled matrix is ​​formed, as shown below:

[0124]

[0125] This matrix also has fix_rows rows and fix_cols columns.

[0126] Step 8, according to the preset number of questions, the corresponding filling box results of each question are obtained, and the positions of the four filling boxes corresponding to the i-th question in the matrix are obtained as follows:

[0127] A_answer=int(i / 20)*80+i%20

[0128] B_answer=int(i / 20)*80+i%20+20

[0129] C_answer=int(i / 20)*80+i%20+40

[0130] D_answer=int(i / 20)*80+i%20+60

[0131] Assuming the corresponding fill mark is [0,0,1,0], the corresponding answer is C. By comparing with the standard answer, you can determine whether the question is scored.

[0132] On the other hand, an embodiment of the present invention also provides a machine-readable card image recognition system based on YOLOv7 model positioning and multiple clustering recognition, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as described in the above scheme.

[0133] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, comprising a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as described in the above scheme.

[0134] The effect of the present invention can be illustrated by the following experiments:

[0135] 1. Experimental conditions: The experiment was implemented using Python 3.8 and tested on an i7 2.40GHz CPU, 8.0GB memory, and Windows 10 platform.

[0136] 2. The experimental contents and results are as follows:

[0137] 1) Customized positioning marker images such as Figure 1 shown.

[0138] 2) Construct a custom data set, collect 301 test paper images through the Internet, synthesize the custom marker map into the test paper image through Python programming, and add rotation, stretching, tilting, and scaling effects to form 43 sub-data sets, such as Figure 2 shown.

[0139] The composite image of the positioning mark map and the test paper image contained in each subclass, taking the 22-class sub-dataset as an example, Figure 3 shown.

[0140] 3. Train the custom data set through YOLOv7 to obtain the custom recognition model best.pt.

[0141] 4. Use best.pt to identify the actual machine-readable card test paper, obtain the mark positioning image and coordinate record file, and the recognized positioning mark is as follows Figure 4 As shown, (a) is the recognition result of the machine-readable card 1, that is, the mark positioning image, and (b) is the coordinate record file of the machine-readable card 1.

[0142] 5. The original image is obviously tilted to the left. The image is extracted and automatically corrected according to the coordinate positions of the four corner frames to obtain the corrected image, such as Figure 5 As shown in the figure, it can be seen that the four corners have obvious marks based on the center point cutting.

[0143] 6. The optimal black and white image obtained after threshold optimization is as follows Figure 6 The left image shows a threshold of 190, and the right image shows a threshold of 150.

[0144] 7. Filter out the rectangular frame based on the best black and white image, and the obtained rectangular frame is as follows Figure 7 .

[0145] 8. Cluster the horizontal x coordinates and vertical y coordinates based on the center coordinates of the rectangular box, and then get the center point coordinates, such as Figure 8 shown.

[0146] 9. Check the rectangle of a question (question 43) separately, such as Fig. 9 , calculate the number of black dots in the rectangular box and the area ratio of the entire rectangular area. If it is greater than 0.5, the box is considered to be filled.

[0147] 10. As Fig.10 As shown in the figure, the result of the recognition in the entire image is the result of comparing the filled-in answer with the marked answer. The correct box is marked in green. The user's filled-in answer is:

[0148] ['B','A','C','A','C','C','C','D','A','A','D','C','D','C','C','D','D','B','A','A','B','C','A',

[0149] 'C','B','B','D','C','B','A','B','B','C','D','B','D','A','C','D','D','B','D','C','A','C','B','B','A',

[0150] 'A','C',",",",",",",",",",",",",'C',",'C',",'C',",'B',",",",",",",",",",","].

[0151] The answer is:

[0152] BACACCCDAADCDCCEDBAABCACBBDCBACBCDBDACDDEDCACBBAAC

[0153] In summary, the experiments show that after simply taking a photo and uploading it to the computer, the batch processing of this method can extract images and identify the answers to fill in, achieving a high accuracy rate, a high degree of automation, and simple operation, which is suitable for batch review of machine-readable card images.

[0154] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0155] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition, characterized in that: The following steps are involved: Step 1, designing clearly distinguishable positioning mark blocks, including the four corner positioning mark blocks of the machine-readable card border area, the basic title number positioning mark block, and the barcode extension positioning mark block; Step 2: Take a photo of the test paper containing the positioning mark block, identify the barcode extension positioning mark block information and the four-corner positioning mark block information in the photo, use the YOLOv7 model to identify the positioning mark block, and record the position of each positioning mark block in a txt document with the corresponding file name; Step 3, extract the image containing the machine-readable card area from the photographed image according to the coordinates of the four-corner positioning mark blocks in step 2 and perform trapezoidal correction. Suppose the read image is a three-channel color image img_source, which is a three-dimensional array; Step 4, convert the image img_source into a grayscale image img_gray, which is a two-dimensional array; Step 5, adaptively adjust the binarization threshold threshold according to the img_gray image, convert all the filled areas and rectangular boxes to white, and the background to black, to obtain the binarized image img_bw; Step 6, according to the optimal binarization threshold determined in step 5, a center point matrix of the filled rectangular frame is formed according to the coordinates of the center points of the rows and columns, and the coordinates of the filled rectangular frame are calculated; Step 7, according to the coordinates of the fill frame obtained in step 6, the proportion of the white point value extended by the central coordinates in the binarized image img_bw is obtained to determine whether the fill frame is filled; Step 8, obtain the corresponding fill-in box result for each question based on the pre-set number of questions, and determine whether the question is scored by comparing it with the standard answer.

2. The machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as claimed in claim 1, characterized in that: The specific implementation of step 5 is as follows; Step 5.1, set the threshold range to [120, 180, 130, 170, 140, 160, 150, 190], a total of 8 values, set the value brought in for the i-th time to be threshold i , get the binary image img_bw i , a total of 8: {img_bw1,img_bw2,img_bw3,img_bw4,img_bw5,img_bw6,img_bw7,img_bw8}; Step 5.2, using the rectangular convolution kernel kernel as a parameter, for img_bw i Perform a closing operation to obtain the corresponding images img_close i , by looping through the img_close i Collect all the inflection points of the contour, collect all the inflection points that can form the contour, for img_close i , get contours of various shapes {contours1, contours2, …contours n }; Step 5.3, all contours {contours1, contours2, ... contours n }Using the rectangular outer frame method, we can find the rectangular area sequence, rectangular width sequence, and rectangular length sequence respectively. We can perform kmeans clustering on these three sequences respectively. We can find the top two clusters with the largest number of items in each sequence, find the mean of each cluster, and then average the means of the top two clusters again to predict the optimal size of the filled rectangular frame in the machine-readable card area. w and fillrec h , set the area range of the filled rectangle to: fillrec_max=1.2*max(maxarea1,maxarea2) fillrec_min=0.8*min(maxarea1,maxarea2) Among them, maxarea1 and maxarea2 are the top two clusters with the largest number of rectangular area sequences; Step 5.4, the area of ​​the filled rectangle changes from {contours1, contours2, …contours n }, and the number of rectangles remaining after this step is k, and we get {areacontours1,areacontours2,…,areacontours k }; Step 5.5: further perform the calculation on the set {areacontours1, areacontours2, …, areacontours k }Through shape (fillrec w ,fillrec h ) to filter and add the rectangular boxes that meet the conditions to the list Here m <k; Step 5.6, find the center point of each filled rectangle selected in step 5.5 to form Step 5.7, perform DBSCAN clustering on the x and y sequences of the filled rectangular boxes obtained in step 5.6 to obtain the classification labels labels_x and labels_y of the center points of all rectangular boxes, and obtain the center point coordinates {x_center1, x_center2…x_center c },{y_center1,y_center2…y_center r } and the number of categories is set to rows out , cols out , where rows out = r, cols out =c; Step 5.8, to avoid no solution, if the number of DBSCAN clusters of the horizontal and vertical coordinate sequences is not equal to the preset fix_rows and fix_cols respectively but close, then the expansion distances eps_inx and eps_iny are slightly adjusted according to the close method, and the final adjusted row and column clustering result is rows out , cols out ; If step 5 still cannot match the preset row and column values, it means that the image is severely tilted and cannot be matched, and manual correction is required; Step 5.9, calculate the number of clusters and the coordinates of the center point of each cluster based on the results of cluster fitting. If the number of clusters in the horizontal and vertical coordinate sequences are equal to the pre-set fix_rows and fix_cols, respectively, where fix_rows and fix_cols represent the number of rows and columns pre-set in the machine-readable card, indicating that the binarization threshold threshold meets the requirements, then retain the cluster center point and terminate the loop. Otherwise, continue to repeat steps 5.1-5.9 until all 8 thresholds are traversed.

3. The machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as claimed in claim 2, characterized in that: In step 5.3, the optimal size of the filled rectangle in the machine-readable card area is fillrec w and fillrec h The calculation formula is as follows: Among them, maxw1 and maxw2 are the top two clusters with the largest number of rectangles in the rectangle width sequence; maxh1 and maxh2 are the top two clusters with the largest number of rectangles in the rectangle length sequence.

4. The machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as claimed in claim 2, characterized in that: In step 5.6, the center point of the rectangle The coordinates of are obtained by taking the average of the coordinates of the upper left corner and the lower right corner of each rectangular box. in, and The center points of the rectangular boxes are The horizontal and vertical coordinates of and Rectangular box contours i The horizontal coordinates of the upper left corner and the lower right corner, and Rectangular box contours i The vertical coordinates of the upper left corner and lower right corner.

5. The machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as claimed in claim 2, characterized in that: In step 5.7, the clustering X-axis extension distance key parameters and Y-axis extension distance key parameters are set as follows: eps_inx=int(fillrec w / 4+4) eps_iny=int(fillrec h / 4+4) fillrec w , fillrec h is the result from step 5.3, int means rounding; dbscanx=DBSCAN(eps=eps_inx,min_samples=fix_cols / 2) dbscany=DBSCAN(eps=eps_iny,min_samples=fix_rows / 2)labels_x=dbscanx.fit_predict(x_list_in)labels_y=dbscany.fit_predict(y_list_in) Among them, min_samples determines the minimum number of points that can form row and column center point clusters, and the extended distances eps_inx and eps_iny affect whether neighboring points within the extended distance range are planned into the same cluster; fix_rows and fix_cols are the number of rows and columns pre-set in the machine-readable card; dbscanx and dbscany are the clustering information lists after row and column clustering respectively, and eps is the clustering extension distance parameter.

6. The machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as claimed in claim 2, characterized in that: In step 5.8, for the set values ​​of fix_rows and fix_cols, the way to fine-tune eps_inx and eps_iny is as follows; If cols out <fix_cols eps_inx=eps_inx-2 Otherwise eps_inx = eps_inx + 2 If rows out <fix_rows eps_iny=eps_iny-2 Otherwise eps_iny=eps_iny+2.

7. The machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as claimed in claim 2, characterized in that: The method of determining whether the fill frame is filled in step 7 is as follows: is the coordinate of the center point of the (i, j)th column. Taking the center point as the coordinate, it is expanded into a rectangular box. The coordinates of the upper left corner and the lower right corner are: Then find the proportion s of the area occupied by white dots in the coordinate range of the rectangular box in the img_bw two-dimensional matrix. If s is greater than the set threshold, the filled box is considered to be filled and the filling mark is 1. Otherwise, the filled box is not filled and the filling mark is 0, thus forming a filled matrix, which also has fix_rows rows and fix_cols columns.

8. The machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as claimed in claim 2, characterized in that: In step 8, the positions of the four filling boxes corresponding to question j in the matrix are set as follows: A_answer=int(j / 20)*80+j%20 B_answer=int(j / 20)*80+j%20+20 C_answer=int(j / 20)*80+j%20+40 D_answer=int(j / 20)*80+j%20+60 Among them, int means rounding. Assuming the corresponding fill mark is [0,0,1,0], the corresponding answer is C.

9. The machine-readable card image recognition system based on YOLOv7 model positioning and multiple clustering recognition is characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as described in any one of claims 1 to 8.

10. A computer-readable storage medium, comprising a stored computer program. When the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the machine-readable card image recognition method based on YOLOv7 model positioning and multiple clustering recognition as described in any one of claims 1 to 8.

Images