A line sorting method with dynamic angle adjustment in book spine segmentation
By using a line sorting method with dynamically adjusted angles, the problem of insufficient accuracy and robustness of spine segmentation in the library management system is solved, achieving efficient and low-cost automatic book identification and classification, and improving the intelligence level of the library management system.
Patent Information
- Application Number
- CN202210147636.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-17
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-02-17
AI Technical Summary
Existing spine segmentation technologies in library management systems suffer from insufficient segmentation accuracy and robustness, especially when faced with varying spine angles and ambient lighting conditions. Traditional methods are time-consuming and complex, while deep learning methods require extensive manual annotation and training time.
A line sorting method with dynamic angle adjustment is adopted for spine segmentation. Through image preprocessing, line detection, angle calculation and rotation, sliding window sorting and line fitting, the method improves the accuracy and efficiency of spine segmentation by utilizing traditional algorithms. This includes grayscale loading, storing line set angles in a list, image rotation, sliding window fitting and least squares fitting.
It achieves efficient and robust spine segmentation, reduces the maintenance cost of book management, improves the intelligence level of the book management system, reduces manual intervention, and is suitable for automatic identification and classification in the book management system.
Smart Images

Figure CN114494270B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of book spine recognition, and in particular to a line sorting method for dynamically adjusting angles in book spine segmentation. Background Art
[0002] Computer vision technology uses cameras and processors to identify, extract and track object features, and finally obtains the required image information through humanized processing. In recent years, although deep learning, as one of the fastest-growing fields in machine learning, has brought broader applications and development prospects to computer vision, there are still some unresolved problems, such as the accuracy of image segmentation and border positioning, as well as time complexity. For example, in robot competitions, we use deep learning to make radar maps, but in robot vision systems, we need to accurately locate the target, and traditional algorithms often get better results than deep learning. In addition, due to the large number of data and neurons, deep learning often requires a long training and running time. Even a very simple neural network, such as a neural network with only one layer and fifty units, will take about five minutes to train. Therefore, when dealing with different problems, we still need to prescribe the right medicine, give full play to the advantages of traditional algorithms and deep learning, and play to their strengths and avoid their weaknesses.
[0003] In the existing library book management system, book inventory management still relies heavily on manual work, and few people have proposed and solved the overall book management solution based on computer vision. The main problem of spine detection is the accuracy and robustness of spine segmentation. Due to the interference of other content on the spine and the variability of ambient lighting, spine segmentation is difficult to maintain in a stable and good state. Therefore, spine segmentation is the focus and difficulty of spine detection and book management.
[0004] There are very few existing technologies related to book spine segmentation. For example, in “A spine positioning method based on text detection” (DOI 10.3969 / j.issn.1672-9722.2020.01.034), “A spine area rough selection method based on text detection is proposed. First, the overall character area in the image is detected by a sequential segmentation algorithm, and then the characters belonging to the same book are added to a similar character set according to the character width and distance. The candidate spine area is calculated according to the character center and character width in the set, and finally the spine area is selected by a support vector machine classifier”. In addition, in “Book recognition method and system based on spine visual information” (application number CN202011383651.5), “the spine images of books with labeled spine segmentation are obtained as training sets, and the training sets are used to train the segmentation method. A deep convolutional neural network model of book spines is constructed to obtain a book spine segmentation model. The book spine segmentation model is used to perform instance segmentation on the collected books on the shelf to obtain multiple book spine images; a book spine classification dataset is constructed by labeling the book category for each book spine image, and a deep convolutional neural network model for book spine classification is trained by the book spine classification dataset to obtain a book spine feature extraction model. The spine feature extraction model is used to extract the spine visual features of each book in the book database, and the spine visual features are collected to construct a book spine visual database; the book spine image to be identified containing multiple book spines is input into the book spine segmentation model for instance segmentation, and the segmentation result is input into the book spine feature extraction model to obtain the visual feature vector of each book spine in the book spine image to be identified, and the visual feature vector is matched with the database to identify the book category of each book spine in the book spine image to be identified. For the former, it can be seen from the pictures listed in the literature that the book spines obtained by this method are basically incomplete. We also conducted similar experiments on book spine segmentation based on text recognition, but due to the fact that the angles of some texts on the book spines vary, there is a certain degree of conflict with our own algorithm, resulting in unsatisfactory results. For the latter, first of all, the data collection and annotation of book spine segmentation is a huge manual project. In addition, due to the huge amount of data and the complexity of the neural network, the time and computing power required for training are also very large, not to mention that two annotations and model training are required. If the accuracy needs to be optimized, the work and time required will increase exponentially. The present invention aims to provide an efficient and easy-to-maintain book spine recognition system, which prefers to use traditional algorithms for book segmentation, has strong robustness, and improves the intelligent management level of book management. Summary of the invention
[0005] In order to solve the above technical problems, the present invention proposes a line sorting method for dynamically adjusting angles in spine segmentation, which is characterized by:
[0006] Step 1, image preprocessing, includes the following steps:
[0007] Step 1.1, load an image in grayscale;
[0008] Step 1.2, obtaining a line set through line detection;
[0009] Step 1.3, calculating the angle of each line in the line set and storing it in a list;
[0010] Step 1.4, angle calculation;
[0011] Step 1.5, rotating the image as a whole according to the rotation picture parameter, rotating the line set obtained by line detection according to the rotation picture parameter to obtain a new line set, and sorting the lines according to the abscissa x1 of the endpoint of the first line in the new line set;
[0012] Step 2, spine line fitting, includes the following stages:
[0013] Step 2.1, sliding window sorting stage;
[0014] Step 2.2, line fitting stage.
[0015] Furthermore, in step 1.4, the angle average of all lines whose lengths are greater than a threshold is taken as the rotation image parameter, and the threshold is 1 / 3 of the image height.
[0016] Furthermore, in step 2.1, the sliding window sorting stage specifically includes:
[0017] Load the line set, set the window value w, the line number is i, and determine whether i is less than w;
[0018] If the line number i is less than w, the first line to the i+wth line are used as the sliding window;
[0019] If the line number i is not less than w, determine whether i+w is greater than the total number of lines;
[0020] If i+w is greater than the total number of lines, the iwth line to the last line is used as the sliding window. If i+w is not greater than the total number of lines, the iwth line to the i+wth line is used as the sliding window.
[0021] Solve the average angle AvgAngle of all lines in the window. If the angle of the line is in the interval [AvgAngle-Thrangle1, AvgAngle+Thrangle1], then sequentially store the line fitting mark, the calculated line length, the coordinates of the two vertices of the line, and the distance from the line to the midpoint of the left border of the image with the average angle as the slope into a list, and sort the list based on the distance.
[0022] Furthermore, the angle average AvgAngle is obtained by:
[0023] First, the angle of each line in the window is obtained, and the lines are sorted based on the angle, and the lines are loaded into the loop in order. Each line is extended to the left and right respectively until the lines on the left and right boundaries do not meet the angle threshold Thrangle2, and the angle sum AngleSum centered on the line is obtained;
[0024] Secondly, take the sum of the largest number of lines, max(AngleSum), and divide it by the number of lines NumLine. The resulting angle is used as the average angle AvgAngle of the main line i of this dynamic window.
[0025]
[0026] Furthermore, in step 2.2, the line fitting stage specifically includes:
[0027] Determine whether each line in the new line set is involved in overfitting, if it is a line involved in overfitting, proceed to the next one; if it is a line not involved in overfitting, extend to the left and right sides with the line as the center;
[0028] Determine whether the left and right lines meet the distance threshold Thrdistance. If so and the line fitting flag is 1, the length of the line is included in the total length. If the left and right lines exceed the distance threshold Thrdistance, determine whether the total length meets the length threshold Thrlength.
[0029] If the length threshold Thrlength is met, the data is put into the list in the order of leftmost line number, rightmost line number, total length, and filter flag;
[0030] If the length of the line being fitted is less than the length of the last fitted line in the list, this line is a short line with a small error of the previous line. If the fitting flag of the previous line participating in this filtering is 1, it is changed to 0, and the least squares fitting is performed. This line is stored in the final list until all lines are judged and the final line list is obtained. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a flow chart of the line sorting method for dynamically adjusting angles in book spine segmentation of the present invention;
[0032] Figure 2 Shown is the position of the resulting line set on the picture after the preparation stage;
[0033] Figure 3 for Figure 2 A partial enlarged view of
[0034] Figure 4 Shown is a flow chart of the sliding window sorting phase;
[0035] Figure 5 Shown is the line fitting flow chart;
[0036] Figure 6 The original image of the acquisition is shown;
[0037] Figure 7 The results of rotation preprocessing are shown;
[0038] Figure 8 The results of line detection are shown;
[0039] Fig. 9 Shown are the line fitting results; DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0041] like Figure 1 The flowchart of the line sorting method for dynamically adjusting angles in book spine segmentation of the present invention is shown. The line sorting method for dynamically adjusting angles in book spine segmentation includes: step 1 image preprocessing and step 2 book spine line fitting.
[0042] Step 1: Image preprocessing specifically includes the following steps:
[0043] Step 1.1, load an image in grayscale.
[0044] Step 1.2, obtain the line set through line detection.
[0045] Step 1.3, calculate the angle of each line and store it in a list.
[0046] like Figure 2 The figure shows the position of the line set obtained after the preprocessing stage on the image. The line set obtained by line detection is arranged from left to right in the figure, where the dots mark the coordinates of each line (x1, y1), and the numbers are the serial numbers of the line in the line set obtained in step 1.2 in the image preprocessing stage.
[0047] Step 1.4, angle calculation. Take the angle average of all lines whose length is greater than the threshold DisThreshold as the parameter for rotating the image. Here DisThreshold is 1 / 3 of the image height.
[0048] Step 1.5, rotate the image as a whole according to the parameters of the rotation image, and rotate the line set obtained by line detection according to this parameter to obtain a new line set, and sort it according to the parameter x1, that is, sort the line set with the horizontal coordinate x1 of the endpoint of the first line in the line set as a reference.
[0049] Step 2: The spine line fitting step includes: step 2.1 sliding window sorting stage and step 2.2 line fitting stage.
[0050] Step 2.1, sliding window sorting phase, its flow chart is as follows Figure 4 shown.
[0051] The window is to select a part of the entire line set, load the line set, and only perform fitting operations on this part. First, set the window value w. In this example, we set the window value w to 100, and then use the dynamic window that extends to both sides of each line as its own sliding window.
[0052] Determine whether i is less than w. If the line number i is less than w, use the first line to the i+wth line as the sliding window. Figure 2 and Figure 3 In the example, for the line segment with line number 2, the line segment with line number 0 is used as a sliding window until the line segment with line number 102 ends.
[0053] If the line number i is not less than w, then determine whether i+w is greater than the total number of lines.
[0054] If i+w is greater than the total number of lines, the iwth line to the last line is used as a sliding window; the remaining lines are centered on themselves, each extending w lines to both sides, and the iwth to i+wth lines are used as sliding windows. In the sliding window of each line, the average angle AvgAngle of all lines in the window is solved, and the angle threshold Thrangle1 is set. In this embodiment, the angle threshold Thrangle1 is assigned to 10. If the angle of the line is in the interval [AvgAngle-Thrangle1, AvgAngle+Thrangle1], the line fitting mark (here set to 1), the calculated line length, the coordinates of the two vertices of the line, the angle mean as the slope, and the distance from the line to the midpoint of the left boundary of the image are stored in a list in order, and the list is sorted by distance, otherwise the line is ignored.
[0055] Here, the line fitting mark is 1, indicating that the line is not involved in the fitting, and the line fitting mark is 0, indicating that the line is involved in the fitting. The average angle AvgAngle is obtained as follows:
[0056] First, the angle of each line in the window is obtained, and the lines are sorted based on the angle. The lines are loaded into the loop in order, and each line extends to the left and right until the lines on the left and right boundaries do not meet the angle threshold Thrangle2. In this embodiment, the angle threshold Thrangle2 is assigned to 10, and the angle sum AngleSum centered on the line is obtained. The sum containing the most lines is taken and divided by the number of lines NumLine. The obtained angle is used as the average angle AvgAngle of the main line i of this dynamic window, that is:
[0057]
[0058] Step 2.2, line fitting, its flow chart is as follows Figure 5 As shown. Fitting judgment is performed on each line in the sorted line set in order. If the line fitting mark shows that it is a line that is not involved in fitting, it is taken as the center and extended to the left and right sides to determine whether the left and right lines meet the distance threshold Thrdistance. In this embodiment, we assign the distance threshold Thrdistance to 15. If it meets and the line fitting mark is 1, its length is included in the total length until it exceeds Thrdistance, and then it is determined whether its total length meets the length threshold Thrlength. In this embodiment, we assign the length threshold Thrlength to half of the image height. If it meets, it is included in the list in the order of the leftmost line number, the rightmost line number, the total length, and the filter mark (set to 1 here). If the length of the line being fitted at this time is less than the length of the last line that has been fitted in the list, then this line is considered to be a short error line of the previous line. Among them, after line detection, the obtained line set consists of line segments obtained by segmenting the correct spine boundary line and very short error line segments close to the boundary line. The error short line refers to a very short line close to the boundary line.
[0059] If the flag of the previous line participating in this filtering is 1, change it to 0, then perform least squares fitting on the previous line and store this line in the final list. Otherwise, jump out of the loop directly and perform fitting judgment on the next line. After all lines are judged, the final line list is obtained and the line fitting is completed.
[0060] Here, the filtering mark is 1, indicating that the line has not participated in filtering, and the line fitting mark is 0, indicating that the line has participated in filtering.
[0061] Figure 6-Figure 9 The present invention takes actual operation as an example to show the detailed workflow:
[0062] First, collect the image, such as Figure 6 As shown; secondly, the image is rotated and preprocessed, and the preprocessing result is as follows Figure 7 As shown, the line detection results are as follows Figure 8 As shown, the line set in the image is then sorted by sliding window, and line fitting is performed according to the sorting result. The fitting result is shown in Fig. 9 shown.
[0063] In the present invention, the sliding window dynamically adjusts the sorting angle of the line detection result set, and the subsequent line fitting is performed based on the aforementioned sorting results. The advantage of this sliding window dynamically adjusting the angle sorting is that the distance from the midpoint of the left side of the image to the line is adjusted using the average angle of all detected lines in the line sliding window, and the line detection results are sorted based on the distance, avoiding the situation where the same spine line is not clustered due to the deviation of the global angle.
[0064] This system can be installed on board or implemented by mobile phone APP, has strong portability and low management and maintenance cost. For the existing book management system, book management is basically carried out manually, with low management efficiency and high management cost. For some large-scale libraries, the cost of managing all books once a day is unacceptable. The present invention aims to propose an efficient and concise book management method, which uses computer vision technology to automatically identify and classify books, and realizes almost real-time book management on this basis, which can greatly reduce the maintenance time of librarians.
[0065] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0066] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A line sorting method for dynamically adjusting angles in book spine segmentation, characterized in that: Step 1, image preprocessing, includes the following steps: Step 1.1, load an image in grayscale; Step 1.2, obtaining a line set through line detection; Step 1.3, calculating the angle of each line in the line set and storing it in a list; Step 1.4, angle calculation; Step 1.5, rotating the image as a whole according to the rotation picture parameter, rotating the line set obtained by line detection according to the rotation picture parameter to obtain a new line set, and sorting the lines according to the abscissa x1 of the endpoint of the first line in the new line set; Step 2, spine line fitting, includes the following stages: Step 2.1, sliding window sorting phase, specifically includes: Load the line set, set the window value w, the line number is i, and determine whether i is less than w; If the line number i is less than w, the first line to the i+wth line are used as the sliding window; If the line number i is not less than w, determine whether i+w is greater than the total number of lines; If i+w is greater than the total number of lines, the iwth line to the last line is used as the sliding window. If i+w is not greater than the total number of lines, the iwth line to the i+wth line is used as the sliding window. Solve the average angle AvgAngle of all lines in the window. If the angle of the line is in the interval [AvgAngle-Thrangle1, AvgAngle+Thrangle1], then sequentially store the line fitting mark, the calculated line length, the coordinates of the two vertices of the line, and the distance from the line to the midpoint of the left boundary of the image with the average angle as the slope into a list, and sort the list with the distance as a reference; Step 2.2, line fitting stage, specifically includes: Determine whether each line in the new line set is involved in overfitting, if it is a line involved in overfitting, proceed to the next one; if it is a line not involved in overfitting, extend to the left and right sides with the line as the center; Determine whether the left and right lines meet the distance threshold Thrdistance. If so and the line fitting flag is 1, the length of the line is included in the total length. If the left and right lines exceed the distance threshold Thrdistance, determine whether the total length meets the length threshold Thrlength. If the length threshold Thrlength is met, the data is put into the list in the order of leftmost line number, rightmost line number, total length, and filter flag; If the length of the line being fitted is less than the length of the last fitted line in the list, this line is a short line with a small error of the previous line. If the fitting flag of the previous line participating in this filtering is 1, it is changed to 0, and the least squares fitting is performed. This line is stored in the final list until all lines are judged and the final line list is obtained.
2. The line sorting method for dynamically adjusting angles in spine segmentation according to claim 1, characterized in that: In the step 1.4, the angle average of all lines whose lengths are greater than a threshold is taken as the rotation image parameter, and the threshold is 1 / 3 of the image height.
3. The line sorting method for dynamically adjusting angles in spine segmentation according to claim 1, characterized in that: The angle mean AvgAngle is obtained as follows: First, the angle of each line in the window is obtained, and the lines are sorted based on the angle, and the lines are loaded into the loop in order. Each line is extended to the left and right respectively until the lines on the left and right boundaries do not meet the angle threshold Thrangle2, and the angle sum AngleSum centered on the line is obtained; Secondly, take the sum of the largest number of lines, max(AngleSum), and divide it by the number of lines NumLine. The resulting angle is used as the average angle AvgAngle of the main line i of this dynamic window.
