A book collection and editing preprocessing method, device and system
By identifying the text direction and cover orientation on the upper surface of the book, and combining the spine direction to judge the direction of the book, the problem that the existing system cannot recognize the direction of the book is solved, and the processing efficiency of the editing system and the accuracy of label printing is improved.
Patent Information
- Application Number
- CN202210793807.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-15
- Filing Date
- 2022-07-07
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-07-07
AI Technical Summary
The existing book collection and editing system cannot effectively identify the book's flip direction and cover orientation, resulting in inefficient processing.
By obtaining the image on the upper surface of the book, identifying the direction of the text, and judging the direction of the book with the cover orientation and the spine orientation of the book, establishing a corresponding relationship table, calling the corresponding label printing template, and improving the work efficiency of the subsequent execution mechanism.
It realizes accurate identification and processing of the direction of book flipping, improves the working efficiency of the editing system, reduces the working complexity of the marking machine, and promptly alerts to the wrong cover status.
Smart Images

Figure CN115157877B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of book processing, and in particular to a book collection and editing preprocessing method, device and system. Background Art
[0002] The library has a huge amount of books, and the workload of borrowing and returning books is heavy and prone to errors. In order to manage books more conveniently and efficiently, it is necessary to print collection code labels, RFID tags or stamps on the pages of books during the book collection and editing process. There are two types of books in the library: left-hand books and right-hand books. The spine of left-hand books is on the left, and the spine of right-hand books is on the right. When the collection and editing system is operating, it is necessary to adaptively process the mixed left-hand and right-hand books. However, the existing collection and editing system has poor adaptive processing capabilities. It is impossible to obtain the characteristics of the left-hand and right-hand books, the text orientation, etc. before the books enter the execution mechanism, and to perform corresponding preprocessing based on these characteristics, resulting in low processing efficiency. Summary of the invention
[0003] The present invention aims to at least solve the technical problems existing in the prior art and provide a book collection and editing preprocessing method, device and system.
[0004] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a book editing and preprocessing method, comprising: obtaining an upper surface image of a book on a conveying mechanism; identifying the text direction of the upper surface image; obtaining the cover orientation or the back cover orientation of the book, obtaining the spine orientation or the opposite side of the spine orientation of the book; judging the book turning direction based on the text direction on the upper surface of the book, the cover orientation or the back cover orientation, and the spine orientation or the opposite side of the spine orientation, the book turning direction including left-turning and right-turning.
[0005] The above technical solution: before the book is collected and processed, it can identify the direction of book turning, so that subsequent execution mechanisms such as marking devices, page turning devices, etc. can take corresponding measures according to the direction of book turning, thereby improving the work efficiency of subsequent execution mechanisms.
[0006] In a preferred embodiment of the present invention, a plurality of triplets are constructed, wherein the triplets include the direction of text on the upper surface of the book, the orientation of the cover, and the orientation of the spine; or, the triplets include the direction of text on the upper surface of the book, the orientation of the back cover, and the orientation of the spine; or, the triplets include the direction of text on the upper surface of the book, the orientation of the cover, and the orientation of the side opposite to the spine; a correspondence table between the triplets and the direction of turning the book is established, and the direction of turning the book is obtained by searching the correspondence table based on the direction of text on the upper surface of the book, the orientation of the cover or the orientation of the back cover, and the orientation of the spine or the orientation of the side opposite to the spine.
[0007] The above technical solution: by establishing a corresponding relationship table, the book turning direction can be quickly obtained, providing information support for the subsequent execution agency's operations, and improving the work efficiency of the subsequent execution agency.
[0008] In a preferred embodiment of the present invention, a label printing template is called according to the text direction, including: if the text direction is the same as the conveying direction of the conveying mechanism, a first label printing template is called, and the text direction of the first label printing template is the same as the conveying direction; if the text direction is opposite to the conveying direction of the conveying mechanism, a second label printing template is called, and the text direction of the second label printing template is opposite to the conveying direction.
[0009] The above technical solution: identifies the direction of the text on the upper surface of the book, and calls the corresponding label printing template according to the text direction. The subsequent marking machine prints a label that conforms to the direction of the book text according to the called label printing template. This can reduce the work complexity of the subsequent marking machine and improve work efficiency.
[0010] In a preferred embodiment of the present invention, the process of obtaining the orientation of the book cover is: performing ISBN number recognition on the upper surface image, if the ISBN number recognition is successful, it is considered that the book cover is facing downward, if the ISBN number recognition is unsuccessful, it is considered that the book cover is facing upward.
[0011] The above technical solution can identify and determine the orientation of the book cover on the transmission mechanism, so that the subsequent execution mechanism can take corresponding measures according to the orientation of the book cover, thereby further improving the work efficiency of the subsequent execution mechanism.
[0012] In a preferred embodiment of the present invention, a cover orientation determination step is further included, specifically including: when the book cover orientation is different from the set cover orientation, an alarm is issued.
[0013] The above technical solution: issues warning reports for books with incorrect cover status, further improving the work efficiency of subsequent execution agencies.
[0014] In a preferred embodiment of the present invention, the process of obtaining the orientation of the spine of a book is: obtaining a side image of the book on the right side of the conveying track, judging whether the side image is a spine image, if the side image is a spine image, then it is considered that the spine of the book is facing the right side of the conveying track, otherwise, it is considered that the spine of the book is facing the left side of the conveying track; or, obtaining a side image of the book on the left side of the conveying track, judging whether the side image is a spine image, if the side image is a spine image, then it is considered that the spine of the book is facing the left side of the conveying track, otherwise, it is considered that the spine of the book is facing the right side of the conveying track.
[0015] The above technical solution can obtain the direction of the book spine, so that subsequent execution mechanisms such as page turning devices can take corresponding measures according to the direction of the book spine, thereby further improving the work efficiency of subsequent execution mechanisms.
[0016] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a book editing and preprocessing device, including: an image acquisition module, which acquires the upper surface image of the book on the conveying mechanism; a text direction recognition module, which recognizes the text direction of the upper surface image; a book cover orientation acquisition module, which is used to acquire the book cover orientation; a book spine orientation acquisition module, which is used to acquire the book spine orientation; a left-right book turning judgment module, which judges the book turning direction based on the text direction, cover orientation and spine orientation of the book upper surface, and the book turning direction includes left-turning and right-turning.
[0017] The above technical solution can identify the direction of turning the book, so that subsequent execution mechanisms such as marking devices and page turning devices can take corresponding measures according to the direction of turning the book, thereby improving the work efficiency of subsequent execution mechanisms.
[0018] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a book collection and editing preprocessing system, including: a first camera, located above the conveying mechanism, for capturing the upper surface image of the book on the conveying mechanism; a processor, receiving the output image of the output image of the first camera, and executing the steps of the book collection and editing preprocessing method described in the first aspect of the present invention.
[0019] The above technical solution: The system can identify the direction of turning the book, so that subsequent execution mechanisms such as marking devices, page turning devices, etc. can take corresponding measures according to the direction of turning the book, thereby improving the work efficiency of subsequent execution mechanisms.
[0020] In order to achieve the above-mentioned purpose of the present invention, according to the fourth aspect of the present invention, the present invention provides a book collection and editing system, comprising a control module, a conveying mechanism, and a camera module and an actuator installed on the conveying mechanism in sequence along the book conveying direction; the camera module is used to obtain the upper surface image of the book on the conveying mechanism; the actuator comprises a left page turning mechanism, a left page turning mechanism driving device, a right page turning mechanism and a right page turning mechanism driving device; the control module is respectively connected to the camera module, the left page turning mechanism driving device and the right page turning mechanism driving device; the control module obtains the book turning direction according to the book collection and editing preprocessing method described in the first aspect of the present invention, when the book is a left-turning book, the left page turning mechanism is started and the right page turning mechanism is closed, when the book is a right-turning book, the right page turning mechanism is started and the left page turning mechanism is closed.
[0021] The above technical solution can allocate the corresponding left-turning mechanism and right-turning mechanism to work according to the book turning direction, has strong adaptability, and improves the working efficiency of the system.
[0022] In a preferred embodiment of the present invention, the actuator also includes a marking device; the control module is connected to the marking device, and the control module obtains the text direction on the upper surface of the book according to the book editing and preprocessing method described in Example 1. If the text direction is the same as the conveying direction of the conveying mechanism, the marking machine is controlled to call the first label printing template to print the label; if the text direction is opposite to the conveying direction of the conveying mechanism, the printer is controlled to call the second label printing template to print the label.
[0023] The above technical solution: identifies the direction of the text on the upper surface of the book, and calls the corresponding label printing template according to the text direction. The subsequent marking machine prints a label that conforms to the direction of the book text according to the called label printing template. This can reduce the work complexity of the subsequent marking machine and improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a flow chart of the book collection and editing preprocessing method in Example 1 of the present invention. DETAILED DESCRIPTION
[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0026] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0027] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.
[0028] Example 1
[0029] This embodiment discloses a book collection and editing preprocessing method, such as Figure 1 As shown, the method includes:
[0030] Step S1, obtaining the upper surface image of the book on the conveying mechanism. The book to be processed is transmitted to the editing and compiling execution mechanism such as the marking machine and the page turning machine through the conveying mechanism. Generally, the book is squared on the conveying mechanism, and the upper surface image of the book can be obtained by shooting the book from top to bottom through the camera.
[0031] Step S2, identifying the direction of the text on the upper surface image. The surface image can be a back cover image or a front cover image. When the upper surface image is a back cover image, since the numerical direction of the ISBN number is the same as the direction of the back cover text, in order to simplify the text direction recognition process, the text direction of the back cover image can also be determined by identifying the numerical direction of the ISBN number. Image-based text direction recognition is a conventional technology in the field, such as the existing OCR (optical character recognition) algorithm, or the technical solution disclosed in the Chinese patent with publication number CN102855477A or CN110443239A, which will not be repeated here.
[0032] Step S3, obtaining the book cover orientation and book spine orientation.
[0033] In this embodiment, preferably, the process of obtaining the book cover orientation is: ISBN number recognition is performed on the upper surface image. If the ISBN number recognition is successful, the book cover is considered to be facing downwards; if the ISBN number recognition is unsuccessful, the book cover is considered to be facing upwards. Specifically, an ISBN number recognition program can be set to sequentially scan the barcode of each area of the upper surface image. If the barcode scanning is successful, it is considered that the ISBN number exists and the upper surface image is the back cover. Otherwise, it is considered that the ISBN number does not exist and the upper surface image is the front cover. It is also possible to perform text recognition on the upper surface image. If "ISBN" can be recognized, the upper surface is considered to be the back cover. If "ISBN" cannot be recognized, the upper surface is considered to be the front cover. The book back cover orientation can also be obtained according to the above-mentioned book cover orientation judgment process, which will not be repeated here.
[0034] In this embodiment, step S3 may also be to obtain the orientation of the book cover and the orientation of the side opposite to the spine, or to obtain the orientation of the book back cover and the orientation of the spine, or to obtain the orientation of the book back cover and the orientation of the side opposite to the spine.
[0035] In this embodiment, it is further preferred that a cover orientation judgment step is further included, specifically including: when the book cover orientation is different from the set cover orientation, an alarm is given. When the book acquisition system specifies the cover orientation of the book entering the system, it is necessary to give an alarm report for the book with the wrong cover orientation so that the relevant staff can handle it or the corresponding rejection mechanism can reject the book.
[0036] In this embodiment, the direction of the book spine can be known in advance. For example, some editing systems require that the book spine be placed in a set direction when the book is online. In this case, the direction of the book spine can be known in advance. The judgment of the book spine direction can also be achieved through the following scheme: a distance sensor is set above the transmission track to measure the distance between the distance sensor and the upper surface of the book below (the book is required to be stationary relative to the transmission channel when judging the direction of the book spine, such as the transmission channel is suspended), and a blowing device is set on the left / right side of the transmission track to blow air toward the side of the book. The direction of the book spine is judged based on whether the output signal of the distance sensor before and after the blowing device blows. If the signal changes, it is considered that the book spine is facing the right / left side of the transmission track. If the signal does not change, it is considered that the book spine is facing the left / right side of the transmission track. Specifically, it can be considered that the amplitude change of the output signal of the distance sensor exceeds plus or minus 2%, and the signal is considered to have changed. The distance sensor is preferably, but not limited to, an existing laser distance sensor. The blowing device is preferably, but not limited to, an existing turbine fan.
[0037] In this embodiment, preferably, in order to improve the accuracy of spine direction judgment and make it easier to implement, the process of obtaining the direction of the book spine includes: obtaining a side image of the book on the right side of the conveyor track, and judging whether the side image is a spine image; if the side image is a spine image, it is considered that the spine of the book is facing the right side of the conveyor track; otherwise, it is considered that the spine of the book is facing the left side of the conveyor track; or, obtaining a side image of the book on the left side of the conveyor track, and judging whether the side image is a spine image; if the side image is a spine image, it is considered that the spine of the book is facing the left side of the conveyor track; otherwise, it is considered that the spine of the book is facing the right side of the conveyor track.
[0038] In this embodiment. Preferably, book spine recognition is performed through a machine learning model. Multiple book spine images are obtained as positive samples, and multiple book page images (opposite sides of the book spine) are obtained as negative samples, and positive samples are labeled "yes", and negative samples are labeled "no", and all samples are divided into training sets, verification sets, and test sets. A neural network structure is established, such as an HRNet network. The neural network structure is trained with the training set, and the trained neural network is tested and verified with the test set and verification set to obtain a book spine recognition model, and the side image is input into the book spine recognition model, and the book spine recognition model outputs "yes" or "no". Based on the above process, the prior art has realized book spine recognition, such as the Chinese patent with publication number CN111401252A. The technical solution disclosed in the Chinese patent with publication number CN104966081A can also be used.
[0039] In this embodiment, the above-mentioned method for determining the direction of the book spine can also be applied to determining the direction of the opposite side of the book spine (ie, the side of the book pages that can be turned), which will not be described in detail here.
[0040] Step S4, judging the book turning direction based on the text direction on the upper surface of the book, the front cover direction or the back cover direction, and the spine direction or the opposite side of the spine direction, the book turning direction includes left turning and right turning.
[0041] In this embodiment, preferably, in order to quickly obtain the direction of turning the book, the following scheme is adopted: construct multiple triplets, the triplets include the text direction on the upper surface of the book, the cover orientation and the spine orientation, or the triplets include the text direction on the upper surface of the book, the back cover orientation and the spine orientation, or the triplets include the text direction on the upper surface of the book, the cover orientation and the opposite side orientation of the spine, or the triplets include the text direction on the upper surface of the book, the back cover orientation and the opposite side orientation of the spine; establish a correspondence table between the triplets and the direction of turning the book, and obtain the direction of turning the book based on the text direction on the upper surface of the book, the cover orientation or the back cover orientation, and the spine orientation or the opposite side orientation of the spine by searching the relationship correspondence table. The direction of turning the book can be obtained by looking up the correspondence table.
[0042] In this embodiment, preferably, the triplet includes the text direction, cover orientation, and spine orientation of the book surface, and the specific search process includes:
[0043] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces upward, and the spine faces the right side of the conveying track, the book is considered to be a left-opening book;
[0044] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces downward, and the spine faces the right side of the conveying track, the book is considered to be a right-opening book;
[0045] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces upward, and the spine faces the right side of the conveying track, the book is considered to be a right-opening book;
[0046] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces downward, and the spine faces the right side of the conveying track, the book is considered to be a left-opening book;
[0047] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces upward, and the spine faces the left side of the conveying track, the book is considered to be a right-opening book;
[0048] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces downward, and the spine faces the left side of the conveying track, the book is considered to be a left-opening book;
[0049] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces upward, and the spine faces the left side of the conveying track, the book is considered to be a left-opening book;
[0050] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces downward, and the spine faces the left side of the conveying track, the book is considered to be a right-opening book.
[0051] In this embodiment, preferably, the triplet includes the text direction on the upper surface of the book, the back cover direction, and the spine direction. The specific search process includes:
[0052] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover is facing downward, and the spine is facing the right side of the conveying track, the book is considered to be a left-opening book;
[0053] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover faces upward, and the spine faces the right side of the conveying track, the book is considered to be a right-opening book;
[0054] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover is facing downward, and the spine is facing the right side of the conveying track, the book is considered to be a right-opening book;
[0055] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover faces upward, and the spine faces the right side of the conveying track, the book is considered to be a left-opening book;
[0056] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover is facing downward, and the spine is facing the left side of the conveying track, the book is considered to be a right-opening book;
[0057] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover faces upward, and the spine faces the left side of the conveying track, the book is considered to be a left-opening book;
[0058] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover is facing downward, and the spine is facing the left side of the conveying track, the book is considered to be a left-opening book;
[0059] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover is facing upward, and the spine is facing the left side of the conveying track, the book is considered to be a right-opening book.
[0060] In this embodiment, preferably, the triplet includes the text direction on the upper surface of the book, the cover orientation, and the orientation of the opposite side of the spine. The specific search process includes:
[0061] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces upward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a left-opening book;
[0062] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces downward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a right-opening book;
[0063] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces upward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a right-opening book;
[0064] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces downward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a left-opening book;
[0065] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces upward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a right-opening book;
[0066] When the direction of the text on the upper surface of the book is the same as the conveying direction, the cover faces downward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a left-opening book;
[0067] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces upward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a left-opening book;
[0068] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the cover faces downward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a right-opening book.
[0069] In this embodiment, preferably, the triplet includes the text direction on the upper surface of the book, the back cover direction, and the direction of the opposite side of the spine. The specific search process includes:
[0070] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover faces downward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a left-opening book;
[0071] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover faces upward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a right-opening book;
[0072] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover faces downward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a right-opening book;
[0073] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover faces upward, and the opposite side of the spine faces the left side of the conveying track, the book is considered to be a left-opening book;
[0074] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover faces downward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a right-opening book;
[0075] When the direction of the text on the upper surface of the book is the same as the conveying direction, the back cover faces upward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a left-opening book;
[0076] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover faces downward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a left-opening book;
[0077] When the direction of the text on the upper surface of the book is opposite to the conveying direction, the back cover faces upward, and the opposite side of the spine faces the right side of the conveying track, the book is considered to be a right-opening book.
[0078] In this embodiment, preferably, calling the label printing template according to the text direction includes: if the text direction is the same as the conveying direction of the conveying mechanism, calling the first label printing template, the text direction of the first label printing template is the same as the conveying direction; if the text direction is opposite to the conveying direction of the conveying mechanism, calling the second label printing template, the text direction of the second label printing template is opposite to the conveying direction. Preferably, a memory is set to store the first label printing template and the second label printing template.
[0079] In this embodiment, further preferably, the step of obtaining the best labeling position is further included, including:
[0080] Step S101, obtaining the upper surface image. Preferably, in order to improve the accuracy of obtaining and reduce interference images, the front image of the page to be labeled can be photographed by a camera, and the upper surface image can be segmented from the front image. Existing image segmentation methods, such as existing mask methods, can be used to segment the upper surface image from the front image. The upper surface image is preferably, but not limited to, the back cover or front cover of a book.
[0081] Preferably, in order to improve processing efficiency, before entering step S102, the upper surface image is subjected to text detection to obtain the text area, and the graphic detection is performed to obtain the graphic area. If the remaining area after subtracting the text area and the graphic area from the upper surface image cannot accommodate a label, the labeling sub-area is set at the non-first line of text or non-mean text of the upper surface image, so as to minimize the visual impact on the reader. If the remaining area can accommodate at least one label, then enter step S102. Text detection can be implemented using existing algorithms, such as the CTPN algorithm disclosed in the website https: / / zhuanlan.zhihu.com / p / 34757009, and the technical solutions disclosed in the Chinese patents with publication numbers CN105205488A, CN103699895A, or CN104050471A, which will not be repeated here. Graphic detection is used to detect various patterns and graphics such as lines, circles, ellipses, triangles, etc. in the upper surface image. Existing graphic detection algorithms can be used, such as the technical solution disclosed in the Chinese patent with publication number CN1264096A, and the existing object detection algorithm based on the solov2 neural network, which will not be repeated here. It is judged whether the remaining area can accommodate the label. If the length of a sub-area of the remaining area is greater than the length of the label and the width is greater than the width of the label, it is considered that the label can be accommodated, otherwise, it is considered that the label cannot be accommodated.
[0082] Step S102: select a labelable area in the upper surface image, the labelable area includes at least one labeling sub-area matching the label size, the proportion of text and graphics in the labelable area is less than or equal to a proportion threshold, and the proportion threshold ranges from 0 to 0.7.
[0083] In this embodiment, selecting a labelable area in the upper surface image may specifically include: obtaining at least one labeling sub-area matching the label size in the area of the page to be labeled, the proportion of text and graphics in the labeling sub-area is less than or equal to a preset proportion threshold, and the labelable area is composed of at least one labeling sub-area. A sliding window matching the label size is set, and the sliding window slides sequentially on the upper surface image, which can be a sequential slide from left to right and from top to bottom. The sliding step can be set to half, one third, or one quarter of the label width, and the sliding window area with a proportion of text and graphics less than or equal to the proportion threshold is selected as the labeling sub-area. Further preferably, in order to minimize the occlusion of text and graphics by labels, the sliding window area with the smallest proportion of text and graphics is selected as the target labeling sub-area among all sliding window areas that meet the condition that the proportion of text and graphics is less than or equal to the proportion threshold, such as selecting a sliding window area with a proportion of 0 as the target labeling sub-area. The text detection and graphic detection in the sliding window area can adopt the existing algorithm, which has been described in detail in the above description and will not be repeated here. The sum of the pixels in the text area obtained by text detection and the pixel points in the graphic area obtained by graphic detection is counted, and the ratio of the sum of the pixels to the total number of pixels in the sliding window is taken as the proportion of the text and image in the sliding window area.
[0084] In this embodiment, in order to improve efficiency and make the extracted labelable area more accurate and meet the requirements, preferably, in step S102, the labelable area is obtained by an intelligent algorithm, that is, the labelable area is selected in the upper surface image, specifically including:
[0085] Step S1021, obtain the energy distribution map of the upper surface image, preferably but not limited to obtaining the energy distribution map through the existing image Fourier transform, such as the technical solution disclosed in the website https: / / blog.csdn.net / denghecsdn / article / details / 78906126. The energy distribution map can be more recognized by the neural network model, and fully considers the characteristics that the text and graphics on the page image are all highlighted parts, and their pixel energy is higher, and the page background is not prominent, and its pixel energy is lower. Therefore, the energy distribution map can effectively distinguish the text, image and background, which is beneficial to the neural network learning feature law of the labelable area acquisition model, and improves the marking accuracy of the labelable area.
[0086] Step S1022, input the energy distribution map into the pre-trained labelable area acquisition model.
[0087] Step S1023: The labelable area acquisition model outputs an energy distribution map of the labelable area.
[0088] Step S1024: Map the labelable area mark in the energy distribution map of the labelable area to the upper surface image, and output the upper surface image of the labelable area. A labeling sub-area sliding window matching the label size can be used to slide in the labelable area, and the area in the labelable area that can contain the labeling sub-area sliding window is output as a labeling sub-area.
[0089] In this embodiment, further preferably, in order to improve the accuracy, stability and reliability of the labelable area acquisition model, the construction process of the labelable area acquisition model includes:
[0090] Step A1, obtaining multiple upper surface images, including at least 2000 upper surface images, one upper surface image is used as a sample, and a labelable area is marked on each sample. The marking method is preferably but not limited to a marking box, and can be manually marked. The upper surface image with the marked labelable area is used as the sample label of the original sample. A portion of the samples have correctly marked labelable areas, where a correctly marked area means that the proportion of text and graphics in the marked labelable area is less than or equal to a proportion threshold, and are recorded as positive sample labels. Another portion of the samples have incorrectly marked labelable areas, where an incorrectly marked area means that the proportion of text and graphics in the marked labelable area is greater than a proportion threshold, and are recorded as negative sample labels.
[0091] Step A2, dividing all samples into a training set, a validation set and a test set, wherein the training set includes positive sample label samples and negative sample label samples, the validation set includes positive sample label samples, and the test set includes positive sample label samples;
[0092] Step A3, constructing a neural network structure of a labelable region acquisition model, wherein the neural network structure is preferably, but not limited to, a HRNet network structure;
[0093] Step A4, using the training set to train the neural network structure, then using the verification set to verify the trained neural network structure, and adjusting the neural network structure parameters according to the verification result to obtain a labelable area acquisition model;
[0094] Step A5: Use the test set to test the labelable area acquisition model to verify its performance.
[0095] Step S103, outputting the position information of at least one labeling sub-area on the page to be labeled. The position information of the labeling sub-area on the page to be labeled is preferably in the form of, but not limited to, the position coordinates of the center point of the labeling sub-area on the coordinate system of the page to be labeled, and the coordinate system of the page to be labeled is a coordinate system x'o'y' with the book width as the x' axis, the book length as the y' axis, and the lower left corner of the upper surface image of the book as the origin o'. Further preferably, in order to facilitate user selection, the position information of at least one labeling sub-area on the page to be labeled can be output to the display, and when displayed, the shape of each labeling sub-area and the center point position coordinates are marked on the upper surface image.
[0096] In this embodiment, further preferably, in order to reduce the occlusion of text and images by the sticker, it also includes outputting the preferred weight of each labeling sub-area while outputting the position information of the labeling sub-area on the page to be labeled. The preferred weight of the labeling sub-area is negatively correlated with the proportion of text and graphics in the labeling sub-area. The larger the proportion of text and graphics in the labeling sub-area, the smaller the preferred weight of the labeling sub-area. Each labeling sub-area is assigned a preferred weight, and the range of the preferred weight is preferably but not limited to 0 to 1. The larger the preferred weight, the better it is recommended to select the labeling sub-area corresponding to the preferred weight.
[0097] In this embodiment, it is further preferred that, in order to improve the visual effect of the label and facilitate people to quickly find the label by observation, it also includes outputting the preferred weight of each labeling sub-area while outputting the position information of the labeling sub-area on the page to be labeled, and the preferred weight of the labeling sub-area is positively correlated with the degree of centering of the labeling sub-area in the width direction of the page to be labeled. The centering degree calculation process is as follows: let the coordinates of the center point of the labeling sub-area be (x', y'), in the coordinate system x'o'y', the coordinates of the lower left corner of the page in the upper surface image are (0, 0), and the coordinates of the lower right corner of the page in the upper surface image are (d, 0), d is the width of the book, then the centering degree of the labeling sub-area in the width direction of the page to be labeled is: the ratio of the minimum value of (d-x') and x' to the maximum value, the numerical range of the centering degree is 0 to 1, the closer to 1, the better the centering of the labeling sub-area.
[0098] In this embodiment, preferably, in order to comprehensively consider the influence of the centering degree midle of the labeling sub-area and the proportion P of text and graphics in the labeling sub-area on the preferred weight, the (midle-P) value is calculated, and the value obtained after normalizing the (midle-P) value is used as the preferred weight of the labeling sub-area.
[0099] In this embodiment, in the above-mentioned labeling position selection step, the proportion of text and graphics in the selected labelable area is less than or equal to the proportion threshold, that is, the ratio of the total area occupied by the text and graphics in the labeling area to the total area of the labelable area is less than or equal to the proportion threshold, and the proportion threshold can be set according to user needs, which can reduce or avoid the label covering the text and graphics on the page to be labeled, improve the page readability, and realize the adaptive and automatic selection of the best labeling position according to the distribution characteristics of the text and graphics on the page to be labeled.
[0100] Example 2
[0101] The present embodiment discloses a book collection and editing preprocessing device, which corresponds to the embodiment 1 and includes: an image acquisition module for acquiring an upper surface image of a book on a conveying mechanism; a text direction recognition module for recognizing the text direction of the upper surface image; a book spine orientation acquisition module for acquiring the book spine orientation; a left-right book turning judgment module for judging the book turning direction based on the text direction on the upper surface of the book, the cover orientation and the spine orientation, the book turning directions including left-turning and right-turning.
[0102] Example 3
[0103] The present embodiment discloses a book collection and editing preprocessing system, comprising: a first camera, located above a conveying mechanism, for capturing an image of an upper surface of a book on the conveying mechanism; a processor, receiving an output image of an output image of the first camera, and executing the steps of the book collection and editing preprocessing method of embodiment 1.
[0104] In this embodiment, preferably, a second camera is further included, and the second camera is located on the right side or the left side of the conveying mechanism, and is used to capture the side image of the book on the conveying mechanism.
[0105] Example 4
[0106] The present embodiment discloses a book collection and editing system, comprising a control module, a conveying mechanism, and a camera module and an actuator sequentially installed on the conveying mechanism along the book conveying direction; the camera module is used to obtain the upper surface image of the book on the conveying mechanism; the actuator comprises a left page turning mechanism, a left page turning mechanism driving device, a right page turning mechanism and a right page turning mechanism driving device; the control module is respectively connected to the camera module, the left page turning mechanism driving device and the right page turning mechanism driving device, and the control module obtains the book turning direction according to the book collection and editing preprocessing method of embodiment 1, when the book is a left-turning book, the left page turning mechanism is started and the right page turning mechanism is closed, when the book is a right-turning book, the right page turning mechanism is started and the left page turning mechanism is closed.
[0107] In this embodiment, the camera module preferably includes, but is not limited to, a first camera and a second camera. The first camera and the second camera can be arranged at the same position of the conveying mechanism. Preferably, both cameras are arranged at the starting end of the conveying mechanism. A mounting frame can be arranged, which is fixedly mounted above the conveying mechanism and connected to the left and right sides of the conveying mechanism. The first camera is mounted above the mounting frame, and downwardly shoots the upper surface image of the book on the conveying mechanism. The second camera is mounted on the left or right side of the conveying mechanism, and is used to shoot the side image of the book.
[0108] In this embodiment, the control module is preferably but not limited to a processing control module such as PLC, PC, ARM, FPGA, etc. The conveying mechanism is preferably but not limited to a roller conveying mechanism or a belt conveying mechanism, and the belt conveying mechanism can be a technical solution disclosed in the Chinese patent publication number CN112537515A.
[0109] In this embodiment, when the book is a left-turning book, the control module controls the driving device of the left page turning mechanism to start the left page turning mechanism, and controls the driving device of the right page turning mechanism to close the right page turning mechanism. When the book is a right-turning book, the control module controls the driving device of the right page turning mechanism to start the right page turning mechanism, and controls the driving device of the left page turning mechanism to close the left page turning mechanism. The left page turning mechanism and the right page turning mechanism are respectively the first left-leaning page turning device and the first right-leaning page turning device in the Chinese patent with publication number CN112537515A, which will not be described in detail here.
[0110] In this embodiment, preferably, the actuator also includes a marking device; the control module is connected to the marking device, and the control module obtains the text direction on the upper surface of the book according to the book editing and preprocessing method of Example 1. If the text direction is the same as the conveying direction of the conveying mechanism, the marking machine is controlled to call the first label printing template to print the label; if the text direction is opposite to the conveying direction of the conveying mechanism, the printing device is controlled to call the second label printing template to print the label.
[0111] In this embodiment, the labeling device may also adopt the first left labeling device or the first right labeling device in the Chinese patent publication number CN112537515A, or the labeling device disclosed in the Chinese patent publication number CN213566881U.
[0112] In this embodiment, preferably, the execution mechanism also includes a left page turning mechanism and a right page turning mechanism; the control module obtains the book turning direction according to the book editing and preprocessing method of Example 1. When the book is a left-turning book, the left page turning mechanism is started and the right page turning mechanism is closed. When the book is a right-turning book, the right page turning mechanism is started and the left page turning mechanism is closed.
[0113] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0114] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
Claims
1. A book collection and editing preprocessing method, characterized in that: include: Acquire an upper surface image of a book on a conveying mechanism; identifying the direction of text in the upper surface image; Get the book cover orientation or back cover orientation, get the book spine orientation or the opposite side of the spine orientation; Determine the book turning direction based on the text direction on the upper surface of the book, the front cover direction or the back cover direction, and the spine direction or the opposite side of the spine direction, wherein the book turning direction includes left-hand turning and right-hand turning; It also includes selecting labelable areas in the upper surface image through intelligent algorithms, including: Step S1021, obtaining an energy distribution map of the upper surface image; Step S1022, inputting the energy distribution map into a pre-trained labelable area acquisition model; Step S1023, the labelable area acquisition model outputs an energy distribution map of the labelable area; Step S1024, mapping the labelable area mark in the energy distribution map of the labelable area to the upper surface image, and outputting the upper surface image of the labelable area; The process of building a labelable area acquisition model includes: Step A1, obtain multiple upper surface images, one upper surface image is used as a sample, and the labelable area is marked on each sample. The upper surface image with the labelable area marked is used as the sample label of the original sample. The labelable area of some samples is correctly marked and recorded as the positive sample label, and the labelable area of the other samples is incorrectly marked and recorded as the negative sample label; Step A2, dividing all samples into a training set, a validation set and a test set, wherein the training set includes positive sample label samples and negative sample label samples, the validation set includes positive sample label samples, and the test set includes positive sample label samples; Step A3, constructing a neural network structure of a labelable region acquisition model; Step A4, using the training set to train the neural network structure, then using the verification set to verify the trained neural network structure, and adjusting the neural network structure parameters according to the verification result to obtain a labelable area acquisition model; Step A5: Use the test set to test the labelable area acquisition model.
2. The book collection and editing preprocessing method according to claim 1, characterized in that: Construct a plurality of triplets, wherein the triplets include the direction of text on the upper surface of the book, the orientation of the cover, and the orientation of the spine, or the triplets include the direction of text on the upper surface of the book, the orientation of the back cover, and the orientation of the spine, or the triplets include the direction of text on the upper surface of the book, the orientation of the cover, and the orientation of the side opposite to the spine; establish a correspondence table between the triplets and the direction of turning the book, and obtain the direction of turning the book based on the direction of text on the upper surface of the book, the orientation of the cover or the orientation of the back cover, and the orientation of the spine or the orientation of the side opposite to the spine.
3. The book collection and editing preprocessing method according to claim 1 or 2, characterized in that: Calling the label printing template according to the text direction includes: if the text direction is the same as the conveying direction of the conveying mechanism, calling the first label printing template, the text direction of the first label printing template is the same as the conveying direction; if the text direction is opposite to the conveying direction of the conveying mechanism, calling the second label printing template, the text direction of the second label printing template is opposite to the conveying direction.
4. The book collection and editing preprocessing method according to claim 1 or 2, characterized in that: The process of obtaining the book cover orientation is: performing ISBN number recognition on the upper surface image, if the ISBN number recognition is successful, it is considered that the book cover is facing downward, if the ISBN number recognition is unsuccessful, it is considered that the book cover is facing upward.
5. The book collection and editing preprocessing method according to claim 3, characterized in that: The process of obtaining the book cover orientation is: performing ISBN number recognition on the upper surface image, if the ISBN number recognition is successful, it is considered that the book cover is facing downward, if the ISBN number recognition is unsuccessful, it is considered that the book cover is facing upward.
6. The book collection and editing preprocessing method according to claim 4, characterized in that: The method also includes a cover orientation judgment step, which specifically includes: when the book cover orientation is different from the set cover orientation, an alarm prompt is given.
7. The book collection and editing preprocessing method according to claim 1, 2, 5 or 6, characterized in that: The process of obtaining the book spine orientation is: Acquire a side image of the book on the right side of the conveying track, and determine whether the side image is a spine image. If the side image is a spine image, it is considered that the spine of the book faces the right side of the conveying track; otherwise, it is considered that the spine of the book faces the left side of the conveying track; Alternatively, a side image of the book is obtained on the left side of the conveying track to determine whether the side image is a spine image. If the side image is a spine image, it is considered that the spine of the book is facing the left side of the conveying track; otherwise, it is considered that the spine of the book is facing the right side of the conveying track.
8. The book collection and editing preprocessing method according to claim 3, characterized in that: The process of obtaining the book spine orientation is: Acquire a side image of the book on the right side of the conveying track, and determine whether the side image is a spine image. If the side image is a spine image, it is considered that the spine of the book faces the right side of the conveying track; otherwise, it is considered that the spine of the book faces the left side of the conveying track; Alternatively, a side image of the book is obtained on the left side of the conveying track to determine whether the side image is a spine image. If the side image is a spine image, it is considered that the spine of the book is facing the left side of the conveying track; otherwise, it is considered that the spine of the book is facing the right side of the conveying track.
9. The book collection and editing preprocessing method according to claim 4, characterized in that: The process of obtaining the book spine orientation is: Acquire a side image of the book on the right side of the conveying track, and determine whether the side image is a spine image. If the side image is a spine image, it is considered that the spine of the book faces the right side of the conveying track; otherwise, it is considered that the spine of the book faces the left side of the conveying track; Alternatively, a side image of the book is obtained on the left side of the conveying track to determine whether the side image is a spine image. If the side image is a spine image, it is considered that the spine of the book is facing the left side of the conveying track; otherwise, it is considered that the spine of the book is facing the right side of the conveying track.
10. A book collection and editing preprocessing device, used to implement the book collection and editing preprocessing method according to any one of claims 1 to 9, characterized in that: include: An image acquisition module, for acquiring an upper surface image of a book on a conveying mechanism; A text direction recognition module, for recognizing the text direction of the upper surface image; A book cover orientation acquisition module is used to acquire the book cover orientation or back cover orientation; A book spine orientation acquisition module, used to acquire the book spine orientation or the orientation of the opposite side of the spine; The book turning direction is determined based on the text direction on the upper surface of the book, the front cover orientation or the back cover orientation, and the spine orientation or the spine opposite side orientation, and the book turning direction includes left-turning and right-turning.
11. A book collection and editing preprocessing system, characterized in that: include: A first camera is located above the conveying mechanism and is used to capture an image of the upper surface of the book on the conveying mechanism; The processor receives the output image of the output image of the first camera and executes the steps of the book editing preprocessing method described in any one of claims 1-9.
12. A book collection and editing system, characterized in that: It includes a control module, a conveying mechanism, and a camera module and an actuator which are sequentially installed on the conveying mechanism along the book conveying direction; The camera module is used to obtain the upper surface image of the book on the conveying mechanism; The actuator includes a left page turning mechanism, a left page turning mechanism driving device, a right page turning mechanism and a right page turning mechanism driving device; The control module is respectively connected to the camera module, the left page turning mechanism driving device and the right page turning mechanism driving device. The control module obtains the book turning direction according to the book editing and preprocessing method described in one of claims 1 to 9. When the book is a left-turning book, the left page turning mechanism is started and the right page turning mechanism is closed. When the book is a right-turning book, the right page turning mechanism is started and the left page turning mechanism is closed.
13. The book collection and editing system according to claim 12, characterized in that: The actuator also includes a marking device; The control module is connected to the marking device. The control module obtains the text direction on the upper surface of the book according to the book editing and preprocessing method described in one of claims 1 to 9. If the text direction is the same as the conveying direction of the conveying mechanism, the marking machine is controlled to call the first label printing template to print the label. If the text direction is opposite to the conveying direction of the conveying mechanism, the printing device is controlled to call the second label printing template to print the label.
