A book positioning method based on computer vision

Abstract

The present invention is a book positioning method based on computer vision, which includes: step S1, taking a photo with a camera to collect the target area of ​​the book and store it as an image; step S2, building and training a text detection model, marking all text in the image with a text area frame; step S3, based on the text area frame, segmenting the image into book instances, thereby obtaining book instances and their location information in the image; step S4, building and training a text recognition model, identifying the text area frame, and merging the text recognition results in the same book instance to obtain the title recognition result of each book instance; step S5, matching the title recognition result with the book name input by the reader to be queried on the bookshelf, and outputting the positioning result of the target book in the image. Through the above method, the present application achieves high-efficiency and high-accuracy book positioning, and improves the efficiency of book borrowing.

Description

Technical Field

[0001] The invention belongs to the field of computer vision design, and in particular relates to a book positioning method based on computer vision. Background Art

[0002] Compared to online resources, which can be accessed anytime, anywhere, searching for specific books in a library is not easy. Readers need to search for the book of interest through the library management system and then search the corresponding shelves until they find it. In real-world scenarios, this process is not only labor-intensive and time-consuming, but also sometimes difficult to find the desired book on the corresponding shelf due to the library's frequent book withdrawals and shelving, resulting in a waste of resources and time.

[0003] To address this issue, researchers have proposed the idea of ​​automated libraries. Leveraging rapidly advancing robotics technology, robotic arms would automatically grasp books from shelves and deliver them to readers via guided robots, aiming to achieve fully automated book retrieval and delivery. Among the numerous books on the shelves, accurately identifying the target book and determining its precise location is crucial. Therefore, most libraries attach RFID tags to books. However, using RFID to determine the book's location presents two challenges: first, due to multipath effects and inter-tag interference, RFID positioning accuracy cannot meet grasping requirements; second, RFID-based book positioning can only locate the tag, not the book itself, making it ineffective in assisting with book grasping. In recent years, with the rapid development of computer hardware and software, computer vision technology has advanced rapidly, bringing significant benefits to various industries. Computer vision, a technology that enables computers to possess human-like visual capabilities, can understand visual data such as images and videos, and is capable of performing tasks such as facial recognition, image classification, and object detection. Therefore, researchers have proposed applying computer vision technology to book positioning.

[0004] Chinese patent CN111814935A discloses a book location method based on an inventory robot, including the following steps: S1. Data acquisition: The robot uses RFID recognition technology to obtain the electronic tag of the book and transmits it, along with the robot's coordinates and antenna height, to a processor module; S2. Data processing and conversion: The book code is extracted from the electronic tag and stored in a data storage module. The processor reads the data, converts it into an array matrix, and performs averaging on each data point; S3. Book location: The processed data in S2 is input into the book location module. After finding the location tag through a network model, the location tag and the book information are transmitted to the data reading module to locate the book. The network model is used for feature extraction, which can more accurately and quickly locate the location of the book, significantly reducing the workload of library administrators and improving work efficiency. However, the method clearly has significant problems in data processing and camera visual positioning.

[0005] Chinese patent CN112464682A discloses a device and method for locating books on a smart bookshelf based on RFID technology. The device comprises: a tag; a routing device, one for each shelf of the smart bookshelf; multiple antennas, each evenly distributed on each shelf; a reader / writer; and a controller for issuing control instructions to control the routing device to select the corresponding antenna, and obtaining the tag information read by the reader / writer, the antenna code corresponding to the tag information, and the RSSI value corresponding to the tag information. If the tag information corresponds to multiple antenna codes, the antenna code with the larger RSSI value is saved. The device has multiple antennas on each shelf to ensure that each tag is recognized. When the same tag is read by multiple antennas, the RSSI value is selected and saved, eliminating the problem of misplaced shelf positioning, thereby resolving the conflict between reading all tags and accurately locating books, and achieving the goal of accurately locating books.

[0006] However, in practical applications, the above technology faces many challenges: in order to meet the needs of robotic arm grasping, the book positioning accuracy must be high enough; and the books in the library are densely arranged and the textures are highly similar. There are many books in the photos, which increases the obstacles to book recognition and makes positioning technology difficult to implement.

[0007] Therefore, developing a book positioning method based on computer vision has a profound impact on improving the utilization efficiency of library books, and also plays an important role in saving readers' time in consulting information. Summary of the Invention

[0008] In order to solve the above problems, a book positioning method based on computer vision is developed to achieve high efficiency and high accuracy of book positioning, thereby improving the efficiency of library book utilization and saving readers' time in reading materials.

[0009] In order to achieve the above effects, the present invention designs a book positioning method based on computer vision;

[0010] A book positioning method based on computer vision, the method comprising:

[0011] Step S1: take a picture with a camera to capture the target area of ​​the book and store it as a picture;

[0012] Step S2: Build and train a text detection model to mark all text in the image with text area boxes;

[0013] Step S3: Segmenting the picture into book instances based on the text region frame, thereby obtaining book instances and location information of the book instances in the picture;

[0014] Step S4: construct and train a text recognition model to recognize the text area frame, and merge the text recognition results in the same book instance to obtain the title recognition result of each book instance;

[0015] Step S5: Match the book title recognition result with the book name input by the reader to be searched on the bookshelf, and output the positioning result of the target book in the picture.

[0016] Preferably, the method of constructing and training the text detection model in step S2 includes:

[0017] Step S21: perform data enhancement on both artificially synthesized and real scene text detection datasets;

[0018] Step S22: Based on the EAST model, modify the model's definition of the "lower" boundary and implement a GPU-accelerated non-maximum suppression algorithm to improve model training and reasoning efficiency;

[0019] Step S23: Based on the enhanced text detection dataset, train the EAST model: first pre-train the model on an artificially synthesized dataset, and then fine-tune the model on a real-scene dataset to obtain a robust text detection model.

[0020] Preferably, in step S3, the method for segmenting book instances includes:

[0021] Step S31, using five parameters {d i |i∈{1,2,3,4}},θ defines the spine box;

[0022] Step S32: applying a straight line segment detection algorithm to the image, extracting continuous long line segments in the image, outputting a line segment graph, and generating a unit normal graph through the line segment graph;

[0023] Step S33: defining an energy function for the spine frame: the energy function reaches a minimum when each side of the spine frame fits the line segment graph and the normal vectors of each side of the spine frame are in the same direction as the normal graph;

[0024] Step S34: Use the text area frame as the initial spine frame, continuously expand the spine frame outward until the energy function reaches a minimum, output the spine frame at this time, and perform screening.

[0025] Preferably, in step S34, the screening method includes:

[0026] Step S341: When a book edge corresponds to two or more spine boxes with similar shapes, non-maximum suppression is applied to the two or more spine boxes with similar shapes, thereby obtaining a more accurate book instance segmentation result;

[0027] Step S342: Eliminate spine frames with an aspect ratio less than 3:1;

[0028] Step S343: When there is overlap between spine frames, screening is performed based on spatial relationships.

[0029] Preferably, the method of constructing and training the text recognition model in step S4 includes:

[0030] Step S41: perform data enhancement on both artificially synthesized and real-scene text recognition datasets;

[0031] Step S42: construct a text recognition model based on the CRNN model;

[0032] Step S43: Based on the enhanced data set, train a text recognition model: first pre-train the model on an artificially synthesized data set, and then fine-tune the model on a real-scene data set to obtain a robust text recognition model.

[0033] Preferably, in step S5, the method for matching book names includes:

[0034] Step S51: Convert the title recognition result and the reader-input title into word vectors respectively. The length of the vector is equal to the number of all words, and the value is the TF-IDF weight of the corresponding word.

[0035] Step S52: Based on the vector, the cosine similarity is used to measure the matching degree between the book title input by the reader and each book title recognition result. The book title result with the highest similarity can be matched with the input book title, and the matching results with similarity lower than the set threshold are eliminated. The matching results with higher confidence and the positioning information of the input book title in the image are output. If multiple book titles are input, the process is repeated multiple times, and a match is performed for each book title input by the reader.

[0036] Preferably, the data enhancement method in step S21 includes: randomly changing the brightness, color, contrast and other features of the image, and randomly rotating and cropping the image.

[0037] Preferably, in the step S1, the process of collecting pictures further includes using a homography transformation matrix to eliminate perspective deformation in the pictures, and the homography transformation matrix is ​​obtained by a vanishing point detection algorithm.

[0038] Preferably, in the step S1, the process of collecting pictures further includes using intrinsic and extrinsic parameters of the camera to perform distortion correction on the collected pictures, and the intrinsic and extrinsic parameters are obtained by calibrating the camera.

[0039] Preferably, before step S1, the method further includes: S0, constructing and training a text detection model and constructing and training a text recognition model;

[0040] The step S2 is replaced by: marking all texts in the picture with a text area frame;

[0041] The step S4 is replaced by: identifying the text area frame, and merging the text recognition results in the same book instance to obtain the title recognition result of each book instance.

[0042] The advantages and effects of this application are as follows:

[0043] 1. This application introduces computer vision into the field of book positioning. Through technologies such as text detection, book instance segmentation, text recognition and text matching, it realizes an efficient book positioning model; thereby achieving high-efficiency and high-accuracy book positioning, and ultimately achieving the technical effect of improving the utilization efficiency of library books and saving readers' time in looking up information.

[0044] 2. This application designs a new book instance segmentation algorithm based on text region boxes, which can achieve millimeter-level positioning of books on the captured image, and accurately determine the location of the book; when readers need to borrow books, this method can quickly and accurately provide the location of the book, thereby instructing the robotic arm to automatically grab the book on the bookshelf and deliver it to the reader through the navigation robot, providing a basis for the realization of an automated library.

[0045] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiment of the present application in conjunction with the accompanying drawings.

[0046] Based on the detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without inventive work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0048] Figure 1 A flowchart of a book positioning method based on computer vision provided by the present invention;

[0049] Figure 2 A diagram showing the positional relationship between the camera and the bookshelf provided by the present invention;

[0050] Figure 3 The effect diagram of the camera perspective deformation correction based on the vanishing point provided by the present invention;

[0051] Figure 3 (a) is a schematic diagram of the camera perspective deformation before correction provided by the present invention;

[0052] Figure 3 (b) is a schematic diagram of the camera perspective deformation correction process provided by the present invention;

[0053] Figure 3 (c) is a schematic diagram of the camera perspective deformation after correction provided by the present invention; Figure 4 A flowchart of building and training a text detection model provided by the present invention;

[0054] Figure 5 This is the effect diagram of text detection on pictures provided by the present invention;

[0055] Figure 6 The flow chart of the present invention for performing book instance segmentation on an image;

[0056] Figure 7 A diagram showing the relative positions of the spine frame provided by the present invention;

[0057] Figure 8 The effect diagram of the book instance segmentation of the picture provided by the present invention;

[0058] Figure 8 (a) is a line segment diagram of a picture provided by the present invention;

[0059] Figure 8 (b) is a normal map of the image provided by the present invention;

[0060] Figure 8 (c) is a schematic diagram of a data frame of a picture provided by the present invention;

[0061] Figure 8 (d) is a schematic diagram of books to be screened with pictures provided by the present invention;

[0062] Figure 8 (e) is a schematic diagram of the final segmentation result of the image provided by the present invention;

[0063] Figure 9 A flowchart of building and training a text recognition model provided by the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the following description, specific details such as specific configurations and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures has been omitted in the embodiments.

[0065] It should be understood that references throughout this specification to "one embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present application. Therefore, the appearance of "one embodiment" or "this embodiment" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0066] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0067] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association object relationship, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0068] The term "at least one" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, at least one of A and B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0069] It should also be noted that, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include," "comprises," or any other variations thereof are intended to cover non-exclusive inclusion.

[0070] Example 1

[0071] This embodiment mainly introduces a book positioning method based on computer vision. For specific methods, please refer to Figure 1 .

[0072] A book positioning method based on computer vision, the method comprising:

[0073] Step S1: take a picture with a camera to capture the target area of ​​the book and store it as a picture;

[0074] Step S2: Build and train a text detection model to mark all text in the image with text area boxes;

[0075] Step S3: Segmenting the picture into book instances based on the text region frame, thereby obtaining book instances and location information of the book instances in the picture;

[0076] Step S4: construct and train a text recognition model to recognize the text area frame, and merge the text recognition results in the same book instance to obtain the title recognition result of each book instance;

[0077] Step S5: Match the book title recognition result with the book name input by the reader to be searched on the bookshelf, and output the positioning result of the target book in the picture.

[0078] Furthermore, a camera is an optical image sensor, and any device that uses the photoelectric conversion function of a photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image falls within the meaning of a camera in this application.

[0079] The target area refers to the area on the bookshelf where the reader needs to search for books. This area is a rectangular area with a width approximately equal to the area between the two vertical baffles of the bookshelf and an aspect ratio equal to the camera resolution aspect ratio. If the reader searches for multiple books and these books are located in different target areas, the query process will be repeated multiple times according to different target areas.

[0080] Based on the camera's acquisition results, a picture of the target area can be obtained, which contains the spine information of the book that the reader is interested in; through the text detection model, all text in the picture can be marked with text area boxes, and these text areas include book titles, book authors, book editions, etc.; then, based on the text area boxes, the picture is segmented into book instances, each book can be separated, and the position of each book instance in the picture can be obtained, so that millimeter-level positioning of the book instance can be achieved, that is, the location of the book can be accurately determined; then a text recognition model is constructed and trained to identify the text area boxes, and the text recognition results in the same book instance are merged. These text recognition results together constitute the title of a book; finally, the book title recognition result is matched with the book name input by the reader to be queried on the bookshelf, and the positioning result of the target book in the picture is output.

[0081] In this way, the book positioning process can be completed by simply taking a picture of the target area with a camera and performing text detection, instance segmentation, text recognition and book title matching on the obtained picture; and by building and training a text detection model and a text recognition model, the accuracy of book title recognition can be significantly improved.

[0082] Optionally, before taking pictures with a camera, the method further includes: adjusting camera parameters according to the bookshelf environment in which the camera is located.

[0083] Different libraries have different shelf locations and lighting conditions. To ensure clear and reliable image capture, camera parameters need to be adjusted accordingly. First, when scanning the target area, the distance between the camera and the bookshelf is properly controlled to ensure that the entire area is captured. Then, based on factors such as the distance between the camera and the bookshelf and the lighting conditions, camera parameters such as focal length, exposure time, gain, and white balance are adjusted to ensure clear and reliable images.

[0084] Optionally, the method further includes: searching for the location of the book in the library data set according to the target book name input by the reader, and instructing the camera to reach a designated area for photo collection.

[0085] The library database here refers to the library used for book location. Since the locations of each bookshelf in a library are fixed, and the placement of books on the shelves is stored in the library database, once the desired book is known, the camera can be guided to the designated area based on the book's location information recorded in the database, instructing the camera to complete the acquisition of the target area.

[0086] Furthermore, the method of constructing and training a text detection model includes:

[0087] Step S21: perform data enhancement on both artificially synthesized and real scene text detection datasets;

[0088] Step S22: Based on the EAST model, modify the model's definition of the "lower" boundary and implement a GPU-accelerated non-maximum suppression algorithm to improve model training and reasoning efficiency;

[0089] Step S23: Based on the enhanced text detection dataset, train the EAST model: first pre-train the model on an artificially synthesized dataset, and then fine-tune the model on a real-scene dataset to obtain a robust text detection model.

[0090] Furthermore, the method of constructing and training a text recognition model includes:

[0091] Step S41: perform data enhancement on both artificially synthesized and real-scene text recognition datasets;

[0092] Step S42: construct a text recognition model based on the CRNN model;

[0093] Step S43: Based on the enhanced data set, train a text recognition model: first pre-train the model on an artificially synthesized data set, and then fine-tune the model on a real-scene data set to obtain a robust text recognition model.

[0094] Furthermore, in step S3, the method for segmenting book instances includes:

[0095] Step S31, using five parameters {d i |i∈{1,2,3,4}},θ defines the spine box;

[0096] Step S32: applying a straight line segment detection algorithm to the image, extracting continuous long line segments in the image, outputting a line segment graph, and generating a unit normal graph through the line segment graph;

[0097] Step S33: defining an energy function for the spine frame: the energy function reaches a minimum when each side of the spine frame fits the line segment graph and the normal vectors of each side of the spine frame are in the same direction as the normal graph;

[0098] Step S34: Use the text area frame as the initial spine frame, continuously expand the spine frame outward until the energy function reaches a minimum, output the spine frame at this time, and perform screening.

[0099] Furthermore, in step S3, the screening method includes:

[0100] Step S341: When a book edge corresponds to two or more spine boxes with similar shapes, non-maximum suppression is applied to the two or more spine boxes with similar shapes, thereby obtaining a more accurate book instance segmentation result;

[0101] Step S342: Eliminate spine frames with an aspect ratio less than 3:1;

[0102] Step S343: When there is overlap between spine frames, screening is performed based on spatial relationships.

[0103] Furthermore, the book instances here refer to book spine instances. Since books in libraries are usually placed with their spines facing outward, the only area of ​​the book that can be collected is the spine, and the key information of the book is also located on the spine. In other words, only the spine is needed to be focused. Therefore, the book instances here specifically refer to the spine instances, and the process of obtaining the spine frame is the book instance segmentation process.

[0104] Optionally, in order to avoid falling into a local optimal solution too early when obtaining the spine frame, resulting in failure to obtain the most accurate spine frame, it is necessary to calculate the energy function value after the adjacent edges are expanded outward.

[0105] In other words, some textures or color blocks on the spine may cause the spine frame to reach a local optimal solution before it completely covers the entire spine frame area, resulting in the obtained spine frame being not accurate enough. Further calculation is required to avoid "early retirement".

[0106] Furthermore, in step S5, the method for matching book names includes:

[0107] Step S51: Convert the title recognition result and the reader-input title into word vectors respectively. The length of the vector is equal to the number of all words, and the value is the TF-IDF weight of the corresponding word.

[0108] Step S52: Based on the vector, the cosine similarity is used to measure the matching degree between the book title input by the reader and each book title recognition result. The book title result with the highest similarity can be matched with the input book title, and the matching results with similarity lower than the set threshold are eliminated. The matching results with higher confidence and the positioning information of the input book title in the image are output. If multiple book titles are input, the process is repeated multiple times, and a match is performed for each book title input by the reader.

[0109] Furthermore, the data augmentation method involves randomly changing image features such as brightness, color, and contrast, as well as randomly rotating and cropping the images. The purpose of data augmentation on these datasets is to further expand the dataset and train a more robust text detection model. The images are also stretched / compressed to 1600 pixels, while maintaining their aspect ratio.

[0110] Furthermore, in step S1, the image acquisition process also includes using a homography transformation matrix to eliminate perspective distortion in the image, and the homography transformation matrix is ​​obtained by a vanishing point detection algorithm. Perspective distortion is mainly introduced because the camera shooting direction is not perpendicular to the spine direction during the acquisition process, which will cause the spine shape in the image to no longer be rectangular, but trapezoidal. The perspective distortion correction process mainly uses a vanishing point detection algorithm to detect two vanishing points in the dominant orthogonal direction of the image, and then calculates the homography matrix based on the two vanishing points, performs a homography transformation on the image, and completes the image correction.

[0111] Furthermore, in the step S1, the image acquisition process also includes using the internal and external parameters of the camera to perform distortion correction on the acquired image, and the internal and external parameters are obtained by calibrating the camera. In order to give the camera a larger viewing angle, so that the target area can be fully acquired even when the distance to the bookshelf is close, the camera generally needs to be equipped with a wide-angle lens, and the wide-angle lens usually has a large distortion. The distortion is mainly introduced due to the deviation of the lens manufacturing accuracy and the assembly process, which leads to the distortion of the original image. The process of distortion correction is mainly based on the internal and external parameters obtained by camera calibration, and the acquired image is subjected to a two-dimensional perspective transformation, which can correct the deformation and distortion of the image caused by lens distortion.

[0112] This application introduces computer vision into the field of book positioning, and implements a fine-grained and efficient book positioning model through text detection, book instance segmentation, text recognition and text matching; thereby achieving highly efficient and accurate book positioning, and ultimately achieving the technical effect of improving the efficiency of library book utilization and saving readers' time in looking up information.

[0113] This application designs a new book instance segmentation algorithm based on text region boxes, which can achieve millimeter-level positioning of books on the captured image, and accurately determine the location of the book; when readers need to borrow books, this method can quickly and accurately provide the location of the book, thereby instructing the robotic arm to automatically grab the book on the bookshelf and deliver it to the reader through the navigation robot, providing a basis for the realization of an automated library.

[0114] Example 2

[0115] Based on the above embodiment 1, this embodiment mainly introduces another book positioning method based on computer vision.

[0116] A book positioning method based on computer vision, the method comprising:

[0117] Step S0: constructing and training a text detection model and constructing and training a text recognition model;

[0118] Step S1: take a picture of the target area with a camera and store it as a picture;

[0119] Step S2: Mark all text in the picture with a text area frame;

[0120] Step S3: Segmenting the picture into book instances based on the text region frame, thereby obtaining book instances and location information of the book instances in the picture;

[0121] Step S4: identifying the text area frame and merging the text recognition results in the same book instance to obtain the title recognition result of each book instance;

[0122] Step S5: Match the book title recognition result with the book name input by the reader to be searched on the bookshelf, and output the positioning result of the target book in the picture.

[0123] Figure 2 This diagram shows the relationship between the camera and bookshelf provided by the present invention. Depending on the lens's viewing angle, the camera is placed parallel to the books on the shelf, approximately 8-10 cm away. If the camera is too close to the books, it may not fully capture the spines of the books on a shelf. If the distance is too far, the spines will be too small to be recognized in the image. The camera is kept still at the moment of capture to avoid artifacts in the captured image. Based on factors such as the distance between the camera and the bookshelf and the lighting conditions on the bookshelf, camera parameters such as focus, exposure time, gain, and white balance are adjusted to ensure clear and reliable images.

[0124] The camera can be calibrated using Zhang Zhengyou's checkerboard calibration method to obtain the camera's internal and external parameters.

[0125] Furthermore, to achieve fully automatic book positioning and avoid interference such as significant camera shake and large changes in the distance between the camera and the bookshelf during the acquisition process, the camera is fixed to a mobile scanning platform. This mobile scanning platform can move alongside the bookshelf, parallel to the long side of the bookshelf, and uses a jack device to support the camera's raising and lowering. This mobile scanning platform provides a relatively stable acquisition environment for the camera, allowing it to capture the target area. Furthermore, to ensure acquisition brightness, appropriate fill lights are deployed on the scanning platform. Since industrial cameras generally lack automatic focus, manual focus adjustment is required. Industrial cameras primarily consist of a lens and a camera. The scanning platform is moved to the side of the bookshelf, and the camera lens is rotated to adjust the camera's focal length until the camera captures the clearest image of the book.

[0126] Furthermore, based on the target book name entered by the reader, the location of the book is queried in the library dataset. The camera is transported to the target area for collection via a mobile platform, and the scan results are distorted and corrected using a calibration matrix, and then saved.

[0127] Furthermore, if the camera cannot be kept perpendicular to the spine during the shooting process, such as Figure 3 As shown in (a), when there is perspective deformation, the spine part is no longer rectangular, and an additional correction step is required: Figure 3 As shown in (b), the vanishing point detection algorithm is used to detect two vanishing points in the dominant orthogonal direction of the image. Then, the homography matrix is ​​calculated based on the two vanishing points, and the image is transformed homographically. The correction result is shown in Figure 3 (c) If this operation is performed, after obtaining the book instance segmentation result, the result needs to be transformed back using the homography matrix.

[0128] like Figure 4 As shown, Figure 4 The flowchart of the present invention for constructing and training a text detection model includes the following steps.

[0129] Step S001: Perform data augmentation on the text detection dataset. The training process requires a synthetic dataset and a dataset of 8,429 real-world images from ICDAR 2013, ICDAR 2015, and ICDAR 2017 MLT. To achieve better detection results, data augmentation is performed on these datasets. This includes randomly changing image features such as brightness, color, and contrast, randomly rotating and cropping the images, and scaling the images to 1600 pixels, while maintaining the aspect ratio.

[0130] Step S002: Based on the EAST model, a text detection model is constructed. The EAST model consists of two branches for text instance segmentation and pixel-level bounding box regression; the geometric shape of the text area in the detection result is determined by five parameters {d i |i∈{1,2,3,4}},θ forms a rotating box and outputs the confidence score of the region. Based on the EAST model, a text detection model is built with ResNet-50 as the backbone network, and two modifications are made:

[0131] The geometry of the original EAST text region is represented by five parameters {d i |i∈{1,2,3,4}},θ is a rotating frame representation, where {d i |i∈{1,2,3,4}} represents the distance from the center of the rotation box to the top, bottom, left, and right boundaries, respectively. The "bottom" boundary here refers to the boundary closest to the bottom of the image, not the bottom line of the text box. Therefore, for semantic convenience and to avoid the problem of discontinuous distance and angle prediction faced by EAST, the "bottom" boundary is defined as the bottom line of the text box. The effect is as follows: Figure 5As shown, the darkest of the four lines in the text box represents the newly defined "lower" boundary.

[0132] In the original EAST, the local-aware non-maximum suppression algorithm takes a long time to execute when text suggestion boxes are densely packed and close together. Therefore, we reimplemented the GPU-accelerated non-maximum suppression algorithm to reduce this time by 10 times.

[0133] Step S003: Based on the enhanced text detection dataset, train the EAST model. First, pre-train the model on the artificial synthetic dataset, and then fine-tune the model on the real scene dataset to obtain a robust text detection model. The text detection effect of this model is as follows: Figure 5 shown.

[0134] like Figure 6 FIG. 1 is a flow chart of performing book instance segmentation on an image in one embodiment. The process includes the following steps.

[0135] Step S31, using five parameters {d i |i∈{1,2,3,4}],θ defines the spine box. In the book positioning scenario, the spine box can be approximated as a rectangular box. Figure 7 As shown in the figure, the spine frame can be represented by the five parameters {d i |i∈{1,2,3,4}},θ represents. For a given point v inside the spine box (not necessarily the center of the spine box), d i represents the distance from the internal point v to the four sides of the spine box, and θ represents the angle between the spine box and the horizontal direction. The other parameters in the figure can be calculated from the above parameters:

[0136]

[0137] e i =[P i-1 ,P i ] (2)

[0138] Step S32: Apply the line segment detection algorithm to the image to extract the continuous long line segments in the image, output the line segment graph, and generate the unit normal graph through the line segment graph. Since the four sides of the spine box are all straight line segments, the book instance segmentation problem can be simplified to the straight line segment detection problem. Therefore, the straight line segment detection algorithm LSD can be applied to the original image to extract the continuous long line segments in the image and draw them on the line segment graph. The line segment graph is as follows: Figure 8 As shown in (a), the straight lines in the image are clearly extracted. This step can reduce the impact of spine text and some texture on book instance segmentation.

[0139] Since books are placed closely together, a “bundle” of edges may be detected between the books, so additional constraints on the spine edges are required in terms of direction. Figure 8 (b) shows the normal map corresponding to the line segment map, which is used to describe the unit normal vector of each line segment.

[0140] Step S33: Define the spine frame energy function. Based on the line segment graph and the normal graph, the following spine frame energy function can be defined:

[0141]

[0142] where p∈e i Indicates that p is a line segment e i A pixel in Represents e i The energy function is minimized when the edges of the spine box fit the line segment graph and the normal vectors of the edges of the spine box align with the normal graph.

[0143] Finally, the entire book instance segmentation process can be simplified to five parameters {d i Energy minimization problem of |i∈{1,2,3,4}},θ:

[0144]

[0145] Step S34: Initialize the spine frame and continuously expand it outward until the energy function reaches a minimum. The resulting spine frame is output and screened. The text area frame is used as the initial spine frame. Define v as the center of the initial frame, d1 and d3 as half the frame height, d2 and d4 as half the frame width, and θ as the angle between the frame and the horizontal at that point.

[0146] Then the four sides of the spine frame are moved outward in turn, that is, the cycle expands {d i |i∈{1,2,3,4}}. Each edge has two states: active and inactive. In each iteration, the active edge will be moved outward by one pixel, and the inactive edge will remain unchanged. Each iteration may change the edge state. Initially, all edges are in the active state. When the energy E(e i ) is a local minimum and satisfies E(e i )<τ, the edge will change from an activated state to an inactivated state; on the contrary, when the edge energy satisfies E(e i )>τ, the inactive edge will become active again. Repeat the above steps until all edges are inactive, and output the data frame at this time. The process is as follows Figure 8 (c) shown.

[0147] In each iteration, the angle θ between the spine frame and the horizontal direction is updated, and the θ that makes the energy function reach the lowest is selected. θ is selected from 11 values ​​in the range [θ′-5°,θ′+5°], that is:

[0148]

[0149] Where θ′ is the initial angle of each iteration.

[0150] In the process of obtaining the spine frame, due to the presence of rectangular blocks or textures on the spine frame, in order to avoid the process falling into a local optimal solution early, resulting in the inability to obtain the most accurate spine frame, such as Figure 8 As shown in (d), we need to make the edge e i The adjacent edge e i-1 With e i+1 Expand outward by δ, denoted as Δe i-1 and Δe i+1 , where Δe i-1 and Δe i+1 is defined as follows:

[0151]

[0152]

[0153] If Δe i-1 and Δe i+1 The energy of is less than τ, then e i Still in the active state, it will continue to expand during the iteration. δ is set to the width of the image.

[0154] Because the spine box starts with the text box, and a single spine area can contain multiple texts, such as the title, author, and publisher, multiple titles may be recognized as multiple texts. This means that the number of spine boxes initialized by this method far exceeds the actual number of spine boxes, and the resulting number of spine boxes is also necessarily far greater than the actual number of spine boxes, potentially including some incorrectly recognized spine boxes, necessitating further screening.

[0155] For the case where the edge of a book corresponds to two or more spine boxes of similar shapes, non-maximum suppression is applied to the spine boxes of similar shapes to obtain more accurate instance segmentation results; spine boxes usually have a large aspect ratio, so spine boxes with an aspect ratio of less than 3:1 are discarded; finally, for those cases where spine boxes overlap each other, such as a spine box containing multiple spine boxes, they are filtered according to the spatial relationship. The final book instance segmentation result is as follows: Figure 8 (e) shown.

[0156] like Figure 9 FIG. 1 is a flowchart of constructing and training a text recognition model in one embodiment. The process includes the following steps.

[0157] Step S011: Data augmentation is performed on both the synthetic and real-world detection datasets. The datasets required for training consist of the synthetic dataset and the ICDAR 2017MLT real-world dataset. To achieve better detection results, data augmentation is performed on both datasets. This includes randomly changing image features such as brightness, color, and contrast, randomly cropping the images, and scaling them to 32 pixels by 120 pixels.

[0158] Step S012: Build a text recognition model based on the CRNN model. This text recognition model is based on the CRNN model. To avoid character segmentation, the CRNN prediction results select words with the lowest CTC loss in the dictionary and use the dictionary for error correction. To speed up the calculation, the candidate set search range can be limited, and a BK tree can be used for efficient search.

[0159] Step S013: Training a CRNN model based on the enhanced text detection dataset. The model is first pre-trained on an artificially synthesized dataset, and then fine-tuned on a real-world dataset to obtain a robust text recognition model.

[0160] Based on the book instance segmentation results, the text recognition results in the same book instance can be merged to obtain the title recognition results of each book instance. At this time, these title recognition results need to be matched with the book title input by the reader. The title recognition results and the reader-input book title are converted into word vectors respectively, where the word vector converted from the title recognition result is {t k |k∈{1,2,…,N}] and the word vector q converted from the reader's input book title are defined as follows:

[0161] t k =[v 1,k ,v 2,k ,…,v W,k ] (8)

[0162] q=[v 1,q ,v 2,q ,…,v W,q ] (9)

[0163] Vector t k The length of q is equal to the number of all words W, and the value is ν x,k with ν x,qis the TF-IDF weight of the corresponding word. Based on the vectors (8) and (9), the cosine similarity is used to measure the matching degree between the reader's input book title and each book title recognition result. The calculation formula is as follows:

[0164]

[0165] By calculating similarity, the system matches the input book title with the most similar results, discards matches with similarities below a set threshold, and outputs the most confident matches and the location information in the image corresponding to the input book title. If there are multiple input book titles, this process is repeated multiple times, performing a match for each reader's input book title.

[0166] Finally, the matching results and the location information of the book instance are summarized and the location result is output. Preferably, the location result can also be notified to the robotic arm through data interaction, allowing the robotic arm to automatically grasp the book and let the navigation robot deliver the book to the reader.

[0167] This method completes the process of building and training a text detection model and building and training a text recognition model in advance. That is, the text detection model can be built and trained in advance, then stored on a server, and then directly retrieved when all text in the image is subsequently marked with text area boxes. Similarly, building and training a text recognition model can also be completed in advance before the camera takes the photo. That is, the text recognition model can be built and trained in advance, then stored on a server, and then directly retrieved when the text area boxes are subsequently recognized.

[0168] Example 3

[0169] Based on Example 1, this example further discusses the problems encountered when improving this application and the improvement methods:

[0170] In the absence of a ready-made dataset for instance segmentation of densely packed books in a bookshelf scenario, using instance segmentation datasets from other scenarios to train instance segmentation models is ineffective. The solution of this application is to creatively propose a traditional algorithm for book instance segmentation to avoid the problem of low segmentation accuracy due to the lack of a corresponding dataset. This book instance segmentation algorithm only requires a neural network based on text detection. The text detection dataset is rich and the detection algorithm is relatively mature, making the overall algorithm easy to implement.

[0171] Adjacent books on a bookshelf are often placed very close to each other, have similar textures, and are placed in various orientations, which poses a challenge to book instance segmentation. The solution of this application is as follows: to address the problem of closely placed books, the unit normal vector of the line segment is added to the spine box energy function to obtain a more accurate spine box representation; for rectangular blocks of color or textures on the spine, a mechanism is designed to avoid "early exit" in order to prevent the process of expanding the spine box from falling into a local optimal solution early; the text area on the spine is usually parallel to the spine box. Based on this parallel relationship, the text area box is selected as the initial spine box to make the spine box recognition direction as accurate as possible.

[0172] In order to meet the frequent book retrieval requests from the large number of readers in the library, the real-time performance of the book positioning technology needs to be highly demanded. The solution of this application is:

[0173] 1. Modules can be highly parallelized. In the entire embodiment 1, four steps have a relatively high time overhead: text detection, straight line segment detection and unit normal map generation, text recognition, and outward expansion of the spine frame. Among them, text detection can be performed in parallel with the straight line segment detection and unit normal map generation steps, and text recognition can be performed in parallel with the outward expansion of the spine frame step. This parallelization almost halves the overall time overhead.

[0174] 2. In the old version, the process of initializing the spine frame is as follows: the entire image is evenly divided into multiple small areas of 80*60 pixels. If the gradient value in the area is greater than 50, the center of the area, {d i = 1|i∈{1,2,3,4}} to construct the initial spine frame. Since the initial spine frame in this scheme is almost a single point, it can be considered a "zero spine frame." The process of expanding from a "zero spine frame" to align with the spine edge is obviously time-consuming. Therefore, using the text area frame as the initial spine frame, rather than starting from a "zero spine frame," can significantly reduce time overhead.

[0175] Example 4

[0176] Based on Examples 1-3, this example mainly introduces the effect verification of this method.

[0177] (1) In order to verify the accuracy of the book instance segmentation algorithm, it is also necessary to compare it with the deep learning algorithm.

[0178] 100 book images were manually annotated, 80 of which were used as training sets and 20 as test sets. The Mask R-CNN model was trained by adding multiple data enhancements. 50 , AP 75The evaluation is performed, with higher values ​​indicating better segmentation accuracy. The evaluation results and performance comparison in Table 1 show that Mask R-CNN performs poorly in dense scenes, especially for tilted books. Our algorithm outperforms deep learning algorithms and achieves finer edges.

[0179]

[0180]

[0181] Table 1 Comparison of book instance segmentation algorithms

[0182] (2) In order to verify the overall performance of book positioning, for each reader's query request, the invention will provide the possible location of the corresponding book on the bookshelf. If there is a location among the k most likely locations provided by the invention with an IoU value greater than 0.5 with the real location, the positioning result is considered accurate. This solution uses four indicators LA top-1 LA top-3 、MAE L 、MAE S The whole scheme is evaluated, where LA1 and LA3 (LA: localization accuracy) represent the positioning accuracy when k=1 and k=3 respectively, and MAE L 、MAE S The mean absolute error (MAE) describes the segmentation error of the long and short edges of the spine box. Furthermore, the present invention is compared with the Mask R-CNN + OCR solution, where the OCR (text detection + text recognition) steps are consistent with those of the present invention. The evaluation results are shown in Table 2:

[0183]

[0184] Table 2 Overall performance comparison of book positioning

[0185] The results show that this discovery achieves higher positioning accuracy and reaches millimeter-level positioning accuracy, which has high practical value.

[0186] The foregoing description is merely a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any variation, modification, replacement, integration, or parameter change to these embodiments, which is within the spirit and principles of the present invention and which achieves the same functionality through conventional substitutions, without departing from the principles and spirit of the present invention, falls within the scope of protection of the present invention.

Claims

1. A book positioning method based on computer vision, characterized in that: The method comprises: Step S1: take a picture with a camera to capture the target area of ​​the book and store it as a picture; Step S2: Build and train a text detection model to mark all text in the image with text area boxes; Step S3: Segmenting the picture into book instances based on the text region frame, thereby obtaining book instances and location information of the book instances in the picture; Step S4: construct and train a text recognition model to recognize the text area frame, and merge the text recognition results in the same book instance to obtain the title recognition result of each book instance; Step S5: matching the book title recognition result with the book title input by the reader to be searched on the bookshelf, and outputting the location result of the target book in the picture; The method for constructing and training the text detection model in step S2 includes: Step S21: Perform data augmentation on both the artificially synthesized and real-scene text detection datasets. The data augmentation used includes randomly changing the brightness, color, and contrast features of the image, randomly rotating and cropping the image, and scaling the image width to a fixed pixel size while maintaining the aspect ratio. Step S22: Based on the EAST model, a text detection model is constructed; based on the EAST model, a text detection model is constructed with ResNet-50 as the backbone network, the definition of the "lower" boundary of the model is modified, and the definition of the "lower" boundary is defined as the bottom line of the text box, and a GPU-accelerated non-maximum suppression algorithm is implemented to improve the efficiency of model training and reasoning; Step S23: Based on the enhanced text detection dataset, train the EAST model: first pre-train the model on the artificial synthetic dataset, and then fine-tune the model on the real scene dataset to obtain a robust text detection model; In step S3, the method for segmenting book instances includes: Step S31, using five parameters {d i |i∈{1,2,3,4}},θ defines the spine box. For a given point v inside the spine box, d i represents the distance from the internal point v to the four sides of the spine box, and θ represents the angle between the spine box and the horizontal direction; Step S32: Apply a straight line segment detection algorithm to the image, apply a straight line segment detection algorithm LSD to the original image, extract long continuous line segments in the image, output a line segment graph, and generate a unit normal graph through the line segment graph; Step S33: Define the spine frame energy function. Based on the line segment graph and the normal graph, define the spine frame energy function as follows: Among them, p∈e i Indicates that p is a line segment e i A pixel in Represents e i The unit normal vector of ; L(p) is the value of p in the line segment graph, and the value range is [0,255]; N(p) is the value of p in the unit normal graph; This energy function can make each side of the spine box fit the line segment graph, and the normal vectors of each side of the spine box are consistent with the direction of the normal graph, reaching the minimum; Step S34: Use the text area frame as the initial spine frame, continuously expand the spine frame outward until the energy function reaches a minimum, output the spine frame at this time, and perform screening.

2. The computer vision-based book positioning method according to claim 1, characterized in that: In step S34, the screening method includes: Step S341: When a book edge corresponds to two or more spine boxes, non-maximum suppression is applied to the two or more spine boxes to obtain a more accurate book instance segmentation result; Step S342: Eliminate spine frames with an aspect ratio less than 3:1; Step S343: When there is overlap between spine frames, screening is performed based on spatial relationships.

3. The computer vision-based book positioning method according to claim 1, characterized in that: The method for constructing and training the text recognition model in step S4 includes: Step S41: perform data enhancement on both artificially synthesized and real-scene text recognition datasets; Step S42: construct a text recognition model based on the CRNN model; Step S43: Based on the enhanced dataset, train the text recognition model: first pre-train the model on the artificial synthetic dataset, and then fine-tune the model on the real scene dataset to obtain a robust text recognition model.

4. The computer vision-based book positioning method according to claim 1, characterized in that: In step S5, the method for matching book names includes: Step S51: Convert the title recognition result and the reader-input title into word vectors respectively. The length of the vector is equal to the number of all words, and the value is the TF-IDF weight of the corresponding word. Step S52: Based on the vector, the cosine similarity is used to measure the matching degree between the book title input by the reader and each book title recognition result. The book title result with the highest similarity can be matched with the input book title, and the matching results with similarity lower than the set threshold are eliminated. The matching results with higher confidence and the positioning information of the input book title in the picture are output. If multiple book titles are input, the above process is repeated multiple times, and a match is performed for each book title input by the reader.

5. The computer vision-based book positioning method according to claim 1, characterized in that: In the step S1, the image acquisition process also includes using a homography transformation matrix to eliminate perspective deformation in the image, and the homography transformation matrix is ​​obtained by a vanishing point detection algorithm.

6. The computer vision-based book positioning method according to claim 1, characterized in that: In the step S1, the process of collecting pictures also includes using the internal and external parameters of the camera to perform distortion correction on the collected pictures, and the internal and external parameters are obtained by calibrating the camera.

7. The computer vision-based book positioning method according to claim 1, characterized in that: Before step S1, the following steps are also included: S0, constructing and training a text detection model and constructing and training a text recognition model; The step S2 is replaced by: marking all texts in the picture with a text area frame; The step S4 is replaced by: identifying the text area frame, and merging the text recognition results in the same book instance to obtain the title recognition result of each book instance.

Images