A book inventory method based on computer vision

Abstract

The present invention introduces a computer vision-based book inventory method, which includes: step S1, obtaining a sequence of images of a single-layer bookshelf through a camera; step S2, performing optical flow estimation on the image sequence to obtain the moving direction and distance of each pixel between two adjacent frames; step S3, constructing and training a rotation instance target detection model, performing instance target detection on the image sequence, and performing instance segmentation to obtain the position of the spine and call number in the current image, and assigning them to the corresponding books; step S4, training a text recognition model to obtain the book title and call number text contained in each image and the position of the text; step S5, performing target tracking to obtain the specific position of each book and all recognition results; step S6, correcting and matching all recognition results to obtain the real book title result, and outputting the book instance position and the matched book information as the inventory result. The above steps ultimately achieve high efficiency and high accuracy in book inventory.

Description

Technical Field

[0001] The invention belongs to the field of computer vision design, and in particular relates to a book inventory method based on computer vision. Background Art

[0002] The book turnover rate in libraries is very high, and new books need to be added frequently, books need to be reversed, and books returned by readers need to be sorted. Errors made by readers when returning books and staff when re-shelving books can lead to books being mis-shelved, which in turn affects the staff's efficiency of re-shelving books and makes it difficult for readers to find the books they want. At present, a large number of libraries still manage their books by scanning book barcodes, which is a time-consuming and labor-intensive process: manual scanning is required, and only one book can be scanned at a time; considering the beauty of the book and the security of the barcode, the barcode is usually affixed to the inside of the book, and the book needs to be opened when scanning. To solve this problem, technicians have proposed the idea of ​​automated libraries. A few libraries have adopted smart bookshelves, which require a complete transformation of the library, which is costly and cumbersome. Therefore, some libraries have adopted a more intelligent inventory method, replacing barcodes that require line-of-sight scanning with RFID chips, and automatically counting books through RFID readers and antennas deployed on mobile platforms. This scanning process does not require manual intervention, greatly improving inventory efficiency.

[0003] Computer vision is a technology that replaces the human eye in target recognition and understanding. Machine vision products such as cameras convert captured images into image signals. The visual system performs various operations on these signals to extract the characteristics of the target and interpret the content of the shot. Compared with humans, computer vision has low cost, fast recognition speed, stable system, and easy integration. Especially in industrial scenarios, which require a lot of repetitive and meticulous labor, computer vision systems can obtain more stable, reliable, and continuous results. It is precisely because computer vision systems can automatically and quickly obtain a large amount of information that they are widely used in industrial manufacturing, quality inspection, medical monitoring, traffic monitoring, identity authentication and other fields. The domestic demand for computer vision systems is increasing.

[0004] Chinese patent CN111814935A discloses a book positioning method based on an inventory robot, which includes the following steps: S1, data acquisition: the robot uses RFID recognition technology to obtain the electronic tag of the book, and transmits it together with the robot coordinates and antenna height to the processor module; S2, data processing conversion: extract the book code from the electronic tag, and store the electronic tag, book code, robot coordinates, and antenna height in the data storage module, read the data through the processor, convert it into the form of an array matrix, and perform average processing on each data; S3, book positioning: input the data processed by S2 into the book positioning module, find the location tag through the network model, and transmit the location tag and book information to the data reading module to realize book positioning. It uses the network model for feature extraction to locate the location of the book more accurately and quickly, greatly reducing the workload of the library administrator and improving work efficiency. However, it obviously has huge problems in data processing and camera visual positioning processing.

[0005] Chinese patent CN112464682A discloses a book positioning device and method for an intelligent bookshelf based on RFID technology, the device includes: a tag; a routing device, the routing device is set corresponding to each layer of the intelligent bookshelf; multiple antennas, each layer of the intelligent bookshelf has multiple antennas evenly distributed; a reader; a controller, which is used to send a control instruction to control the routing device to select the corresponding antenna to work, and obtain the tag information read by the reader, the antenna code corresponding to the tag information, and the RSSI value corresponding to the tag information, wherein if the tag information corresponds to multiple antenna codes, the antenna code with a larger RSSI value is saved. The device sets multiple antennas on each shelf, which can ensure that each tag is identified. When the same tag is read by multiple antennas, the RSSI value is selected and saved, which can eliminate the problem of positioning the wrong shelf, thereby solving the contradiction between reading all tags and accurate positioning, and achieving the purpose of accurately locating books.

[0006] The rapid development of computer vision technology has promoted the intelligent upgrading and transformation of various industries, and has also brought new methods and opportunities for library inventory. The above-mentioned RFID library inventory has been fully automated, but in libraries that have never deployed it, it is necessary to manually insert chips into books first. Computer vision technology is a technology that can be used out of the box and does not require any prior preparation. In addition, RFID high-frequency inventory still has a certain probability of missing reading even at a very close inventory distance. On the basis of high-frequency inventory, adding computer vision technology can greatly make up for the missed books because the camera will not miss any book. However, in actual application, the above technology faces many challenges: in order to meet the needs of robotic arm grasping, the book positioning accuracy must be high enough; and the books in the library are densely arranged, the textures are highly similar, and there are many books in the photos, which increases the obstacles to book recognition and makes positioning technology difficult to implement. Summary of the invention

[0007] To solve the above problems, without the need to modify the environment, computer vision technology is used to give accurate recognition results in a shorter time, achieving high efficiency and high accuracy in library inventory, making it easier for library administrators to locate misplaced books, and allowing readers to find the books they need faster and more conveniently.

[0008] In order to achieve the above effect, the present invention designs a book inventory method based on computer vision.

[0009] A book inventory method based on computer vision comprises the following steps:

[0010] Step S1, recording a video of a single-layer bookshelf by a camera, and splitting the video frame by frame to obtain a picture sequence;

[0011] Step S2: perform optical flow estimation on the image sequence to obtain optical flow data, and further obtain the moving direction and distance of each pixel between two adjacent frames;

[0012] Step S3, constructing and training a rotation instance object detection model, performing instance object detection on the image sequence, and further performing instance segmentation, obtaining the positions of the book spine and call number in the current image, and assigning the call number to the corresponding book according to the coordinates;

[0013] Step S4, training a text recognition model, performing text recognition on the images of the book spine and call number area identified in the image sequence, and obtaining the book title and call number text contained in each image and the location of the text;

[0014] Step S5: Based on the position coordinates of the books and call numbers identified in the picture sequence and the corresponding text, the corresponding relationship between the book instances contained in the two pictures is obtained according to the optical flow data, and the target is tracked to track the appearance of the same book or call number instance in all pictures, so as to obtain the specific position of each book on a single-layer bookshelf and all the recognition results of the same book in different pictures;

[0015] Step S6: For all recognized texts of each book or call number instance, correction and matching are performed based on the candidate set of the book database to obtain the real book title result, and the book instance location and matching book information are output as the inventory result.

[0016] Preferably, in step S2, the method for estimating the optical flow of the picture sequence includes:

[0017] Step S21, construct an optical flow estimation model based on PWC-Net. The core of the model is to use a multi-scale network to predict the optical flow, start calculating the optical flow from a low-resolution image, input low-resolution optical flow data to a higher-resolution network step by step, calculate a new high-resolution optical flow, and finally obtain the optical flow data of the original image size;

[0018] Step S22, pre-shooting a number of videos of real books on bookshelves, and training an optical flow estimation model in a self-supervised manner on the captured videos, so that the model can learn how to move the pixels of the previous picture to the pixels of the next picture;

[0019] Step S23, starting from the first picture in the picture sequence, sequentially calculate the optical flow data between two adjacent pictures; and scale the estimated optical flow data accordingly according to the ratio of the original picture size to the model input size.

[0020] Preferably, in step S3, the method of constructing and training a rotation instance object detection model includes:

[0021] Step S301, synthesizing an instance segmentation data set based on a real-life bookshelf picture;

[0022] Step S302: Use real images and synthetic data sets to train the target detection model Mask-RCNN based on the rotated candidate box to obtain a robust and generalizable model:

[0023] Step S303, in the first stage, training based on the synthetic data set to obtain a fully trained first-stage model that is applicable to multiple scenarios and has strong generalization;

[0024] Step S304, the second stage, fine-tuning training based on real pictures, so that the first-stage model can better adapt to the real scene, correct the inconsistency problem of training data distribution caused by noise in the synthetic data set, and obtain a high-precision target detection model that fits the real prediction scene.

[0025] Preferably, in step S3, the method for performing instance segmentation on the picture sequence includes:

[0026] Step S311: Send each image in the image sequence to the object detection model to obtain the bounding boxes of all the spine instances and call number instances that appear; specifically, the model outputs a rotated bounding box, which contains not only the coordinates of the original rectangle but also its rotation angle;

[0027] Step S312, according to the obtained position coordinates and tilt angle of the rectangular frame, the four corner coordinates of the rotational bounding box are calculated, and the image area enclosed by the rotational bounding box is the actual recognized book spine or call number instance;

[0028] Step S313, filtering the bounding box results of the book spine and call number instances with too small areas, and determining whether the four corner coordinates of the bounding box are within the boundary of the image. If there are corner points that are out of range, the corresponding instance recognition results are discarded;

[0029] Step S314: For the book spine instance bounding box, divide it into two groups of line pairs, namely, two left and right lines and two upper and lower lines. Only the left and right line pairs are taken, and the left and right lines are respectively extended to the upper and lower edges of the image to obtain a polygon composed of four intersection points as a new instance bounding box, which is equivalent to completing the division of the left and right sides of the book;

[0030] Step S315: the above operation does not need to be performed on the call number instance bounding box, and the original bounding box range is directly taken as the segmentation result;

[0031] Step S316: Non-maximum suppression is applied to the book spine and call number instance segmentation frames respectively, and only instances with high confidence and less overlap with other instances are retained as the final segmentation results.

[0032] Preferably, in step S4, the method of training a text recognition model and performing text recognition includes:

[0033] Step S401: Train a text recognition model based on the ICDAR 2019-LSVT Chinese text dataset. The text recognition model consists of two models: detection and recognition. The detection model is built based on the DB and trained with mixed precision. The recognition model consists of CRNN.

[0034] Step S402: using the trained text recognition model to identify the position coordinates of the text instances appearing in the image sequence and the corresponding text;

[0035] Step S403: for each image in the image sequence, according to the segmented book spine and call number positions and the identified text box positions, the text corresponding to the text box is assigned to each book spine and call number instance, and the text corresponding to each book spine and call number box is the splicing result of the assigned text; thus, the position box of each book spine and call number and the corresponding identified text, i.e., <image, position coordinate, identified text> pair, are obtained; the book title recognition result includes information such as the book title, author, publisher, etc., and the call number recognition result includes information such as the English letters of the call number and the library name, wherein the Chinese characters in the call number need to be removed;

[0036] Step S404, according to the spine and call number coordinates obtained by instance segmentation, calculate which spine instance the call number belongs to, assign the call number identification text to the spine instance, and finally obtain the result pair of <image, book location, book title and call number identification text>.

[0037] Preferably, in step S5, the target tracking process includes: calculating based on the optical flow data, and moving the centers of all book spines and call number instances on the previous image to the area of ​​the book spine or call number instance in the next image through the optical flow, so as to match the repeated book instances that have appeared in the previous and next images, and aggregating the <image, book position, book title and call number recognition text> pairs belonging to the same book, and obtaining all recognition results of the same book in different frames, which are in the form of <instance coordinates, book title and call number text recognition sequence>; at this time, the coordinates of each book instance are global coordinates in a layer of bookshelf, and the start or end position of the bookshelf is the origin.

[0038] Preferably, in step S6, the method for correcting and matching based on the candidate set of the book database includes:

[0039] For the <instance coordinates, book title and call number text recognition sequence> obtained by target tracking, select the recognition result with the highest confidence from the recognition sequence: first perform an exact match of the call number in the library database (a match is successful only when all characters are the same). If any match in the call number recognition sequence is successful, the matched book information is used as the inventory result; otherwise, perform book title matching, perform word segmentation on any recognition result in the book title recognition sequence, select candidates with the same words after word segmentation in the local candidate set of the current layer of books, and then use TF-IDF to calculate its similarity with the current recognized book title, and take the book title with the highest similarity that exceeds the set similarity threshold. If the number of occurrences is the same, then select the correction result with the highest score.

[0040] Preferably, in step S301, the synthetic instance segmentation dataset includes real book images and synthetic book images; and the synthesis method thereof includes:

[0041] Step S3011: deploy a camera in a real bookshelf scene, and move the camera at a constant speed to take pictures of books on different layers of the bookshelf, thereby obtaining several pictures of the real bookshelf;

[0042] Step S3012, based on the real bookshelf pictures taken, synthesize and enhance the instance segmentation data set: cut out the spine of each book from the real picture, and perform data enhancement such as random rotation, random deformation, random change of illumination, random addition of white noise, random addition of light spots, etc. to obtain several new book spine instances;

[0043] Step S3013: construct a background image library based on real empty bookshelf background images and background images randomly selected from the ImageNet dataset, randomly extract a certain number of backgrounds from them, randomly paste the above-synthesized new spine instances to random positions in these backgrounds, and obtain the four corner position coordinates of the new spine instances in the new backgrounds, and automatically add them to the synthesized annotation file.

[0044] Preferably, in step S6, before correcting and matching the recognition results of the book title text and the call number text, it also includes: querying the information of the books on the shelf at the corresponding shelf level from the library database according to the shelf number and layer number scanned by the current camera as a set of candidate books for matching.

[0045] Preferably, in step S1, the video frame-by-frame splitting method includes: reading the video frame by frame, performing distortion correction, image rotation and other operations on each frame of the picture, so as to obtain a distortion-free front view of the bookshelf.

[0046] The advantages and effects of this application are as follows:

[0047] 1. This application uses a rotating instance target detection module to solve the problems of a large number of missed detections and low accuracy of traditional instance segmentation in scenarios with dense books and tilted books; at the same time, it uses a target tracking algorithm to ensure that the same book has multiple opportunities to be detected by instance segmentation and multiple text recognitions, and allows comprehensive consideration of recognition results in different frames, so as to find the one with the highest confidence as the final result, which greatly solves the problems of missed book inventory and inaccurate inventory results.

[0048] 2. From the perspective of user experience, this application uses instance segmentation and target tracking technology to obtain the global position coordinates of each book on the corresponding shelf layer, which is convenient for library administrators to locate misplaced books and allows readers to find the books they need faster and more conveniently.

[0049] 3. This application uses computer vision related technologies to conduct library inventory, which does not require any modification to the environment and can be deployed directly.

[0050] 4. The computer vision recognition module implemented in this application based on deep learning technology has high robustness and reliability, and can give accurate recognition results in a short time.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings as follows.

[0052] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0054] Figure 1 A flowchart of the computer vision-based book inventory method provided for this application;

[0055] Figure 2 A diagram showing the relationship between the deployment positions of the camera and the bookshelf provided for this application;

[0056] Figure 3 The structure diagram of the optical flow estimation model provided for this application;

[0057] Figure 4 The result graph of optical flow estimation for the image sequence provided by this application;

[0058] Figure 5 Flowchart for constructing synthetic datasets provided for this application;

[0059] Figure 6 A structural diagram of the rotation instance target detection model provided in this application;

[0060] Figure 7 This is the effect diagram of instance segmentation of the image sequence provided by this application;

[0061] Figure 8 The effect diagram of text recognition on the image sequence provided by this application;

[0062] Fig. 9 This is a diagram showing the effect of target tracking on a sequence of images provided in this application. DETAILED DESCRIPTION

[0063] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, rather than all of the embodiments. In the following description, specific details such as specific configuration and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures is omitted in the embodiment.

[0064] It should be understood that the references to "one embodiment" or "this embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "one embodiment" or "this embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0065] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0066] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association object relationship, indicating that there can be two relationships. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0067] The term "at least one" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, at least one of A and B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0068] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusions.

[0069] Example 1

[0070] This embodiment mainly introduces the overall design of a book inventory method based on computer vision.

[0071] A book inventory method based on computer vision, the method comprising:

[0072] Record video with camera and obtain video of single-layer bookshelf;

[0073] The video is split frame by frame to obtain a picture sequence, an optical flow estimation model is trained in a self-supervised manner, optical flow estimation is performed on each of the pictures, and the moving direction and distance of each pixel between two adjacent frames are further obtained;

[0074] Implementing an object detection model based on a rotating candidate box to improve instance segmentation accuracy in scenarios where books are densely arranged: constructing a synthetic dataset and training an object detection model, performing instance object detection on the image sequence, and further performing instance segmentation to obtain the position coordinates of each book and call number instance that appears, and assigning the call number to the corresponding book based on the coordinates;

[0075] Training a text recognition model to perform text recognition on the images of book spine instances and call number regions identified in the image sequence, and obtaining the book title and call number text contained in each image and the location of the text;

[0076] Based on the position coordinates of the books and call numbers identified in the image sequence and the corresponding text, the corresponding relationship between the book instances contained in each pair of images is obtained according to the optical flow data, and target tracking is performed to track the occurrence of the same book or call number instance in all images, thereby obtaining the specific position of each book on a single-layer bookshelf and all the recognition results of the same book in different images;

[0077] For all recognized texts of each book or call number instance, correction and matching are performed based on the candidate set of the book database to obtain the real book title result, and the book instance location and matching book information are output as the inventory result.

[0078] Among them, the camera is an optical image sensor, and any device that uses the photoelectric conversion function of a photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is in a corresponding proportional relationship with the light image falls within the meaning of the camera in this application.

[0079] The video of a single-layer bookshelf can be a video of books on one layer of a bookshelf, or a video of books on the same layer of different bookshelves at the same height, or a video of several grids on one layer of a bookshelf, which should be shot by a camera with a video shooting function. The picture sequence refers to the video frames extracted from the video at a certain interval by a computer program. Since it is necessary to calculate the optical flow data between adjacent pictures, the contents of adjacent pictures are required to overlap to a certain extent, otherwise the optical flow calculation will fail or be wrong. Optical flow is a parameter that describes the movement direction and movement distance of pixels in adjacent pictures, that is, to which pixel point in the next picture the pixel point in the previous picture moves.

[0080] Among them, the process of building and training the optical flow estimation model, the rotation instance target detection model and the text recognition model can be completed in advance before the camera is deployed to record the video. That is, the three models can be trained with pre-shot videos or pictures, and then the trained model weights are saved on the server. When deploying and running the visual inventory program, you only need to directly load the corresponding weights.

[0081] Among them, the environmental factors such as bookshelves, books and lighting in different libraries vary greatly. Before deploying to a new library for operation, the existing model weights need to be further fine-tuned. The specific steps are: shoot a small amount of videos or pictures in the new library, and for the old model that needs fine-tuning, use a lower learning rate to train new data until the verification effect reaches a high level. Stop fine-tuning, replace the old weight file on the server with the new model weight file after fine-tuning, and restart the inventory program.

[0082] Each book instance will appear in multiple frames in the video. When tracking and locating each unique book spine instance in the video, it is necessary to know in which frames each instance appears and the correspondence between the instances in these frames, which can be described by the optical flow data between frames; build and train an optical flow estimation model based on PWC-Net, calculate the optical flow parameters of the image sequence, and obtain the pixel correspondence of the image sequence; build and train a Mask-RCNN model for rotating target detection, perform spine segmentation to separate multiple books arranged continuously in the image, and achieve pixel-level positioning of the books; perform call number segmentation to distinguish the area belonging to the call number on each book spine instance, and reduce the difficulty of identifying the call number text; use optical flow and pixel-level segmentation results to track all instances, and calculate the previous The centers of all book spines or call number instances on an image are moved to the area of ​​a book spine or call number instance in the next image through optical flow, so as to match the repeated book and call number instances that have appeared in the previous and next images, aggregate the <image, position coordinate> pairs belonging to the same book or the same call number, and calculate the global position of the instance in the current layer as the result of target tracking; build and train a text recognition model based on DB and CRNN, perform text recognition and aggregation on all the occurrence positions of each instance after target tracking, and obtain multiple text recognition results of the same book title or call number; query the book database of the inventory library to obtain the book information of the current bookshelf layer, correct the errors in the book title or call number text recognition results and match them as the final inventory result.

[0083] Optionally, before turning on the camera to record the bookshelf video, it also includes: installing a suitable fill light and adjusting the brightness according to the light intensity of the bookshelf; selecting a suitable working distance, i.e. the distance between the camera and the bookshelf, and adjusting various parameters of the camera according to the arrangement of the books and the focal length and viewing angle of the camera used.

[0084] Preferably, the process of recording a video of a single layer of books on a bookshelf by using a camera includes: fixing the camera on a movable platform, for the bookshelf layer to be photographed, fixing the working distance and the moving speed according to the ambient light and the arrangement of the books, focusing the camera on the photographed books, and adjusting the camera parameters such as exposure time, gain, white balance, etc., to ensure that the camera can capture very clear and reliable results; setting the camera to video mode, and moving the platform at a constant speed to scan until the end of the bookshelf, to obtain a video of the books on one layer of the bookshelf.

[0085] Preferably, the video is segmented into a sequence of pictures frame by frame, and the picture sequences mentioned later in this application all refer to a series of pictures obtained by segmenting the video. In order to accurately calculate the optical flow between video frames with minimal computational overhead, it is necessary to adjust the moving speed so that the interval between the picture sequences cannot be too small, ensuring a low computational overhead, and there must be a certain overlap between adjacent pictures to ensure the accuracy of the obtained optical flow data.

[0086] Optionally, the camera parameters are calibrated, and the image sequence is corrected using the calibrated parameters to eliminate barrel distortion of the image.

[0087] Optionally, before correcting and matching the recognition results of the book title and call number text, the method further includes: querying the information of the books on the shelf at the corresponding shelf level from the library database according to the shelf number and level number currently scanned by the camera, as a set of candidate books for matching.

[0088] Optionally, the process of estimating the optical flow of the image sequence includes:

[0089] Build an optical flow estimation model based on PWC-Net. The core of the model is to use a multi-scale network to predict the optical flow, start calculating the optical flow from a low-resolution image, gradually input low-resolution optical flow data into a higher-resolution network, calculate a new high-resolution optical flow, and finally obtain the optical flow data of the original image size.

[0090] Pre-shoot several videos of real books on bookshelves, and use self-supervision to train the optical flow estimation model on the videos, so that the model can learn how to move the pixels of the previous picture to the pixels of the next picture.

[0091] Starting from the first image in the image sequence, the optical flow data between two adjacent images is calculated sequentially; the estimated optical flow data is scaled accordingly according to the ratio of the original image size to the model input size.

[0092] Optionally, the process of building and training a rotated instance object detection model includes:

[0093] Synthesize instance segmentation dataset based on real bookshelf pictures;

[0094] The Mask-RCNN object detection model based on the Rotated Region Proposal is trained using real images and synthetic datasets to obtain a robust and generalizable model:

[0095] In the first stage, we train on synthetic datasets to obtain a fully trained one-stage model that is applicable to multiple scenarios and has strong generalization capabilities.

[0096] In the second stage, fine-tuning training is performed based on real images to enable the first-stage model to better adapt to real scenes, correct the inconsistent distribution of training data caused by noise in the synthetic data set, and obtain a high-precision target detection model that fits the real prediction scene.

[0097] Optionally, the process of synthesizing a segmented dataset includes:

[0098] Based on the real-life pictures of books on the bookshelf, the corresponding books are cut out from the pictures according to the marked positions of the book spine and call number using perspective transformation. Since the call number is in the spine area, after cutting out the book instance, the coordinates of the cut call number can be obtained by calculating the relative positions of the two.

[0099] Take a picture of a real bookshelf without books as a background, and randomly extract several natural background pictures from the ImageNet dataset. Combine the two background pictures to form a candidate synthetic background picture library.

[0100] Randomly extract a picture from the background picture library, randomly expand its size, and randomly increase or decrease its contrast to simulate various scenes in the real environment;

[0101] Paste several cut books into the background image to synthesize a new image. When pasting books, randomly generate intervals between the previous book or image boundary, and randomly stretch, rotate, change the lighting, add random light spots and noise, adjust the contrast and clarity, etc. Paste the transformed book instance to the calculated background image position, and use this position coordinate as the new annotation until the generated pasting position coordinate exceeds the background image boundary. According to the above process, a large number of synthetic book images and corresponding annotations can be generated;

[0102] The coordinates of the call number should also be transformed accordingly according to the pasting position. In addition, the coordinate transformation of the call number can also be separated from the spine transformation, and a separate translation, coordinate transformation, illumination transformation and other operations can be performed, as long as the transformed coordinates are still within the spine enclosed range.

[0103] Optionally, the process of performing instance segmentation on the picture sequence includes:

[0104] Each image in the image sequence is fed into the object detection model to obtain the bounding box of all the spine instances and call number instances. Specifically, the model outputs a rotated bounding box, which contains not only the coordinates of the original rectangle, but also its rotation angle;

[0105] According to the obtained position coordinates and tilt angle of the rectangular frame, the coordinates of the four corners of the rotating bounding box are calculated, and the image area enclosed by the rotating bounding box is the real recognized book spine or call number instance;

[0106] Filter the bounding box results of the book spine and call number instances with too small an area, and determine whether the four corner coordinates of the bounding box are within the boundary of the image. If there are corner points that are out of range, remove the corresponding instance recognition results;

[0107] For the book spine instance bounding box, divide it into two groups of line pairs: left and right lines and top and bottom lines. Only the left and right line pairs are taken, and the left and right lines are extended to the top and bottom edges of the image respectively. The polygon composed of four intersection points is obtained as the new instance bounding box, which is equivalent to completing the left and right side segmentation of the book.

[0108] For the call number instance bounding box, the above operation does not need to be performed, and the original bounding box range is directly taken as the segmentation result;

[0109] For the book spine and call number instance segmentation frames, non-maximum suppression is applied respectively, and only instances with high confidence and less overlap with other instances are retained as the final segmentation results. In other words, there may be some overlaps between the bounding boxes of tilted books and vertical books, but their overlapping areas account for a small proportion of the area of ​​the two bounding boxes, so both will be retained, ensuring that the tilted and vertical books in the image can be segmented at the same time.

[0110] Optionally, the text recognition process includes:

[0111] The text recognition model is trained based on the ICDAR 2019-LSVT Chinese text dataset. The text recognition model consists of two models: detection and recognition. The detection model is built based on DB and trained with mixed precision; the recognition model is composed of CRNN.

[0112] Use the trained text recognition model to identify the location coordinates of text instances and the corresponding text that appear in the image sequence;

[0113] For each image in the image sequence, according to the segmented spine and call number positions, and the identified text box positions, the text corresponding to the text box is assigned to each spine and call number instance. The text corresponding to each spine and call number box is the concatenation result of the assigned text. In this way, the position box of each spine and call number and the corresponding identified text are obtained, that is, the <image, position coordinates, identified text> pair. The book title recognition result includes information such as the book title, author, and publisher. The call number recognition result includes information such as the English letters of the call number and the library name. The Chinese characters in the call number need to be removed.

[0114] According to the spine and call number coordinates obtained by instance segmentation, calculate which spine instance the call number belongs to, assign the call number identification text to the spine instance, and finally obtain the result pair of <image, book location, book title and call number identification text>.

[0115] Optionally, the target tracking process includes:

[0116] Based on the optical flow data, the center of all book spines and call number instances on the previous image is calculated and moved to the area of ​​the book spine or call number instance in the next image through the optical flow, so as to match the repeated book instances that have appeared in the previous and next images, and aggregate the <image, book location, book title and call number recognition text> pairs belonging to the same book to obtain all the recognition results of the same book in different frames, which are changed to <instance coordinates, book title and call number text recognition sequence>. At this time, the coordinates of each book instance are the global coordinates in a bookshelf layer, and the start or end position of the bookshelf is the origin.

[0117] Optionally, the process of correcting the text recognition result of matching book titles and call numbers according to the inventory library book database includes:

[0118] For the <instance coordinates, book title and call number text recognition sequence> obtained by target tracking, select the recognition result with the highest confidence from the recognition sequence: first perform an exact match of the call number in the entire library database (a match is considered successful only when all characters are the same). If any match in the call number recognition sequence is successful, the matched book information is used as the inventory result; otherwise, perform book title matching, perform word segmentation on any recognition result in the book title recognition sequence, and select candidates with the same words after word segmentation in the local candidate set of the current layer of books. Then, use TF-IDF to calculate the similarity score between the current book title and the candidate, and select the candidate book title with the highest score as the correction result. Finally, select the correction result with the largest number of occurrences from the book title correction results in the queue as the inventory result. If the number of occurrences is the same, then select the correction result with the highest score.

[0119] Among them, the local candidate set of books in the current layer only belongs to the currently scanned bookshelf layer and is a subset of the entire library data set.

[0120] From the perspective of user experience, the use of instance segmentation and target tracking technology can obtain the global position coordinates of each book on the corresponding shelf layer, which is convenient for library administrators to locate misplaced books and allows readers to find the books they need faster and more conveniently. On the one hand, the use of computer vision-related technologies for library inventory does not require environmental modification and can be directly deployed; on the other hand, the computer vision recognition module based on deep learning technology is highly robust and reliable, and can give accurate recognition results in a short time. Furthermore, from a technical perspective, the rotation instance target detection module can solve the problems of a large number of missed detections and low accuracy of traditional instance segmentation in scenes with dense books and tilted books; the target tracking algorithm can ensure that the same book has multiple opportunities to be detected by instance segmentation and multiple text recognitions, and allows comprehensive consideration of recognition results in different frames, so as to find the one with the highest confidence as the final result, which greatly solves the problems of missed inventory of books and inaccurate inventory results.

[0121] Example 2

[0122] Based on the above-mentioned embodiment 1, this embodiment mainly introduces a specific method for book inventory based on computer vision.

[0123] Figure 1 The following is a flowchart of a computer vision-based book inventory method in one embodiment. The method includes the following contents:

[0124] Step 1, recording a video with a camera to obtain a video of books on a single layer of a bookshelf, and splitting the video frame by frame to obtain a sequence of images of a whole layer;

[0125] Step 2: Build and train a rotation optical flow estimation model, perform optical flow estimation on all adjacent image pairs in the image sequence, obtain the optical flow data corresponding to the image sequence, and identify the moving direction and distance of the image pixels;

[0126] Step 3: Build and train a rotation instance object detection model, locate the instance of each image in the image sequence, and obtain the location coordinates of the book and call number instance contained in each image;

[0127] Step 4: training a text recognition model to perform text recognition on the images of the book or call number instance appearing in each image, obtaining the recognized text of each instance, and assigning the call number recognized text to the corresponding book instance;

[0128] Step 5: Target tracking is performed based on the optical flow and the image coordinate information of the book and call number instances, tracking the moving direction and distance of each book between adjacent images, and finally obtaining the global coordinates of each book instance on the current shelf layer, and merging the book title and call number recognition results belonging to the same book in different images;

[0129] Step 6, correcting the book and call number identification text, and then obtaining a list of matching results of the book title and call number of the same book based on the inventory book database;

[0130] Step 7: From the matching result list of the book title and call number of the same book, the most likely matching item is calculated according to the call number exact match and the book title matching score. Finally, the book global position and the matching book information are combined as the inventory result.

[0131] Among them, the process of building and training the rotation target model in step 3 can be completed in advance before step 1; the order of step 4 text matching and step 5 target tracking can be exchanged, and the order of step 5 target tracking and step 6 correct matching can also be exchanged, as long as text recognition and correct matching can be performed on all occurrences of the same book.

[0132] like Figure 2 The figure shows the deployment position relationship diagram of the camera and the bookshelf in one implementation. The specific deployment requirements are as follows:

[0133] The camera's shooting position is fixed, and the camera is placed parallel to the mobile platform at the same height as the target scanning bookshelf. Considering that the horizontal viewing angle of general industrial cameras is larger than the vertical viewing angle, the camera is installed vertically on the mobile platform, and the distance from the bookshelf is further adjusted according to the size of the camera's designed horizontal viewing angle to ensure that the camera can capture all the book information on a complete shelf, and will not capture the books on the upper or lower shelves.

[0134] After fixing the camera, adjust the camera's white balance value. Since industrial cameras do not have an autofocus function, you need to rotate the camera lens for manual focus to make the text of the book clearly visible. In actual operation, shoot from left to right in a uniform motion parallel to the bookshelf. In the implementation scheme proposed in this application, a moving speed of 0.1m / s is used. In order to ensure that there is no smearing of text in the captured video, it is necessary to select a suitable camera exposure time. The longer the exposure time, the greater the displacement of the text smear in the picture; the lower the exposure time, although it can reduce the smear, the video brightness will be reduced, and the shooting result will be too dark. It is necessary to deploy a fill light at a suitable position on the mobile platform and adjust the brightness so that the final exposure time value is within the range of 2000-4000μs. If the brightness still does not achieve the expected effect after adding the fill light, the gain value can be appropriately increased.

[0135] Each different camera must be calibrated individually to correct distortion: print a chessboard picture, take a sufficient number of chessboard photos, and use Zhang Zhengyou's chessboard calibration method to calculate the camera's internal and external parameters based on this set of photos.

[0136] Taking the mobile scanning of books on one layer of a bookshelf as an example, a video of the books on the current layer is recorded, and the video is cut frame by frame to obtain a picture sequence in which adjacent pictures overlap each other. Distortion correction is performed on each picture in the picture sequence according to the calculated camera parameters, and the pictures are saved as the shooting results of the current layer. According to the shelf number of the currently counted bookshelf, the corresponding book information is obtained from the database of the inventory library, and together with the picture sequence, it constitutes the collected data.

[0137] like Figure 3 As shown, this is the network structure diagram of the optical flow estimation model PWC-Net. Figure 4 This is a result diagram of optical flow estimation for a picture sequence in one implementation method. The specific process is:

[0138] Build an optical flow estimation model PWC-Net, shoot videos of books on several bookshelves, and train the optical flow estimation model based on the captured videos in a self-supervised manner;

[0139] The optical flow data is obtained by estimating the optical flow of the image sequence, which describes which pixel in the previous image corresponds to which pixel in the next image after it moves. Figure 4 The optical flow estimation results of several points in the original image are shown.

[0140] like Figure 5 As shown, it is a flowchart of building and training a rotation instance target detection model in one implementation method, and the process includes the following steps.

[0141] Step 301 , select a bookshelf in a certain area of ​​a library, deploy cameras, take multiple sets of real book images in different environments, and mark the bounding box coordinates of the book spine and call number on each image.

[0142] Step 302 , take pictures of several empty bookshelves and collect several background pictures from the ImageNet dataset as a background picture library for the synthetic segmentation dataset.

[0143] Step 303, cut out book instances from the real book images, randomly select a number of instances, perform random stretching, rotation, illumination change, add random light spots and noise, adjust contrast and clarity, etc.; randomly select a background from the background image library, perform random size expansion and contrast transformation, and then paste the selected instances onto the background at random intervals.

[0144] Step 304 , generating synthetic annotation data based on the coordinate information of the book spine and the call number instance on the new background, and forming a synthetic data set together with the synthetic image.

[0145] Step 305 , training a Mask-RCNN model for rotation instance detection based on the synthetic data set to obtain a one-stage model, which is suitable for multiple scenarios and has strong generalization.

[0146] Step 306, fine-tune the training of the first-stage model based on real pictures, so that the first-stage model can better adapt to the real scene, correct the data distribution inconsistency problem caused by the noise in the synthetic data set, and finally obtain a high-precision target detection model that conforms to the real prediction scene.

[0147] like Figure 6 As shown in the figure, it is a structural diagram of the Mask-RCNN model for rotated target detection. The main difference between it and the original Mask-RCNN is that the RPN network outputs candidate bounding boxes with rotation angles. Figure 7 FIG. 1 is a diagram showing the effect of performing instance segmentation on a picture sequence in an implementation method. Figure 7 (a) is the original image to be detected. Figure 7 (b) is the rotation bounding box obtained by target detection. Figure 7 (c) is a schematic diagram of extending the left and right line segments of the spine bounding box. Figure 7 (d) is the instance segmentation result.

[0148] The Mask-RCNN model for object detection for rotated instances includes a rotated candidate box generation network and an image classification network. The rotated candidate box generation network calculates the input image and outputs a rotated bounding box that may contain a book spine or call number instance. The images in these rotated bounding boxes are then sent to the image classification network to identify whether the content is a book spine instance, a call number instance, or a background. Finally, the coordinates of the bounding box and the corresponding instance category are used as the network's object detection results. Figure 7 (a) The process of segmenting the spine and call number instances is as follows:

[0149] Will Figure 7 (a) Input the trained two-stage rotation instance object detection model to obtain the rotated bounding boxes corresponding to the initially recognized book and call number instances, and filter the bounding boxes with too small areas and remove the bounding boxes that exceed the image boundary, and obtain the following: Figure 7 (b) The rotation bounding box;

[0150] The spine of the book is segmented on both sides, and the left and right lines of the identified spine bounding box are extended to the upper and lower edges of the image to obtain four intersection points, forming a new spine segmentation frame. Since the function of the call number bounding box is to distinguish the call number area from the spine instance, there is no need to extend the line. The effect after extending the left and right line segments of the spine bounding box is as follows Figure 7 (c);

[0151] There may be overlaps between the instance segmentation frames obtained above, especially when the generalization effect of the target detection model is not ideal (such as encountering a new library with a very different environment, but the model has not been fine-tuned yet). The overlap is more serious. At this time, it is necessary to perform non-maximum suppression on the segmentation frames of the book spine and call number instances respectively, and remove the segmentation frames that overlap a lot with other results but have low confidence, so as to ensure the uniqueness and stability of the segmentation results for each instance. Non-maximum suppression will reduce the number of segmentation results to a certain extent, but will increase the accuracy of segmentation. The effect of non-maximum suppression on Figure 7(c) is shown in the figure. Figure 7 (d) It should be noted that since the overlapping area between the segmentation boxes of the tilted book and the vertical book is extremely small, the segmentation box of the tilted book will still be retained and the recall rate of instance segmentation will not change.

[0152] like Figure 8 As shown in FIG. 1 , it is a diagram showing the effect of text recognition on a picture sequence in an implementation method. Figure 8 (a) is the text recognition result of the original image. Figure 8 (b) Figure 8 (c) Figure 8 (d) shows three examples selected from the book instances segmented from the original image. Each example corresponds to a book and has the corresponding book title and call number text recognition results (the recognized call number is in brackets).

[0153] A text recognition model is built based on DB and CRNN, and after training on the ICDAR 2019-LSVT Chinese text dataset, the trained model is used to recognize the original image to obtain the following Figure 8 (a) only contains the recognized text on the image and the position of each text, and does not know which book or call number instance the text belongs to. Therefore, it is necessary to determine which book or call number instance the center of each recognized text box belongs to based on the instance segmentation box coordinates of the current image, and assign its text to the corresponding instance. Finally, for multiple text boxes assigned to the same instance, the text in them can be spliced ​​to form the recognized text of the book title or call number.

[0154] According to the spine and call number coordinates obtained by instance segmentation, calculate which spine instance the call number belongs to, assign the call number identification text to the spine instance, and finally obtain the result pair of <image, book location, book title and call number identification text>.

[0155] like Fig. 9 As shown in FIG. 1 , it is a diagram showing the effect of target tracking on a picture sequence in one implementation. Fig. 9 (a) Fig. 9 (b) are two adjacent pictures randomly selected from the picture sequence. Fig. 9 (a) is the left picture, Fig. 9 (b) is the right picture. Fig. 9 (a) and Fig. 9 (b) The process of book target tracking is as follows:

[0156] Step 501, extract from the optical flow data Fig. 9 (a) and Fig. 9 (b) The optical flow between the two images obtains the position coordinate information of each book spine instance from the instance segmentation results of the two images, and obtains the recognition text and call number corresponding to each book spine instance from the text recognition results.

[0157] Step 502, according to Fig. 9 The coordinates of the bounding box of the book spine instance in (a) are used to calculate the coordinates of the center point. Fig. 9 Position representation of each spine instance in (a).

[0158] Step 503, calculate Fig. 9 The center point of each book spine instance in (a) is moved to Fig. 9 (b) The moved center point is called the new center point. Calculate which of the new center points belongs to Fig. 9 (b) Which of the book spine instances bounding box is obtained? Fig. 9 The spine example in (a) is similar to Fig. 9 (b) The correspondence of the book spine instance.

[0159] Step 504: Fig. 9 (a) and Fig. 9 In (b), the segmentation results and text recognition results of the matching book spine are merged, that is, the <image, book location, book title and call number recognition text> pairs belonging to the same book are aggregated to obtain multiple coordinates and multiple text recognition results corresponding to the same book instance, and the absolute position of the same book in the global coordinate system composed of the two images is calculated, which is in the form of <instance coordinates, book title and call number text recognition sequence>.

[0160] As you can see, Figure 8 There are some errors in the text recognition results, and the recognized book publishers, authors, and additional book information are mixed with the book titles. Especially when the book titles contain complex letters, Chinese characters, or special fonts, the recognition errors are the most serious. Target tracking only merges these fuzzy recognized texts, so they cannot be used as the final inventory results. It is still necessary to correct and match the book titles and call numbers of the recognition results merged by target tracking.

[0161] For the title and call number text recognition sequence of the same book, firstly, the call number sequence is accurately matched directly in the whole library data set. If any recognized call number is matched successfully, the corresponding book information is used as the recognition result of the current instance; otherwise, when the call number matching fails, the title recognition sequence and the title of the current layer book candidate set are segmented respectively, and then for each recognized title in the title recognition sequence, the items with the same words after segmentation are found in the candidate set, and the similarity scores of the to-be-matched title and the candidate title are calculated using TF-IDF, and the one with the highest score is selected as the matching result of the current title, and finally the result that has been matched the most times is selected from the corrected title matching sequence and output as the inventory result. Finally, the global coordinate information of the book instance and the matched book information are aggregated, and the inventory result is presented to the user in the form of a graphical interface or report.

[0162] Therefore, in a specific implementation scheme of the present application, on the one hand, the deep learning technology in computer vision is introduced into the field of library inventory where data sets are difficult to obtain. By means of synthetic data sets, a rotation instance target detection model and a text recognition model are constructed and trained, and the optical flow estimation model can be trained by self-supervision. Good results can be achieved without a large amount of source data acquisition and manual annotation, and it has good feasibility and high accuracy. On the other hand, for missed recognition or wrong recognition caused by environmental changes or insufficient model generalization performance, the scheme can use the target tracking algorithm to identify each frame of the video, increasing the possibility of a book being recognized, and to a certain extent solving the problem of wrong recognition caused by angle problems, and improving the recognition accuracy. In addition, this scheme can accurately locate the global coordinates of the book based on the optical flow and instance segmentation results, so that librarians can quickly sort out misplaced books, and readers can also find the books they want to borrow more quickly.

[0163] The above description is only the preferred embodiment of the present invention, which does not limit the protection scope of the present invention. For those skilled in the art, the present invention can be modified and varied in various ways. Within the spirit and principle of the present invention, any change, modification, replacement, integration and parameter change of these embodiments by conventional substitution or capable of achieving the same function without departing from the principle and spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A book inventory method based on computer vision, characterized in that: It includes the following steps: Step S1, recording a video of a single-layer bookshelf by a camera, and splitting the video frame by frame to obtain a picture sequence; Step S2: perform optical flow estimation on the image sequence to obtain optical flow data, and further obtain the moving direction and distance of each pixel between two adjacent frames; Step S3, constructing and training a rotation instance object detection model, performing instance object detection on the image sequence, and further performing instance segmentation, obtaining the positions of the book spine and call number in the current image, and assigning the call number to the corresponding book according to the coordinates; Step S4, training a text recognition model, performing text recognition on the images of the book spine and call number area identified in the image sequence, and obtaining the book title and call number text contained in each image and the location of the text; Step S5: Based on the position coordinates of the books and call numbers identified in the picture sequence and the corresponding text, the corresponding relationship between the book instances contained in the two pictures is obtained according to the optical flow data, and the target is tracked to track the appearance of the same book or call number instance in all pictures, so as to obtain the specific position of each book on a single-layer bookshelf and all the recognition results of the same book in different pictures; Step S6: For all recognized texts of each book or call number instance, correction and matching are performed based on the candidate set of the book database to obtain the real book title result, and the book instance location and matching book information are output as the inventory result.

2. A computer vision-based book inventory method according to claim 1, characterized in that: In step S2, the method for estimating the optical flow of the image sequence includes: Step S21, construct an optical flow estimation model based on PWC-Net. The core of the model is to use a multi-scale network to predict the optical flow, start calculating the optical flow from a low-resolution image, input low-resolution optical flow data to a higher-resolution network step by step, calculate a new high-resolution optical flow, and finally obtain the optical flow data of the original image size; Step S22, pre-shooting a number of videos of real books on bookshelves, and training an optical flow estimation model in a self-supervised manner on the captured videos, so that the model can learn how to move the pixels of the previous picture to the pixels of the next picture; Step S23, starting from the first picture in the picture sequence, sequentially calculate the optical flow data between two adjacent pictures; and scale the estimated optical flow data accordingly according to the ratio of the original picture size to the model input size.

3. A computer vision-based book inventory method according to claim 1, characterized in that: In step S3, the method for constructing and training a rotation instance target detection model includes: Step S301, synthesizing an instance segmentation data set based on a real-life bookshelf picture; Step S302: Use real images and synthetic data sets to train the target detection model Mask-RCNN based on the rotated candidate box to obtain a robust and generalizable model: Step S303, in the first stage, training based on the synthetic data set to obtain a fully trained first-stage model that is applicable to multiple scenarios and has strong generalization; Step S304, the second stage, fine-tuning training based on real pictures, so that the first-stage model can better adapt to the real scene, correct the inconsistency problem of training data distribution caused by noise in the synthetic data set, and obtain a high-precision target detection model that fits the real prediction scene.

4. The computer vision-based book inventory method according to claim 1, characterized in that: In step S3, the method for performing instance segmentation on the picture sequence includes: Step S311: Send each image in the image sequence to the object detection model to obtain the bounding boxes of all the spine instances and call number instances that appear; specifically, the model outputs a rotated bounding box, which contains not only the coordinates of the original rectangle but also its rotation angle; Step S312, according to the obtained position coordinates and tilt angle of the rectangular frame, the four corner coordinates of the rotational bounding frame are calculated, and the image area enclosed by the rotational bounding frame is the truly recognized book spine or call number instance; Step S313, filtering the bounding box results of the book spine and call number instances with too small areas, and determining whether the four corner coordinates of the bounding box are within the boundary of the image. If there are corner points that are out of range, the corresponding instance recognition results are discarded; Step S314: For the book spine instance bounding box, divide it into two groups of line pairs, namely, two left and right lines and two upper and lower lines. Only the left and right line pairs are taken, and the left and right lines are respectively extended to the upper and lower edges of the image to obtain a polygon composed of four intersection points as a new instance bounding box, which is equivalent to completing the division of the left and right sides of the book; Step S315: the above operation does not need to be performed on the call number instance bounding box, and the original bounding box range is directly taken as the segmentation result; Step S316: Non-maximum suppression is applied to the book spine and call number instance segmentation frames respectively, and only instances with high confidence and less overlap with other instances are retained as the final segmentation results.

5. The computer vision-based book inventory method according to claim 1, characterized in that: In step S4, the method of training the text recognition model and performing text recognition includes: Step S401: Train a text recognition model based on the ICDAR 2019-LSVT Chinese text dataset. The text recognition model consists of two models: detection and recognition. The detection model is built based on the DB and trained with mixed precision. The recognition model consists of CRNN. Step S402: using the trained text recognition model, identifying the position coordinates of the text instances appearing in the image sequence and the corresponding text; Step S403: for each image in the image sequence, according to the segmented book spine and call number positions and the identified text box positions, the text corresponding to the text box is assigned to each book spine and call number instance, and the text corresponding to each book spine and call number box is the splicing result of the assigned text; thus, the position box of each book spine and call number and the corresponding identified text, i.e., <image, position coordinate, identified text> pair, are obtained; the book title recognition result includes the information of the book title, author, and publisher, and the call number recognition result includes the information of the English letters of the call number and the library name, wherein the Chinese characters in the call number need to be removed; Step S404, according to the spine and call number coordinates obtained by instance segmentation, calculate which spine instance the call number belongs to, assign the call number identification text to the spine instance, and finally obtain the result pair of <image, book location, book title and call number identification text>.

6. The computer vision-based book inventory method according to claim 1, characterized in that: In step S5, the target tracking process includes: based on the optical flow data, calculating the area of ​​the spine or call number instance in the next image to which the center of all book spines and call number instances on the previous image moves after the optical flow, so as to match the repeated book instances that appeared in the previous and next images, and aggregate the <image, book position, book title and call number recognition text> pairs belonging to the same book to obtain all recognition results of the same book in different frames, which are changed into <instance coordinates, book title and call number text recognition sequence>; at this time, the coordinates of each book instance are global coordinates in a layer of bookshelf, and the start or end position of the bookshelf is the origin.

7. The computer vision-based book inventory method according to claim 1, characterized in that: In step S6, the method for correcting and matching based on the candidate set of the book database includes: For the <instance coordinates, book title and call number text recognition sequence> obtained by target tracking, select the recognition result with the highest confidence from the recognition sequence: first perform an exact match of the call number in the library database (a match is successful only when all characters are the same). If any match in the call number recognition sequence is successful, the matched book information is used as the inventory result; otherwise, perform book title matching, perform word segmentation on any recognition result in the book title recognition sequence, select candidates with the same words after word segmentation in the local candidate set of the current layer of books, and then use TF-IDF to calculate its similarity with the current recognized book title, and take the book title with the highest similarity that exceeds the set similarity threshold. If the number of occurrences is the same, then select the correction result with the highest score.

8. The computer vision-based book inventory method according to claim 3, characterized in that: In the step S301, the synthetic instance segmentation data set includes real book images and synthetic book images; The synthesis method thereof includes: Step S3011: deploy a camera in a real bookshelf scene, and move the camera at a constant speed to take pictures of books on different layers of the bookshelf, thereby obtaining several pictures of the real bookshelf; Step S3012, based on the real bookshelf pictures taken, synthesize and enhance the instance segmentation data set: cut out the spine of each book from the real picture, and perform random rotation, random deformation, random lighting change, random addition of white noise, and random addition of light spot data enhancement to obtain several new book spine instances; Step S3013: construct a background image library based on real empty bookshelf background images and background images randomly selected from the ImageNet dataset, randomly extract a certain number of backgrounds from them, randomly paste the above-synthesized new spine instances to random positions in these backgrounds, and obtain the four corner position coordinates of the new spine instances in the new backgrounds, and automatically add them to the synthesized annotation file.

9. The computer vision-based book inventory method according to claim 3, characterized in that: In the step S6, before correcting and matching the recognition results of the book title text and the call number text, it also includes: according to the shelf number and layer number scanned by the current camera, querying the library database for the information of the books on the corresponding shelf layer as a set of matching candidate books.

10. The computer vision-based book inventory method according to claim 1, characterized in that: In the step S1, the video frame-by-frame splitting method includes: reading the video frame by frame, performing distortion correction and image rotation operations on each frame of the picture, and obtaining a distortion-free front view of the bookshelf.

Images