A method for locating books

Abstract

The embodiments of the present disclosure provide a method for positioning a book. First, based on the book information, the corresponding original spine image is obtained from the original spine image library; a bookshelf image with a marking pattern is obtained; the bookshelf image with the marking pattern is input into the trained marking pattern positioning model to obtain the positioning information of the marking pattern; based on the positioning information of the marking pattern, the original spine image is deflected to obtain a deflected spine image; the degree of matching between the deflected spine image and the bookshelf image with the marking pattern is calculated, and if the matching degree reaches a threshold, the positioning information of the deflected spine image is calculated and the positioning information is output. The present disclosure solves the problem that during the book positioning process, the original spine image is taken from the front, while the spine image on the image to be matched is a "side view" and the shooting angle is unknown. It can significantly improve the accuracy and stability of book positioning.

Description

Technical Field

[0001] The present disclosure relates to the field of book information management, and in particular to a method for locating books. Background Art

[0002] Rapid book location is widely used in library management. Leveraging computer vision technology, a camera captures an image of the spine of a book placed on a shelf and compares it with pre-stored spine images to quickly locate the book. This method requires no human intervention and is simple and easy to use.

[0003] However, this method presents a serious problem in practice. Due to the camera's mounting angle, it's difficult to ensure the camera is facing the book. This means that most spine images are only side views, and the angle of capture is unknown. Pre-stored spine images are mostly front views. Therefore, directly using image matching to identify the spine can result in significant errors, making it impossible to accurately locate the book, severely impacting the effectiveness of this method. Summary of the Invention

[0004] The present disclosure provides a book positioning method, apparatus, device and storage medium.

[0005] According to a first aspect of the present disclosure, a method for locating a book is provided. The method comprises: obtaining a corresponding original book spine image from an original book spine image library based on book information; obtaining a bookshelf image with a marking pattern; inputting the bookshelf image with the marking pattern into a trained marking pattern positioning model to obtain positioning information of the marking pattern; deflecting the original book spine image based on the positioning information of the marking pattern to obtain a deflected book spine image; calculating a degree of match between the deflected book spine image and the bookshelf image with the marking pattern; and if the degree of match reaches a threshold, calculating positioning information of the deflected book spine image and outputting the positioning information.

[0006] According to the above aspects and any possible implementation, an implementation is further provided, wherein the deflection processing includes: obtaining a deflection calculation formula using the positioning information of the marking pattern; and performing deflection processing on the original spine image based on the deflection calculation formula.

[0007] According to the aspects described above and any possible implementation method, an implementation method is further provided, wherein obtaining the deflection calculation formula includes: obtaining the initial coordinates of the marking pattern, wherein the initial coordinates are the pixel coordinates of the marking pattern on the bookshelf image taken without deflection; calculating the deflection relationship between the initial coordinates and the positioning information of the pattern, and obtaining the deflection matrix as the deflection calculation formula.

[0008] According to the aspects and any possible implementation methods described above, an implementation method is further provided, in which the training method of the marking pattern positioning model includes: obtaining a set of bookshelf images with marking patterns through multi-angle shooting; using a calibration tool to use the pixel coordinates of the four vertices of the marking pattern as labels on each frame of the collected image to form a training data set; using the training data set to train a preset marking pattern positioning model to obtain a trained marking pattern positioning model.

[0009] According to the above aspects and any possible implementation manner, there is further provided an implementation manner, wherein the marking pattern is fixedly set at a selected position of the bookshelf.

[0010] As described above, an implementation method is further provided, in which the method for constructing the original spine image library includes: photographing the spine of each book in all the books from the front; cropping to obtain a complete and non-deflected original spine image; and labeling each original spine image with book information; storing the original spine image and book information in a database that supports queries to obtain the original spine image library.

[0011] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the matching degree includes: the matching degree algorithm includes Harris, SIFT, SURF, and FAST.

[0012] According to a second aspect of the present disclosure, a book positioning device is characterized in that it includes: an original spine image unit, which is used to obtain a corresponding original spine image from an original spine image library based on book information; a spine image unit, which is used to obtain a bookshelf image with a marking pattern; a positioning unit, which is used to input the bookshelf image with the marking pattern into a trained marking pattern positioning model to obtain positioning information of the marking pattern; a deflection unit, which is used to deflect the original spine image based on the positioning information of the marking pattern to obtain a deflected spine image; a matching unit, which is used to calculate the matching degree between the deflected spine image and the bookshelf image with the marking pattern, and if the matching degree reaches a threshold, calculate the positioning information of the deflected spine image, convert the positioning information of the deflected spine image into highly readable position information, and output the position information.

[0013] According to a third aspect of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect of the present disclosure.

[0014] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect of the present disclosure.

[0015] It can be seen that in the embodiment of the present application, the marking pattern is first used to locate and extract the direction of the camera. Then the original spine image is deflected based on the direction of the camera, and finally the deflected image is used to locate the book. This solves the problem that the original spine image is taken from the front, while the spine image on the image to be matched is a "side view" and the shooting angle is unknown. The accuracy and stability of spine image matching can be significantly improved, and the effect of automatic book positioning can be improved. It should be understood that the content described in the invention content section is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0017] Figure 1 is a block diagram of a book positioning system according to an embodiment of the present disclosure;

[0018] Figure 2 is a flow chart of a method for locating books according to an embodiment of the present disclosure;

[0019] Figure 3 is a flow chart of performing deflection processing on an original book spine image according to an embodiment of the present disclosure;

[0020] Figure 4 is a flow chart of obtaining a deflection calculation formula according to an embodiment of the present disclosure;

[0021] Figure 5 is a flow chart of training a marker pattern positioning model according to an embodiment of the present disclosure;

[0022] Figure 6 is a flow chart of a method for constructing an original book spine image library according to an embodiment of the present disclosure;

[0023] Figure 7 FIG2 shows a block diagram of a book positioning device 700 according to an embodiment of the present disclosure;

[0024] Figure 8 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0026] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0027] This disclosure provides a method for book location based on deflected spine images. This method solves the problem of the original spine image being taken from the front, while the spine image on the image to be matched is a side view with an unknown shooting angle. This significantly improves the accuracy and stability of spine image matching, enhancing the effectiveness of automatic book location.

[0028] The present disclosure can be used in the following scenarios: (1) A librarian needs to help a user find the specific location of a book. (2) When a book is placed in the wrong location, the librarian needs to locate the book. (3) A robot completes a book search and return action.

[0029] Figure 1 It is a block diagram of a book positioning system according to an embodiment of the present disclosure.

[0030] like Figure 1 As shown, the system 100 includes an image acquisition device 110 , an image annotation device 120 , and a book positioning device 130 .

[0031] The purpose of the image acquisition device 110 is to collect three types of images. The first type is a collection of images containing marking patterns. The second type is a library of book spine images. The third type is a real-time image of the bookshelf.

[0032] Image annotation device 120 includes a display and a processing program. Its task is to assist in generating training data for marker pattern recognition. First, the display is used to extract all images of the same bookshelf and a particular marker pattern from an image collection. All images containing the marker pattern are then viewed and processed for each frame.

[0033] The book locating device 130 includes a processing program. This program is associated with the real-time image acquisition portion of an image acquisition device. It also includes an image processing method. The processing program receives one or more frames of images of each bookshelf or a specific bookshelf from the real-time image acquisition device. The device also includes a training program for a marker pattern recognition model. The recognition model automatically identifies the marker pattern in the real-time acquired image and a positioning model whose output includes the coordinates of the four vertices of the pattern box.

[0034] The book positioning device 130 also includes basic calibration information for the marking pattern, a bookshelf plane image, and real coordinates. It also includes a conversion program. By inputting the coordinates of the four vertices of the marking pattern into the program, the program outputs a conversion matrix.

[0035] The book positioning device 130 also includes image deflection and image matching functions. It uses the camera's pointing angle or a transformation matrix to deflect the original book spine image. It then determines whether the deflected spine image matches the real-time bookshelf image. If a match is found, the book's location information is output.

[0036] Figure 2 4 is a flowchart of a method for locating a book according to an embodiment of the present disclosure.

[0037] refer to Figure 2 ,

[0038] At 210,

[0039] Based on the book information, the corresponding original spine image is obtained from the original spine image library.

[0040] In some embodiments, basic information about the book to be located is first obtained, including the title, author, and keywords. A search function is then used to search the original spine image based on this basic information. If the database returns multiple books, a book to be located is selected, and the original spine image or visual feature data of the spine of the book to be located is extracted from the database.

[0041] At 220,

[0042] Get the image of the bookshelf with the marked pattern,

[0043] In some embodiments, an image capture device is used to capture real-time images of the bookshelf. During the capture process, the image capture device must be aligned with the bookshelf to ensure that each frame contains the marking pattern. The captured images require real-time marking pattern location and image matching to avoid the captured images not reflecting the current book positions, resulting in invalid location information.

[0044] At 230,

[0045] Input the bookshelf image with the marking pattern into the trained marking pattern positioning model to obtain the positioning information of the marking pattern,

[0046] In some embodiments, the complete bookshelf image or the features of the bookshelf image are input into the trained marker pattern positioning model, and the output of the model is the pixel coordinates of the marker pattern.

[0047] At 240,

[0048] Based on the positioning information of the marking pattern, the original spine image is deflected to obtain a deflected spine image.

[0049] In some embodiments, the camera pointing angle parameters or two sets of pixel coordinates of the same marker are used to restore the deflection matrix and deflect the original book spine image. This is because image matching is based on the visual features of the image. The real-time acquired image has problems such as deformation, which will affect the calculation of visual features. Since the book spine image is generally narrow, this deformation will more seriously affect the visual features of the image. Here, after the original image is deflected using the camera angle, the deflected image has the same deformation characteristics as the real-time acquired image. Therefore, the accuracy of image matching can be improved.

[0050] At 250,

[0051] Calculate the matching degree between the deflected book spine image and the bookshelf image with the marking pattern. If the matching degree reaches a threshold, calculate the positioning information of the deflected book spine image and output the positioning information.

[0052] In some embodiments, the real-time collected images are processed by first extracting all spine regions of the entire bookshelf image, segmenting each book, and extracting the spine image of each cut book or calculating visual features.

[0053] In some embodiments, image matching is performed on each spine image segmented from the bookshelf image using the features of the deflected spine image. The matching process first extracts image features, then generates feature descriptors. Finally, the features of the two images are matched based on the similarity of the descriptors. The matching algorithm ultimately outputs the similarity between the two images. Specific matching algorithms include, but are not limited to, Harris, SIFT, SURF, and FAST. Finally, by comparing a preset similarity threshold, it is determined whether a single bookshelf contains a deflected spine image.

[0054] In some embodiments, the relative coordinates of the corresponding cut-out book spine image are compared with the relative coordinates of each area of ​​the bookshelf to determine the pixel coordinates of the book to be located.

[0055] In some embodiments, a structural diagram of the bookshelf is obtained at the same time. The structural diagram should include the number of columns, the number of layers, and the partitions. According to the structural diagram, the bookshelf is divided into different areas, which can be quickly located. Each area has descriptive information, for example, 2 layers and 2 columns, area A. This can narrow the book positioning range and facilitate book locking. The structural diagram needs to be a design drawing of the same scale or a front-facing non-deflected image. The coordinate range of each area is defined.

[0056] In some embodiments, the coordinate transformation relationship is used to calculate the relative coordinates of each area on the bookshelf. For example, for area A on the 2nd floor and 2nd column, the corresponding relative coordinate range is [(Xa1, Ya1), (Xa2, Ya2)(Xa3, Ya3)(Xa4, Ya4)].

[0057] In some embodiments, the area of ​​the corresponding bookshelf is determined based on the pixel coordinates of the book to be located and the relative coordinates of each area, and the final output is the description information of the area.

[0058] Figure 3 The flowchart of the deflection process of the original book spine image according to the embodiment of the present disclosure is shown in FIG.

[0059] refer to Figure 3 ,

[0060] At 310,

[0061] The deflection calculation formula is obtained using the positioning information of the marking pattern.

[0062] In some embodiments, the four vertex coordinates output by the input marking pattern positioning device are used to deduce the pointing angle of the camera using the PNP (Perspective-N-Point) algorithm. Then, based on the pointing angle of the camera, the image is deflected. The deflection relationship between the pixel coordinates and the real coordinates can also be calculated to calculate the deflection matrix.

[0063] In some embodiments, the deflection relationship is established using the relative coordinates of the four vertices of the marking pattern and the coordinates of the four vertices of the marking pattern on the structure map.

[0064] At 320,

[0065] Deflection processing is performed on the original spine image based on the deflection calculation formula.

[0066] In some embodiments, each pixel of the original spine is transferred to a corresponding position in the deflected spine image through deflection processing. When all pixels are transferred, a difference processing is performed on pixels without values ​​in the deflected spine image to obtain a complete deflected spine image.

[0067] Figure 44 is a flow chart of obtaining a deflection calculation formula according to an embodiment of the present disclosure.

[0068] refer to Figure 4 ,

[0069] At 410,

[0070] Acquire the initial coordinates of the marking pattern, where the initial coordinates are the pixel coordinates of the marking pattern on the bookshelf image taken without deflection.

[0071] In some embodiments, a camera is first mounted on a bookshelf. The camera is aligned with the marking pattern from a vertical perspective under ideal lighting conditions, ensuring that the marking pattern is ideally illuminated, free of obstructions, and free of deflection or distortion. A standard image of the entire bookshelf is captured. The pixel coordinates of the marking pattern are obtained from the standard image of the bookshelf as initial coordinates.

[0072] At 420,

[0073] Calculate the deflection relationship between the initial coordinates and the positioning information of the pattern, and obtain the deflection matrix as the deflection calculation formula.

[0074] In some embodiments, in two-dimensional space, the deflection matrix is ​​also called a coordinate transformation matrix or an affine transformation matrix.

[0075] Figure 5 4 is a flowchart of training a marker pattern positioning model according to an embodiment of the present disclosure.

[0076] refer to Figure 5 ,

[0077] At 510,

[0078] By shooting from multiple angles, we can obtain a set of bookshelf images with marked patterns.

[0079] In some embodiments, a marker pattern is placed at a fixed, unobstructed location within a bookshelf, and initial coordinates of the marker pattern are obtained. This can be obtained using an image captured vertically. Alternatively, the marker pattern can be placed proportionally to a bookshelf structure diagram to obtain a composite image, and the coordinates of the marker pattern on the composite image are then obtained as the initial coordinates. The purpose of obtaining the initial coordinates of the marker pattern is to support the subsequent acquisition of the deflection matrix.

[0080] In some embodiments, one or more cameras are used to photograph the bookshelf at different locations.

[0081] In some embodiments, an image acquisition device is used to capture an image set of a marking pattern. The camera needs to capture images from multiple angles and orientations. The camera needs to be mounted on a rack or robot that can move, shift up and down, and change angles. Before acquisition, the visible angle interval values, acquisition orientation, and acquisition height must be pre-set. These ranges must cover the possible setting ranges of subsequent real-time image acquisition equipment. During the acquisition process, the rack needs to be aligned with the bookshelf orientation to ensure that each frame contains the marking pattern.

[0082] In some embodiments, the collected bookshelf image is expanded. The expansion can be performed using methods such as angle deflection, noise superposition, and image blurring. The benefit of the expansion process is that it increases the amount and richness of the training data.

[0083] At 520,

[0084] Using the calibration tool, the pixel coordinates of the four vertices of the marking pattern are used as labels on each frame of the collected image to form a training data set.

[0085] In some embodiments, the expanded image set is annotated. For each image frame, a marker pattern is first annotated with a box. The box must completely and tightly encompass the marker pattern. The processing program extracts the coordinates of the box's four vertices [(x1, y1), (x2, y2), (x3, y3), (x4, y4)]. The vertex coordinates can be pixel coordinates of the image. After the vertex coordinates are extracted, a training dataset is formed.

[0086] At 530,

[0087] The preset marking pattern positioning model is trained using the training data set to obtain the trained marking pattern positioning model.

[0088] In some embodiments, the preset marker pattern positioning model is a neural network model. The training data set is input into the neural network model for training. If there are multiple bookshelves and multiple icon placement patterns, multiple neural network models are required to ensure the accuracy of image matching.

[0089] The input of the neural network model is a frame of image, and the output is the coordinates of the four vertices of the box corresponding to the marking pattern. If the model accuracy is not ideal, first consider replacing the marking pattern. In order to facilitate pattern recognition, the marking pattern needs to be designed in advance. The main design principles include that the marking pattern has high recognition and obvious differences in color features, texture features, shape features and bookshelf background. The marking pattern is preferably square to reduce the complexity of image segmentation. If the accuracy of the training model is lower than 99%, it means that the recognition of the marking pattern is low and the marking pattern should be redesigned.

[0090] Figure 6 4 is a flowchart of a method for constructing an original book spine image library according to an embodiment of the present disclosure.

[0091] refer to Figure 6 ,

[0092] At 610,

[0093] Take a frontal photo of the spine of each book in the entire collection.

[0094] In some embodiments, using an image acquisition device, first secure the camera to a rack. The books to be captured need to be placed in the center of the rack. Capturing can be done on a single or multiple books. The camera needs to capture images of the book spines from a vertical perspective under ideal lighting conditions. If capturing multiple books, ensure that each book has ideal lighting, is not covered, and is not deflected or deformed.

[0095] At 620,

[0096] Cut to get the complete original spine image without deflection.

[0097] In some embodiments, a display is used to manually or automatically cut and obtain a spine image. After the image is cut, a regular and complete spine image of a single book is extracted. A unique number is assigned to each spine image.

[0098] At 630,

[0099] Label each original spine image with book information,

[0100] In some embodiments, book information is added to the spine image manually or automatically, such as the title, introduction, author, category, etc. It is necessary to ensure that the spine images of the same book are associated with the same book information.

[0101] At 640,

[0102] Store the original spine image and book information in a database that supports queries, and obtain the original spine image library.

[0103] In some embodiments, the book information is associated with a spine image library, and the spine image of the book to be located can be extracted from the spine image library using the book information.

[0104] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0105] The above is an introduction to the method embodiment. The following is a further explanation of the solution disclosed in the present disclosure through an apparatus embodiment.

[0106] Figure 7 A block diagram of a book positioning device 700 according to an embodiment of the present disclosure is shown.

[0107] The apparatus 700 may be included in Figure 1 Of the 130. Figure 7 As shown, the apparatus 700 includes:

[0108] The original spine image unit 710 is used to obtain the corresponding original spine image from the original spine image library based on the book information;

[0109] The bookshelf image unit 720 is used to obtain a bookshelf image with a marking pattern;

[0110] A positioning unit 730 is configured to input the bookshelf image with the marking pattern into a trained marking pattern positioning model to obtain positioning information of the marking pattern;

[0111] A deflection unit 740 is configured to perform deflection processing on the original spine image based on the positioning information of the marking pattern to obtain a deflected spine image;

[0112] The matching unit 750 is used to calculate the matching degree between the deflected book spine image and the bookshelf image with the marking pattern. If the matching degree reaches a threshold, the positioning information of the deflected book spine image is calculated, the positioning information of the deflected book spine image is converted into highly readable position information, and the position information is output.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0114] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0115] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0116] Figure 8A schematic block diagram of an electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0117] The electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a ROM 802 or a computer program loaded from a storage unit 808 into a RAM 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An I / O interface 805 is also connected to the bus 804.

[0118] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0119] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the marking pattern location method. For example, in some embodiments, the book location method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the book location method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the book location method by any other suitable means (e.g., via firmware).

[0120] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0126] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0127] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for locating a book, comprising: Based on the pre-acquired basic information of the book to be located, the corresponding original spine image is obtained from the original spine image library; Get an image of a bookshelf with a marked pattern; Inputting the bookshelf image with the marking pattern into the trained marking pattern positioning model to obtain positioning information of the marking pattern; Based on the positioning information of the marking pattern, the original spine image is deflected to obtain a deflected spine image; Calculating a degree of matching between the deflected book spine image and the bookshelf image with the marking pattern, and if the degree of matching reaches a threshold, calculating positioning information of the deflected book spine image and outputting the positioning information; The deflection process includes: Obtaining a deflection calculation formula using the positioning information of the marking pattern; Performing deflection processing on the original book spine image based on the deflection calculation formula; The deflection calculation formula includes: Acquire initial coordinates of the marking pattern, where the initial coordinates are pixel coordinates of the marking pattern on the bookshelf image, where the bookshelf image is captured without deflection of the marking pattern; Calculating the deflection relationship between the initial coordinates and the positioning information of the pattern, and obtaining a deflection matrix as a deflection calculation formula; The step of calculating the degree of matching between the deflected book spine image and the bookshelf image with the marking pattern, and if the degree of matching reaches a threshold, calculating positioning information of the deflected book spine image and outputting the positioning information comprises: Extract all spine areas of the bookshelf image with the marking pattern, segment each book, and extract the segmented spine image of each book; use the deflected spine image to perform image matching on each segmented spine image to obtain the matching degree between the deflected spine image and each spine image; by comparing with a preset matching degree threshold, determine whether the bookshelf image contains the deflected spine image; if so, compare the relative coordinates of the corresponding spine image with the relative coordinates of each area of ​​the bookshelf to determine the pixel coordinates of the book to be located; determine the area of ​​the corresponding bookshelf based on the pixel coordinates of the book to be located and the relative coordinates of each area, and output description information of the area.

2. The method according to claim 1, characterized in that The training method of the marking pattern positioning model includes: By shooting from multiple angles, a set of bookshelf images with marked patterns is obtained; Using the calibration tool, the pixel coordinates of the four vertices of the marking pattern are used as labels on each frame of the acquired image to form a training data set; The training data set is used to train the preset marking pattern positioning model to obtain the trained marking pattern positioning model.

3. The method according to claim 1, characterized in that The marking pattern is fixedly arranged at a selected position of the bookshelf.

4. The method according to claim 1, wherein The method for constructing the original book spine image library includes: Take a front-facing photo of the spine of each book in the collection; The original spine image is obtained by cropping. And annotate each original book spine image with book information; The original book spine image and book information are stored in a database that supports querying, and an original book spine image library is obtained.

5. The method according to claim 1, wherein The matching degree includes: The matching algorithms include Harris, SIFT, SURF, and FAST.

6. A book positioning device, characterized in that: include: The original book spine image unit is used to obtain the corresponding original book spine image from the original book spine image library based on the pre-acquired basic information of the book to be located; a bookshelf image unit, configured to obtain a bookshelf image with a marking pattern; a positioning unit, configured to input the bookshelf image with the marking pattern into a trained marking pattern positioning model to obtain positioning information of the marking pattern; a deflection unit, configured to perform deflection processing on the original spine image based on the positioning information of the marking pattern to obtain a deflected spine image; a matching unit, configured to calculate a degree of matching between the deflected book spine image and the image of the bookshelf with the marking pattern, and if the degree of matching reaches a threshold, calculate positioning information of the deflected book spine image, convert the positioning information of the deflected book spine image into highly readable position information, and output the position information; The deflection unit is specifically used for: Obtaining a deflection calculation formula using the positioning information of the marking pattern; Performing deflection processing on the original book spine image based on the deflection calculation formula; The deflection calculation formula includes: Acquire initial coordinates of the marking pattern, where the initial coordinates are pixel coordinates of the marking pattern on the bookshelf image, where the bookshelf image is captured without deflection of the marking pattern; Calculating the deflection relationship between the initial coordinates and the positioning information of the pattern, and obtaining a deflection matrix as a deflection calculation formula; The matching unit is specifically used for: Extract all spine areas of the bookshelf image with the marking pattern, segment each book, and extract the segmented spine image of each book; use the deflected spine image to perform image matching on each segmented spine image to obtain the matching degree between the deflected spine image and each spine image; by comparing with a preset matching degree threshold, determine whether the bookshelf image contains the deflected spine image; if so, compare the relative coordinates of the corresponding spine image with the relative coordinates of each area of ​​the bookshelf to determine the pixel coordinates of the book to be located; determine the area of ​​the corresponding bookshelf based on the pixel coordinates of the book to be located and the relative coordinates of each area, and output description information of the area.

7. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are configured to cause the computer to execute the method according to any one of claims 1 to 5.

Images