Book label recognition device, book label recognition system, book label recognition method, and computer program

Abstract

To provide a book label recognition device, a book label recognition system, a book label recognition method, and a computer program that are expected to be easily introduced to an existing library.SOLUTION: In a book label recognition system in which a photographing device 1, a server device 3, and a terminal device 5 communicate via a network N, the server device 3 being a book label recognition device comprises: an image acquisition unit 31b that acquires an image obtained by photographing bookshelves; a label detection unit 31c that detects a label attached to each book stored in the bookshelves from the acquired image; a character recognition unit 31d that recognizes characters described on the detected label; a bookshelf determination unit 31e that determines whether the book is stored in a right place on the basis of, the characters recognized by the character recognition unit 31d; and a notification processing unit 31f that notifies a user of a result of determination made by the determination unit.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a book label recognition device, a book label recognition system, a book label recognition method, and a computer program that perform a process of recognizing a label attached to a book stored in a bookshelf of a library.

Background Art

[0002] Conventionally, books or materials (hereinafter simply referred to as books) handled in libraries, reading rooms, rental bookstores, etc. (hereinafter simply referred to as libraries) are stored and kept, for example, in a plurality of bookshelves or bookracks (hereinafter simply referred to as bookshelves). On a book, a label on which identification information or the like corresponding to the type of the book is described is attached to a predetermined position such as the back cover, and the book is managed based on the information described on this label.

[0003] In Patent Document 1, a storage management system is proposed that includes a book classification label incorporating an RF-ID element and an RF-ID reader installed on a bookshelf, and performs location management of books stored on the bookshelf. In this storage management system, attempts are made to improve the reliability of reading by having the RF-ID reader switch between a plurality of antenna unit parts and communicate with the RF-ID element.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the storage management system described in Patent Document 1, it is necessary to attach a classification label for books with an RF-ID element built in, and it is necessary to install an RF-ID reader on the bookshelf. Considerable time and cost are required to introduce it into an existing library. In particular, a huge amount of time is required for the work of attaching labels with RF-ID elements built in to all of the thousands to tens of thousands of books in the collection.

[0006] The present invention has been made in view of such circumstances, and an object thereof is to provide a book label recognition device, a book label recognition system, a book label recognition method, and a computer program that can be expected to facilitate introduction into an existing library.

Means for Solving the Problems

[0007] A book label recognition device according to an embodiment includes an acquisition unit that acquires an image of a bookshelf, a detection unit that detects a label attached to a book stored on the bookshelf from the acquired image, and a recognition unit that recognizes characters described on the detected label.

Effects of the Invention

[0008] In the case of an embodiment, it can be expected to facilitate the introduction of an existing library such as a device or system for recognizing book labels.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0010] A specific example of a book label recognition system according to an embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, and is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0011] <System Overview> FIG. 1 is a schematic diagram showing a configuration example of a book label recognition system according to the present embodiment. The book label recognition system according to the present embodiment is a system for managing the storage locations of a plurality of books stored in a library. Specifically, the book label recognition system according to the present embodiment determines whether a plurality of books stored in a library are stored in an appropriate bookshelf where they should be stored based on identification information and the like described on a label attached to the book, and notifies information regarding books stored in an incorrect storage location to library staff or the like.

[0012] In the book label recognition system according to this embodiment, for example, a label on which identification information indicating the type of this book or the like is described is pre - attached to a predetermined position such as the back cover of a book. The label to be attached to a book does not necessarily need to have a communication function or the like, and may be a seal or a piece of paper on which the identification information is printed. Also, the label does not need to be dedicated to the book label recognition system according to this embodiment, and existing labels attached to books in a library can be diverted. Note that it is preferable that the label is attached at a position where it can be visually recognized while the book is stored on a bookshelf.

[0013] In the book label recognition system according to this embodiment, an imaging device 1 for imaging a bookshelf is used. The imaging device 1 is a device in which a camera is mounted on a moving mechanism such as wheels or legs, and can move (self - run) inside the library to image the bookshelf. In FIG. 1, an imaging device 1 provided with wheels as a moving mechanism is illustrated, but the imaging device 1 may be one that moves by flying such as a drone. Also, in the book label recognition system according to this embodiment, the movement of the imaging device 1 is not performed by manual operation by a staff member or the like, but is performed by autonomous control or remote control by a server device 3 or the like.

[0014] For example, at a predetermined timing such as after the library closes or before it opens, the imaging device 1 moves inside the library along a predetermined route and sequentially images a plurality of bookshelves provided inside the library. The imaging device 1 has a function of wireless communication by, for example, a wireless LAN (Local Area Network) or a mobile phone communication network, and transmits an image of the bookshelf on which the books are stored to the server device 3.

[0015] The server device 3 of the book label recognition system according to this embodiment may be installed, for example, in a library, or may be installed, for example, in a location separate from the library. The server device 3 acquires the image captured by the imaging device 1 and performs a process of detecting a label attached to the back cover or the like of a book on a bookshelf shown in the image. Further, the server device 3 determines the type of the book to which this label is attached by performing a process of recognizing identification information such as characters or numbers described on the detected label. The server device 3 also stores in advance the correspondence between the identification information described on the label of the book and the bookshelf in which the book corresponding to this identification information should be stored. Based on the stored correspondence and the identification information recognized from the label of the book, the server device 3 determines whether or not this book is stored in an appropriate bookshelf. When it is determined that the book is not stored in an appropriate bookshelf, the server device 3 notifies the terminal device 5.

[0016] The terminal device 5 is a device used by a user such as a library staff member, and is, for example, an information processing device used for inventory management of library books. The terminal device 5 may be, for example, a PC (personal computer) installed in a library, or may be a portable device such as a smartphone or a tablet terminal device. The terminal device 5 that has received the notification from the server device 3 notifies the user of information regarding a book that is not stored in an appropriate bookshelf.

[0017] <Device Configuration> Figure 2 is a block diagram showing the configuration of the imaging device 1 according to the present embodiment. The imaging device 1 according to the present embodiment includes a processing unit 11, an imaging unit 12, a storage unit 13, a moving mechanism 14, a communication unit 15, and the like. The processing unit 11 is configured using an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The processing unit 11 reads and executes the program 13a stored in the storage unit 13, thereby performing various processes such as a process of controlling the imaging by the imaging unit 12 and a process of controlling the moving mechanism 14 to self-drive.

[0018] The imaging unit 12 includes a lens, an imaging element, and the like. The imaging element is, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), etc. It captures the image converged by the lens and outputs the data of the image obtained by the imaging.

[0019] The storage unit 13 is configured using a non-volatile memory element such as a flash memory or an EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 13 stores various programs executed by the processing unit 11 and various data necessary for the processing of the processing unit 11. In the present embodiment, the storage unit 13 stores the program 13a executed by the processing unit 11. The storage unit 13 may also store the data of the image captured by the imaging unit 12.

[0020] In the present embodiment, the program 13a is written in the storage unit 13, for example, at the manufacturing stage of the imaging device 1. Further, for example, the program 13a may be acquired by the imaging device 1 through communication from a remote server device or the like that distributes it. Further, for example, the program 13a may be provided in a manner recorded on a recording medium such as a memory card or an optical disk, and the imaging device 1 may read the program 13a from the recording medium and store it in the storage unit 13. Further, for example, the program 13a may be read by a writing device from what is recorded on the recording medium and written into the storage unit 13 of the imaging device 1. The program 13a may be provided in a mode of distribution via a network or in a mode of being recorded on a recording medium.

[0021] The moving mechanism 14 includes a plurality of wheels for self-propelling the imaging device 1, one or a plurality of motors for rotating the wheels, a gear mechanism for transmitting the rotation of the motor to the wheels, a drive circuit for driving the motor, and the like. For example, the moving mechanism 14 drives the motor according to a control command from the processing unit 11 and rotates the wheels to move the imaging device. Further, the moving mechanism 14 is provided with various sensors such as an ultrasonic sensor for detecting obstacles and a gyro sensor for detecting the position and orientation of the own device. The moving mechanism 14 gives the detection results by these sensors to the processing unit 11. Note that the moving mechanism 14 does not have to be by wheels, and any mechanism may be adopted.

[0022] The communication unit 15 communicates with various devices via a network N such as a mobile phone communication network, a wireless LAN (Local Area Network), or the Internet. In the present embodiment, the communication unit 15 communicates with the server device 3 and transmits the data of the image captured by the imaging unit 12 to the server device 3. The communication unit 15 transmits the data given from the processing unit 11 to other devices and gives the data received from other devices to the processing unit 11.

[0023] In the book label recognition system according to the present embodiment, it is assumed that the movement and shooting of the imaging device 1 are performed under the control of the server device 3. The imaging device 1 transmits the detection results of various sensors of the movement mechanism 14 to the server device 3, receives a control command related to movement or shooting from the server device 3, and the processing unit 11 controls the operations of each unit according to the received control command, thereby performing shooting by the shooting unit 12 and movement by the movement mechanism 14. However, the imaging device 1 may be configured to autonomously move and shoot.

[0024] FIG. 3 is a block diagram showing the configuration of the server device 3 according to the present embodiment. The server device 3 according to the present embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, and the like. In the present embodiment, the description is made assuming that processing is performed by one server device, but a plurality of server devices may perform distributed processing.

[0025] The processing unit 31 is configured using an arithmetic processing device such as a CPU, MPU, or GPU, a ROM, a RAM, and the like. By reading and executing the server program 32a stored in the storage unit 32, the processing unit 31 performs various processes such as a process of controlling the operation of the imaging device 1, a process of detecting a label of a book from an image captured by the imaging device 1, a process of recognizing characters described on the label, a process of determining whether the book is stored in an appropriate bookshelf, and a process of notifying the user of information regarding a book stored in an inappropriate bookshelf.

[0026] The storage unit 32 is configured using a large-capacity storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data necessary for the processing of the processing unit 31. In the present embodiment, the storage unit 32 stores the server program 32a executed by the processing unit 31, and is provided with a bookshelf DB (database) 32b that stores information regarding books stored in the bookshelf. The bookshelf DB 32b is a database that stores, for example, associating identification information indicating the type of book with identification information assigned to the bookshelf. The storage unit 32 also temporarily stores the image acquired from the imaging device 1.

[0027] In the present embodiment, the server program 32a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disk. The server device 3 reads the server program 32a from the recording medium 99 and stores it in the storage unit 32. However, the server program 32a may be written in the storage unit 32, for example, at the manufacturing stage of the server device 3. Also, for example, the server program 32a may be acquired by the server device 3 through communication from another remote server device or the like that distributes it. For example, the server program 32a may be read by a writing device from what is recorded on the recording medium 99 and written into the storage unit 32 of the server device 3. The server program 32a may be provided in a form of distribution via a network or in a form recorded on the recording medium 99.

[0028] The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, and the Internet. In the present embodiment, the communication unit 33 communicates with the imaging device 1 and the terminal device 5 via the network N. The communication unit 33 transmits the data given from the processing unit 31 to other devices and gives the data received from other devices to the processing unit 31.

[0029] Note that the storage unit 32 may be an external storage device connected to the server device 3. Also, the server device 3 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software. Further, the server device 3 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that receives operation inputs, or a display unit that displays images.

[0030] Also, in the server device 3 according to the present embodiment, by the processing unit 31 reading and executing the server program 32a stored in the storage unit 32, a photographing control processing unit 31a, an image acquisition unit 31b, a label detection unit 31c, a character recognition unit 31d, a bookshelf determination unit 31e, a notification processing unit 31f, etc. are realized in the processing unit 31 as software functional units.

[0031] The photographing control processing unit 31a performs processing for controlling operations such as the movement and photographing of the photographing device 1. For example, the server device 3 has data of an in-library map indicating the positions of a plurality of bookshelves and obstacles provided in the library, and the photographing control processing unit 31a grasps the position of the photographing device 1 in the library based on the detection results of various sensors obtained from the photographing device 1. The photographing control processing unit 31a transmits control commands regarding the movement and photographing to the photographing device 1 so that the photographing device 1 moves within the library along a predetermined route and photographs the bookshelf at a predetermined position. Note that, for processing such as the creation of the in-library map data and the movement control of the photographing device 1 in the library, for example, the technology of SLAM (Simultaneous Localization and Mapping) can be used. Since SLAM is an existing technology, detailed description thereof is omitted.

[0032] The image acquisition unit 31b performs a process of receiving and acquiring the image transmitted by the imaging device 1 through the communication unit 33. For the received image, the image acquisition unit 31b acquires information such as the position and orientation of the imaging device 1 when this image was taken, for example, from the imaging device 1 or from the control result of the imaging control processing unit 31a, and determines which bookshelf the image was taken of based on this information. The image acquisition unit 31b attaches the identification information of the photographed bookshelf to the image acquired from the imaging device 1 and stores it in the storage unit 32. Note that the photographed images stored in the storage unit 32 may be erased from the storage unit 32 at an appropriate timing after the subsequent processes such as label detection are completed.

[0033] The label detection unit 31c performs a process of detecting the label attached to the book from the image of the bookshelf acquired by the image acquisition unit 31b. In the present embodiment, the label detection unit 31c detects the label by either one or both of a method of detecting the label using a learning model pre-learned by machine learning and a method of detecting the label based on rules by performing predetermined image processing and calculations on the image. The details of these label detection methods will be described later.

[0034] The character recognition unit 31d performs a process of recognizing the characters described on the label detected by the label detection unit 31c. The character recognition unit 31d extracts one or a plurality of image regions in which the label is depicted from the photographed image based on the detection result of the label detection unit 31c, and performs a character recognition process on each of the extracted image regions. For the character recognition process, for example, the technology of OCR (Optical Character Recognition) is used. Since OCR is an existing technology, a detailed description thereof is omitted. Further, the character recognition unit 31d may perform a process of recognizing the characters described on the label using a learning model for character recognition learned by machine learning, deep learning, or the like.

[0035] The bookshelf determination unit 31e performs a process of determining whether a book is stored in an appropriate bookshelf based on the identification information of the bookshelf attached to the image by the image acquisition unit 31b, the information stored in the bookshelf DB 32b of the storage unit 32, and the characters of the label recognized by the character recognition unit 31d. The bookshelf determination unit 31e determines the type of the book to which this label is attached based on the characters of the label recognized by the character recognition unit 31d. The bookshelf determination unit 31e refers to the bookshelf DB 32b based on the identification information indicating the type of this book, and acquires the identification information of the bookshelf in which this book should be stored. The bookshelf determination unit 31e determines whether the identification information of the bookshelf acquired from the bookshelf DB 32b matches the identification information of the bookshelf attached to the image by the image acquisition unit 31b. The bookshelf determination unit 31e determines that the book is stored in an appropriate bookshelf when both pieces of identification information match, and determines that the book is not stored in an appropriate bookshelf when both pieces of identification information do not match.

[0036] The notification processing unit 31f performs a process of notifying the terminal device 5 of the determination result of the bookshelf determination unit 31e. For example, when the bookshelf determination unit 31e determines that a book is not stored in an appropriate bookshelf, the notification processing unit 31f transmits information about this book to the terminal device 5, causing the terminal device 5 to notify the user. At this time, the notification processing unit 31f transmits, for example, an image of the bookshelf in which the book to be notified is stored and information about the position of the label attached to the book to be notified to the terminal device 5. The terminal device 5 that has received these can notify the user which book is the notification target by, for example, superimposing and displaying a rectangular frame surrounding the book to be notified or its label on the image of the bookshelf.

[0037] FIG. 4 is a block diagram showing the configuration of the terminal device 5 according to the present embodiment. The terminal device 5 according to the present embodiment includes a processing unit 51, a storage unit (storage) 52, a communication unit (transceiver) 53, a display unit (display) 54, an operation unit 55, and the like. The terminal device 5 is a device used by a user such as a librarian in a library, and can be configured using an information processing device such as a smartphone, a tablet terminal device, or a personal computer.

[0038] The processing unit 51 is composed of an arithmetic processing unit such as a CPU or MPU, a ROM, etc. By reading and executing the program 52a stored in the storage unit 52, the processing unit 51 performs various processes such as receiving information transmitted from the server device 3 and notifying the user.

[0039] The storage unit 52 is configured using a non-volatile memory element such as a flash memory. The storage unit 52 stores various programs executed by the processing unit 51 and various data necessary for the processing of the processing unit 51. In the present embodiment, the storage unit 52 stores the program 52a executed by the processing unit 51. In the present embodiment, the program 52a is distributed by a remote server device or the like, and the terminal device 5 acquires this through communication and stores it in the storage unit 52. However, the program 52a may be written into the storage unit 52, for example, at the manufacturing stage of the terminal device 5. For example, the terminal device 5 may read the program 52a recorded on a recording medium 98 such as a memory card or an optical disk and store it in the storage unit 52. For example, the program 52a recorded on the recording medium 98 may be read by a writing device and written into the storage unit 52 of the terminal device 5. The program 52a may be provided in a distribution mode via a network or in a mode recorded on the recording medium 98.

[0040] The communication unit 53 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, and the Internet. In the present embodiment, the communication unit 53 communicates with the server device 3 via the network N. The communication unit 53 transmits data given from the processing unit 51 to other devices and gives data received from other devices to the processing unit 51.

[0041] The display unit 54 is configured using a liquid crystal display or the like, and displays various images, characters, and the like based on the processing by the processing unit 51. The operation unit 55 receives the user's operation and notifies the processing unit 51 of the received operation. For example, the operation unit 55 receives the user's operation by means of a mechanical button or an input device such as a touch panel provided on the surface of the display unit 54. Also, for example, the operation unit 55 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the terminal device 5.

[0042] Also, in the terminal device 5 according to the present embodiment, the processing unit 51 reads and executes the program 52a stored in the storage unit 52, whereby the display processing unit 51a and the like are realized in the processing unit 51 as software functional units. Note that the program 52a may be a program dedicated to the information processing system according to the present embodiment, or may be a general-purpose program such as an Internet browser or a web browser.

[0043] The display processing unit 51a receives the information transmitted by the server device 3, and performs processing to display characters, images, and the like on the display unit 54 based on the received information. As described above, when the server device 3 determines that a book is not stored in an appropriate bookshelf, the server device 3 transmits information regarding this book to the terminal device 5. The display processing unit 51a that has received the information from the server device 3 can notify the user, for example, by displaying a rectangular frame surrounding the book or its label that is incorrectly stored with respect to the image of the bookshelf. Also, the display processing unit 51a may notify the user by displaying information such as the identification information of the appropriate bookshelf where the incorrectly stored book should be stored and its location.

[0044] <Label Detection Processing> FIG. 5 is a schematic diagram showing the appearance of a book and a label. In the book label recognition system according to this embodiment, a label is pre-attached to each book stored in the library. The label is, for example, attached to the back cover of the book as shown in the figure. However, in the case of a book whose back cover area is smaller than the label, it may be attached to the front cover, the back cover, etc., or may be attached across the back cover and the back cover. The label is preferably attached to a visible location when the book is stored on the bookshelf.

[0045] In this embodiment, the label attached to the book is rectangular, and a quadrilateral frame line surrounding the periphery and a boundary line dividing the area within this frame into three areas: an upper area, a middle area, and a lower area are drawn. The label is one on which identification information called "claim symbol" is described. In this embodiment, the "claim symbol" includes three pieces of identification information: "classification number", "author symbol", and "volume number". The "classification number" is described in the upper area of the label, the "author symbol" is described in the middle area, and the "volume number" is described in the lower area. The "classification number" is identification information indicating the type of the book, the "author symbol" is identification information assigned to the author of the book, and the "volume number" is identification information indicating which volume it corresponds to in the case of a book in a series. Note that the illustrated label is just an example and is not limited to this. The shape of the label and the information described on the label may vary from library to library.

[0046] In this embodiment, it is assumed that the bookshelf for storing each book is determined by the "classification number". That is, each bookshelf and the "classification number" of the book are determined in advance. For the sake of simplicity of explanation, in this embodiment, it is assumed that a book with one "classification number" is stored in one bookshelf. However, in reality, a bookshelf may store books with a plurality of "classification numbers". In this case, for example, the "classification number" of the books to be stored may be determined for each shelf such as the first row, the second row... of the bookshelf. Also, there may be a case where a plurality of bookshelves are assigned as storage locations for one "classification number". In this case, for example, a plurality of pieces of identification information such as "classification number" and "author symbol" may be determined for the bookshelf.

[0047] In addition, the book label recognition system according to this embodiment determines whether books with a "classification number" suitable for the bookshelf are stored. However, the book label recognition system may further determine whether the books stored on the bookshelf are arranged in the order of "author symbols" (such as alphabetical order or kana order), or whether books in a series are arranged in the order of "volume numbers", etc. How to make a determination based on the information described on the book label may be appropriately set by the system designer or library staff, etc.

[0048] FIG. 6 is a schematic diagram showing an example of the arrangement of bookshelves in a library. In the illustrated example, for example, 18 bookshelves A to R are provided in the library. The 18 bookshelves A to R are arranged in a 3-row × 6-column matrix, and are adjacent to each other so as to form a group of 6 bookshelves. A passage for staff and users to pass through is provided between the groups of bookshelves. Note that the arrangement and number of the illustrated bookshelves are examples and are not limited thereto.

[0049] In the book label recognition system according to this embodiment, for example, after the library closes, the photographing device 1 moves inside the library to photograph the bookshelves. The path along which the photographing device 1 moves is predetermined. For example, in this figure, the path of the photographing device 1 is shown by a dashed arrow. The photographing device 1 moves along this predetermined path, takes a photograph by the photographing unit 12 in front of each bookshelf, and transmits the image of the photographed bookshelf to the server device 3. In this embodiment, the order in which the photographing device 1 photographs the bookshelves is predetermined. For example, the photographing is performed in the order of bookshelf A → B → C → F → I → E → H → D → G → J → M → K → N → L → O → R → Q → P. Note that this photographing order is an example and is not limited thereto. In addition, when there are bookshelves on both the left and right sides with respect to the traveling direction, the photographing device 1 can photograph the left and right bookshelves in order by changing the photographing direction of the photographing unit 12 or changing the direction of the photographing device 1.

[0050] The server device 3 that has received an image from the imaging device 1 can determine which bookshelf each image was taken of by associating the order in which a plurality of images were received with a predetermined imaging order of the bookshelves. Further, the imaging device 1 may attach its own position information or the like to the image by, for example, a sensor or the like and transmit it. In this case, the server device 3 can determine the correspondence between the image and the bookshelf based on the position information in addition to the reception order of the images. Alternatively, information such as an identification number may be described at a predetermined position of each bookshelf, and the server device 3 may recognize the identification number of the bookshelf from the captured image. The information such as the identification number in this case may be, for example, characters or numbers recognizable by a person, or may be, for example, a two-dimensional code or the like.

[0051] Also, in the book label recognition system according to the present embodiment, it is assumed that the imaging device 1 captures the entire one bookshelf in one imaging, but it is not limited to this. The imaging device 1 may capture one bookshelf in multiple times. Also, in the book label recognition system according to the present embodiment, the image captured by the imaging device 1 is transmitted to the server device 3 as a still image, but it is not limited to this, and a moving image may also be used.

[0052] FIG. 7 is a schematic diagram showing a configuration example of the bookshelf DB32b. The bookshelf DB32b provided in the server device 3 according to the present embodiment is a database that stores information regarding the types of books stored in the bookshelf, and is, for example, a database that stores by associating a "bookshelf ID" and a "classification symbol". The "bookshelf ID" is identification information uniquely assigned to the bookshelves installed in the library, and is, for example, information such as bookshelf A, bookshelf B,.... The "classification symbol" is information included in the "claim symbol" of the label attached to the book as described above, and is information indicating the type of the book. The server device 3 can determine whether the books are stored in appropriate bookshelves by referring to the bookshelf DB32b.

[0053] In the book label recognition system according to the present embodiment, the server device 3 performs a process of detecting a label attached to a book stored in the bookshelf from an image of the bookshelf photographed by the photographing device 1. The server device 3 according to the present embodiment performs label detection by adopting either one or both of the following two methods. The first method for performing label detection is a method using a learning model, and the second method is a rule-based method for performing predetermined image processing and calculations on an image.

[0054] (Label detection method using a learning model) FIG. 8 is a schematic diagram showing a configuration example of a learning model for performing label detection. The learning model according to the present embodiment is a learned learning model that has been subjected to learning processes such as machine learning, deep learning, or reinforcement learning in advance. The learning model performs predetermined calculations on various input information and outputs the calculation results. In the storage unit 32 of the server device 3, data such as coefficients and thresholds of functions defining this calculation are stored together with the server program 21a as a learning model.

[0055] In this embodiment, the learning model has a neural network structure in which a plurality of neurons are interconnected. Since it is an existing technology, a detailed description is omitted. A neuron is an element that performs operations on a plurality of inputs and outputs one value as an operation result. A neuron has information such as weighting coefficients and thresholds used in the operations. The learning model of the neural network includes an input layer that receives the input of one or more pieces of data, an intermediate layer that performs arithmetic processing on the data received in the input layer, and an output layer that aggregates the arithmetic results of the intermediate layer and outputs one or more values. Learning processes such as deep learning and reinforcement learning are processes of setting appropriate values for the coefficients and thresholds of each neuron constituting the neural network using a large number of pre-given learning data (teacher data). The learning model according to this embodiment is a learned model in which deep learning using teacher data is performed on the learning model of the neural network, and is learned by a method such as the gradient descent method, the stochastic gradient descent method, or the error backpropagation method. Note that the learning process of the learning model may be performed by the server device 3 or may be performed by other devices.

[0056] The server device 3 of the book label recognition system according to this embodiment extracts a plurality of images with an area corresponding to the label of a book from the image of the bookshelf photographed by the photographing device 1, and sequentially inputs the extracted images into the learning model. The learning model outputs the confidence level as to whether the input image is a book label or not as a numerical value in the range of, for example, 0 to 1. The server device 3 determines whether the extracted image is a label by determining whether the numerical value of the confidence level output by the learning model exceeds a preset threshold value. Further, the server device 3 sequentially inputs a plurality of images extracted from the image of the bookshelf into the learning model, and performs determinations based on the confidence levels output by the learning model in sequence, so that all labels included in the image of the bookshelf can be detected. The server device 3 can extract the position where the image determined to be a label was extracted as the position where the label is written in the image of the bookshelf, and acquire the position information of the plurality of labels as a detection result. Note that, as the learning model, a CNN (Convolution Neural Network) model including a convolutional layer, a pooling layer, etc. for extracting feature amounts from the input image can be adopted.

[0057] For the learning of the learning model, for example, teacher data in which a flag (a value of 1 or 0) indicating whether this image is a label or not is associated with images of labels with various colors, angles, etc. and images that are not labels is used. For example, by photographing the labels attached to the books in the library where this system is planned to be introduced, the designer of this system, library staff, etc. can create teacher data. Also, the creation of teacher data may be performed by the server device 3 or the like.

[0058] (Label detection method based on rules) FIG. 9 is a flowchart showing a processing procedure when label detection is performed based on rules. The server device 3 according to the present embodiment performs image processing for converting a color image into grayscale on the image of the bookshelf acquired from the imaging device 1 (step S1). The server device 3 performs image processing for noise removal on the grayscale-converted image (step S2). For noise removal, for example, image processing using a filter such as an average value filter or a median filter can be performed. The server device 3 performs image processing for binarizing the grayscale image from which noise has been removed (step S3). The server device 3 performs image processing for shrinking on the binarized image (step S4). The image processing for shrinking is a process of shrinking connected components and finally leaving one point for one connected component. The server device 3 performs image processing for contour extraction on the shrunk binary image (step S5).

[0059] Next, the server device 3 extracts, for example, a rectangular region based on the contour information extracted by these image processes, and determines whether the rectangular region is a label by checking whether the area of the extracted rectangular region corresponds to the area of the label (step S6). The server device 3 determines a rectangular region having an area corresponding to the label as a label, and outputs the label detection result by drawing the contour of the rectangular region determined as a label on the original bookshelf image (step S7), and ends the process.

[0060] Note that the image processing shown in the flowchart of this figure is an example and is not limited thereto. The server device 3 may detect the label of the book from the bookshelf image by combining various other image processes.

[0061] In the book label recognition system according to this embodiment, the server device 3 performs a process of detecting a label of a book from an image of a bookshelf using at least one of the above two methods. The server device 3 may also perform a process of detecting a label using both of the above two methods. In this case, for example, the server device 3 finally uses the one detected as a label by both methods as the label detection result, and does not finally treat the label detected by only one of the methods as a label, so that the two methods can be combined.

[0062] Based on the detection result of the label from the image of the bookshelf, the server device 3 performs a process of extracting the label image from the original image. For one or more label images extracted from the image of the bookshelf, the server device 3 performs character recognition processing such as OCR on each of them. Based on the result of the character recognition processing, the server device 3 acquires information such as "classification number", "author symbol", and "volume number" described on each label.

[0063] Regarding this character recognition process, there may be cases where the server device 3 cannot recognize characters from the image detected and extracted as a label. This may be, for example, when the width of the back cover of a book is smaller than the width of the label and a part of the label is attached to the back cover of the book, or when a small book is stored between large books and a part of the label is hidden, or when a book is tilted and stored in a bookshelf and the characters in the extracted label image are tilted, etc.

[0064] If the server device 3 fails to recognize the characters on the label detected from the image of the bookshelf, it may perform re - shooting using the imaging device 1. In re - shooting, the server device 3 performs movement control to move the imaging device 1 closer to, for example, the position where the book with the label that could not be recognized is stored, and performs shooting so that the unrecognizable label becomes easier to recognize. In this case, the server device 3 may perform controls such as magnifying and shooting the label that could not be recognized, or tilting the shooting unit according to the tilt of the book to shoot the label. However, if the characters still cannot be recognized even after re - shooting, the server device 3 stores information such as the position of the label that cannot be recognized as a character - unrecognizable label.

[0065] FIG. 10 and FIG. 11 are flowcharts showing the procedure of the process performed by the server device 3 according to the present embodiment. The processing unit 31 of the server device 3 according to the present embodiment starts this process at a predetermined timing, for example, after the library closes. The shooting control processing unit 31a of the processing unit 31 controls the operation of the imaging device 1 by performing wireless communication therewith, and moves the imaging device 1 along a predetermined path in the library (step S21). When the imaging device 1 reaches a predetermined shooting position such as the front position of the bookshelf, the shooting control processing unit 31a shoots the bookshelf (step S22).

[0066] The image acquisition unit 31b of the processing unit 31 acquires the image of the bookshelf shot by the imaging device 1 by receiving it via the communication unit 33 (step S23). The image acquisition unit 31b determines which bookshelf the acquired image was shot of based on information such as the position or order where the acquired image was shot (step S24). The image acquisition unit 31b attaches the ID of the bookshelf as a determination result to the acquired image and stores it in the storage unit 32.

[0067] The label detection unit 31c of the processing unit 31 performs a process of detecting a label attached to a book stored in the bookshelf on the image acquired in step S23 (step S25). At this time, the label detection unit 31c uses either or both of a method of detecting a label using the learning model shown in FIG. 8 or a method of detecting a label based on a rule shown in FIG. 9 to detect a label from the image of the bookshelf taken.

[0068] Based on the label detection result in step S25, the character recognition unit 31d of the processing unit 31 extracts the label image from the image of the bookshelf acquired in step S23 (step S26). The character recognition unit 31d performs a character recognition process on the image of one or more labels extracted in step S26 by a technique such as OCR (step S27). The character recognition unit 31d determines whether it has been able to recognize the characters described on the label for all the detected labels (step S28). If there is a label for which the characters cannot be recognized (S28: NO), the processing unit 31 returns the process to step S21, moves the imaging device 1 to a position suitable for imaging, for example, a label for which the characters cannot be recognized, and re-images the label.

[0069] When the characters of all the labels have been recognized (S28: YES), the bookshelf determination unit 31e of the processing unit 31 acquires the "classification symbol" described on the label based on the result of the character recognition (step S29). The bookshelf determination unit 31e refers to the bookshelf DB32b of the storage unit 32 based on the ID of the bookshelf determined in step S24 (step S30), and acquires the "classification symbol" of the books to be stored in this bookshelf. The bookshelf determination unit 31e determines whether the "classification symbol" acquired from this label matches the "classification symbol" of the books to be stored in this bookshelf for all the labels detected from the image, and stores the information of the book with a label for which both "classification symbols" do not match as a book stored in an inappropriate place (step S31).

[0070] The processing unit 31 determines whether or not the above determination has been completed for all the bookshelves provided in the library (step S32). If the processing has not been completed for at least one bookshelf (S32: NO), the processing unit 31 returns the processing to step S21 and performs the above processing for the next bookshelf. If the processing has been completed for all the bookshelves (S32: YES), the notification processing unit 31f of the processing unit 31 notifies the user's terminal device 5 of the result of the determination process as to whether the books are stored in the appropriate bookshelves (step S33), and ends the processing.

[0071] FIG. 12 is a schematic diagram showing an example of a notification screen displayed on the terminal device 5. The server device 3 according to the present embodiment transmits the result obtained by the processing of the flowcharts shown in FIGS. 10 and 11 to the terminal device 5. At this time, the server device 3 transmits, for example, an image of the bookshelf photographed by the photographing device 1, the ID of the bookshelf shown in the image, the position of the book or its label stored in an inappropriate location, the "classification symbol" obtained from the label of the book stored in an inappropriate location, the ID of the appropriate bookshelf in which the book with this "classification symbol" should be stored, and information such as the position of the book or its label for which the characters of the label could not be recognized to the terminal device 5.

[0072] The display processing unit 51a of the terminal device 5 that has received the information from the server device 3 displays the notification screen shown in FIG. 12 on the display unit 54 based on the received information. In the illustrated notification screen, images of three bookshelves A to C photographed by the photographing device 1 are joined together and displayed as one image. The images of the plurality of bookshelves are joined together so as to have the same arrangement as the arrangement of the plurality of bookshelves installed in the library. The image processing for joining the images of the plurality of bookshelves may be performed by the server device 3 or the terminal device 5. Further, the display processing unit 51a of the terminal device 5 displays a character string of the bookshelf ID assigned to this bookshelf above each bookshelf.

[0073] In addition, the display processing unit 51a notifies the location of books with inappropriate storage locations by superimposing, for example, a rectangular frame that surrounds a book or its label for which the storage location has been determined to be inappropriate, on the image of the bookshelf. In the illustrated example, a rectangular frame that surrounds the back cover of a book for which the storage location has been determined to be inappropriate is superimposed on the image of the bookshelf. In this case, the display processing unit 51a can detect, based on the position of the label detected by the server device 3, a rectangular region of a size within a predetermined range including the label from the image of the bookshelf, and superimpose a rectangular frame on this region as the back cover of the book to which the label is attached. Note that the detection of the back cover of the book may be performed by the server device 3 instead of the terminal device 5.

[0074] In addition, the display processing unit 51a notifies books for which the characters of the label could not be recognized and for which it was not possible to determine the storage location, by superimposing a dashed rectangular frame that surrounds the book or its label for which the characters of the label could not be recognized, on the image of the bookshelf. In this figure, books for which the storage location has been determined to be inappropriate are surrounded by a solid rectangular frame, and books for which the characters of the label could not be recognized are surrounded by a dashed rectangular frame, thereby indicating to the user that the notification contents are different. However, this is just an example, and for example, frames of different colors may be displayed, frames of different shapes may be displayed, or the difference in the notification contents may be expressed by other methods.

[0075] In addition, the display processing unit 51a overlays and displays an information display area for displaying information about the book to be notified or text such as a message on an empty area or the like in the image of the bookshelf. In this example, the display processing unit 51a displays a rectangular area with rounded corners as the information display area, and connects the rectangular frame surrounding the corresponding book and the information display area with a lead line to show the correspondence. In this example, the display processing unit 51a displays, in the information display area, a message notifying the original storage location, such as "Move to bookshelf G" and "Move to bookshelf D", for the books determined to have an inappropriate storage location. However, this message is just an example and is not limited to this. For example, the classification symbol of the book may be displayed, or a message notifying that the storage location of the book is inappropriate, such as "The storage location is inappropriate", may be displayed, or other messages may be displayed. Also in this example, the display processing unit 51a displays, in the information display area, a message such as "The label cannot be recognized" for the books for which the characters of the label could not be recognized. Note that the display processing unit 51a may display other messages for the books for which the characters of the label could not be recognized, or may not display a message for the books for which the characters of the label could not be recognized.

[0076] Note that the display processing unit 51a can display various information on the notification screen of this figure, such as the number of books for which labels are detected in each bookshelf, the classification symbol identified from the detected labels, the number of labels for which characters could be recognized, the number of labels for which characters could not be recognized, or the number of books with an inappropriate storage location.

[0077] <Summary> The book label recognition system according to the present embodiment having the above configuration is such that the server device 3 acquires an image of a bookshelf photographed by the photographing device 1, detects a label attached to a book stored in the bookshelf from the acquired image, and recognizes characters described on the detected label. As a result, it is possible to introduce a device or system that recognizes labels attached to books in an existing library without the need to perform operations such as newly attaching labels or the like having a communication function to a large number of books stored in the library and introducing a bookshelf having a communication function.

[0078] Also, in the book label recognition system according to the present embodiment, the photographing device 1 is provided with a moving mechanism 14 that moves between bookshelves in the library and a photographing unit 12 that photographs the bookshelves. As a result, for example, the photographing device 1 can be moved between bookshelves along a predetermined path by remote control or autonomous control or the like to photograph a plurality of bookshelves provided in the library, so that the work burden on users such as library staff can be reduced.

[0079] Also, in the book label recognition system according to the present embodiment, when the discrimination accuracy of the characters of the label detected from the image is low (when it cannot be discriminated), the server device 3 controls the photographing device 1 to re-photograph the bookshelf. As a result, it can be expected to improve the recognition rate and recognition accuracy of the book label based on the photographed image of the photographing device 1.

[0080] Also, in the book label recognition system according to the present embodiment, the bookshelf DB32b of the server device 3 stores in association the ID of the bookshelf and the "classification symbol" of the book to be stored in this bookshelf. The server device 3 identifies the bookshelf shown in the acquired image, and determines the correctness of the storage location of the book based on the correspondence information stored in the bookshelf DB32b, the identification result of the bookshelf, and the characters recognized from the label of the book, and notifies the determination result. As a result, it can be expected to accurately determine the correctness of the storage location of the book and notify the determination result to the user or the like.

[0081] Also, in the book label recognition system according to the present embodiment, the server device 3 transmits information regarding books stored in inappropriate bookshelves to the terminal device 5, and causes the terminal device 5 that has received this information to display, for example, an image of the bookshelf in which this book is stored and a rectangular frame surrounding the book, thereby notifying the bookshelf in which this book is stored and the storage location in this bookshelf. As a result, a user such as a library staff member can easily grasp the books stored in inappropriate bookshelves.

[0082] Also, in the book label recognition system according to the present embodiment, the server device 3 transmits information regarding books for which a label has been detected but the characters cannot be identified to the terminal device 5, and causes the terminal device 5 that has received this information to notify the books for which the characters on the label cannot be recognized. As a result, a user such as a library staff member can grasp the books for which a label has been detected but the characters cannot be recognized, that is, the books for which the correctness of the storage location cannot be determined.

[0083] Also, in the book label recognition system according to the present embodiment, the server device 3 causes the terminal device 5 to display the image acquired from the imaging device 1 and information such as a rectangular frame indicating the book to be notified stored in the bookshelf shown in this image. As a result, a user such as a library staff member can easily grasp which book on the bookshelf is the book to be notified.

[0084] (Modification Example 1) In the above-described embodiment, the server device 3 acquires the image of the bookshelf photographed by the imaging device 1, and the server device 3 performs processes such as detection and recognition of the label based on the acquired image. However, the present invention is not limited to this. For example, the imaging device 1 may perform some or all of the processes such as detection of the label, recognition of the characters of the label, determination of the correctness of the storage location of the book, and notification of the determination result to the terminal device 5 based on the image of the bookshelf photographed by itself. When the imaging device 1 performs the front part of these processes, the book label recognition system may not include the server device 3.

[0085] (Modification Example 2) In the above-described embodiment, the imaging device 1 is provided with the moving mechanism 14. However, the present invention is not limited to this, and the imaging device 1 may not be provided with the moving mechanism 14. In this case, for example, a librarian or the like in a library can use the imaging device 1 to capture an image of a bookshelf and transmit the captured image from the imaging device 1 to the server device 3. Also, in this case, when the terminal device 5 provided in the library is a portable device such as a smartphone or a tablet terminal having a photographing unit, for example, the imaging device 1 and the terminal device 5 may be the same device. Further, the terminal device 5 may be a wearable device such as AR (Augmented Reality) glasses. Furthermore, the terminal device 5 may be configured to perform processes such as detection and recognition of labels. In this case, the functions of the imaging device 1, the server device 3, and the terminal device 5 can be realized by one device.

[0086] (Modification Example 3) In the above-described embodiment, the library label recognition system is configured to recognize a label and notify the user. However, the present invention is not limited to this. For example, the imaging device 1 may be provided with a book conveyance mechanism, such as an arm unit for gripping books and a storage unit for storing a plurality of books, and may convey books stored in an inappropriate bookshelf to an appropriate bookshelf.

[0087] The embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Reference Numerals

[0088] 1 Imaging device 3 Server device 5 Terminal device 11 Processing unit 12 Photographing unit 13 Storage unit 13a Program 14 Moving mechanism 15 Communication unit 31 Processing unit 31a Shooting control processing unit 31b Image acquisition unit 31c Label detection unit 31d Character recognition unit 31e Bookshelf determination unit 31f Notification processing unit 32 Memory unit 32a Server program 32b Bookshelf DB 33 Communication unit 51 Processing unit 51a Display processing unit 52 Memory unit 52a Program 53 Communication unit 54 Display unit 55 Operation unit

Claims

1. An acquisition unit that acquires an image of a bookshelf; A detection unit that detects a label attached to a book stored in the bookshelf from the acquired image; A recognition unit that extracts an image area of the label detected from the image and recognizes characters described on the label based on the extracted image area; A notification unit that notifies a book with a label to which the recognition unit cannot recognize characters; Comprising: When the recognition unit cannot recognize the characters of the label, the acquisition unit acquires an image of the bookshelf re-photographed, the detection unit detects the label based on the acquired image, and the recognition unit recognizes the characters described on the detected label; The notification unit performs a process of displaying on a display unit an image obtained by splicing a plurality of images obtained by photographing a plurality of bookshelves acquired by the acquisition unit according to the arrangement of the plurality of bookshelves, and information on books with labels to which the recognition unit cannot recognize characters and that are stored in the bookshelves shown in the image; A book label recognition device.

2. A moving mechanism that moves between bookshelves in a library; A photographing unit that photographs the bookshelf; The book label recognition device according to claim 1, comprising:

3. A storage unit that stores in association the bookshelf and identification information of books to be stored in the bookshelf; An identification unit that identifies the bookshelf shown in the acquired image; A determination unit that determines whether the storage location of a book is correct based on the information stored in the storage unit, the identification result of the identification unit, and the characters recognized by the recognition unit; A notification unit that notifies the determination result of the determination unit; The book label recognition device according to claim 1 or claim 2, comprising:

4. The notification unit notifies, for a book stored in an inappropriate bookshelf, the bookshelf in which the book is stored and the storage location in the bookshelf; The book label recognition device according to claim 3.

5. A photographing device having a moving mechanism that moves between bookshelves in a library and a photographing unit that photographs the bookshelf; An acquisition unit that acquires an image photographed by the photographing device, a detection unit that detects a label attached to a book stored in the bookshelf from the acquired image, a recognition unit that extracts an image area of the label detected from the image and recognizes characters described on the label based on the extracted image area, and a notification unit that notifies a book with a label to which the recognition unit cannot recognize characters; Comprising: When the recognition unit cannot recognize the characters on the label, the imaging device re-images the bookshelf by the imaging unit. When the recognition unit cannot recognize the characters on the label, the information processing device acquires an image obtained by the acquisition unit re-imaging the bookshelf, the detection unit detects a label based on the acquired image, and the recognition unit recognizes the characters described on the detected label. The notification unit performs a process of displaying on a display unit an image obtained by splicing together a plurality of images obtained by respectively photographing a plurality of bookshelves according to the arrangement of the plurality of bookshelves, and information indicating a book with a label to which the recognition unit cannot recognize characters attached thereto on the bookshelf shown in the image. Book label recognition system.

6. Acquire an image of a bookshelf. Detect a label attached to a book stored on the bookshelf from the acquired image. Extract the image area of the label detected from the image, and recognize the characters described on the label based on the extracted image area. When the characters on the label cannot be recognized, acquire an image obtained by re-imaging the bookshelf, detect a label based on the acquired image, and recognize the characters described on the detected label. Cause an information processing device to perform a process of displaying on a display unit an image obtained by splicing together a plurality of images obtained by respectively photographing a plurality of bookshelves according to the arrangement of the plurality of bookshelves, and information indicating a book with a label to which characters cannot be recognized attached thereto on the bookshelf shown in the image, and notify a book with a label to which characters cannot be recognized attached thereto. Book label recognition method.

7. On a computer, Acquire an image of a bookshelf. Detect a label attached to a book stored on the bookshelf from the acquired image. Extract the image area of the label detected from the image, and recognize the characters described on the label based on the extracted image area. When the characters on the label cannot be recognized, acquire an image obtained by re-imaging the bookshelf, detect a label based on the acquired image, and recognize the characters described on the detected label. Cause an information processing device to perform a process of displaying on a display unit an image obtained by splicing together a plurality of images obtained by respectively photographing a plurality of bookshelves according to the arrangement of the plurality of bookshelves, and information indicating a book with a label to which characters cannot be recognized attached thereto on the bookshelf shown in the image, and notify a book with a label to which characters cannot be recognized attached thereto. A computer program that causes processing to be executed.

Images