Library book intelligent checking method, equipment and device based on visual technology

Abstract

The invention relates to the technical field of image feature recognition, in particular to a library book intelligent checking method, equipment and device based on a visual technology. According to the method, a spine region is screened through the form and depth information of a straight line division region in a spine image, and the irregular placement degree of the spine region is analyzed through the boundary depth change and inclination condition of the spine region; and marking a shielding analysis area, and analyzing the shielding degree and performing shielding marking for checking by combining the irregular placement degree and depth conditions of the spine areas on the two sides of the marked shielding analysis area. The situation that the books are shielded is analyzed in combination with the depth form change, the comprehensive and accurate degree of book information recognition is improved, and the book checking efficiency is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of image feature recognition technology, and more specifically to a method, equipment, and apparatus for intelligent inventory of library books based on visual technology. Background Technology

[0002] Book inventory is the process of confirming that all books in a library are in their correct shelf positions, ensuring the effective use of the library's book retrieval functions. Manual inventory is time-consuming, labor-intensive, inefficient, and prone to errors. Currently, computer vision technology is commonly used. Wheeled robots equipped with cameras capture images of the bookshelves, identify the spine information of the books, and perform the inventory. The steps are roughly as follows: capture images, segment the spine regions of the books within the images, identify the text information in the spine regions, match it with known book information, and determine if the book's position is correct.

[0003] Existing technologies often use LSD line detection algorithms to segment and identify the spine region of books. This requires a relatively complete frontal image of the spine for book inventory recognition. However, during the robot's imaging process, some books are tilted or obscured due to the random placement by readers. This obstruction between books affects information acquisition during the book inventory process, making it impossible to fully reveal book information, thus affecting the efficiency of book inventory and making the inventory work incomplete. Summary of the Invention

[0004] To address the technical problem in existing technologies where book occlusion affects information acquisition during machine book inventory, preventing complete information capture and thus impacting inventory efficiency, this invention aims to provide a visual technology-based intelligent book inventory method, device, and apparatus for libraries. The specific technical solution adopted is as follows: In a first aspect, the present invention provides a method for intelligent inventory of library books based on visual technology, the method comprising: The bookshelf spine image and the depth value of each pixel are obtained by using a depth camera; the suspected spine areas are determined by using a line segment detection algorithm to filter out straight lines and the distribution of depth values ​​in the bookshelf spine image. By analyzing the morphological distribution and depth uniformity of suspected spine regions, potential spine indicators for these regions are obtained; spine regions are then selected from the suspected spine regions based on these potential spine indicators. Based on the depth changes of each spine region at different boundary sections and the tilt of the bookshelf, combined with possible spine indices, the irregularity of each spine region is obtained; the occlusion analysis area is marked by the regional distribution between adjacent spine regions; and the occlusion degree index of the occlusion analysis area is obtained based on the depth distribution of the occlusion analysis area and the irregularity and area of ​​the spine regions on both sides. Occlusion is marked and inventoried based on occlusion level indicators.

[0005] Furthermore, the method for obtaining the suspected spine region includes: The LSD line detection algorithm is used on the bookshelf spine image to obtain the initial boundary edge in each bookshelf spine image; the angle between each initial boundary edge and the horizontal line is calculated, and the initial boundary edges with an angle smaller than the preset tilt angle are screened out to obtain the region boundary. For any two region boundaries, the angle between the two region boundaries is taken as the degree of intersection; when there are no other region boundaries between the two region boundaries with an angle less than or equal to the degree of intersection, the two region boundaries are taken as the spine boundaries. The mean depth value of all pixels in the bookshelf spine image is used as the segmentation depth value; among the pixels with a depth value greater than the segmentation depth value, the mode of the depth value is used as the non-book depth value; the area where the pixels corresponding to the non-book depth value exist is used as the non-book area; outside the non-book area, the area between every two adjacent spine boundaries composed of pixels with a depth value less than the non-book depth value is used as each suspected spine area.

[0006] Furthermore, the method for obtaining the possible indicators of the book spine includes: The cosine of the intersection degree between the spine boundaries of each suspected spine region is used as the boundary parallelism index of each suspected spine region; the minimum bounding rectangle of each suspected spine region is obtained, and the ratio of the area of ​​the minimum bounding rectangle to the area of ​​the suspected spine region is used as the shape regularity index of each suspected spine region; the ratio of the boundary parallelism index to the shape regularity index of each suspected spine region is used as the morphological feature value of each suspected spine region. The depth feature value of each suspected spine region is obtained by negatively correlating the depth range of all pixels in each suspected spine region. By combining the morphological and depth characteristics of each suspected spine region, the possible spine indices for each suspected spine region are obtained.

[0007] Furthermore, the method for obtaining the degree of irregular placement includes: For any spine region, obtain the minimum bounding rectangle of the spine region, and take the two long sides of the minimum bounding rectangle and the short side closest to the upper boundary of the bookshelf spine image as the analysis edges; at the center point of each analysis edge of the minimum bounding rectangle, sort the depth values ​​of consecutive pixels along the gradient direction of the depth value to obtain the gradient reduction sequence of each analysis edge; in the gradient reduction sequence, the depth value of each index is greater than the depth value of the next index; Obtain the total number of indices in the gradient reduction sequence; take the largest total number between two long sides of the analysis edge as the lateral rotation degree; take the total number of short sides of the analysis edge as the longitudinal rotation degree; calculate the product of the lateral rotation degree and the longitudinal rotation degree as the rotation deviation index of the spine region; The angle between the long side of the minimum bounding rectangle of the spine region and the vertical direction is normalized and used as the tilt deviation index of the spine region; the possible spine indices of the spine region are negatively correlated to obtain the characteristic influence index of the spine region. By combining the characteristic influence index, rotation deviation index, and tilt deviation index of the spine region, the irregularity of the spine region is obtained.

[0008] Furthermore, the method for marking the occlusion analysis area includes: In the suspected spine region, the non-spine region between each two adjacent spine regions is taken as each occlusion analysis region; when the non-spine region in the suspected spine region is the first suspected spine region on the left and right sides, the corresponding non-spine region is taken as the occlusion analysis region.

[0009] Furthermore, the method for obtaining the occlusion level index includes: For any occlusion analysis area, calculate the product of the area of ​​the spine region on each side of the occlusion analysis area and the degree of irregular placement, and use it as the local occlusion probability on each side of the occlusion analysis area; sum the local occlusion probability values ​​on both sides of the occlusion analysis area to obtain the occlusion probability index of the occlusion analysis area. By performing a negative correlation mapping on the coefficient of variation of the depth values ​​of all pixels in the occlusion analysis area, possible indicators of the spine features of the occlusion analysis area can be obtained. By combining the occlusion probability index and the spine feature probability index of the occlusion analysis area, the occlusion degree index of the occlusion analysis area is obtained.

[0010] Furthermore, the step of marking and inventorying occlusions based on occlusion degree indicators includes: Areas with occlusion levels exceeding a preset occlusion threshold are designated as suspected occlusion areas. If book information cannot be identified during inventory checks of suspected occlusion areas, an auxiliary prompt is provided and the inventory is re-checked.

[0011] Furthermore, the method for obtaining the spine region includes: The suspected spine regions where the spine index is greater than the preset spine threshold are defined as spine regions.

[0012] Secondly, the present invention also provides a smart inventory device for library books based on visual technology, comprising: The acquisition module is used to acquire images of the bookshelf spines and the depth value of each pixel position through a depth camera; The spine region analysis module is used to identify suspected spine regions by analyzing the straight lines and depth value distribution in the spine image of the bookshelf using a straight line segment detection algorithm; it obtains potential spine indicators for the suspected spine regions by analyzing the shape distribution and depth uniformity of the suspected spine regions; and it filters spine regions from the suspected spine regions based on the potential spine indicators. The occlusion analysis module is used to obtain the irregularity of each spine region based on the depth value changes at different boundary parts of each spine region and the tilt of the bookshelf, combined with possible spine indicators; to mark the occlusion analysis area by the regional distribution between adjacent spine regions; and to obtain the occlusion degree index of the occlusion analysis area based on the depth distribution of the occlusion analysis area and the irregularity and area of ​​the spine regions on both sides. The inventory analysis module is used to mark occlusions and conduct inventory checks based on occlusion level indicators.

[0013] Thirdly, the present invention also provides a library book intelligent inventory device based on vision technology, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the library book intelligent inventory method based on vision technology described above.

[0014] The present invention has the following beneficial effects: This invention uses the shape and depth information of linearly divided regions in a book spine image for initial screening of the spine area. Then, by analyzing the boundary depth changes and tilt of the spine area, the degree of irregularity in placement can be determined. Deviations from an ideal flat placement provide a data foundation for subsequent analysis of occlusion levels. Occluded portions that may be missed are marked. Combining irregular placement and depth information, the degree of occlusion is analyzed and marked for inventory checks, reducing omissions due to occlusion. This invention combines depth and shape changes to analyze occlusion between books, improving the comprehensiveness and accuracy of book information identification and ensuring the efficiency of book inventory work. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1A flowchart illustrating a visual technology-based intelligent inventory method for library books, provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of a bookshelf spine with obstruction provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of a wheeled robot bookshelf provided in one embodiment of the present invention; Figure 4 A schematic diagram of a bookshelf spine image provided in one embodiment of the present invention; Figure 5 This is a side view of the depth of a bookshelf provided in one embodiment of the present invention; Figure 6 This is a front view of a bookshelf area provided in one embodiment of the present invention; Figure 7 A schematic diagram of a book with occlusion caused by its size, provided as an embodiment of the present invention; Figure 8 This is a schematic diagram of an irregularly arranged book arrangement according to an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a visual technology-based intelligent inventory method, device, and apparatus for library books proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a visual technology-based intelligent inventory method, equipment, and apparatus for library books provided by the present invention.

[0020] When identifying and segmenting the spine region of books in an image, large books may obscure smaller books, leading to missed book data or incorrect spine region information. This results in incomplete book inventory work, with some books not being placed in the designated location, causing the search function to fail.

[0021] Example 1: Incorporating image depth information and combining it with the position of lines in the image to determine occlusion and visibility helps in the smooth progress of inventory checks. Please refer to [link / reference needed]. Figure 1 The diagram illustrates a flowchart of a visual technology-based intelligent inventory method for library books according to an embodiment of the present invention. The method includes the following steps: S1: Obtain the spine image of the bookshelf and the depth value of each pixel position using a depth camera; use the straight line segment detection algorithm to filter out the straight lines and the distribution of depth values ​​in the spine image of the bookshelf to determine the suspected spine area.

[0022] The obscured portion of a book, relative to other unobscured books, is often located between the spines of the books. Furthermore, within the bookshelf area of ​​the image, there is a certain depth difference between the book area and the non-book area. The presence of an obscured book is determined based on its location, depth, and spine characteristics. Please refer to [link to image]. Figure 2 The illustration shows a schematic diagram of a bookshelf spine with obstruction, according to an embodiment of the present invention.

[0023] First, during the inventory process, images of the books are acquired. A wheeled robot equipped with a depth camera photographs each section of the bookshelves in the library, and the images are preprocessed to obtain images of the book spines. Image preprocessing can specifically include image denoising and enhancement, etc., and is a well-known technique in the field, so it is not limited thereto. A depth camera is a device that can capture the depth information of each pixel in an image, that is, the distance between the object and the camera; the value of each pixel represents the distance from the camera to the object. Please refer to [link to relevant documentation]. Figure 3 This illustration shows a schematic diagram of a wheeled robot bookshelf for taking pictures, according to an embodiment of the present invention. Please refer to [link / reference]. Figure 4 The illustration shows a schematic diagram of a bookshelf spine provided in one embodiment of the present invention.

[0024] Because the spine edge of a book exhibits a clearly distinct straight line compared to its surroundings, an LSD straight line detection algorithm can be used to assist in determining potential occlusion between spine regions. This algorithm detects straight line segments in the resulting image that may represent the spine edge for preliminary region identification. In this embodiment of the invention, the method for obtaining suspected book regions includes: First, the LSD line detection algorithm is applied to the bookshelf spine images to obtain the initial boundary edges in each spine image. The angle between each initial boundary edge and the horizontal line is calculated, and initial boundary edges with angles smaller than a preset tilt angle are filtered out to obtain the region boundaries. Since books easily occlude short edges when occlusion occurs, the analysis mainly focuses on the straight lines in the vertical direction. In this embodiment, the preset tilt angle is set to 45 degrees; angles lower than this are considered not to belong to the vertical boundary of the book and are not included in the occlusion analysis.

[0025] Furthermore, considering that the spines of the books are ideally parallel, for any two region boundaries, the angle between the two region boundaries is taken as the degree of intersection. If the angle between the region boundaries is 0, it means that the region boundaries are parallel to each other, which is very likely to be two points on the spine.

[0026] Therefore, when there are no other regional boundaries between the two regional boundaries, and the angle between the two regional boundaries is less than or equal to the degree of intersection, it indicates that the two regional boundaries are adjacent and parallel to each other, which can characterize the edge of the book spine, and the two regional boundaries are taken as the spine boundary.

[0027] The mean depth value of all pixels in the bookshelf spine image is used as the segmentation depth value. Among pixels with a depth value greater than the segmentation depth value, the mode of the depth value is taken as the non-book depth value. In cases where the depth value is large, the areas with high depth values ​​and a large distribution are the areas without books. Please refer to [link / reference]. Figure 5 This shows a side view of the depth of a bookshelf provided in an embodiment of the present invention.

[0028] The region containing pixels whose depth values ​​are not corresponding to those of books is designated as the non-book region. The remaining regions, excluding the non-book regions, are considered areas containing books. Each book's spine region is divided by its spine edges. The region between each pair of adjacent spine boundaries, consisting of pixels with depth values ​​less than those of non-books, is designated as each suspected spine region. (See also...) Figure 6 The diagram shows a front view of a bookshelf area provided in an embodiment of the present invention.

[0029] S2: Obtain potential spine indicators for the suspected spine regions by analyzing their morphological distribution and depth uniformity; then, select spine regions from the suspected spine regions based on these potential spine indicators.

[0030] Since books are not always arranged in an ideal, compact, and vertical manner, and due to the wear and tear or tilting of books, the suspected spine area may be a gap between adjacent books that cannot be identified as non-book parts due to the unfolding of the pages, or it may be a part of the book that is being obscured. Therefore, it is necessary to further analyze the possible spines by examining their shape and depth to determine the spine area for subsequent analysis of obscuration.

[0031] Considering that the shape of the spine is close to a regular rectangle and the depth of the spine plane is uniform, preferably, in this embodiment of the invention, the method for obtaining the possible indicators of the spine includes: First, the cosine value of the intersection degree between the spine boundaries of each suspected spine region is used as the boundary parallelism index for each suspected spine region. The larger the cosine value of the intersection degree, the more likely the spine boundaries are to be ideally parallel, and the more the corresponding region conforms to the characteristics of a spine region. Next, the minimum bounding rectangle of each suspected spine region is obtained. The ratio of the area of ​​the minimum bounding rectangle to the area of ​​the suspected spine region is used as the shape regularity index for each suspected spine region. The area can be represented by the total number of pixels. The closer the region is to the bounding rectangle, i.e., the smaller the shape regularity index, the more the region shape approximates a rectangle, and the more it conforms to the morphological characteristics of the region.

[0032] Therefore, the ratio of the boundary parallelism index to the shape regularity index of each suspected spine region is used as the morphological characteristic value of each suspected spine region. The larger the boundary parallelism index and the smaller the shape regularity index, the more the suspected spine region conforms to the characteristics of a spine region in terms of shape. It should be noted that obtaining the minimum bounding rectangle is a well-known technique familiar to those skilled in the art, and will not be elaborated here.

[0033] Furthermore, the depth value range of all pixels in each suspected spine region is negatively correlated to obtain the depth feature value of each suspected spine region. The smaller the range, the more uniform the depth in the region, and the higher the probability that it represents a spine plane. Therefore, the region better matches the characteristics of a spine region. In this embodiment of the invention, negative correlation means that the dependent variable, i.e., the depth feature value, increases as the independent variable, i.e., the depth value range, decreases. The negative correlation mapping adopts a negative exponential form. The negative correlation mapping method is a well-known technique in the art, and the choice of the negative correlation function is not elaborated or limited here.

[0034] Finally, by combining the morphological and depth feature values ​​of each suspected spine region, a potential spine index for each suspected spine region is obtained. In this embodiment of the invention, the product of the morphological and depth feature values ​​is used as the potential spine index for each suspected spine region. The larger the morphological and depth feature values, the more significant the characteristics of the spine region, and the higher the probability that the suspected spine region is a spine part.

[0035] Therefore, spine regions can be further filtered from suspected spine regions using a spine probability index. In this embodiment of the invention, suspected spine regions with a spine probability index greater than a preset spine threshold are designated as spine regions. The preset spine threshold is set to 0.8, and the specific value can be adjusted by the implementer according to the specific implementation scenario; no limitation is imposed here.

[0036] S3: Based on the depth changes of each spine region at different boundary sections and the tilt of the bookshelf, combined with possible spine indicators, obtain the irregularity of each spine region; mark the occlusion analysis area by the regional distribution between adjacent spine regions; obtain the occlusion degree index of the occlusion analysis area based on the depth distribution of the occlusion analysis area and the irregularity and area of ​​the spine regions on both sides.

[0037] Improper placement of returned books may cause larger books to partially obscure smaller ones, or the spine of a book may be tilted and obscured due to insufficient depth. The information of the obscured book may not be properly identified, leading to it being missed during inventory checks. This type of obscuration occurs between the spine areas of the books, possibly within detectable spine regions. Although some spine characteristics are lost, the obscured book still retains depth information and is located between other books. Please refer to [link to relevant documentation]. Figure 7 The illustration shows a schematic diagram of a book with a size that causes obstruction, provided by an embodiment of the present invention. The front view shows the possible obstruction of small books by books on both sides when viewing the bookshelf from the front, and the top view shows the possible obstruction of small books when viewing the bookshelf from above.

[0038] Due to irregular placement, the tight fit between the spines may be compromised, causing the book to tilt or rotate. Please refer to [link / reference needed]. Figure 8 This illustration shows a schematic diagram of an irregular arrangement of books according to an embodiment of the present invention. The books are tilted at an angle to the bookshelf. The rotation of the books causes a change in the planar depth in the image. Therefore, the irregular arrangement can be analyzed from the perspectives of depth change and tilt angle.

[0039] Preferably, in this embodiment of the invention, the method for obtaining the degree of irregular placement includes: First, we analyze the possible depth changes. For any spine region, we obtain the smallest bounding rectangle of the spine region. We take the two long sides of the smallest bounding rectangle and the short side closest to the upper boundary of the bookshelf spine image as the analysis sides. That is, when analyzing the possible rotation changes, we perform rotation analysis through the two sides and the upper edge of the long side of the spine.

[0040] At the center point of each analysis edge of the minimum bounding rectangle, the depth values ​​of consecutive pixels are sorted along the gradient direction of the depth value to obtain the gradient decrease sequence of each analysis edge. When the book undergoes rotation and occlusion, the rotation will cause the left or right side of the spine to protrude or the top to occlude. The depth change along the edge shows a trend of first decreasing and then increasing. When making a judgment, the degree of decrease of the trend of each analysis edge can reflect the possibility of rotation occlusion.

[0041] Therefore, by analyzing the number of consecutive pixels at the edge center whose depth decreases, the degree of the decreasing trend can be characterized. In the process of sorting the gradient decreasing sequence, the depth value of each index is greater than the depth value of the next index. The sorting of depth values ​​is only carried out when there are consecutive pixels that meet the condition. When there are no pixels that meet the condition, the sorting is stopped to obtain the sequence.

[0042] Further, the total number of indices in the gradient reduction sequence is obtained; the magnitude of this total number reflects the degree of rotation trend. The maximum total number between the two longer sides of the analysis edge is taken as the lateral rotation degree, reflecting the degree of lateral rotational occlusion of the spine. Then, the total number of the shorter sides of the analysis edge is taken as the longitudinal rotation degree, reflecting the degree of longitudinal rotational occlusion. By calculating the product of the lateral and longitudinal rotation degrees, a rotational deviation index for the spine region is obtained. Combining the deviations in both directions, the degree of irregularity in book rotation is determined.

[0043] Furthermore, the angle between the longest side of the smallest bounding rectangle of the spine region and the vertical direction is normalized and used as an indicator of the tilt deviation of the spine region. Since the spine of the books is photographed from the front of the bookshelf, that is, the four sides of the image are parallel to the sides of the bookshelf, the angle between the spine region and the vertical direction reflects the degree of tilt. The larger the angle, the higher the degree of tilt and the higher the possibility of irregular placement.

[0044] Further, a negative correlation mapping was performed on the possible spine indices of the spine region to obtain the characteristic influence index of the spine region. Due to irregular placement, the spine region will change. The spine region is relatively poor in feature representation compared to the ideal and regular flat spine region. Therefore, the lower the possible spine index, the higher the degree of irregular placement may occur.

[0045] Finally, by combining the characteristic influence index, rotation deviation index, and tilt deviation index of the spine region, the irregularity of the spine region is obtained. In this embodiment of the invention, the characteristic influence index, rotation deviation index, and tilt deviation index of the spine region are multiplied to obtain the irregularity of the spine region. The larger the characteristic influence index, rotation deviation index, and tilt deviation index, the greater the degree of flatness deviation of the spine region and the more significant the irregularity.

[0046] Since the occluded portion may be occluded from both sides, each area where occlusion may occur is first marked. In this embodiment of the invention, the method for marking the occlusion analysis area includes: In the suspected spine region, the non-spine area between each two adjacent spine regions is taken as each occlusion analysis area. When the non-spine area in the suspected spine region is the first suspected spine area on the left and right sides, it indicates that there may also be occlusion in the first non-spine area on both sides of the bookshelf. The corresponding non-spine area is taken as the occlusion analysis area.

[0047] An analysis of the degree of occlusion is performed on areas that may be obstructed. This is achieved by considering the irregular arrangement and size of the spine regions on both sides, combined with the potential depth characteristics of the spines, to obtain an occlusion degree index. Preferably, in this embodiment of the invention, the method for obtaining the occlusion degree index includes: First, for any occlusion analysis area, calculate the product of the area size of the spine region on each side of the occlusion analysis area and the degree of irregular placement. This product is used as the local occlusion probability on each side of the occlusion analysis area. For any spine region on any side, the higher the degree of irregular placement and the larger the area size, the greater the probability of placement occlusion on that side and the higher the degree of area coverage after occlusion, indicating a more severe occlusion situation.

[0048] By considering the possible occlusion on both sides, the sum of the local occlusion probabilities on both sides of the occlusion analysis area is used to obtain the occlusion probability index of the occlusion analysis area.

[0049] Furthermore, a negative correlation mapping is performed on the coefficients of variation of the depth values ​​of all pixels in the occlusion analysis area to obtain possible indicators of the spine features of the occlusion analysis area. The coefficient of variation reflects the dispersion of depth values ​​in the occlusion analysis area. If it is only the gap between the spines, the depth data aggregation is worse; conversely, when the spine is likely occluded, the depth data is more concentrated. It should be noted that the coefficient of variation is a well-known technique familiar to those skilled in the art. The coefficient of variation represents the ratio of the standard deviation to the mean of the data. The smaller the coefficient of variation, the more concentrated the data and the more stable the mean. This will not be elaborated further here.

[0050] Finally, by combining the occlusion probability index and the spine feature probability index of the occlusion analysis area, an occlusion degree index of the occlusion analysis area is obtained. In this embodiment of the invention, the product of the occlusion probability index and the spine feature probability index of the occlusion analysis area is used as the occlusion degree index of the occlusion analysis area. The larger the occlusion probability index and the spine feature probability index are, that is, the larger the occlusion degree index is, the more likely the occlusion analysis area is a severely occluded spine area.

[0051] S4: Mark and inventory occlusions based on occlusion level indicators.

[0052] Severely occluded areas are marked using an occlusion level index. When problems arise during the inventory of marked books, timely reminders and maintenance are facilitated, aiding in the smooth progress of the inventory. During the inventory process, optical character recognition (OCR) technology is used to extract text information from the spines of the books and compare it with the book location information database. The obtained book titles are matched with the information in the database, and the position of the matched book in the image is compared with its correct position. Books with discrepancies are output, along with their names, incorrect positions, and correct positions, to assist in the completion of the inventory process.

[0053] For obscured areas, inventory information is easily overlooked. Therefore, in this embodiment of the invention, the obscuration analysis area with an obscuration degree index greater than a preset obscuration threshold is recorded as a suspected obscured area, indicating the presence of obscured book spines. When book information cannot be identified during inventory of a suspected obscured area, it indicates that there is obstruction due to placement rather than borrowing, requiring auxiliary prompts. After the books are reorganized, the inventory can be re-counted to address the obstruction, improving the comprehensiveness and accuracy of the inventory.

[0054] In summary, this invention uses the shape and depth information of the linearly divided regions in the spine image to initially screen the spine area. Then, by analyzing the boundary depth changes and tilt of the spine area, the degree of irregularity in placement can be determined. The deviation from the ideal flat placement provides a data foundation for subsequent analysis of occlusion. Occluded parts that may be missed are marked. Combining the irregularity and depth of placement, the degree of occlusion is analyzed and marked for inventory, reducing omissions due to occlusion. This invention combines depth and shape changes to analyze occlusion between books, improving the comprehensiveness and accuracy of book information identification and ensuring the efficiency of book inventory work.

[0055] Example 2: The present invention also provides a smart inventory device for library books based on vision technology, comprising: The acquisition module is used to acquire images of the bookshelf spines and the depth value of each pixel position through a depth camera; The spine region analysis module is used to identify suspected spine regions by analyzing the straight lines and depth value distribution in the spine image of the bookshelf using a straight line segment detection algorithm; it obtains potential spine indicators for the suspected spine regions by analyzing the shape distribution and depth uniformity of the suspected spine regions; and it filters spine regions from the suspected spine regions based on the potential spine indicators. The occlusion analysis module is used to obtain the irregularity of each spine region based on the depth value changes at different boundary parts of each spine region and the tilt of the bookshelf, combined with possible spine indicators; to mark the occlusion analysis area by the regional distribution between adjacent spine regions; and to obtain the occlusion degree index of the occlusion analysis area based on the depth distribution of the occlusion analysis area and the irregularity and area of ​​the spine regions on both sides. The inventory analysis module is used to mark occlusions and conduct inventory checks based on occlusion level indicators.

[0056] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional module units as needed, that is, the internal structure of the device can be divided into different functional module units to complete all or part of the functions described above. Since the specific implementation process of the intelligent library book inventory device based on vision technology in this embodiment is the same as the specific implementation process of the intelligent library book inventory method based on vision technology described above, it will not be described in detail here.

[0057] Example 3: The present invention also provides a library book intelligent inventory device based on vision technology, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the library book intelligent inventory method based on vision technology described above.

[0058] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0059] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for intelligent inventory management of library books based on visual technology, characterized in that, The method includes: The bookshelf spine image and the depth value of each pixel are obtained by using a depth camera; the suspected spine areas are determined by using a line segment detection algorithm to filter out straight lines and the distribution of depth values ​​in the bookshelf spine image. By analyzing the morphological distribution and depth uniformity of suspected spine regions, potential spine indicators for these regions are obtained; spine regions are then selected from the suspected spine regions based on these potential spine indicators. Based on the depth changes of each spine region at different boundary sections and the tilt of the bookshelf, combined with possible spine indices, the irregularity of each spine region is obtained; the occlusion analysis area is marked by the regional distribution between adjacent spine regions; and the occlusion degree index of the occlusion analysis area is obtained based on the depth distribution of the occlusion analysis area and the irregularity and area of ​​the spine regions on both sides. Occlusion is marked and inventoried based on occlusion level indicators.

2. The intelligent inventory method for library books based on visual technology according to claim 1, characterized in that, The method for obtaining the suspected spine region includes: The LSD line detection algorithm is used on the bookshelf spine image to obtain the initial boundary edge in each bookshelf spine image; the angle between each initial boundary edge and the horizontal line is calculated, and the initial boundary edges with an angle smaller than the preset tilt angle are screened out to obtain the region boundary. For any two region boundaries, the angle between the two region boundaries is taken as the degree of intersection; when there are no other region boundaries between the two region boundaries and the two region boundaries respectively with an angle less than or equal to the degree of intersection, the two region boundaries are taken as the spine boundaries. The mean depth value of all pixels in the bookshelf spine image is used as the segmentation depth value; among the pixels with a depth value greater than the segmentation depth value, the mode of the depth value is used as the non-book depth value; the area where the pixels corresponding to the non-book depth value exist is used as the non-book area; outside the non-book area, the area between every two adjacent spine boundaries composed of pixels with a depth value less than the non-book depth value is used as each suspected spine area.

3. The intelligent inventory method for library books based on visual technology according to claim 2, characterized in that, The methods for obtaining the possible indicators of the book spine include: The cosine of the intersection degree between the spine boundaries of each suspected spine region is used as the boundary parallelism index of each suspected spine region; the minimum bounding rectangle of each suspected spine region is obtained, and the ratio of the area of ​​the minimum bounding rectangle to the area of ​​the suspected spine region is used as the shape regularity index of each suspected spine region; the ratio of the boundary parallelism index to the shape regularity index of each suspected spine region is used as the morphological feature value of each suspected spine region. The depth feature value of each suspected spine region is obtained by negatively correlating the depth range of all pixels in each suspected spine region. By combining the morphological and depth characteristics of each suspected spine region, the possible spine indices for each suspected spine region are obtained.

4. The intelligent inventory method for library books based on visual technology according to claim 1, characterized in that, The method for obtaining the irregularity degree includes: For any spine region, obtain the minimum bounding rectangle of the spine region, and take the two long sides of the minimum bounding rectangle and the short side closest to the upper boundary of the bookshelf spine image as the analysis edges; at the center point of each analysis edge of the minimum bounding rectangle, sort the depth values ​​of consecutive pixels along the gradient direction of the depth value to obtain the gradient reduction sequence of each analysis edge; in the gradient reduction sequence, the depth value of each index is greater than the depth value of the next index; Obtain the total number of indices in the gradient reduction sequence; take the largest total number between two long sides of the analysis edge as the lateral rotation degree; take the total number of short sides of the analysis edge as the longitudinal rotation degree; calculate the product of the lateral rotation degree and the longitudinal rotation degree as the rotation deviation index of the spine region; The angle between the long side of the minimum bounding rectangle of the spine region and the vertical direction is normalized and used as the tilt deviation index of the spine region; the possible spine indices of the spine region are negatively correlated to obtain the characteristic influence index of the spine region. By combining the characteristic influence index, rotation deviation index, and tilt deviation index of the spine region, the irregularity of the spine region is obtained.

5. The intelligent inventory method for library books based on visual technology according to claim 1, characterized in that, The method for marking the occlusion analysis area includes: In the suspected spine region, the non-spine region between each two adjacent spine regions is taken as each occlusion analysis region; when the non-spine region in the suspected spine region is the first suspected spine region on the left and right sides, the corresponding non-spine region is taken as the occlusion analysis region.

6. The intelligent inventory method for library books based on visual technology according to claim 5, characterized in that, The method for obtaining the occlusion level index includes: For any occlusion analysis area, calculate the product of the area of ​​the spine region on each side of the occlusion analysis area and the degree of irregular placement, and use it as the local occlusion probability on each side of the occlusion analysis area; sum the local occlusion probability values ​​on both sides of the occlusion analysis area to obtain the occlusion probability index of the occlusion analysis area. By performing a negative correlation mapping on the coefficient of variation of the depth values ​​of all pixels in the occlusion analysis area, possible indicators of the spine features of the occlusion analysis area can be obtained. By combining the occlusion probability index and the spine feature probability index of the occlusion analysis area, the occlusion degree index of the occlusion analysis area is obtained.

7. The intelligent inventory method for library books based on visual technology according to claim 1, characterized in that, The process of marking and inventorying occlusions based on occlusion degree indicators includes: Areas with occlusion levels exceeding a preset occlusion threshold are designated as suspected occlusion areas. If book information cannot be identified during inventory checks of suspected occlusion areas, an auxiliary prompt is provided and the inventory is re-checked.

8. The intelligent inventory method for library books based on visual technology according to claim 1, characterized in that, The method for obtaining the spine region includes: The suspected spine regions where the spine index is greater than the preset spine threshold are defined as spine regions.

9. A visual technology-based intelligent inventory device for library books, used to implement the visual technology-based intelligent inventory method for library books as described in claim 1, characterized in that, include: The acquisition module is used to acquire images of the bookshelf spines and the depth value of each pixel position through a depth camera; The spine region analysis module is used to identify suspected spine regions by analyzing the straight lines and depth value distribution in the spine image of the bookshelf using a straight line segment detection algorithm; it obtains potential spine indicators for the suspected spine regions by analyzing the shape distribution and depth uniformity of the suspected spine regions; and it filters spine regions from the suspected spine regions based on the potential spine indicators. The occlusion analysis module is used to obtain the irregularity of each spine region based on the depth value changes at different boundary parts of each spine region and the tilt of the bookshelf, combined with possible spine indicators; to mark the occlusion analysis area by the regional distribution between adjacent spine regions; and to obtain the occlusion degree index of the occlusion analysis area based on the depth distribution of the occlusion analysis area and the irregularity and area of ​​the spine regions on both sides. The inventory analysis module is used to mark occlusions and conduct inventory checks based on occlusion level indicators.

10. A library book intelligent inventory device based on vision technology, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a smart inventory of library books based on a vision technology method as described in any one of claims 1 to 8.

Images