A book inventory system and robot based on machine vision

Abstract

The present invention provides a book inventory robot based on machine vision, which belongs to the field of machine vision technology and includes: a region division module, which is used to control a visual camera to move to a designated position of a corresponding divided region to obtain a region image; a set construction module, which is used to lock the surrounding areas of each divided region, and construct a surrounding overlapping set in combination with the determined radiation range of the divided region; a reconstruction module, which is used to perform region reconstruction on the region image of the corresponding divided region according to the surrounding overlapping set to obtain a first image; a boundary recognition module, which is used to splice all first images, and perform book boundary recognition on the spliced ​​image, and mark closed-loop boundaries and non-closed-loop boundaries; a boundary extension processing module, which is used to perform boundary extension processing on the corresponding non-closed-loop boundaries according to the closed-loop boundaries; and a text recognition module, which is used to recognize and summarize the texts within all closed-loop boundaries to obtain book summary information. The process saves time and effort and greatly improves the efficiency of inventory.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a book inventory system and a robot based on machine vision. Background Art

[0002] Book inventory is a basic task in library collection management. By counting the actual number of books, we can understand the flow of books, including whether they are on the shelf, on loan, lost or damaged. Inventory can not only help us understand the construction of library collection resources, but also find abnormalities in the borrowed bibliographic data.

[0003] The common inventory is that the library managers check and enter the books one by one. However, due to the large number of books, this process is time-consuming and labor-intensive, resulting in extremely low inventory efficiency.

[0004] Therefore, the present invention proposes a book inventory system and a robot based on machine vision. Summary of the invention

[0005] The present invention provides a book inventory system and robot based on machine vision, which are used to obtain regional images by photographing book storage racks through visual cameras, and then process the images and automatically identify book boundaries, which saves time and labor and greatly improves inventory efficiency.

[0006] The present invention provides a book inventory system based on machine vision, comprising:

[0007] A region division module is used to divide the inventory area into regions based on the arrangement of the book storage shelves in the inventory area and according to the shooting width of the visual camera, and control the visual camera to move to a designated position corresponding to the divided area to obtain a regional image;

[0008] A set construction module is used to lock the surrounding area of ​​each divided area, and to construct a surrounding overlapping set of the corresponding divided area in combination with the designated position of each divided area and the radiation range of the divided area determined by shooting based on the corresponding designated position by the visual camera;

[0009] A reconstruction module, used for performing regional reconstruction of the edge range of the regional image of the corresponding divided area according to the neighboring overlap set to obtain a first image;

[0010] A boundary recognition module is used to splice all first images related to the same sub-bookshelf in the book storage rack to obtain a second image, and perform book boundary recognition on each second image, marking the closed-loop boundary and non-closed-loop boundary of each bookshelf grid in the same sub-bookshelf;

[0011] A boundary extension processing module is used to perform boundary extension processing on the corresponding non-closed-loop boundary according to the closed-loop boundary of each bookshelf grid to obtain a new closed-loop boundary;

[0012] The text recognition module is used to identify and summarize the text within all closed-loop boundaries to obtain book summary information.

[0013] Preferably, the area division module includes:

[0014] A vector construction unit, used to construct a shooting vector according to the shelf height and shelf width of each sub-bookshelf in the book storage shelf and in combination with the shooting width of the visual camera;

[0015] The segmentation unit is used to input the shooting vector into the area division model, automatically generate the shooting segmentation lines of the sub-bookshelves, and obtain a plurality of divided areas.

[0016] Preferably, the center position of each divided area is the designated position;

[0017] The book storage rack is assembled from sub-storage racks with the same frame structure, and each sub-storage rack includes a plurality of bookshelf grids, and the length and width of each bookshelf grid are equal.

[0018] Preferably, the reconstruction module comprises:

[0019] A coordinate mapping unit, used to map the regional image corresponding to the divided area into a first coordinate system, and at the same time, map each radiation range in the surrounding overlapping set into a second coordinate system;

[0020] An alignment processing unit, used for performing position alignment processing on the mapped first coordinate system and the second coordinate system, and determining a comparison array of the same position point in each radiation range;

[0021] a clarity comparison unit, configured to determine the retained elements in the comparison array according to the element clarity of each comparison element in the comparison array;

[0022] If the retained element is the original element of the corresponding divided area, it is retained unchanged;

[0023] Otherwise, the original elements of the corresponding partitioned area are replaced to obtain the retained elements;

[0024] A replacement unit is used to obtain a first image according to all replacement results.

[0025] Preferably, the clarity comparison unit comprises:

[0026] A comparison subunit, for retaining the corresponding comparison element based on the first coordinate system if the element definition r01 of the comparison element based on the first coordinate system in the comparison array is greater than or equal to the element definition r02 of the corresponding comparison element based on the second coordinate system;

[0027] Otherwise, determine the range clarity of the radiation range to which the corresponding reference element in the second coordinate system belongs in the reference array, and determine the allowable optimization amount of the corresponding reference element in the first coordinate system in combination with the overall clarity of the corresponding divided image;

[0028]

[0029] Among them, qf represents the corresponding range clarity; qz represents the corresponding overall clarity; max represents the maximum value symbol; ∝1 represents the variable function;

[0030] The replacement subunit is used to upgrade and replace the clarity of the corresponding comparison element based on the first coordinate system according to the allowable optimization amount.

[0031] Preferably, the boundary extension processing module includes:

[0032] A cropping determination unit, used to lock each non-closed-loop boundary in the corresponding bookshelf grid based on a first position of the corresponding second image, and determine a cropping boundary of the corresponding non-closed-loop boundary in combination with a closed-loop boundary in the bookshelf grid that is closest to the corresponding non-closed-loop boundary based on a second position of the corresponding second image;

[0033] an enhancement determination unit, configured to crop the corresponding second image according to the cropping boundary to obtain a third image, determine an enhancement multiple for the third image according to the length of the corresponding non-closed-loop boundary itself and the length of a straight line connecting the first and last points of the non-closed-loop boundary, and perform enhancement processing on the third image to obtain a fourth image;

[0034]

[0035] Wherein, zn represents the enhancement multiple for the third image; Lc represents the length of the corresponding non-closed loop boundary itself; Lsw represents the length of the straight line connecting the first and last points of the corresponding non-closed loop boundary; L b represents the average length of all closed-loop boundaries involved in the second image; Indicates all L b The variance of Zc is: Zc represents the boundary length of the cropping boundary corresponding to the third image; C b represents the boundary length corresponding to the closed loop boundary in the third image; Indicates the floor symbol;

[0036] a re-recognition unit, configured to re-recognize the book boundary of the fourth image, and if a closed-loop boundary is present in the recognition result after extending the corresponding non-closed-loop boundary, retain the closed-loop boundary;

[0037] If it does not exist, the upper and lower protruding points of the non-closed loop boundary are locked to obtain a protruding frame image, and at the same time, the text position is recognized on the fourth image;

[0038] A calibration unit, used for mapping and comparing the recognized text position with the highlight frame diagram, and performing a first calibration on the upper and lower highlight points in the highlight frame diagram that have an upper and lower comparison relationship, and performing a second calibration on the upper and lower highlight points in the highlight frame diagram that do not have a comparison relationship;

[0039] A cutting point judgment unit, used to judge whether the corresponding second calibration point is a closed-loop cutting point according to the protruding direction and protruding height of the second calibration point, the connecting straight line with the two closest opposite calibration points and the text position in the middle area between the two connecting straight lines, and in combination with the final reference height of the burr points of all closed-loop boundaries involved in the corresponding bookshelf grid and the book storage attributes of the corresponding bookshelf grid;

[0040] If yes, keep the corresponding second calibration point; otherwise, remove the corresponding second calibration point;

[0041] A new boundary determination unit, used to connect two points in a vertical comparison relationship with a straight line according to the first calibration result to obtain a new closed-loop boundary;

[0042] A cutting line is set according to the retained second calibration point and in combination with the text position of the middle area involved in the corresponding second calibration point to obtain a new closed-loop boundary.

[0043] Preferably, the cutting point determination unit comprises:

[0044] The distance determination subunit is used to determine a first number of upper burr points and a second number of lower burr points of each closed-loop boundary involved in the corresponding bookshelf grid, and determine a first reference distance between adjacent upper burr points and a second reference distance between adjacent lower burr points in combination with the height of each burr point and the boundary length of the corresponding closed-loop boundary;

[0045] A height determination subunit, used to determine a first reference height and a second reference height according to a first height of an upper burr point and a second height of a lower burr point corresponding to a closed-loop boundary;

[0046] The final determination subunit is used to determine the corresponding final reference height according to all the first reference distances, the first reference heights, the second reference distances and the second reference heights.

[0047] Preferably, the text recognition module includes:

[0048] A combination analysis unit is used to perform a combination analysis of the author name, the book title and the call number on the recognized text within each closed-loop boundary to determine the number of complete comparisons. If the number of complete comparisons is 1 and the number of incomplete comparisons is 0, then the combination information under the number of complete comparisons is retained;

[0049] An adaptation and combination unit, for retaining the combination information under the complete control number if the complete control number is 1 and the incomplete control number is not 0, and adapting and combining the information under the incomplete control number with the information under the incomplete control number within the side closed loop boundary, and retaining the information;

[0050] The summarizing unit is used to summarize all the retained combination information to obtain book summary information.

[0051] A robot is used for applying any of the above-mentioned machine vision-based book inventory systems.

[0052] Compared with the prior art, the beneficial effects of this application are as follows:

[0053] The book storage racks are photographed by visual cameras to obtain regional images, and then the images are processed and the book boundaries are automatically identified. The process saves time and effort and greatly improves inventory efficiency.

[0054] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0055] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0057] Figure 1 This is a structural diagram of a book inventory system based on machine vision in an embodiment of the present invention;

[0058] Figure 2 This is a structural diagram of the radiation range in an embodiment of the present invention;

[0059] Figure 3 A structural diagram of a non-closed-loop boundary and a closed-loop boundary in an embodiment of the present invention;

[0060] Figure 4 A structural diagram of a clipping boundary in an embodiment of the present invention;

[0061] Figure 5 It is a structural diagram of the comparison relationship in the embodiment of the present invention. DETAILED DESCRIPTION

[0062] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0063] The present invention provides a book inventory system based on machine vision, such as Figure 1 As shown, including:

[0064] A region division module is used to divide the inventory area into regions based on the arrangement of the book storage shelves in the inventory area and according to the shooting width of the visual camera, and control the visual camera to move to a designated position corresponding to the divided area to obtain a regional image;

[0065] A set construction module is used to lock the surrounding area of ​​each divided area, and to construct a surrounding overlapping set of the corresponding divided area in combination with the designated position of each divided area and the radiation range of the divided area determined by shooting based on the corresponding designated position by the visual camera;

[0066] A reconstruction module, used for performing regional reconstruction of the edge range of the regional image of the corresponding divided area according to the neighboring overlap set to obtain a first image;

[0067] A boundary recognition module is used to splice all first images related to the same sub-bookshelf in the book storage rack to obtain a second image, and perform book boundary recognition on each second image, marking the closed-loop boundary and non-closed-loop boundary of each bookshelf grid in the same sub-bookshelf;

[0068] A boundary extension processing module is used to perform boundary extension processing on the corresponding non-closed-loop boundary according to the closed-loop boundary of each bookshelf grid to obtain a new closed-loop boundary;

[0069] The text recognition module is used to identify and summarize the text within all closed-loop boundaries to obtain book summary information.

[0070] Preferably, the center position of each divided area is the designated position;

[0071] The book storage rack is assembled from sub-storage racks with the same frame structure, and each sub-storage rack includes a plurality of bookshelf grids, and the length and width of each bookshelf grid are equal.

[0072] In this embodiment, only books with author names, book titles and call numbers on their spines are counted, that is, the books counted in this application all have author names, book titles and call numbers on their spines, thereby facilitating machine vision analysis of the books and achieving effective inventory.

[0073] In this embodiment, the area division is implemented based on the shooting vector.

[0074] In this embodiment, for example, the inventory area is divided into area A, area B, and area C. At this time, the visual camera is placed at the center of area A, area B, and area C to obtain area images of area A, area B, and area C.

[0075] In this embodiment, the surrounding area refers to other areas that are adjacent to the divided area, such as Figure 2 As shown, region B and region C are adjacent to region A. At this time, region B and region C can be regarded as neighboring regions.

[0076] In this embodiment, the radiation range refers to the surrounding area. Although the surrounding area is determined in advance, during the actual shooting process, after the surrounding area is shot and covered as a whole, a part of area A will be covered to some extent. That is, the regional image of area B not only includes the entire area B itself but also a part of area A, and the covered part of area A is regarded as the radiation range. It should be noted that there is no overlap in the radiation ranges of the surrounding areas of the same divided area. Even if there is overlap, the range needs to be determined to ensure non-overlapping, and the attribution can be randomly determined to belong to any surrounding area. This is mainly to avoid overlap, and to facilitate the comparison of only two elements in the array. For example, there is also area D. At this time, the radiation range of area D partially overlaps with the radiation range of area C. At this time, the partially overlapping part can be attributed to the radiation range of area D or the radiation range of area C.

[0077] In this embodiment, the neighbor overlap set = {the emission range corresponding to the adjacent neighbor areas}.

[0078] In this embodiment, the edge range refers to the radiation range that overlaps with the surrounding area in the divided area, that is, Figure 2 The radiation range based on area B and area C involved.

[0079] In this embodiment, since the first image is for each area, the second image can be obtained by edge splicing all the first images, and each sub-bookshelf contains a number of divided areas.

[0080] In this embodiment, the closed-loop boundary and the non-closed-loop boundary of each bookshelf grid are marked for the book itself, and the closed-loop boundary refers to the spine of the book 1 being closed in the form of a parallelogram, and the non-closed-loop boundary refers to the failure to close the book in the form of a parallelogram, that is, the closure is unreasonable, and the form of the non-closed-loop boundary is as follows: Figure 3 shown.

[0081] In this embodiment, the boundary extension process refers to performing a closed-loop process on a non-closed-loop boundary to obtain a new closed-loop boundary.

[0082] In this embodiment, the text is identified and summarized by identifying the author's name, the book title and the call number, and these three can be matched, and the book summary information summarizes and lists all the identified books.

[0083] The beneficial effect of the above technical solution is: the regional image is obtained by shooting the book storage rack with a visual camera, and then the image is processed and the book boundaries are automatically identified, which saves time and labor and greatly improves the inventory efficiency.

[0084] The present invention provides a book inventory system based on machine vision, wherein the area division module comprises:

[0085] A vector construction unit, used to construct a shooting vector according to the shelf height and shelf width of each sub-bookshelf in the book storage shelf and in combination with the shooting width of the visual camera;

[0086] The segmentation unit is used to input the shooting vector into the area division model, automatically generate the shooting segmentation lines of the sub-bookshelves, and obtain a plurality of divided areas.

[0087] In this embodiment, the shooting vector = {shelf height shelf width shooting width}.

[0088] In this embodiment, the area division model is based on different storage rack sizes and shooting widths, and the corresponding effective dividing lines are divided into samples to obtain the training of the neural network model. Therefore, the shooting dividing lines can be automatically generated based on the model, thereby realizing the reasonable division of the storage racks. The main purpose is to ensure that each area after division can achieve the full shooting of the books on the storage rack under the corresponding shooting width to avoid missing books.

[0089] The beneficial effect of the above technical solution is: by constructing vectors and model analysis, it is easy to achieve effective segmentation of storage racks, ensure comprehensive shooting of storage racks to avoid omissions, and thus ensure an accurate basis for inventory counts.

[0090] The present invention provides a book inventory system based on machine vision, wherein the reconstruction module comprises:

[0091] A coordinate mapping unit, used to map the regional image corresponding to the divided area into a first coordinate system, and at the same time, map each radiation range in the surrounding overlapping set into a second coordinate system;

[0092] An alignment processing unit, used for performing position alignment processing on the mapped first coordinate system and the second coordinate system, and determining a comparison array of the same position point in each radiation range;

[0093] a clarity comparison unit, configured to determine the retained elements in the comparison array according to the element clarity of each comparison element in the comparison array;

[0094] If the retained element is the original element of the corresponding divided area, it is retained unchanged;

[0095] Otherwise, the original elements of the corresponding partitioned area are replaced to obtain the retained elements;

[0096] A replacement unit is used to obtain a first image according to all replacement results.

[0097] In this embodiment, the first coordinate system and the second coordinate system are completely consistent, that is, they are aligned in position, and the purpose of mapping the radiation range is to adjust the clarity of the image in the corresponding area.

[0098] In this embodiment, the reference array includes two elements, one is the clarity of the pixel point at position A1 based on the first coordinate system, and the other is the clarity of the pixel point at position A1 based on the second coordinate system, and the element clarity is the clarity of the corresponding pixel point, and the clarity of the pixel point at position A1 can be regarded as a reference element.

[0099] In this embodiment, the retained element is one of the two elements in the reference array, that is, by comparing the clarity of the two elements in the reference array, the element with greater clarity is regarded as the retained element. If the element with greater clarity is the clarity of the pixel point at position A1 in the first coordinate system (the original element corresponding to the divided area), it is retained.

[0100] Preferably, the clarity comparison unit comprises:

[0101] A comparison subunit, for retaining the corresponding comparison element based on the first coordinate system if the element definition r01 of the comparison element based on the first coordinate system in the comparison array is greater than or equal to the element definition r02 of the corresponding comparison element based on the second coordinate system;

[0102] Otherwise, determine the range clarity of the radiation range to which the corresponding reference element in the second coordinate system belongs in the reference array, and determine the allowable optimization amount of the corresponding reference element in the first coordinate system in combination with the overall clarity of the corresponding divided image;

[0103]

[0104] Among them, qf represents the corresponding range clarity; qz represents the corresponding overall clarity; max represents the maximum value symbol; ∝1 represents the variable function;

[0105] The replacement subunit is used to upgrade and replace the clarity of the corresponding comparison element based on the first coordinate system according to the allowable optimization amount.

[0106] In this embodiment, the clarity of the radiation range may refer to the clarity of the radiation range of area A, etc.

[0107] In this embodiment, the overall clarity refers to the clarity of the corresponding divided images, and the clarity is determined by the resolution. The higher the resolution, the higher the clarity.

[0108] In this embodiment, the purpose of setting the allowed optimization amount is to adjust the clarity of pixels whose clarity is not very high, so as to ensure the reliability of subsequent recognition of the boundary of the image.

[0109] The beneficial effect of the above technical solution is: the purpose of not adopting global adjustment is to avoid the situation where the picture is too dirty, and the clarity of the relevant points is adjusted in a targeted manner to avoid the situation where the picture appears unnatural due to excessive adjustment of clarity.

[0110] The present invention provides a book inventory system based on machine vision, wherein the boundary extension processing module comprises:

[0111] A cropping determination unit, used to lock each non-closed-loop boundary in the corresponding bookshelf grid based on a first position of the corresponding second image, and determine a cropping boundary of the corresponding non-closed-loop boundary in combination with a closed-loop boundary in the bookshelf grid that is closest to the corresponding non-closed-loop boundary based on a second position of the corresponding second image;

[0112] an enhancement determination unit, configured to crop the corresponding second image according to the cropping boundary to obtain a third image, determine an enhancement multiple for the third image according to the length of the corresponding non-closed-loop boundary itself and the length of a straight line connecting the first and last points of the non-closed-loop boundary, and perform enhancement processing on the third image to obtain a fourth image;

[0113]

[0114] Wherein, zn represents the enhancement multiple for the third image; Lc represents the length of the corresponding non-closed loop boundary itself; Lsw represents the length of the straight line connecting the first and last points of the corresponding non-closed loop boundary; L b represents the average length of all closed-loop boundaries involved in the second image; Indicates all L bThe variance of Zc is: Zc represents the boundary length of the cropping boundary corresponding to the third image; C b represents the boundary length corresponding to the closed loop boundary in the third image; Indicates the floor symbol;

[0115] a re-recognition unit, configured to re-recognize the book boundary of the fourth image, and if a closed-loop boundary is present in the recognition result after extending the corresponding non-closed-loop boundary, retain the closed-loop boundary;

[0116] If it does not exist, the upper and lower protruding points of the non-closed loop boundary are locked to obtain a protruding frame image, and at the same time, the text position is recognized on the fourth image;

[0117] A calibration unit, used for mapping and comparing the recognized text position with the highlight frame diagram, and performing a first calibration on the upper and lower highlight points in the highlight frame diagram that have an upper and lower comparison relationship, and performing a second calibration on the upper and lower highlight points in the highlight frame diagram that do not have a comparison relationship;

[0118] A cutting point judgment unit, used to judge whether the corresponding second calibration point is a closed-loop cutting point according to the protruding direction and protruding height of the second calibration point, the connecting straight line with the two closest opposite calibration points and the text position in the middle area between the two connecting straight lines, and in combination with the final reference height of the burr points of all closed-loop boundaries involved in the corresponding bookshelf grid and the book storage attributes of the corresponding bookshelf grid;

[0119] If yes, keep the corresponding second calibration point; otherwise, remove the corresponding second calibration point;

[0120] A new boundary determination unit, used to connect two points in a vertical comparison relationship with a straight line according to the first calibration result to obtain a new closed-loop boundary;

[0121] A cutting line is set according to the retained second calibration point and in combination with the text position of the middle area involved in the corresponding second calibration point to obtain a new closed-loop boundary.

[0122] In this embodiment, the second image is an image of the entire sub-bookshelf, and the sub-bookshelf includes a number of shelf grids, such as Figure 4 As shown, the books are placed on the bookshelf grid, and the first position of the non-closed loop boundary refers to the Figure 4 a1 in the figure, the closed-loop boundary closest to the corresponding non-closed-loop boundary is based on the second position a2 of the second image. At this time, the determined cropping boundary is the outermost boundary including the closest closed-loop boundary and the non-closed-loop boundary, such as c1. At this time, the result of cropping according to c1 is the third image.

[0123] In this embodiment, the non-closed loop boundary refers to a boundary with unreasonable closure, that is, a closure that is not in the form of a parallelogram.

[0124] In this embodiment, the length itself is Figure 4 The length of the thick continuous line segment between points d1 and d2, and the length of the straight line connecting the first and last points is Figure 4 The length of the thin continuous line segment between points d1 and d2.

[0125] In this embodiment, when the value of zn is 0, it is determined that the third image is not enhanced;

[0126] If the value of zn is not 0, the third image is enhanced accordingly according to the enhancement factor.

[0127] In this embodiment, after the image enhancement processing, the image may be re-identified to obtain an extended boundary, that is, the original non-closed-loop boundary is closed.

[0128] In this embodiment, if Figure 5 As shown, the upper and lower protruding points of the non-closed-loop boundary b1 are b01, b02, b03, b04, and b05, respectively. Among them, b01, b02, and b03 are located above the boundary and are regarded as upper protruding points, and b04 and b05 are located below the boundary and are regarded as lower protruding points. Because, in the process of intelligent drawing of the boundary, there will be mutations in the drawing line, because there is a large gap between the corners of each book, and because there is a certain small arc in the boundary, different mutations are prone to occur in the recognition process. At this time, the position with mutations is regarded as a protruding point and then the protruding point is marked. At this time, a protruding framework diagram is obtained. Figure 5 Where wb1 and wb2 are text positions.

[0129] In this embodiment, text position recognition is to lock the position where the text exists in the image, which can be used as a basis for the existence of an upper and lower comparison relationship. Because the text on the spine of each book is generally located in the center, after the initial rough locking of the position, the unreasonable comparison of the upper and lower protruding points can be effectively avoided. For example, before relying on the text position, the comparison relationships are determined to be: b01 and b04, b02 and b05. After relying on the text position, the comparison relationships are determined to be: b01 and b04, b03 and b05. This is to improve the accuracy of the comparison. Because if the comparison between b02 and b05 is continued, the area after the connection between the two is the area where some text positions exist or no text positions exist, which is unreasonable. At this time, the first calibration is performed on b01 and b04, b03 and b05, and the second calibration is performed on b02. It should be noted that the area after the points after the comparison are connected includes the text position.

[0130] In this embodiment, the protrusion direction refers to whether the direction of the corresponding point is upward or downward.

[0131] In this embodiment, the protruding height refers to the protruding height of the point based on the horizontal line, which can be an upper height or a lower height.

[0132] In this embodiment, the connecting straight lines of the two closest opposite calibration points are the connecting straight lines of b02 and b03 and the connecting straight lines of b02 and b04. The two middle areas connected to Xi'an are the middle parts of the connecting straight lines formed by b02 and b03 and the connecting straight lines formed by b02 and b04.

[0133] In this embodiment, the burr point refers to any protruding position point existing on a straight line.

[0134] In this embodiment, the book storage attribute refers to the type of the corresponding book, such as literature type, ancient poetry type, etc. Generally, books of literature type are thicker.

[0135] In this embodiment, determining whether the corresponding second calibration point is a closed-loop cutting point includes:

[0136]

[0137] Among them, Px is the corresponding judgment coefficient; g t Indicates the protruding height of the second calibration point; r w Indicates the position value of the corresponding text position; z r Indicates the conversion coefficient based on the position value; zx tf Indicates the conversion coefficient of protrusion direction to height; g s represents the book thickness matched by the book storage attribute of the corresponding bookshelf grid; h0 represents the distance between the second calibration point and the first calibration point with the closest corresponding relationship; g a represents the average thickness of all closed-loop boundaries involved in the corresponding bookshelf grid; z g Indicates the conversion factor based on thickness;

[0138] When px is greater than 0.7, it is determined to be a closed-loop cutting point, and the value range of Px is 0 to 1.

[0139] In this embodiment, a blank connected area in the middle area is found, and a straight line that can penetrate the area vertically, does not occupy a text position in the penetration line, and has one end connected to the second calibration point is regarded as a cutting line.

[0140] The beneficial effect of the above technical solution is: by locking the non-closed-loop boundary and the closed-loop boundary, the cropping boundary is effectively determined and the subsequent determination of the enhancement multiple is convenient to achieve reasonable enhancement of the image, that is, the initial extension of the non-closed-loop boundary is achieved through enhancement, and the subsequent use of the highlight framework diagram and the upper and lower highlight point comparison analysis process is continued to achieve a reasonable extension of the second calibration point straight line, to ensure the reasonable closure of the non-closed-loop boundary as much as possible, to achieve effective inventory of books, and to ensure inventory efficiency and accuracy.

[0141] The present invention provides a book inventory system based on machine vision, wherein the cutting point judgment unit comprises:

[0142] A distance determination subunit, used to determine a first number of upper burr points, a second number of lower burr points and a boundary thickness of the corresponding closed-loop boundary of each closed-loop boundary involved in the corresponding bookshelf grid, and to determine a first reference distance between adjacent upper burr points and a second reference distance between adjacent lower burr points;

[0143] A height determination subunit, used to determine a first reference height and a second reference height according to a first height of an upper burr point and a second height of a lower burr point corresponding to a closed-loop boundary;

[0144] The final determination subunit is used to determine the corresponding final reference height according to all the first reference distances, the first reference heights, the second reference distances and the second reference heights.

[0145] In this embodiment, the burr point at the upper boundary is regarded as an upper burr point, and the burr point at the lower boundary is regarded as a lower burr point.

[0146] In this embodiment, the first reference distance=(border thickness / first number+average value of lengths between adjacent upper burr points) / 2.

[0147] The second reference distance=(border thickness / second number+average value of lengths between adjacent lower burr points) / 2.

[0148] In this embodiment, the first reference height = the average value of all first heights;

[0149] Second reference height=average value of all second heights.

[0150] In this embodiment, the final reference height=(first reference height+second reference height) / 2-min(first reference distance, second reference distance) / (1+first number+second number).

[0151] The beneficial effect of the above technical solution is: through comprehensive analysis of the number and height of burr points, the thickness of the boundary, etc., the final reference height is effectively determined, providing a basis for judging whether it is a closed-loop cutting point.

[0152] The present invention provides a book inventory system based on machine vision, wherein the text recognition module comprises:

[0153] A combination analysis unit is used to perform a combination analysis of the author name, the book title and the call number on the recognized text within each closed-loop boundary to determine the number of complete comparisons. If the number of complete comparisons is 1 and the number of incomplete comparisons is 0, then the combination information under the number of complete comparisons is retained;

[0154] An adaptation and combination unit, for retaining the combination information under the complete control number if the complete control number is 1 and the incomplete control number is not 0, and adapting and combining the information under the incomplete control number with the information under the incomplete control number within the side closed loop boundary, and retaining the information;

[0155] The summarizing unit is used to summarize all the retained combination information to obtain book summary information.

[0156] In this embodiment, for example, the recognition result is: author u1, book title u2, call number r01. At this time, author u1, book title u2, call number r01 are completely corresponding, for example, Shi Tiesheng--I and the Altar of Earth--Su01, and the number of complete comparisons is 1.

[0157] In this embodiment, for example, the recognition results are: author u1, author u2, book title u2, call number r01. At this time, it is determined that author u1, book title u2 and call number r01 are completely corresponding, and author u2 is redundant. At this time, the number of complete comparisons is 1, and the number of incomplete comparisons is also 1. It should be noted that when there is an incomplete situation within the closed-loop boundary, it must also exist in the adjacent closed-loop boundary. At this time, matching can be performed.

[0158] The beneficial effect of the above technical solution is: by identifying the text in the closed-loop boundary and based on the one-to-one matching of the author's name and the book title, reasonable adaptation is effectively achieved, the integrity and rationality of the summary are guaranteed, and the inventory efficiency is improved.

[0159] The present invention provides a robot for applying any of the above-mentioned machine vision-based book inventory systems.

[0160] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A book inventory system based on machine vision, characterized in that: include: A region division module is used to divide the inventory area into regions based on the arrangement of the book storage shelves in the inventory area and according to the shooting width of the visual camera, and control the visual camera to move to a designated position corresponding to the divided area to obtain a regional image; A set construction module is used to lock the surrounding area of ​​each divided area, and to construct a surrounding overlapping set of the corresponding divided area in combination with the designated position of each divided area and the radiation range of the divided area determined by shooting based on the corresponding designated position by the visual camera; A reconstruction module, used for performing regional reconstruction of the edge range of the regional image of the corresponding divided area according to the neighboring overlap set to obtain a first image; A boundary recognition module is used to splice all first images related to the same sub-bookshelf in the book storage rack to obtain a second image, and perform book boundary recognition on each second image, marking the closed-loop boundary and non-closed-loop boundary of each bookshelf grid in the same sub-bookshelf; A boundary extension processing module is used to perform boundary extension processing on the corresponding non-closed-loop boundary according to the closed-loop boundary of each bookshelf grid to obtain a new closed-loop boundary; The text recognition module is used to identify and summarize the text within all closed-loop boundaries to obtain book summary information.

2. The machine vision-based book inventory system according to claim 1, characterized in that: The area division module comprises: A vector construction unit, used to construct a shooting vector according to the shelf height and shelf width of each sub-bookshelf in the book storage shelf and in combination with the shooting width of the visual camera; The segmentation unit is used to input the shooting vector into the area division model, automatically generate the shooting segmentation lines of the sub-bookshelves, and obtain a plurality of divided areas.

3. The machine vision-based book inventory system according to claim 2, characterized in that: The center position of each divided area is the designated position; The book storage rack is assembled from sub-storage racks with the same frame structure, and each sub-storage rack includes a plurality of bookshelf grids, and the length and width of each bookshelf grid are equal.

4. The machine vision-based book inventory system according to claim 1, characterized in that: The reconstruction module comprises: A coordinate mapping unit, used to map the regional image corresponding to the divided area into a first coordinate system, and at the same time, map each radiation range in the surrounding overlapping set into a second coordinate system; An alignment processing unit, used for performing position alignment processing on the mapped first coordinate system and the second coordinate system, and determining a comparison array of the same position point in each radiation range; a clarity comparison unit, used for determining the retained elements in the comparison array according to the element clarity of each comparison element in the comparison array; If the retained element is the original element of the corresponding divided area, it is retained unchanged; Otherwise, the original elements of the corresponding partitioned area are replaced to obtain the retained elements; A replacement unit is used to obtain a first image according to all replacement results.

5. The machine vision-based book inventory system according to claim 4 is characterized in that: The clarity comparison unit comprises: A comparison subunit, for retaining the corresponding comparison element based on the first coordinate system if the element definition r01 of the comparison element based on the first coordinate system in the comparison array is greater than or equal to the element definition r02 of the corresponding comparison element based on the second coordinate system; Otherwise, determine the range clarity of the radiation range to which the corresponding reference element in the second coordinate system belongs in the reference array, and determine the allowable optimization amount of the corresponding reference element in the first coordinate system in combination with the overall clarity of the corresponding divided image; Among them, qf represents the corresponding range clarity; qz represents the corresponding overall clarity; max represents the maximum value symbol; ∝1 represents the variable function; The replacement subunit is used to upgrade and replace the clarity of the corresponding comparison element based on the first coordinate system according to the allowable optimization amount.

6. The machine vision-based book inventory system according to claim 1, characterized in that: The boundary extension processing module includes: A cropping determination unit, used to lock each non-closed-loop boundary in the corresponding bookshelf grid based on a first position of the corresponding second image, and determine a cropping boundary of the corresponding non-closed-loop boundary in combination with a closed-loop boundary in the bookshelf grid that is closest to the corresponding non-closed-loop boundary based on a second position of the corresponding second image; an enhancement determination unit, configured to crop the corresponding second image according to the cropping boundary to obtain a third image, determine an enhancement multiple for the third image according to the length of the corresponding non-closed-loop boundary itself and the length of a straight line connecting the first and last points of the non-closed-loop boundary, and perform enhancement processing on the third image to obtain a fourth image; Wherein, zn represents the enhancement multiple for the third image; Lc represents the length of the corresponding non-closed loop boundary itself; Lsw represents the length of the straight line connecting the first and last points of the corresponding non-closed loop boundary; L b represents the average length of all closed-loop boundaries involved in the second image; Indicates all L b The variance of Zc is: Zc represents the boundary length of the cropping boundary corresponding to the third image; C b represents the boundary length corresponding to the closed loop boundary in the third image; Indicates the floor symbol; a re-recognition unit, configured to re-recognize the book boundary of the fourth image, and if a closed-loop boundary is present in the recognition result after extending the corresponding non-closed-loop boundary, retain the closed-loop boundary; If it does not exist, the upper and lower protruding points of the non-closed loop boundary are locked to obtain a protruding frame image, and at the same time, the text position is recognized on the fourth image; A calibration unit, used for mapping and comparing the recognized text position with the highlight frame diagram, and performing a first calibration on the upper and lower highlight points in the highlight frame diagram that have an upper and lower comparison relationship, and performing a second calibration on the upper and lower highlight points in the highlight frame diagram that do not have a comparison relationship; A cutting point judgment unit, used to judge whether the corresponding second calibration point is a closed-loop cutting point according to the protruding direction and protruding height of the second calibration point, the connecting straight line with the two closest opposite calibration points and the text position in the middle area between the two connecting straight lines, and in combination with the final reference height of the burr points of all closed-loop boundaries involved in the corresponding bookshelf grid and the book storage attributes of the corresponding bookshelf grid; If yes, keep the corresponding second calibration point; otherwise, remove the corresponding second calibration point; A new boundary determination unit, used to connect two points in a vertical comparison relationship with a straight line according to the first calibration result to obtain a new closed-loop boundary; A cutting line is set according to the retained second calibration point and in combination with the text position of the middle area involved in the corresponding second calibration point to obtain a new closed-loop boundary.

7. The machine vision-based book inventory system according to claim 6, characterized in that: The cutting point judgment unit comprises: The distance determination subunit is used to determine a first number of upper burr points and a second number of lower burr points of each closed-loop boundary involved in the corresponding bookshelf grid, and determine a first reference distance between adjacent upper burr points and a second reference distance between adjacent lower burr points in combination with the height of each burr point and the boundary length of the corresponding closed-loop boundary; A height determination subunit, used to determine a first reference height and a second reference height according to a first height of an upper burr point and a second height of a lower burr point corresponding to a closed-loop boundary; The final determination subunit is used to determine the corresponding final reference height according to all the first reference distances, the first reference heights, the second reference distances and the second reference heights.

8. The machine vision-based book inventory system according to claim 1, characterized in that: The text recognition module comprises: A combination analysis unit is used to perform a combination analysis of the author name, the book title and the call number on the recognized text within each closed-loop boundary to determine the number of complete comparisons. If the number of complete comparisons is 1 and the number of incomplete comparisons is 0, then the combination information under the number of complete comparisons is retained; An adaptation and combination unit, for retaining the combination information under the complete control number if the complete control number is 1 and the incomplete control number is not 0, and adapting and combining the information under the incomplete control number with the information under the incomplete control number within the side closed loop boundary, and retaining the information; The summarizing unit is used to summarize all the retained combination information to obtain book summary information.

9. A robot, characterized in that: Used to apply the machine vision-based book inventory system described in any one of claims 1-8.

Images