File checking method and system based on machine vision

Abstract

The invention provides a machine vision-based archive checking method and system. The method comprises the following steps of: acquiring images of each stored archive of an archive outfit; splicing the archive images of the target layer; using an image detection algorithm to detect the archive storage unit area in the spliced image to obtain the unit storage area of each archive; detecting and segmenting the file box ridge target of each unit storage area by using an image segmentation algorithm, and counting the number of segmented areas; extracting a single segmented area of the file box, performing image correction according to the inclination angle of the segmented area, and correcting the file box into a vertical angle; file number area image extraction is carried out on the corrected file box image, file numbers are extracted by using an OCR identification technology, and a file number sequence of files stored in the layer is determined; comparing the counted number with a pre-stored ledger number of the layer of archives, and determining whether the number of the archives is correct or not; and comparing the determined file number sequence with pre-stored ledger information of the layer of file to determine whether the layer of file is missing or not and whether the storage position is correct or not. According to the invention, the in-warehouse state of the archives in the warehouse can be judged, and whether the in-warehouse position of the archives in the warehouse is correct can also be judged.

Description

Technical Field

[0001] The invention belongs to the field of archive inventory, and in particular relates to an archive inventory method and system based on machine vision. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Inventory work in archive warehouses is a basic and important task in archive management. It can not only check the number of archives in the collection and standardize archive management, but also understand the collection situation and improve archive services. When archive warehouses conduct regular comprehensive or partial inventory task management, they mostly use manual handheld inventory, inventory carts, inventory robots, etc. However, facing the huge number of archives, manual inventory alone is time-consuming, inefficient, and has a high error rate.

[0004] With the development of electronic information technology, the introduction of label technologies such as RFID and QR codes in modern intelligent archive warehouse management can greatly improve the efficiency of archive inventory. RFID electronic tags have many advantages: their readers do not require visible light sources, are penetrating, can directly read information through external materials, can process multiple radio frequency tags at the same time, batch process, simplify labor intensity, have a low probability of error, and are not greatly affected by environmental conditions. However, this inventory method requires RFID tag processing of archives, and RFID tags need to be pasted on archive boxes, and the cost of renovating old archives is high. In addition, the archives in the archive warehouse are arranged closely. When using RFID equipment for inventory, due to interference between RFID tags, it is impossible to determine whether the location of the archives is accurate during inventory. QR codes also require a lot of manual pasting, QR code information writing and other manual operations, and after long-term use, QR codes are easily damaged and dirty, resulting in inaccurate recognition. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a file inventory method and system based on machine vision. The present invention can not only determine the storage status of files in the warehouse, but also determine whether the storage location of files in the warehouse is correct.

[0006] It should be noted that the archives referred to in the present invention can be various forms of original records with preservation value directly formed by people in various social activities, existing in the form of tangible storage media, and are a general term for all materials in the form of text, graphics, audio and video, such as books, CDs, videotapes, etc.

[0007] According to some embodiments, the present invention adopts the following technical solutions:

[0008] A file inventory method based on machine vision includes the following steps:

[0009] Obtain target images from each stored file;

[0010] Sequentially stitch the images of the file boxes in different areas of the same face of the same compact rack;

[0011] Use image detection algorithm to detect each unit storage area in the spliced ​​image. Each section and each row of the compact shelving is a unit storage area.

[0012] Using the image segmentation algorithm, extract the single segmented area of ​​the file box, count the number and position of the segmented area, the segmented area is the spine area of ​​a file box, convert the image coordinates of the segmented area into world coordinates, confirm the location information of the file in the unit storage area, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle;

[0013] Extract the file number area image from the corrected file box image, extract the file number using OCR recognition technology, save the corresponding position information, and determine the file number sequence and position of the file in the storage layer;

[0014] Compare the counted number with the number of pre-stored records of the archives at this level to confirm whether the number of archives is correct;

[0015] Compare the determined file number sequence, file location and pre-stored ledger information of the files in this area to confirm whether the files in this area are missing and whether the storage location is correct.

[0016] As an optional implementation, an image acquisition device at the end of the mechanical arm of the automatic archive storage and retrieval robot is used to acquire images of the file boxes of each stored file in a predetermined order;

[0017] The order of reservation is the same as the order of image acquisition when storing ledger information;

[0018] The predetermined order is from the outermost side to the innermost side of the file holder, and from the uppermost end to the lowermost end.

[0019] As an optional implementation, the file number and title information are set on the back of the file box.

[0020] As an optional implementation method, the number of ledgers and ledger information of each level of archives are pre-stored in the background system.

[0021] As an optional implementation, the images of the archive storage areas of the target area are stitched together using an image stitching algorithm based on SURF feature point matching.

[0022] As an optional implementation, the image detection algorithm is an image detection algorithm based on an improved YOLOv8 model, and the process of improving the YOLOv8 model is as follows: the feature fusion layer of the backbone network part only selects the sampling feature maps extracted by the P3 convolution layer and the P5 convolution layer for fusion, and the feature fusion method adopts a cascade method to obtain a fused feature map output;

[0023] Use Swi sh instead of SiLU as the activation function.

[0024] As an optional implementation, the image segmentation algorithm is to use the YOLOv8 detection algorithm to obtain the file box area, crop the file box area, and apply the Unet algorithm to the cropped image to segment the file box spine image.

[0025] A file inventory system based on machine vision, comprising:

[0026] An image acquisition module is used to acquire target images of each stored file;

[0027] An image stitching module is used to sequentially stitch images of various file boxes in various areas on the same face of the same compact rack;

[0028] The image detection module is used to detect each unit storage area in the spliced ​​image using an image detection algorithm. Each section and each row of the compact rack is a unit storage area.

[0029] An image segmentation module is used to extract a single segmented area of ​​the file box using an image segmentation algorithm, count the number and position of the segmented areas, where the segmented area is the spine area of ​​a file box, convert the image coordinates of the segmented area into world coordinates, confirm the location information of the file in the unit storage area, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle;

[0030] An image correction module is used to extract a single segmented area of ​​the file box, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle;

[0031] The OCR recognition module is used to extract the file number area image from the corrected file box image, extract the file number using OCR recognition technology, and save the corresponding position information to determine the file number sequence of the stored files in this layer;

[0032] The comparison module is used to compare the statistical quantity with the pre-stored ledger quantity of the archives in this layer to confirm whether the number of archives is correct; to compare the determined file number sequence, archive location and the pre-stored ledger information of the archives in this area to confirm whether the archives in this area are missing and whether the storage location is correct.

[0033] As an optional implementation, it further includes a file management robot and a processor, wherein the file management robot is used for accessing files and collecting images;

[0034] The processor is used to store the number and information of archive ledgers.

[0035] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in any one of the above methods are completed.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention innovatively provides an archive inventory method based on machine vision, which collects images of archives on the shelves in sequence, and then uses image stitching technology to synthesize images of the archives on a certain layer of the compact shelving, uses the target detection algorithm of the improved YOLOv8 model to identify the archive area on the shelves of a certain section of the compact shelving, and then performs archive spine area segmentation according to the target detection result, performs single archive box spine image correction on the segmented area, extracts the file number area using the prior information of the archive box spine position, and then uses OCR technology to extract the file number, compares the number of archive boxes detected on this layer with the file number information and the archive ledger, completes the inventory of the archives on the current layer of the compact shelving, and repeats the above process to complete the inventory of all the archives on the compact shelving in the archive warehouse. The overall process is simple, does not require major changes to robots and archive cabinets / archive rooms, and has high recognition efficiency.

[0038] The present invention innovatively provides an archive inventory system based on machine vision, which realizes rapid inventory of archives, reduces the cost of archive RFID and QR code processing, and improves the efficiency of archive inventory; it also improves the target detection speed, and combines with OCR recognition technology to realize the inventory of the number of archives in the library and the accurate inventory of the archives' location on the shelf.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0041] Figure 1 The flowchart of an embodiment is shown in FIG. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0044] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0045] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0046] For a clear description, the following terms are explained first:

[0047] Archives: It is a collection of original records of various forms with preservation value directly formed by people in various social activities. It exists in the form of tangible storage media. It is a general term for all kinds of materials in the form of text, graphics, audio and video, such as books, CDs, videotapes, etc.

[0048] File box: a box / shell used to hold each file.

[0049] Filing cabinet / compact shelving: A cabinet / shelf used to store files, which can contain multiple layers, each with a certain amount of storage space.

[0050] File management robot: The file management robot is a truss-type robot, consisting of a column, a guide rail and a robotic arm. The end of the robotic arm is equipped with a clamping device to realize automatic storage and retrieval of files. The end can also be equipped with a camera to collect work images.

[0051] Archives inventory: Archives inventory refers to the process of comprehensively sorting, checking, organizing, compiling and cataloging the archives of one or more units. The main purpose of paper archives inventory is to confirm whether the number of archives in the library is consistent with the archives ledger, and whether the location of the archives in the library is consistent with the ledger.

[0052] Embodiment 1

[0053] A file inventory method based on machine vision, such as Figure 1 As shown, the following steps are included:

[0054] Obtaining the file box image of each stored file;

[0055] Stitching the images of the archive boxes of the target layer;

[0056] Using the image detection algorithm, the archive area of ​​each layer of a certain section of the compact shelving in the spliced ​​image is detected to obtain the archive storage area of ​​each section of the compact shelving;

[0057] Using the image segmentation algorithm, the detected area is segmented again, and the number and position of the segmented areas are counted;

[0058] Extract a single segmented area of ​​the file box, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle;

[0059] Extract the file number area image from the corrected file box image, use the CRNN character recognition model to automatically identify each file number area, identify the file number information, and determine the file number sequence of the files stored in this layer;

[0060] Compare the counted number with the number of pre-stored records of the archives at this level to confirm whether the number of archives is correct;

[0061] Compare the determined file number sequence with the pre-stored ledger information of the files at this level to confirm whether the files at this level are missing and whether the storage location is correct.

[0062] Of course, in some embodiments, an inventory of all stored files may be performed, so the above process steps may be performed iteratively, for example, one file cabinet at a time, or one file cabinet at a time.

[0063] In these embodiments, image acquisition can utilize the image acquisition device at the end of the mechanical arm of the automatic archive storage and retrieval robot to perform image acquisition of archives stored on the compact shelving in a sequence from top to bottom and from outside to inside.

[0064] In some embodiments, an image acquisition device at the end of the mechanical arm of the automatic file storage and retrieval robot is used to acquire images of the file boxes of each stored file in a predetermined order;

[0065] The order of reservation is the same as the order of image acquisition when storing ledger information.

[0066] In this embodiment, the file number information is set on the back of the file box.

[0067] The number of records and record information of each layer of files are stored in advance. Before the files are put on the shelves, the file information and shelf location are input into the background management system in advance. The file location is the storage location number of the compact shelving, such as 3 rows of compact shelving with 3 sections, 1 layer and 4 grids on the A side.

[0068] In this embodiment, the image stitching algorithm based on SURF feature point matching can be used to stitch the image of each file box of the target layer. Of course, in other embodiments, other image stitching algorithms can be used to stitch the image of each file box.

[0069] In this embodiment, the image detection algorithm is a training area detection model, that is, an image detection algorithm based on an improved YOLOv8 model. The process of improving the YOLOv8 model is as follows: the feature fusion layer of the backbone network part only selects the sampling feature maps extracted by the P3 convolution layer and the P5 convolution layer for fusion, and the feature fusion method adopts a cascade method to obtain a fused feature map output;

[0070] Swi sh is used instead of Si LU as the activation function to further reduce the amount of calculation.

[0071] In this embodiment, the image detection algorithm detects a total of N×M archive storage areas on each file rack, where N is the number of archive sections in this column, and M is the number of layers of this column of archive compact racks. A file box segmentation model can be constructed and trained to perform file spine image segmentation on each file storage area. The file spine segmentation algorithm is an image segmentation network based on the combination of YOLOv8 and Unet to obtain a file spine segmentation image.

[0072] In this embodiment, the existing image correction method can be used to correct the spine of the file box to a vertical angle. In addition, the file number area image is extracted based on the prior position information of the title and file number information on the spine of the file box, the file number is extracted using OCR recognition technology, and the file number sequence of each layer of the file is counted.

[0073] According to the regional order statistics, analyze whether the files are stored according to the file number information, and feedback the information to the background or host computer. Compare with the ledger of the files of this layer pre-stored in the background or host computer to confirm whether the files of this layer are missing and whether the storage location is correct.

[0074] If they are the same, then there is no missing file in that layer and the storage location is correct. If the quantity is incorrect and the information comparison results are different, then there may be missing files or incorrect storage location.

[0075] The storage status can be checked manually or based on the comparison results.

[0076] Embodiment 2

[0077] A file inventory system based on machine vision, comprising:

[0078] An image acquisition module is used to acquire the image of the file box of each stored file;

[0079] An image stitching module, used for stitching the images of the archive boxes of the target layer;

[0080] An image detection module is used to detect the archive storage area of ​​each grid of the compact rack in the spliced ​​image using an image detection algorithm to obtain the archive storage area of ​​a certain section and a certain layer of the compact rack;

[0081] An image segmentation module is used to perform image segmentation of the spine area of ​​the detected archive storage area using an image segmentation algorithm, and to count the number of segmented areas;

[0082] An image correction module is used to extract a single segmented area of ​​the file box, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle;

[0083] The OCR recognition module is used to extract the file number area image from the corrected file box spine image, extract the file number using OCR recognition technology, and determine the file number sequence of the files stored in this layer;

[0084] The comparison module is used to compare the statistical quantity with the pre-stored ledger quantity of the archives at this level to confirm whether the number of archives is correct; to compare the determined file number sequence with the pre-stored ledger information of the archives at this level to confirm whether the archives at this level are missing and whether the storage location is correct.

[0085] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A file inventory method based on machine vision, characterized in that: The following steps are involved: Obtain target images from each stored file; Sequentially stitch the images of the file boxes in different areas of the same face of the same compact rack; Use image detection algorithm to detect each unit storage area in the spliced ​​image. Each section and each row of the compact shelving is a unit storage area. Using the image segmentation algorithm, extract the single segmented area of ​​the file box, count the number and position of the segmented area, the segmented area is the spine area of ​​a file box, convert the image coordinates of the segmented area into world coordinates, confirm the location information of the file in the unit storage area, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle; Extract the file number area image from the corrected file box image, extract the file number using OCR recognition technology, save the corresponding position information, and determine the file number sequence and position of the file in the storage layer; Compare the counted number with the number of pre-stored records of the archives at this level to confirm whether the number of archives is correct; Compare the determined file number sequence, file location and pre-stored ledger information of the files in this area to confirm whether the files in this area are missing and whether the storage location is correct.

2. The method for file inventory based on machine vision as claimed in claim 1, characterized in that: The image acquisition device at the end of the mechanical arm of the automatic file storage and retrieval robot is used to obtain the image of the file box of each stored file in a predetermined order; The order of reservation is the same as the order of image acquisition when storing ledger information; The predetermined order is from the outermost side to the innermost side of the file holder, and from the uppermost end to the lowermost end.

3. The method for file inventory based on machine vision as claimed in claim 1, characterized in that: The file box has file number and title information arranged on the back of the box.

4. The method for file inventory based on machine vision as claimed in claim 1, characterized in that: The number of ledgers and ledger information of each level of archives are pre-stored in the background system.

5. The method for file inventory based on machine vision as claimed in claim 1, characterized in that: The images of each archive storage area in the target area are stitched together using an image stitching algorithm based on SURF feature point matching.

6. The method for file inventory based on machine vision as claimed in claim 1, characterized in that: The image detection algorithm is an image detection algorithm based on an improved YOLOv8 model. The process of improving the YOLOv8 model is as follows: the feature fusion layer of the backbone network only selects the sampled feature maps extracted by the P3 convolution layer and the P5 convolution layer for fusion, and the feature fusion method adopts a cascade method to obtain a fused feature map output; Swish is used instead of SiLU as the activation function.

7. The method for file inventory based on machine vision as claimed in claim 1, characterized in that: The image segmentation algorithm is to use the YOLOv8 detection algorithm to obtain the file box area, crop the file box area, and apply the Unet algorithm to the cropped image to segment the file box spine image.

8. A file inventory system based on machine vision, characterized in that: include: An image acquisition module is used to acquire target images of each stored file; An image stitching module is used to sequentially stitch images of various file boxes in various areas on the same face of the same compact rack; The image detection module is used to detect each unit storage area in the spliced ​​image using an image detection algorithm. Each section and each row of the compact rack is a unit storage area. An image segmentation module is used to extract a single segmented area of ​​the file box using an image segmentation algorithm, count the number and position of the segmented areas, where the segmented area is the spine area of ​​a file box, convert the image coordinates of the segmented area into world coordinates, confirm the location information of the file in the unit storage area, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle; An image correction module is used to extract a single segmented area of ​​the file box, perform image correction according to the inclination of the segmented area, and correct the file box to a vertical angle; The OCR recognition module is used to extract the file number area image from the corrected file box image, extract the file number using OCR recognition technology, and save the corresponding position information to determine the file number sequence of the stored files in this layer; The comparison module is used to compare the statistical quantity with the pre-stored ledger quantity of the archives in this layer to confirm whether the number of archives is correct; to compare the determined file number sequence, archive location and the pre-stored ledger information of the archives in this area to confirm whether the archives in this area are missing and whether the storage location is correct.

9. The machine vision-based file inventory system according to claim 8, characterized in that: Also included is a file management robot and a processor, wherein the file management robot is used for accessing files and collecting images; The processor is used to store the number and information of archive ledgers.

10. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method according to any one of claims 1 to 7 are completed.

Images