Goods shelf inspection method and system based on machine vision and storage medium

Abstract

The invention discloses a goods shelf inspection method and system based on machine vision and a storage medium, and relates to the technical field of goods shelf inspection. The method comprises the following steps of: 1, acquiring visual data of a goods shelf through image acquisition equipment, establishing a motion path of the acquisition equipment through a big data technology after the data is acquired, and identifying the motion path; 2, processing the image by using computer vision and a deep learning algorithm, extracting key information, analyzing data after key data is extracted, and building a storage database after data analysis; and step 3, reference data and verification rules of the standard information are set in advance, an analysis result is compared with the standard information in the storage database, and an inspection report is generated and comprises a stockout alarm, a price error and a display error. According to the invention, uninterrupted inspection work can be realized, the inspection speed is far higher than that of manual inspection, the accuracy during inspection is higher, human negligence and errors are reduced, data are objective and reliable, and quantification can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of shelf inspection, in particular to a shelf inspection method and system based on machine vision and a storage medium. BACKGROUND

[0002] Shelf inspection refers to a work process of regularly or irregularly checking, collating and counting stored goods according to certain plans, processes and standards, and the fundamental purpose is to ensure that the state, quantity, position and information of the goods are consistent with the expectations or records, so as to ensure the accuracy, safety and efficiency of the warehousing, retail or logistics link. In the conventional inspection process, the inspection method used is mostly manual inspection, which is not only low in efficiency but also high in cost. Moreover, the results of manual inspection are difficult to conveniently organize into an inspection report and store, and when traceability processing is required later, effective traceability cannot be achieved. SUMMARY

[0003] The present application aims to provide a shelf inspection method and system based on machine vision and a storage medium to solve the problems raised in the background.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical solution: a shelf inspection method based on machine vision, comprising the following steps: Step 1: Obtain visual data of the shelf through an image acquisition device, and after obtaining the data, build the motion path of the acquisition device through big data technology and identify the motion path; Step 2: Process the image using computer vision and deep learning algorithms to extract key information, analyze the data after extracting the key data, and build a storage database; Step 3: Set the standard data and verification rules of the standard information in advance, compare the analysis results with the standard information in the storage database, and generate an inspection report including out-of-stock alerts, price errors and display errors.

[0005] Preferably, the image acquisition device in step 1 includes an inspection robot, a drone or a handheld device. The inspection robot can autonomously move along the shelf, stop in front of each shelf according to the preset motion path, trigger the camera to take pictures, and usually take multiple pictures in succession to splice into a full shelf view. Preferably, the area data of the collection area and the position of the shelf in the area are collected in advance before the motion path is set. After the collection is completed, the area is divided into N regions. The internal shelves of each region are identified according to the preset rules. After the identification is completed, a motion path is developed for each region, and the path is identified using N paths. Preferably, the positioning and navigation used by the mobile robot in step one adopts a simultaneous localization and mapping technology and a two-dimensional code or machine vision calibration marker system; The imaging system comprises a 2D camera, a 3D camera or a depth camera and an illumination system, the 2D camera is used for commodity identification and out-of-stock detection; the 3D camera or the depth camera is used for acquiring depth information to determine the number of commodities, detect the thickness of the row of stacked commodities and identify the partially occluded commodities; Preferably, in step two, distortion correction, brightness or contrast adjustment, deblurring and the like are performed on the image, while the target is detected by a core algorithm module, image classification and optical character recognition and the like are performed after detection, and a business logic module sets the inspection rules; the data stored in the storage database includes storage of commodity standard images, inventory unit information, plan and price information and the like, after data storage is completed, each piece of data is identified by a time stamp, classification is completed according to the identification, and when looking up later, the time stamp is queried; Preferably, in step two, a deep learning target detection model is used for target detection analysis, the model is trained in advance by labeled data, can directly detect and identify each instance of the commodities in the image, and outputs the boundary box and the category of the commodities; image classification combined with prior positions is used to segment the shelf image to obtain the storage grid, and the segmented storage grid image is cropped out and sent to an image classification model to judge the corresponding inventory unit or whether the storage grid is empty, and when it is judged to be empty, the detection record of the storage grid is directly deleted.

[0006] Preferably, in step three, multi-angle shooting is performed by the inspection robot, a 3D camera is used to perceive from multiple angles, and an algorithm is used to predict the occluded target; an incremental learning or online learning technology is used to establish a model updating process; a lightweight neural network model is used to optimize the model structure, and cloud collaborative computing is used to offload the heavy computing tasks to the server.

[0007] Preferably, in step three, an inspection storage model is built, and the inspection storage model is specifically constructed to include a unified identifier, a mapping relationship is established, and a query limit is set; The shelf inspection system based on machine vision comprises an image acquisition unit, an analysis processing unit and a post-processing unit; The image acquisition unit acquires visual data of the shelf through an image acquisition device, builds a motion path of the acquisition device through big data technology after the data is acquired, and identifies the motion path; The analysis processing unit uses computer vision and deep learning algorithms to process the image, extracts key information, analyzes the data after extracting the key data, and builds a storage database; The post-processing unit sets reference data and verification rules of standard information in advance, compares the analysis result with the standard information in the storage database, and generates an inspection report, including an out-of-stock alarm, a price error and a display error; A computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the machine vision-based shelf inspection method according to any one of the preceding items; Compared with the prior art, the present application has the beneficial effects that the present application can realize uninterrupted inspection work, the inspection speed is much higher than manual work, the accuracy during inspection is higher, human negligence and errors are reduced, the data is objective and reliable, quantification can be realized, digital reports are generated, data insights are provided for inventory management and supply chain optimization, and the labor cost and losses caused by out-of-stock and price errors can be reduced in the long run. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0009] Figure 1 The method flowchart provided by the embodiments of the present application is provided. Figure 2 The system structure block diagram provided by the embodiments of the present application is provided. DETAILED DESCRIPTION

[0010] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0011] Please refer to Figure 1 The present application provides a technical solution: a machine vision-based shelf inspection method, comprising the following steps: Step one: acquiring visual data of the shelf through an image acquisition device, after acquiring the data, building the motion path of the acquisition device through big data technology, and marking the motion path; The image acquisition device in step one includes an inspection robot, a drone or a handheld device, the inspection robot can move along the shelf autonomously, the robot stops in front of each shelf according to the preset motion path, triggers the camera to take pictures, and usually takes multiple pictures in succession to splice into a full shelf view.

[0012] In step one, the area data of the collection area and the positions of the shelves in the area need to be collected in advance before the establishment of the motion path; after the collection is completed, the area is divided, and the divided areas are identified as first area, second area, …, and Nth area; for the shelves inside each area, they are identified according to the "area identification + serial number" rule; after the identification is completed, motion paths are developed for each area, and the paths are identified as first area path, second area path, …, and Nth area path.

[0013] In the process of dividing the shelf area, the shelf layer plate in the image is separated from the background, and each independent storage space is further segmented, and the layer plate edge is extracted by using edge detection and straight line detection; deep learning: using a semantic segmentation model to directly mark the layer plate and storage space at the pixel level.

[0014] The positioning and navigation of the mobile robot in step one use simultaneous localization and mapping technology and a two-dimensional code / machine vision positioning marker system. The imaging system includes a 2D camera, a 3D camera / depth camera, and an illumination system. The 2D camera identifies the goods and detects out-of-stock; the 3D camera / depth camera is used to obtain depth information, judge the number of goods, and detect the stacking face thickness of stacked goods, and identify partially obscured goods; the illumination system: built-in LED fill light, suitable for different area lighting conditions to obtain clear and stable images, reduce environmental interference during collection.

[0015] Step two: using computer vision and deep learning algorithms to process images and extract key information, and after extracting key data, analyzing the data, and building a storage database after data analysis; In step two, the image is corrected for distortion, brightness / contrast is adjusted, and deblurring is performed, and the target is detected using a core algorithm module, and after detection, image classification and optical character recognition are performed, and the rules of the inspection are set through a business logic module. The data stored in the storage database includes storage of standard images of goods, inventory unit information, plan and price information, etc. After storage is completed, each piece of stored data is identified, and the identification method uses a timestamp to identify, and through identification, classification is performed, and in the later stage, the timestamp is used for query.

[0016] In the step two, a deep learning target detection model is used to directly detect and identify each instance of the goods in the image during target detection analysis, the deep learning target detection model is trained in advance through labeled data, and can output the boundary box and category of the goods, image classification + prior location, the storage location is obtained through segmentation, and the image of the segmented storage location is cropped out and sent to an image classification model to determine which inventory unit it is or whether it is empty, and when it is determined to be empty, it is directly deleted. In the step two, the out-of-stock and inventory are detected, and the detected specific inventory units are directly counted after the detection target is output; the empty storage location is judged; the depth camera data is used to estimate the thickness of the goods arrangement surface and the remaining inventory quantity through 3D depth information.

[0017] In the step two, price recognition analysis is performed, which is realized through optical character recognition, specifically, the optical character recognition technology is used to recognize the numbers on the price tag in or near the boundary box of the goods detection, and the recognized price is compared with the standard price in the storage database, and an alarm is given when the comparison result is inconsistent; the recognized goods position and quantity are compared with the standard display plan through display compliance analysis; appearance detection is performed to check whether the goods packaging is damaged or expired, and the staff is directly notified for processing when damage or expiration is found.

[0018] Step three: the standard information baseline data and verification rules are set in advance, the analysis result is compared with the standard information in the storage database, and a patrol inspection report is generated, including out-of-stock alarm, price error and display error.

[0019] In the step three, a problem processing model is developed, which specifically includes the problem of occlusion: specifically, the front row of goods occludes the back row of goods; the solution is to use a 3D camera to perceive from multiple angles, and the algorithm predicts the occluded target, so that the patrol robot can take multiple angle photos; the problem of light change: specifically, uneven light, reflection and shadow in the working area; the solution is to equip with active light source, use high dynamic range imaging technology, and enhance the data in the algorithm training data; the problem of large number of inventory units and fast update: specifically, new products are put on the shelf, old products are taken off the shelf, and the model needs to be updated continuously; the solution is to establish an efficient model updating process; use incremental learning or online learning technology; the problem of computing resources and real-time performance: specifically, the algorithm needs to be run in real time on an embedded device or edge server; the solution is to use a lightweight neural network model, optimize the model structure, and use cloud collaborative computing to offload heavy computing tasks to the server.

[0020] Step three involves building an inspection storage model. The specific construction steps of this model include: a unified identifier, generating a globally unique ID for each inspection, specifically the name of the directory in object storage, the foreign key of all relevant records in the relational database, and the unique identifier of the report file; establishing a mapping relationship, where a table in the relational database stores the web address of the report file, allowing easy retrieval of the corresponding original image and detailed results via the ID; and setting query limits, which typically involves defining the data type, storage time, keywords within the data, and ID name. Query limit settings can significantly optimize high write speeds, high compression, and fast aggregation queries for time-series data, satisfying both storage and query monitoring requirements.

[0021] In addition, this application also proposes a machine vision-based shelf inspection system, such as... Figure 2 As shown. The system includes an image acquisition unit, an analysis and processing unit, and a post-processing unit; The image acquisition unit acquires visual data of the shelf through the image acquisition device. After acquiring the data, it constructs the movement path of the acquisition device through big data technology and identifies the movement path. The analysis and processing unit uses computer vision and deep learning algorithms to process images, extract key information, analyze the extracted key data, and build a storage database. The post-processing unit pre-sets the benchmark data and verification rules for standard information, compares the analysis results with the standard information in the stored database, and generates an inspection report, including out-of-stock alerts, price errors, and display errors. In addition, this application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the machine vision-based shelf inspection method as described in any of the preceding claims.

[0022] Compared with existing technologies, the beneficial effects of this invention are that it can achieve uninterrupted inspection work, with an inspection speed far exceeding that of manual inspection, and higher accuracy during inspection, reducing human negligence and errors. The data is objective and reliable, and can be quantified. By generating digital reports, it provides data insights for inventory management and supply chain optimization. In the long run, it can reduce labor costs and losses caused by stockouts and mispricing.

[0023] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0024] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method of shelf inspection based on machine vision, characterized in that The method comprises the following steps: Step 1: Obtain visual data of the shelf through an image acquisition device. After obtaining the data, build the motion path of the acquisition device through big data technology, and identify the motion path; Step 2: Process the image using computer vision and deep learning algorithms to extract key information. After extracting the key data, analyze the data and build a storage database after the data analysis; Step 3: Set the standard information benchmark data and verification rules in advance. Compare the analysis results with the standard information in the storage database to generate an inspection report, including out-of-stock alerts, price errors, and display errors. 2.The machine vision based shelf inspection method of claim 1, wherein: The image acquisition device in step 1 includes an inspection robot or a camera. The inspection robot can move autonomously along the shelf, stop in front of each shelf according to the preset motion path, trigger the camera to take pictures, and usually take multiple pictures in succession to splice into a full shelf view.

3. The machine vision-based shelf inspection method of claim 2, wherein: Before setting the motion path in step 1, the area data of the collection area and the position of the shelf in the area need to be collected in advance. After the collection is completed, the area is divided into N regions. The internal shelves of each region are identified according to the preset rules; After the identification is completed, the motion path for each region is developed, and the path is identified using N paths.

4. The machine vision-based shelf inspection method of claim 3, wherein: The positioning and navigation of the mobile robot in step 1 use simultaneous localization and mapping technology and a two-dimensional code or machine vision positioning marker system; The imaging system includes a 2D camera, a 3D camera or a depth camera, and an illumination system. The 2D camera is used for product identification and out-of-stock detection; the 3D camera or depth camera is used to obtain depth information to determine the number of goods, detect the stacking face thickness of stacked goods, and identify partially obscured goods.

5. The machine vision-based shelf inspection method of claim 4, wherein: In step 2, the image is subjected to distortion correction, brightness or contrast adjustment, deblurring, etc. At the same time, the target is detected through the core algorithm module, and after detection, image classification and optical character recognition are performed, and the inspection rules are set by the business logic module; The data stored in the storage database includes storage of standard product images, inventory unit information, plan and price information, etc. After data storage is completed, each piece of data is identified using a timestamp, and classification is completed based on the identification. In the later stage, the timestamp is queried to find.

6. The machine vision-based shelf inspection method of claim 5, wherein: In step 2, when performing target detection analysis, a deep learning target detection model is used. This model is trained in advance using labeled data, which can directly detect and recognize each instance of a product in an image, while outputting the bounding box and category of the product. Using image classification combined with prior positions, the shelf image is segmented to obtain the storage grid, and the segmented storage grid image is then cropped and sent to an image classification model to determine the corresponding inventory unit or whether the storage grid is empty. When it is determined to be empty, the detection record of the storage grid is deleted directly.

7. The machine vision-based shelf inspection method of claim 1, wherein: The step three further comprises multi-angle shooting by the inspection robot, sensing from multiple angles by the 3D camera, and predicting the occluded target through the algorithm; a model updating process is established, incremental learning or online learning technology is adopted; a lightweight neural network model is adopted, the model structure is optimized, cloud collaborative calculation is adopted, and the heavy calculation task is unloaded to the server.

8. The machine vision-based shelf inspection method of claim 7, wherein: The inspection storage model in the step three is built, and the inspection storage model is specifically built, comprising a unified identifier; A mapping relationship is established; and a query limit is set.

9. A machine vision based shelf inspection system characterized in that: The shelf inspection system is applicable to the shelf inspection method based on machine vision in any one of claims 1-8, comprising an image acquisition unit, an analysis processing unit and a post-processing unit; The image acquisition unit obtains visual data of the shelf through an image acquisition device, builds a motion path of the acquisition device through big data technology after obtaining the data, and identifies the motion path; The analysis processing unit uses computer vision and deep learning algorithm to process the image, extracts key information, analyzes the data after extracting the key data, and builds a storage database; The post-processing unit sets the standard data and verification rules of the standard information in advance, compares the analysis result with the standard information in the storage database, generates an inspection report, and includes an out-of-stock alarm, a price error and a display error.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the guest flow statistical method based on deep learning in any one of claims 1-8.

Images