Inventory estimation system and method based on background difference and application
By applying a background difference-based inventory estimation system in commercial refrigerators, the problems of low inventory inventory efficiency and high hardware requirements in the existing technology are solved, and high-precision and low-cost inventory monitoring effect are achieved.
Patent Information
- Application Number
- CN202411816695.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-05-02
AI Technical Summary
The inventory of existing commercial refrigerators is inefficient, and the target detection method that relies on deep learning models has problems such as insufficient data volume, poor robustness, high hardware requirements and frequent updates.
The inventory estimation system based on background difference is adopted, and the image processing module, the foreground segmentation module, the foreground processing module and the inventory estimation module are used to identify the product area and estimate the inventory quantity. The system does not require a large amount of data labeling, has low hardware requirements, strong adaptability, and can be directly deployed on terminal devices.
It significantly reduces data costs and time costs, improves the accuracy of inventory monitoring, reduces hardware costs and system maintenance complexity, and reduces operating costs.
Smart Images
Figure CN119919053A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of automation and intelligent monitoring of commercial refrigerators, and in particular to an inventory estimation system, estimation method and application based on background difference. Background Art
[0002] Commercial refrigerators refer to refrigerated or frozen freezers used to store ice cream, beverages, dairy products, quick-frozen foods, food materials, etc. in commercial channels such as supermarkets, cold drink shops, frozen food shops, hotels and restaurants. When operating, merchants need to replenish goods regularly according to the inventory of goods in the refrigerator. Manual inventory counting is inefficient, and commodity recognition methods are usually used for inventory counting and estimation. At present, the target detection method is mainly used in the field of commodity recognition, which requires the preparation of relevant training data sets. However, due to insufficient data volume, the robustness of the model is often poor. In addition, deep learning models usually have high requirements for the operating environment, large model size, and frequent updates, which makes the algorithm face high hardware requirements and frequent remote upgrades when deployed to the terminal. Summary of the invention
[0003] In order to overcome the above problems existing in the prior art, the present invention proposes an inventory estimation system, estimation method and application based on background difference.
[0004] The technical solution adopted by the present invention to solve its technical problem is: a stock estimation system based on background difference, comprising an image processing module, a foreground segmentation module, a foreground processing module, and a stock estimation module, wherein the image processing module is used to correct the captured image and intercept the target area; the foreground segmentation module is used to separate the commodity and the basket area to obtain a foreground mask image; the foreground processing module is used to process the foreground mask image obtained by the foreground segmentation module to obtain a foreground mask image; and the stock estimation module obtains the stock quantity of the commodity by measuring the volume of the commodity in the foreground mask image.
[0005] In the above-mentioned inventory estimation system based on background difference, the image processing module includes a distortion correction unit and a target area interception unit. The distortion correction unit is used to correct the barrel distortion in the captured image so that the image conforms to the actual geometric proportion; the target area interception unit is used to calibrate the effective basket area, and intercept and scale the calibrated basket area.
[0006] In the above-mentioned inventory estimation system based on background difference, the foreground segmentation module generates a difference map by comparing the image after placing goods with the image without placing goods, and determines whether each pixel belongs to the foreground or the background according to a set threshold.
[0007] In the above-mentioned inventory estimation system based on background difference, the foreground processing module uses a morphological method to optimize the image obtained by the foreground segmentation module.
[0008] The above-mentioned inventory estimation system based on background difference, the Susou inventory estimation module calculates the distance from the edge of the product area to the boundary of the basket, uses a preset depth estimation table, maps the distance to the stacking depth, calculates the pixel volume of the product and maps the pixel volume to the actual volume, and divides the total product volume by the volume of a single standard product to obtain the actual inventory quantity of the product.
[0009] A method for inventory estimation based on background difference includes the following steps: Step 1: collect image data of the products after they are placed through the camera inside the device, and correct the collected images and intercept the target area through the image processing module; Step 2, processing the image obtained in step 1 through a foreground segmentation module to obtain a foreground mask image; Step 3, performing morphological processing on the foreground mask image obtained in step 2 by a foreground processing module to obtain a foreground mask image; Step 4, on the foreground mask image obtained in step 3, obtain the edge characteristics and stacking depth information of the goods, combine the area of the goods area, calculate the pixel volume of the goods, and convert the pixel volume into actual volume, divide the total product volume by the volume of a single standard product, and get the actual inventory quantity of the goods.
[0010] In the above-mentioned inventory estimation method based on background difference, the step 2 is specifically as follows: based on multiple background images, Gaussian mixture modeling is performed on the RGB channels of each pixel, and the probability density model that best matches the background distribution is iteratively estimated through the EM algorithm; the image obtained in step 1 is input into the probability density model, and when the image of the placed goods is input, the pixel points are matched with the background model, and the pixel points that match the background Gaussian distribution are judged as the background, and the pixel points that match the foreground Gaussian or do not match any Gaussian distribution are judged as the foreground.
[0011] In the above-mentioned inventory estimation method based on background difference, the morphological processing method in step 3 includes an opening operation and a closing operation. The opening operation is used to eliminate small noise points and retain the main structure of the image; the closing operation is used to eliminate small depressions and burrs and separate adhesion areas.
[0012] An application of the above-mentioned inventory estimation system based on background difference or the above-mentioned inventory estimation method based on background difference in a commercial refrigerator.
[0013] The beneficial effects of the present invention are as follows: (1) The present invention does not require a large amount of data annotation and training: Compared with the target detection method that relies on deep learning models, the present invention realizes commodity area recognition through background difference and traditional image processing methods, without the need to collect and annotate a large amount of data, which significantly reduces data costs and time costs.
[0014] (2) Low hardware requirements and strong adaptability: Traditional deep learning models usually require high computing power and frequent algorithm updates. The present invention adopts a lightweight algorithm to achieve inventory estimation through formulas and fixed parameters. It can be directly deployed on terminal devices, has strong adaptability and low operating environment requirements, saving hardware costs.
[0015] (3) Low model update frequency: Due to the use of background difference and feature induction methods, there is no need to frequently update the algorithm or model parameters, which reduces the complexity and cost of system maintenance and avoids frequent remote updates.
[0016] (4) Accurate area recognition and inventory estimation: Through image distortion correction, target area capture and morphological operations, the accuracy of product area recognition is ensured. The stacking depth of the product is calculated by combining the distance between the edge of the product and the edge of the basket, which can more accurately estimate the inventory ratio and effectively improve the accuracy of inventory monitoring.
[0017] (5) Lower operating costs: Compared with cloud computing, the method of the present invention runs directly on the terminal, reducing the dependence on network bandwidth and cloud servers, saving traffic and cloud computing costs, and providing operators with a more cost-effective solution. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a schematic diagram of the process of the present invention Figure 2 It is a schematic diagram of foreground segmentation of the present invention; Figure 3 It is a schematic diagram of establishing the background model of the present invention; Figure 4 is a top view of a basket in an embodiment of the present invention; Figure 5 Schematic diagram of the mapping function between basket height and basket width in an embodiment of the present invention. DETAILED DESCRIPTION
[0019] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0020] The present invention discloses an inventory estimation system based on background difference, including an image processing module, a foreground segmentation module, a foreground processing module and an inventory estimation module. Through modular design, the image data of goods in the refrigerator can be efficiently processed and the inventory can be estimated. The overall process is as follows: Figure 1 As shown, the specific module functions and inventory estimation methods are described as follows.
[0021] Image processing includes two parts: distortion correction and target area capture: 1. Distortion correction: Use the calibration method to correct barrel distortion of the captured image to make the image content more consistent with the actual scene and ensure the accuracy of subsequent analysis.
[0022] 2. Target area capture: Since the position of the horizontal cabinet camera is fixed, the target area is captured using preset coordinate points. The affine transformation matrix is calculated through the coordinate points, and the image is transformed and cropped to only display the effective area where the goods are actually stored in the refrigerator.
[0023] 1. Image processing module First, take pictures of the market inventory with a fixed camera. Since the directly taken pictures are limited by the camera shooting angle and fisheye lens distortion, they cannot be used directly, so image processing is required first. The first step for the image to enter the processing flow is correction. The chessboard calibration method of OpenCV is used to correct the image distortion, and the barrel distortion in the captured image is corrected to make the image conform to the actual geometric proportion. Secondly, according to the shape structure and layout characteristics of the refrigerator, the effective basket area is calibrated, and the area is intercepted and scaled to exclude invalid areas (such as cabinet doors, surrounding backgrounds, etc.) to reduce interference and optimize the accuracy of subsequent processing steps.
[0024] 2. Foreground Segmentation Module In order to separate the product and basket areas and achieve effective foreground segmentation, a background difference method (such as Figure 2 As shown in Figure 1). By comparing the image with the products placed and the image without the products placed (empty basket image), a difference image is generated, and then each pixel is judged as foreground or background based on the set threshold. In order to improve the adaptability of the model and avoid the interference of lighting changes or shadows caused by a single background image, a background model composed of multiple background images is constructed. Specifically, based on multiple background images, Gaussian mixture modeling is performed on the RGB channels of each pixel, and the probability density model that best fits the background distribution is iteratively estimated through the EM algorithm. The background model is established as follows: Figure 3 When an image of products is input, the pixels are matched with the background model: pixels that match the background Gaussian distribution are judged as the background, and pixels that match the foreground Gaussian or do not match any Gaussian distribution are judged as the foreground. This method can obtain a more accurate foreground mask image, providing reliable segmentation results for subsequent processing.
[0025] 3. Foreground processing module The generated binary foreground mask may produce noise and misjudgment due to factors such as light and reflection, so further post-processing is required. Morphological methods are used for optimization, including opening and closing operations:
[0026] The opening operation is used to remove small noise points and retain the main structure of the image. The specific process is erosion followed by dilation: erosion removes small noise points and isolated pixels in the foreground, and dilation restores the overall shape and size of the foreground object.
[0027] The closing operation fills small holes in the segmentation result by dilating first and then corroding, eliminating small depressions and burrs on the boundary and separating the adhesion areas.
[0028] After completing the morphological processing, the interior of the refrigerator is divided into multiple sub-areas based on the preset basket area boundary lines, and the foreground of each sub-area is processed separately.
[0029] 4. Inventory estimation module Based on the extracted foreground mask image, the edge characteristics and stacking depth information of the product are further obtained. By calculating the distance from the edge of the product area to the border of the basket, the pixel distance is converted into the stacking height using the preset height estimation function. Figure 4 As shown in FIG. 1 , the height mapping of the basket is presented as the width of the basket edge in the image. The specific height mapped to the position of the edge in the image can be obtained through actual measurement, and a height estimation function can be fitted according to the corresponding mapping points. In this embodiment, the mapping function is fitted by measuring the width of the basket in the image and the actual depth of the basket. Figure 5 As shown, Figure 5 The horizontal axis is the width of the basket in the picture (pixel distance), and the vertical axis is the actual depth of the basket (actual height). The fitted height estimation function is y=4.34527990e-08*x 3 -1.00040377e-04*x 2 + 1.09517865e-01*x +1.54591436e-01. According to the height estimation function, the pixel distance is measured to obtain the actual stacking height of the product.
[0030] Combined with the area of the product area, the pixel volume of the product is calculated; then the pixel volume is converted into the actual volume based on actual measurement. Finally, the actual inventory quantity of the product is obtained by dividing the total product volume by the volume of a single standard product.
[0031] In another specific embodiment, a ruler is attached to the wall of the basket, and the depth of the basket can be displayed on the photo taken by the camera.
[0032] In this embodiment, the specific estimation method of the inventory estimation system is: take inventory pictures through a camera. Since the directly taken pictures are limited by the camera shooting angle and fisheye lens distortion, they cannot be used directly, so image processing is required first. Image processing includes two parts: distortion correction and target area capture:
[0033] 1. Distortion correction: Use the calibration method to correct barrel distortion of the captured image to make the image content more consistent with the actual scene and ensure the accuracy of subsequent analysis.
[0034] 2. Target area capture: Since the position of the horizontal cabinet camera is fixed, the target area is captured using preset coordinate points. The affine transformation matrix is calculated through the coordinate points, and the image is transformed and cropped to only display the effective area where the goods are actually stored in the refrigerator.
[0035] In the foreground segmentation stage, the foreground area is obtained by differentiating the actual image in the market from the preset background model. By setting a threshold, each pixel is judged as foreground or background, and a binary foreground mask is generated, in which the white area represents the product (foreground) and the black area represents the background. The background model is modeled by a Gaussian mixture model (MOG), which is pre-trained using a certain amount of background images without products. Compared with the average method, the MOG model is more background-adaptive and can more accurately distinguish between background and products.
[0036] Next, the system performs morphological operations on the foreground mask, such as dilation and closing operations, to fill in small holes in the foreground template to ensure the integrity of the product edge. The complete edge contour of the product is then obtained through a contour extraction algorithm.
[0037] In the inventory estimation stage, the depth of the product is estimated based on the extracted edge contour of the product and the reference position of the freezer basket. The distance between the edge of the product and the upper edge of the basket can be used to preliminarily determine the stacking depth of the product, and the inventory ratio can be estimated based on the area information of the product. In this way, this method can achieve efficient inventory estimation without the need for additional data collection and annotation for model training.
[0038] The estimation system and estimation method disclosed in this embodiment are applied to commercial refrigerators. This method not only improves the accuracy of commodity inventory estimation in the commercial refrigerators, but also can effectively adapt to light changes and background interference in complex environments, providing technical support for the intelligent management of commercial refrigerators.
[0039] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.
Claims
1. A background difference-based inventory estimation system, characterized in that: It includes an image processing module, a foreground segmentation module, a foreground processing module, and an inventory estimation module. The image processing module is used to correct the captured image and intercept the target area; the foreground segmentation module is used to separate the commodity and basket area to obtain a foreground mask image; the foreground processing module is used to process the foreground mask image obtained by the foreground segmentation module to obtain a foreground mask image; The inventory estimation module obtains the inventory quantity of the commodity by measuring the volume of the commodity in the foreground mask image.
2. The inventory estimation system based on background difference according to claim 1, characterized in that: The image processing module includes a distortion correction unit and a target area interception unit. The distortion correction unit is used to correct the barrel distortion in the captured image so that the image conforms to the actual geometric proportion; the target area interception unit is used to calibrate the effective basket area and intercept and scale the calibrated basket area.
3. The inventory estimation system based on background difference according to claim 1, characterized in that: The foreground segmentation module generates a difference image by comparing the image after placing the goods and the image without placing the goods, and determines whether each pixel belongs to the foreground or the background according to a set threshold.
4. The inventory estimation system based on background difference according to claim 1, characterized in that: The foreground processing module uses a morphological method to optimize the image obtained by the foreground segmentation module.
5. The inventory estimation system based on background difference according to claim 1, characterized in that: The Susou inventory estimation module calculates the distance from the edge of the product area to the boundary of the basket, uses the preset depth estimation table to map the distance to the stacking depth, calculates the pixel volume of the product and maps the pixel volume to the actual volume, and divides the total product volume by the volume of a single standard product to obtain the actual inventory quantity of the product.
6. A method for inventory estimation based on background difference, characterized in that: The steps include: Step 1: collect image data of the products after they are placed through the camera inside the device, and correct the collected images and intercept the target area through the image processing module; Step 2, processing the image obtained in step 1 through a foreground segmentation module to obtain a foreground mask image; Step 3, performing morphological processing on the foreground mask image obtained in step 2 by a foreground processing module to obtain a foreground mask image; Step 4, on the foreground mask image obtained in step 3, obtain the edge characteristics and stacking depth information of the goods, combine the area of the goods area, calculate the pixel volume of the goods, and convert the pixel volume into actual volume, divide the total product volume by the volume of a single standard product, and get the actual inventory quantity of the goods.
7. The inventory estimation method based on background difference according to claim 6, characterized in that: The step 2 specifically includes: based on multiple background images, Gaussian mixture modeling is performed on the RGB channels of each pixel, and the probability density model that best matches the background distribution is iteratively estimated through the EM algorithm; the image obtained in step 1 is input into the probability density model, and when an image of placed goods is input, the pixels are matched with the background model, and the pixels that match the background Gaussian distribution are judged as the background, and the pixels that match the foreground Gaussian or do not match any Gaussian distribution are judged as the foreground.
8. The inventory estimation method based on background difference according to claim 6, characterized in that: The morphological processing method in step 3 includes an opening operation and a closing operation. The opening operation is used to eliminate small noise points and retain the main structure of the image; the closing operation is used to eliminate small depressions and burrs and separate adhesion areas.
9. Application of an inventory estimation system based on background difference as claimed in any one of claims 1 to 5 or an inventory estimation method based on background difference as claimed in any one of claims 6 to 8 in a commercial refrigerator.