A system and a method for dimension estimate of products on the shelf

Abstract

The invention relates to a system and a method that estimates the dimensions of the products in the shelf units (10) in stores and warehouses, which contain many shelves (8), shelf space (9), different types of products, product groups and repeat products (7). The dimensions of the product on the shelves (8), for which no information is known about its dimensions, are estimated by comparing it with other products on the same shelf, whose dimensions are known or estimated, based on the image taken by an image capturing device (11). For this purpose, the image processing system (12) detects the shelf (8) / shelf space (9) on the image taken of the shelf unit (10), image rectification, product and location detection within the shelves (8) or calculates the dimension of the product by taking the products, whose dimensions are known, as reference.

Description

[0001] A SYSTEM AND A METHOD FOR DIMENSION ESTIMATE OF PRODUCTS ON THE SHELF

[0002] Technical Field

[0003] The invention relates to estimating real-world dimensions of products on the shelf in the retail industry.

[0004] The invention particularly relates to a system and a method that allows the dimensions of the product on the display shelves and about which no information is known, to be estimated by comparing it with other products on the same shelf, whose dimensions are known or estimated through the image taken with the rendering device.

[0005] The State of Art

[0006] There are two basic approaches in distance and dimension measurement systems. The first of these approaches is sensor-dependent systems such as laser, LIDAR, ultrasonic, etc., and the second is the systems based on recreating the third (depth) dimension in the resulting two-dimensional image.

[0007] Sensor-dependent distance and dimension measurement systems are currently the most common solution. This solution brings processing loads such as sensor dependency, high processor usage, and data type conversions. The performance and consistency of the sensors used depend on the user. These systems, which are fed with instant data, can fail as a result of misusages. When more than one test is tried, it may give outputs very different from each other depending on the user's mistakes.

[0008] Another method is to estimate product dimensions from the image. In this method, help is taken from additional hardware to get out of the two-dimensional world of the image. For this purpose, depth information is obtained as additional data from three-dimensional cameras or stereo cameras are used where the image is taken from two different angles. Depth information, which is the third dimension, is tried to be obtained by using the difference between two images with the methods using stereo images. The size information of the object at a known / estimated distance can be estimated with the help of depth information. These approaches bring additional hardware costs and cannot be used in every environment.

[0009] The product dimension obtained in these two approaches used is an instant decision about the product being measured. In other words, it is concluded that the dimension of the product is the dimension measured at that moment. However, in real life, sensors can make errors and algorithms and sensors can produce results with a certain error tolerance. Similarly, in some cases, the front of the product being examined may be partially blocked by different objects. Even if the product can still be understood despite the partial blocking, its estimated dimensions may be affected by this blocking. Directly implementing the resulting dimension decision may refer to directly accepting and processing these errors.

[0010] The packaging of the object whose dimensions are measured may be changed by the manufacturer over time for different reasons. In cases where the change to be made in the product packaging is not excessive and the change in dimensions does not affect the recognition of the product, estimating the dimension based on a single image leads to results with a high error margin.

[0011] Mechanisms that only estimate dimensions with the image at the time of processing produce and use static results. In other words, a dimension estimate is made in the relevant image only within the scope of the data obtained from that image. This estimate is used in subsequent processes. Static data obtained from a single image in this way is extremely sensitive to the scene, rendering device, resolution, and sensor hardware.

[0012] The patent numbered TR 2015 / 02320 in the prior art relates to the retail product recognition, counting and positioning system. The system comprises a mobile application in the mobile device used to enter images of retail shelves into the system, a panorama creator, which turns images into a high-resolution shelf panorama, a product segmenter that determines and frames the location information of products, a recognition engine that recognizes segmented product images, a product verifier that evaluates the results and presents them to the user and a server with reporting elements that compiles the results in detail of market, shelf type, shelf location, product ID and quantity and turns them into summary data. In said system, products are identified by searching in an image database, but there is no mention of a calculation regarding the dimensions of the product or a structure for determining the dimensions of an unidentified product.

[0013] The application numbered CA3177901 A1 relates to a method for planning the deployment of image sensors in the store in the retail industry. In the method, the product is determined from the images of the products, while the height of the products is also calculated. As a result of the data obtained, the appropriate position for the camera to be deployed to display the shelves in the store is determined. However, in the mentioned method, product information and dimensions are taken from a database where they are already registered. There is no mention of a method to obtain such information for an unregistered product.

[0014] The patent numbered TR 2018 / 00200 comprises an electronic processing system, an image processing circuit, a product arrangement circuit and a scoring circuit in the form of an application. The image processing circuit is configured to receive an image of products presented in a presentation structure having sections, to identify the sections, and to identify a related product within each identified section. The product arrangement circuit is configured to determine an arrangement of the identified products within defined sections of the presentation structure and to compare the determined arrangement with a product arrangement template. The scoring circuit is configured to generate a product arrangement score in response to a comparison of the determined arrangement to the product arrangement template. Said system aims to place the products on the shelves in a determined and desired order. For this purpose, it uses information such as type, shape and dimension of the products by pulling them from a place where they are registered. It is not possible to detect said features from the images taken from the camera.

[0015] Similar to the patent above, the application numbered EP4293592A1 also relates to a method and a system for displaying products on shelves in the desired order. The sizes of the products are calculated by using a vector convergence technique and then by center clustering, which automatically eliminates outliers. However, in said application, there is no mention of determining shelves for image correction and applying perspective correction by using the directions of these shelves. As a result, it was deemed necessary to make a development in the relevant technical field due to the negativities described above and the inadequacy of existing solutions on the subject.

[0016] Object of the Invention

[0017] The object of the invention is to estimate the dimensions of the product on the display shelves in the retail sector, whose dimensions are not known, by comparing it with other products on the same shelf, whose dimensions are known or estimated, through the image taken with the rendering device.

[0018] The invention operates with standard two-dimensional camera images. It does not require any additional hardware. Basically, it estimates the dimension of a product whose dimensions are unknown, compared to the product next to it, which is on the same shelf as the product, whose dimensions are known.

[0019] Since the error margin of the information obtained from an estimate is relatively high, the estimates made in the method are updated as long-term estimates as different images are seen. Instead of looking at a single image result, the dimension of the recognized product is constantly updated by using the running average method. In this way, errors that may arise from different reasons in a single image (such as incorrect product recognition, partial blocking of the product, deformation effect on the packaging) will be minimized. As the number of views increases, the error margin will decrease.

[0020] The running average method is minimally affected by dimensional changes in the packaging as long as the product is identified correctly. In addition to being resistant to dimension change, as the image is seen, it has an adaptation feature as the dimension estimate makes convergence according to the last seen ones. In other words, as the number of views increases, the estimates are updated towards the final packaging size.

[0021] In photographic images, images of the three-dimensional world are visualized on a two- dimensional plane. This visual change forms a perspective difference for different parts of the image. That means, when we consider the image of a scene, for example, the dimension corresponding to 10 pixels in the middle of this image is not the same as the dimension corresponding to 10 pixels at the top of the image. Similarly, the dimensions in the middle and the right or left side are not the same. For this reason, image correction is performed to eliminate the perspective difference on the image.

[0022] For image correction, shelves are detected, and perspective correction is applied by using the directions of shelves. Starting from the idea that in the real world the shelves will be parallel to the ground and therefore to each other, visual correction is performed so that the shelf planes are parallel to each other. In this way, the perspective difference in the right, left and middle of the image, that is, in the horizontal plane, is minimized. Variations in dimension estimates of products on the same shelf but in different positions are minimized.

[0023] The structural and characteristic features and all the advantages of the invention will be understood more clearly by means of the drawings given below and the detailed description written with references to these drawings.

[0024] Description of the Drawings

[0025] Figure 1 is a front view of the shelf unit in the system, which is subject of the invention. Figure 2 is a view from the left profile of the shelf unit in the system, which is subject of the invention.

[0026] Figure 3 shows the shelf structure in the system, which is the subject of the invention.

[0027] Figure 4 is a perspective view of the shelf unit in the system, which is subject of the invention.

[0028] Figure 5 shows the data processing flow in the system, which is the subject of the invention.

[0029] Figure 6 shows the flow chart of the method, which is the subject of the invention.

[0030] Drawings do not necessarily need to be scaled, and details not necessary for understanding the present invention may be omitted.

[0031] Description of Piece References

[0032] 1. Product- 1

[0033] 2. Product-2

[0034] 3. Product-3 4. Product group- 1

[0035] 5. Product group-2

[0036] 6. Product group-3

[0037] 7. Repeat product

[0038] 8. Shelf

[0039] 9. Shelf space

[0040] 10. Shelf unit

[0041] 1 1 . Displaying device

[0042] 12. Image processing system

[0043] 13. Calculating device

[0044] 14. Database

[0045] Detailed Description of The Invention

[0046] In this detailed description, preferred embodiments of the invention are described only for a better understanding of the subject and in a way that does not form any limiting effect.

[0047] The invention relates to a system and a method that estimates the dimensions of the products in the shelf units (10) in stores and warehouses, which contain many shelves (8), a shelf space (9), different types of products, product groups and a repeat product (7). In the system, there is an image capturing device (1 1 ) that takes photographs of the shelf unit (10), shelves (8), products, product groups and repeat products at different angles. The image processing system (12), which detects the shelf (8) / shelf space (9) on the image taken of the shelf unit (10), performs the image rectification, the recognition of the products located in the shelves (8). The system (12) calculates product dimension by reference to product or products whose dimensions are known. The dimension information of the referenced products and the calculated product are kept in the database (14) and updated when necessary.

[0048] In the method applied for an example shelf unit (10) where the system of the invention is applied and whose flow is given in Figure 6, the image of the shelf unit (10) is first taken with an image capturing device (11 ). The shelf (8) and shelf space (9) are detected by operating the image processing system (12) on this image. If the shelf unit (10) to which the resulting image belongs is viewed straight ahead, the shelf spaces (9) will appear as a smooth rectangle. If the image taken is not straight ahead, perspective distortions will occur. In order to overcome this perspective distortion, image correction is performed by using the detected shelf (8) and shelf spaces (9). In the rectified view, the shelves (8) are parallel to each other on the horizontal axis. The products (1 , 2, 3), the product groups (4, 5, 6) and the repeat products (7) on the shelves (8) are also straight and upright as if seen from the straight ahead, and the difference in horizontal perspective disappears.

[0049] The products (1 , 2, 3) are objects that have different widths, heights and depths and can be found in different shapes. The product groups (4, 5, 6) represent the product groups in which these products (1 , 2, 3) are found in different numbers. The repeat products (7) are repeat products or product groups that contain the same products in different numbers on different shelves (8). The shelf (8) is a platform that is located in the shelf unit (10), allowing the display and storage of different types of products (1 , 2, 3), product groups (4, 5, 6) and repeat products (7), where the products of different dimensions and types are placed.

[0050] Product locations within the shelves (8) are detected and it is recognized which product it is by using different algorithmic modules of the image processing system (12) on the rectified shelf (8) images. The products are associated with the shelves (8) to which they belong by calculating which shelf (9) each product is placed on. The product for which the dimension calculation will be performed is compared only with the other products on the shelf space (9) it is located on. The product groups located on the shelf space (9) are formed by placing the same product side by side. Calculation is made on individual products from these product groups.

[0051] The height and width values of individual products (1 , 2, 3) and repeat products (7) occupied in the image are measured in pixels from each shelf space (9) and the product groups (4, 5, 6) on that shelf (8) with the dimension calculation module. The database (14) comprises dimension information (cm, mm, etc.) for at least one previously measured or recorded product. If the dimensions of any product located on the same shelf (9) for the individual product on the processed shelf space (9) are included in the database (14), the dimension calculation module calculates the actual dimensions of the product (cm, mm, etc.) by comparing the height and width pixel lengths. • If there is no estimate in the database (14) regarding the product groups (4, 5, 6), the estimates of these groups (4, 5, 6) are accumulated until they reach a certain number. Outlier analysis is performed within the accumulated estimates, and outliers, if any, are removed from the estimates. The average of the remaining values represents the first estimate value for the product(s) examined. This value is recorded in the database (14) as the first estimate.

[0052] • If there is an estimate value previously recorded in the database (14) regarding the product groups (4, 5, 6), an outlier analysis is performed between the instant estimate and previous estimates. If the instant estimate is too far below or too far above previous estimates, this estimate is ignored. If it is not an outlier, the running average is calculated between the previous estimates and the instant measurement. The estimate in the database (14) is updated with the newly calculated estimate.

[0053] As a result of the iterative progress of these updates, the products are converged to their actual dimensions.

[0054] In the images taken from the image capturing device (1 1 ), there is also a perspective difference in the vertical axis. This perspective difference means that the pixel-to-height ratio of different shelves (8) is also different in our proposed approach. However, it is not possible to establish a relationship between the shelves (8) arranged one above the other. The distance between each shelf (8) may be different in the visual and in the real world. For this reason, while the dimension is estimated by comparing the image, the product comparison is made individually for each shelf (8). For each shelf (8), a product located on that shelf (8) and whose dimension is unknown, is compared with another product located only on its own shelf (8) and whose dimension is known.

[0055] Under outlier analysis, the first estimate of product dimension is not given immediately at first sight. A certain number of observations about the product are expected to occur in order to form the first valid estimate. The first estimate is made after these observations, after the outliers are put out. Because it may be possible to identify the wrong product at the initial stage. Particularly, if the initial dimension estimate is due to incorrect recognition, it may take a long time for the system to correct this estimate and get closer to the correct estimate. With each image, dimension estimates for the products recognized in the image are updated. Estimating by looking at the running average in order to be least affected by packaging and dimension changes leads to more accurate estimates. While these estimates are updated, a certain number of previously observed dimension estimates of the same product are kept in the database (14) and the dimension in the current image is compared with the previous estimates. If the current estimate is too small or too large compared to previous estimates, said evaluation is ignored as it is thought that the product has been identified incorrectly or there has been a packaging change that cannot be correct.

[0056] If the allowable dimension change is small, it will cause the packaging to become less resistant to dimension change. If the allowable dimension change is large, it increases the error in correct dimension estimate due to incorrect recognition. Outlier analysis is used to prevent these errors. With this whole approach, fixed, static data turns into living data. A constantly live and updated product dimension information database (14) is obtained.

[0057] Particularly, in the systems where different images are constantly processed in a flow, dimension estimate can be done in the background for images flowing through the system. Products whose dimension information has been previously estimated and whose dimension information has been collected in the background can be used when needed.

Claims

CLAIMS1. A system that estimates the dimensions of the products in the shelf units (10) in stores and warehouses, which contain many shelves (8), shelf space (9), different types of products, product groups and repeat products (7), characterized by comprising:• an image capturing device (1 1 ) that takes photographs of the shelf unit (10), shelves (8), products, product groups and repeat products at different angles,• an image processing system (12) that detects the shelf (8) / shelf space (9) on the image taken of the shelf unit (10), image rectification, product and location detection within the shelves (8), and calculates the product dimension by reference to the product or products whose dimensions are known,• a database (14) where the reference products and the dimension information of the calculated product are kept.

2. The product dimension estimate system according to claim 1 , characterized by comprising the image processing system (12) that makes an outlier analysis between the last estimate and previous estimates in case there is an estimate recorded in the database (14) for the product whose dimension is calculated, ignores this estimate if the last estimate is much lower or higher than the previous estimates, if the last estimate is not an outlier, calculates the running average between the previous estimates and the last measurement and updates the estimate in the database (14) with the newly calculated estimate.

3. A method that estimates the dimensions of the products in the shelf units (10) by detecting the shelves (8) and shelf space (9) by the image processing system (12) from the image of the shelf unit (10) taken with an image capturing device (1 1 ), characterized by comprising the following process steps:• the image processing system (12) performs image rectification according to the detected shelves (8) and shelf spaces (9),• the product is detected on the rectified image,• the product or products whose dimensions are known, for each shelf space (9), are estimated by taking them as reference,• initial estimates for each product are accumulated for outlier analysis,the outlier values in the accumulated estimates are discarded and the remaining values are averaged, the first size estimate of the product is recorded in the database (14).

4. The product dimension estimate method according to claim 3, characterized in that if there is a record of the estimated product in the database (14):• the outlier analysis is performed between the estimate and previously recorded measurements,• the running average of the current estimate with the recorded measurements is taken,• the size estimate of the product is updated in the database (14).

Images