Automated method for adjusting the field of view parameters of an in-store imaging device.

Abstract

The present invention relates to a method for controlling an imaging device having at least one controllable field of view parameter and configured to capture images of an item placement tool, the method comprising the steps of: continuously capturing images (CAPTx) of the item placement tool using the imaging device, the capturing step including a step of controlling the at least one controllable field of view parameter to be modified between two successive image captures; determining an instrument coverage score for each of the successively captured images (Sfc); selecting a reference image from the successively captured images as the image having the best instrument coverage score (SLC); and setting the at least one controllable field of view parameter to that used when capturing the reference image.

Description

[[Technical Field]]

[0001] The field of the present invention is the field of computer vision-based inventory management systems and methods for warehouses or retail stores. More specifically, the present invention relates to capturing satisfactory images of fixtures mounted with shelf devices. [[Background Art]]

[0002] Retail store shelves are generally organized into gondolas. Each gondola comprises a plurality of rows, and each row comprises a plurality of shelf labels located adjacent to articles. The shelf labels are arranged along the front edge of the shelf and display information related to the articles offered for sale, such as price, price per weight, and the name of the article.

[0003] It is widely known to use electronic shelf labels (hereinafter referred to as "ESL") on shelves in order to enable easy and quick updating of article information and reduce operational costs. The product information displayed on the screen of one ESL is remotely controlled via radio frequency.

[0004] For example, when a gondola is re-ordered, or in the case of seasonal products, the position of articles for sale on the shelf may change over time. In recent years, efforts have been made to create "rearograms", that is, planograms that provide a realistic view of the front of the gondola. Rearograms take into account any changes made to the allocation of shelf space to articles, or changes made to the allocation of ESLs to articles. A rearogram therefore means a reliable and up-to-date representation of the actual shelf visible to customers in a sales area.

[0005] Providing a reliable and complete realogram enables the development of several useful applications, such as specific promotional content that takes into account the customer's position in front of the shelf or the geolocation of items within the sales area, thereby accelerating product replenishment and / or picking by sales area personnel. To monitor the layout of items within the shelves and complete the realogram, an option is to install an imaging system in the sales area. The imaging system preferably provides a real-time view of the shelves in the sales area. Based on the images or videos provided by the imaging system, image processing methods are performed for several applications, including automatic detection of empty shelf space, automatic detection of ESLs, and verification of the fit between the actual and expected faces of items.

[0006] A commonly used imaging system is a camera fixed to the ceiling or other strategic location in the sales area. Each camera is positioned so that its field of view faces the gondola. Another imaging system described in international application WO2021 / 009244 A1 is a camera that can be mounted directly to a shelf, similar to an ESL, and is therefore capable of capturing images of the gondola it faces.

[0007] When a new camera is installed in a sales area, or when a camera is moved to a new location within a sales area, personnel in the sales area may use a mobile device capable of interacting with the inventory management server via a wireless connection. The mobile device may be further configured to establish short-range communication, such as NFC (Near Field Communication), with the camera in order to register the camera with the inventory management server by providing the camera with initial settings (such as a store identifier and authentication information for connecting to the inventory management server). The mobile device may also be used to send calibration commands to the camera to adjust the camera's controllable field of view parameters, such as the camera's field of view direction.

[0008] In any case, since this adjustment is performed by humans, it is inevitably prone to errors, especially when the calibration process must be repeated for multiple cameras. As a result, the images captured by the cameras may turn out to be not satisfactory enough for the image processing method to function correctly.

[0009] Furthermore, after installation, cameras may be moved accidentally, for example by the client, resulting in deviations from the initial settings and the capture of inappropriate images. [Overview of the Initiative]

[0010] The present invention aims to overcome at least one of the aforementioned drawbacks.

[0011] For this purpose, the present invention relates to a method for controlling an imaging device having at least one controllable field of view parameter and configured to capture an image of an object placement apparatus. The method comprises the following automated setup steps, i.e., — A step of continuously acquiring images of the article placement device using the imaging device, wherein the acquisition step includes controlling the at least one controllable field of view parameter so as to be modified between two consecutive image acquisitions, — A step of determining the instrument coverage score for each of the sequentially acquired images, — A step of selecting a reference image from among the sequentially acquired images having the best instrument coverage score, — The process includes the step of setting the field of view parameter used when acquiring the reference image to the at least one controllable field of view parameter.

[0012] Accordingly, the present invention provides an automatic setup for a camera configured to capture images of an item placement fixture, the camera having adjustable field of view parameters so that the images captured by the camera are of sufficient quality for the intended use in a computer vision-based inventory management method, in particular in terms of providing a satisfactory scope of application for the fixture.

[0013] A preferred but non-limiting embodiment of this method is as follows:

[0014] The step of determining the instrument coverage score of one of the sequentially acquired images includes the steps of identifying the instrument in the image, determining an instrument matching polygon that demarcates the identified instrument, and calculating the deviation of the instrument matching polygon from the center alignment in the image.

[0015] The neural network processing unit determines the equipment covering score for each of the sequentially acquired images.

[0016] The step of determining the equipment coating score of one of the sequentially acquired images includes: identifying a label and / or product in the image; determining a point cloud having points corresponding to the identified label and / or product; and calculating the deviation of the determined point cloud from the center alignment in the image.

[0017] The method further includes the step of transmitting the continuously acquired images to an image processing server configured to perform the step of selecting the reference image from the imaging device.

[0018] The method further includes, once the reference image is selected, sending a setting command to the imaging device to cause the image processing server to set the field of view parameters used when acquiring the reference image to the at least one controllable field of view parameter.

[0019] Said method further comprises the following automated operating steps, namely: - capturing a current image of said article placement apparatus using said imaging device; - calculating a bias between said current image and a pre-evaluated image; - determining an adjustment to be performed on said at least one controllable field of view parameter based on said bias; - controlling said at least one controllable field of view parameter based on said determined adjustment.

[0020] Said step of calculating said bias comprises comparing said current image with said pre-evaluated image, and calculating a similarity score.

[0021] Said step of calculating said bias comprises determining an apparatus coverage score of said current image, and comparing said apparatus coverage score of said current image with said apparatus coverage score of said pre-evaluated image.

[0022] Said method further comprises re-performing said automated setting step when said bias exceeds a calibration threshold.

[0023] Said method further comprises, after said step of controlling said at least one controllable field of view parameter, said step of capturing a new image of said article placement apparatus.

[0024] Said at least one controllable field of view parameter comprises a field of view direction of said imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other aspects, objects, advantages and features of the present invention will become more apparent upon reading the following detailed description of preferred embodiments of the present invention, which is provided by way of non-limiting example and made with reference to the accompanying drawings.

[0026] [Figure 1] It is a block diagram showing steps executed in a setting phase of a camera according to the present invention. [Figure 2] It shows a camera and an image processing server that can be involved in the setting phase according to Figure 1. [Figure 3] It is a block diagram showing steps implemented in an operation phase of a camera according to a possible embodiment of the present invention. [Figure 4] It shows a camera and an image processing server that can be involved in the operation phase according to Figure 3. [Figure 5] It shows a first image of an appliance captured by a camera, together with its corresponding appliance matching polygon. The first image does not provide satisfactory coverage of the appliance. [Figure 6] It shows a second image of an appliance captured by a camera, together with its corresponding appliance matching polygon. The second image provides satisfactory coverage of the appliance. [Figure 7] It shows a third image of an appliance captured by a camera, together with its corresponding appliance matching polygon. The polygon is cropped. DETAILED DESCRIPTION OF EMBODIMENTS FOR CARRYING OUT THE INVENTION

[0027] The present invention relates to an automated method for controlling an imaging device having at least one controllable field-of-view parameter and configured to capture images of article placement appliances arranged in a warehouse or a retail store. The at least one controllable field-of-view parameter may include an adjustable focal length and / or an adjustable field-of-view direction. The imaging device may, for example, comprise a motorized lens that can be rotated by an actuator about one or more rotation axes, for example about one horizontal rotation axis, to enable driving the field-of-view direction vertically up and down perpendicular to the ground.

[0028] In a preferred embodiment, the imaging device is configured to be removably and securely fitted into a receiving portion of a shelf support located on the shelf edge of the shelf facing the article placement fixture. Preferably, the article placement fixture comprises a shelf equipped with a shelf device such as an ESL.

[0029] The method according to the present invention includes a setup step which may be performed as an initial automatic setup process when an imaging device is newly installed facing an article placement fixture.

[0030] Referring to Figure 1, the automatic setup stage includes a first setup step CAPTx in which images of the article placement fixture are successively acquired using an imaging device, the acquisition including controlling at least one controllable field of view parameter to be modified between two successive image acquisitions. As an exemplary example, ten images of the article placement fixture may be successively acquired while modifying the field of view parameter of the imaging device. For example, the successive images may be acquired while changing the field of view direction of the imaging device. For this purpose, the field of view direction may be increased by, for example, 10° between two successive images, for example, vertically raised from bottom to top.

[0031] Referring further to Figure 1, the setup step includes a second setup step Sfc which determines an instrument coverage score for each of the sequentially captured images. The instrument coverage score of an image represents the degree to which the image covers the instrument, thus providing proper capture of the instrument.

[0032] In one embodiment, determining the instrument coverage score of one image from a series of captured images comprises identifying an instrument in the image, determining an instrument matching polygon that demarcates the identified instrument, and calculating the deviation of the instrument matching polygon from the center alignment in the image. The area of ​​the feature matching polygon may also be considered in determining the instrument coverage score, with significant areas in the image leading to a better score.

[0033] In a preferred embodiment, determining the instrument coverage score is performed by a neural network processing unit. The neural network processing unit may be pre-trained for this task, for example based on supervised learning, to determine instrument-matching polygons in an image of the instrument and to estimate the degree of center alignment of the instrument-matching polygons in the image. The neural network processing unit may also be trained to identify shelves and to estimate the horizontality of the identified shelves as a measure of the instrument coverage score. In one embodiment, the neural network processing unit comprises a first neural network dedicated to detecting shelves in the captured image and having their polygonal contours, and a second neural network dedicated to detecting shelf levels having their full length. The determined instrument-matching polygons are quite complex and may, in some cases, be slightly too large or too small. Identifying shelf levels helps to compensate for these defects, thereby improving the accuracy of the matching polygons.

[0034] Figures 5, 6, and 7 show different images I1, I2, and I3 of the instrument captured by the camera, along with their corresponding instrument matching polygons P1, P2, and P3. Taking only vertical alignment as an example, each of these images I1, I2, and I3 can be separated into two equal parts by the image horizontal central axis MA, and calculating the deviation of the fixture matching polygons from the center alignment in the image may involve determining the geometric features B1, B2, and B3 of the feature matching polygons P1, P2, and P3 (for example, the centroid along the vertical axis in this example) and calculating the distance between the geometric features B1, B2, and B3 and the image horizontal central axis MA. In Figures 5 and 7, the centroids B1 and B3 are spaced away from the image mid-axis so that the feature matching polygons P1 and P3 are not considered to be centered. In contrast, in Figure 6, the centroid B2 is perfectly fitted on the image horizontal mid-axis MA, and the feature matching polygon P2 is considered to be centered.

[0035] Referring again to Figure 1, the setup phase includes a third setup step SLC, which selects a reference image from among the sequentially captured images as the image having the best instrument coverage score, i.e., the image that is estimated to provide the best capture of the instrument (for example, because the entire instrument is captured in the central part of the image). The setup step includes a fourth setup SET, which sets at least one controllable field of view parameter to the one used when capturing the reference image.

[0036] In possible embodiments, selection is performed only for images having a fixture coverage score that exceeds a selection threshold. This allows for the ignoring of feature matching polygons that are clearly not associated with a suitable image, such as polygon P3 in Figure 7, which do not include the entire fixture but rather have a cut-off fixture (for example, because the line of sight is too low or too high to capture the entire fixture).

[0037] In another embodiment, which may be implemented independently of or in conjunction with the instrument matching polygon embodiment, determining the instrument coverage score of one of the sequentially captured images includes identifying a label and / or product in the image and, based on the identified label and / or product, calculating the deviation of the instrument from center alignment in the image. When implemented in conjunction with the fixture matching polygon embodiment, this embodiment may be useful for automatically correcting the determined matching polygon.

[0038] Identifying labels and / or products within an image may include identifying points corresponding to the labels and / or products, such as their center points. Labels may include light indicators, and identifying labels within an image may include, for example, detecting the blinking of light indicators using a neural network. The identified points form a point cloud, and calculating the deviation of the instrument from its center alignment within the image may include determining the geometric features of the point cloud (e.g., the centroid along the vertical axis in one example) and calculating the distance between the geometric features and the horizontal mid-axis of the image.

[0039] The above example illustrates the setting of the viewing angle from bottom to top. Those skilled in the art will understand that the present invention is not limited to this vertical adjustment, but extends to finer adjustments, such as adjustments along both the vertical and horizontal axes using both the horizontal and vertical mid-axis lines of the image.

[0040] Referring to Figure 2, the method may include transmitting continuously captured images IMGx from the imaging device 10 to an image processing server 20 configured to perform the selection of a reference image. As shown in Figure 2, the image processing server 20 may include a neural network processing unit 30 configured to determine the instrument coverage score for each of the continuously captured images. As also shown in Figure 2, once a reference image is selected, the image processing server 20 may send a setting command SET-Cd to the imaging device 20 to set at least one controllable field of view parameter to the one used when capturing the reference image.

[0041] The method according to the present invention may further include an automated operation step that can be performed periodically or in a specific order after the initial setup step for each newly captured image.

[0042] Referring to Figure 3, the operation steps may include a first operation step CAPTc in which an imaging device is used to capture a current image of the object placement apparatus; a second operation step BIAS in which a bias is calculated between the current image and a pre-evaluated image (initially a reference image from the setup stage); a third step DET in which adjustments to be made to at least one controllable field of view parameter based on the bias; and a fourth step ADJ in which at least one controllable field of view parameter is controlled based on the determined adjustments.

[0043] In one embodiment, the steps DET, which determine the adjustment, and ADJ, which control at least one controllable viewing parameter, are performed only if the calculated bias exceeds a deviation threshold. This avoids consuming the imaging device's battery when the current image is considered close to a pre-evaluated image and therefore sufficiently satisfactory.

[0044] In one embodiment, calculating the bias includes comparing the current image with a pre-evaluated image and calculating a similarity score.

[0045] In another embodiment, calculating the bias includes determining the instrument coverage score of the current image (in the same manner as performed in the setup phase) and comparing the instrument coverage score of the current image with the instrument coverage score of a pre-evaluated image. In this embodiment, as in the setup phase, the instrument coverage score may be calculated by a neural network processing unit.

[0046] In both of these embodiments, the calculated bias may be compared to a threshold, and if the bias exceeds the calibration threshold, the setting step may be repeated. In addition, the method may include comparing the bias to a first calibration threshold and a second calibration threshold that is greater than the first calibration threshold. If the bias then exceeds the second calibration threshold, the setting step is repeated by sequentially capturing the same number of images as in the initial setup stage; on the other hand, if the bias exceeds the first calibration threshold but does not exceed the second calibration threshold, the setting step is repeated by sequentially capturing a reduced number of images compared to the initial setup stage.

[0047] In addition, the labels and / or product point clouds described above can be used in both of these embodiments. More specifically, calculating the bias may involve comparing the point cloud determined for the current image with the points determined for the pre-evaluated image, and then controlling the field parameters to match the point clouds.

[0048] The method may further include, after the step of controlling the ADJ of at least one controllable field of view parameter, capturing a new image of the article placement fixture and updating the pre-evaluated image to the new image. By replacing the pre-evaluated image with the new image, the method avoids the increase in errors over time that would otherwise be observed if the reference image in the setup stage were retained as the pre-evaluated image, and thus the entire process remains stable over time.

[0049] Referring to Figure 4, the operational step may include sending the current image IMGc from the imaging device 10 to the image processing server. As shown in Figure 4, the image processing server 20 may include a neural network processing unit 30 configured to determine the instrument coverage score of the current image. As also shown in Figure 4, once the adjustments to be performed are determined, the image processing server 20 may send an adjustment command ADJ-Cd to the imaging device 20 so that at least one controllable field of view parameter can be matched to the one used when capturing the pre-evaluated image.

[0050] The present invention is not limited to the methods described above, but also extends to the image processing servers described above. Such an image processing server has at least one controllable display parameter and can communicate with an imaging device configured to capture images of an object placement apparatus. The image processing server further comprises a processing unit configured, in particular, to perform a setting step of determining the Sfc in the setting phase, selecting the SLC, and setting the ST, and / or performing an operation step of calculating the BIA in the operation phase, determining the DET, and controlling the ADJ.

[0051] The present invention further extends to a computer program product that, when the program is executed by the computer, includes instructions that cause the computer to perform a setup step to determine Sfc, select SLC, set the ST in the setup phase, and / or perform an operation step to calculate BIA, determine DET, and control the ADJ in the operation phase.

Claims

1. An automated method for controlling an imaging device having at least one controllable field of view parameter and configured to capture an image of an object placement device, wherein the method is: — A step of continuously acquiring images of the article placement device using the imaging device, wherein the acquisition step includes controlling the at least one controllable field of view parameter so as to be modified between two consecutive image acquisitions, — A step of determining the item placement and equipment coverage score for each of the sequentially captured images, — A step of selecting the image with the best item placement and fixture coverage score from the sequentially acquired images as a reference image, An automated method comprising the step of setting the field of view parameter used when acquiring the reference image to the at least one controllable field of view parameter.

2. The automated method according to claim 1, wherein the step of determining the article placement fixture coverage score of one of the sequentially acquired images includes the steps of identifying the article placement fixture in the image, determining an article placement fixture matching polygon that demarcates the identified article placement fixture, and calculating the deviation of the article placement fixture matching polygon from the center alignment in the image.

3. The automated method according to claim 1, wherein the step of determining the article placement fixture covering score for each of the sequentially acquired images is performed by a neural network processing unit.

4. The automated method according to claim 1, wherein the step of determining the article placement fixture coverage score of one of the sequentially acquired images includes the steps of identifying a label and / or product in the image, determining a point cloud having points corresponding to the identified label and / or product, and calculating the deviation of the determined point cloud from the center alignment in the image.

5. The automated method according to any one of claims 1 to 4, further comprising the step of transmitting the continuously acquired images from the imaging device to an image processing server configured to perform the step of selecting the reference image.

6. The automated method according to claim 5, further comprising the step of sending a setting command to the imaging device to cause the image processing server to set the field of view parameters used when acquiring the reference image to the at least one controllable field of view parameter.

7. - The step of capturing a current image of the article placement device using the imaging device, — A step of calculating the bias between the current image and the pre-evaluated image, — A step of determining, based on the bias, the adjustment to be performed on the at least one controllable field of view parameter, —The automated method according to claim 1, further comprising the step of controlling the at least one controllable field of view parameter based on the adjustment determined.

8. The automated method according to claim 7, wherein the step of calculating the bias includes the steps of comparing the current image with the pre-evaluated image and calculating a similarity score.

9. The automated method according to claim 7, wherein the step of calculating the bias includes the steps of determining the article placement fixture coverage score of the current image and comparing the article placement fixture coverage score of the current image with the article placement fixture coverage score of the pre-evaluated image.

10. The steps include determining whether the calculated bias exceeds a calibration threshold, An automated method according to any one of claims 7 to 9, further comprising the steps of: continuously acquiring images of the article placement device using the imaging device; determining an article placement device coverage score for each of the continuously acquired images; selecting an image having the best article placement device coverage score from among the continuously acquired images as a reference image; and repeating the step of setting the field of view parameter used when acquiring the reference image to the at least one controllable field of view parameter.

11. The automated method according to any one of claims 7 to 9, further comprising the step of acquiring a new image of the article placement apparatus after the step of controlling the at least one controllable field of view parameter.

12. The automated method according to claim 1, wherein the at least one controllable field of view parameter includes the field of view direction of the imaging device.

13. An image processing server capable of communicating with an imaging device having at least one field of view parameter, and configured to capture images of an object placement device, comprising a processing unit configured to perform each step of the automated method described in claim 1.

14. It is a computer program, A computer program that, when executed by a computer, includes instructions causing the computer to perform each step of the automated method described in claim 1.

Images