Systems and methods for image recognition

Abstract

An image recognition method for planogram compliance, comprising the steps of: receiving an input image of a target area to be analysed for planogram compliance; detecting one or more objects in the input image; optionally, detecting one or more gaps in the input image; calculating image similarity scores of the one or more objects detected and a plurality of product images; identifying an associated product for at least one of the one or more objects detected, based on the calculated image similarity scores, and assessing the target area based on the one or more objects detected, the one or more gaps detected and / or the associated products identified of the one or more objects.

Description

Systems and methods for image recognitionTECHNICAL FIELD

[0001] The present disclosure generally relates to image recognition techniques, and in particular, to systems and methods involving image recognition techniques for planogram compliance.BACKGROUND OF THE INVENTION

[0002] A planogram represents a plan of items displayed on shelves in bays in a retail environment such as supermarkets. It is generally used to recommend product arrangement to make products more visualisable to customers who find and purchase a particular product. This approach also assists with ensuring sufficient shelf space and stock allocation for each product. A business may decide on a planogram for planning purpose, and in store personnel then sets up the display shelves in accordance with the planogram provided.

[0003] There are various challenges and constraints for existing planogram compliance processes. For example, in a busy retail environment, products on display shelves can be disorganised due to customers’ interaction with the products. Some existing systems attempt to automate the planogram compliance process by relying on captured photos of product shelves. Factors such as background clutter, illumination, rotational invariance, deformation, occlusion, and blurriness of the photos can adversely impact the accuracy of such systems due to the systems not being configured to account for poor quality photos.

[0004] Some planogram compliance systems require complex technical infrastructure in order to analyse captured photos of the shelves. For example, the captured photos of the shelves may have to be validated overseas or by third parties, adding further processing time and costs to the planogram compliance process.

[0005] Any reference to or discussion of any document, act or item of knowledge in this specification is included solely for the purpose of providing a context for the present invention. It is not suggested or represented that any of these matters or any combination thereof formed at the priority date part of the common general knowledge, or was known to be relevant to an attempt to solve any problem with which this specification is concerned.SUMMARY OF THE INVENTION

[0006] In a first aspect, the present disclosure provides an image analysis method for planogram compliance, comprising the steps of: receiving an input image of a target area to be analysed for planogram compliance; detecting one or more objects in the input image; optionally, detecting one or more gaps in the input image;calculating image similarity scores of the one or more objects detected and a plurality of product images; identifying an associated product for at least one of the one or more objects detected, based on the calculated image similarity scores; assessing the target area based on the one or more objects detected, the one or more gaps detected and / or the associated products identified of the one or more objects.

[0007] Preferably, the method additionally comprises: detecting one or more shelves, used to support the one or more objects.

[0008] Preferably, the detected one or more shelves are used to filter out the one or more objects that are located outside of the target area.

[0009] In some embodiments, the step of detecting objects in the input image further comprises one or more of the following steps: extracting features from the input image by a convolutional neural network (CNN); passing the extracted features through a series of layers to predict class probabilities and coordinates of bounding boxes; dividing the input image into a grid of cells, wherein each cell is responsible for predicting a set of bounding boxes and class probabilities; filtering the predicted bounding boxes to remove overlapping bounding boxes and select a bounding box with the highest probability; outputting a set of predicted bounding boxes and class labels for each object in the input image.

[0010] Preferably, the step of detecting the one or more objects and the one or more shelves in the input image is carried out by a machine learning model, wherein the machine learning model is configured in a format suitable to run on a user device.

[0011] In one embodiment, the machine learning model is converted into CoreML format.

[0012] Preferably, the machine learning model is configured to run on the user device without requiring network connectivity.

[0013] Preferably, the step of detecting one or more gaps in the input image further comprises: segmenting an input image and detecting labels such as “gap” or “not a gap” to the segments, wherein a gap indicates an empty space that does not have a product.

[0014] Preferably, a segment is a mask that has a shape of a polygon.

[0015] Preferably, the step of detecting one or more gaps is carried out by a gap detection machine learning model, configured in a format suitable to run on a user device.

[0016] Preferably, the machine learning model is converted into CoreML format. Preferably, the gap detection machine learning model is configured to run on the user device without relying on network connectivity. .

[0017] Preferably, the step of calculating image similarity scores further comprises: calculating a vector representation of a detected object in the input image; calculating a cosine similarity score of the vector representation of the detected object, and the product images in the library, to product a list of similarity scores; selecting a product image that has a highest similarity score as an associated product of the object; determining product information of the associated product.

[0018] Preferably, the step of determining product information of the associated product comprises determining GTIN of the product.

[0019] In some embodiments, the input image is a photo of a retail store. .

[0020] In a second aspect, the present disclosure provides a method for estimation of planogram compliance, comprising: capturing an input image on a user device; performing the image recognition method of the first aspect, on the input image; estimating planogram compliance by comparing output of the image recognition method with a planogram.

[0021] In some embodiments, the method for estimation of planogram compliance further comprises: constructing the planogram based on intended product display in the target area.

[0022] In some embodiments, the method for estimation of planogram compliance further comprises: indicating to a user that a non-compliant condition exists, if one or more of the following condition exists: the number of detected gaps exceeds a predetermined limit, there are no overlapping detected objects, and / orthere is a mismatch between the associated product and the planogram.

[0023] In some embodiments, the method for estimation of planogram compliance further comprises: indicating to a user that planogram compliance is achieved, if the one or more identified associated products substantially match with the planogram.

[0024] In a third aspect, the present disclosure provides a user device configured to carry out the method of the first or second aspect.

[0025] In this specification, terms such as ‘comprises’, ‘comprising’, ‘includes’, ‘including’, or similar terms are intended to mean a non-exclusive inclusion, such that a method, system or apparatus that comprises a list of elements does not include those elements solely, but may well include other elements not listed.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] An example of the invention will now be described with reference to the accompanying drawings, in which:

[0027] Figure 1 illustrates an example of a planogram;

[0028] Figure 2 illustrates a logical workflow of an image recognition method in accordance with the present disclosure;

[0029] Figure 3 shows an interface of a labelling tool used for manual annotation of objects in a given image;

[0030] Figure 4 shows an interface of a labelling tool used for manual annotation of shelves in a given image;

[0031] Figure 5 shows an example of an input image of the image recognition method;

[0032] Figure 6 illustrates gaps that can be detected in the input image by the gap detection model;

[0033] Figure 7 shows an example of an image that has gaps which are manually labelled by the labelling tool;

[0034] Figure 8 illustrates an example of the U-net architecture;

[0035] Figure 9 shows how a planogram compliance report may be generated;

[0036] Figure 10 shows an interface of an application configured to let a user capture an image of a target bay;

[0037] Figure 11 shows an application including an augmented view of a planogram during image capture;

[0038] Figure 12 shows a captured image which is aligned with the augmented planogram;

[0039] Figure 13 shows compliance results generated by the user application, and

[0040] Figures 14 and 15 indicate how additional product images with different conditions may be added into the application by a user.DETAILED DESCRIPTION

[0041] A planogram represents a plan of items displayed on shelves in various bays in a retail environment, such as a supermarket. The process of ensuring compliance of product arrangement in accordance with a given planogram usually comprises the following main steps.

[0042] First, a business prepares a plan for product display in a retail environment. The plan is converted into a visual representation which indicates how products should be displayed and arranged on various shelves. This step is generally known as planogram construction. The planogram may be constructed for a particular product category, a shelf, or a bay including a number of shelves.

[0043] Next, the actual display of products is captured in one or more photos. To assess planogram compliance, the one or more captured photos are compared with the constructed planogram to determine whether the products are being displayed as planned.

[0044] The present disclosure provides image recognition techniques enabled by a plurality of machine learning models to support automation of the planogram compliance process, which also aims to improve accuracy and efficiency of the compliance process as compared to existing systems.

[0045] It will be appreciated that the image recognition method described herein may be adapted for other uses, and not limited to planogram compliance only.1. Planogram Construction

[0046] A planogram constructed by a retailer is a representation of a desired product arrangement, and typically comprises one or more of fixtures, products and product images. An example of a planogram 10 constructed by a retailer is shown in Figure 1.

[0047] A fixture is a layout of a bay 105 that includes one or more shelves 100 used to support products 103 on display. The layout may include spacings between the one or more shelves 100, also known as notches 101 . In the example illustrated in Figure 1 , the bay 105 includes five product shelves 100, each allowing a plurality of products 103 to be displayed.

[0048] A product 103 is usually characterized by one or more of the following characteristics: manufacturer, brand, product description, global trade item number (GTIN), Universal Product Code (UPC), Australian Product Number (APN), barcode, dimensions such as height, width, depth, number of facings, position on a shelf or in a bay, and similarthereof. The characterizing information of a product can be stored in a product library. The product library may additionally comprise one or more images of the products, known as product images. The extent of the visual representation of a product may depend on the actual product library used, the product library may comprise highresolution front images of products, or it may comprise additional views of the product from more viewing angles. In at least one embodiment, each product may be associated with four product images, including a front, left, right and top view of the product. Preferably, the product images included in the product library are relatively high resolution images.

[0049] The fixtures and the product information per bay can be represented in a pre-defined format, for example, in tabular format, such as a spreadsheet or in an XML document (extensible markup language). A planogram is then constructed based on information recorded in the predefined format. In one example, the constructed planogram can be an image comprising groundtruth bounding boxes, where each bounding box is a rectangle surrounding a product on a shelf, and the planogram is a visual representation of the intended product arrangement chosen by the retailer as illustrated in Figure 1 .

[0050] In one form, the bounding boxes can be defined as ground-truth data. The groundtruth data may be used to calculate a compliance score, for example, a Visibility Compliance Rating, which is a weighted representation of variables as, Ranged or Out of Stock, Compliant Product and Position (otherwise known as Planogram “Drift”), and Share of Shelf (represented as front facing Linear CM2). The weightings of the three variables as a combined score in the calculation can differ based on configurations chosen by a user. If all of the products on display match with the bounding boxes contained in the planogram, then the bay would be compliant.2. Image recognition method

[0051] One or more photos of a bay or a shelf are captured by in-field personnels using an image capturing device, such as a camera. The one or more photos then become the input images of an image recognition method in accordance with the present disclosure to support the planogram compliance process. The image recognition method assists with automated identification of objects, shelves, location of objects with respect to shelves, as well as identification an associated product for each identified object in the input images. In at least one embodiment, the image recognition method is configured to extract information such as pixel-based gaps, shelves, objects and GTIN of the products associated with the objects detected in the input images.

[0052] Figure 2 illustrates an embodiment of a logical workflow of the image recognition method 20 according to the present disclosure.2.1 Object and shelf detection

[0053] The image recognition method comprises a number of processing modules for correctly identifying products that are on display and their respective display locations.

[0054] The first module is an object and shelf detection module 201 , which receives an input image and detects one or more objects in the input image. An object 113 represents a product 103 that is on display. Additionally, the object and shelf detection module 201 is also configured to detect one or more shelves in the input image. A shelf 100 comprises a surface on which the products 103 are supported. Preferably, the detected shelves are used to filter out products that are located outside of a target bay to be analysed by the image recognition method, but are captured in an input image provided to the image recognition method. This can often occur when a target bay is positioned closely to its neighbouring bays or when shelves are stacked vertically.Similarly, the detected objects may also be used to filter out gaps detected by the image recognition method as will be described further below.

[0055] In at least one form, each of the detected objects and shelves is surrounded by a bounding box by the object and shelf detection module. In at least one embodiment, the image recognition method comprises a machine learning model, such as YOLOv5, to detect objects and / or shelves present in the input images. Other suitable machine learning models may be used as alternatives.

[0056] The operational workflow of the machine learning model may comprise the following three procedures:

[0057] Model Training - To train the object and shelf detection machine learning model, a training dataset that comprises images as well as the annotated bounding boxes with labels such as “object” or “shelf in each image is created. The training dataset may be split into an 80:20 ratio where 80% of the dataset is used fortraining purpose, and 20% of the dataset is used for evaluation purpose as will be further described below.

[0058] To enhance accuracy of the machine learning model, manual annotation of data may be required when generating the training dataset. An exemplary labelling tool is illustrated in Figures 3 and 4. This labelling tool allows a user to create rectangular shaped bounding boxes around objects 113 and shelves 100 in the training image.

[0059] During the labelling process for training the object and shelf detection model, the products 103 captured in the input image are labelled as "object," while the shelves 100 are labelled as "shelf, as indicated in Figures 3 and 4. Preferably, the annotation file uses the same data format forthe machine learning model used for object and shelf detection (for example YOLO data format). In one embodiment, the shelves 100 in a target bay are labelled, whereas the shelves positioned outside of the target bay are not labelled. Preferably, a substantial portion or all of the products 103 in the image should be labelled, as partial labelling of the products can be insufficient for model training. Furthermore, if there are gaps in the bay where no products are present, no labels are required for those specific positions.

[0060] Figure 3 shows that a series of bounding boxes have been created by a user to indicate objects contained in the given image. The corners of the bounding boxes are denoted by round dots in Figure 3. Importantly, during the annotation process, each bounding box contains one object only. This alignment also helps to evaluate the performance of the similarity model as will be described below. The exported data for this labelling process comprises the store images as well as the annotations for the images in the training dataset.

[0061] Figure 4 shows an interface of the labelling tool being used to create bounding boxes and labels for shelves.

[0062] Evaluation of the model - The machine learning model is subject to an evaluation stage which assesses model training loss and training accuracy. The model performs satisfactorily if training loss decreases, and the accuracy increases during the training phase. The checkpoint that consists of the best configurations of the model is, then, saved for the evaluation purpose. During the evaluation stage, such a checkpoint is used to generate an evaluation report about theaforementioned 20% of the training dataset. This report includes class-based accuracy of the model as well as a confusion matrix that is used to evaluate the performance of the model based on true positives, false positives, true negatives and false negatives. According to the reported metrics, the retailer may opt to use this checkpoint in the inference stage or discard it and re-train or fine-tune the model with more data to achieve a suitable level of accuracy.

[0063] Inference - The best checkpoint of the machine learning model is selected for production use. Furthermore, the trained model needs to be converted into an appropriate machine learning model format, such as the Open Neural Network Exchange (ONNX)1 format to operate on an web interface or CoreML2 format to operate on mobile devices, for example iOS devices, such as an iPhone or an iPad.

[0064] By way of example, a PyTorch MobileNet model may be converted to CoreML model format with steps listed below.1. Load the PyTorch Model: the pre-trained MobileNet model from the PyTorch library is initialised and loaded from a checkpoint file.2. Wrap Model: the MobileNet model is wrapped in a custom Imagesimilarity class that is designed to handle image embeddings.3. Prepare Input: a dummy input tensor is created to simulate a single image with the required dimensions for the model.4. Trace Model: it is then possible to trace the MobileNet model with the dummy input, and optimises it for inference.5. Define CoreML Shapes: the input and output shapes of the CoreML model are specified, allowing for variable batch sizes and image dimensions.6. Set Preprocessing: the preprocessing steps for the CoreML model are defined, including scaling and bias values for image normalisation.7. Convert to CoreML: the traced MobileNet model is converted into CoreML format, setting the input as an image and the output as a tensor of features.8. Add Metadata: Metadata and descriptions for the CoreML model, including author information, are then added.9. Save CoreML Model: the CoreML model is saved to the specified output path.

[0065] Converting the machine learning model to CoreML format allows the machine learning model to run on a smart user device, such as an iPhone or an iPad. Without model conversion, the model can be run on servers and accessed remotely via the internet or 'cloud'. However, such operation requires network connectivity, as well as multiple CPU cores and a large amount of RAM and VRAM. In one embodiment, the model could require as many as 4 CPU cores, 22GB of RAM, and 16GB of VRAM when running on the cloud. In comparison, such resource requirements are greatly reduced if the model is converted into CoreML format, consuming around 200MB of RAM and no VRAM to run on an user device, such as an iPhone. Enabling the machine learning modelto run on a user device also completely eliminates the need for network connectivity, which can be unstable or unavailable in some locations. This is particularly the case for large warehouses which are essentially metal boxes that attenuate or completely block cell services. Running a machine learning model on a user device also helps keep data private, and enabling the machine learning model to be integrated into, or used by an smart device application. In at least one embodiment, the smart device application is an application downloadable from Apple App Store or Google Play Store or similar.

[0066] The steps listed above may also be used to convert other machine learning models described herein into CoreML format.2.1.1 YOLOV5

[0067] YOLO (You Only Look Once) is a real-time object detection algorithm that uses a convolutional neural network (CNN) to predict bounding boxes and class probabilities of objects in input images.

[0068] The YOLO algorithm divides an input image into a grid of cells, and for each cell, it predicts the probability of the presence of an object and bounding box coordinates of the object. It also predicts the class of the object.

[0069] The YOLO algorithm processes an input image in a single pass, making it fast and efficient for detecting objects in input images. The process of the YOLO algorithm can be broken down into the following exemplary processing steps:

[0070] First, an input image is passed through a CNN to extract features from the image.

[0071] The features are then passed through a series of fully connected layers, which predict class probabilities and bounding box coordinates.

[0072] The image is then divided into a grid of cells, and each cell is responsible for predicting a set of bounding boxes and class probabilities.

[0073] The output of the model is a set of bounding boxes and class probabilities for each cell.

[0074] The bounding boxes are then filtered using a post-processing algorithm such as nonmax suppression to remove overlapping boxes and choose the box with the highest probability. The final output by the YOLO algorithm is a set of predicted bounding boxes and class labels for each object in the image.

[0075] YOLOv5 is an improved version of the YOLO algorithm with a key difference from the previous versions, which is the use of a more efficient neural network architecture called EfficientDet, based on EfficientNet architecture. The EfficientDet architecture is designed to be efficient in terms of computation and memory usage while also achieving high accuracy.

[0076] It will be appreciated that YOLOv5 is an example of a suitable object detection algorithm that may be used by the present invention. Other objection detection algorithms, such as YOLOv8 may be used as an alternative. Similar to YOLOv5, it processes an input image in a singlepass, detects objects included in the input image, and produces a set of bounding boxes to indicate the objects detected in the input image. The output of the object detection algorithm is then provided to an image similarity calculation module as will be described further below.2.2 Gap detection

[0077] In a constructed planogram, there may be very minimal or no spacing between adjacent products as the shelves will be filled with products, or the products may be arranged in overlapping locations. The image recognition method comprises a gap detection module 202 configured to detect gaps in the input images. In one embodiment, the bay 105 is considered non compliant with the planogram if there are detected gaps in the input images which is usually an indication of empty space on shelves 100.

[0078] In at least one embodiment, the image recognition method uses a machine learning model such as UNet, for semantic image segmentation to detect gaps in the input images. The output of UNet contains colour image pixels where each colour pixel associates with a label. The label may indicate that the colour pixel is 'not a gap', or it is 'a gap'. In real-world scenarios, there may be infinitesimal gaps between products. However, such gaps are not significant to the overall results. In this instance, the bounding boxes of the detected objects can be used to filter out such gaps. For example, if the detected gaps are within the area of a particular bounding box of an object or a shelf, then the labels of the detected gaps will be modified to indicate that they are “not a gap”.

[0079] In one embodiment, UNet was used to segment an input image and detect labels such as “gap” or “not a gap” to those segments. A segment is a mask that has a shape of the polygon. Figure 5 shows an example of an input image. Figure 6 illustrates potential positions and locations of gaps that can be detected in the input image by the gap detection model.

[0080] The operational workflow of the gap detection module consists of the following three procedures:

[0081] Training - To train the UNet model, a training dataset that consists of images and annotations per image where an annotation is a polygon with a label such as “not a gap” or “gap” is prepared. It is worth noting that an RGB image can be presented as colour pixels where a pixel has three channels such as red, green and blue. The training dataset is split into 80:20 ratio where 80% of the dataset is used for the training purpose, and 20% of the dataset is used the evaluation purpose.

[0082] Next, a suitable labelling tool such Label Studio3 may be used to label polygons for a selected number of retailers that were used to train and evaluate the gap detection model. During the labelling process, a user is required to create a polygon for each “gap” or “not a gap” area. Furthermore, the way of providing the label for each polygon is similar to that in the labelling task for the object and shelf detection task. For instance, the user creates a polygon covering the gap area and annotates it with the “gap” label. On the other hand, for image area that is not a gap, the user creates a polygon covering that area and annotates it with the “not a gap” label. These labels will be converted into numeric format such as “0” for “not a gap” and “1 ” for “gap” by the labelling tool. The exported data consists of the original images as well as the corresponding masked images of each original one. Figure 7 shows an example of an image that has gaps which are manually labelled by the labelling tool. In one configuration, a user may choose to only annotate gaps in thetraining image but leave out areas that are not gaps. In this way, the areas of the image that are not annotated as a gap are automatically labelled as “no gap” during the training stage.

[0083] Evaluation - While training the model, the training loss of the model is reported. Such training loss indicates how well the model segments the images and assigns labels for those segments. A lowertraining loss suggests smallergaps between expected segments (i.e., annotated by the user) and generated segments by the machine learning model. It also indicates a higher possibility of correctly matching the expected labels (i.e., provided by the user) and the predicted labels of those segments. During the evaluation stage, the checkpoint of the model is utilised to generate an evaluation report about the aforementioned 20% of the dataset. According to the reported metrics, a retailer may opt to use this checkpoint in the inference stage or discard it and re-train or fine-tune the model with more data to improve accuracy.

[0084] Inference - The best checkpoint of the model is selected for the production use. Furthermore, the machine learning model is converted into either the Open Neural Network Exchange (ONNX) format to operate on the web interface or the CoreML format to operate on mobile devices, for example iOS devices such as an iPhone or iPad, as described above.2.2.1 U-NET

[0085] U-net was previously used for biomedical image segmentation. Its architecture can be broadly considered as an encoder network followed by a decoder network. Figure 8 illustrates an example of the U-net architecture.1

[0086] The encoder is the first half in the architecture diagram illustrated in Figure 8. It usually is a pre-trained classification network like VGG / ResNet where convolution blocks are applied, followed by a maxpool down sampling to encode the input image into feature representations at multiple different levels.

[0087] The decoder is the second half of the architecture shown in Figure 8. The goal is to semantically project the discriminative features (lower resolution) learnt by the encoder onto the pixel space (higher resolution) to get a dense classification. The decoder consists of upsampling and concatenation followed by regular convolution operations.

[0088] In some embodiments, the penalised cost is the sum of cross entropy loss and dice loss, and the goal in the training stage is to minimise the loss value.2.3 Image similarity

[0089] The image recognition method 10 comprises an image similarity calculation module 203 configured to identify the corresponding product of a detected object in the input images. As mentioned above, the image similarity calculation module 203 receives the output of the object and shelf detection module 201. In the product library, there are one or more high-resolution product images of a product with the associated GTIN. The image similarity module generates similarity scores between detected objects in the input images and high-resolution images contained in the product library, to identify the product associated with each detected object.1https: / / developers.arcgis.com / python / guide / how-unet-works /

[0090] The image similarity module calculates a vector representation of each object. Next, a cosine similarity between the vector representation of the object and vector representation of each of the product images is calculated as the similarity score between a pair of vectors. This results in a list of similarity scores of the high-resolution product images. The GTIN of the first ranking high-resolution image is then selected to obtain the GTIN of the detected object in the input images.

[0091] Vector representations of product images are generated during inference, where a single product image is transformed and passed through a neural network to obtain its vector. Additionally, during inference additional vectors of the bay currently being examined are created from a library of product images to compare the image taken by a user’s camera against. In this step, embeddings for a batch of images using a DataLoader are created, the resulting vectors are stored in a dictionary keyed by labels.

[0092] In one embodiment, a suitable machine learning model such as ResNet50 with triplet loss mechanism is used to generate the vector representation of the product images and the cosine similarity to generate the similarity score between a pair of vectors. The operational workflow of this module consists of the following three procedures:

[0093] Training - To train the ResNet50 model for the purpose of image similarity matching, a training dataset is prepared which consists of high-resolution products images, the GTIN of each product and the corresponding images with real-world conditions (including background clutter, illumination, occlusion, lighting conditions, and the other conditions) of each GTIN. Such images with real-world conditions are annotated by the retailer. A list of triplets are, then, constructed where each triplet contains an image with the real-world condition (also known as 'anchor'), a positive sample (for example, a high-resolution image of the matching GTIN), and a negative sample (for example, a high-resolution image of a non-matching GTIN). For instance, if there are 3 GTINs where each GTIN has 2 high-resolution images and 4 images with the real-world conditions, there will be 24 triplets (2 x 4 x 3) constructed. Such triplets are then used to train the model.

[0094] A triplet is represented as: triplet (anchor, positive, negative). The basic idea is to formulate a loss such that it pulls (anchor and positive) together, and push (anchor and negative) away by a margin. This may be represented as: distanced, p) + margin < distanced, n).

[0095] Evaluation - While training the model, the training loss of the model is reported. Such training loss indicates the distance between similar images and dissimilar images. In other words, it shows how well the model pushes the dissimilar images away and pulls the similar images together. The lower training loss indicates better the performance of the model. The checkpoint of the model is, then, saved according to the lowest training loss and used to evaluate the performance of the model a particular store. Given a store image with annotation, the accuracy of the model is evaluated by two folds such as bay or shelf. According to the reported metrics, a retailer may opt to use this checkpoint in the inference stage or discard it and fine-tune the model with more quality data that may consist incorrectly predicted products by the model to achieve the higher accuracy.

[0096] Inference - The best checkpoint of the model is selected for the production use. Furthermore, this needs to be converted into either the Open Neural Network Exchange (ONNX) format to operate on the web interface or the CoreML format to operate on mobile devise, for example an iOS device, such as an iPhone or an iPad.

[0097] To improve the accuracy of the image similarity model, as many unique product images as possible are used for the training dataset. This helps the model learn the representation of each product. Additionally, high resolution product images as well as images with real-world conditions are both included in the training dataset. The quality of this group of images is identified by the following features: colours, shapes and text labels of the product.

[0098] The following conditions may exist in the input images: background clutter, illumination, rotational invariance, deformation, occlusion, blurriness, and inclusion. Each input image may be affected by one or more of such conditions. The quality of this group of images is identified by the following features: colours, shapes, text labels on the image, and the condition of image. It is crucial to have at least one training image per condition because it helps the image similarity module to learn representation of each condition and mapping the representation of the product image to the image with real-world condition.

[0099] Similar to other module training processes, a manual labelling process assists with improving accuracy of the model. A corresponding tool may be developed which supports such process of the image similarity module. This tool may be used as follows:• A user uploads product images of a particular product category (e.g., hair care, dental health, or other product categories).• For each product image, the user uploads images with different real-world conditions.• The system keeps track of training status of the uploaded images.• After labelling, the user exports data fortraining the image similarity model in a particular product category.• After training the image similarity model, the user imports metadata to update the training status of each uploaded image.Figures 14 and 15 indicate how different images may be uploaded to achieve more accurate training of the model. In Figure 15, varying illumination and blurred images of a selected shampoo product may be incorporated in the training dataset.2.3.1 ResNet50

[0100] ResNet50 is an image classification machine learning model. One of its key features is the use of residual connections, which allow the model to learn a set of residual functions that map input to desired output. These residual connections enable the model to learn much deeper architectures than was previously possible, without suffering from vanishing gradients.

[0101] The architecture of ResNet50 is divided into four main parts: convolutional layers, identity block, convolutional block, and the fully connected layers. The convolutional layers areresponsible for extracting features from the input image, while the identity block and convolutional block are responsible for processing and transforming these features. Finally, the fully connected layers are used to make the final classification.

[0102] The convolutional layers in ResNet50 consist of several convolutional layers followed by batch normalisation and ReLU activation. These layers are responsible for extracting features from the input image, such as edges, textures, and shapes. The convolutional layers are followed by max pooling layers, which reduce the spatial dimensions of the feature maps while preserving the most important features.

[0103] The identity block and convolutional block are the key building blocks of ResNet50.

[0104] The identify block is a simple block that passes the input through a series of convolutional layers and adds the input back to the output. This allows the network to learn residual functions that map the input to the desired output. The convolutional block is similar to the identity block, but with the addition of a 1x1 convolutional layer that is used to reduce the number of filters before the 3x3 convolutional layer.

[0105] The final part of ResNet50 is the fully connected layers. These layers are responsible for making the final classification. The output of the final fully connected layer is fed into a softmax activation function to produce the final class probabilities.

[0106] In one embodiment, the ResNet50 model is used in conjunction with triplet loss to train a network that learns to differentiate between similar and dissimilar images. The process involves:1 . Initialising the ResNet50 model and freezing its layers except for the fully connected layer, which is fine-tuned during training.2. Defining the T ripletMarginLoss criterion with a specified margin to measure the distance between an ch or- positive and anchor-negative pairs.3. Setting up an optimizer, specifically SGD (Stochastic gradient descent), to update the model's parameters based on the computed loss.4. During training, batches of triplets are processed and the triplet loss is backpropagated to update the model's weights.5. In the validation phase, embeddings for the validation set are calculated and the cosine similarity is used to evaluate the model's accuracy.6. The best model state is saved based on the minimum loss achieved during training.

[0107] Vector representations of an image may be created by the machine learning model as follows:1 . Iterate over a collection of X and Y co-ordinate data entries, each potentially containing a bounding box specifying a region of interest within the image.2. For each entry with a bounding box, the corresponding image region is extracted and converted it into a PIL (Python Image Library) image object.3. This image region is then resized to specific dimensions (for example, 128x256 pixels in this case) and transformed into a tensor using the ToTensor transformation.4. A series of preprocessing steps are applied, including normalisation with predefined mean and standard deviation values, to prepare the image tensor for the neural network.5. The preprocessed image tensor is fed into a preloaded neural network model, which performs a forward pass and generates the vector representation.6. The resulting vector is detached from the current computation graph and moved to the CPU, making it a standalone tensor that represents the image mathematically.

[0108] The process above is repeated for each image or image region in the loop, resulting in a collection of vectors that mathematically represent the visual content of the input image, suitable for further analysis such as similarity comparison.

[0109] In another embodiment, the machine learning model used for image similarity calculation can be Mobilenet, instead of ResNet50.

[0110] Each of the machine learning models described above may be converted into a suitable format for operation on a user device. This enables the entire image recognition method described above to be configured to run on a portable device such as a mobile phone or a tablet device, without relying on any external information accessible via wired or wireless networks. This means that the image recognition method is able to perform its tasks without Wi-Fi or cellular data networks. It also eliminates the need of relying on complex technical infrastructure to validate the input images.3. Compliance Report Generation

[0111] A compliance report aims to validate whether the product arrangement as represented by the input images is compliant with the provided planogram. In one embodiment, the output of the compliance report comprises X% of ranging, Y% of facings wide, Z% of correct placement, and any other metrics selected by the retailer, as illustrated in Figure 9.

[0112] Products that are ranged or stocked in a retailer are allocated a position on a shelf within a category. “Facings Wide” represents the amount of allocated horizontal facings from left to right or right to left. “Facings High” represents the amount of vertical facings from bottom to top or top to bottom. “Facings Deep” represents the amount of facings from the front to the back of the shelf. With reference to Figure 1 again, the top shelf is two facings wide and two facings high.

[0113] A metric in a compliance report can be derived from rules provided by a retailer. In one embodiment, an expert decision tree can be developed based on rules provided by a retailer. For example, the retailer may be asked to provide a set of rules that help to identify the label of the bounding box of a product on the planogram (for example, “not compliant”, “maybe compliant”, and “compliant”). The expert decision tree may comprise additional rules based on observed actions carried out by the retailer.

[0114] A compliance report can then be generated based on the output of the image recognition method and the expert decision tree. An exemplary decision tree may be as follows:1 . Are there any image pixels in the bounding box of the planogram? a. If yes, go to Rule 2 b. If no, go to Rule 32. Are the gap pixels accounted for the large portion of the total pixels in the bounding box? a. If yes, that bounding box is marked as “not compliant” b. If no go to Rule 33. Is there any detected bounding box overlapping with the bounding box of the planogram? a. If yes, go to Rule 4 b. If no, the bounding box of the planogram is marked as “not compliant”.4. Are there 2 or more detected bounding boxes overlapping with the bounding box of the planogram? a. If yes, go to Rule 5 b. If no, go to rule 65. Do any detected bounding boxes have the matched GTIN? a. If yes, the bounding box of the planogram is marked as “maybe compliant”. b. If no, the bounding box of the planogram is marked as “not compliant”.6. Does the detected bounding box have the matched GTIN? a. If yes, the bounding of the planogram is marked as “compliant”. b. If no, the bounding box of the planogram is marked as “maybe compliant”.

[0115] With this expertise decision tree, the product arrangement is not compliant according to the planogram when one of the following has been detected:- the number of detected gaps in shelves exceeds a limit, or- there is a mismatch between the GTIN of the detected object and the GTIN of the product at that position in the planogram.

[0116] On the other hand, the store is compliant when the GTINs of all detected objects are matched with the GTINs of the products regarding their positions in the planogram.4. Evaluation of the image recognition method4.1 Evaluation datasets

[0117] Different evaluation datasets are used to evaluate the following models: the object and shelf detection model, the gap detection model, and the image similarity model. The information about the dataset for each model is detailed below.• Object and shelf detection: The SKU1 10k dataset and a supplementary dataset created by a retailer are used for the training purpose. The images of the training datasets comprise manually annotated objects and shelves as mentioned above. The SKU110k dataset contains 11 ,762 images with more than 1 .7 million annotated bounding boxes captured in densely packed scenarios from thousands of supermarket stores2. The object detection model was fine-tuned with the SKU110k dataset. The SKU110k dataset does not have any annotated data for the shelf detection task. The supplementary dataset is created by a retailer, which comprises 40 images, wherein each image in the store dataset comprises a large number of annotated objects and shelves. The supplementary dataset was utilised to fine-tune the checkpoint of the model trained on the SKU110k dataset for the purpose of detecting both objects and shelves.• Gap detection: there are very limited datasets that are specific to the purpose of segmenting gaps in an input image. The gap detection training dataset is prepared by including store images where each image has annotated polygons as masks that were labelled “gap”, and the remaining areas that were labelled “not a gap”.• Image similarity: The supplementary dataset created for the object and shelf detection as well as the high-resolution product images from the SKU product library were utilised to train and evaluate the image similarity model. For example, the GTINs of unique products from 20 images are identified and utilised to select high-resolution images with the font face of products. On the other hand, the annotated objects in the store dataset are then associated with their corresponding product. Given the product GTINs and their associated annotated objects, the triplets of images are, then, constructed. When there are 226 unique products where each product has 1 high- resolution image and 4 annotated objects on average, a total of 204,304 triplets in total are created (4 * 226 * 226).4.2 Object and shelf detection module

[0118] The object and shelf detection module performs the task of detecting position of identified objects, and classifying a label for that object. The performance of the classification step may be measured using the following metrics: accuracy, precision, recall, F1 score, and mean average precision (mAP).

[0119] The parameter mAP is used due to characteristics of multi-classification of the object and shelf detection module. Furthermore, there are four important scenarios that help to understand the evaluation metrics such as:2https: / / github.com / eg4000 / SKU110K_CVPR19true positive, where the object is classified correctly according to the true label;• false positive, where the object is classified incorrectly according to the true label;• true negative, where the object is classified correctly according to the false label, and• false negative, where the object is classified incorrectly according to the false label.The evaluation metrics may be calculated as follows:• Accuracy: It is calculated based on the total number of true positives and true negatives over the total number of objects or shelves.• Precision: It is calculated based on the number of true positives over the total number of true positives and false positives.• Recall: It is calculated based on the number of true positives over the number of true positives and false negatives.• F1 -score: It is calculated based on the precision and the recall to measure the overall accuracy of the model according to both positive and negative cases• mAP: It is the mean of the average of the precision among classes. For example, if there are 2 classes of objects such as a product and a shelf, mAP will be the mean of the sum of the average prevision among those classes.4.3 Gap detection

[0120] As mentioned above, the gap detection module is configured to identify the gaps in the input images. This is achieved by shaping polygons and predicting the label for each polygon. The performance of the machine learning model is evaluated based on two metrics such as the cross entropy loss and the Dice loss. The cross entropy loss evaluates the dissimilarity between the predicted probability distribution and the ground truth labels. Such loss function effectively penalises misclassification and rewards the model when it generates the accurate segmentation results. On the other hand, the Dice loss measures similarity between the predicted segmentation mask and the ground truth mask. Such loss rewards the model when it produces the segmentation masks that closely align with the ground truth masks. The overall loss function of the UNet model is the sum of the cross entropy loss and the Dice loss. The lower the loss value, the better the model performs.

[0121] Applying the aforementioned labelling process, “gap” annotations for 30 infield hair care product images have been created. It is worth noting that the arrangement of products may vary as per category. Hence, it is important to have more labelled data in such cases to improve the robustness of the model. Moreover, the annotation process for an image can take a few minutes on average. Given the annotated data, the U-Net model is trained and evaluated for gap detection. The training loss and the evaluation loss were around 0.00015. Based on an empirical analysis, the model could detect the majority of gaps on the image.4.4 Image similarity

[0122] The image similarity model is trained and evaluated based on the triplet loss mechanism which aims to enforce the embedding space to have closer distances between images of the same GTIN and larger distances between images of different GTINs. Furthermore, the model is evaluated based on accuracy of matching products on a particular bay of the given store image.

[0123] Given a store image of one bay, the accuracy of the image similarity can be calculated by two folds such as:

[0124] Bay - The products in that bay are gathered to construct the search space. For instance, the bay may consists of 70 products. Hence, the search space contains 70 items. The number of products that has the correct GTIN over all products in one bay.

[0125] Shelf - A bay may be constructed by stacking N shelves. Hence, the search space can vary between one shelf and N shelves. Furthermore, the window sliding approach may be adopted to ensure that the model is evaluated with all products in that bay. For instance, if a bay has 6 shelves and the size of the window is 2, there will be 3 search spaces including:(1) the first shelf and the second shelf,(2) the third shelf and the fourth shelf, and(3) the fifth shelf and the sixth shelf.

[0126] The accuracy of the image similarity model is calculated based on the average of the accuracy of the model in individual windows.

[0127] The image similarity model is trained and evaluated based on the triplet loss mechanism which aims to enforce the embedding space to have closer distances between images of the same GTIN and larger distances between images of different GTINs. Furthermore, the mode is evaluated based on accuracy of matching products on a particular bay of the given store image.5. Operation of the image recognition method and compliance estimation on a user device

[0128] The image recognition method and the steps of determining planogram compliance mentioned above can be integrated into an application that is configured to run on a mobile user device such as iPhone or iPad. The benefit of this mode of operation is that it completely eliminates the need of network connectivity while allowing for instant determination of planogram compliance and providing feedback to retailers.

[0129] Figure 10 shows an interface of the application configured to let a user capture a photo of a bay, by using a built-in camera of the user device. The captured photo then becomes the input image of the image recognition method. Preferably, the constructed planograms are stored in the application as well. Before a photo is captured by the user device, the user is prompted to select a constructed planogram via the interface of the application.

[0130] When taking a photo of a target bay, the application overlays the selected planogram in an imaging area of the user device, such that the user has an augmented view of the planogram as well as the actual appearance of the bay. The augmented view helps the user position and angle the camera of the user device such that the augmented view of the planogram closely aligns with the photo to be taken. Figure 11 shows an example of an augmented planogram 10a projected onto the imaging view of the camera. The augmented planogram 10a is misaligned with the actual appearance of the target bay. Figure 12 shows an example of an augmented planogram whichproperly aligns with a captured image on the user device. This will be a good angle to capture a high quality photo of the target bay to achieve accurate compliance results and improve the accuracy of the analysis.

[0131] The augmented view as illustrated in Fig 11 can be based on positioning data from the planogram, which indicates what products should be expected in the photo position. The information extracted from the captured photo by the various modules can be used for training the machine learning model. When there is a discrepancy, an administrator may be required to log into a web portal to provide corrections, then the corrected data is used in machine learning training to improve the machine learning model.

[0132] Figure 13 displays a comparison of a captured image 200 of a shelf and its intended planogram 10b, and the compliance results generated by the application. The compliance results can be displayed to the user using different colours or by any other suitable means. In Figure 13, green dots 30 indicate that correct products have been positioned at the designated locations on the shelf. Yellow dots 40 may suggest that wrong products are detected in the allocated location. Red dots (not shown in Figure 13) may be used to indicate that no products are detected, suggesting those products are potentially out of stock.

[0133] In one form, the application not only provides instant, real-time planogram results, but also supports a continuous loop of machine learning model training and tuning. For example, the labelling tool shown in Figures 3, 4, 7 may also be integrated into the application, allowing instant and on-going model training on the user device.

[0134] In one example, the operational flow of the application on a user device may be as follows. After a photo of a target bay is captured on the device, it is processed by the object and shelf detection module to produce bounding boxes where it detects products. The application then extracts the images of the products from each of these bounding boxes and uses the image similarity module comprising an image similarity machine learning model to produce similarity vectors for all the product catalogue. The similarity module then uses cosine similarity to compare the similarity vectors to find the closest match, this is then used as the 'best match' for this position.

[0135] The data that is indicative of “best match”, “expected product in photo position”, and “match status” can be recorded in a database. The database may be accessible by the user application via data communication networks, or it may be stored locally on storage device of the user device. This data is then used to provide new training data for the machine learning model to re-train it. In addition to providing corrections on the user device at time of capture (Figure 13), the user may also be allowed to provide corrections remotely (e.g. on an administrator portal) to indicate what the correct product should be.

[0136] Embodiments of the invention are generally directed to image recognition techniques and methods involving machine learning models, the results of which may be used for planogram compliance assessment. It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

CLAIMS1 . An image recognition method for planogram compliance, comprising the steps of: receiving an input image of a target area to be analysed for planogram compliance; detecting, by a machine learning model, one or more objects in the input image; optionally, detecting one or more gaps in the input image; calculating image similarity scores of the one or more objects detected and a plurality of product images; identifying an associated product for at least one of the one or more objects detected, based on the calculated image similarity scores; assessing the target area based on the one or more objects detected, the one or more gaps detected and / or the associated products identified of the one or more objects, wherein the machine learning model is configured in a format suitable to run on a user device without requiring network connectivity.

2. The image recognition method of claim 1 , wherein the step of detecting one or more objects in the input image additionally comprises: detecting one or more shelves, used to support products.

3. The image recognition method of claim 2, wherein the detected shelves are used to filter out objects that are located outside of the target area.

4. The image recognition method of any one of claims 1 to 3, wherein the step of detecting objects in the input image further comprises: extracting features from the input image by a convolutional neural network (CNN); passing the extracted features through a series of layers to predict class probabilities and coordinates of bounding boxes; dividing the input image into a grid of cells, wherein each cell is responsible for predicting a set of bounding boxes and class probabilities; filtering the predicted bounding boxes to remove overlapping bounding boxes and select a bounding box with the highest probability; outputting a set of predicted bounding boxes and class labels for each object in the input image.

5. The image recognition method of any one claims 1 to 4, wherein the step of detecting one or more gaps in the input image further comprises: segmenting an input image and detecting labels such as “gap” or “not a gap” to those segments, wherein a gap indicates an empty space that does not have a product.

6. The image recognition method of any one of claims 1 to 5, wherein the step of detecting one or more gaps is carried out by a gap detection machine learning model, configured in a format suitable to run on a user device, such as CoreML.

7. The image recognition method of claim 6, wherein the gap detection machine learning model is configured to run on the user device without relying on network connectivity.

8. The image recognition method of any one of claims 1 to 7, wherein the step of calculating image similarity scores further comprises: calculating a vector representation of a detected cropped object out of the input image; calculating a cosine similarity score of the vector representation of the detected object, and the product images in the library, to product a list of similarity scores; selecting a product image that has a highest similar score, wherein the product represented by the product image is determined as an associated product of the object, and determining product information of the associated product.

9. The image recognition method claim 8, wherein the step of determining product information comprises determining GTIN of the product.

10. The image recognition method of any one of claims 1 to 9, wherein input image is a photo of a retail store, and the target area is a target bay including one or more shelves.

11. A method for estimation of planogram compliance, comprising: selecting a pre-defined planogram; capturing an input image on a user device of a target area; performing the image recognition method of any one of claims 1 to 10, on the user device; estimating planogram compliance by comparing output of the image recognition method with the pre-defined planogram.

12. The method of claim 11 , wherein the step of capturing an input image further comprises providing an augmented view including an overlay of the pre-defined planogram over a camera view, such that the user device may align the target area with the pre-defined planogram.

13. The method of claim 11 or 12, wherein method further comprises:indicating to a user that a non-compliant condition exists, if one or more of the following condition exists: the number of detected gaps exceeds a predetermined limit, there are no overlapping detected objects, or there is a mismatch between the associated product and the planogram.

14. The method of any one of claims 11 to 13, further comprising generating a compliance score based on the percentage of detected objects matching the pre-defined planogram’s product information and positions.

15. The method of any one of claims 11 to 14, further comprising generating a share of shelf report indicating the proportion of shelf space occupied by each product or brand.

16. The method of any one of claims 11 to 15, further comprising: indicating to a user that planogram compliance is achieved, if all of the determined, associated products match with the pre-defined planogram.

17. A user device configured to carry out the method as claimed in any one of claims 1 to 16.

Images