Tag identification

Abstract

A system for identifying tags includes an imaging sensor and a processor. The imaging sensor acquires one or more images of one or more tags from light reflected from one or more tags on a tagged item. The processor receives one or more images and a library of tag types, and is configured to use one or more images to determine feature metrics using machine learning algorithms based on image processing, image manipulation, and / or image correction, and to use the feature metrics and the library of tag types to determine the tag type of one or more tags in one or more images based on maximal determination, bounding box generation, tag candidate patch extraction, tag candidate segmentation, tag candidate feature metric determination, and / or comparison with a model, and to determine the confidence level of the tag type, and to provide the determined tag type in response to the confidence level being above a threshold level.

Description

Technical Field

[0001] [Cross - reference to Related Applications] This application claims priority to U.S. Provisional Patent Application No. 63 / 456,974, entitled TAG IDENTIFICATION, filed on April 4, 2023, the content of which is incorporated herein by reference for all purposes.

Background Art

[0002] Producers or resellers of articles (including raw materials and components of such articles), particularly but not limited to high - value articles, include, for example, manufacturers, as well as other operators involved in the entire supply and distribution chain such as suppliers, processors, wholesalers, retailers, repackagers, and retailers, etc. These are facing the problem of article counterfeiting. Counterfeiting includes the replacement, dilution, addition, or omission of raw materials or components of an article when compared to its intended product specifications, as well as false representation regarding the intended sales process of the packaged article or deviation from the intended sales process. Since counterfeit products are sold instead of genuine articles, this leads to potential loss of revenue. Also, there may be health hazards or product - related damages caused by using counterfeit products instead of genuine ones. For example, counterfeit products may have different performance or not function at all compared to genuine articles. This is particularly serious in industries that can affect health and safety, such as industries related to pharmaceuticals, dietary supplements, medical devices, food and beverages, construction, transportation, and defense.

[0003] To achieve favorable health outcomes in a more cost-effective and timely manner, healthcare providers need to focus not only on the efficacy of specific medications but also on adherence to health management plans. Understanding when, where, and how often medications are prescribed by physicians, dispensed accurately and in a timely manner from pharmacies, received by patients, and consumed by patients helps to understand and validate the effectiveness of the overall health management plan. For the reliability of the collected information, it is important to record and collect data for appropriate analysis and research, while also ensuring the underlying identity of the medication at each stage.

[0004] Invisible tags, visible tags, and / or semi-invisible tags (e.g., microtags, tagants, physical markers, and chemical markers), including those in edible and ingestible forms, are known methods for identifying and authenticating various items, including drug product volume tagging in solid oral form, food, packaging materials, and product labels. However, there is no convenient machine-readable method to identify the sparsity of such tags and to distinguish between the various tag types.

[0005] Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows one embodiment of a tag identification system. [Figure 2A] This flowchart shows one embodiment of the process for generating the first model. [Figure 2B] This flowchart shows one embodiment of the process for generating a second model. [Figure 3A] This is a flowchart illustrating one embodiment of the process for classifying tags. [Figure 3B] This flowchart illustrates one embodiment of the process for classifying additional identification features. [Modes for carrying out the invention]

[0007] The present invention can be implemented in various forms, including processes, apparatus, systems, compositions of materials, computer program products embodied on computer-readable storage media, and / or processors, such as processors configured to execute instructions stored in memory coupled to the processor and / or instructions provided by memory. In this specification, these implementations, or any other forms the present invention may take, may be referred to as "technologies." Generally, the order of the steps of the disclosed processes may be modified within the scope of the invention. Unless otherwise specified, components such as processors or memory described as configured to perform a task may be implemented as general components temporarily configured to perform a task at a given time, or as specific components manufactured to perform a task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0008] A detailed description of one or more embodiments of the present invention is provided below, along with accompanying drawings illustrating the principles of the present invention. While the present invention is described in relation to such embodiments, it is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses numerous alternative forms, modifications, and equivalents. In order to provide a complete understanding of the present invention, numerous specific details are described below. These details are provided for illustrative purposes, and the present invention may be carried out in accordance with the claims even without some or all of these specific details. For clarity, known technical matters in the art related to the present invention are not described in detail in order to avoid unnecessarily obscuring the present invention.

[0009] A system for identifying tags is disclosed. The system comprises an imaging sensor and a processor. The imaging sensor acquires one or more images of one or more tags from light reflected from one or more tags on a tagged article. The processor is configured to receive one or more images and a library of tag types. Using one or more images, the processor is configured to determine a set of feature metrics. The determination of the set of feature metrics is based on machine learning algorithms and on at least one of image processing, image manipulation, or image correction. Using the set of feature metrics and the library of tag types, the processor is configured to determine the tag type of one or more tags in one or more images. Determining the tag type is based on at least one of maximal determination, bounding box generation, tag candidate patch extraction, tag candidate segmentation, tag candidate feature metric determination, or comparison with a model. The processor is further configured to determine the confidence level of the tag type. In response to the confidence level being above a threshold level, the processor provides the determined tag type.

[0010] In some embodiments, a library of tag types includes a set of feature metric values, each set of feature metric values ​​corresponding to a specific tag type and used to identify a specific tag type. In various embodiments, image processing, image manipulation, and / or image correction include one or more of the following: image segmentation (e.g., semantic segmentation or instance segmentation), synthesis of images, patch extraction, feature extraction, feature compression, deep feature extraction, feature fusion, feature classification, confidence level determination, image cropping or binning, white balance correction, background color subtraction, etc.

[0011] Disclosed is an integrated, self-contained system for the rapid detection and identification of tag types on tagged articles (e.g., specific types of microtags, tagants, chemical markers, physical markers, lugate filters, interference filters, pigments, flakes, platelets, luminescent agents, granules, etc.). The system comprises an imaging sensor, a focusing optical system, and a data acquisition and processing computer platform. In various embodiments, the system includes one or more illumination sources, sample fixation, automated sample loading, scanning, analysis, and / or automated image acquisition to enhance its ability to detect and identify different tag types on tagged articles (e.g., lugate microtag types). In various embodiments, the color of the tag or tag type is a result of the tag's inherent chemical or physical material properties, or a result of one or more coatings on the tag surface.

[0012] In various embodiments, the system is integrated into inline or offline inspection systems (for example, in pharmaceutical or food processing and packaging facilities). In some embodiments, the system is integrated into distributed computing and communication environments.

[0013] In some embodiments, ambient light (e.g., indirect light, sunlight, indoor lighting, streetlights, etc.) projects an image of the sample onto the image sensor via a focusing optical system. In some embodiments, light from a directional illumination source (e.g., a smartphone flash) is used to project an image of the sample onto the image sensor. In some embodiments, a tag in the sample is illuminated with light of a specific color (e.g., laser light) to induce a response from the tag (e.g., a fluorescent marker or fluorescent dye). The image sensor renders the captured digital image by a processing computer (e.g., a smartphone chipset).

[0014] The imaging sensor is used to collect one or more images of one or more tags on a tagged item (e.g., tagged tablets, tagged packaging, tagged packaging labels, tagged food, etc.), where each image in the series represents an image of the tagged item at a single point in time and / or location or physical orientation and / or at a specific wavelength, or wavelength range or color band.

[0015] In various embodiments, one or more composite images are generated from two or more images. For example, in multispectral and hyperspectral imaging systems, each image in a full image dataset contains a wavelength range narrower than the entire wavelength range encompassed by the full image dataset. For example, a multispectral dataset may include measuring reflectance in four other (narrower) bands, including the blue, green, and red wavelengths of the visible spectrum and the near-infrared band, while individual images in a hyperspectral "data cube" may contain a wavelength range of only 1-2 nm. The number of individual images in a multispectral dataset is generally in the range of 3-20, while the number of individual images in a hyperspectral data cube often contains hundreds of consecutive spectral bands. As used herein, "composite image" refers to a collection of individual images in a full image dataset or data cube.

[0016] In the following, the term “composite image” is used to refer to any appropriate combination of one or more composite images and / or one or more individual images.

[0017] In various embodiments, a collection image provides an indication of the presence and / or density of tag types (i.e., the number of tags per region, image, or sample). In various embodiments, one or more thresholds based on statistical likelihood are applied when analyzing the collection image to provide binary or multiplexed answers or responses (e.g., the presence or absence of a particular tag type, the presence and / or density of each of the tag types, flags, or warnings). In some embodiments, thresholds are applied to the output of confidence levels from one or more models (e.g., the confidence level of a first model must be greater than a certain threshold t1, and the confidence level of a second model must be greater than a certain threshold t2). In some embodiments, thresholds are applied to individual images and / or collection images (e.g., the minimum number of tags in a single image must be greater than a certain threshold t3, and the minimum number of tags in a set of collection images must be greater than a certain threshold t4). In various embodiments, these thresholds are applied so that all conditions are met, some of these conditions are met, and / or a conditional list of conditions is met in order to determine a response or answer (for example, the intensity of the pixel to be processed must be greater than a certain threshold t5, or the intensity of the pixel to be processed must be greater than a certain threshold t7 if the hue of the pixel is within a given range defined by a certain threshold t6). In some embodiments, the conditions of each threshold must be met for a tag candidate to be classified as a particular tag type. In some embodiments, the condition of each threshold is that the tag must be above or below a given threshold.

[0018] In some embodiments, additional identifying features of the tagged article are determined. For example, the additional identifying features include one or more of the following: Quick Response (QR) code, barcode, two-dimensional matrix, data matrix, logo, serial number, article shape, article size, brightness, color, mark, symbol, or randomly serialized markers. In some embodiments, the additional identifying features are used in combination with the identified tag type to identify and authenticate the tagged article (i.e., to verify the authenticity of the tagged article).

[0019] In various embodiments, an Optical Character Recognition (OCR) method is used to read text and / or a feature detection algorithm is used to identify other types of symbols imprinted on tagged articles. For example, the text and other symbol types include one or more of the following: debossed marks on pharmaceutical tablets, unique identifiers imprinted on drug capsules, product information printed on labels (e.g., U.S. drug code numbers), logos, serial numbers, item size, item shape, and item color. For example, in the case of a solid oral dosage form of a human drug product, the imprinted code on the tablet or capsule, used in conjunction with the product size, shape, and color, can be used to identify the drug product, one or more active ingredients of the product, strength, and manufacturer or distributor. In some embodiments, the OCR method utilizes the Tesseract OCR engine. In various embodiments, the OCR method includes one or more open-source methods (e.g., OCRopus, Kraken, Calamari OCR, Keras OCR, EasyOCR, etc.) or commercially available OCR application programming interfaces (e.g., Amazon Textract, Amazon Rekognition, Google Cloud Vision, Microsoft Azure Computer Vision, Cloudmersive, Free OCR, Mathpix, etc.). In various embodiments, Hu moments, histograms of oriented gradients, key-point matching, or neural network algorithms are used for logo detection.

[0020] In various embodiments, additional identification features are used before determining the tag type. For example, reading a QR code on a tagged item allows for preloading of the correct machine learning model for image acquisition settings and / or tag classification. In various embodiments, the image acquisition settings and / or correct machine learning model for tag classification include one or more of the following: focusing algorithm, focusing position, exposure algorithm, exposure value, white balance algorithm, white balance setting, selection of one or more channels or bands, image processing configuration settings, segmentation configuration settings, tag filtering configuration settings, tag density specifications, scan configuration settings, or any other suitable feature detection algorithm and / or configuration settings. For example, a barcode on a tagged item allows the system user to load and display relevant product information (for example, this allows the system user to verify that the correct product was identified before item authentication).

[0021] In the disclosed system, a collective image of tags on tagged items is passed to a classification algorithm (e.g., a tag labeling or typing algorithm). In some embodiments, the image is segmented before being passed to the classification algorithm.

[0022] For example, pixels representing tags are segmented from the image background, and a binary segmentation mask is output for each image. The binary segmentation mask is used to select relevant pixels to outline a tag or tag cluster. A tag cluster is a group of two or more physically close tags, and each individual tag in a tag cluster cannot be fully outlined by its own bounding box. In some embodiments, image background pixels are used as part of a tag filtering algorithm (for example, to ensure that a tag is bright enough against its background, or to normalize the optical properties of a tag by incorporating background subtraction techniques).

[0023] Feature extraction empirically determines the features of interest (e.g., size, shape, color, intensity, chroma, etc.) to extract from each tag patch in order to generate a set of feature metrics for each tag patch. For example, feature metrics (e.g., one or more tag characteristics) considered significant criteria for tag type identification, and / or the associated statistical thresholds for each tag type established as being indicative of significance, are used to generate a set of feature metrics for each tag type. In some embodiments, the features are manually determined by a human user to generate a set of feature metrics for each tag patch. In various embodiments, the features and / or feature metrics are automatically determined.

[0024] As used herein, the term "feature" is commonly used as a general characteristic (e.g., shape, color, intensity, etc.), and a "feature metric" is a more specific value of a general characteristic (e.g., circular, red, bright, etc.) useful in the classification of tags and / or product types. In various embodiments, the level of specificity of the feature metrics required is determined by generating and training a classification model (described herein) to meet the desired level of reliability. For example, there are various types of red tablets in the market, but multiple other characteristic features (e.g., dosage, size, active ingredient, etc.) vary by product. Distinguishing one red tablet product from another based only on the red feature metric may require a more specific determination of the feature metric (e.g., the exact hue, chroma, lightness of the color, etc.). Alternatively, when additional discriminative feature metrics are combined and used to classify or identify a given product, the specificity of the color may not need to be as stringent as when using only a general color family. Hereinafter, the terms "feature" or "feature metric" may be used interchangeably, and thus are not intended to limit their appropriate interpretation by those skilled in the art.

[0025] In various embodiments, the feature metric includes one or more of size, shape, color value, chroma value, intensity value, color value standard deviation, chroma value standard deviation, intensity value standard deviation, relative color value, relative chroma value, or relative intensity value, and the relative color value, relative chroma value, or relative intensity value is with respect to the image background surrounding one or more tag types. In various embodiments, a color, chroma, or intensity feature metric includes any of an absolute value, a standard deviation, and / or a relative value. In various embodiments, color is determined using a Hue, Saturation, Lightness (HSL) color model, a Hue, Saturation, Value (HSV) color model, a Red, Green, Blue (RGB) color model, or any other suitable type of color model.

[0026] In some embodiments, feature compression is used to reduce the data size while maximizing information. The resulting compressed version of the original tag patch reduces training time and prediction time and reduces computer memory requirements.

[0027] In some embodiments, deep feature extraction is used to extract additional tag features (e.g., latent tag features) by a deep learning method. For example, latent tag features include features, patterns, and differences that are not obvious or distinguishable by human observation but are determined to be significant by a deep learning method. Deep supervised learning can be used to extract additional features of interest to generate tag features that are difficult and costly to manually encode and latent features embedded in the data.

[0028] Feature fusion generates a combined list of features for each patch by adding empirically determined features to the list of extracted features generated from a deep feature extraction process.

[0029] Feature classification classifies tag types using deep supervised learning models. Patch features (and / or feature metrics) of tag images are passed to machine learning algorithms (e.g., support vector machines), residual neural network classification algorithms (e.g., ResNet-based classification algorithms, or any other suitable type of neural network) that are trained on sets of feature metrics corresponding to various tag types in order to generate a class prediction model.

[0030] After training, these algorithms can be used to recognize, identify, and mathematically group feature metrics associated with specific tag types of interest for unknown items. Such characteristics include, for example, size, shape, color value, saturation value, intensity value, color value standard deviation, saturation value standard deviation, intensity value standard deviation, relative color value, relative saturation value, and relative intensity value.

[0031] In various embodiments, tag type classification is based on morphology (i.e., shape) and / or intensity (i.e., lightness distribution). In some embodiments, algorithms utilize both color and spatial (e.g., shape and size) characteristics in their training.

[0032] In various embodiments, one or more machine learning models are trained using hierarchical feature learning (i.e., "hierarchical learning"). For example, the first layer of a neural network model is trained to learn representations of basic (low-level) features such as edges. Further lower layers of the network output representations of more specific features corresponding to parts of an object (e.g., corners, protrusions). After the last layer, the representations become high-level, allowing the network to clearly separate different objects, having learned to distinguish between different classes.

[0033] In various embodiments, one or more neural network models are trained using spectral data (e.g., spectral data obtained from tags detected by a bounding box model). For example, spectral data obtained from a spectral imaging sensor (e.g., spectral imager, multispectral imager, hyperspectral imager, etc.). In various embodiments, one or more neural network models are trained to match observed spectral data with a library of spectral data corresponding to known tag types.

[0034] In various embodiments, the output of one or more neural network models provides a class label for the identified tag type (e.g., to the system user). In some embodiments, a single tag candidate must be classified by one or more models to identify the final class (e.g., for a tag candidate to be ultimately identified as a valid tag class for the system user, it must be classified as a valid tag class by Model 1 and also by Model 2). In some embodiments, a single tag candidate must be classified by one model, and other image characteristics must be classified as valid classes for the final class to be identified (e.g., the tag candidate must be classified as a valid tag class by Model 3, and the image must be classified as a valid logo class by Model 4). In some embodiments, a tag may need to pass through multiple models to be properly classified. As an example, considering two models (Model A and Model B), for a tag to be identified as Class 1, the system may require that the tag be classified as Class 1 in Model A but also as Class 0 in Model B, classified as Class 1 in both Model A and Model B, classified as Class 1 in either Model A or Model B, or classified as Class 1 by an AND condition, an OR condition, or any other suitable combination of conditions. In various embodiments, the provided class label for an identified tag type includes associated metadata. For example, associated metadata may include date and time, confidence level, number and / or density of identified tag types (i.e., number of tags per unit area), and a plot or graph of associated spectral data.

[0035] In various embodiments, when the presented tagged articles are sparsely tagged, methods for anomaly detection and targeting at the pixel and subpixel scales based on statistical machine learning algorithms are used to rapidly locate and identify individual tags. Sparse distribution of samples may result from economic considerations to minimize the cost of the tags applied and / or to maintain the aesthetic and / or visual concealment of the tagged samples.

[0036] In various embodiments, the disclosed system provides a method for rapidly locating and identifying various tags, the method including (i) determining whether a sufficient number of tags having a tag type of interest have been found to indicate that a sample can be authenticated (e.g., minimum number of tags, threshold confidence level, etc.), (ii) determining that a sufficient number of tags have not been found, with or without the tag type of interest, to indicate that a sample cannot be authenticated (e.g., inconclusive or doubtful), or (iii) determining that a sufficient number of tags without the tag type of interest have been found to indicate that a sample is suspicious or mislabeled.

[0037] In some embodiments, the system is controlled by a user via a computer interface (i.e., a User Interface (UI)). In various embodiments, the UI provides means for (i) specifying which product to identify, (ii) specifying which tag type(s) to identify, (iii) specifying which untagped feature metric(s) to identify, (iv) specifying which algorithm(s) to apply (e.g., tags only, one or more untagped feature metric(s) only, tags and untagped feature metric(s), etc.), (v) specifying additional manually entered or automatically captured information about a particular sample, system, or user, (vi) specifying which metadata to report, (vii) providing additional product and / or system background information, and / or (viii) initiating a process to automatically search for and identify tags. For example, 1. Enter the tag types(s) you are interested in into the UI. 2. Manually load and / or place the samples. 3. Set the threshold within the UI. 4. Report the results via the UI (e.g., the number of tag candidates found, the number of tag candidates found that exceed the threshold confidence level, the number of tag patches that exceed the selected confidence level and include the tag type of interest, the number of tag patches that do not include the tag type of interest, and pass / fail output for each tag type of interest). In various embodiments, sample loading and / or placement are automated. For example, an automated sample slide loading and unloading or handling system is integrated into the system to facilitate and rapid automated analysis of large sample sets. In various embodiments, the UI provides options that allow the user to calibrate and / or train algorithms for tag types of interest via guided processing (e.g., an application wizard). For example, the selection of a calibration option guides the user through the process of using a calibration standard (e.g., an optical grid standard, a white balance calibration standard, or any other suitable calibration standard).

[0038] The system addresses the limitations of other machine learning identification techniques (e.g., machine learning identification techniques useful for detecting and identifying tag types) by providing a convenient machine-readable system and method that can identify the sparsity of tags and distinguish one tag type from another.

[0039] Figure 1 shows one embodiment of a tag identification system. In the illustrated example, the tag identification system 100 comprises a light source 102, tagged items 104, a focusing lens 106, an imaging sensor 108, a control unit 110, a central processing unit (CPU) 112, a memory 114, a database 116, and a display 118.

[0040] In the illustrated example, a light source 102 (e.g., ambient light, light from a directional light source) illuminates the tagged article, and the light reflected from the tagged article 104 is guided through a focusing lens 106 to an image sensor 108. An instruction set residing in memory 114 and operating on the CPU 112 instructs the control unit 110 to acquire and receive images of the tagged article 104. One or more images of the tagged article 104 are received by the CPU 112 for processing and / or storage in the database 116. A display 118 provides a user interface that enables the initiation of tag identification processing and the reception of results regarding the tag type. In some embodiments, one or more images of the tagged article 104 are further analyzed for additional features that enable the identification of the tagged article and tag type. In various embodiments, the identified additional features (and / or additional feature metrics) and the identified tag type are used to authenticate the tagged article 104.

[0041] In various embodiments, the light source 102 includes broadband light, narrowband light, coherent light, incoherent light, collimated light, focused light, or any other suitable type of light. In various embodiments, the tagged article 104 includes tagged tablets, tagged packaging, tagged packaging labels, tagged food, or any other suitable tagged article. For example, the tagged article 104 includes pharmaceutical products, food, tablets, capsules, labels, containers, seeds, consumer goods (or parts thereof), electronic materials (or parts thereof), industrial products (or parts thereof), or packaging.

[0042] In various embodiments, the focusing lens 106 includes a plurality of optical components (e.g., lenses, filters, windows, optical flats, prisms, polarizers, beam splitters, waveplates, optical fibers, mirrors, retroreflectors, etc.). In some embodiments, the focusing lens 106 is also used to shape light from a light source 102 (e.g., to illuminate a tagged item 104 with shaped light).

[0043] Light from the light source 102 is reflected from the tagged item 104 and then enters the image sensor 108. In various embodiments, the image sensor 108 includes solid-state sensors, complementary metal-oxide-semiconductor (CMOS) sensors, charge-coupled device (CCD) sensors, gaze-type arrays, red-green-blue (RGB) sensors, infrared (IR) sensors, RGB and IR sensors, Bayer pattern color sensors, multi-band sensors, or monochrome sensors.

[0044] In some embodiments, the CPU 112 is used to control and / or adjust the measurement. For example, the CPU 112 is used to indicate whether to turn the light source 102 on or off, to indicate whether to use the image sensor 108 to acquire an image (e.g., an image of the tagged item 104), to receive data from the image sensor 108, to receive data from the light source 102 (e.g., on / off status), to display an interface (e.g., on the display 118), to receive commands, or for any other appropriate function that the CPU 112 performs when making a measurement.

[0045] In some embodiments, the tag identification system 100 shown in Figure 1 is configured as a compact, self-contained, battery-powered portable system suitable for field use (e.g., in a remote operation environment). In some embodiments, the tag identification system 100 includes a mobile device (e.g., a smartphone or tablet). In some embodiments, the tag identification system 100 includes a microscope. In various embodiments, the tag identification system 100 includes thermal protection and / or ingress protection.

[0046] In various embodiments, the image sensor 108 is used to acquire one or more images of a tagged article 104 illuminated by one or more of ambient light, broadband light, narrowband light, coherent light, incoherent light, collimated light, focused light, or any other suitable type of light (e.g., images of one or more tags on the tagged article 104). In various embodiments, multiple image sensors and / or sensor types are used during the operation of the tag identification system 100 (e.g., when training the tag identification system 100). In various embodiments, partial processing of the images acquired by the image sensor 108, such as adjusting the image white balance, image resolution, etc. (e.g., based on known sensor responses) using a calibration table, is performed as part of or immediately after image acquisition. In various embodiments, acquiring one or more additional images of the tagged article 104 (e.g., acquiring the "next image") repositions the tagged article 104 within the field of view of the image sensor 108 (e.g., as part of training the tag identification system 100, as part of product authentication, etc.). For example, the tag identification system 100 is trained to recognize tags and / or product types based on the diverse spatial and / or directional positional relationships of the tagged article 104 with respect to the image sensor 108, and / or based on the diverse types of image sensors.

[0047] In some embodiments, the CPU 112 is used to determine the type of each of one or more tags on the tagged article 104, and a machine learning algorithm is used for determining the tag type, and the determination of the tag type is based on at least a part of one or more of the following: maximal determination, bounding box generation, tag candidate patch extraction, tag candidate segmentation, tag candidate feature metric calculation, or comparison with a model (e.g., a machine learning model). In various embodiments, the CPU 112 determines the tag type using one or more of the following machine learning algorithms, namely, support vector machine models, neural network models, bounding box models, clustering algorithms, or classifier algorithms.

[0048] In various embodiments, the tag identification system 100 provides a determined tag type for the tagged article 104 and / or provides relevant metadata generated during the analysis of aggregate images obtained from the tagged article 104. In some embodiments, the tag identification system 100 provides identified product information for the tagged article 104. In some embodiments, the tag identification system 100 verifies the authenticity of the tagged article 104 (e.g., determining authenticity using the identified product information and identified tag type of the tagged article 104). For example, the tag type, relevant metadata, product information, and / or product authenticity of the tagged article 104 are provided to the system user via the display 118 or to a cloud-based database for user review.

[0049] In various embodiments, the system user is instructed to contact others based on the information provided. For example, if a tagged item 104 is determined to be a suspicious product by the tag identification system 100, the user is instructed to contact the appropriate individual(s) (e.g., a member of the security team, quality control personnel, a healthcare professional, etc.). In some embodiments, the tag identification system 100 includes integrated email or telephone support. In some embodiments, the tag identification system 100 redirects the system user to another source of product information. For example, a QR code on a tagged item is used to direct the system user to a webpage containing relevant product information, or a serial number read by the tag identification system 100's OCR algorithm redirects the system user to query a cloud-based database to display item information.

[0050] In some embodiments, the tag identification system 100 includes a login function that allows only authorized users to use the system. In some embodiments, the tag identification system 100 includes a calendar reminder function to instruct the system user when to authenticate tagged items. For example, the system user is prompted to take prescribed tagged drug products to ensure patient compliance with improved health outcomes.

[0051] In some embodiments, the tag identification system 100 includes a manual user override. For example, if product information about a tagged item 104 cannot be read from the aggregate image, for example, if the OCR algorithm cannot read or misreads the debossed mark on a tagged pharmaceutical tablet, the system user is prompted to manually add the information. For example, the system user adds the debossed mark information to the tag identification system 100 via the display 118 (e.g., a touchscreen user interface). In some embodiments, a manual user override is used to stop image acquisition (for example, if the CPU 112 cannot automatically stop image acquisition, or if the user decides to interrupt the process).

[0052] In some embodiments, the tag identification system 100 is trained to identify multiple tag types. In some embodiments, the tag identification system 100 is trained to distinguish one tag type from a second tag type. In some embodiments, the tag identification system 100 is used to generate a set of feature metrics (e.g., a set of feature metrics useful for training the tag identification system) from ground truth images of known tag types. Ground truth tag images are typically “true and accurate” tag images created and verified by one or more human experts. In various embodiments, the set of feature metrics obtained from ground truth images of known tag types is used by the tag identification system 100 to identify tag types on a tagged article 104 and / or to distinguish a first type of one or more tags on the tagged article 104 from a second tag type.

[0053] In various embodiments, ground truth information is information about an image (e.g., an image of a tagged sample) or sample (e.g., a tagged sample) that is known to be real or true, provided by direct observation and measurement (i.e., empirical evidence), as opposed to information provided by inference. For example, “known sample” and “known image” include samples and images having corresponding ground truth information for each sample and image.

[0054] In some embodiments, the tag identification system 100 is trained to identify a tagged article 104. For example, the tag identification system 100 is configured and / or trained to recognize additional identifying features of the tagged article 104 in order to identify the tagged article 104. For example, the additional identifying features include one or more of the following: a quick-response code, a barcode, a two-dimensional matrix, a logo, a serial number, the shape of the article, brightness, color, a mark, a symbol, a randomly serialized marker, or any other suitable additional identifying features. In various embodiments, the tag identification system 100 is configured and / or trained to read text or other character information imprinted on the tagged article 104 in order to identify the tagged article 104. In various embodiments, the tag identification system 100 is configured and / or trained to identify and authenticate the tagged article 104 using the additional identifying features in combination with the identified tag type. In various embodiments, the tag identification system 100 is configured and / or trained to identify tagged articles 104 using additional identification features without using identified tag types.

[0055] In various embodiments, the tag recognition system 100 is trained using one or more neural network models. For example, it may be a Convolutional Neural Network (CNN), a Region-based Convolutional Neural Network (R-CNN), a Fast R-CNN model, a Faster R-CNN model, a YOLO model belonging to the You Only Look Once (YOLO) model family, or any other suitable neural network model.

[0056] Figure 2A is a flowchart illustrating one embodiment of the process for generating a first model. In some embodiments, the process in Figure 2A is performed using the tag identification system 100 of Figure 1. In the illustrated example, a known sample is received in 200. For example, a known sample containing tags is received by a user of the tag identification system, and the sample and one or more tag types are known (e.g., determined by direct observation or measurement). For example, multiple tablets are received to which known tag types are applied (e.g., applied to the surface of the tablets via a tablet coating). In various embodiments, different samples containing the same one or more tag types are received. For example, tablets of different sizes, shapes, colors, markings, etc., all having the same one or more tag types are received. In some embodiments, an untagged sample is received. For example, an untagged sample is used to train a first model to recognize an untagged sample or to recognize additional feature metrics of a sample.

[0057] In step 202, the next sample is selected from known samples. For example, the next tablet is selected from a batch of known tagged tablet samples. In step 204, the next sample is positioned under the imaging sensor. For example, the next tablet selected from the batch of known tablet samples is positioned under the imaging sensor at the desired coordinates (e.g., by hand, using a fixture or mount). For example, the sample is positioned by adjusting its position relative to the imaging sensor until one or more tags are observable within the field of view of the tag identification system (e.g., observable by a human user).

[0058] In 206, an image acquisition setting is selected. For example, an image acquisition setting is selected that includes one or more illumination sources, region of interest, focusing algorithm, focusing position, exposure algorithm, exposure value, white balance algorithm, white balance setting, channel or one or more band selection, image processing configuration setting, segmentation configuration setting, tag filtering configuration setting, tag density specification, scan configuration setting, or any other appropriate configuration setting.

[0059] In 208, an image set is acquired and stored. For example, one or more images of one or more tags on a known sample are acquired and stored. For example, one or more images generated by light reflected from one or more tags are acquired and stored by the imaging sensor. In various embodiments, the image sensor includes a solid-state sensor, a CMOS sensor, a CCD sensor, a gaze-type array, an RGB sensor, an IR sensor, an RGB and IR sensor, a Bayer pattern color sensor, a multi-band sensor, a monochrome sensor, or any other suitable type of imaging sensor.

[0060] For example, an image set is acquired using selected image acquisition settings and stored in computer memory or an associated database. In various embodiments, the image set includes multiple images (e.g., 3, 8, 12, 15, 20, 30, 43, 50, 100, 200, 500, or any other suitable number of images). In some embodiments, the number of images in the image set is determined empirically by testing the generated model against known (or ground truth) images to achieve a desired confidence level when classifying tags in the images (e.g., 60%, 70%, 82%, 95%, 98%, 99.3%, or any other suitable confidence level). In various embodiments, the number of images in the image set is determined or limited by the availability of known samples.

[0061] In various embodiments, one or more images are stored in computer memory and / or in a data storage unit (e.g., a data storage unit of a computer system controlling a tag identification system). In various embodiments, one or more images are stored as image files, e.g., TIFF files, PNG files, JPG files, or any other suitable image file format. In various embodiments, one or more images are provided to a system user (e.g., displayed on a monitor of a control computer system) and / or provided to a software application that enables the user to interact with one or more images.

[0062] In step 210, it is determined whether there is a next sample. For example, it is determined whether there is a next tablet remaining in a batch of known tagged tablet samples that have not been imaged. If it is determined that there is no next sample, the process returns to step 202. If it is determined that there is a next sample, the process proceeds to step 212.

[0063] In 212, an image set is provided. For example, an image set from all imaged tablet samples is provided to a computer system (e.g., the memory and / or associated database of a computer system that is part of a tag identification system).

[0064] In step 214, the following image is selected. For example, the following image is selected from a set of images obtained from an imaged tablet sample. In step 216, the following image is preprocessed. For example, the image preprocessing includes one or more of the following: cropping or binning of the image, white balance correction, exposure correction (e.g., automatically or via a calibration table), background color subtraction, background correction (i.e., adjusting for the fact that the surrounding target color may affect the perceived color of the tag), noise reduction, etc.

[0065] In some embodiments, preprocessing includes checking the overall image quality. Image quality attributes considered when determining the overall image quality include one or more of the following: image sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, vignetting, exposure accuracy, chromatic aberration (i.e., color fringing), lens flare, color moiré, and / or data compression and transmission loss (e.g., low-quality Joint Photographic Experts Group (JPEG) images), over-sharpening "halos," loss of fine, low-contrast detail, and other visual artifacts, including reproducibility of the result. In various embodiments, image quality attributes are classified in terms of measuring only one specific type of degradation (e.g., blur, block, or ringing), or by considering all possible signal distortions, i.e., multiple types of artifacts.

[0066] In various embodiments, image quality is evaluated using objective or subjective methods. For example, image quality is evaluated and / or quality scores are provided using image quality methods including one or more of single-stimulus, dual-stimulus, full-reference, reduced-reference, no-reference, or any other suitable image quality methods. For example, images are passed through an image quality filter (e.g., manual or automatic image quality filter) to determine the quality score of each image, and the quality score of each image is used to decide whether to discard the image or use it to generate a first model. In some embodiments, the following image preprocessing is not performed (i.e., step 216 of the processing is skipped).

[0067] In 218, tag candidates are identified. For example, a blob detection method is used to identify a region in the next image that exhibits a different characteristic (e.g., brightness or color) from the entire image background. A blob is a region of an image where several characteristics are constant or nearly constant, and all points within a blob can be considered similar to each other in some sense. For example, a blob detection method includes one or more algorithms based on determining extrema (e.g., local maximums, local minimums, maximum values ​​of a particular color, etc.), color thresholding, differential calculus, etc. (e.g., based on the derivative of a function with respect to position, etc.). In some embodiments, the blob detection method used to identify tag candidates is trained to look for characteristics (e.g., brightness, color, size, etc.) that match the image of the tag.

[0068] In step 220, candidate tag patches are extracted. For example, individual tag and / or tag cluster candidate patches in the following image are cropped from the entire image background. Each individual tag or tag cluster candidate patch includes a localized region that surrounds and contains the image of the tag or tag cluster candidate.

[0069] In some embodiments, patch extraction involves cropping one or more images to generate one or more patch images containing individual tag images and / or clusters of tag images. In some embodiments, morphology and tag contours are used to select clusters of consecutive pixels in a mask that form clusters of tags. This allows training of clustering or extraction algorithms on “atomic” units of interest within the image, minimizing the input size to multidimensional deep networks and enabling preprocessing and normalization on a per-tag basis rather than on the entire image. The processing can be further refined for individual tags using watershed segmentation and erosion / dilation techniques.

[0070] For example, identified blobs in the following image are extracted from the image background after utilizing a bounding box model to localize and / or detect images of tag candidates. In various embodiments, the bounding box model includes one or more of the following: Tensor Flow models, convolutional neural networks, region-based convolutional neural networks (R-CNNs), Fast R-CNN models, Faster R-CNN models, YOLO models belonging to the You Only Look Once (YOLO) model family, EdgeBoxes models, or any other suitable bounding box models.

[0071] In step 222, it is determined whether the next image exists. For example, it is determined whether the next image remains in the set of images obtained from a batch of known tagged samples. In response to the determination that the next image exists, the process returns to step 214. In response to the determination that the next image exists, the process proceeds to step 224.

[0072] In step 224, it is determined whether there are enough tag candidates. In response to the determination that there are insufficient tag candidates, the process proceeds to 238, where a negative result is provided (e.g., provided to the system user via a display in the tag identification system), and the process terminates. In response to the determination that there are enough tag candidates, the process proceeds to 226. For example, if no tag candidates are found, the process cannot continue because there is no data to generate a model. If too few tag candidates are found, it may be determined that such a model is not worth generating and / or testing, and / or that a model generated with too few data points is unlikely to be sufficiently reliable. The determination of whether there are enough tag candidates is an empirically based judgment resulting from working with the process in Figure 2A. For example, it may be empirically determined that a minimum of 30, 50, 100, 200, 392, 5000, 9812, 1,000,000, or any other appropriate minimum number of tag candidates must be found before determining that there are enough tag candidates to continue the process. In some embodiments, a sufficient number of tag candidates is determined empirically by testing the generated model against known (or ground truth) images to achieve a desired confidence level (e.g., 60%, 70%, 82%, 95%, 98%, 99.3%, or any other appropriate confidence level) when classifying tags in the next image. In some embodiments, to determine if a tag candidate is eligible, it is determined whether any of the tag candidate patches extracted from the image match (e.g., exceed a threshold confidence value) a known tag feature in a library of tag types.

[0073] In step 226, a tag candidate segmentation mask is generated. For example, a binary segmentation mask is generated for each tag candidate patch (and / or tag candidate cluster patch) extracted from the following image. Using this mask, only the relevant pixels of that tag candidate are selected, background pixels are zeroed out, and the image is trimmed into small patches that precisely cover the area of ​​the tag candidate or tag candidate cluster.

[0074] For example, a binary segmentation mask is generated using pixels representing tag candidates segmented from an image background. In computer graphics, if a given image is intended to be placed on a background, transparent areas can be specified via a binary mask. Thus, for each intended image, there are actually two bitmaps: the actual image, which is given pixel values ​​with all bits set to 0 in unused areas, and an additional mask, which is given pixel values ​​with all bits set to 0 in the corresponding image area and values ​​with all bits set to 1 in the surrounding area. In some embodiments, a binary segmentation mask is generated for each individual tag image and each cluster of tags.

[0075] Tag candidate segmentation algorithms can be classified into semantic segmentation and instance segmentation. Semantic segmentation involves dividing an image into different semantic parts and assigning each pixel to a class (e.g., tagging the foreground or background). Instance segmentation attempts to identify each instance of the same class by separately detecting and depicting all single tag candidates shown in the image.

[0076] For example, a semantic segmentation program localizes tag boundaries within an image, distinguishes these tag boundaries from the image background, and generates a pixel-level binary mask. Using the mask, tag patches are cropped or extracted from the image background for comparison with a set of known tag features in a library of tag types. In various embodiments, classical imaging techniques for tag segmentation (e.g., watershed transform, morphological image processing, Laplacian of Gaussian (LoG) filters, pixel classification, etc.) are used to detect tag boundaries. In various embodiments, artificial intelligence (AI) imaging techniques (e.g., U-Net (Convolutional Network for Biomedical Image Segmentation), Mask R-CNN, etc.) are used for tag segmentation.

[0077] For example, images of tag candidates are processed to detect and segment the tag pixels from the image background before being fed to a classification algorithm (e.g., a tag feature classification algorithm). In some embodiments, the aggregate image represents the amount of transmitted energy for each set or range of wavelengths of the sample. This provides the spectral reflectance characteristics of the sample. In some embodiments, image segmentation includes divisions of pixels belonging to one or more tag candidates in the spectral image. In some embodiments, the pixel divisions include determining the foreground and background.

[0078] In various embodiments, the aggregate images are processed using a combination of one or more machine learning clustering algorithms (e.g., principal component analysis, K-means, spectral angle mapping initially trained on samples of known tag types, or any other suitable algorithm).

[0079] Principal Component Analysis (PCA) is used to simplify the complexity of high-dimensional data while preserving trends and patterns by transforming the data into fewer dimensions to summarize its features. K-means is an iterative clustering algorithm used to classify data into a specified number of groups by starting with an initial set of randomly determined cluster centers. Each pixel in the image is then assigned to the nearest cluster center by distance, and each cluster center is recalculated as the centroid of all pixels assigned to the cluster. This process is repeated until a desired threshold is reached. Spectral Angle Mapping (SAM) compares a given spectrum to a known spectrum, treats both as vectors, calculates the "spectral angle" between them, and groups them using a threshold based on that angle.

[0080] In some embodiments, using a strictly unsupervised model eliminates the need to manually determine semantic labels. For example, in the field of machine learning, there are primarily two types of tasks: supervised and unsupervised. The main difference between the two types is that supervised learning uses ground truth, meaning there is prior knowledge about the output values ​​that the training samples should correspond to. Therefore, the goal of supervised learning is to learn a function that best approximates the observable input-output relationship in the data, given the data and samples of the desired output. Unsupervised learning, on the other hand, does not have labeled outputs, so its goal is to infer the natural structure that exists within the set of data points. Semantic labeling or semantic segmentation involves assigning class labels to pixels. In a broader sense, the task involves assigning each pixel a label that best matches the local features of that pixel and the labels estimated for the surrounding pixels, based on a consistency model learned from the training data. Semantic labeling, when done manually, is laborious and requires a large amount of domain knowledge. Therefore, by using a strictly unsupervised model, the need to manually determine semantic labels is eliminated.

[0081] Semantic labeling can be contrasted with instance labeling. Instance labeling, or instance segmentation, differs from semantic segmentation. Semantic segmentation refers to the process of associating each pixel of an image with a class label, such as person, flower, or car. It treats multiple objects of the same class as a single entity. Conversely, instance segmentation treats multiple objects of the same class as separate individual instances. For example, suppose we have a Street View input image containing multiple people, cars, buildings, etc. If only group objects belonging to the same category are desired, for example, distinguishing all cars from all buildings, this is a semantic segmentation task. If it is desired to distinguish each individual within each category, for example, people, this becomes an instance segmentation task. Furthermore, if it is desired to perform division at both the category and instance levels, this becomes a panoptic segmentation task.

[0082] In some embodiments, conventional K-means clustering algorithms are sufficient for background-versus-foreground detection. In some embodiments, segmentation performance can be further improved by scaling to a deep autoregressive segmentation model that defines two clusters by separating them based on mutual information.

[0083] In some embodiments, the original image is converted to grayscale, then a thresholding method is applied, and the final output is a binary image with a gray and black mask. Thresholding is a type of image segmentation that modifies the pixels of an image to make it easier to analyze. In thresholding, the image is converted from color or grayscale to a binary image, i.e., simply a black and white image. Most frequently, thresholding is used as a way to select a region of interest in an image, ignoring parts of interest (e.g., separating the foreground from the background). In some embodiments, the foreground and background of the tagged image are used to generate a binary segmentation mask. In some embodiments, the tag morphology and contours are used to generate a binary segmentation mask. In some embodiments, no tag candidate segmentation mask is generated (i.e., step 226 of the process is skipped), and the process proceeds to 228.

[0084] In step 228, it is determined whether a sufficient number of tag candidates pass one or more Quality Control (QC) checks. For example, it is determined whether a sufficient number of tag candidates have sufficient image quality to be useful for generating a first model. In various embodiments, the tag candidate QC check includes verifying whether the spectral characteristics of the tag candidates are within, above, or below a certain threshold limit (e.g., absolute or relative to the background). In various embodiments, the spectral characteristics are determined using a Hue, Saturation, Luminance (HSL) color model, a Hue, Saturation, Value (HSV) color model, a Red-Green-Blue (RGB) color model, or any other suitable type of color model. The relative HSV parameter may be in the form of a relative difference (i.e., tag intensity minus background intensity) or a relative ratio (i.e., tag intensity divided by background intensity).

[0085] In response to a determination that a sufficient number of tag candidates will not pass one or more QC checks, the process proceeds to 238, a negative result is provided (for example, to the system user via a display in the tag identification system being used), and the process terminates. In response to a determination that a sufficient number of tag candidates will pass one or more QC checks, the process proceeds to 230.

[0086] In some embodiments, a QC check is performed before determining whether there are enough tag candidates (i.e., the order of steps 224 and 228 of the process is reversed). In some embodiments, no QC check is performed (i.e., step 228 of the process is skipped).

[0087] In step 230, tag feature metrics are determined. For example, tag feature metrics considered significant for tag type identification, and / or relevant statistical thresholds for each tag type established as significant, are determined for each tag image. In some embodiments, the tag feature metrics are manually determined by a human user to generate a set of feature metrics for each tag image (for example, when using the processing in Figure 2A for the first time to analyze a batch of previously unstudied known samples).

[0088] In various embodiments, the tag feature metric includes one or more of the following: size, shape, color value, saturation value, intensity value, color value standard deviation, saturation value standard deviation, intensity value standard deviation, relative color value, relative saturation value, or relative intensity value, where the relative color value, relative saturation value, or relative intensity value is relative to the image background surrounding one or more tag types.

[0089] In some embodiments, tag feature metrics are determined automatically (for example, by a machine learning model trained to identify relevant tag feature metrics). After training, these models can be used to recognize, identify, and mathematically group feature metrics associated with a specific tag type of interest for known samples.

[0090] In various embodiments, one or more machine learning models are trained using hierarchical feature learning (i.e., "hierarchical learning"). For example, the first layer of a neural network model is trained to learn representations of basic (low-level) features such as edges. Further lower layers of the network output representations of more specific features corresponding to parts of an object (e.g., corners, protrusions). After the last layer, the representations become high-level, allowing the network to clearly separate different objects, having learned to distinguish between different classes.

[0091] In various embodiments, one or more neural network models are trained using spectral data (e.g., spectral data obtained from tags detected by a bounding box model). For example, spectral data obtained from a spectral imaging sensor (e.g., spectral imager, multispectral imager, hyperspectral imager, etc.). In various embodiments, one or more neural network models are trained to match observed spectral data with a library of spectral data corresponding to known tag types.

[0092] In various embodiments, the output of one or more neural network models provides (e.g., to a system user) class labels for identified tag types. In various embodiments, the provided class labels for identified tag types include associated metadata. For example, associated metadata may include date and time, confidence level, number and / or density of identified tag types (i.e., number of tags per unit area), and plots or graphs of associated spectral data.

[0093] In various embodiments, when the presented tagged articles are sparsely tagged, methods for anomaly detection and targeting at the pixel and subpixel scales based on statistical machine learning algorithms are used to rapidly locate and identify individual tags. Sparse distribution of samples may result from economic considerations to minimize the cost of the tags applied and / or to maintain the aesthetic and / or visual concealment of the tagged samples.

[0094] In some embodiments, a first set of tag feature metrics is generated by extracting features of interest from each tag or tag cluster image. For example, to generate a first set of tag feature metrics for each tag image, morphological features of interest (e.g., size, shape, and intensity distribution) are empirically determined and extracted from each tag image (e.g., metrics considered significant criteria for identification, and their respective associated statistical thresholds established as indicating significance). In some embodiments, the features of interest are determined manually by a human user. In some embodiments, feature extraction includes determining a first set of one or more morphological features of interest from each tag image in order to generate a first set of feature metrics.

[0095] In some embodiments, a second set of feature metrics is generated by extracting additional features of interest from each tagged image. In some embodiments, additional morphological tag features (e.g., latent tag features) are extracted by a deep learning method (i.e., "deep feature extraction"). Deep supervised learning can be used to extract additional features of interest to generate morphological features that are difficult and costly to encode manually, or latent features embedded in the data. For example, latent tag features include features, patterns, and differences that are not obvious or discernible to human observation but are determined to be significant by a deep learning method. In some embodiments, deep feature extraction includes determining a second set of one or more morphological features of interest from the tagged images in order to generate a second set of feature metrics.

[0096] In various embodiments, deep feature extraction is achieved by flattening tagged images and feeding them pixel by pixel into a Fully Connected Neural Network (FCNN) or a One-Dimensional Convolutional Neural Network (1D CNN). In some embodiments, for example when using 2D multichannel patches, a Feature Pyramidal Network (FPN) is used to convolve the input tagged images at multiple scales before flattening the features, enabling scale-invariant inputs (e.g., tags of different sizes or images of different magnifications). In various embodiments, the FPN or other suitable neural network model outputs a list of features extracted for each tagged image input.

[0097] In some embodiments, a complete set of tag feature metrics is generated. For example, a complete set of tag feature metrics is generated by "merging" (or combining) (i.e., "feature fusion") a first set of feature metrics. In some embodiments, feature fusion includes adding a second set of feature metrics to a first set of feature metrics to generate a complete set of feature metrics.

[0098] In some embodiments, tagged images (and / or associated tag segmentation masks) are compressed to reduce image file size while maximizing information. To reduce training and prediction times and avoid large memory requirements, tagged images are compressed into embedding features using principal component analysis (PCA). This can be improved (though it incurs considerable memory and computational costs) by implementing a three-dimensional convolutional neural network (3D CNN) autoencoder. An autoencoder is a type of artificial neural network used to learn efficient coding of unlabeled data (i.e., unsupervised learning). The coding is validated and improved by attempting to regenerate the input from the coding. Autoencoders typically learn a representation (i.e., coding) of a dataset for dimensionality reduction by training the network to ignore non-significant data (i.e., "noise"). Compressing to three features enables 2D classification instead of 3D classification, thereby saving time and cost. In some embodiments, feature compression involves spectrally compressing the tagged images to produce compressed tagged images.

[0099] In 232, settings for model generation are provided. For example, settings specific to the acquisition, preprocessing, identification, extraction, segmentation, quality checking, and detection (and / or measurement of feature metrics) of known sample images are provided to the tag identification system's processor. In some embodiments, the settings for model generation are stored in a database (e.g., the tag identification system's database). In some embodiments, the settings for model generation are stored together in a configuration file for later recall (e.g., by a system user).

[0100] In step 234, a first model is generated. For example, a first model is generated that utilizes one or more of the following machine learning algorithms: a support vector machine model, a neural network model, a bounding box model, a clustering algorithm, or a classifier algorithm. In some embodiments, one or more neural network models are used to classify tagged images. The neural network model used to classify tagged images (i.e., the “classifier model”) assigns class labels to one or more tagged images outlined by bounding boxes. In some embodiments, the classifier model uses sharp edge detection. In some embodiments, the classifier model uses machine vision. In some embodiments, the classifier model uses machine learning techniques. For example, the classifier model uses one or more neural network models to classify tagged images detected by the bounding box model. In various embodiments, the one or more neural network models used by the classifier model are the same models, different models, or a combination of the same or different models as those used by the bounding box model.

[0101] In 236, a first model is provided. For example, the first model is provided to a system user of the tag identification system (for example, for verification and testing purposes). In some embodiments, the first model is stored in a database (for example, the database of the tag identification system or any other suitable database).

[0102] In step 238, a negative result is provided and the process terminates. For example, a negative result is provided to a system user of the tag identification system (e.g., via the display of the tag identification system) and the process terminates. In some embodiments, the negative result is stored in a database (e.g., the database of the tag identification system or any other suitable database). For example, a negative result is provided that includes a list of reasons why the process could not produce a first model (e.g., a shortage of identified tag candidates, a shortage of tag candidates that passed the QC check, the actual number of identified tag candidates and / or tag candidates that passed the QC check, or any other suitable information).

[0103] Figure 2B is a flowchart illustrating one embodiment of the process for generating a second model. In some embodiments, the process in Figure 2B is performed using the tag identification system 100 of Figure 1 (e.g., the CPU 112 in Figure 1). In the illustrated example, at 250, a known sample is received. For example, a known sample having one or more additional identification features is received by the user of the tag identification system.

[0104] Additional identifying features and feature metrics include one or more of the following: quick response (QR) codes, barcodes, two-dimensional (2D) matrices, logos, serial numbers, item shape, item size, brightness, color, marks, symbols, randomly serialized markers, text and other symbol types, or any other appropriate additional identifying features. For example, text and other symbol types include one or more of the following: debossed marks on pharmaceutical tablets, unique identifiers engraved on drug capsules, product information printed on labels (e.g., U.S. drug code numbers), logos, serial numbers, etc.

[0105] In various embodiments, known samples are received, with or without one or more tag types (for example, to generate and / or train a second model capable of determining one or more additional distinguishing features of known samples). For example, multiple tablets (e.g., tablets in tablet form) with known debossed marks imprinted on them are received to generate a second model capable of classifying known samples based on additional distinguishing features (i.e., known debossed marks). In various embodiments, multiple additional distinguishing features are used to classify known samples. For example, multiple tablets, each having the same size, shape, color, and debossed marks, are used to generate a second model capable of classifying known samples using multiple additional distinguishing features.

[0106] In step 252, the next sample is selected from known samples. For example, the next tablet is selected from a batch of known tablet samples (e.g., tablets with known debossed marks). In step 254, the next sample is positioned under the imaging sensor. For example, the next tablet selected from the batch of known tablet samples is positioned under the imaging sensor at the desired coordinates (e.g., by hand, using a fixture or mount). For example, the sample is positioned by adjusting its position relative to the imaging sensor until one or more additional identifying features become observable within the field of view of the tag identification system (e.g., observable by a human user).

[0107] In 256, an image acquisition setting is selected. For example, an image acquisition setting is selected that includes one or more selections of the following: one or more illumination sources, region of interest, focusing algorithm, focusing position, exposure algorithm, exposure value, white balance algorithm, white balance setting, one or more channel or band selections, image processing configuration setting, segmentation configuration setting, tag filtering configuration setting, tag density specification, scan configuration setting, or any other appropriate configuration setting.

[0108] In 258, an image set is acquired and stored. For example, one or more images of one or more additional distinguishing features on a known sample are acquired and stored. For example, one or more images produced by light reflected from one or more additional distinguishing features are acquired and stored by the imaging sensor. In various embodiments, the image sensor includes a solid-state sensor, a CMOS sensor, a CCD sensor, a gaze-type array, an RGB sensor, an IR sensor, an RGB and IR sensor, a Bayer pattern color sensor, a multi-band sensor, a monochrome sensor, or any other suitable type of imaging sensor.

[0109] For example, an image set is acquired using selected image acquisition settings and stored in computer memory or an associated database. In various embodiments, the image set includes multiple images (e.g., 3, 8, 12, 15, 20, 30, 43, 50, 100, 200, 500, or any other suitable number of images). In some embodiments, the number of images in the image set is determined empirically by testing the generated model against known (or ground truth) images to achieve a desired confidence level when classifying tags in the images (e.g., 60%, 70%, 82%, 95%, 98%, 99.3%, or any other suitable confidence level). In various embodiments, the number of images in the image set is determined or limited by the availability of known samples.

[0110] In various embodiments, one or more images are stored in computer memory and / or in a data storage unit (e.g., a data storage unit of a computer system controlling a tag identification system). In various embodiments, one or more images are stored as image files, e.g., TIFF files, PNG files, JPG files, or any other suitable image file format. In various embodiments, one or more images are provided to a system user (e.g., displayed on a monitor of a control computer system) and / or provided to a software application that enables the user to interact with one or more images.

[0111] In step 260, it is determined whether there is a next sample. For example, it is determined whether there is a next tablet remaining in a batch of known tablet samples that have not been imaged. If it is determined that there is no next sample, the process returns to step 252. If it is determined that there is a next sample, the process proceeds to step 262.

[0112] In 262, an image set is provided. For example, an image set from all imaged tablet samples is provided to a computer system (e.g., the memory and / or associated database of a computer system that is part of a tag identification system).

[0113] In 264, an algorithm is selected to determine additional distinguishing feature metrics. For example, an algorithm is selected to determine additional distinguishing feature metrics that depend on one or more specific distinguishing feature metrics that have been identified. In some embodiments, one or more algorithms are selected to best identify one or more feature metrics. In some embodiments, one or more algorithms for determining additional distinguishing feature metrics are combined into a single overall algorithm specific to one or more distinguishing feature metrics that have been identified. In some embodiments, an algorithm follows a hierarchy of additional distinguishing feature metric determination. In various embodiments, the algorithm compares the detected additional distinguishing feature metrics with a library of known additional distinguishing feature metrics associated with known product types. For example, before determining the color of a tablet, the shape of the tablet is first determined and compared with a library of known additional distinguishing feature metrics to determine if this shape exists in the known library. If the shape does not exist in the library, model generation can be paused (e.g., by an algorithm that queries the system user whether to continue) to update the library and include new samples. In some embodiments, the library (e.g., a library of known feature metrics for known product types) is dynamic. For example, as the library grows in the process of classifying multiple, new, and additional product types, additional “additional distinguishing feature metrics” may be needed to classify tags or product types that have a subset of similar distinguishing features. In some embodiments, new tag types are added to the library of tag types after the system has been trained to distinguish them from known tag types in the library of tag types. In some embodiments, tag types correspond to a predefined set of feature metric values. In some embodiments, the predefined set of feature metric values ​​corresponding to known tag types is modified as needed as the library of tag types grows.

[0114] In various embodiments, an optical character recognition (OCR) method is used to read text and / or a feature detection algorithm is used to identify other types of symbols imprinted on tagged articles. For example, the text and other symbol types include one or more of the following: debossed marks on pharmaceutical tablets, unique identifiers imprinted on drug capsules, product information printed on labels (e.g., U.S. drug code numbers), logos, serial numbers, item size, item shape, item color, etc. For example, in the case of a solid oral dosage form of a human drug product, the imprinted code on the tablet or capsule, used in conjunction with the product size, shape, and color, can be used to identify the drug product, one or more active ingredients of the product, strength, and manufacturer or distributor. In some embodiments, the OCR method utilizes the Tesseract OCR engine. In various embodiments, the OCR method includes one or more open-source methods (e.g., OCRopus, Kraken, Calamari OCR, Keras OCR, EasyOCR, etc.) or commercially available OCR application programming interfaces (e.g., Amazon Textract, Amazon Rekognition, Google Cloud Vision, Microsoft Azure Computer Vision, Cloudmersive, Free OCR, Mathpix, etc.). In various embodiments, Hu moments, histograms of oriented gradients, key-point matching, or neural network algorithms are used for logo detection.

[0115] In some embodiments, additional identification features include relative inter-tag information. For example, relative distance and / or average distance between tags may be used to distinguish different product classes. In some embodiments, the relative inter-tag information includes relative distances between tags of multiple classes. In some embodiments, the relative inter-tag information includes spectral parameters (e.g., relative hue, relative saturation, difference and / or distribution of relative values). In various embodiments, the relative inter-tag information includes relative frequency distributions of two or more different tag classes.

[0116] In various embodiments, one or more additional identifying features provide product-level identification. In various embodiments, one or more additional identifying features provide instance-level identification of a product.

[0117] In various embodiments, additional identification features are used before determining the tag type. For example, reading a QR code on a tagged item allows for preloading of the correct machine learning model for image acquisition settings and / or tag classification. In various embodiments, the image acquisition settings and / or correct machine learning model for tag classification include one or more illumination sources, regions of interest, focusing algorithms, focusing positions, exposure algorithms, exposure values, white balance algorithms, white balance settings, one or more channel or band selections, image processing configuration settings, segmentation configuration settings, tag filtering configuration settings, tag density specifications, scan configuration settings, or any other suitable feature detection algorithms and / or configuration settings. For example, a barcode on a tagged item allows the system user to load and display relevant product information (for example, this allows the system user to verify that the correct product was identified before item authentication).

[0118] In 266, the following image is selected. For example, the following image is selected from the image set obtained from the imaged tablet sample.

[0119] In 268, the following image preprocessing is performed: For example, image preprocessing includes one or more of the following: image cropping or binning, white balance correction, exposure correction (e.g., automatically or via a calibration table), background color subtraction, background correction (i.e., adjusting for the fact that the surrounding target color may affect the perceived color of the tag), noise reduction, etc.

[0120] In some embodiments, preprocessing includes checking the overall image quality. Image quality attributes considered when determining the overall image quality include one or more of the following: image sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, vignetting, exposure accuracy, lateral chromatic aberration (i.e., color fringing), lens flare, color moiré, and / or data compression and transmission loss (e.g., low-quality JPEG), over-sharpening "halo," loss of fine, low-contrast detail, and other visual artifacts, including the reproducibility of the result. In various embodiments, image quality attributes are classified in terms of measuring only one specific type of degradation (e.g., blur, block, or ringing), or they are classified considering all possible signal distortions, i.e., multiple types of artifacts.

[0121] In various embodiments, image quality is evaluated using objective or subjective methods. For example, image quality is evaluated and / or quality scores are provided using image quality methods including one or more of single-stimulus, dual-stimulus, full-reference, reduced-reference, no-reference, or any other suitable image quality methods. For example, images are passed through an image quality filter (e.g., manual or automatic image quality filter) to determine the quality score of each image, and the quality score of each image is used to decide whether to discard the image or use it to generate a first model. In some embodiments, the following image preprocessing is not performed (i.e., step 268 of the processing is skipped).

[0122] In 270, additional distinguishing feature metrics are determined. For example, additional distinguishing features and / or related additional distinguishing feature metrics that are considered significant for product type identification, and / or related statistical thresholds for each product type that are established as significant are determined. For example, an additional distinguishing feature of color is considered significant for product type identification, but it is determined that the exact color (i.e., related additional distinguishing feature metric) is required to distinguish one product from another. For example, red may be considered helpful in classifying or narrowing down a product type to one of several possible product types that are also "red," but it is determined that the exact shade of red (e.g., red at a wavelength of 632 nm) is required to distinguish a particular product from other products that are red but have a different hue (i.e., a different color wavelength). Furthermore, for example, red at a wavelength of 632 nm associated with a statistical threshold of + / - 3 nm is used to form an acceptable criterion for distinguishing a product from other products that are red but have a different hue.

[0123] For example, additional identifying features and feature metrics include one or more of the following: quick response (QR) codes, barcodes, two-dimensional (2D) matrices, logos, serial numbers, item shape, item size, brightness, color, marks, symbols, randomly serialized markers, text and other symbol types, or any other appropriate additional identifying features.

[0124] In various embodiments, one or more additional distinguishing features and / or additional distinguishing feature metrics are determined. For example, multiple tablets, each having the same size, shape, color, and debossed mark, are determined and used to generate a second model that can classify known samples using multiple additional distinguishing features. In some embodiments, the additional distinguishing features are manually determined by a human user to generate a set of additional distinguishing feature metrics for each product type. In various embodiments, the additional distinguishing features and / or additional distinguishing feature metrics are determined automatically.

[0125] In various embodiments, an optical character recognition (OCR) method is used to read text and / or a feature detection algorithm is used to identify other types of symbols imprinted on tagged articles. For example, the text and other symbol types include one or more of the following: debossed marks on pharmaceutical tablets, unique identifiers imprinted on drug capsules, product information printed on labels (e.g., U.S. drug code numbers), logos, serial numbers, etc. For example, in the case of human drug products in solid oral dosage form, the imprinted code on the tablet or capsule, used in conjunction with the size, shape, and color of the product, can be used to identify the drug product, one or more active ingredients of the product, strength, and manufacturer or distributor. In some embodiments, the OCR method utilizes the Tesseract OCR engine. In various embodiments, the OCR method includes one or more open-source methods (e.g., OCRopus, Kraken, Calamari OCR, Keras OCR, EasyOCR, etc.) or commercially available OCR application programming interfaces (e.g., Amazon Textract, Amazon Rekognition, Google Cloud Vision, Microsoft Azure Computer Vision, Cloudmersive, Free OCR, Mathpix, etc.). In various embodiments, Hu moments, histograms of oriented gradients, key-point matching, or neural network algorithms are used for logo detection. In various embodiments, object detection algorithms are used to identify objects (e.g., alone or in combination with other algorithms as identifying features).

[0126] In step 272, it is determined whether the next image exists. For example, it is determined whether the next image remains in the image set obtained from a batch of known samples. In response to the determination that the next image exists, the process returns to step 266. In response to the determination that the next image exists, the process proceeds to step 274.

[0127] In step 274, it is determined whether there are sufficient additional distinguishing feature metrics. If it is determined that there are not sufficient additional distinguishing feature metrics, the process proceeds to step 284, where a negative result is provided (e.g., to the system user via a display in the tag identification system being used), and the process terminates. If it is determined that there are sufficient additional distinguishing feature metrics, the process proceeds to step 276. For example, if no additional distinguishing feature metrics are found, the process cannot continue because there is no data to generate a model. If too few additional distinguishing feature metrics are found, it may be determined that such a model is not worth generating and / or testing, and / or that a model generated with too few data points is unlikely to be sufficiently reliable. The determination of whether there are sufficient additional distinguishing feature metrics is an experience-based judgment resulting from working with the process in Figure 2B. For example, it may be empirically determined that a minimum of 30, 50, 100, 200, 392, 5000, 9812, 1,000,000, or any other appropriate minimum number of additional discriminant feature metrics (i.e., individual measurements of one or more additional discriminant feature metrics) must be found before determining that there are enough additional discriminant feature metrics to continue processing. In some embodiments, the number of sufficient additional discriminant feature metrics is determined empirically by testing the generated model against known (or ground truth) images to achieve a desired confidence level (e.g., 60%, 70%, 82%, 95%, 98%, 99.3%, or any other appropriate confidence level) when classifying samples in the next image. In some embodiments, it is determined whether any of the additional discriminant feature metrics from the image match (e.g., exceed a threshold confidence value) an additional discriminant feature metric of a known tag in a library of sample types.

[0128] In 276, it is determined whether there are enough additional discriminant feature metrics that pass one or more QC checks. For example, it is determined whether there are enough additional discriminant feature metrics of sufficient quality to be useful for generating a second model. In various embodiments, the QC check of the additional discriminant feature metrics includes checking whether the HSV parameter (e.g., absolute intensity or intensity relative to the background) of the additional discriminant feature metric is within, above, or below a certain threshold limit. The relative HSV parameter may be in the form of a relative difference (i.e., intensity of the additional discriminant feature metric minus background intensity) or a relative ratio (i.e., intensity of the additional discriminant feature metric divided by background intensity).

[0129] In response to a determination that sufficient additional identification feature metrics do not pass one or more QC checks, the process proceeds to 284, a negative result is provided (e.g., to the system user via a display in the tag identification system being used), and the process terminates. In response to a determination that additional identification feature metrics pass one or more QC checks, the process proceeds to 278.

[0130] In some embodiments, a QC check is performed before determining whether there are sufficient additional distinguishing feature metrics (i.e., the order of steps 274 and 276 of the process is reversed). In some embodiments, no QC check is performed (i.e., step 276 of the process is skipped).

[0131] In various embodiments, additional identification feature metrics utilize existing libraries and / or functions. For example, the libraries and functions required to enable QR code recognition may only require the configuration of appropriate image acquisition settings. In various embodiments, the configuration of the libraries and functions used is required. For example, the libraries and functions required to enable OCR may require specifying the number of characters and / or the format of those characters, but image acquisition, preprocessing, determination of feature metrics and / or associated quality score comparison may not be required to generate a second model.

[0132] In some embodiments, existing libraries and functions are combined with existing databases to provide product information based on identification features. For example, a database may be provided that enumerates all possible barcodes so that appropriate product information is provided as output when a barcode is recognized.

[0133] In some embodiments, ground truth images are provided in combination with existing algorithms. For example, in key-point template matching, ground truth images of additional discriminant features are provided and serve as comparison templates from which other images are compared.

[0134] In section 278, settings for model generation are provided. For example, settings specific to image acquisition, algorithm selection, image preprocessing, determination of additional discriminative feature metrics, and checking the quantity and quality of feature metrics in known sample images are provided to the tag recognition system's processor. In some embodiments, the settings for model generation are stored in a database (e.g., the tag recognition system's database). In some embodiments, the settings for model generation are stored together in a configuration file for later recall (e.g., by a system user).

[0135] In step 280, a second model is generated. For example, a second model is generated that utilizes one or more of the following models: an OCR model, an object detection model, a support vector machine model, a neural network model, a bounding box model, a clustering algorithm, or a classifier algorithm. In some embodiments, the OCR model utilizes the Tesseract OCR engine. In various embodiments, the OCR method includes one or more open-source methods (e.g., OCRopus, Kraken, Calamari OCR, Keras OCR, EasyOCR, etc.) or commercial OCR application programming interfaces (e.g., Amazon Textract, Amazon Rekognition, Google Cloud Vision, Microsoft Azure Computer Vision, Cloudmersive, Free OCR, Mathpix, etc.). In various embodiments, Hu moments, gradient histograms, key-point matching, or neural network algorithms are used for logo detection. In various embodiments, an object detection algorithm is used to identify objects, functioning as a distinguishing feature alone or in combination with other algorithms.

[0136] In some embodiments, one or more neural network models are used to classify sample images. The neural network model used to classify the sample images (i.e., the “classifier model”) assigns class labels to one or more portions of the image outlined by bounding boxes. In some embodiments, the classifier model uses sharp edge detection. In some embodiments, the classifier model uses machine vision. In some embodiments, the classifier model uses machine learning techniques. For example, the classifier model uses one or more neural network models to classify sample images detected by the bounding box model. In various embodiments, the one or more neural network models used by the classifier model are the same models, different models, or a combination of the same or different models as those used by the bounding box model.

[0137] In various embodiments, the second model includes additional requirements to limit the possible output responses of the second model and / or to improve the reliability of the output results. For example, a minimum number of keypoints may be defined for the keypoint matching algorithm to ensure sufficient reliability that an image can be compared to a keypoint template.

[0138] In 282, a second model is provided. For example, the second model is provided to a system user of the tag identification system (for example, for verification and testing purposes). In some embodiments, the second model is stored in a database (for example, the database of the tag identification system or any other suitable database).

[0139] In step 284, a negative result is provided and the process terminates. For example, a negative result is provided to a system user of the tag identification system (e.g., via the display of the tag identification system) and the process terminates. In some embodiments, the negative result is stored in a database (e.g., the database of the tag identification system or any other suitable database). For example, a negative result is provided that includes a list of reasons why the process could not generate a second model (e.g., lack of additional identifying feature metrics identified, lack of additional identifying feature metrics that passed the QC check, the actual number of additional identifying feature metrics identified and / or that passed the QC check, or any other suitable information).

[0140] Figure 3A is a flowchart illustrating one embodiment of a process for classifying tags. In some embodiments, the process in Figure 3A is performed using the tag identification system 100 of Figure 1 (e.g., the CPU 112 in Figure 1). In the illustrated example, a library of first models and tag types is received in 300. For example, a set of feature metrics corresponding to the first models and known tag types generated by the process in Figure 2A (i.e., the “library of tag types”) is received by a computer system (e.g., a computer system controlling the tag identification system). For example, the library of tag types includes a set of tag feature metrics and / or relevant statistical thresholds considered significant for tag type identification. In some embodiments, the library of tag types includes a set of feature metric values, each set of feature metric values ​​corresponding to a particular tag type and used to identify a particular tag type.

[0141] In various embodiments, the tag feature metric includes one or more of size, shape, color value, saturation value, intensity value, color value standard deviation, saturation value standard deviation, intensity value standard deviation, relative color value, relative saturation value, or relative intensity value, where the relative color value, relative saturation value, or relative intensity value is relative to the image background surrounding one or more tag types. In various embodiments, the color is determined using a hue, saturation, luminance (HSL) color model, a hue, saturation, value (HSV) color model, a red-green-blue (RGB) color model, or any other suitable type of color model.

[0142] In 302, an unknown sample is received. For example, a sample containing one or more tag types is received by a user of the tag identification system, and the sample and / or one or more tag types are unknown. For example, a tablet to be identified is received to which an unknown tag type has been applied (e.g., applied to the surface of the tablet via a tablet coating).

[0143] In 304, the unknown sample is placed beneath the imaging sensor. For example, the unknown sample is placed beneath the imaging sensor at a desired coordinate (e.g., by hand, using a fixture or mount). For example, the sample is placed by adjusting its position relative to the imaging sensor until the sample is observable within the field of view of the tag identification system (e.g., observable by a human user).

[0144] In 306, the following images are acquired: For example, an image of an unknown sample is acquired by the imaging sensor of the tag identification system. For example, the image is acquired using a predetermined image acquisition setting incorporated into the first model, and this acquired image is routed to the computer memory and / or associated database of the tag identification system. In various embodiments, the image sensor includes a solid-state sensor, a CMOS sensor, a CCD sensor, a gaze-type array, an RGB sensor, an IR sensor, an RGB and IR sensor, a Bayer pattern color sensor, a multi-band sensor, a monochrome sensor, or any other suitable type of imaging sensor.

[0145] In various embodiments, images are stored in computer memory and / or in a data storage unit (e.g., a data storage unit of a computer system controlling a tag identification system). In various embodiments, images are stored as image files, e.g., TIFF files, PNG files, JPG files, or any other suitable image file format. In various embodiments, images are provided to a system user (e.g., displayed on a monitor of a control computer system) and / or provided to a software application that enables the user to interact with the images.

[0146] In 308, the following image preprocessing is performed: For example, image preprocessing includes one or more of the following: image cropping or binning, white balance correction, exposure correction (e.g., automatically or via a calibration table), background color subtraction, background correction (i.e., adjusting for the fact that the surrounding target color may affect the perceived color of the tag), and noise reduction.

[0147] In some embodiments, preprocessing includes checking the overall image quality. Image quality attributes considered when determining the overall image quality include one or more of the following: image sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, vignetting, exposure accuracy, lateral chromatic aberration (i.e., color fringing), lens flare, color moiré, and / or data compression and transmission loss (e.g., low-quality JPEG), over-sharpening "halo," loss of fine, low-contrast detail, and other visual artifacts, including the reproducibility of the result. In various embodiments, image quality attributes are classified in terms of measuring only one specific type of degradation (e.g., blur, block, or ringing), or they are classified considering all possible signal distortions, i.e., multiple types of artifacts.

[0148] In various embodiments, image quality is evaluated using objective or subjective methods. For example, image quality is evaluated and / or quality scores are provided using image quality methods including one or more of single-stimulus, dual-stimulus, full-reference, reduced-reference, no-reference, or any other suitable image quality methods. For example, images are passed through an image quality filter (e.g., a manual or automatic image quality filter) to determine the quality score of each image, and the quality score of each image is used to decide whether to discard the image or use it to generate a first model. In some embodiments, the following image preprocessing is not performed (i.e., step 308 of the processing is skipped).

[0149] In 310, tag candidates are identified. For example, a blob detection method is used to identify a region in the next image that exhibits a different characteristic (e.g., brightness or color) from the entire image background. A blob is a region of an image where several characteristics are constant or nearly constant, and all points within a blob can be considered similar to one another in some sense. For example, a blob detection method includes one or more algorithms based on determining extrema (e.g., local maximums, local minimums, maximum values ​​of a particular color, etc.), color thresholding, differential calculus, etc. (e.g., based on the derivative of a function with respect to position, etc.). In some embodiments, the blob detection method used to identify tag candidates is trained to look for characteristics (e.g., brightness, color, size, etc.) that match the image of the tag. In some embodiments, the blob detection parameters are provided by a first model.

[0150] In step 312, it is determined whether there is one or more tag candidates. If it is determined that there are no one or more tag candidates, the process proceeds to step 328. For example, if no tag candidates are found in the next image, the process is instructed to retrieve another image for retrying (assuming the loop timeout has not been reached) because there is no data to use for classifying the tag type. If it is determined that there is one or more tag candidates, the process proceeds to step 314.

[0151] In 314, candidate tag patches are extracted. For example, individual tag and / or tag cluster candidate patches in the following image are cropped from the entire image background. Each individual tag or tag cluster candidate patch includes a localized region that surrounds and contains the tag or tag cluster candidate image. In some embodiments, patch extraction involves cropping one or more images to generate one or more patch images containing individual tag images and / or clusters of tag images. In some embodiments, morphology and tag contours are used to select clusters of consecutive pixels in a mask that form a cluster of tags. In some embodiments, the candidate tag patch extraction parameters are provided by a first model.

[0152] For example, identified blobs in the following image are extracted from the image background after utilizing a bounding box model to localize and / or detect images of tag candidates. In various embodiments, the bounding box model includes one or more of the following: Tensor Flow models, convolutional neural networks, region-based convolutional neural networks (R-CNNs), Fast R-CNN models, Faster R-CNN models, YOLO models belonging to the You Only Look Once (YOLO) model family, EdgeBoxes models, or any other suitable bounding box models.

[0153] In some embodiments, to determine if a tag candidate patch is eligible, it is determined whether any of the tag candidate patches extracted from the image match (e.g., exceed a threshold confidence value) a known tag feature in a library of tag types.

[0154] In 316, a tag candidate segmentation mask is generated. For example, a binary segmentation mask is generated for each tag candidate patch (and / or tag candidate cluster patch) extracted from the following image. Using this mask, only the relevant pixels of the tag candidate are selected, background pixels are zeroed out, and the image is trimmed into small patches that precisely cover the area of ​​the tag candidate or tag candidate cluster. For example, a binary segmentation mask is generated using pixels representing tag candidates segmented from the image background. In some embodiments, a binary segmentation mask is generated for each individual tag image and each cluster of tags. In some embodiments, the pixel segmentation includes determining the foreground and background. In some embodiments, the foreground and background of the tag image are used to generate the binary segmentation mask. In some embodiments, the tag candidate segmentation mask parameters are provided by a first model.

[0155] In some embodiments, the original image is converted to grayscale, a thresholding method is then applied, and the final output is a binary image with a gray and black mask. In some embodiments, a binary segmentation mask is generated using tag morphology and contours. In some embodiments, image segmentation includes divisions of pixels belonging to one or more tag candidates in a spectral image.

[0156] In various embodiments, tag boundaries are detected using classical imaging techniques for tag segmentation (e.g., watershed transform, morphological image processing, Laplacian of Gaussian (LoG) filtering, pixel classification, etc.). In various embodiments, artificial intelligence (AI) imaging techniques (e.g., U-Net (Convolutional Network for Biomedical Image Segmentation), Mask R-CNN, etc.) are used for tag segmentation. In various embodiments, the following images are processed using a combination of one or more machine learning clustering algorithms (e.g., principal component analysis, K-means, spectral angle mapping initially trained on samples of known tag types, or any other suitable algorithm). In some embodiments, conventional K-means clustering algorithms are sufficient for background vs. foreground detection. In some embodiments, segmentation performance can be further improved by scaling to a deep autoregressive segmentation model that defines two clusters by separating them based on mutual information. In some embodiments, no tag candidate segmentation mask is generated (i.e., step 316 of the process is skipped), and the process proceeds to 318.

[0157] In 318, it is determined whether a sufficient number of tag candidates pass one or more QC checks. For example, it is determined whether a sufficient number of tag candidates have sufficient image quality to provide tag classification. In various embodiments, the tag candidate QC check includes verifying whether the tag HSV parameters are within, above, or below a certain threshold limit (e.g., either absolute or relative to the background). The relative HSV parameters may be in the form of a relative difference (i.e., tag intensity minus background intensity) or a relative ratio (i.e., tag intensity divided by background intensity).

[0158] If it is determined that there are not enough tag candidates and one or more QC checks, the process proceeds to 328. If it is determined that there are not enough tag candidates and one or more QC checks, the process proceeds to 320.

[0159] In some embodiments, a QC check is performed before determining whether there are one or more tag candidates (i.e., the order of steps 318 and 312 of the process is reversed). In some embodiments, no QC check is performed (i.e., step 318 of the process is skipped).

[0160] In step 320, the tag candidate feature metrics are determined. For example, the tag candidate feature metrics considered significant for tag type identification, and / or the relevant statistical thresholds for each tag type established as significant, are determined by the first model for each tag candidate image.

[0161] In various embodiments, the tag candidate feature metric includes one or more of size, shape, color value, saturation value, intensity value, color value standard deviation, saturation value standard deviation, intensity value standard deviation, relative color value, relative saturation value, or relative intensity value, where the relative color value, relative saturation value, or relative intensity value is relative to the image background surrounding one or more tag types. In various embodiments, the color is determined using a hue, saturation, luminance (HSL) color model, a hue, saturation, value (HSV) color model, a red-green-blue (RGB) color model, or any other suitable type of color model.

[0162] In some embodiments, a first set of tag candidate feature metrics is generated by a first model that extracts features of interest from each tag or tag cluster image. In some embodiments, a second set of tag candidate feature metrics is generated by the first model that extracts additional features of interest from each tag image (e.g., using deep feature extraction). In some embodiments, a complete set of tag candidate feature metrics is generated (e.g., generated by the first model). For example, a complete set of tag candidate feature metrics is generated by fusing the first and second sets of tag candidate feature metrics (i.e., "feature fusing"). In some embodiments, feature fusing includes adding the second set of tag candidate feature metrics to the first set of tag candidate feature metrics in order to generate a complete set of tag candidate feature metrics.

[0163] In 322, the tag classification and confidence level are determined. For example, the tag classification and confidence level are determined by a first model. For example, it is determined whether the sample contains multiple tags of a tag type that exceed a threshold confidence level, so the sample is considered to contain a tag type. For example, the tag type is determined by classifying one or more image features of the tag and / or tag cluster and comparing them with known tag feature classifications or types in a library of tag types. For example, the image features of one or more tags and / or tag clusters include spectral and spatial characteristics associated with a particular tag type of interest. Such characteristics include, for example, spectral profiles, spectral signatures, the shape and size of the imaged tags, etc. In various embodiments, the application of the first model for determining the tag classification provides a binary confidence level or reports the confidence as a separate metric (e.g., distance).

[0164] In some embodiments, tag classification includes determining the type of tag in one or more tag candidate segmentation masks (for example, using determined tag candidate feature metrics). In various embodiments, the tag type includes one or more of the following tag types, namely microtags, tagants, chemical markers, physical markers, lugate filters, interference filters, pigments, flakes, platelets, or granules, or any other suitable type of tag. For example, a tag includes one or more of silicon, silicon dioxide, potassium aluminum silicate, mica, titanium dioxide, colored or dyed metals and metallized substrates, polymer materials, combinations of high-refractive-index and low-refractive-index thin films, or any other materials with different properties from the bulk medium in which the tag is embedded (for example, for identification purposes).

[0165] In some embodiments, tag classification utilizes tag shape and / or size information determined from one or more tag candidate segmentation masks. In some embodiments, classification labels include characteristics of the input image (e.g., intensity, shape, or color aspect) summarized by numerical vectors (i.e., feature vectors).

[0166] In some embodiments, each tag candidate segmentation mask is classified (e.g., by assigning a class label to each tag candidate segmentation mask) using probabilities generated by a classification algorithm (e.g., a classification algorithm that is part of a first model). In some embodiments, classification is achieved using an FCNN. In various embodiments, tag type classification utilizes a hybrid deep learning framework, a Long-Short Term Memory (LSTM) network, a deep residual network (e.g., one of the ResNet algorithm families such as ResNet50, ResNet50 with Keras, ResNet-101, ResNet-152, etc.), and / or a 1D CNN.

[0167] In some embodiments, one or more machine learning models (e.g., one or more neural network models that are part of a first model) are used to classify the tag candidate segmentation masks. These may be, for example, support vector machine models, neural network models, bounding box models, clustering algorithms or classifier algorithms, or any other suitable machine learning model.

[0168] In some embodiments, determining a tag classification tag involves matching a candidate tag feature metric with known feature classifications and / or types in a library of tag types that exceed a confidence threshold level (e.g., above 70%, 75%, 80%, 90%, 95%, 98%, or any other appropriate confidence threshold level).

[0169] In step 324, it is determined whether the confidence level of the tag classification exceeds a first threshold confidence level. For example, it is determined whether the confidence level of the tag classification exceeds a first threshold confidence level of 98% confidence. In response to the determination that the confidence level of the tag classification does not exceed the first threshold confidence level, the process proceeds to step 328. In response to the determination that the confidence level of the tag classification exceeds the first threshold confidence level, the process proceeds to step 326. In various embodiments, the threshold confidence level provided by the first model is in the form of a binary threshold, a probability, or any other suitable metric related to a particular model algorithm (e.g., distance, number of keypoints, etc.).

[0170] In 326, a tag classification is provided and the process ends. For example, the tag classification is provided to a system user (e.g., displayed on a monitor of a control computer system) and / or to a software application that enables the user to interact with a program running on the tag identification system, and the process ends. For example, if an unknown sample is a tablet tagged with TruTag® microtags, a tag classification is provided indicating that the tablet has been found to contain one or more TruTag microtags. In some embodiments, the authenticity of the unknown sample is also confirmed (e.g., it is considered authentic by the presence of a known security tag, such as a TruTag microtag). In some embodiments, a confidence level is provided. For example, the tablet is authenticated as genuine with a confidence level of 99%, the result is provided and recorded in the database of the tag identification system, and the user is given the option to analyze another known sample. In various embodiments, the process in Figure 3B is performed simultaneously with (i.e., in parallel with) the process in Figure 3A, or before or after (i.e., in series) the process in Figure 3A, and the user is provided with the product type of an unknown sample, and similarly, the tag type is indicated (for example, indicating that the product is authentic).

[0171] For example, a tablet is identified as a genuine oxycodone hydrochloride tablet containing 10 mg of the active ingredient. In some embodiments, additional identifying features are displayed to the user (e.g., a photograph of the tablet, a manufacturer's logo, a National Drug Code (NDC), an expiration date, or any other appropriate additional identifying information). For example, such additional identifying feature information is stored as metadata in a library of tag types. For example, the relationship between relevant product metadata and TruTag microtags is pre-established and associated with the unique spectral signature of the TruTag microtag. In another example, this is a combination of additional identifying information that provides a link to relevant product metadata stored in a library of tag types (e.g., tablet shape, color, size, debossed mark symbol, etc.) and one or more unique attributes of the TruTag microtag (e.g., spectral signature, microtag quantity (e.g., quantity per unit area), reflectance, or lightness value, etc.).

[0172] In 328, it is determined whether the loop has reached a timeout. For example, whether the loop has reached a timeout is determined based on a predefined configuration setting in the first model, i.e., a configuration setting defined by a limited number of attempts to classify the tag type successfully, i.e., a configuration setting defined by a system user, a configuration setting defined by time, a configuration setting defined by the number of images processed, or any other suitable configuration setting defined in which a timeout has been reached. In response to determining that the loop has not reached a timeout, processing returns to 306. In response to determining that the loop has reached a timeout, processing proceeds to 330. In some embodiments, the timeout is configured to allow the next image indefinitely until manual intervention is performed by the system user.

[0173] In step 330, a negative result is provided and the process terminates. For example, a negative result is provided to a system user of the tag identification system (e.g., via the display of the tag identification system) and the process terminates. In some embodiments, the negative result is stored in a database (e.g., the database of the tag identification system or any other suitable database). For example, a negative result is provided that includes a list of reasons why the process could not classify the unknown tag type (e.g., a lack of identified tag candidates, a lack of tag candidates that passed the QC check, the actual number of identified tag candidates and / or tag candidates that passed the QC check, or any other suitable information).

[0174] Figure 3B is a flowchart illustrating one embodiment of the process for classifying additional identification features. In some embodiments, the process in Figure 3B is performed using the tag identification system 100 of Figure 1 (e.g., CPU 112 in Figure 1). In the illustrated example, a second model and product type library is received at 350. For example, a set of additional identification feature metrics corresponding to the second model and known product types generated by the process in Figure 2B (i.e., the "product type library") is received by a computer system (e.g., a computer system controlling the tag identification system). For example, the product type library includes a set of additional identification feature metrics and / or relevant statistical thresholds (i.e., product type identifications) that are considered significant for the additional identification feature classification.

[0175] Additional identification features and feature metrics include one or more of the following: quick response (QR) codes, barcodes, two-dimensional (2D) matrices, logos, serial numbers, article shapes, article sizes, brightness, color, marks, symbols, randomly serialized markers, text and other symbol types, or any other suitable additional identification features. For example, text and other symbol types include one or more of the following: debossed marks on pharmaceutical tablets, unique identifiers engraved on drug capsules, product information printed on labels (e.g., US drug code numbers), logos, serial numbers, etc. By determining one or more additional identification features and / or feature metrics, the tag identification system can identify product types associated with the determined additional identification features and / or feature metrics within a library of product types. In some embodiments, providing additional identification feature classifications is synonymous with identifying and providing product types.

[0176] In 352, an unknown sample is received. For example, a sample containing one or more tag types is received by a user of the tag identification system, and the sample and / or one or more tag types are unknown. For example, a tablet to be identified is received to which an unknown tag type has been applied (e.g., applied to the surface of the tablet via a tablet coating). In some embodiments, a sample that does not contain any tag types is received by a user of the tag identification system, and the sample is unknown. For example, the user of the tag identification system is only interested in identifying a product type, regardless of whether it contains a security tag or other tag types.

[0177] In 354, the unknown sample is placed beneath the imaging sensor. For example, the unknown sample is placed beneath the imaging sensor at a desired coordinate (e.g., by hand, using a fixture or mount). For example, the sample is placed by adjusting its position relative to the imaging sensor until the sample is observable within the field of view of the tag identification system (e.g., observable by a human user).

[0178] In 356, the following images are acquired: For example, an image of an unknown sample is acquired by the imaging sensor of the tag identification system. For example, the image is acquired using a predetermined image acquisition setting incorporated into the second model, and this acquired image is routed to the computer memory and / or associated database of the tag identification system. In various embodiments, the image sensor includes a solid-state sensor, a CMOS sensor, a CCD sensor, a gaze-type array, an RGB sensor, an IR sensor, an RGB and IR sensor, a Bayer pattern color sensor, a multi-band sensor, a monochrome sensor, or any other suitable type of imaging sensor.

[0179] In various embodiments, images are stored in computer memory and / or in a data storage unit (e.g., a data storage unit of a computer system controlling a tag identification system). In various embodiments, images are stored as image files, e.g., TIFF files, PNG files, JPG files, or any other suitable image file format. In various embodiments, images are provided to a system user (e.g., displayed on a monitor of a control computer system) and / or provided to a software application that enables the user to interact with the images.

[0180] In 358, the following image preprocessing is performed: For example, image preprocessing includes one or more of the following: image cropping or binning, white balance correction, exposure correction (e.g., automatically or via a calibration table), background color subtraction, background correction (i.e., adjusting for the fact that the surrounding target color may affect the perceived color of the tag), and noise reduction.

[0181] In some embodiments, preprocessing includes checking the overall image quality. Image quality attributes considered when determining the overall image quality include one or more of the following: image sharpness, noise, dynamic range, tone reproduction, contrast, color accuracy, distortion, vignetting, exposure accuracy, lateral chromatic aberration (i.e., color fringing), lens flare, color moiré, and / or data compression and transmission loss (e.g., low-quality JPEG), over-sharpening "halo," loss of fine, low-contrast detail, and other visual artifacts, including the reproducibility of the result. In various embodiments, image quality attributes are classified in terms of measuring only one specific type of degradation (e.g., blur, block, or ringing), or they are classified considering all possible signal distortions, i.e., multiple types of artifacts.

[0182] In various embodiments, image quality is evaluated using objective or subjective methods. For example, image quality is evaluated and / or quality scores are provided using image quality methods including one or more of single-stimulus, dual-stimulus, full-reference, reduced-reference, no-reference, or any other suitable image quality methods. For example, images are passed through an image quality filter (e.g., a manual or automatic image quality filter) to determine the quality score of each image, and the quality score of each image is used to decide whether to discard the image or use it to generate a first model. In some embodiments, the following image preprocessing is not performed (i.e., step 358 of the processing is skipped).

[0183] In 360, additional distinguishing feature metrics are determined. For example, additional distinguishing features and / or related additional distinguishing feature metrics that are considered significant for product type identification, and / or related statistical thresholds for each product type that have been established as significant are determined. Additional distinguishing features and feature metrics include one or more of the following: quick response (QR) codes, barcodes, two-dimensional (2D) matrices, logos, serial numbers, article shapes, article sizes, brightness, color, marks, symbols, randomly serialized markers, text and other symbol types, or any other suitable additional distinguishing features. For example, text and other symbol types include one or more of the following: debossed marks on pharmaceutical tablets, unique identifiers engraved on drug capsules, product information printed on labels (e.g., US drug code numbers), logos, serial numbers, etc. In some embodiments, the parameters used to determine the additional distinguishing features are provided by a second model.

[0184] For example, an additional distinguishing feature of color may be considered significant for product type identification, but it may be determined that the exact color (i.e., the relevant additional distinguishing feature metric) is needed to distinguish one product from another. For instance, red may be considered helpful in classifying or narrowing down a product type to one of several possible product types that are also "red," but it may be determined that the exact shade of red (e.g., red at a wavelength of 632 nm) is needed to further distinguish a particular product from other products that are red but have a different hue (i.e., a different color wavelength). Furthermore, for example, red at a wavelength of 632 nm associated with a statistical threshold of + / - 3 nm may be used to form an acceptable criterion for distinguishing a product from other products that are red but have a different hue.

[0185] In various embodiments, one or more additional distinguishing features and / or additional distinguishing feature metrics are determined. In various embodiments, the additional distinguishing features and / or additional distinguishing feature metrics are determined automatically.

[0186] In various embodiments, an optical character recognition (OCR) method is used to read text and / or a feature detection algorithm is used to identify other types of symbols imprinted on the tagged article. In some embodiments, the OCR method utilizes the Tesseract OCR engine. In various embodiments, the OCR method includes one or more open-source methods (e.g., OCRopus, Kraken, Calamari OCR, Keras OCR, EasyOCR, etc.) or commercially available OCR application programming interfaces (e.g., Amazon Textract, Amazon Rekognition, Google Cloud Vision, Microsoft Azure Computer Vision, Cloudmersive, Free OCR, Mathpix, etc.). In various embodiments, Hu moments, histograms of oriented gradients, key-point matching, or neural network algorithms are used for logo detection.

[0187] In step 362, it is determined whether there are sufficient additional distinguishing feature metrics. If it is determined that there are not sufficient additional distinguishing feature metrics, the process proceeds to step 372. If it is determined that there are sufficient additional distinguishing feature metrics, the process proceeds to step 364. For example, if no additional distinguishing feature metrics are found, the process cannot continue without acquiring the next image because there is no data to identify the product type. If too few additional distinguishing feature metrics are found, it may be predetermined that the results are ambiguous or otherwise unreliable. The determination of whether there are sufficient additional distinguishing feature metrics is an empirically based judgment resulting from testing with the process in Figure 3B (for example, after training the process using the process in Figure 2B). For example, it may be empirically determined that a minimum of 3, 5, 10, or any other appropriate minimum number of additional distinguishing feature metrics must be found before determining that there are sufficient additional distinguishing feature metrics to continue the process and achieve the desired level of confidence.

[0188] In 364, it is determined whether a sufficient number of additional distinguishing feature metrics pass one or more QC checks. For example, it is determined whether there are additional distinguishing feature metrics of sufficient quality to be useful for classifying additional distinguishing feature metrics (e.g., to identify the product type of an unknown sample). In various embodiments, the QC check of an additional distinguishing feature metric includes verifying whether the HSV parameter of the additional distinguishing feature metric (e.g., absolute intensity or intensity relative to the background) is within, above, or below a certain threshold limit. The relative HSV parameter may be in the form of a relative difference (i.e., the intensity of the additional distinguishing feature metric intensity minus the background intensity) or a relative ratio (i.e., the intensity of the additional distinguishing feature metric intensity divided by the background intensity).

[0189] If it is determined that there are not enough additional distinguishing feature metrics to pass one or more QC checks, the process proceeds to 372. If it is determined that there are enough additional distinguishing feature metrics to pass one or more QC checks, the process proceeds to 366.

[0190] In some embodiments, a QC check is performed before determining whether there are sufficient additional distinguishing feature metrics (i.e., the order of steps 364 and 362 of the process is reversed). In some embodiments, no QC check is performed (i.e., step 364 of the process is skipped).

[0191] In 366, additional identifying feature classifications are determined. For example, additional identifying feature classifications are determined by a tag identification system to identify the product type of an unknown sample. In some embodiments, it is determined whether any of the additional identifying feature metrics from the following image match (e.g., exceed a threshold confidence value) any known additional identifying feature metrics in a product type library. For example, in the case of a solid oral dosage form of a human drug product, the imprinted code on the tablet or capsule, used in conjunction with the size, shape, and color of the product, can be used to identify the drug product, one or more active ingredients, strength, and manufacturer or distributor. This is made possible by using a look-up table stored in a product type library, and the determined additional identifying feature metrics are linked to relevant drug product information for a given product type.

[0192] In step 368, it is determined whether the confidence level of the additional discriminant feature classification exceeds a second threshold confidence level. For example, it is determined whether the confidence level of the additional discriminant feature classification exceeds a second threshold confidence level of 99.5% confidence. In response to the determination that the confidence level of the tag classification does not exceed the second threshold confidence level, the process proceeds to step 372. In response to the determination that the confidence level of the tag classification exceeds the second threshold confidence level, the process proceeds to step 370. In various embodiments, the threshold confidence level provided by the first model is in the form of a binary threshold, a probability, or any other suitable metric related to a particular model algorithm (e.g., distance, number of keypoints, etc.).

[0193] In step 370, additional identification feature classifications are provided, and the process ends. For example, the additional identification feature classifications are provided to the system user (e.g., displayed on a monitor of the control computer system) and / or to a software application that enables the user to interact with a program running on the tag identification system, and the process ends. For example, if the unknown sample is a tablet, an additional identification feature classification indicating the tablet type is provided. In some embodiments, a confidence level is also provided. For example, the tablet is identified as an oxycodone hydrochloride tablet with a confidence level of 99%. In some embodiments, the authenticity of the unknown sample is also confirmed (e.g., it is considered authentic by the presence of a known security tag, such as a TruTag microtag).

[0194] In various embodiments, the process in Figure 3A is performed simultaneously with (i.e., in parallel with) the process in Figure 3B, or before or after (i.e., in series with) the process in Figure 3B. For example, the user is provided with a response corresponding to a combination of the results of both the tag type determination and the determination of additional identification features (e.g., product type, tag type, product authenticity, etc.). In some embodiments, the process in Figure 3B is not performed at all. In some embodiments, the first and second models are combined into a single model.

[0195] For example, a tablet is identified as a genuine oxycodone hydrochloride tablet containing 10 mg of the active ingredient. In some embodiments, additional identifying features are displayed to the user (e.g., a photograph of the tablet, a manufacturer's logo, an NDC code, an expiration date, or any other appropriate additional identifying information). For example, such additional identifying feature information is stored as metadata in a product type library. For example, the relationship between relevant product metadata and TruTag microtags is pre-established and associated with the unique spectral signature of the TruTag microtag. In another example, this is a combination of additional identifying information that provides a link to relevant product metadata stored in a product type library (e.g., tablet shape, color, size, debossed mark symbol, etc.) and one or more unique attributes of the TruTag microtag (e.g., spectral signature, microtag quantity (e.g., quantity per unit area), reflectance, or lightness value, etc.).

[0196] In step 372, it is determined whether the loop has reached a timeout. For example, whether the loop has reached a timeout is determined based on a predefined configuration setting within the second model, i.e., a configuration setting defined by the system user, a configuration setting defined by time, a configuration setting defined by a limited number of attempts to successfully classify additional identifying features or feature metrics, a configuration setting defined by the number of frames, or any other suitable configuration setting defined in which a timeout has been reached. In response to determining that the loop has not reached a timeout, processing returns to step 356. In response to determining that the loop has reached a timeout, processing proceeds to step 374. In some embodiments, the timeout is configured to allow for an infinite number of subsequent images until manual intervention is performed by the system user.

[0197] In 374, a negative result is provided and the process terminates. For example, the negative result is provided to a system user of the tag identification system (e.g., displayed on the tag identification system's monitor) and / or to a software application that allows the user to interact with a program running on the tag identification system, and the process terminates. In some embodiments, the negative result is stored in a database (e.g., the tag identification system's database or any other suitable database). For example, the negative result is provided which includes a list of reasons why the process was unable to classify additional identification features (e.g., lack of identified additional identification feature metrics, lack of additional identification feature metrics that passed the QC check, the actual number of additional identification features identified and / or that passed the QC check, or any other suitable information).

[0198] While the embodiments described above are explained in some detail for clarity, the present invention is not limited to the details described. Many alternative ways of carrying out the present invention exist. The disclosed embodiments are illustrative and not limiting.

Claims

1. An imaging sensor that acquires one or more images of one or more tags from light reflected from one or more tags on a tagged item, Processor and Equipped with, The aforementioned processor, Upon receiving one or more of the aforementioned images, Receive a library of tag types, Determining a set of feature metrics using one or more of the aforementioned images, wherein the determination of the set of feature metrics is based on a machine learning algorithm and on at least one part of one or more of image processing, image manipulation, or image correction. Determining the tag type of one or more tags in one or more images using the set of feature metrics and the library of tag types, wherein the determination of the tag type is based on at least one of the following: maximal determination, bounding box generation, tag candidate patch extraction, tag candidate segmentation, tag candidate feature metric determination, and comparison with a model. Determine the reliability level of the aforementioned tag type, In response to the reliability level exceeding a threshold level, the determined tag type is provided. A system configured in such a way.

2. The system according to claim 1, wherein the set of feature metrics is determined for each of the one or more images.

3. The system according to claim 1, wherein the set of feature metrics is determined for each of the one or more tags in the one or more images.

4. The system according to claim 1, wherein one of the one or more tags includes a microtag, tagant, chemical marker, physical marker, lugate filter, interference filter, pigment, flake, platelet, or granule.

5. The system according to claim 4, wherein the tag comprises one or more, or a combination of, any other materials having different properties from silicon, silicon dioxide, potassium aluminum silicate, mica, titanium dioxide, colored or dyed metals and metallized substrates, polymer materials, combinations of high refractive index thin films and low refractive index thin films, or any other materials in which the tag is embedded for identification purposes.

6. The system according to claim 1, wherein the tagged articles include pharmaceutical products, food products, tablets, capsules, labels, containers, seeds, consumer goods (or parts thereof), electronic materials (or parts thereof), industrial products (or parts thereof), or packaging.

7. The system according to claim 1, wherein the processor is further configured to determine one or more additional identifying features of the tagged article.

8. The system according to claim 7, wherein one of the one or more additional identification features includes one or more of the following: a quick response code, a barcode, a two-dimensional matrix, a data matrix, a logo, a serial number, an article shape, brightness, color, a mark, a symbol, or randomly serialized markers.

9. The system according to claim 1, wherein the processor is further configured to determine the identification of the tagged article based on at least one of the additional identification features of the tagged article.

10. The system according to claim 1, wherein the set of feature metrics includes one or more tag characteristics that are considered significant for determining the tag type, and / or statistical thresholds established as indicating importance for each, and is used to generate a set of feature metrics for each tag type.

11. The system according to claim 10, wherein the set of characteristic metrics includes one or more of size, shape, color, saturation, or intensity.

12. The system according to claim 11, wherein the color, saturation, or intensity includes an absolute value, a standard deviation, or a relative value.

13. The system according to claim 11, wherein the color is a result of the tag's inherent chemical or physical material properties, or a result of one or more coatings on the tag's surface.

14. The system according to claim 10, wherein the set of characteristic metrics is determined automatically.

15. The system according to claim 10, wherein the set of characteristic metrics is manually determined by a human user.

16. The system according to claim 1, wherein one or more ground truth images of one or more known tag types are used to train the system.

17. The system according to claim 1, wherein the new tag type is added to the library of tag types after the system has been trained to distinguish between known tag types and new tag types in the library of tag types.

18. The system according to claim 1, wherein the tag type corresponds to a set of values ​​for a predefined feature metric.

19. The system according to claim 18, wherein the predefined set of feature metric values ​​corresponding to known tag types is modified as necessary as the library of tag types grows.

20. The system according to claim 1, wherein one or more images are acquired using a mobile device.

21. The system according to claim 20, wherein the mobile device includes a smartphone, a microscope, or a tablet.

22. The system according to claim 1, wherein image segmentation includes divisions of pixels belonging to the tag type within one or more images.

23. The division of the aforementioned pixels includes determining the foreground and background, The foreground and background are used to generate a binary segmentation mask. The system according to claim 22.

24. The system according to claim 1, wherein the image sensor includes a solid-state sensor, a CMOS sensor, a CCD sensor, a gaze-type array, an RGB sensor, an IR sensor, an RGB and IR sensor, a Bayer pattern color sensor, a multi-band sensor, or a monochrome sensor.

25. The system according to claim 1, wherein determining the tag type is done using one or more machine learning algorithms, including support vector machines, neural network models, bounding box models, clustering algorithms, and / or classifier algorithms.

26. Receiving one or more images, wherein the imaging sensor acquires one or more images of one or more tags from light reflected from one or more tags on a tagged article. Receive a library of tag types, Using a processor to determine a set of feature metrics using one or more images, wherein determining the set of feature metrics is based on at least one part of one or more of image processing, image manipulation, or image correction using a machine learning algorithm. Determining the tag type of one or more tags in one or more images using the set of feature metrics and the library of tag types, wherein determining the tag type is based on at least one of the following: maximal determination, bounding box generation, tag candidate patch extraction, tag candidate segmentation, tag candidate feature metric determination, and / or comparison with a model. Determine the reliability level of the aforementioned tag type, In response to the reliability level exceeding a threshold level, the determined tag type is provided. A method that includes doing so.

27. A computer program product that is embodied in a non-temporary computer-readable storage medium, Receiving one or more images, wherein the imaging sensor acquires one or more images of one or more tags from light reflected from one or more tags on a tagged article. Receive a library of tag types, Determining a set of feature metrics using one or more of the aforementioned images, wherein determining the set of feature metrics is based on at least one part of one or more of image processing, image manipulation, or image correction using a machine learning algorithm. Determining the tag type of one or more tags in one or more images using the set of feature metrics and the library of tag types, wherein determining the tag type is based on at least one of the following: maximal determination, bounding box generation, tag candidate patch extraction, tag candidate segmentation, tag candidate feature metric determination, and comparison with a model. Determine the reliability level of the aforementioned tag type, In response to the reliability level exceeding a threshold level, the determined tag type is provided. A computer program product that includes computer instructions for use.

Images