Imaging-based drug content verification in robotic preparation systems

Abstract

The label image information is used to automatically identify and / or verify the identification of those pharmaceutical substances in conjunction with the automatic preparation of pharmaceutical formulations from the pharmaceutical substances. In some examples, the textual content of a substance container is identified using OCR. This encompasses a substantial portion of the tag information intended for manual inspection. As a cross-check, one or more additional visual aspects of the tag, such as a graphical appearance aspect in addition to the textual content itself, are evaluated. Optionally, the risk of misrecognition is evaluated. Optionally, due to the low risk of actual misrecognition, incomplete matching with templates is allowed, and / or for template matches that are close but potentially fuzzy, it is deemed that its risk is non-negligible, and then mitigation actions are taken.

Description

[0001] Related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 470,353, filed June 1, 2023.

[0002] The content of this reference is incorporated into this paper in its entirety.

[0003] Technical Field and Background Technology In some embodiments of the invention, the invention relates to the field of imaging in robotics, and more specifically, but not exclusively, to implementations of systems for specifying and / or verifying drug dosages in robotic preparation.

[0004] An automated pharmaceutical preparation system starts with reagents that can be supplied separately in multiple containers, and then mixes pharmaceutical substances into a common container.

[0005] Pharmaceutical substances are typically supplied in labeled containers. Labeling is generally governed by regulations and best practices, within which individual manufacturers have a degree of design autonomy. Summary of the Invention

[0006] Based on some example aspects of the subject matter currently described, a method for identifying a pharmaceutical substance is provided, comprising: accessing at least one image of a label of a pharmaceutical substance container; accessing one or more label templates, each label template containing data including: text content of a reference label area and at least one indication of a graphic appearance in the reference label area; comparing the image with the one or more label templates, including a comparison of both the text content and the at least one indication of the graphic appearance; and determining an identifier of the pharmaceutical substance based on the result of the comparison.

[0007] Based on some examples of the topics currently described, the text content contains at least one of the generic name of the pharmaceutical substance and the brand name of the pharmaceutical substance.

[0008] Based on some examples of the topic currently being described, the text content specifies the amount of pharmaceutical substance contained in the container.

[0009] Based on some examples of the topic currently described, the quantity is specified by both numbers and units.

[0010] Based on some examples of the topics currently described, the text content indicates the manufacturer's name of the pharmaceutical substance.

[0011] Based on some examples of the topic currently described, each tag template encodes the text content as a numerical sequence of code points.

[0012] Based on some examples of the subject matter currently described, at least one indicator of the graphic appearance includes the image intensity value in the reference label area.

[0013] Based on some examples of the subject matter described herein, at least one indication of the graphic appearance includes a pixel map representation of the reference label area.

[0014] Based on some examples of the subject currently described, at least one indication of the graphical appearance includes an eigenvector for a set of eigenvalue features.

[0015] Based on some examples of the subject currently described, the comparison includes determining label information used in the comparison, and the determined label information includes determining at least some text content displayed in the image of the label, and determining an indication of the graphic appearance in the image of the label.

[0016] Based on some examples of the topic currently described, both the determined text content and the determined indications of the graphic appearance are derived from the same area of ​​the label's image.

[0017] Based on some examples of the topic described above, an OCR engine is used to determine the identified text content of the image of the label.

[0018] Based on some examples of the topic currently described, the determined indications of the graphic appearance include a pixel map selected from the image of the label.

[0019] Based on some examples of the topic currently described, the determined indications of the graphic appearance include feature vectors calculated from the image of the label.

[0020] Based on some examples of the topic currently described, the text content of each tag template is assigned to each of the multiple fields of the tag template's data in the corresponding section, and each field also includes a field identifier.

[0021] Based on some examples of the topic currently described, each field is associated with a section of a reference label, which contains the corresponding portion of the text content for that field.

[0022] Based on some examples of the subject currently described, each field is associated with a corresponding indication in at least one of the indicators of the graphic appearance, and the corresponding indication of the graphic appearance indicates the graphic appearance in the associated area of ​​the reference label.

[0023] Based on some examples of the topics described here, field identifiers at least identify a field for the brand name of the pharmaceutical substance, a field for the generic name of the pharmaceutical substance, and a field for the quantity of the pharmaceutical substance.

[0024] Based on some examples of the topic described here, the field identifier identifies the name of the manufacturer used for pharmaceutical substances.

[0025] Based on some examples of the topic currently being described, the container is a small bottle.

[0026] Based on some examples of the subject matter described herein, the method includes normalizing at least one image according to the three-dimensional structure of the pharmaceutical substance container for comparison with one or more label templates.

[0027] Based on some examples of the subject currently described, the method includes normalizing at least one image based on any one or more of focus quality, illumination level, image contrast, and image perspective shortening, in order to compare it with one or more label templates.

[0028] Based on some examples of the subject currently described, the comparison includes identifying text in at least one region of an image that corresponds in position to text content in a region of a reference label.

[0029] According to some examples of the subject matter currently described, the comparison includes: determining that there is a partial mismatch of text content between one or more label templates and text content identified in at least one image of the label, and determining that there is a sufficient correspondence between at least one image of the label and at least one indication of appearance in a region of a reference label; and assigning an identifier associated with one or more label templates when determining the identifier of a pharmaceutical substance, despite the partial mismatch of text content.

[0030] Based on some examples of the subject matter described herein, the method includes controlling an imager to image at least one image of a label on a pharmaceutical substance container.

[0031] Based on some examples of the subject matter described herein, the method includes controlling a manipulator to manipulate a pharmaceutical substance container to display it to an imager, while simultaneously controlling an image or imaging the pharmaceutical substance container.

[0032] Based on some examples of the subject matter currently described, the method includes receiving a container of pharmaceutical substances.

[0033] Based on some examples of the subject matter described herein, accessing one or more templates includes accessing at least a first template based on the experimental identifier of the pharmaceutical substance, and accessing at least a second template based on the similarity between the second template and the first template.

[0034] Based on some examples of the topics currently described, similarity is determined using a measure of the Hamming distance between the first and the template.

[0035] Based on some exemplary aspects of the subject matter currently described, a system for preparing a pharmaceutical substance is provided, the system including a processor and a memory, wherein the memory contains instructions instructing the processor to perform the following operations: accessing at least one image of a label of a pharmaceutical substance container; accessing one or more label templates, each label template containing data including: text content of a reference label area and at least one indication of a graphic appearance in the reference label area; comparing the image with the one or more label templates, including a comparison of both the text content and the at least one indication of the graphic appearance; and determining an identifier of the pharmaceutical substance based on the result of the comparison.

[0036] Based on some examples of the subject matter described herein, the system includes an imager and an image controller, and the memory contains instructions that instruct the processor to control the image controller to image at least one image of the tag.

[0037] Based on some examples of the subject matter currently described, the system includes a manipulator, and the memory contains instructions that instruct a processor to control the manipulator to position a tag to be imaged within at least one image of the tag.

[0038] Based on some examples of the subject matter currently described, the memory contains instructions that direct a processor to execute any variation of the method for identifying the pharmaceutical substance specified above. For method variations that include a controlled imager, the system includes an imager controller and optionally an imager. For method variations that include a controlled manipulator, the system optionally includes a manipulator.

[0039] Based on some examples of methods in the subject currently described, a method is provided for registering a label template for identifying pharmaceutical substances, the method comprising: accessing at least one image of a reference label; identifying text content in one or more areas of the at least one image; generating one or more corresponding indications of the graphic appearance in the areas of the at least one image; and storing the text content as a label template associated with the corresponding indications of the graphic appearance by area.

[0040] Based on some examples of the topic described here, recognizing text content involves performing optical character recognition (OCR) on at least one image of a reference label.

[0041] Based on some examples of the topic currently described, one or more zones are zones that identify text content by performing OCR.

[0042] Based on some examples of the topic currently described, the method includes assigning text content from a corresponding section to each of a plurality of fields stored in a tag template, each field also including a field identifier.

[0043] Based on some examples of the topic currently described, the assignment is performed according to a dictionary of expected text content patterns in multiple fields.

[0044] Based on some examples of the topic currently described, the assignment is performed according to at least one of the following: the text content itself, the graphic appearance of the area in at least one image in which the text content appears, and the surrounding environment of the area in at least one image.

[0045] Based on some examples from the topic currently being described, heuristics include regular expressions.

[0046] Based on some examples of the topic currently described, heuristics specify the relative positioning of two or more sections of text content.

[0047] Based on some examples of the topic currently described, heuristics specify the relative font size of one or more sections of text content.

[0048] Based on some examples of the topic currently described, at least one image of the reference label is an image of a container that is labeled with the reference label.

[0049] Based on some examples of the topics currently described, at least one image of the reference label is an example image from a regulatory database.

[0050] Based on some example aspects of the subject matter currently described, a system is provided that is configured to register label templates for identifying pharmaceutical substances, the system including a processor and a memory storing instructions, wherein the instructions instruct the processor to: access at least one image of a reference label; identify text content in one or more areas of the at least one image; generate one or more corresponding indications of the graphic appearance in the areas of the at least one image; and store the text content as a label template associated with the corresponding indications of the graphic appearance by area.

[0051] Based on some examples of the subject matter described herein, the memory contains instructions that instruct a processor to control an image controller to image at least one image of a reference label.

[0052] Based on some examples of the subject matter described herein, the memory contains instructions that instruct a processor to control a manipulator to position the reference label in at least one image of the label when the reference label to be imaged is fixed to the container.

[0053] Based on some examples of the topic currently described, the memory contains instructions that instruct the processor to execute any variation of the method for registering the tag template specified above.

[0054] Based on some example aspects of the subject currently described, a method is provided for adjusting registered tag templates to enhance uniqueness, the method comprising: accessing a first tag template specifying a first set of features, the first set of features identifying a tag image as an instance of a first tag type upon matching; accessing a second tag template specifying a second set of features, the second set of features identifying a tag image as an instance of a second tag type upon matching; determining that an estimated error distance between the first and second tag templates is within a threshold range of potential tag template conflicts; and adding at least one specified feature to the first set of features, selecting the at least one specified feature to increase the estimated error distance.

[0055] Based on some examples of the topic currently described, the Hamming distance between the text content of the first and second tag templates is used to estimate the error distance.

[0056] Based on some examples of the topic currently described, at least one specified feature added includes the distinguishing colors of the first and second label templates.

[0057] Based on some examples of the topic currently described, at least one specified feature added includes a feature that distinguishes the text areas of the first and second label templates.

[0058] Based on some examples of the topic currently described, the characteristics of a text area include one or more of the following: text content, text area size, text area orientation, and glyph size within the text area.

[0059] Based on some examples of the subject matter currently described, a system is provided configured to adjust registered tag templates to enhance uniqueness, the system including a processor and a memory storing instructions, wherein the instructions instruct the processor to: access a first tag template specifying a first set of features, the first set of features recognizing a tag image as an instance of a first tag type upon matching; access a second tag template specifying a second set of features, the second set of features recognizing a tag image as an instance of a second tag type upon matching; determine that the estimated error distance between the first and second tag templates is within a threshold range of potential tag template conflicts; and add at least one specified feature to the first set of features, selecting the at least one specified feature to increase the estimated error distance.

[0060] Based on some examples of the subject currently described, the memory contains instructions that instruct the processor to perform any variation of the method relating to adjusting the registered tag template specified above.

[0061] Based on some example aspects of the subject matter currently described, a method is provided for confirming a unique and valid match between a label image and a label template, the method comprising: determining that the label image corresponds to a first label template, the first label template specifying a first set of features sufficient to identify an imaging label as an instance of a first label type; selecting a second label template specifying a second set of features based on an estimated similarity between the second label template and the first label template; determining that the correspondence between the label image and the second label template is within a threshold range of potential label template conflicts; and issuing an alert based on the determination.

[0062] Based on some examples of the subject matter currently described, a system is provided that includes a processor and a memory storing instructions to the processor to perform a method for confirming a unique and valid match between a label image and a label template specified above.

[0063] Alternatively, any suitable component of the system described by the memory having instructions for the processor can be implemented as a fixed circuit system element in the processing circuit system; for example, using one or more application-specific integrated circuits.

[0064] Alternatively, any processing instructions and data elements described herein may be provided on a non-transitory computer-readable medium.

[0065] Based on some exemplary aspects of the subject matter described herein, a method for ensuring the safety of robotic pharmaceutical preparation using a processing circuit system is provided, the method comprising: a) receiving an image of a label on a pharmaceutical substance container; b) classifying the received image as data indicating the properties of the pharmaceutical substance using a trained machine learning model; c) performing optical character recognition (OCR) on the received image, thereby generating OCR text data; and d) utilizing the contents of the pharmaceutical substance container in pharmaceutical preparation in response to a positive correlation between the OCR text data and the properties of the pharmaceutical substance.

[0066] Based on some examples of the subject matter currently described, the method further includes: e) issuing an alert in response to a negative correlation between OCR text data and the properties of pharmaceutical substances.

[0067] Based on some exemplary aspects of the subject matter described herein, a system is provided to ensure the safety of robotic pharmaceutical preparation, employing a processing circuit system configured to: a) receive an image of a label on a pharmaceutical substance container; classify the received image into data indicating the properties of the pharmaceutical substance using a trained machine learning model; perform optical character recognition (OCR) on the received image, thereby generating OCR text data; and d) utilize the contents of the pharmaceutical substance container in pharmaceutical preparation in response to a positive correlation between the OCR text data and the properties of the pharmaceutical substance.

[0068] Based on some exemplary aspects of the subject matter currently described, a computer program product is provided, the computer program product including a non-transitory computer-readable storage medium storing program instructions that, when read by a processing circuitry system, cause the processing circuitry system to execute a computerized method for ensuring the safety of robotic pharmaceutical preparation, the method comprising: a) receiving an image of a label of a pharmaceutical substance container; b) classifying the received image as data indicating the properties of the pharmaceutical substance using a trained machine learning model; c) performing optical character recognition (OCR) on the received image, thereby generating OCR text data; and d) utilizing the contents of the pharmaceutical substance container in pharmaceutical preparation in response to a positive correlation between the OCR text data and the properties of the pharmaceutical substance.

[0069] Based on some exemplary aspects of the subject matter described herein, a security-enhanced pharmaceutical preparation apparatus is provided, including a processing circuit system (PC) configured to: a) obtain data of desired content indicating a pharmaceutical dosage (PD) using an identification device; b) lock a receiving container of the PD into the pharmaceutical preparation apparatus; c) initiate a pharmaceutical preparation sequence based on the desired content using robotic capabilities to create the PD; d) at least in response to the successful completion of the pharmaceutical preparation sequence, perform an image scan of the created PD using a camera, thereby generating a PD image; and e) in response to detecting data in the PD image depicting the desired content indicating the PD using imaging processing techniques, unlock the PD from the pharmaceutical preparation apparatus using robotic capabilities.

[0070] According to some examples of the subject matter described herein, the PC is further configured to obtain data indicating desired content of the PD by obtaining the identifier of the PD; and wherein the PC is further configured to, after a), transmit the identifier to a control server; and wherein the PC is configured to receive data indicating desired content from the control server, the desired content being based on the identifier; and wherein the PC is further configured to, after d), transmit an indication of successful completion of the pharmaceutical preparation sequence to the control server; thereby avoiding the need for direct access to the electronic medical record (EMR) system by the pharmaceutical preparation device and achieving privacy-preserving integration of pharmaceutical preparation and the EMR system.

[0071] Based on some examples of the topic currently described, the data indicating the desired content of the PD is a scan of a label, which includes at least one of a group consisting of barcodes, QR codes, and identifier text.

[0072] Based on some examples of the topics described here, the PC is configured to lock the receiving container before initiating a drug preparation sequence.

[0073] Based on some examples of the subject matter described herein, the PC is configured to lock the receiving container after initiating a drug preparation sequence.

[0074] Based on some examples of the topic described here, the PC is configured to lock the receiving container in response to instructions from the control server.

[0075] Based on some example aspects of the subject matter currently described, a non-transitory computer-readable medium is provided that stores a database of templates for identifying drug labels, each template including multiple fields identified according to the type of information represented by the fields, and each field being associated with a text string representing text in a section of the drug label and an indication of the graphic appearance in the section.

[0076] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. While similar or equivalent methods and materials to those described and materials herein may be used in the practice or testing of embodiments of this disclosure, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including its definitions, shall prevail. Furthermore, materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

[0077] As those skilled in the art will understand, aspects of this disclosure can be embodied as a system, method, or computer program product. Therefore, aspects of this disclosure can take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining hardware and software aspects (all collectively referred to herein as "circuit," "module," or "system") (e.g., a method can be implemented using a "computer circuit system"). Furthermore, some embodiments of this disclosure can take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon. Implementations of methods and / or systems of some embodiments of this disclosure can involve performing and / or completing selected tasks manually, automatically, or in combination thereof. Furthermore, according to the actual instruments and apparatus of some embodiments of methods and / or systems of this disclosure, several selected tasks can be implemented by hardware, by software, or by firmware and / or by a combination thereof, for example, using an operating system.

[0078] For example, hardware for performing selected tasks according to some embodiments of this disclosure may be implemented as a chip or circuit. As software, selected tasks according to some embodiments of this disclosure may be implemented as multiple software instructions executable by a computer using any suitable operating system. In some embodiments of this disclosure, one or more tasks performed in the method and / or by the system are performed by a data processor (also referred to herein as a "digital processor," meaning a data processor operating using digital bit sets), such as a computing platform for executing multiple instructions. The instruction execution element of the processor may include, for example, one or more microprocessor chips, ASICs, and / or FPGAs. Optionally, the data processor includes volatile memory for storing instructions and / or data and / or non-volatile memory for storing instructions and / or data, such as magnetic hard disks and / or removable media. Optionally, network connectivity is also provided. Optionally, a display and / or a user input device, such as a keyboard or mouse, are also provided. Any of these embodiments is more generally referred to herein as an example of a computer circuit system.

[0079] Any combination of one or more computer-readable media may be used in some embodiments of this disclosure. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compressed optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may also contain or store information for use by such programs, such as data constructed in a manner recorded in the computer-readable storage medium such that a computer program can access it as, for example, one or more tables, lists, arrays, data trees, and / or other data structures. In this document, a computer-readable storage medium that records data in a form that can be retrieved as a group of digital bits is also referred to as a digital memory. It should be understood that, in some embodiments, a computer-readable storage medium may optionally also be used as a computer-writable storage medium when it is not inherently read-only and / or is in a read-only state.

[0080] In this document, a data processor is referred to as being "configured" to perform data processing actions, provided that it is coupled to a computer-readable medium to receive instructions and / or data from it, process them, and / or store the processing results in the same or another computer-readable medium. The processing performed (optionally with respect to data) is specified by instructions, the effect of which is that the processor operates according to the instructions. The actions of processing may be referred to by one or more other terms, either additionally or alternatively; for example: comparison, estimation, determination, calculation, identification, association, storage, analysis, selection, and / or transformation. For example, in some embodiments, a digital processor receives instructions and data from a digital memory, processes the data according to the instructions, and / or stores the processing results in the digital memory. In some embodiments, "providing" the processing results includes one or more of transmitting, storing, and / or presenting the processing results. Presentation may optionally include displaying on a display, providing audio instructions, printing on a printout, or giving the results in a form acceptable to human senses.

[0081] A computer-readable signal medium may contain a propagated data signal embodying computer-readable program code, such as in baseband or as part of a carrier wave. This propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and may transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, device, or apparatus.

[0082] Program code embodied on a computer-readable medium and / or data used therefrom may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0083] Computer program code for performing operations of some embodiments of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. Additionally or alternatively, sequences of logical operations (optionally corresponding to computer instructions) can be embedded in the design of an ASIC and / or the configuration of an FPGA device. The program code can be executed entirely on the user's computer, partially on the user's computer (e.g., as a standalone software package), partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can connect to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The following describes some embodiments of the present disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create components for implementing the functions / actions specified in the flowcharts and / or one or more block diagram blocks.

[0085] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device or other apparatus to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing containing instructions that implement the functions / actions specified in flowcharts and / or one or more block diagram blocks.

[0086] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0087] Some of the methods described in this paper are typically designed for use by computers only; and may be infeasible or impractical for human experts to perform entirely manually. Human experts who wish to perform similar tasks manually (such as inspecting objects) might expect to use entirely different methods, such as leveraging expert knowledge and / or the pattern recognition capabilities of the human brain, which would be far more efficient than manually performing the steps of the methods described in this paper. Attached Figure Description

[0088] This document describes some embodiments of the present disclosure by way of example only, with reference to the accompanying drawings. Reference will now be made in detail to the drawings, and it should be emphasized that details are shown by way of example and for the purpose of illustrative discussion of embodiments of the present disclosure. In this regard, the description taken in conjunction with the drawings will make it clear to those skilled in the art how embodiments of the present disclosure can be practiced.

[0089] In the attached diagram: Figure 1A Example robotic pharmaceutical preparation systems are shown, illustrating some examples based on currently published topics; Figure 1B Examples of pharmaceutical container labels are shown schematically, based on some examples of currently published topics; Figure 2 This is a flowchart illustrating a method for registering labels for use in pharmaceutical preparation systems, based on some examples of currently disclosed topics; Figure 3 This schematically illustrates elements assigned to fields in a structured template, based on some examples of currently published topics. Figure 4 This is a flowchart of a method labeled as container verification, based on some examples of currently publicly available topics; Figure 5 It is a flowchart of a method for tag identification and / or verification based on some examples of currently publicly available topics; Figure 6 This is a schematic flowchart illustrating methods for adjusting registered tag verification templates to enhance uniqueness based on some examples of currently publicly available topics; Figure 7 This is a schematic flowchart illustrating methods for detecting obfuscation risks during tag template matching, based on some examples of currently available topics. Figure 8 Examples of robotic injection preparation systems employing machine learning-based validation of pharmaceutical ingredients are shown, based on some examples of currently disclosed topics. Figure 9A Examples of drug labels that can be used in a robotic injection preparation system employing machine learning-based drug ingredient verification are shown, based on some examples of currently disclosed topics. Figure 9BThis illustrates, based on some examples of currently published topics, the rotation of a container in front of an online scanning camera; and Figure 10 The flowchart illustrates an example method for machine learning-based drug ingredient validation, based on some examples of currently published topics. Detailed Implementation

[0090] In some embodiments of the invention, the invention relates to the field of imaging in robotics, and more specifically, but not exclusively, to implementations of systems for specifying and / or verifying drug dosages in robotic preparation.

[0091] Overview A broad aspect of some examples of the currently disclosed subject matter involves the use of label image information in the automatic identification and / or verification of pharmaceutical substances. In some examples, the identification and / or verification of pharmaceutical substances is combined with the preparation of pharmaceutical formulations using the pharmaceutical substances; for example, automated preparation via a pharmaceutical preparation system.

[0092] Pharmaceutical preparation systems transfer pharmaceutical substances between containers, including, for example, extracting substances from containers, injecting substances into containers, and / or mixing substances within containers. It should be understood that, for example, the activities and outcomes of automated pharmaceutical preparation systems need careful validation, taking into account the risks associated with administering medication to patients. Specifically, under the assumption of direct human technician involvement in pharmaceutical preparation tasks, certain aspects of how information is transferred along the pharmaceutical distribution chain have been developed. Specifically, information on the labels of pharmaceutical containers can be structured to reduce confusion for human readers without necessarily presenting completely equivalent information in a standardized form for machine reading. For example, machine-readable identifiers (where present) are often numbers that reference the required information but do not themselves describe it. This introduces a dependency potentially associated with risk. Alternatively or additionally, utilizing human-readable information potentially includes important cross-checking of machine-readable information and / or reducing the risk of confusion that may arise when human operators rely on label information that differs from that used by the automated system they are supervising.

[0093] Whenever automated systems replace human intervention, there is a potential need to provide those systems with the ability to convert human-targeted information sources into machine representations of the information. Furthermore, a potential advantage is that this capability is self-verifiable and / or its reliability is easily analyzed.

[0094] Problems with automatic drug label recognition Regulatory agencies around the world have made considerable efforts to ensure that pharmaceutical materials are clearly and completely labeled.

[0095] However, standards can vary between different regulatory jurisdictions. Furthermore, drug labels are often not regular and standardized in every detail. Besides carrying many different pieces of information, they can be visually complex. Textual information can be provided in various fonts, sizes, and colors. Some information may be stylized (made partially decorative) in the form of logos or other trademarks. The information provided on the label can include fixed text (e.g., name and substance quantity) and variable text (e.g., batch number and expiration date). Text is often oriented in multiple directions (e.g., two or more from left to right, right to left, top to bottom, and / or bottom to top). Additionally, labels may use manufacturer-selected coloring and / or graphic designs to achieve visual differentiation of drug containers (e.g., based on dosage) and / or grouping (e.g., based on substance type and / or manufacturer). This type of variability presents potential problems for machine-based recognition of drug labels, especially since this variability may appear in future label designs (therefore, it needs to be adapted to this label type in a fairly generic way).

[0096] Machine-oriented labeling, such as barcodes and / or OR codes, is commonly present, but its use is not universally standardized or mandatory. Even when present, machine-oriented labeling can provide incomplete information. For example, it may simply provide a compactly written numerical indication uniquely assigned to a particular pharmaceutical manufacturer and formulation. Determining the actual specifications of a pharmaceutical container based on such a numerical indication may require further access to a database that indexes the numerical indication to information such as the manufacturer's actual name, the active ingredient of the drug, and / or the dosage contained therein. Even when the label itself actually provides all the necessary dosage information in some other (human-readable) form, this potentially introduces an external dependency into the identification of the contents of the labeled container. The reliance of automated systems solely on such machine-oriented labeling also, at least in principle, leads to a potential discrepancy between what a person handling the container will read and what a machine handling the container will read.

[0097] The label may be applied incorrectly (e.g., bubbling, wrinkling, and / or incorrect orientation) and / or may contain printing errors. The label may be damaged during storage and / or handling, thereby reducing its readability.

[0098] To complicate matters further (at least for automated label recognition), labels on two medicine containers can sometimes be very similar in almost every aspect, even if they differ on key factors such as the total amount of the drug substance contained in the container. For example, in the text content, differences as small as a single digit in a number or a single character in a unit name can be critical in distinguishing between safe and unsafe doses. Similarly, different doses of the same substance from the same manufacturer may differ only in one or two digits of the machine-directed identification code.

[0099] Furthermore, the three-dimensional shape of the tagged container (e.g., cylindrical shape) may result in the label not having a single imaging viewpoint that clearly contains all the information required for automatic container identification.

[0100] While manufacturers can attempt to reduce the risk of label confusion for human readers through techniques such as different printed layouts of text, different use of colors, different patterns, and / or different overall packaging sizes and / or shapes, content that may be readily and intuitively obvious to humans is not necessarily in the prompts that automated systems are configured to utilize.

[0101] Structured templates for tag recognition In some examples of the currently disclosed topics, at least some of these problems are mitigated by creating and / or using structured templates to identify labels for pharmaceutical substance containers (e.g., vials).

[0102] Supplementary specifications for label content Some examples of the currently disclosed topics involve the creation and / or use of structured templates that contain supplementary specifications for the labeled content of one or more zones: text content, and some additional representations or methods (different from the recognized character sequences of the text) for evaluating the graphical appearance associated with said text content. For example, regarding Figure 3 Examples describing complementary specified data in a structured template.

[0103] The structured template is also referred to herein as a "label template". A structured template is constructed relative to one or more reference labels. Reference labels can originate from an actual instance of a label container. Reference labels can be provided separately from a specific container (e.g., as a flat, still unused sticker). Reference labels can be a digital representation and / or specification of a label and / or derived from a digital representation and / or specification of a label; for example, an image file. Optionally, information from multiple reference labels and / or reference label images can be combined to generate the structured template.

[0104] Text content can be understood as tagged information conveyed using character glyphs, regardless of the geometric details of the actual glyphs used. Specifically, text content can be equivalently represented in different numerical forms. For example, text content can optionally be represented as a sequence of character code points; such as character code points specified by Unicode, ASCII, or another text encoding standard. If compression is applied to the numerical representation of the text content, the compression is reversible, allowing the recovery of the character code points in the same sequence as before compression.

[0105] The remaining label information is assigned to a category of graphic appearance. In a structured template, this may optionally be indicated by a representation of intensity values ​​in an example image of the label and / or a portion thereof; these intensity values ​​may be, for example, the intensity values ​​of a pixel array and / or intensity values ​​associated with one or more geometrically specified shapes. In a structured template, the graphic appearance of the label area is not necessarily represented with fully recoverable fidelity. For example, lossy compression methods may be applied to the data representing the graphic appearance as an image.

[0106] In some examples, graphical appearance is indicative, but not necessarily representational (however, graphical appearance representation is considered a type of indicative). For example, a graphical appearance can be statistically indicative as, for instance, a histogram of intensity values ​​in a region. In another example, hashing or vectorization algorithms can be used that are expected to produce the same or similar numerical results for image regions that are “sufficiently similar” under some relevant metric, but do not contain enough information to allow reconstruction of the image region itself, and may not even represent intensity values. Some examples of graphical appearance evaluation using machine learning products (e.g., the learned weight set of a neural network) can be considered “indicative,” for example, as long as their results indicate which label type a certain label instance belongs to, rather than representing the label appearance itself. In other uses of machine learning products, vectorization can be reversibly converted into some kind of graphical representation, for example, used as a feature vector of a set of feature values.

[0107] When at least one of the two labeling and identification methods occasionally fails, combining the two methods can reduce risk. In pharmaceutical manufacturing, the risk of error can be considered a critical risk because misidentification of substances and / or materials can lead to patient harm. Conversely, the requirement for absolute certainty in a single label identification method may not be technically feasible. For example, a single method may be error-prone and / or probabilistic. Alternatively, from a risk analysis perspective, relying on the absolute effectiveness of individual methods may be considered unacceptable. For example, even if the method itself is generally “perfect” in assessing a match, it can be considered susceptible to systematic risks arising from its application environment (e.g., database errors or abnormal lighting conditions).

[0108] Especially relevant to text content: While (irreversible) hashing algorithms can optionally be used as part of the manipulation of text content representation, the direct or explicit recoverable representation of the character code point sequence of the marked area in a structured template is a potential advantage, as stated above.

[0109] Even in the case of automatic tag recognition, this constraint helps to preserve individual tag instances as self-contained indicators naming their contents. This "self-contained" nature corresponds to the meaning of tags as self-contained indicators that are also understandable to the human individual reading them. Specifically, names and quantities are directly indicated, rather than merely indirectly and through reference, such as through the accompanying figure marks assigned to such information. It should be understood that "self-contained" in this sense is for information typically learned from reading tags and does not require the indications to convey the importance of common conventions, such as the background conventions of language and linguistic representations that are accessible to the human reader of the tag.

[0110] The following concerns the contribution of graphic appearance representation: Although modern optical character recognition (OCR) typically excels in recognizing text present in images, the possibility of character misrecognition is high enough that it is preferably considered a risk to automated label recognition systems. For example, character misrecognition can occur when fonts (especially stylized fonts that may exist as brand or manufacturer names) have unusual glyph forms and / or glyph positioning, when character sequences are anomalous, and / or simply due to imaging conditions such as focus quality and imaging noise.

[0111] Therefore, in some examples of the currently disclosed topic, graphical appearances are used to correct errors in text content recognition and / or verify text content that has been correctly recognized. For a given tag instance that matches a certain tag template, one or both false positives (incorrect match with the tag template) and false negatives (failed to match the tag template correctly) can be resolved.

[0112] For example, a potential advantage is that automatic recognition is robust to OCR errors; roughly to some extent, it is also correctly recognized as usable by the automated system when the label is clear and correct to a human reader. For instance, rejecting a substance container for use in pharmaceutical preparation might be unacceptable due to a single character error in text recognition, especially if such an error is an artifact limited by the OCR engine. At the same time, preferably, the automated system should not over-accept errors.

[0113] Considering a supplementary category of automatic label recognition implementations that omit text content recognition (e.g., OCR): label matching based solely on, for example, one form of (non-text) pixel pattern matching can also be considered risky. Depending on the method used, the risk may be attributed, for example, to the inherent probabilistic nature of the method, and / or to the use of threshold-based parameters, where the threshold setting is somewhat arbitrary.

[0114] There are potential risks in relying solely on the visual appearance of tags while ignoring the actual determination of the "self-contained" information carried by tags for the average human reader. For example, the risk lies in the possibility that a matching template might be misidentified in the database (leading to errors that a person seeing the tag wouldn't make), or subtle tag tampering might "trick" an automated system into misclassifying it, while human users would only notice the plain text content. Using OCR-based methods, the textual content of the tag and the tag's identifier are closely linked, thereby reducing this specific type of bias risk.

[0115] Structured recognition of tag content fields Some examples of the currently disclosed subject matter involve the creation and / or use of structured templates that assign corresponding field identifiers to each of the multiple specific sections of a label, specifying the meaning of the data it describes. Field identifiers are, for example, one of “brand name,” “generic name,” “material quality,” or “manufacturer.” However, field identifiers are not necessarily given by the name of the field (if one exists); for example, fields can be given arbitrary strings of identifiers, numbers, and / or stored in a predetermined order.

[0116] The location of a region can be identified using any suitable data. For example, the location of a rectangular region can be identified by its horizontal and vertical offset from the label boundaries, as well as its horizontal and vertical extent. Non-rectangular regions are not excluded. In some preferred examples, structured templates restrict the location where information from fields can appear. This offers a potential advantage by reducing the likelihood of unexpected similarities between different labels. It should be noted that OCR engines typically include modules that perform the task of recognizing regions containing text. Therefore, the recognized text is associated with a specific region at its origin.

[0117] Field values ​​do not necessarily have to be text. Optionally, the graphic appearance is evaluated for one or more non-textual features of the label, such as the logo shape and / or background color. In some examples, features of the container construction that differ from the label itself are incorporated into the structured template. For example, the size and / or shape of the container can be considered as descriptive field values. In some examples, the container size and / or shape are referenced in the structured template to aid in the geometric normalization of the label. For example, a label surrounding a cylindrical portion of a vial may have areas that appear perspective shortened, out of focus, and / or disproportionate. Optionally, information about the shape of the container to which the label has been applied is used to help determine which parts of the label image are valid for analysis, and / or how parts of the label image should be transformed (e.g., during preprocessing) for use in structured template matching.

[0118] While field values ​​are typically conveyed by an image of a specific area of ​​the label, field values ​​can in some cases be global characteristics of the label; for example, its size and / or shape. Label areas associated with different fields are not necessarily mutually exclusive; for example, the color and text content of the same area can be associated with different corresponding fields. Rewritten text (two overlapping texts) is also not excluded. While labels designed for clarity will generally avoid self-overlapping text, certain imaging conditions (e.g., translucency of a glass vial) can cause translucency from one label section to another. In another example of a similar rewritten text, partially transparent or semi-transparent labels can be applied to overlap each other.

[0119] Optionally, for text that is not important and / or secondary in terms of the recognition requirements, such as text indicating stored and / or processed information that has no particular significance for the recognition purpose, the text content and / or graphic appearance can be evaluated, but may differ in their specific wording and / or graphic appearance. In some examples, such text may contain repetitions of information available in an important field, and optionally, such repetitions are identified and incorporated into the template matching process (e.g., optionally verifying the consistency of two matching information fragments).

[0120] Optical character recognition of tag information In some examples, optical character recognition (OCR) techniques are used, such as using OCR libraries previously known in the art (e.g., Tesseract or EasyOCR) to determine the text content (e.g., the values ​​of fields).

[0121] As mentioned above, the typical output of an OCR subroutine indicates both: the location where text is found in the image; and the determined text content (which characters it contains and in what order). In some examples, 3D label geometry (the label shape once applied to the container) is used to help transform the image into a form suitable for OCR.

[0122] In some examples, the text within the labels is freely recognized by the OCR engine, including the free recognition of text location. In some examples, data stored in available structured templates is used to constrain the location of searchable text and / or the type of text that can be seen at a certain location. In some examples, free-location OCR and location-constrained OCR are mixed; for example, location-constrained OCR is used to limit template comparisons to possible matches, but free-location OCR is used to avoid imposing excessive constraints on the analysis of the actual labels being examined, which could interfere with match determination.

[0123] Alternatively, in some examples, the first stage of the OCR algorithm can be run, which only identifies the location where text is visible on the label (e.g., relative to the label edge or other landmarks), without recognizing the text itself. This can then be used to select a reasonable subset of the structured templates that are being attempted to match it in detail.

[0124] To help reduce the number of comparisons computed, the selection of structured templates for comparison may optionally be based on previous (recent) results and / or expected matching structured templates. Optionally, structured templates considered similar to the most recently matched templates may also be included. This has the potential advantage, for example, by helping to avoid over-commitment to such expected and / or historical results. Optionally, the structured template search may be expanded when the expected template match is weak (e.g., below a certain scoring threshold).

[0125] The text content (at least when it is correct) directly indicates information important for the use of the pharmaceutical substance. For a correct match, an exact match with the text description of the need is possible and preferred, but not necessarily required.

[0126] If the text content of the label itself is correct to some extent, but is misidentified by OCR, it may be sufficient to refrain from using the pharmaceutical substance as a precaution. While erroneous disposal may lead to inefficiency, it can optionally be corrected by a human operator.

[0127] To potentially reduce this inefficiency, certain types of typical OCR error recognitions may occur, which are considered to be within acceptable tolerances; for example, replacing "1" with "l" or "0" with "O" when everything else is correctly recognized, and / or when this has previously been approved by human oversight. In some cases (e.g., when the text content is stylized and / or uses unusual fonts), the level of permissible OCR errors can optionally be increased (e.g., the number of errors and / or the list of permissible but incorrect character replacements). The discussion of tolerance is further described below.

[0128] For patients, the greatest potential risk is that misidentified (e.g., OCR-recognized) text strings might happen to resemble the expected text characters, allowing misplaced medicine containers that should be prohibited to be accepted for use. Restrictions on field values ​​reduce this possibility to some extent. However, risk assessments may still need to consider cases where containers differ critically in the dosage they contain, but have the same field layout (e.g., they have the same manufacturer, brand name, and generic substance name, all located in the same area on their labels).

[0129] More generally, for example, regarding Figure 4 Frame 416 Figure 5 box 512 and Figure 2 Section 216 discusses the use and / or implementation of OCR and / or text content recognition.

[0130] Tag information verification based on graphic appearance To mitigate the risks associated with tag recognition, there is a potential synergistic effect in the mixed use of graphic appearance with text content (e.g., as recognized using OCR). The graphic appearance within the recognized text area can be characterized in one or more aspects. Examples include: the size of the area; the size, spacing, and / or font of the recognized letterforms; and / or the arrangement of pixel values ​​within the same text area of ​​the image. Optionally, the graphic appearance can be further transformed and / or measured to be stored in a structured template of the tag; for example, compression, conversion to contours, thresholding, evaluation of color distribution (histogram), or other image transformations and / or analyses. As mentioned above, the graphic appearance can be at least partially represented (e.g., as geometric features and / or intensity), or indicated only by indicators that can be compared to it (e.g., hash values).

[0131] For textual content, information about the three-dimensional shape of the label applied to the container can be used to generate appropriate geometric transformations for image comparison with a structured template. Even when photographed from different angles, such as when a label wrapped around a circular vial is rotated differently relative to an imager (e.g., an optical imaging camera), geometric transformations may allow for comparison of label image regions. The transformation can be chosen deterministically (e.g., based on knowledge of the container shape recorded in the template) or selectively (e.g., through a process that minimizes iterative errors) to optimize the similarity between the image and the template. Optionally, the geometric transformation is chosen as "global" (e.g., with relatively few parameters) so that imperfections in the label's appearance are not ignored by the transformation's parameters.

[0132] Alternatively, when identifying and / or verifying the identifiers on drug substance labels, any one or more of several different methods may be used as the basis for a second criterion of using graphic appearance to supplement textual content.

[0133] In some examples, the structured template contains a pixel map of the text area. When used in label recognition, it may be required not only that the recognized text content of the label fully matches the text specified by the template, but also that the pixel map of the text area corresponds to the pixel map of the template (e.g., matching according to some analytical metric, optionally within some tolerance). This may detect cases where text is misrecognized due to label distortion (e.g., creases on the label)—distortions that affect the relative positioning of text characters, thus preventing a match of the overall graphic appearance of the area. Conversely (e.g., to mitigate OCR false negatives), errors in OCR recognition of text can optionally be overcome by a sufficiently close match based on a comparison of pixel map values.

[0134] Another approach (as an addition or alternative to the above) uses metrics derived from the process of recognizing text content. These can include, for example, the entire range of the text and / or the position of individual characters.

[0135] Another approach involves optionally using differential imaging (image subtraction and / or division) after appropriate normalization to account for differences in, for example, focus, illumination level, and / or image contrast. The comparison criterion could be “zero difference,” with some noise tolerance, for example, based on calculations and / or tolerances set based on test images with and / or without defects in the labeled image. Optionally, differences in different sub-regions are expected to be similar in magnitude, and if they are not similar, they are optionally marked as an indication of mismatch. For example, a misplaced letter might cause a sub-region to “stand out” with a higher error than other letters, possibly different from the character identifier that OCR can assign to the region.

[0136] In another approach, geometric transformations are allowed to minimize the difference between the template image region and the imaging label region to achieve any desired local or global offset. The transformation itself is then examined to determine if it contains indications of defects in the label, such as introducing discontinuities not visible in the template image region.

[0137] In some examples, machine learning is used to help identify non-textual features of the labels. For example, for each label to be distinguished by the system, one or more training examples are created; for example, images of the labels viewed from one or more rotational directions, optionally with actual and / or synthetically generated label defects. In some examples, training is extended to use actual and optionally synthetically generated labels (e.g., examples available from a regulatory database of labels) consisting of a large database of typical examples. Variable regions (e.g., regions given batch numbers and / or expiration dates) may be optionally excluded, and / or generated so that they are also variable in the training dataset.

[0138] In some examples, machine learning receives feedback based on whether it correctly classifies a label into a specific label type; for example, regardless of whether it correctly assigns different views of a label example to the same label category. Alternatively, machine learning reinforces classification based on features in the label; such as key characteristics like manufacturer, brand name, generic name, and / or quantity of contents. Optionally, training is hierarchical, for example, it includes training to distinguish between “critical regions” (regions that directly provide identification information) and “non-critical regions” of the label. Subsequent training focused on the identification itself can potentially benefit from this limitation on which regions of the label are most prominent in identifying the pharmaceutical content described by the label.

[0139] For example, regarding Figure 4 Box 416 and Figure 5 Box 514 discusses the use and / or implementation of graphical appearance comparisons in label identification and / or verification.

[0140] Assessment of the risk of misidentification Some examples of the currently disclosed topics involve the assessment of the risk of misidentification of labels on pharmaceutical substance containers.

[0141] For automated label identification and / or verification methods involving thresholds and / or potential boundary conditions, there may be issues with determining the associated risks. For example, it may be valuable to be able to assess the risk level of false positives or false negatives and / or to provide methods for selecting algorithm parameters that keep the risk at an acceptable level.

[0142] While OCR recognition is prone to errors, a potentially useful feature is that its errors are inherently “countable.” This has a potential advantage for generating Hamming distance metrics (and / or similar metrics), where the “proximity” (i.e., distance or similarity) of a labeled image to a structured template can be determined based on how many character changes are needed for them to match precisely. Alternatively, errors can be weighted to indicate more significant and less significant substitution errors; for example, substituting “o” for “0” could be weighted to add less “distance” than substituting “w” for “q” because the glyph shapes are more similar, and / or because the correct value can be more easily guessed given that an error has occurred. Additionally or alternatively, actual examples of substitution errors and / or the error rate can be observed and used to score the significance of errors.

[0143] Specifically, because there are relatively few characters on the label, the Hamming distance between text strings (character sequences) can be calculated very quickly. This can optionally be used to reduce the overall number of image-to-image comparisons performed as part of image verification by comparing only the label image with a structured template that has fields with similar text values. Even largely general comparisons are possible; for example, comparisons with every known label of a pharmaceutical substance registered in the jurisdiction, or comparisons with every known label within a broad category (e.g., packaging type and / or manufacturer).

[0144] In some examples of the currently disclosed subject, distance calculations are used to determine where and / or how many OCR errors are permissible when matching a particular label against a structured template. Where the structured template database is accepted to substantially cover the entire domain of possible labels, increasing the error count in a given text field (e.g., "manufacturer") may be acceptable, as long as this does not lead to any possibility of incorrect cross-identification. It should be noted that if high-quality matches are produced for other fields, the domain of label substitutes that need to be considered for a particular field may be significantly narrowed. For example, once the manufacturer has been confidently identified, consideration of error distance can be limited to only knowing the quality of the goods that the given manufacturer is to package. In some examples, based on reasonable options available given the text values ​​determined for one or more other fields of the structured template, the vocabulary that the OCR engine can use to identify the text in the first field is selected (e.g., limited).

[0145] Hamming distance can optionally be used to determine the risk of misidentification (and / or its relative lack thereof). The situation where two labels in the database (i.e., their corresponding structured templates) are very similar to each other in their text and / or roughly as similar to examples of a particular label can optionally be considered as indicating a risk of confusion.

[0146] In the former case, mitigation can optionally be introduced as part of the registration, allowing a larger number of distinguishing features to be used later as part of the structured template matching. For example, label color can be introduced as a distinguishing factor in some sub-regions of a label, even if it is not usually explicitly considered and / or required.

[0147] In the latter case, even if a good match has been found, mitigation can take the form of issuing a human intervention alert, because it may be uncertain whether the good match is genuine or whether a small number of errors in label recognition have led to a false match. Alternatively, the case of a “tight match” in the OCR space (e.g., a small Hamming distance in text values) may optionally lead to greater weighting of the classification of non-OCR aspects of structured template matching (e.g., image analysis-based methods) and / or machine learning products.

[0148] It should be noted that "distance" evaluation is not limited to Hamming distance, nor is it limited to use with textual information. For example, an image of a label (the entire label or any suitable portion thereof) can optionally be "vectorized" into a histogram of its pixel values ​​(each value of the histogram provides a dimension), and then appropriately normalized. The vector distance between two such histograms can then be readily computed and used as the basis for evaluating the similarity between the imaged label and a structured template.

[0149] In some examples, machine learning is used to create a “label vector space”; that is, a logical space of label features (distinguished by machine learning), into which a given label image (optionally including associated OCR-recognized text) can be encoded as a vector. Each label image can then be compared to the encoded vectors of a structured template to determine which such vector they are most similar to. As described with respect to Hamming distance, there is also a potential advantage for detecting ambiguous situations. It should be noted that machine learning can spontaneously incorporate relevant non-textual features (e.g., the unique use of color and / or pattern in otherwise closely related labels) as part of developing a vector space that sufficiently distinguishes examples from the training set.

[0150] However, it should be noted that validating machine learning products themselves (e.g., verifying that they do not produce erroneous indications during use, or even determining the likelihood that this might occur) can be a difficult task. In contrast, it may be easier to infer risk levels for algorithms designed to be effective. Therefore, when selecting a specific design for an automated identification and / or verification system for pharmaceutical container labels, the ability to convincingly demonstrate that the risk level is computable (e.g., rather than merely observable in test results that may or may not fully reflect the conditions of use) should be considered.

[0151] Registration of structured templates Some examples of the currently disclosed topics involve registering labels for pharmaceutical substance containers into a database of structured templates for such labels.

[0152] The process of creating structured templates for labels (and / or label plus container) is referred to herein as “template registration” (i.e., registering structured templates describing labels for use by pharmaceutical preparation systems). In this document, for example, regarding… Figure 2 Describe the registration method for structured templates.

[0153] The data on the label of a medicine container is not necessarily arranged according to a completely self-descriptive scheme or one that is entirely subject to customary and / or regulatory requirements. Template registration may optionally occur in conjunction with any suitable combination of automated and manual or auxiliary processes.

[0154] Optionally, in some examples, the field identifiers (their “meanings”) of the text area of ​​the label are assigned by intervention from a human operator responsible for registering specific labels for use by the label recognition system. In some preferred examples, automatically recognized label information (e.g., OCR results) is presented to the human operator, along with user interface options to annotate and / or correct these results appropriately.

[0155] For example, a user interface is provided to a human operator, configured to allow indication of where a field with a specific identifier appears on a label (e.g., along with which OCR-recognized text). Optionally, the operator corrects and / or actually provides the text value from the label.

[0156] In some examples, field identifiers are automatically assigned based on heuristics applied to the text content and one or both of the text content’s graphical appearance, location and / or context (with or without subsequent human correction and / or confirmation via the user interface).

[0157] It should be understood that the application of heuristics can be structured in any suitable manner. For example, several heuristics can be evaluated, and the results can work together to aid in identification. In another example, a heuristic may be dominant when it “matches” certain text, but otherwise considered indeterminate, with the result then used to query other heuristics for evaluation.

[0158] In some examples, the appropriate field identifier for the text content is determined by utilizing a vocabulary with field restrictions—for example, a vocabulary constructed using manufacturer names, brand names, and / or generic substance names. Matches within the vocabulary assigned to a particular field identifier are interpreted as indications that the text content is a value of said field identifier.

[0159] Alternatively, heuristics used for text content can themselves be compactly described as regular expressions. For example, a regular expression:

[0160] This can be roughly described in natural language as "an integer in grams, milligrams, or micrograms" (optionally followed by an asterisk); optionally followed by "per vial", "per container", " / vial", or " / container"; with an optional space separator. To allow for OCR errors, common errors can be explicitly included (e.g., allowing o or O instead). (Allowing 'l' to replace '1'). Some regular expression languages ​​allow specifying the number and / or type of errors for a single character as part of the regular expression itself.

[0161] As an example of a heuristic not based on the text value itself: the text "500 mg" somewhere on the label may indicate the actual amount of the vial's contents, but there may be other numbers and units indicated on the label. Optionally, heuristics for assessing the likelihood that "500 mg" refers to the amount of the vial's contents may include, for example, relatively large font size, and / or text that is relatively closer to and / or similar in orientation to the text giving the brand name and / or generic name of the substance.

[0162] Specifically (but not exclusively), when the text content of two different structured templates is similar, additional heuristics and / or operations can be applied to attempt to identify distinguishing features. For example, the color used in association with text can be considered distinguishable. In another example, text areas outside the text area associated with a specific field can be subjected to OCR, or simply their size, position, and / or shape can be identified and evaluated. Alternatively, the graphic appearance (distinct from the text content) can be evaluated for such areas or for any other areas of a label.

[0163] In some examples, structured templates are registered entirely based on automated processes; for example, registration based on label images obtained from a regulatory database of such labels. Such structured templates can potentially be used to determine the risk of unintended conflicts with fully registered templates (e.g., those that have been reviewed and / or corrected by human users), even if not considered to be entirely trusted to identify the substance container. For example, after a user registers a new label, the system can determine that the structured template of the new label is close enough to an automatically registered label (optionally unreviewed) that there is a risk of conflict. The system can then present the automatically registered structured template to the user for any additional verification and / or correction; potentially including the definition of further distinguishing fields to reduce the likelihood of misidentification. Optionally, the system applies heuristics to automatically create one or two distinguishing field information within the structured template.

[0164] In another example, new registration information (e.g., corrections made by a human user to a structured template of a tag) may optionally be applied to related tags. A tag being "related" can be determined, for example, based on image and / or text content similarity. For instance, two or more tags may share the same manufacturer, but partial registration is maintained after automatic registration because it is difficult to identify the manufacturer's name from the stylized text. Once the user provides the manufacturer's name for one of the tags (the corresponding image area is available), the system has information that can be used to identify the same pixel pattern in other tags and associate it with the same manufacturer name. This may optionally be done automatically; or automatically using, for example, user confirmation and / or correction of the input portion.

[0165] definition The subject matter of this disclosure generally relates to robotic pharmaceutical preparation systems, and more specifically, to fluid transfer stations within robotic pharmaceutical preparation systems. It should be understood that, for brevity and clarity, the examples described herein (with reference to the accompanying drawings and others) are subsets of components of a pharmaceutical preparation system; for example, specific aspects of an overall fluid transfer device assembly are described. Furthermore, examples directly similar to those described herein should be understood to be covered within the scope of this disclosure. This includes, for example, specific embodiments and / or combinations of elements and / or subsystems that differ in detail from those explicitly described but are similar in function and / or functional role.

[0166] Pharmaceutical preparation system: Embodiments of the robotic pharmaceutical preparation system and its fluid transfer station described herein are configured to perform operations related to the transfer of pharmaceuticals between different fluid transfer devices.

[0167] A robotic pharmaceutical preparation system (which may alternatively be referred to as a "robotic system") according to the subject matter of this disclosure comprises elements and / or subsystems, such as robotic stations, robotic arms, motors, control units, and / or other mechanisms that operate to perform, control, and / or verify fluid delivery. These elements are optionally designed and / or described as constituting units and / or subsystems and / or comprised of units and / or subsystems operable to perform activities related to the preparation of a pharmaceutical product intended for administration to a patient. For example, a robotic system may include one or more automated or partially automated subsystems, each including at least one manipulator at least partially controlled by a controller unit (equivalently referred to as a controller or control unit). The controller units themselves may be arranged, for example, hierarchically and / or communicatively with each other in a network to coordinate the overall operation of the pharmaceutical preparation system.

[0168] The pharmaceutical preparation system described herein facilitates fluid transfer between a container and a fluid transfer unit specified by a fluid transfer device. The latter typically acts as an intermediate element (usually, but not necessarily, an "active" element, e.g., an element that causes fluid movement by generating pressure), and the former is considered either a fluid source or a fluid receiving element (usually, but not necessarily, a "passive" element). Fluid transfer can be assisted by a fluid transfer connector. However, a typical intermediate fluid transfer unit can still optionally operate as an initial source of fluid (e.g., in the form of a pre-filled syringe provided at the start of pharmaceutical preparation) and / or as a final container for the fluid (e.g., in the form of a filling unit, which is then transferred forward to another process, such as delivery to a patient, storage, or another purpose). The intermediate role of the container as both a fluid source and a fluid receiver is also not excluded.

[0169] Embodiments of delivery devices may include, for example, one or more catheters, pumps, syringes, vials, intravenous bags, adapters, and / or needles. Optionally, these elements are consumables and / or accessories for a pharmaceutical preparation system. However, alternatively, such elements are considered components of a pharmaceutical preparation system. The term vial is specifically used herein as a term for a container in which contents, beginning in a solid and / or relatively concentrated form, are initially dissolved and / or diluted. In other examples, the vial contains a certain amount of fluid, regardless of whether it was initially intended as a container for solids and / or concentrates. In other examples, the vial may still retain substances in solid and / or concentrated forms. However, the term vial is not limited to this list of examples.

[0170] Fluid: As used herein, “fluid” generally includes pharmaceuticals, diluents, salt solutions, water, or any other fluid used in the preparation of pharmaceuticals. More specifically, fluid can be understood as being provided as a liquid, although the use of gaseous fluids is not excluded, provided their properties are consistent with those described herein.

[0171] Fluid transfer: "Fluid transfer" between a container assembly and a fluid transfer assembly is carried out through openings formed in the ports of the container assembly or fluid transfer assembly and / or through openings formed in the diaphragms of the container assembly or fluid transfer assembly.

[0172] Diaphragm: In this document, "diaphragm" generally refers to a membrane configured to close an inlet to a part of its constituent device. A diaphragm on a container or container connector (also known as a container diaphragm) seals the container. A diaphragm on a fluid transfer assembly (also known as a fluid transfer connector diaphragm) prevents or blocks entry into and / or through a fluid transfer conduit. Typically, diaphragms are made of a resilient, puncture-resistant material. Such materials can be polymers with elastic properties, like rubber. For example, vials are often fitted with caps that integrate diaphragms.

[0173] Container: In performing fluid transfer, the robotic system operating according to this disclosure may optionally manipulate and / or inspect containers embodied differently. As described herein, "container" may optionally refer to any one or more of the following: syringe, IV bag, elastic pump, vial, bottle, ampoule, syringe, tubing, conduit, or generally any vessel or container suitable for containing fluid or liquid. It should also be understood that a container may be any other element operating as a component of a fluid transfer device, with or without a connector (or "adapter") for establishing fluid communication between the container and other fluid transfer components. For example, a container may be a vial with a vial adapter, or an intravenous bag with a point adapter. Access to the container may be through a container diaphragm, which may be a diaphragm of the container cap or may be part of a connector. In fluid transfer using fluid pressure changes and / or differential pressure, the container characteristically experiences fluid transfer due to pressure changes within and / or generated by the fluid transfer assembly to which the container is connected.

[0174] Vial: As mentioned herein, a “vial” (e.g., a container) can be a sealable vessel, such as an ampoule or bottle, containing a liquid or powdered medicine, for example, made of glass or plastic. Vials can be single-use or reusable. Vials can be tubular or bottle-shaped, with a neck portion near the opening. The top of the vial can be capped, for example, with a septum. The shape of a vial is generally fixed, and its internal volume is specifically constant, although the volume can be filled to a greater or smaller extent. In some examples, a vial is also a preparation container in which the supplied substance is dissolved, diluted, and / or reconstituted in preparation for further operations, such as transfer to a syringe. Therefore, operations performed on a vial typically involve one or both of injecting and removing a fluid. Between these two, there may be mixing operations, such as dissolving and / or diluting the originally contained substance with the injected fluid. Vials are typically supplied for manual preparation options and are not necessarily standardized in size. Vials can be labeled to suit manual operation, but this presents potential disadvantages for automated operations, such as obstructing the view of the vial's contents.

[0175] Container Assembly: As mentioned herein, a “container assembly” may comprise: a separate container, or a container to which a container connector is mounted. The term “vial assembly” is used equivalently, although examples embodying aspects of this disclosure do not necessarily include vials in the strict sense (e.g., ampoules may be present instead). A diaphragm for at least partially sealing the vial inlet may be positioned as part of the vial itself and / or part of the container connector (equivalently referred to as a “vial adapter” or “container adapter”). The container connector may include means that can be mounted onto the vial to facilitate the transfer of the vial itself (by gripping onto the adapter rather than gripping the vial) and / or facilitate the transfer of fluid into or out of the vial. The container connector may provide protected (e.g., “closed” and / or sterile) contact with the contents of the vial. The container connector may be a single-use or reusable sterile device. The vial in the container assembly may be partially shielded by elements attached thereto (e.g., a manipulator holding it for operations such as oscillation or fluid exchange).

[0176] Manipulator: As referred to herein, a “manipulator” can include structures and / or mechanisms configured to controllably interact with at least one container (e.g., a container loaded onto a system) and / or other components or structures of a pharmaceutical preparation system. A manipulator can be configured to move at least one container. A manipulator can be configured to induce or facilitate a fluid transfer process; for example, transferring fluid from one container to another, involving, for example, aspirating and / or inserting (e.g., injecting) fluid. A manipulator can include a robotic arm, platform, robotic station, or a combination thereof configured to manipulate the container and / or fluid transfer assembly. A manipulator can include actuators, such as motors, for amplifying its operation. Some manipulators are also referred to herein as “stirring.” In some examples, a stirrer includes a manipulator that provides stirring-specific capabilities (e.g., the ability to oscillate, separate from the ability of the manipulator to selectively position the manipulated container at a target location). In some examples, a simpler feature of a stirrer is the ability to hold a container (e.g., a vial) while the stirrer itself moves in a stirring manner, for example, oscillating in place to move the contents of the container. In some examples, the movement of the stirrer introduces vortex motion into the fluid contents of the container. In some examples, the stirrer moves to disturb the surface boundary regions of the fluid contents of the container; for example, to produce splashing and / or momentary droplet separation. In some examples, the stirrer generates an electric current in the fluid contents of the container. In some examples, the stirrer introduces motion into the contents of the vial, causing the mixing of two material phases of the vial's contents, for example, suspending and / or dissolving a solid material in a liquid material, and / or suspending and / or dissolving two liquid materials in a common material phase.

[0177] The manipulator is not necessarily implemented as an "arm" itself, even in descriptions associated with such terms. Fluid transfer can occur when the container is engaged (e.g., clamped) by the manipulator. In the example, the manipulator (e.g., a "clamp" or "plunger arm") may include one or more actuators for engaging the syringe and / or pulling or pushing the plunger of the syringe. The syringe may then engage with a vial. The manipulator may optionally be configured to manipulate other types of fluid containers, such as vials, IV bags, tubing, and / or another suitable container.

[0178] Controller: In this document, the equivalent terms “controller” and “controller unit” generally refer to a circuit system configured to command certain aspects of the behavior of a controlled element, such as the operation of an actuator (which may in turn be the actuator of a manipulator), the operation of a sensor, and / or the operation of an imager. In some examples, the controller includes a computerized circuit system (also referred to herein as a “processing circuit system”) configured to perform operations according to a set of instructions stored in a memory readable by the controller, which may be executed, for example, by a central processing unit (CPU), one or more processors, processor units, and / or microprocessors. Alternatively or additionally, in some examples, the controller uses a digital signal processor (DSP), a field-programmable gate array (FPGA), a specialized application-specific integrated circuit (ASIC), or another device. Alternatively or additionally, in some examples, the controller or controller unit includes one or more analog circuits (e.g., amplifier-based feedback) and / or low-level logic gate-based control circuits. In some examples, the control unit may include one or more mechanism controllers. The controller unit may include any component for controlling elements in a robotic pharmaceutical preparation system and may include at least one of an analog control circuit system, a synchronization unit, and a processor.

[0179] Imager: In this document, the term "imager" refers to any device that operates to produce an image of a target. "Optical" imaging can generally be understood as imaging involving light (including visible light), but this is not mandatory. For example, infrared and / or ultraviolet wavelengths, in addition to or alternative to visible light wavelengths, can be used for imaging where appropriate. An example of an imager is an optical camera; for example, a camera equipped with one or more transparent lenses and a light sensor that can be read out to produce a digital image. Optionally, scanning imaging methods are used, for example, imaging the reflectivity of laser illumination scanned from the target back to the sensor. Optionally, interferometric imaging is used, for example, for tracking small deformations and / or movements. Imaging using radiant energy other than visible light is not excluded; for example, acoustic energy, electromagnetic wavelengths outside the visible spectrum, and / or particle (with mass) radiation imaging. Optionally, contact imaging is performed, for example, moving a contact probe along the imaging target to confirm the accuracy of its positioning and / or measure one or more contours of its shape.

[0180] In this document, characteristics and / or values ​​may be referred to as “expected” or “targeted.” It should be understood that these terms refer to the technical representation of the appropriate corresponding state and / or quantity. Optionally, such representation is digital, for example, as in the case of a target stored by a digital computing circuit system. Optionally, such representation is analog and / or mechanical; for example, represented by a pointer, knob, weight, or other element or arrangement of elements.

[0181] Before explaining at least one embodiment of this disclosure in detail, it should be understood that this disclosure, in its application, is not necessarily limited to the details of the construction and arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings. The features described in this disclosure, including those of the invention, can have other embodiments or can be practiced or carried out in various ways.

[0182] Now for reference Figure 1A It shows an example robotic pharmaceutical preparation system 101, which is based on some examples of the currently disclosed subject matter.

[0183] System controller 100 includes processing circuitry 105 and communicates with imager controller 140, which in turn operates imager 145 to perform overall imaging of the drug container 175 and / or the label 190 of the drug substance container 175, as these containers are selected for use and / or validation by drug preparation system 101. Drug preparation system 101 optionally includes container manipulator 170 operable to manipulate the drug container 175 to different positions relative to imager 145, e.g., different rotational and / or translational positions.

[0184] Optionally, the movement of the imager 145 and / or the movement of the pharmaceutical substance container 175 may be used to select the field of view 185 of the imager 145 for positioning it for imaging.

[0185] The processing circuitry system 105 includes a processor 110 and a memory 115, which can be configured to perform imager control and / or container manipulation. Additionally, in some examples, the memory 115 stores instructions for any one or more of the template matcher 121, OCR engine 125, drug registerer 139, and drug validator 135. These instructions are executed by the processor 110 during appropriate operation, for example, regarding... Figure 2 and Figures 4 to 7 As described. Memory 115 stores a label specification database 137, which is used and / or modified, for example, regarding... Figure 2 and Figures 4 to 7 As described.

[0186] Optionally, the drug preparation system 101 as per [the relevant information] Figure 8It is implemented as described in the drug preparation system 810, wherein the template matcher 121 is implemented as an ML-based label classifier 120.

[0187] In some examples, the drug preparation system 101 operates under the control of a user interface 161.

[0188] Now for reference Figure 1B This illustration schematically shows an example of a pharmaceutical container label 190 based on some examples of currently disclosed topics. The text written is placeholder text, which will replace any actual text that will be displayed on the actual label instance.

[0189] Certain areas of label 190 are specifically used to identify and / or verify the substance contained in container 175 to which image label 190 is attached. Examples of such areas containing textual content are brand name 192, generic name 196, manufacturer's name 199, and substance quality indicator 194. In some examples, brand name 192, generic name 196, and substance quality indicator 194 are the information required to establish identification of container 175. In some examples, manufacturer's name 199 is also used, for example, if specified for use by a human operator of pharmaceutical preparation system 101.

[0190] Label 190 typically contains other fixed information, represented as text and / or by a graphic appearance. The example shown includes a QR code 197A, storage and processing information 197B, dosage form 198, and manufacturer's mark 193. For the same label, other information may be variable; for example, batch number and expiration date 195. In some examples, the image label 190 contains potentially distinguishing graphic information, such as a background shading 191. In some examples, any such information may optionally be used to identify and / or verify the contents of container 175.

[0191] Now for reference Figure 2 It is a flowchart illustrating a method for registering a label for use by a pharmaceutical preparation system 101, based on some examples of the currently disclosed subject matter.

[0192] At box 210, in some examples, the pharmaceutical preparation system 101 receives a tagged vial instance to register as a structured template. Alternatively, in some examples, an existing image (e.g., an image from an online regulatory database of pharmaceutical substance labels) is available, in which case the flowchart may optionally skip to start at box 216, or optionally skip to box 212 and then to box 216.

[0193] Optionally, at box 212, the specifications of the vial contents (e.g., the contents of container 175) are received. This can be received, for example, from a database, and / or from user input at user interface 161. In some examples, the specifications of the vial contents at box 212 are skipped, and the vial contents are instead specified as part of the operation at box 216 (i.e., directly from the information shown on label 190 itself).

[0194] At box 214, in some examples, imaging is performed on the vial or other container 175, including imaging the label 190. The resulting image will be used as a registration image in the registration of the structured template. Imaging is performed, for example, using imager 145, under the control of imager controller 140 and / or system controller 100. Optionally, more than one image is obtained. For example, at least three images may be obtained, each rotated approximately 120° from a previous image, or otherwise rotated, to ensure that all or substantially all images of the label 190 are clearly imaged. In some examples, a line scan imager is used, where imaging is performed substantially continuously as the container 175 rotates. Optionally, more than one set of images is obtained. In some examples, the images contain an overall view of the container 175. This can be useful in determining the shape and / or size of the container 175, which may aid in identifying and / or verifying the contents of the container 175.

[0195] Optionally, preprocessing is performed on the image obtained in box 214. For example, the label image may be geometrically normalized to produce a label image that renders the label as if it were unfolded substantially flat, or rendered in another normalized configuration. In some examples, one or more label images are obtained from an existing database of labels; for example, a database maintained by a regulatory agency that regulates drug labeling.

[0196] In some examples, at box 216, the label text content corresponding to the bottle contents specifications in box 212 is identified. In some examples, this is performed using OCR. Optionally, user 161's input is used to provide and / or correct the text content.

[0197] Appropriately assign field identifiers to the text content; for example, identifiers for "brand name," "generic name," "material quality," and / or "manufacturer name." In some examples, the label text content itself is used to determine the specifications of the bottle contents. For example, heuristics are used to assign appropriate field identifiers to the text and / or graphic content determined by OCR based on the identified content.

[0198] Optionally, there is an opportunity for a human operator to correct the specifications of the vial contents; for example, starting with information automatically obtained from the text content of the label, and / or information in the specifications of the vial contents at box 212.

[0199] In some examples, at box 218, a structured template corresponding to the information identified at box 216 is stored in the label specification database 137. The stored information preferably includes both the identified text content (e.g., OCR-recognized and / or user-provided and / or edited characters) and graphic data, from which the text content is determined and / or the graphic data carries the text content on the label. Preferably, these two types of information are linked. For example, they may be stored in a combined data structure (different from other graphic data / text content data structures), referenced / stored via the same field identifier, and / or associated with the same and / or substantially overlapping area location specifications on the label. For example, in Figure 3 The example shown is of a field.

[0200] Optionally, the registration information of one or more known or partially known tags, carrying information related to the currently registered tag, is also updated. For example, the manufacturer name matching the manufacturer name of the currently entered tag may be optionally updated in the database for records where the manufacturer was not previously identified, and where there is a match for manufacturer information, such as a similar logo graphic.

[0201] Now for reference Figure 3 The diagram schematically illustrates elements of fields assigned to structured template 220 according to some examples of currently disclosed topics. Specifically, elements of brand name 192, generic name 196, product quality indicator 194, manufacturer name 199, and background shading 191 are shown. In the illustrated example, text content and graphic content (pixel images) are shown. In some examples, the position, size, and / or shape of the pixel images corresponding to the identified content are also saved as part of the structured template. Optionally, the structured template contains information about the shape of container 175 (e.g., relative to label 190). Optionally, the structured template contains one or more images from which a label registration record is constructed. Optionally, the total label size and shape are determined and stored as part of the label registration record.

[0202] Now for reference Figure 4 It is a flowchart of a method labeled as container verification, based on some examples of currently public topics.

[0203] At box 410, in some examples, a tagged instance of a vial (or other container 175) to be verified for its contents is received (e.g., received by the pharmaceutical preparation system 101).

[0204] At box 412, in some examples, container 175 is imaged, containing the image of the label; for example, as regarding Figure 2The imaging of the image label registration at box 214 is described. Optionally, label imaging is optimized for one or more specific structural templates selected as "expected" (e.g., based on current settings and / or recent historical expectations). For example, imaging may be performed to obtain a normalized image of the text in certain regions of the label, without necessarily involving imaging of other regions where the relevant text is not expected. If label matching fails later, imaging may optionally be re-performed (differently and / or more generally) during a second attempt at label recognition.

[0205] Depending on the implementation details, the operations in boxes 414 and 416 may optionally be sequential and / or mixed. In some examples, at least one characteristic of the label imaged in box 412 is determined, and one or more structured templates are selected for access (and more detailed comparisons) based on a match with the determined at least one characteristic. For example, an OCR engine may be used to determine the position of one or more lines of text in the label, or to determine the text content itself. In another example, graphic features of the label are determined, such as the shape of the label and / or the background color in certain areas. The determined characteristics are used as lookup values, for example, in the label specification database 137, to the set of available structured templates.

[0206] Alternatively or in some examples: at least one parameter is determined based on at least one selected structured template (e.g., from the label specification database 137), the parameter being used to analyze the image of the label imaged in box 412. For example, the desired position of the text area is determined, and if there is no text in the area and / or if the text in the area is not aligned with the desired position, the label does not match the structured template.

[0207] Therefore, at box 414, in some examples, one or more vial verification templates are accessed (e.g., stored in the label specification database 137 and / or as per the information provided). Figure 2 (The described registered structured template). Optionally, all available structured templates can be accessed. In some examples, for instance, the structured templates accessed are pre-selected for access based on the history of most recent matches and / or the current configuration of the drug preparation system 101. In some examples, structured templates are accessed based on partial representations from the label image of box 212, such as based on the location of text found in the label image of box 212, based on label size, and / or based on another characteristic of the label image.

[0208] At box 416, in some examples, a label matching with the validation template (that is, with the structured template accessed in box 414) is performed. Optionally, this includes any suitable combination of comparison operations. Optionally (e.g., when the match is poor by a certain comparison criterion), only a partial comparison is performed. In some examples, the initial phase of the comparison is chosen to be computationally relatively "cheap"; for example, comparing where text areas are located and / or what text characters are identified in those areas.

[0209] In some examples, at least where the structured template is ultimately considered to match the label image, the comparison includes both text content (e.g., text content determined by OCR) and graphical content at the pixel data level. For example, the pixel image below the corresponding area can be compared with the matching text content. In some examples, the match in the text content is confirmed based on the area location.

[0210] Optionally, fields of the structured template are matched against corresponding regions in the label image. Optionally, matches with structured template fields are accepted in different (e.g., shifted and / or widened) regions of the label image. In some examples, field identifiers in the structured template are used to constrain the interpretation of the text content in the label; for example, field identifiers may optionally limit the vocabulary used by the OCR engine when recognizing text in specific regions of the label image.

[0211] When comparing text content, either an exact match or a "fuzzy" match between the structured template text and the OCR-recognized image text may be accepted. In the case of a fuzzy match, one or more character substitutions, omissions, and / or insertions may be optionally allowed. Optionally, differences in the text content may be weighted or otherwise distinguished based on the nature of the differences. For example, similar-shaped glyphs (such as...) The swap of O can be considered a less significant error than the swap of glyphs with very different appearances. In another example, certain substitutions that merge two glyphs into one glyph can be weighted less, for example... For .

[0212] To compare the pixel-level appearance of graphic content (optionally including text without character identification information), any suitable comparison metric and / or method may be used. For example, image regions may be compared in a differencing manner (e.g., using subtraction and / or division). Optionally, a statistical metric, such as the mean squared error of pixel values, may be used. Optionally, normalization may be performed before the comparison; for example, normalization may be used to ensure that the label region is appropriately comparable to the structured template region in parameters such as position, scale, orientation, contrast, and / or focus quality.

[0213] In some examples, a machine learning product is used to compare images, which has been trained to report whether two pixel images originated from images of the same label type (e.g., whether they are each an imaging instance of a specific label, containing formatting and text content). In some examples, pixel images are tested to determine whether they are corresponding sub-regions of a label. In some examples, pixel images (e.g., those with structured templates) are tested to determine whether they are sub-regions of the overall label image. In some examples, the training data contains images of the same label type with variable information such as varying batch numbers and / or expiration dates. In some examples, the training data contains images of the same label type under different lighting, orientation, and / or focus conditions. In some examples, the training data contains images of the same label type with different defects; for example, wrinkled, bubbly, misoriented, and / or missing or otherwise damaged areas.

[0214] In some examples, tag matching is required, particularly for certain fields in a structured template. For instance, brand name, generic name, and product quality might be needed. Optionally, manufacturer name matching might be used. In some examples, additional fields of the structured template are provided and matched. For example, when there is a potential risk of cross-identifying two similarly structured tags (e.g., tags of types with nearly identical text content for the required fields), matching additional information to reduce the risk of misidentification is a potential advantage.

[0215] Optionally, a full comparison is performed on the structured templates based on the similarity between some structured templates and the current "best-matching" structured template, so as to increase the confidence of the best-match to be significantly higher than that of any potential competing matches. This similarity is determined based on any suitable metric, such as Hamming distance, or a relevant metric, such as a weighted Hamming distance adjusted by error significance.

[0216] Optionally, the operation of multiple image repeating frames 414 and 416 for the image label 190 of container 175 may be performed. This may help reduce the likelihood that the label 190 is misidentified due to imaging defects such as line scan errors, illumination artifacts, and / or transient electrical artifacts.

[0217] Optionally, at box 418, in some examples, the label identifier identified in boxes 414-416 will be compared with the desired contents of container 175. For example, the intended contents of container 175 may be referenced (directly or indirectly) in a digitally specified protocol for mixing pharmaceutical preparations via pharmaceutical preparation system 101. Alternatively or separately, the identification and / or verification information of container 175 may be provided separately from and / or together with the preparation protocol information.

[0218] In some examples, an alert is issued if there is a mismatch between the demand and the actual identified identifier on the label 190 of container 175. In some examples, a proper match between the demand and the actual identifier is required before allowing further operation of the pharmaceutical preparation system 101 using container 175 is permitted.

[0219] Now for reference Figure 5 This is a flowchart illustrating methods for label identification and / or verification based on some examples of currently disclosed topics. In some examples, the operations in boxes 510-516 correspond to... Figure 4 The operations in boxes 414-416. In some examples, the operations in boxes 510-514 correspond to... Figure 2 Operations in boxes 214-216 (template registration).

[0220] In some examples, at box 510, at least one tag image is accessed. This could be an image of the tag being registered (e.g., as...). Figure 2 (as part of the method of operation), or an image of the label 190 on container 175, which is part of the pharmaceutical preparation operation, for example as Figure 4 The method is part of the identification and / or verification.

[0221] As previously mentioned, the label image may optionally be a single image composed of multiple images. Optionally, the label image is normalized to facilitate proper comparison with other images. Optionally, several images are accessed so that boxes 512-514 can be repeated (for example) to verify consistency.

[0222] In box 512, in some examples, an OCR engine is used to determine the text content of label 190. The text content may optionally be associated with a specific image location identified from it.

[0223] At box 514, in some examples, the graphic appearance of label 190 is characterized. The characterization may optionally include selecting pixel image data (e.g., intensity data) from regions of text recognized by the OCR engine. Optionally, the pixel image data itself is used as a representation of the graphic appearance. Optionally, the pixel image data is generalized, abstracted, transformed into a vector representation of its features, and / or otherwise transformed.

[0224] At box 516, in some examples, if the method is registered as part of the container verification template (e.g., as...), Figure 2 If executed as part of the method, text content and graphic appearance data are stored. Optionally, the text content and graphic appearance data are linked through a common tag area associated with them. Optionally, more than one field is stored for the text content; for example, fields for each of several different pharmaceutical substance properties, such as substance brand name, substance generic name, substance quality and / or substance manufacturer.

[0225] Alternatively, at box 516, if the method is performed as part of identifying and / or verifying a specific container 175 and label 190, the determined text content and graphic appearance identified in boxes 512-514 are used to perform a comparison with one or more container verification templates. Again, more than one field may be involved in the comparison, and different fields of the text content may be clearly associated with different meanings, as described for verification template registration.

[0226] The comparisons may optionally reference (and use) the text content and graphical appearance in any suitable order (including mixed orders). For example, computationally inexpensive comparisons may be performed first to exclude most container validation templates from further consideration, where only a full set of comparisons is performed on some or all of the remaining templates. Optionally, the comparisons of all features may be performed substantially simultaneously (e.g., by parallel processing).

[0227] The matching of text content can be either exact or allow a certain amount of errors (e.g., character omissions, insertions, or substitutions, the type of which is consistent with OCR errors that may occur from time to time).

[0228] Image appearance matching can be performed in any or more of several ways. For example, pixel map data can be compared directly after appropriate normalization to remove and / or minimize imaging variability. Methods for comparing pixel map data include, for example, differencing methods, statistical methods, and other measures that identify feature locations (e.g., edges and / or corners), their orientation, and / or the arrangement of features characterizing the image. In some examples, pixel map data is abstracted into histogram data, or otherwise “fingerprinted.”

[0229] Optionally, a set of machine learning-based weights is used to classify the graphical appearance of a label as "similar" or "different" from a given container validation template, and / or to classify the appearance of a label selected from any number of container validation templates. In some examples, the graphical appearance is characterized by the vectorization of the image region, such as the vectorization to a feature vector, where the component magnitudes indicate the intensity of certain feature values ​​("features") representing those features in the image region.

[0230] Optionally, a full comparison is performed on at least one incompletely matching template to confirm that the probability of any template other than the best-matching template being the correct template is very low. Optionally, the best-matching template is only considered the correct template—for example, as long as no other options can be considered as remaining. Or, for example, if the best-matching template does not have sufficiently similar text content and / or graphic appearance to the container label 190 being evaluated, the result can be reported as a failure to identify container 175 and / or its contents.

[0231] Now for reference Figure 6 This is a schematic flowchart illustrating a method for adjusting the registered tag verification template to enhance uniqueness based on some examples of currently publicly available topics.

[0232] As part of risk assessment and / or mitigation, it may be useful to perform a check characterizing the “error distance” between two label validation templates (i.e., two different container validation templates; these two terms should be considered interchangeable). Error distance can be understood as the number of errors that must occur before one label template can be confused with another. Optionally, mitigation may be performed to increase, for example, the error distance between the label templates initially closed.

[0233] In box 610, in some examples, the first label template is accessed. In box 612, in some examples, the second label template is accessed. In box 614, in some examples, the error distance between the two label templates is determined.

[0234] Considering the text content, the error distance can be easily calculated as the number of character insertions, deletions, and / or substitutions that must be performed to effectively transform the text content of one tag template into the text content of another. Optionally, some errors may have higher or lower weights, for example, depending on their probability (e.g., replacing 'o' with 'o'). Alternatively, replacing 1 with 'l' can be considered a more likely substitution, contributing relatively less to the error distance compared to replacing characters with different glyphs. For different data fields, the error distance can optionally be calculated separately.

[0235] Considering graphic appearance data, error distances can be calculated based on metrics that are appropriate for the representation of the graphic appearance data itself. This could involve directly converting data from one template to data from another; for example, calculating the sum of absolute values ​​of pixel value changes, measuring histogram value differences, or determining vector component differences.

[0236] At decision box 616, in some examples, it is determined whether there is a significant risk of confusion between the two label templates. Optionally, the risk is assessed using instances of the labels against a conceptual comparison. For example, if the level of “error noise” in the selected (e.g., hypothetical or observed) comparison accuracy could lead to errors reaching a value of 1 / 2 the error distance between the two label templates, it can be anticipated that in some cases, ambiguous conclusions will be produced.

[0237] If the determination in box 616 indicates possible instance confusion, then optionally, at box 620, a mitigation action is performed. A general category of such mitigation actions is adding one or more additional features to one or both of the label templates. This may optionally be performed automatically (e.g., using text content of secondary importance and / or selected graphic appearance metrics, such as color or overall label size). Alternatively or alternatively, the operator may be prompted to provide feature indications, such as distinguishing the regions of interest of the two label templates.

[0238] Another general category of mitigation is to “tagged” one or two templates to gain extra attention so that action will be taken later when they match sufficient ambiguity.

[0239] If the error distance is considered not to involve significant risk, the flowchart may optionally end at box 618 without implementing mitigation.

[0240] Now for reference Figure 7 This is a schematic flowchart illustrating a method for detecting the risk of obfuscation during tag template matching, based on some examples of currently published topics.

[0241] At box 710, in some examples, multiple validation template matches are accessed. These are optionally determined relative to the instance of container 175 and its label 190, for example, as per [reference to...]. Figure 4-5 And / or as described in the Overview section. At box 712, in some examples, error distances are determined. For example, error distances are determined between matching templates themselves, and / or between the matching template and the container tag instance being evaluated. Error distances may optionally be calculated, for example, as per [reference to...]. Figure 6 And / or as described in the overview section. At box 714, in some examples, the calculated error distance is compared to the requirement. For example, it may be required that the statistical risk of a certain error identification does not exceed a certain threshold. Optionally, some error identification scenarios are considered more critical than others, and the attention threshold is increased accordingly.

[0242] In decision box 716, in some examples, if the determined error distance is considered not to involve significant risk, the flowchart may optionally end at box 718 without implementing mitigation.

[0243] Alternatively, at box 720, mitigation can be implemented. Optionally, this includes stopping automated operation and / or alerting the operator to potential confusion. Optionally, the operator is authorized (e.g., through operation via a user interface) to accept, reject, or correct obviously ambiguous identifications made by the drug preparation system. Optionally, the operator is required to confirm the ambiguous situation and clear and / or reset the error conditions before the system continues its operation.

[0244] Now for reference Figure 8 It illustrates an example robotic injection preparation system 810 employing machine learning-based drug property verification, based on some examples of currently disclosed topics.

[0245] In some examples, system controller 100 includes processing circuitry system 105, including, for example, processor 110 and memory 115.

[0246] Processor 110 can be a suitable hardware-based electronic device with data processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), a dedicated application-specific integrated circuit (ASIC), one or more cores of a multi-core processor, etc. Processor 110 may also include, for example, multiple processors, multiple ASICs, virtual processors, combinations thereof, etc.

[0247] Memory 115 may optionally include any suitable type of volatile and / or non-volatile memory; for example, a single physical memory component or multiple physical memory components. Memory 115 may be configured, for example, to store data and / or instructions used in computation. The memory itself may contain a non-transitory computer-readable storage medium.

[0248] The processing circuit system 105 can be configured to execute any one or more of a plurality of functional modules according to computer-readable instructions stored on a non-transitory computer-readable storage medium. Hereinafter, such functional modules are referred to as being included in the processing circuit system. These modules may include, for example, a label classifier 120 based on label-based machine learning, an optical character recognition (OCR) engine 125, and a drug verifier 135.

[0249] The pharmaceutical container 175 can be held or manipulated by a suitable mechanism, such as a gripper or a robotic arm. An optical character recognition (OCR) engine 125 can perform optical character recognition on a received image of the label 190 of the pharmaceutical container 175. For example, the OCR engine 125 can generate a text string based on the characters it reads from the pharmaceutical label 190.

[0250] The machine learning-based label classifier 120 can utilize a trained machine learning model to determine pharmaceutical parameters associated with the label of a pharmaceutical product. The label classifier 120 can be understood as a specific example of the template matcher 121, for example, as per [reference to...]. Figure 1A As described. In some examples, a machine learning-based label classifier 120 receives labeled images and generates from them data indicating drug parameters associated with the labels used during the training phase.

[0251] System controller 100 may be operatively connected to container manipulator 170, for example, via a bus or network connection. In some examples, system controller 100 may command container manipulation operations; for example, command container manipulator 170 to position pharmaceutical container 175 for imaging by imager 145 (e.g., positioning it at a camera field of view distance 185). Other operations performed on container 175 may optionally include picking up, placing, and positioning for substance transfer; for example, transferring fluid from and / or into a syringe, container 175 may be momentarily connected to syringe as part of pharmaceutical preparation.

[0252] System controller 100 may be operatively connected to a syringe manipulation subsystem (not shown); for example, via a bus or network connection. In some examples, system controller 100 may command the syringe manipulation subsystem to operate; for example, to command it to aspirate a specific amount (e.g., volume) of fluid (e.g., a drug to be injected into a patient) into a syringe barrel (not shown).

[0253] In some examples, imager 145 includes a digital camera configured to optically scan, for example, a label 190 of a pharmaceutical substance container 175. In some examples, imager 145 is a line scan camera that moves laterally or rotatably relative to label 190 to capture different portions of label 190 (label 190, camera 145, or both can move), and then provides these portions to imager controller 140, which performs image processing to fuse a series of photographs into a single image (e.g., a panoramic image).

[0254] The imager controller 140 can be operatively connected to the imager 145 and the system controller 100.

[0255] Imager 145 can be positioned such that it captures a digital image of the medicine container from a specific distance, referred herein as imager field of view 185.

[0256] Imager controller 140 can be operatively connected to imager 145. Imager controller can implement imager control methods and can supply digital images to system controller 100.

[0257] System controller 100 can be operatively connected to imager controller 140 and can implement system control methods, such as those described below. Figure 10 The system control methods described.

[0258] Now for reference Figure 9A It shows example image 900 of a label for a medicine container, based on some examples of currently disclosed topics. Also referenced... Figure 9BThe image shows the rotation of a container 175 in front of a line scanning camera 945, according to some examples of the currently disclosed subject matter. In the example shown, the line scanning camera 945 has been operated while the labeled container 175 rotates several times in front of it. Some images are repeated 901A-901D with a portion of the recognized text, followed by confidence score labels.

[0259] Now for reference Figure 10 It shows a flowchart of an example sequence of machine learning-based drug property verification in a robotic injection preparation system, based on some examples of currently disclosed topics.

[0260] At box 310, in some examples, the processing circuitry 105 (e.g., drug verifier 135) receives an image of the label on the drug substance container from, for example, an imager controller 140.

[0261] In box 320, in some examples, the processing circuitry 105 (e.g., OCR engine 125) performs OCR on the received image. OCR can produce various text strings representing the properties of a drug, such as: • Drug Name • Manufacturer •quantity • Production date •batch number • Validity period The processing circuit system 105 can utilize these pharmaceutical properties for various purposes.

[0262] By way of a non-limiting example: the processing circuit system 105 can perform the preparation of the drug injection in a specific order when a smaller container (e.g., a container holding a relatively small amount of drug substance) is provided, and perform the preparation of the drug injection in a different manner when a larger container is used.

[0263] In addition, the processing circuit system 105 can perform critical security checks based on the obtained text string, such as detecting incorrect or expired medicines.

[0264] However, as mentioned above, text generated by OCR agencies may contain errors due to reasons such as damaged labels or lighting issues.

[0265] At box 330, in some examples, the processing circuit system 105 (e.g., a machine learning-based label classifier 120) classifies the received image, for example, using a machine learning model that classifies labeled images as pharmaceutical property data (e.g., using a machine learning model trained as described above). The pharmaceutical properties resulting from this classification may include, for example: • Drug Name • Manufacturer •quantity • Production date •batch number • Validity period Alternatively, the classification determines the identification of the container, and based on the identification of the container, the nature of the medicine can be determined, for example, by searching.

[0266] At box 338, in some examples, processing circuitry system 105 (e.g., drug validator 135) determines whether the drug properties and / or identifiers generated by the machine learning classification of the labels are consistent with the text string determined by the OCR. For example, in some examples, the drug properties generated by the classification may be provided as a text string, and processing circuitry system 105 (e.g., drug validator 135) may determine whether the text strings are the same. In some other examples, processing circuitry system 105 (e.g., drug validator 135) may confirm that the manufacturer indicated by the classification is the same as the manufacturer indicated by the OCR text string, etc.

[0267] According to decision box 340, in some examples: if the processing circuitry 105 (e.g., drug verifier 135) detects a lack of consistency between the determined drug parameters and the text string exported by the OCR, it can issue an alert at box 345. Otherwise, at box 350, it can continue processing (e.g., using the contents of the container).

[0268] General situation As used herein, the term “about” in relation to quantity or value means “within ±10% of”.

[0269] The terms “comprises”, “comprising”, “includes”, “including”, “having”, and their conjugates mean “including but not limited to”.

[0270] The term “composed of” means “included in and limited to”.

[0271] The term "consistently made of" means that a composition, method, or structure may contain additional ingredients, steps, and / or portions, provided that the additional ingredients, steps, and / or portions do not substantially alter the fundamental and novel characteristics of the claimed composition, method, or structure.

[0272] As used herein, unless the context clearly indicates otherwise, the singular forms “a / an” and “the” include plural indicators. For example, the terms “compound” or “at least one compound” can include multiple compounds, including mixtures thereof.

[0273] The terms “example” and “exemplary” are used herein to mean “serving as an example, illustration, or description.” Any embodiment described as “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or as excluding the incorporation of features from other embodiments.

[0274] As used herein, the term “optionally” means “provided in some embodiments and not provided in others.” Any specific embodiment of this disclosure may include multiple “optional” features, except where such features conflict to some extent.

[0275] As used herein, the term "method" refers to the manner, means, techniques, and procedures used to accomplish a given task, including but not limited to those known to practitioners in the fields of chemistry, pharmacology, biology, biochemistry, and medicine, or those that are readily developed based on known manner, means, techniques, and procedures.

[0276] As used herein, the term “treatment” includes the elimination, substantial inhibition, slowing or reversal of the progression of a condition, substantial improvement of the clinical or aesthetic symptoms of a condition, or substantial prevention of superficial signs of the clinical or aesthetic symptoms of a condition.

[0277] Throughout this application, embodiments can be presented with reference to the range format. It should be understood that the use of the range format is for convenience and brevity only and should not be construed as a fixed limitation on the scope of the description herein. Therefore, the description of a range should be considered as specifically disclosing all possible subranges and individual numerical values ​​within said ranges. For example, a description of a range such as "from 1 to 6" should be considered as specifically disclosing subranges such as "from 1 to 3", "from 1 to 4", "from 1 to 5", "from 2 to 4", "from 2 to 6", "from 3 to 6", etc.; and individual numbers within said ranges, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the width of the range.

[0278] Whenever a range of numbers is indicated herein (e.g., “10-15”, “10 to 15”, or any pair of numbers connected by such range indications), it means any number (fraction or integer) contained within the indicated range boundaries, including the range boundaries, unless the context explicitly states otherwise. The phrases “range” / “variable range” / “ranges” between the first and second indicating numbers, and “to,” “at most,” “until,” or “through” (or another such range indication term) for the first indicating number and “range” / “variable range” / “ranges” for the second indicating number, are used interchangeably herein and mean including the first and second indicating numbers and all fractions and integers in between.

[0279] Although this disclosure has been described in conjunction with specific embodiments, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be readily apparent. Therefore, it is intended to cover all such alternatives, modifications, and variations falling within the spirit and broad scope of the appended claims.

[0280] It should be understood that, for clarity, certain features described in the context of individual embodiments of this disclosure may also be provided in combination in a single embodiment. Conversely, for simplicity, various features described in the context of a single embodiment may also be provided individually or in any suitable sub-combination or, where appropriate, in any other described embodiments of this disclosure. Certain features described in the context of various embodiments should not be considered essential features of those embodiments unless the embodiments are invalid without those elements.

[0281] The applicant intends that all publications, patents, and patent applications mentioned in this specification be incorporated herein by reference in their entirety as if each individual publication, patent, or patent application were specifically and separately attributed to be incorporated herein by reference. Furthermore, any reference or identification of any reference in this application should not be construed as an admission that such reference is available as prior art to this disclosure. In the sense of using section headings, they should not be construed as necessarily limiting. Additionally, any priority documents of this application are hereby incorporated herein by reference in their entirety.

Claims

1. A method of identifying a drug substance, comprising: accessing at least one image of a label of a container of a drug substance; accessing one or more label templates, each label template's data containing: text content of a region of a reference label, and at least one indication of graphical appearance in the region of the reference label; comparing the image to the one or more label templates, including comparison to both the text content and the at least one indication of graphical appearance; and determining an identity of the drug substance from a result of the comparison.

2. The method of claim 1, wherein the text content includes at least one of a generic name of the drug substance and a brand name of the drug substance.

3. The method of any one of claims 1-2, wherein the text content specifies an amount of the drug substance contained in the container.

4. The method of claim 3, wherein the amount is specified by both a number and a unit indication.

5. The method of any one of claims 1-4, wherein the text content indicates a manufacturer name of the drug substance.

6. The method of any one of claims 1-5, wherein each label template encodes the text content as a sequence of numbers of code points.

7. The method of any one of claims 1-5, wherein the at least one indication of graphical appearance comprises image intensity values in the region of the reference label.

8. The method of claim 7, wherein the at least one indication of graphical appearance comprises a pixel map representation of the region of the reference label.

9. The method of any one of claims 1-7, wherein the at least one indication of graphical appearance comprises a feature vector for a set of feature value features.

10. The method of any one of claims 1-9, wherein the comparing includes determining label information used in the comparing, and the determining label information includes determining at least some text content displayed in the image of the label, and determining an indication of graphical appearance in the image of the label.

11. The method of claim 10, wherein both the determined text content and the determined indication of graphical appearance are derived from a same region of the image of the label.

12. The method of any one of claims 10-11, wherein the determined text content of the image of the label is determined using an OCR engine.

13. The method of any one of claims 10-12, wherein the determined indication of graphical appearance comprises a pixel map selected from the image of the label.

14. The method of any one of claims 10-13, wherein the determined indication of graphical appearance comprises a feature vector computed from the image of the label.

15. The method of any one of claims 1-14, wherein the text content of each label template is assigned in a respective portion to each of a plurality of fields of the data of the label template, each field further including a field identification.

16. The method of claim 15, wherein each field is associated with a zone of the reference label, the zone containing a respective portion of textual content of the field.

17. The method of claim 16, wherein each field is associated with a respective one of the at least one indication of graphical appearance, and the respective indication of graphical appearance indicates a graphical appearance in the associated zone of the reference label.

18. The method of any one of claims 15 to 17, wherein the fields identify at least a field identifying a brand name for the drug substance, a field for a generic name of the drug substance, and a field for a quantity of the drug substance.

19. The method of any one of claims 15 to 18, wherein the fields identify a field identifying a name of a manufacturer for the drug substance.

20. The method of any one of claims 1 to 19, wherein the container is a vial.

21. The method of any one of claims 1 to 20, comprising normalizing the at least one image to the one or more label templates according to a three-dimensional structure of the drug substance container.

22. The method of any one of claims 1 to 21, comprising normalizing the at least one image to the one or more label templates according to any one or more of a quality of focus, a level of illumination, an image contrast, and an image perspective foreshortening.

23. The method of any one of claims 1 to 22, wherein the comparing comprises identifying text in a zone of the at least one image that corresponds in position to a position of the textual content in the zone of the reference label.

24. The method of any one of claims 1 to 23, wherein: the comparing comprises: determining that there is a partial mismatch of textual content between one of the one or more label templates and textual content identified in the at least one image of the label, and determining that the at least one image of the label is in sufficient correspondence with one of the at least one indication of appearance in the zone of a reference label; and despite the partial mismatch of textual content, assigning the identification associated with the one of the one or more label templates in the determining the identification of the drug substance.

25. The method of any one of claims 1 to 24, comprising controlling an imager to image the at least one image of the label of the drug substance container.

26. The method of claim 25, comprising controlling a manipulator to manipulate the drug substance container to present it to the imager while the image is being controlled or the drug substance container is being imaged.

27. The method of any one of claims 25 to 26, comprising receiving the drug substance container.

28. The method of any one of claims 1 to 27, wherein said accessing one or more templates comprises accessing at least a first template according to a tentative identity of the drug substance, and accessing at least a second template according to a similarity of the second template to the first template.

29. The method of claim 28, wherein said similarity is determined according to a measure of Hamming distance between the first and second templates.

30. A system for drug substance manufacturing, the system comprising a processor and a memory, wherein the memory contains instructions instructing the processor to: access at least one image of a label of a drug substance container; access one or more label templates, each label template containing data of: text content of a zone of a reference label, and at least one indication of graphical appearance in the zone of the reference label; compare the image to the one or more label templates, including comparison to both the text content and the at least one indication of graphical appearance; and determine an identity of the drug substance according to a result of the comparison.

31. The system of claim 30, comprising an imager and an imager controller; wherein the memory contains instructions instructing the processor to control the image controller to image the at least one image of the label.

32. The system of any one of claims 30 to 31, comprising a manipulator configured to manipulate the drug substance container; wherein the memory contains instructions instructing the processor to control the manipulator to position a label to be imaged in the at least one image of the label.

33. A method of registering a label template for use in identifying a drug substance, the method comprising: accessing at least one image of a reference label; identifying text content in one or more zones of the at least one image; generating one or more respective indications of graphical appearance in the zones of the at least one image; and storing the text content as a label template associated by zone with the respective indications of graphical appearance.

34. The method of claim 33, wherein said identifying text content comprises performing optical character recognition (OCR) on the at least one image of the reference label.

35. The method of claim 34, wherein the one or more zones are zones in which text content is identified by said performing OCR.

36. The method of any one of claims 33 to 35, comprising assigning the text content in respective portions of the text content to each of a plurality of fields stored in the label template, each field further comprising a field identity.

37. The method of claim 36, wherein said assigning is performed according to a dictionary of expected text content patterns in the plurality of fields. ​ 38. The method of any of claims 36-37, wherein the assignment is performed according to a heuristic automatically applied to the textual content according to at least one of the textual content itself, a graphical appearance of a region of the at least one image in which the textual content appears, and a surrounding environment of the region in the at least one image.

39. The method of claim 38, wherein the heuristic comprises a regular expression.

40. The method of any of claims 38-39, wherein the heuristic specifies a relative positioning of two or more of the portions of the textual content.

41. The method of any of claims 38-40, wherein the heuristic specifies a relative font size of one or more of the portions of the textual content.

42. The method of any of claims 33-41, wherein the at least one image of the reference label is an image of a container labeled with the reference label.

43. The method of any of claims 33-41, wherein the at least one image of the reference label is an example image from a regulatory database.

44. A system configured to register a label template for identifying a drug substance, the system comprising a processor and a memory storing instructions, wherein the instructions instruct the processor to: access at least one image of a reference label; identify textual content in one or more regions of the at least one image; generate one or more respective indications of a graphical appearance in the regions of the at least one image; and store the textual content as a label template associated by region with the respective indications of graphical appearance.

45. The system of claim 44, comprising an imager and an image controller; wherein the memory contains instructions instructing the processor to control the image controller to image the at least one image of the reference label.

46. The system of any of claims 44-45, comprising a manipulator configured to manipulate a drug substance container; wherein the memory contains instructions instructing the processor to control the manipulator to position a reference label in the at least one image of the label when the reference label is affixed to a container to be imaged.

47. A method of adjusting a registered label template to enhance uniqueness, the method comprising: accessing a first label template specifying a first set of features that, when matched, identify a label image as an instance of a first label type; accessing a second label template specifying a second set of features that, when matched, identify a label image as an instance of a second label type; determining that an estimated error distance between the first and second label templates is within a threshold range of a potential label template conflict; and adding at least one specified feature to the first set of features, the at least one specified feature selected to increase the estimated error distance.

48. The method of claim 47, wherein a Hamming distance between text content of the first and second label templates is used to estimate the error distance.

49. The method of any one of claims 47-48, wherein the at least one added specified feature comprises a distinguishing color of the first and second label templates.

50. The method of any one of claims 47-49, wherein the at least one added specified feature comprises a feature of a text region of the first and second label templates that distinguishes the first and second label templates.

51. The method of claim 50, wherein the feature of the text region comprises one or more of text content, a size of the text region, an orientation of the text region, and a size of a glyph in the text region.

52. A system configured to adjust a registered label template to enhance uniqueness, the system comprising a processor and a memory storing instructions, wherein the instructions instruct the processor to: access a first label template specifying a first set of features that, when matched, identify a label image as an instance of a first label type; access a second label template specifying a second set of features that, when matched, identify a label image as an instance of a second label type; determine that an estimated error distance between the first and second label templates is within a threshold range of a potential label template conflict; and add at least one specified feature to the first set of features, the at least one specified feature selected to increase the estimated error distance.

53. A method of confirming a unique valid match of a label image to a label template, the method comprising: determining that the label image corresponds to a first label template specifying a first set of features sufficient to identify an imaged label as an instance of a first label type; selecting a second label template specifying a second set of features according to an estimated similarity of the second label template and the first label template; determining that a correspondence of the label image to the second label template is within a threshold range of a potential label template conflict; and issuing an alert according to the determination.

54. A non-transitory computer-readable medium storing a database of templates for identifying labels of pharmaceutical products, each template comprising a plurality of fields identified according to a type of information represented by the field, and each field associated with a text string representing text in a region of the label of the pharmaceutical product and an indication of a graphical appearance in the region.

Images