System and method for automatically orienting product containers

Abstract

A computer-implemented method for automatically orienting product containers includes obtaining a reference product label image associated with a batch of product containers on a production line and based on a perceived label feature of the reference product label image, using a trained deep learning model to compute a perceived center of the reference product label image. A production line camera obtains image slices of respective product labels of individual product containers of the batch of product containers. Based on the obtained image slices of the respective product labels and a determined label center based on the computed perceived center of the reference product label image, a rotation angle for rotating the individual product containers is computed. The computed rotation angle is communicated to a controller to effect rotation of the individual product containers.

Description

Technical Field

[0001] Embodiments of this disclosure relate to systems and methods for orienting product containers in a desired manner (e.g., for packaging purposes). In particular, one or more embodiments of this disclosure relate to a technique for automating the orientation of product containers on a production line based on product label features and graphics. Background Technology

[0002] In many industrial production processes, especially in the food and beverage industry, products are transported in containers such as bottles and cans. Product labels are typically affixed to the outer surface of the product container and may contain graphic / features such as brand logos, Universal Product Codes (UPCs), nutritional information, etc. Labeled containers may arrive at the packaging machine at random orientations or rotational angles relative to the corresponding product labels. Labeled containers are expected to be oriented visually consistent and / or appealing when conveyed to the packaging machine so that the end consumer can optimally observe or utilize the final packaged or ready-made product arrangement. Summary of the Invention

[0003] In short, various aspects of this disclosure provide a technique for automatically orienting product containers on a production line based on product label features, reducing operator intervention or configuration.

[0004] A first aspect of this disclosure provides a computer-implemented method for automatically orienting product containers on a production line, wherein each product container has a product label disposed around its outer surface. The method includes acquiring a reference product label image associated with a batch of product containers on the production line. The method further includes calculating a perceptual center of the reference product label image using a trained deep learning model based on perceptually relevant label features of the reference product label image. The method also includes acquiring image slices of the corresponding product labels of individual product containers within the batch of product containers via a production line camera. The method further includes calculating a rotation angle of the individual product container based on the acquired image slices of the corresponding product labels and a label center determined based on the calculated perceptual center of the reference product label image. The method also includes transmitting the calculated rotation angle to a controller to achieve rotation of the individual product container from an initial orientation to a final orientation based on the calculated rotation angle.

[0005] Other aspects of this disclosure implement the features of the above-described methods in the form of computing systems and computer program products.

[0006] Additional technical features and benefits can be achieved through the technology disclosed herein. Embodiments and aspects of this disclosure are described in detail herein and are considered part of the claimed subject matter. For a better understanding, refer to the detailed description and accompanying drawings. Attached Figure Description

[0007] The foregoing and other aspects of this disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. For ease of identification of any element or action discussed, one or more of the most significant digits in the reference numerals refer to the figure number in which the element or action is first introduced.

[0008] Figure 1 This is a schematic diagram illustrating a system for automatically orienting product containers on a production line according to an exemplary embodiment.

[0009] Figure 2 This is an illustrative embodiment with reference to a product label image.

[0010] Figure 3 It is an illustrative visualization of the perceived relevant label features located on the reference product label image.

[0011] Figure 4 This is an exemplary representation of the perceptual center of a reference product label image.

[0012] Figure 5 This is a schematic flowchart illustrating an exemplary technique for training and deploying a rotation angle classifier that calculates rotation angles at runtime.

[0013] Figure 6 This is an exemplary representation of using a template matching algorithm at runtime to infer the patch location on a reference product label image that matches a captured image slice of the product label.

[0014] Figure 7 A computing system capable of supporting the automatic orientation of product containers in a production line, according to the disclosed embodiment, is shown. Detailed Implementation

[0015] Various techniques related to the system and method will now be described with reference to the accompanying drawings, wherein the same reference numerals always denote the same elements. The drawings discussed below and the various embodiments used to describe the principles of this disclosure in this patent document are for illustrative purposes only and should not be construed as limiting the scope of this disclosure in any way. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged apparatus. It should be understood that functions described as being performed by certain system elements can be performed by multiple elements. Similarly, for example, an element can be configured to perform functions described as being performed by multiple elements. Many of the innovative teachings of this application will be described with reference to exemplary, non-limiting embodiments.

[0016] There are various container orientation systems available to orient cans, bottles, or other containers of various shapes in a common direction. For example, a servo-based orientation system can use sensors (such as a vision system) to perform the task at high speed to detect arbitrary actual container positions and then rotate the container to the desired orientation accordingly.

[0017] To configure an orientation system for a new product or label, production engineers typically specify a baseline product label / graphic for the orientation system, as well as the desired orientation of the product container relative to that label. This manual engineering step can limit how quickly a production line can switch from one product to another, or introduce a new product label for the same product.

[0018] This disclosure provides an artificial intelligence (AI)-based solution that can identify, with high accuracy but without any prior knowledge of the product label, which part of the product label should form the front of the label to be displayed to the consumer. The AI-based solution is based on the understanding that product labels are typically designed by marketing experts and graphic designers, thus following common design patterns and best practices to attract consumer attention. For example, certain label features (such as large text or product / brand logos) are often intended to be eye-catching parts of the product label and are therefore positioned at the front of the label. Furthermore, certain label features (such as barcodes / UPCs, nutrition facts, etc.) may be intended to be positioned away from the front of the product label, for example, for scanning / pricing reasons. For instance, in multi-packaged product containers, it might be desirable for the barcode of the individual container to face inwards to prevent customers from scanning the individual product barcode instead of the package's barcode for pricing.

[0019] In at least some embodiments, the proposed AI-based solution enables a fully automated system that automatically determines the perception center of a reference product label and, when presented on a production line with randomly oriented product containers, calculates rotation angles to rotate the containers to the correct orientation, for example, for packaging purposes, without operator configuration or intervention. In some embodiments, depending on the confidence level of the calculated perception center of the product label, the AI-based solution can be assisted by user confirmation or by using a user-specified front cover of the product label.

[0020] The disclosed implementation reduces the overall engineering effort required for automated system integration and label configuration of new product containers arriving on the production line, and thus provides faster transitions when switching from one product or product label to another on the production line. Furthermore, the disclosed implementation performs label sensing with high accuracy while providing runtime performance for inference and alignment comparable to existing solutions.

[0021] Now turn to the attached diagram. Figure 1 A system 100 for automatically orienting product containers 104 on a production line 102, according to an exemplary embodiment, is shown. In particular, the disclosed system 100 can be used to automatically orient labeled product containers 104 based on label features and graphics on product labels.

[0022] Product container 104 can have various shapes and typically (but not necessarily) has an axisymmetric outer surface. For example, the most common types of product containers used in the food and beverage industry include cans, bottles, etc. Product labels may be attached around the outer surface of each product container 104 or otherwise positioned. Product labels may extend completely around the outer surface of the product container 104 (i.e., 360 degrees). As an illustration, in Figure 1 In the product container 104, the black and white borders depict various label features of the corresponding product label (e.g., brand logo, UPC code, etc.).

[0023] like Figure 1 As shown, incoming product containers 104 may have a random orientation relative to their product labels. The randomly oriented product containers 104 can be individually rotated by a specified rotation angle via orientation unit 106 to ensure that the output product containers 104 have a consistent orientation relative to the product labels, as illustrated. In an exemplary configuration, orientation unit 106 may include one or more clamps 108 (e.g., suction-based clamps) for holding individual product containers 104 and servo-controlled motors 110 coupled to the respective clamps 108. Motors 110 may be individually controlled based on corresponding control signals 126 from controller 124 to rotate the clamped product containers 104 by a specified rotation angle. In embodiments, the output (re)oriented product containers 104 may be conveyed to a packaging machine, where a group of product containers 104 may be packaged into a single package (e.g., a six-can beverage). The disclosed embodiments are capable of determining the rotation angle of individual product containers based on AI, allowing the end consumer to optimally observe or utilize the final package or off-the-shelf product arrangement.

[0024] According to the illustrated embodiment, system 100 may include a production line camera 112 arranged to scan each product container 104 conveyed onto production line 102 to acquire its image. Due to the 3D shape of the product container 104, production line camera 112 may capture only image slices of the corresponding product label. As used herein, the term "image slice" refers to an image of a portion or fragment of a product label acquired via a partial (e.g., 180-degree) scan of the production line camera 112. Production line camera 112 may suitably include a high-speed camera configured to capture image slices of the corresponding product label of an individual product container 104 in a batch of product containers on production line 102. For example, in one embodiment shown herein, production line camera 112 and its illumination system for generating flashes may be triggered based on a precisely timed trigger signal 128 from a controller 124, which may also control the position and speed (e.g., the speed of a conveyor system) of the product containers 104. Image slices acquired for individual product containers 104 may be transmitted as data signals 114 to a computing system 116 to calculate the corresponding rotation angles of those individual product containers 104 during runtime.

[0025] The computing system 116 may include, for example, an industrial PC, an edge computing device, or any other computing system including one or more processors and memory storing algorithm modules executable by the one or more processors. The algorithm modules include at least a visual tag perception engine 118 and a rotation angle calculation engine 120. The various engines described herein, including the visual tag perception engine 118 and the rotation angle calculation engine 120, including their components, may be implemented by the computing system 116 in various ways, such as as hardware and through programming. Programming of engines 118 and 120 may take the form of processor-executable instructions stored on a non-transitory machine-readable storage medium, and the hardware of the engines may include processors that execute those instructions. The processing power of the systems, devices, and engines described herein, including the visual tag perception engine 118 and the rotation angle calculation engine 120, may be distributed among multiple system components, such as multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements. References below... Figure 7 An embodiment of the computing system applicable to this application is described.

[0026] Still referencing Figure 1The visual label perception engine 118 can be configured to acquire a reference product label image associated with the batch of product containers 104 on production line 102, and calculate the perceptual center of the reference product label image using a trained deep learning model based on the perceptually relevant label features of the reference product label image. The visual label perception engine 118 can be implemented as part of an offline system for initial calibration setup. The rotation angle calculation engine 120 can be configured to calculate the rotation angle of individual product containers 104 at runtime based on image slices of the corresponding product labels acquired via production line camera 112 and a label center determined based on the perceptual center of the calculated reference product label image. The perceptual center of the calculated reference product label image can be determined as the label center used to calculate the rotation angle. In some embodiments, depending on the confidence level associated with the calculated perceptual center, the label center used to calculate the rotation angle can be the calculated perceptual center or can be derived from a user-specified front portion of the label.

[0027] The computing system 116 can transmit the calculated rotation angle of each product container 104 as a data signal 122 to the controller 124. In an embodiment, the controller 124 may include a programmable logic controller (PLC) or any other type of industrial controller, configured to convert the data signal 122 into a control signal 126 and transmit the control signal 126 to the appropriate servo control motor 110 associated with the corresponding product container 104 to achieve rotation of the product container 104 from an initial orientation to a final orientation based on the calculated rotation angle.

[0028] In one implementation, such as Figure 1 As shown, reference product label images used for initial calibration can be acquired via a 360-degree scan of reference product containers associated with the batch of product containers. Typically, but not necessarily, all product containers in the same batch may have identical product labels. The reference product container may be one of product containers 104 belonging to a new batch of product containers arriving on production line 102, or it may represent a different product container in the batch. The reference product container (e.g., product container 104) may be mounted on a turntable 130, allowing camera 132 to perform a 360-degree scan of the reference product label. A controller (e.g., controller 124) may transmit control signals 134 to control the rotation and speed of the turntable 130 and issue appropriately timed trigger signals 136 to trigger camera 132 and its illumination system for generating synchronized flashes. The 360-degree scanned reference product label image may be transmitted by camera 132 as a data signal 138 to computing system 116 to use visual label perception engine 118 to determine the perception center of the reference product label image. In an alternative embodiment, the reference product label image may be obtained directly from the product manufacturer.

[0029] Figure 2 An illustrative embodiment of a reference product label image 200 is shown. For illustration, the horizontal axis of the reference product label image 200 represents angular distance. As shown, the reference product label image 200 may include a two-dimensional planar or non-distorted image representing a 360-degree layout of the reference product label.

[0030] According to the disclosed implementation, the visual label perception engine 118 ( Figure 1 At least one deep learning model can be used to detect perceptually relevant label features on the reference product label image 200. This deep learning model may include one or more pre-trained feature detection models / neural networks. In one exemplary implementation, the deep learning model can be trained on a training dataset comprising images of multiple different product labels via a supervised learning process using classification labels with known perceptually relevant label features for those images (of the dataset). This allows the trained deep learning model to detect and locate perceptually relevant label features on the reference product label image. In a suitable implementation, the training dataset can be obtained from an existing OpenLogo dataset comprising product labels for a large number of known products.

[0031] For model training, product labels in the training dataset can have bounding box annotations of perceptually relevant label features for different categories. These bounding box annotations can define the classification labels used by the deep learning model. Perceptually relevant label features for different categories can include, but are not limited to, large text, brand logos, barcodes (e.g., UPC codes), geometric shapes, coarse or high color contrast features, nutrition facts tables, etc. The visual label perception engine 118 can use a trained deep learning model to detect and locate perceptually relevant label features on reference product label images, for example, through bounding boxes. Figure 3 As shown, the exemplary output obtained from the trained deep learning model can be visualized, in which the detected perceptually relevant label features, namely large text, geometry, nutrition facts table and UPC code, are located on the reference product label image 200 by bounding boxes 302, 304, 306 and 308, respectively.

[0032] According to the disclosed implementation, perceived relevant label features or feature categories can be grouped into "interesting" and "uninteresting" categories. "Interesting" label features may include those that define the portion of the product label intended to form the front of the product label. Examples of "interesting" label features include, but are not limited to, large text, large geometric shapes, high color contrast features, brand logos, etc. "Uninteresting" label features may include those that define the portion of the product label intended to be positioned away from the front. Examples of "uninteresting" label features include, but are not limited to, barcodes (e.g., UPC codes), tables (e.g., nutrition facts tables), etc.

[0033] By identifying the locations of "interesting" and "uninteresting" label features, the visual label perception engine 118 can algorithmically calculate the perceptual center of a reference product label image. In one implementation, the visual label perception engine 118 can calculate the perceptual center of the reference product label image based on a weighted combination of the locations of perceptually relevant label features detected on the reference product label image. For example, each detected perceptually relevant label feature can be assigned a weight that may depend on variables such as the size (e.g., angular range) of the detected label feature, whether the detected label feature is an "interesting" or "uninteresting" label feature, and other variables. In one implementation, to ensure that the calculated perceptual center of the reference product label is located further away from "uninteresting" label features, negative weights can be assigned to "uninteresting" label features, or lower weights can generally be assigned relative to "interesting" label features.

[0034] To illustrate an exemplary method, in Figure 3 In the illustrated embodiment, the location of each detected perception-related label feature can be defined by the center of the corresponding bounding box 302, 304, 306, 308 along the horizontal or angular axis. These locations are... Figure 3 The values ​​are denoted as X1, X2, X3, and X4, respectively. The corresponding weight assigned to each of the detected perception-related label features can be determined based on the width (i.e., angular range) of the corresponding bounding boxes 302, 304, 306, and 308 along a horizontal or angular axis. Furthermore, the weights associated with "features of interest" (e.g., corresponding to bounding boxes 302 and 304) can include positive values, while the weights associated with "features of non-interest" (e.g., corresponding to bounding boxes 306 and 308) can include negative values. The location of the perception center can be determined as the weighted average of the locations X1, X2, X3, and X4 of each of the detected perception-related label features. In some embodiments, the calculated perception center can be located by a bounding box having a defined width around the location of the calculated perception center.

[0035] Figure 4 An exemplary representation of the calculated perceptual center of the reference product label is described herein. In this document, the perceptual center of the reference product label image 200 is defined by a structure having a perceptual center around position X. C The location is defined by a bounding box 402 with a defined width. For example, as described above, the position X can be determined based on a weighted combination of positions X1, X2, X3, and X4. C As shown in the figure, assigning lower or negative weights to the "uninteresting" label features (corresponding to bounding boxes 306 and 308) can cause the calculated perceptual center to be located further away from those "uninteresting" label features.

[0036] In an alternative implementation, the visual label perception engine 118 may use a deep learning model (e.g., one or more neural networks) trained on a dataset comprising a large number of product label images to compute the perceptual center of a reference product label image, wherein the images in the dataset may each be annotated with known label centers. For example, the model may utilize initial layers of the neural network to extract label features (which may include perceptually relevant label features as described herein) and subsequent layers to learn the spatial relationship between the extracted label features and the annotated label centers.

[0037] Therefore, the visual label perception engine 118 can identify the location of the perception center of a new product label without any prior knowledge of the product label or even the product or brand. Thus, the disclosed implementation provides a fully universal method that does not require user input to set up the system, wherein the AI-based visual label perception engine 118 can autonomously determine what the “important” parts of an image are based on a single image of a reference product label for the customer to see.

[0038] Continue to refer to Figure 1 In some implementations, the computing system 116 can transmit a data signal 142 representing the perceptual center of the calculated reference product label image to a human-machine interface (HMI) device 140. The HMI device 140 can be configured to visualize and display the front of a label to an operator based on the calculated perceptual center of the reference product label, thereby providing transparency to the underlying orientation process. The HMI device 140 can also be used to assist in AI-based calculations of the perceptual center, particularly in cases of low confidence levels associated with the calculated perceptual center. The confidence level associated with the calculated perceptual center can be determined, for example, based on a confidence score associated with detected perceptually relevant label features inferred from a deep learning model.

[0039] In one implementation, if the determined confidence level associated with the calculated perception center is below a predetermined threshold, user input can be sought via HMI device 140 for confirmation or use of a user-specified label front to cover the center of the displayed label front. User confirmation or coverage can be transmitted by HMI device 140 as data signal 144 to computing system 116. Based on the user input transmitted via data signal 144, the label center used by rotation angle calculation engine 120 to calculate the rotation angle can be either the perception center calculated by visual label perception engine 118 or the calculated center of the user-specified label front. In at least some implementations, if the determined confidence level associated with the calculated perception center is above a predetermined threshold, the calculated perception center can be used to automatically determine the rotation angle of an individual product container without any operator intervention.

[0040] During runtime, the rotation angle calculation engine 120 can execute an inline inference loop that may involve acquiring image slices of the product labels of each product container 104 on the production line 102, transmitted by the production line camera 112 (via data signal 114), and calculating the desired offset rotation angle of the product container 104 based on the acquired image slices and the label center of a determined reference product label image. The calculated rotation angle can be used to rotate the product container 104 via the controller 124 to achieve a desired / consistent orientation relative to the corresponding product label, for example, as described above. An exemplary implementation of the rotation angle calculation engine 120 is described below.

[0041] In a first embodiment, the rotation angle calculation engine 120 can use a trained rotation angle classifier to infer the offset rotation angle from the acquired image slices. The rotation angle classifier can include a neural network, and suitably includes a shallow convolutional neural network capable of performing fast inference loops for high-speed production lines. The rotation angle classifier can be trained using a dataset generated by performing image patch enhancements corresponding to different regions of a reference product label image, the images having different angular offsets from the label center. The angular offsets can be used to define classification labels for the image patches. A properly trained rotation angle classifier can provide high accuracy in calculating the desired rotation angle for each product container.

[0042] Figure 5 An exemplary flow 500 suitable for training a rotation angle classifier such as those described above is shown. The various modules described herein, including patch 504, enhancer / data transformer 506, and offset calculator / label generator 512, including their components, can be implemented by a computing system such as computing system 116 in various ways, e.g., as hardware and through programming.

[0043] Patch 504 can be used to generate image patches from a reference product label image 502. Each image patch can define an area of ​​the reference product label image 502, for example, corresponding to a 180-degree field of view. In an exemplary embodiment, image patches can be generated by cropping 360 / (x-1) patches from the reference product label image 502 in steps of x° from 0° to (360°-x°).

[0044] The enhancer / data transformer 506 can add noise and warping to the image patch to simulate how a product label looks when wrapped around an axisymmetric (e.g., cylindrical) product container. Additionally, the enhancer / data transformer 506 can match eigen and extrinsic camera properties (e.g., lighting, resolution, etc.) and simulate camera distortion on the image patch to simulate actual image slices captured by the production line camera 112. The enhancer / data transformer 506 can add and combine these modifications to generate a sufficiently large and realistic training dataset 508.

[0045] The offset calculator / label generator 512 can determine an angular offset for each image patch relative to the label center 510 of the determined reference product label image 502. The determined label center 510 can be the calculated perceptual center of the reference product image 502. Alternatively, if the confidence level associated with the calculated perceptual center is low (e.g., below a predetermined threshold), as described above, the label center 510 can be determined based on user input. For example, the angular offset can be determined by locating the center of each image patch and calculating the angular distance from the center of the image patch to the label center of the determined reference product label image 502. The angular offset thus determined can be used to define the classification label 514.

[0046] The training dataset 508 and the corresponding labels 514 can be used to train the rotation angle classifier 516 via a supervised learning process. Once trained and validated, the rotation angle classifier 516 can be deployed to the runtime system to automatically infer the offset rotation angle from image slices acquired via the production line camera 112.

[0047] In a second embodiment, the rotation angle calculation engine 120 may utilize a template matching algorithm. Template matching is a technique in computer vision used to search for and locate a template image within a larger parent image. In the context of this application, template matching algorithms can provide high runtime speed for calculating the required rotation angle for each product container. Suitable implementations may involve using template matching functionality provided by the OpenCV library.

[0048] Figure 6This is an exemplary representation of using a template matching algorithm at runtime to infer the location of a patch on a reference product label image that matches a captured image slice of the product label. In the illustrated embodiment, image slice 602 represents a distorted image with a 180-degree field of view, acquired via production line camera 112 (via data signal 114), showing a portion of a product label wrapped around a cylindrical product container. The template matching algorithm can be performed using image slice 602 as a template image and a reference product label image 200 as a parent image. When executed, the template matching algorithm can slide template image 602 over parent image 200 (e.g., as in 2D convolution) and compare template image 602 with patches on parent image 200 (e.g., a 180-degree patch in this embodiment) to locate a matching patch 604 on parent image 200. In an implementation using OpenCV, depending on the template matching criterion used, the minMaxLoc function in OpenCV can be used to determine the matching patch 604 as either the global minimum (when using TM_SQDIFF) or the global maximum (when using TM_CCORR or TM_CCOEFF).

[0049] The rotation angle can be determined by calculating the angular offset between the matching patch 604 and the determined label center. For example, it can be determined by positioning the center X of the matching patch 604. M The angular offset is determined by calculating the angular distance from the center of the matching patch 604 to the label center of the determined reference product label image 200. In the illustrated embodiment, the determined label center is the calculated perception center X of the reference product label image 200. C Alternatively, if the calculated sensory center X... C The associated confidence level is low (e.g., below a predetermined threshold), as described above, for example, the label center can be determined based on user input.

[0050] Figure 7 An embodiment of a computing system 700 capable of supporting automated orientation of product containers in a production line, according to the disclosed embodiments, is shown. In the embodiment, the computing system 700 may include one or more of an industrial PC, an edge computing device, etc. The computing system 700 includes at least one processor 710, which may take the form of a single processor or multiple processors. The processor 710 may include a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a microprocessor, or any hardware device adapted to execute instructions stored on a memory including a machine-readable medium 720. The machine-readable medium 720 may take the form of any non-transitory electronic, magnetic, optical, or other physical storage device storing executable instructions such as visual tag perception instructions 722 and rotation angle calculation instructions 724, etc. Figure 7 As shown. Thus, the machine-readable medium 720 can be, for example, random access memory (RAM), such as dynamic RAM (DRAM), flash memory, spin-torque memory, electrically erasable programmable read-only memory (EEPROM), storage drive, optical disc, etc.

[0051] The computing system 700 can execute instructions stored on the machine-readable medium 720 via the processor 710. Executing the instructions (e.g., visual tag perception instruction 722 and rotation angle calculation instruction 724) can cause the computing system 700 to perform any of the technical features described herein, including any features of the visual tag perception engine 118 and rotation angle calculation engine 120 as described above.

[0052] The systems, methods, devices, and logic described above, including the visual tag perception engine 118 and the rotation angle calculation engine 120, can be implemented in many different ways through many different combinations of hardware, logic, circuitry, and executable instructions stored on a machine-readable medium. For example, these engines may include circuitry in a controller, microprocessor, or application-specific integrated circuit (ASIC), or may be implemented using discrete logic or components, or combinations of other types of analog or digital circuitry combined on a single integrated circuit or distributed among multiple integrated circuits. Products such as computer program products may include a storage medium and machine-readable instructions stored on that medium, which, when executed in an endpoint, computer system, or other device, cause that device to perform operations according to any of the above descriptions, including operations based on any feature of the visual tag perception engine 118 and the rotation angle calculation engine 120. The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device.

[0053] The systems, devices, and processing power of the engines described herein, including the visual label perception engine 118 and the rotation angle calculation engine 120, can be distributed among multiple system components, such as across multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements. Parameters, databases, and other data structures can be stored and managed separately, can be merged into a single memory or database, can be logically and physically organized in many different ways, and can be implemented in many ways, including data structures such as linked lists, hash tables, or implicit storage mechanisms. Programs can be multiple parts of a single program (e.g., subroutines), separate programs, distributed across several memories and processors, or implemented in many different ways, such as in the form of libraries (e.g., shared libraries).

[0054] The systems and processes illustrated in the accompanying drawings are not unique. Other systems, processes, and menus can be derived from the principles of this disclosure to achieve the same purpose. Although this disclosure has been described with reference to specific embodiments, it should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the present design can be made by those skilled in the art without departing from the scope of this disclosure.

Claims

1. A computer-implemented method for automatically orienting product containers on a production line, wherein each product container has a product label disposed around the outer surface of the product container, the method comprising: Obtain a reference product label image associated with a batch of product containers on the production line. Based on the perceptually relevant label features of the reference product label image, a trained deep learning model is used to calculate the perceptual center of the reference product label image. Image slices of the corresponding product labels of individual product containers in the batch of product containers are acquired via cameras on the production line. The rotation angle of the individual product container is calculated based on the acquired image slices of the corresponding product label and the label center determined based on the calculated perceptual center of the reference product label image. The calculated rotation angle is transmitted to the controller to rotate individual product containers from initial orientation to final orientation based on the calculated rotation angle.

2. The method according to claim 1, characterized in that, The reference product label image is obtained by scanning a reference product container associated with the batch of product containers in a 360-degree manner.

3. The method according to any one of claims 1 and 2, characterized in that, The deep learning model is trained on a dataset containing images of multiple different product labels via a supervised learning process, using known perceptually relevant label features of images with multiple different product labels. The trained deep learning model is configured to detect and locate perceptually relevant label features on the reference product label image.

4. The method according to claim 2, characterized in that, The perception-related label features include "interesting" label features that define the portion of the product label that forms the front of the product label, and "uninteresting" label features that define the portion of the product label that is located away from the front.

5. The method according to claim 4, characterized in that, The “interesting” label features include one or more features selected from the group consisting of: large text, large geometry, and high color contrast features, and wherein the “uninteresting” label features include one or more features selected from the group consisting of: barcodes and tables.

6. The method according to claim 2, characterized in that, The perceptual center of the reference product label image is calculated based on a weighted combination of the corresponding positions of the perceptually relevant label features detected on the reference product label image.

7. The method according to claim 6, characterized in that, Depending on whether the detected perception-related label feature is an "interesting" label feature or an "uninteresting" label feature, a weight is assigned to each detected perception-related label feature, wherein the assigned weight is configured to locate the calculated perception center away from the detected perception-related label feature that is an "uninteresting" label feature.

8. The method according to any one of claims 1 to 2, characterized in that, The calculated perception center is determined as the label center used to calculate the rotation angle.

9. The method according to any one of claims 1 to 2, characterized in that, Also includes: Based on the calculated sensing center of the reference product label, the front of the label is displayed via a human-machine interface (HMI) device. If the determined confidence level associated with the computer sensing center is lower than a predetermined threshold, the HMI device will request user input to confirm or use a user-specified label front to cover the center of the displayed label front. Specifically, based on the user input, the label center used to calculate the rotation angle is determined as either the perceptual center calculated based on the deep learning model or the calculated center of the label front specified by the user.

10. The method according to any one of claims 1 to 2, characterized in that, The rotation angle is calculated by feeding the acquired image slices into a trained rotation angle classifier to produce an inference output indicating the rotation angle. The rotation angle classifier is trained using a dataset, which is generated by augmenting image patches corresponding to different regions of the reference product label image, the image patches having different angular offsets from the label center, the angular offsets being used as classification labels for the image patches.

11. The method according to any one of claims 1 to 2, characterized in that, The rotation angle is calculated in the following way: The acquired image slices are used as templates to perform a template matching algorithm to locate matching patches in the reference product label image, and Calculate the angular offset between the matching patch and the center of the tag.

12. A non-transitory computer-readable storage medium, characterized in that, Includes instructions that, when processed by a computer, configure the computer to perform the method according to any one of claims 1 to 11.

13. A computing system, characterized in that, include: At least one processor; as well as The memory stores instructions that, when executed by the at least one processor, configure the computing system to perform the method according to any one of claims 1 to 11.

14. A system for automatically orienting product containers on a production line, characterized in that, Each product container has a product label disposed around the outer surface of the product container, and the system includes: A production line camera is configured to capture image slices of the corresponding product labels of individual product containers within a batch of product containers on the production line. Computing systems, including: One or more processors, and A memory storing algorithm modules executable by the one or more processors, the algorithm modules comprising: A visual label perception engine is configured to use a trained deep learning model to calculate the perceptual center of a reference product label image based on perceptually relevant label features of the reference product label image, and A rotation angle calculation engine is configured to calculate the rotation angle of an individual product container based on image slices of the corresponding product labels acquired via the production line camera and a label center determined based on the perceptual center of a calculated reference product label image. The controller rotates the individual product container from its initial orientation to its final orientation based on the calculated rotation angle.

15. The system according to claim 14, characterized in that, It also includes human-machine interface (HMI) devices, which are configured to: The label front is displayed based on the calculated perceptual center of the reference product label, and If the determined confidence level associated with the computer's perception center is below a predetermined threshold, then user input is requested for confirmation, or the center of the displayed label front is covered using a user-specified label front. Specifically, based on the user input, the label center used by the rotation angle calculation engine to calculate the rotation angle is determined as either the perceptual center calculated based on the deep learning model or the calculated center of the front of the label specified by the user.

Images