Digital labeling of recycled items

Abstract

A plastic item, such as a beverage bottle, may convey two different digital watermarks encoded using two different signaling protocols. The first printed label watermark conveys a retail payload, including a Global Trade Item Number (GTIN) used by point-of-sale scanners in retail stores to identify and price the item at checkout. The second plastic texture watermark may convey a recycling payload, including data identifying the plastic composition. The use of two different signaling protocols ensures that the point-of-sale scanner does not spend its limited time and computing resources decoding the recycling watermark, which may lack the data required for retail checkout. In some embodiments, recycling equipment advantageously uses both types of watermarks to identify the plastic composition of the item (e.g., using an associated database to associate the GTIN with the plastic type), thereby increasing the proportion of items that are correctly identified for sorting and recycling. In other embodiments, the plastic item (or the label thereon) carries only a single watermark. Many other features and arrangements are also described in detail.

Description

[0001] Related application data

[0002] In the United States, this application is a continuation-in-part of patent application 16 / 435,292 filed on June 7, 2019 (published as US20190306385), and also claims priority to the following provisional patent applications: 62 / 818,051 filed on March 13, 2019; 62 / 830,318 filed on April 5, 2019; 62 / 836,318 filed on April 19, 2019 26; 62 / 845,230 filed May 8, 2019; 62 / 854,754 filed May 30, 2019; 62 / 923,274 filed October 18, 2019; 62 / 956,493 filed January 2, 2020; 62 / 967,557 filed January 29, 2020; and 62 / 968,106 filed January 30, 2020. Background Art

[0003] There is an urgent need to increase the percentage of plastic items that are reused or recycled.

[0004] Document US20040156529 of the present applicant teaches that the plastic surface of a 3D object can be textured by thermoplastic molding to form a machine-readable digital watermark conveying a multi-bit payload.

[0005] Document US20040086151 of the applicant teaches how to use injection molding (eg using vacuum molding or pressure molding) to produce a 3D object with a digital watermark.

[0006] Document US20020099943 of the applicant teaches that an object can convey two watermarks, one formed in the topology of the object surface and one formed by printing.

[0007] Document US20150016712 of the present applicant teaches that a 3D object can be identified using watermark or image fingerprint data, and the identification data can be linked to the recycling information of the object. For example, the identification data can be linked to a recycling code indicating whether the object is made of polyethylene terephthalate, high-density polyethylene, polyvinyl chloride, etc.

[0008] The applicant's document US20150302543 similarly teaches that the payload of a watermark formed on a plastic object can convey or can be linked to the recycling code of the object. Additionally, the '543 document teaches that a waste sorting device equipped with a camera can classify an incoming material stream based on decoded watermark data. The document US20180345323 of FiliGrade B.V. also discloses sensing recycling information from a watermarked plastic bottle and separating waste streams based on the decoded information.

[0009] The applicant's current work improves on the above techniques and provides many additional features and advantages. Summary of the Invention

[0010] In one aspect, the present technology relates to defining a data pattern of spaced-apart binary elements at positions included within a regular 2D position grid. The pattern defines a first fixed reference signal and a first variable data signal. The first reference signal aids a decoder in the geometric registration and extraction of the first variable data signal, the decoder presenting a camera-captured image depicting the physical counterpart of the data pattern. The arrangement further includes shaping a 3D surface topology pattern of a mold according to a smoothed counterpart of the data pattern. The topology pattern includes peaks or depressions having smooth cross-sections to facilitate demolding of the molded part from the mold.

[0011] In a particular embodiment, the 2D position grid defines M candidate positions in which the binary elements can be located. The binary elements are located in 25%, 20%, 10% or less of these M candidate positions.

[0012] In another particular embodiment, the 2D position grid includes an N×N position grid, each position grid having a grayscale or floating-point value of a first reference signal corresponding thereto. The first variable data signal includes an M×M position array, where M < N, and each position in the M×M position array has a binary value of a variable data signal corresponding thereto. The M×M first variable data signal values are interpolated to produce an N×N interpolation array. In doing so, the first variable data signal is converted from a binary form to a grayscale or floating-point form. The corresponding values of the first reference signal and the interpolated first variable data signal are then summed at each of the N×N positions at a weighted ratio, thereby producing a sum array of N×N values. A threshold operation is applied to the sum array of these N×N values to identify the values, and then the positions in the N×N grid corresponding to these extreme values are marked with binary elements.

[0013] In another specific embodiment, the 2D position grid defines M positions, each position having a corresponding value of the first reference signal. Each value represents the relative darkness of the reference signal at that position. The first variable data signal includes binary symbols, each symbol being associated with a corresponding position within the regular 2D position grid. In order to generate the data pattern, the values ​​of the first data signal are sorted to produce a ranking of the N darkest positions. The P darkest positions are then identified in the ranking of the N darkest positions (leaving Q other positions). Each of the P positions is marked with a binary element. Of the remaining Q positions, each position is marked or unmarked (with a binary element) depending on whether the corresponding binary symbol of the variable data signal has the first value or the second value.

[0014] The mould can be produced using the arrangement just described and can be used to form plastic containers. In such containers, the formed plastic conveys a first variable data signal that can be read by a suitable decoder, such as in a waste recycling facility.

[0015] In some arrangements, the plastic container is also provided with a label. The label may include a print pattern defining a second fixed reference signal and a second variable data signal. The second reference signal facilitates geometric registration and extraction of the second variable data signal by a decoder that presents a camera-captured image depicting the print pattern. Typically, the second fixed reference signal is different from the first fixed reference signal, and / or the second variable data signal is different from the first variable data signal.

[0016] A recycling system that processes two different such plastic containers may process one based on its plastic texture pattern and the other based on its printed label pattern. That is, a computer processor of such a recycling system may use a first registration signal (in textured plastic pattern) to geometrically register to a first variable data signal on a first container, extract the first variable data signal, and sort the first container for recycling based on the extracted first variable data signal. It may further use a second registration signal (in printed label pattern) to geometrically register to a second variable data signal on a second container, extract the second variable data signal, and sort the second container for recycling based on the extracted second variable data signal.

[0017] In another aspect, a plastic container is shaped to convey information. More specifically, the container is shaped to convey a texture pattern of spaced elements at locations within a regular 2D position grid. The pattern defines a first fixed reference signal and a first variable data signal. The first reference signal facilitates geometric registration and extraction of the first variable data signal by a decoder that presents a camera-captured image depicting the container. In such a container, the 2D position grid may define M candidate locations where the elements may be located. The elements are located at 25%, 20%, 10%, or less of the M candidate locations.

[0018] Typically, the other 75%, 80%, 90% or more of these M candidate locations follow the nominal contour of the plastic container, for example, remaining smooth and simply following the cylindrical shape of the bottle. That is, within the boundaries of the texture pattern, a large portion of the container surface area remains unchanged.

[0019] On the other hand, a plastic container conveys a plastic texture pattern and a printed label pattern. The plastic texture pattern includes spaced-apart elements at positions within a regular 2D position grid. The plastic pattern defines a first fixed reference signal and a first variable data signal. The first reference signal facilitates geometric alignment and extraction of the first variable data signal by a decoder, which presents a camera-captured image depicting the container. Similarly, the printed label pattern includes spaced-apart elements at positions within a regular 2D position grid. Likewise, the printed label pattern defines a second fixed reference signal and a second variable data signal. The second reference signal facilitates geometric alignment and extraction of the second variable data signal by a decoder, which presents a camera-captured image depicting the container. In this arrangement, the second fixed reference signal is different from the first fixed reference signal, and / or the second variable data signal is different from the first variable data signal.

[0020] In one particular arrangement, the second fixed reference signal is different from the first fixed reference signal. In another particular arrangement, the second variable data signal is different from the first variable data signal.

[0021] In yet another aspect, a plastic container conveys a texture pattern comprising spaced-apart elements at positions within a regular 2D position grid. The pattern defines a first fixed reference signal and a first variable data signal. The first reference signal facilitates geometric registration and extraction of the first variable data signal by a decoder, which presents a camera-captured image depicting the container. In this arrangement, the 2D position grid defines M candidate positions at which the elements may be located. The elements are actually located at 20%, 20%, 10% or less of the M candidate positions. Likewise, at most of the M candidate positions, the container follows the nominal profile as described above.

[0022] Another aspect relates to a method of marking a container to convey a multi-symbol payload. The method includes generating a data pattern that encodes the payload. The pattern includes spaced-apart elements at positions within a regular 2D position grid. The pattern defines a fixed reference signal and a variable data signal. The reference signal facilitates geometric registration and extraction of the variable data signal by a decoder that presents a camera-captured image depicting a physical counterpart of the data pattern.

[0023] The method further includes forming a physical counterpart to the pattern on the container by printing or texturing. In this arrangement, the 2D position grid defines M positions, each position being associated with a corresponding value of the reference signal. Each value represents the relative darkness of the reference signal at that position. The variable data signal includes binary symbols, each symbol being associated with a corresponding position within the regular 2D position grid.

[0024] More specifically, the data pattern asterisk is generated using the ranked values ​​of the reference signal to produce a ranking of the N darkest positions. The P darkest positions in the ranking of the N darkest positions are then identified (leaving Q other positions). Each of these P positions is marked with a binary element. The remaining Q positions are marked or unmarked with a binary element depending on whether the corresponding binary symbol of the variable data signal has a first value or a second value.

[0025] Another aspect of the present technology relates to a recycling system including an optical reader and plastic bottles. The plastic bottles are labeled. The label is coated with a first digital pattern encoding a first identifier. The texture of the plastic carries a second digital pattern encoding a different second identifier. In this case, decoding of either identifier by the optical reader enables the recycling system to sort the bottles by plastic type.

[0026] Another aspect of the technology relates to a recycling system, also including an optical reader and a plastic bottle. The bottle is at least partially wrapped in a sleeve. The sleeve is previously printed with an ink mark, that is, in a flat form. The printed sleeve is then wrapped around the bottle and tightly fitted to the bottle by heat shrinking. The ink mark on the sleeve includes a machine-readable code that is geometrically deformed by the heat shrinking, but can still be read by the optical reader to control the sorting of the plastic bottle for recycling.

[0027] Another aspect of the technology relates to a point of sale system that can be used for checkout in a retail store. The system includes an optical reader and a plastic bottle. The bottle has a printed label coated with a first digital pattern encoding a first identifier, and the plastic is textured with a second digital pattern encoding a second identifier. The optical reader is configured to decode the first identifier but not the second identifier.

[0028] Yet another aspect of the technology relates to a bottle comprising a plastic container beneath an ink-printed label. The ink-printed label comprises a machine-readable code that can classify the underlying plastic container by plastic type.

[0029] The foregoing and other arrangements are described in detail in the following detailed description, which refers to the accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A-1Q There are illustrated several different forms into which the plastic surface may be shaped to convey digital watermark data.

[0031] Figure 2A Illustrated is one form into which a plastic surface may be shaped to convey ternary digital watermark data.

[0032] Figure 2B Another form is shown in which a plastic surface can be shaped to convey multi-state or continuous tone digital watermark data.

[0033] Figure 3A , 3B , 3C and 3D show the Voronoi, Delauney, Traveling Salesman and Brick patterns.

[0034] Figure 4 The peaks of the reference signal defining the printed digital watermark in the spatial frequency (Fourier amplitude) domain are shown.

[0035] Figure 4A Shows Figure 4 A magnified view showing radial lines passing through the different peaks.

[0036] Figures 5A-5C shows various reference signals that can be used with plastic texture digital watermarks, adjusted to avoid Figure 4 The printed watermark reference signal is interfered with.

[0037] Figures 6A-6C They are shown respectively Figures 5A-5C The spatial domain (pixel domain) counterpart of the reference signal.

[0038] Figure 7 A piece of the "snake" watermark pattern is shown, with which a mold for a plastic bottle can be formed.

[0039] Figure 8 , 9 , 10 and 11 show different systems for capturing images of items on a conveyor belt in a recycling facility.

[0040] Fig.12An arrangement is shown which provides variable illumination to achieve contrast enhancement.

[0041] Fig.13 Certain processes performed in an illustrative recycling facility are detailed.

[0042] Fig.14 Provides information about Fig.13 Further details of illustrative recovery equipment.

[0043] Fig.15 Blocks are shown arranged in an overlapping manner across an image frame as employed in certain embodiments of the present technology.

[0044] Fig.16 It shows that in certain embodiments of the present technique, each neighboring block is analyzed in a dense cluster when a watermark signal is found in the block.

[0045] Fig.17 exist Fig.15 The context of the image frame is shown Fig.16 A dense cluster of , which is located where one of the blocks is found to include a reference signal.

[0046] Fig.18 It shows a dense cluster of each neighboring block being analyzed when a glare area is detected in an image frame in certain embodiments of the present technology.

[0047] Fig.19 A subblock histogram used in a method for triggering a block analysis is shown.

[0048] Fig. 20 The help illustrates different methods for trigger analysis.

[0049] Fig.21A Blocks are shown arranged along the inlet side of an image frame, and a dense cluster of neighboring blocks located in the inlet block where a watermark reference signal is detected, as may occur in certain embodiments of the present technology.

[0050] Fig. 21B Shown in Fig.21A A frame captured shortly after the frame where the cluster of analysis blocks has corresponded to the conveyor motion making the frame progress downward.

[0051] Fig. 21C Shown in Fig. 21B A frame captured shortly after frame , shows the detection of the watermark reference signal, which results in a second cluster of blocks.

[0052] Fig.21D Shown in Fig. 21C A frame captured shortly after the frame shows two clusters of analysis blocks progressing further down the frame according to the conveyor motion.

[0053] Fig. 22 The intra-frame combination of watermark element data is illustrated.

[0054] Fig.23 Another example of intra-frame combining of watermark element data is illustrated.

[0055] Fig.24 Illustration of a camera / lighting arrangement that uses light transmitted through a plastic object to the camera.

[0056] Fig.25 A partially assembled lighting module is depicted.

[0057] Fig.26A and 26B An alternative arrangement is illustrated by which diffuse lighting may be achieved.

[0058] Fig. 27 An arrangement using multiple cameras is schematically illustrated.

[0059] Fig.28 An arrangement is schematically illustrated in which the camera has a split field of view, a portion of which is occupied by specular reflections of objects.

[0060] Fig.29 An arrangement is schematically illustrated in which first and second light sources are positioned and operated alternately to optimize the capture of specular and diffuse light reflected from an object.

[0061] Fig.30 Represents three overlapping 32×32 watermark element image patches, as well as depictions of all individual watermark elements.

[0062] Fig.31 The diagram illustrates the determination of the center of gravity of an object (here a transparent, clear beverage bottle) by the position of the block in which the watermark reference signal is detected.

[0063] Fig.32A and 32B A recycling system including a box-addressed sorting diverter is illustrated.

[0064] Figures 33A-33D Various camera and light configurations are illustrated.

[0065] Fig.34 The watermark detection corresponding to different LED channels is illustrated.

[0066] Fig.35 The watermark detection corresponding to different LED channels is illustrated.

[0067] Fig.36A and 36B Illustrated is the bin value determination.

[0068] Fig.37 Diagram of an example ecosystem where Fig.32A and 32B The recovery system shown in . DETAILED DESCRIPTION

[0069] There is a growing demand for highly reliable identification of plastic objects, for example for sorting waste streams.

[0070] Digital watermarks are advantageous for this purpose because they can be applied to materials of various types and shapes. In addition, watermarks can be spread over containers and / or their labels in a manner that enhances readability, even when the object is damaged, soiled, or partially obscured.

[0071] The digital watermark provides a 2D optical code signal that enables machine vision in waste sorting systems to figure out the type of material in each object and sort the waste stream accordingly. The encoded signal imparted into the container via 3D printed molds, laser textured molds, and etched molds (described below) can be used to sort the containers in various recycling environments.

[0072] According to one aspect of the technology, plastic items are encoded with two different watermarks: one watermark printed - usually via ink - on a label attached to the item (or printed on the item itself), and one formed by 3D texturing of the plastic surface.

[0073] Printed watermarks typically convey a retail payload that is primarily designed for use with point-of-sale terminal scanners, e.g., containing or pointing to product name, price, weight, expiration date, packaging date, etc., to identify and price the item at retail checkout. Textured watermarks typically include a payload that can be used for recycling, e.g., containing or pointing to data related to plastics. Each watermark typically lacks some or all of the information conveyed by the other watermark.

[0074] Importantly, in most embodiments, the two watermarks (retail and recycling) use different signaling protocols. Applicants have discovered that the time interval between reading the retail watermark by a typical point-of-sale retail scanner before the next image frame arrives for analysis is very short. If the retail watermark and the recycling watermark are depicted in the same image frame, they must be quickly distinguishable - otherwise the scanner may not be able to successfully decode the retail watermark before the next image frame arrives. Part of this specification teaches how to make the two watermarks easily distinguishable so that the retail scanner does not waste precious milliseconds trying to decode the recycling watermark, thereby helping to ensure reliable retail checkout operations.

[0075] The primary method of making retail watermarks and recycling watermarks easily distinguishable is through the use of different signaling protocols (including, for example, different reference signals, different encoding protocols, and / or different output formats.) Through these differences, point-of-sale scanners can reliably discern retail watermarks, and recycling systems can reliably discern recycling watermarks, without the risk of a point-of-sale scanner accidentally expending effort trying to decode the payload from a recycling watermark, thereby causing confusion.

[0076] Despite the differences in watermark signaling protocols, the recycling system is preferably also configured with a watermark processing module that is adapted to read the retail watermark (as well as the recycling watermark) and discern from the retail watermark information that can be used for plastic recycling purposes (typically by reference to a database that correlates the retail watermark payload data with plastic information). Thus, no matter which watermark the recycling system reads from an item, the system will have information to control the correct sorting of the items by plastic type.

[0077] As described above, the signaling protocols of the two watermarks can differ in a variety of ways, for example, including a reference signal and / or the encoding algorithm used. The reference signal (sometimes called a calibration signal, synchronization signal, grid signal, or registration signal) of each watermark serves as a synchronization component, enabling the geometric pose of the watermark (as depicted in the captured image) to be discerned so that the payload can be correctly extracted. An exemplary reference signal is a constellation of multiple peaks in the spatial frequency domain. The first of the two watermarks may include a first reference signal that is absent from the second. (The latter watermark may include a different reference signal, for example, consisting of peaks of different frequencies, different peak phases, and / or a different number of peaks.)

[0078] The encoding algorithms may differ in the process of encoding the data and / or the format in which the encoded data is expressed. For example, a print watermark encoding algorithm may employ a signaling protocol in which the resulting watermark format is a square with a side length of 0.85 inches, a watermark element resolution of 150 per inch, and conveys data structured as a 128×128 array of element positions. In contrast, the signaling protocol employed by a texture watermark encoding algorithm may produce squares of different sizes (typically less than 0.85 inches on a side), have a watermark element resolution other than 150 per inch, and / or convey data structured as a different than a 128×128 array. The two different signaling protocols employed in the two watermarks may have different payload capacities, for example, one with a variable message portion that can convey 48 bits and the other with a variable message portion that can convey half or one-third of that payload capacity.

[0079] The two encoding algorithms may additionally or alternatively differ in the error correction encoding method used (if any), the redundancy rate employed, the number of bits in the signature string output by the error correction encoder, the CRC method used (if any), the scrambling key used to scramble the signature string output by the error correction encoder to produce a scrambled signature string, the expansion key used to generate a large number of randomized "chips" from each bit of the scrambled signature string, the scattering table data defining the spatial placement of each of these "chips" in the output watermark pattern, etc. The decoding algorithm may differ accordingly.

[0080] The inability of one watermark reader (e.g., a point-of-sale retail watermark reader) to read another type of watermark (e.g., a recycled watermark) may be due to any of the just-mentioned differences between the watermarks, e.g., regarding their geometric reference signals, output formats, signaling protocols, encoding / decoding algorithms, etc.

[0081] Each watermark payload typically consists of fixed and variable message parts. The fixed part typically includes data that identifies the signaling protocol used. The variable message part typically includes multiple fields. For printed retail watermarks, one field typically conveys the Global Trade Item Number (GTIN), while other fields may convey GS1-defined Application Identifier codes (e.g., indicating weight, expiration date, etc.). Plastic identification information may be conveyed in printed retail watermarks in the form of such Application Identifier codes, although such AI codes are not currently part of the GS1 standard.

[0082] Some recycling systems employ two watermark readers, a first reader configured to apply a first watermark reading algorithm (e.g., to read a retail watermark using a first signaling protocol) and a second reader configured to apply a different second reading algorithm (e.g., to read a recycling watermark using a second signaling protocol). Each such reader is incapable of reading the other type of watermark. Other recycling systems employ a single reader configured to read both types of watermarks. Still other systems employ a hybrid arrangement in which certain components are shared (e.g., to perform common FFT operations) while other components are dedicated to one type of watermark or the other.

[0083] To ensure reliable reading of the watermark regardless of the location of the item in the waste stream, the watermark is preferably visible from multiple viewpoints of the item. For example, it is desirable to form a recycled, textured watermark on several surfaces of each item, including the front and back. Likewise, a retail printed watermark is ideally formed on opposite sides of each item, such as on the front and back labels.

[0084] To most effectively apply watermark reading efforts, certain embodiments of the present technology examine image pixel blocks for clues that suggest the presence of watermark data. Only image blocks where such clues are found are further analyzed for watermarks. Many of these clues are detailed and include detection of glare spots (an area of ​​pixels where each pixel has a value above a threshold), detection of a set of spatial image frequencies corresponding to a watermark reference signal, classifier output indicating that a pixel block likely depicts a plastic item, classifier output indicating that a pixel block likely does not depict a conveyor belt, a determination based on previous images that a majority of sub-blocks within the block have average values ​​within 1, 2, 3, or 4 numbers of a histogram peak, detection of signals associated with conveyor belt markings, detection of salt / pepper markings, and a variety of other techniques for distinguishing promising image blocks from other image blocks. When a small patch of promising imagery is identified, it is typically analyzed to determine if both retail and recycling watermarks are present.

[0085] In some embodiments, when an image block is found to be promising, the determination also triggers an examination of multiple nearby image blocks. An incoming frame of an image may initially be divided into blocks at a first density (e.g., with a first pixel spacing or overlap). When a promising block is found, other blocks are examined at a greater density, e.g., separated from the promising block by a smaller pixel spacing or a greater overlap. Relatedly, when a promising block is found in one frame, watermark data at different locations may be analyzed in the next frame based on the conveyor speed and frame capture rate - corresponding to predicted motion of an item depicted in the promising block.

[0086] Because images of both types of watermarks are analyzed, sometimes two instances of the same type of object (e.g., two identical 12-ounce Pepsi bottles) will be sorted based on the reading of two different watermarks. That is, the plastic type of the first bottle may be identified by its printed watermark, while the plastic type of the second bottle may be identified by its textured watermark. Despite reading different watermarks, both will be diverted to the same recycling destination.

[0087] Special Arrangements

[0088] Digital watermark information is read from image data depicting plastic objects in the waste stream. This information can indicate the type of plastic (e.g., polyethylene terephthalate, high-density polyethylene, low-density polyethylene, polypropylene, polycarbonate, etc.) and can also convey other information useful for recycling. Diverters and other mechanisms in the automated sorting system are controlled based on such watermark information to direct plastic objects to appropriate destinations for recycling or reuse.

[0089] Digital watermarks (hereafter watermarks) are printed on the packaging of many products and are often used to encode a Global Trade Item Number, or GTIN (much like the ubiquitous 1D UPC barcode), but in a visually non-intrusive way. Point-of-sale scanners in retail stores can detect and decode the watermark data, use it to look up the product's identity and price, and add it to the shopper's checkout register. The watermark data is typically organized into squares that are redundantly tiled edge-to-edge across some or all of the printing on the product. Because the watermark data is spatially dispersed, scanners can read the data from different views of the product (e.g., from the front and back views of a beverage bottle).

[0090] Most typically, the watermark data is hidden as subtle variations in the brightness and / or chromaticity of the pixels that make up the package image. Sometimes the watermark can take the form of an inconspicuous pattern of dots that may be scattered, for example, on a self-adhesive label affixed to a plastic fresh food container.

[0091] To keep costs down, point-of-sale scanners often use simple processors. Such scanners typically devote most of their energy to finding and decoding 1D barcodes; watermark reading is sometimes an afterthought. A scanner that captures 30 frames per second only needs 33 milliseconds to process each frame, spending most of that time on barcode reading. Only a few milliseconds are available for watermark reading.

[0092] There are two parts to watermark reading: finding the watermark and decoding the watermark.

[0093] In an illustrative implementation, finding a watermark (sometimes referred to as watermark detection) involves analyzing frames of captured images to locate a known reference signal. The reference signal can be a characteristic constellation of peaks in the 2D Fourier amplitude domain (also called the spatial frequency domain). In the spatial (pixel) domain, such a reference signal takes the form of a collection of two-dimensional sinusoidal sums of different spatial frequencies across the watermark block. Figure 5A An illustrative reference signal in the Fourier amplitude domain is shown; Fig. 6A This same reference signal in the spatial domain is shown. The frequency is preferably an integer value to ensure continuity along the edges of the watermark block. When an object with such a known reference signal is depicted in a captured image, its specific presentation reveals the scale, rotation and translation of the watermark payload data also present in the image.

[0094] The watermark payload data is encoded by watermark elements ("waxels") occupying a 2D array of positions - typically 128 x 128 elements in size. The array may span an area of, for example, 0.85 or 1.7 inches on a side, depending on whether the watermark is formed at a resolution of 150 or 75 watermark elements per inch (WPI). Such blocks are tiled in a repeating array throughout the package along with the reference signal.

[0095] Once the scale, rotation and translation of the watermark are known from analysis of the reference signal depicted in the captured image, the watermark payload can be decoded. The decoder samples the captured image at locations corresponding to the originally encoded 128×128 array of data and uses these sampled values ​​when decoding the original watermark payload. (Convolutional encoding is typically used, for example, to convert the 48-bit payload into a string of 1024 data, which is then redundantly interspersed among the 16,384 locations of the 128×128 element watermark block.)

[0096] These and other details of watermarking techniques, including those described in the patent documents identified herein, are familiar to skilled artisans.

[0097] In one particular embodiment of the present technology, a plastic container conveys two watermarks - one formed by label printing and a second formed by texturing of the plastic surface, such as by molding. (The label may include a substrate that is printed and applied to the container, or it may include printing applied directly to the container.)

[0098] Plastics can be molded in a variety of ways, including blow molding, injection molding, rotational molding, compression molding, and thermoforming. In each such process, the heated plastic resin is shaped according to the mold. By forming a pattern on the mold surface, a mutual pattern is formed on the surface of the resulting plastic product. If the pattern in the mold is customized (for example, by CNC milling, laser etching, or 3D printing) to have the shape of the watermark pattern (the change in brightness / chromaticity is converted into mold height, depth, angle, reflectivity, or local curvature), the resulting plastic product will have a surface texture corresponding to the watermark. This pattern in the plastic surface can be sensed by the optical method detailed below.

[0099] Figure 1A-1Q is an illustration of a representative surface texture.

[0100] Most of the textures depicted are illustrated as 2D cross-sections of a 3D surface - only one dimensional modulation is shown. For clarity, the textures are shown on a flat surface. Naturally, most plastic containers are curved in at least one dimension.

[0101] Additionally, for clarity, Figure 1A-1Q Most of the illustrations show markers with only two states. Specific examples include the "sparse" point markers detailed in patent publications US20170024840, 20190139176, and WO / 2019 / 165364. Other two-state markers include line art patterns such as Voronoi, Delaunay, traveling salesman, and bricks, as detailed in published applications WO / 2019 / 113471 and US20190378235, and shown in Figure 3A ,3B , 3C, and 3D. (The Voronoi pattern is implemented by forming a grid of flashes (here, triangles) whose vertices are located at positions corresponding to the sparse array of points. The Delaunay pattern is the dual of the Voronoi pattern, where the flashes take the form of polygons with different numbers of sides. The traveling salesman pattern is implemented by defining a traveling salesman path that visits each point in the sparse array of points. The brick pattern is implemented by placing vertical line segments at the locations of the points in the sparse array of points and forming horizontal lines at the middle locations, thus defining a rectangular flash.)

[0102] Figure 1A Labeled to show the binary state of each watermark element. A "1" here is represented by a relatively raised segment; a "0" is represented by the nominal baseline elevation of the plastic surface (which can be seen as relatively depressed compared to the depicted "1" state). The nominal elevation of the plastic surface is shown by the dotted line.

[0103] Figure 1B and Figure 1A Similar, but the sharp corners have been rounded (e.g., by low pass filtering) to help the formed plastic release from the mold. This rounding can be used in any embodiment to smooth the sharp corners.

[0104] Figure 1C An embodiment is shown in which the transition between states is ramped and successive "1" values ​​include a brief return to the opposite nominal surface level. Figure 1D yes Figure 1C The angled transition further facilitates demoulding and, depending on the lighting, facilitates optical inspection.

[0105] In some embodiments, Figure 1B , 1C Each of the raised protrusions in 1D and 1D can be a rounded protrusion with only a small flat area or no flat area at its peak.

[0106] Figure 1E A feature showing a "1" state may be a surface that is non-parallel to the nominally flat surface characterizing the "0" state. Figure 1F yes Figure 1E A variation showing that the "1" state does not need to be raised; it can simply be tilted.

[0107] Figure 1G An arrangement is shown in which the "1" and "0" states are each tilted in different directions relative to the nominal surface of the plastic. (As shown, the tilt directions may differ by 180 degrees or by 90 degrees.) This tilt causes light to be preferentially reflected in different directions, making the mark more visible to a watermark reader.

[0108] Although Figures 1A-1G has been described and illustrated as including segments above the nominal surface, but it will be appreciated that such encodings may similarly (and perhaps more commonly) include segments depressed below the nominal surface. (Watermark encoding / reading is generally independent of polarity - upward or downward.) An example is the formation of Figures 3A-3D and Figure 7 Lines used in patterns. Combinations of elevations and depressions can also be used naturally.

[0109] Figure 1H Illustration of the favorable dispersion phenomenon associated with curved surfaces. For most placements of cameras and light sources, for a curved surface, incident light (shown by the larger arrows) will reflect from the surface at different angles (shown by the smaller arrows), with some of it reflecting toward the camera, creating a bright flash. In contrast, for a flat surface, essentially all of the incident illumination will reflect in one direction - most likely away from the camera. As a result, flat surfaces will generally appear dark to the camera, while curved surfaces will generally feature a bright flash. (If, by chance, a flat surface reflects toward the camera, an "inversion" occurs, with the flat surface appearing brighter than the curved surface.)

[0110] Fig. 1I The dispersion and focusing phenomena associated with a surface having bumps and dimples are shown. The bumps act as described above - to disperse incident light over a wide range of angles. In contrast, the curved dimples act as focusing elements. The focusing caused by the dimples results in more light being reflected in the general direction of the light source than the dispersion caused by the bumps. Assuming the camera is relatively close to the light source (e.g., within 10 degrees of the illuminated surface), the dimples will appear brighter than the bumps in the image captured by the camera. (The dashed line shows a nominal plastic surface.)

[0111] (Here and elsewhere, it will be appreciated that the light source and camera may be placed differently than shown in the figures. They may be very close together, such as within single digit degrees, or further apart. The light may strike the surface straight down (90° incidence) or at an angle, such as at an angle of incidence of 80°, 60°, 30°, or less.)

[0112] Fig. 1I The arrangement can be extended to three surface features, bumps, dimples, and planes, e.g. Figure 1J As shown. The flat surface will be like the combination Figure 1H Therefore, Figure 1J The arrangement is an example of a surface that can be used for ternary signal encoding, reflecting variously a moderate amount of light (i.e., the flash caused by the bumps), a large amount of light (i.e., the focused reflection caused by the pits), and an extreme (usually dark, but sometimes bright, caused by the flat areas).

[0113] Figure 1J Another aspect of surface shaping is also illustrated, which can be used in any embodiment: the size of the protrusions does not have to be similar to the indentations. In this example, the protruding bumps are deeper than the indentations. Relatedly, the protruding bumps have a smaller radius of curvature than the indentations. The opposite is also possible.

[0114] Figure 1K The figure shows that adjusting the surface height has little, if any, effect on the reflected light pattern. transition , which is the function that defines the surface height Derivative .

[0115] exist Figure 1K In the example, light incident at point B on the plastic surface will be reflected with the same direction and intensity as light incident at point D. The two surfaces are at different elevations, but are parallel. Conversely, light incident at point A will be reflected with a different intensity and direction than light incident at point C. At point A, the surface derivative is negative (decreasing with increasing altitude as you move to the right). At point C, the surface derivative is positive. Assuming the camera is located near a light source, almost no incident light will be reflected back to the camera from point A, and virtually all incident light will be reflected back to the camera from point C. Therefore, Figure 1K The flat-bottomed pit shown in cross section has three reflective zones: one along the flat bottom, one with a negative derivative, and one with a positive derivative. Placing the light source (and a nearby camera) as shown, the camera will sense reflected flash from the latter zone, with no reflection from the first two zones.

[0116] Similar phenomena also come from flat-topped raised bumps, such as Figure 1L The leftmost side of the bump has a positive derivative and reflects a flash of light back to the camera. The flat top does not reflect light back to the camera, nor does the rightmost side of the bump (and its negative derivative).

[0117] (It will be appreciated that the results described depend on the light source being located to the left of the forming surface. If the light is located to the right, some of the results are reversed.)

[0118] Naturally, the shape of the curve can be tailored to optimize performance, for example to favor reflection from a particular direction relative to the incident light source.

[0119] Generally speaking, bumps and dots with a rounded, flat appearance are preferred because they tend to reflect light more omnidirectionally than component shapes with straight edges.

[0120] Notwithstanding the previous sentence, another advantageous method of surface texturing is to employ retroreflective features, such as dimples in the shape of 3D corner reflectors in the plastic. The property of a 3D corner reflector is that light is reflected back to its source over a wide range of incident angles. Figure 1MThis property is shown in two dimensions; it extends to three dimensions as well.

[0121] Angular indentations can be made on the plastic surface where the watermark should have a bright appearance (e.g. the "1" state), and can be omitted where the watermark should have a dark appearance (e.g. the "0" state). The deepest "point" of the indentation can be rounded; what is important is that most of the surface extents are perpendicular to each other.

[0122] Figure 1N A fragment of a square 128x128 watermark element watermark block is shown, showing 16 watermark elements. Some are indented with retroreflective 3D corner reflectors according to the data being encoded (e.g., representing a "1" signal); others remain flat (e.g., representing a "0" signal). Fig.1O A segment using triangular watermark elements is shown, organized in a hexagonal array. Again, depending on the data being encoded, some are indented with retroreflective 3D corner reflectors; others are not.

[0123] In a variant embodiment, the two states of the signal tiling are not expressed by a corner reflector or a flat surface. Instead, a corner reflector is formed at the location of each watermark element. The two states differ in the treatment of the three orthogonal surfaces (facets) that define the concave reflector. The "1" state is characterized by a smooth surface that reflects light with relatively little scattering. The "0" state is characterized by a textured (e.g., rough or matte) surface that reflects light with relatively more scattering. The first type of reflector is efficient; the second type of reflector is inefficient. However, to a human observer, these two characteristics make the surface have a uniform texture that is essentially indistinguishable. Figure 1P Such an arrangement is shown (the rough corner reflectors are shown by grey watermark elements).

[0124] If square watermark elements are formed in plastic at a density of 75 per inch, each watermark element spans an area of ​​0.0133 inches on a side. Therefore, the width of each corner reflector indentation is equal to or less than this value. The higher the density of watermark elements, the smaller the size.

[0125] Naturally, in a retroreflective arrangement the camera should be placed as close to the light source as possible, ideally so that the angular distance between the two (as seen from the conveyor belt) is less than ten degrees.

[0126] Figure 1PThe texturing of some surfaces in an arrangement may be used in other arrangements, including the other depicted arrangements. That is, certain areas of the plastic surface may be roughened or matte treated to increase scattering, while other areas may remain smooth to provide more specular light reflection. In some embodiments, the plastic surface has no depressions or elevations for encoding watermark data. Instead, the encoding is achieved entirely through scattering texturing of different areas without otherwise disturbing the nominal shape of the item.

[0127] Figure 1Q A small fragment of the plane is shown in the 3D view, marked with 3 sparse points - here in the form of surface depressions.

[0128] While many of the shown surfaces can encode two signaling states, and some can encode three states, more generally, multi-ary encodings can be used.

[0129] Figure 2A Another form of ternary encoding is shown, where the signal consists of -1, 0, and 1 elements. A "-1" represents a tilt in one direction, a "1" represents a tilt in another direction, and a "0" represents a tilt halfway between the other two directions. Many other such forms can be devised naturally, for example, by including depressions from the nominal plastic surface that reflect Figures 1A-1F protrusions. Quaternary coding can be achieved using four different surface tilts at consecutive 90 degree angles. Quinary coding can be achieved by using four tilts for orthogonal coding and a fifth state as the nominal surface of the plastic. Higher order multi-level coding can be achieved by expanding the tilt set.

[0130] (As in the other figures, Figure 2A Surfaces in the image may be roughened, for example with a matte or frosted finish, to scatter some light into directions other than pure reflection. )

[0131] In addition to multi-ary encoding, the present technique is also applicable to so-called "continuous tone" watermarks, which have a variety of intermediate states between two extreme values. Typically, the reference signal is continuously valued (or valued by a large set of quantization steps), and the sum of this reference signal and the multi-ary payload pattern representation produces a continuous tone watermark. The continuous values ​​of the watermark elements in these marks can be represented by localized surface heights or slopes. Figure 2B Such markings are conceptually illustrated.

[0132] The patterns just discussed show that the shape extends to both surfaces of the plastic medium, such as the top and bottom (or the inside and outside of the bottle). Sometimes, only one surface (such as the outside) is shaped, while the other surface is smooth.

[0133] While texturing of plastics using a molding die is most common, other molding methods can be used. Laser or chemical etching is an example, and results in a surface marked with depressions that correspond (by amplitude or tilt) to the spatial variations of the watermark signal. (Laser etching is well suited for serialization - where each instance of an item is encoded differently.)

[0134] In some embodiments, the plastic surface is locally treated to present a matte or frosted finish, rather than a glossy finish. In this case, the watermark itself can be formed as a pattern consisting of matte and glossy watermark elements. Matte textures are achieved by molding or otherwise treating the surface to obtain a measure of surface roughness, such as vertical variations on the order of one-tenth of a micron or half a micron or more.

[0135] In an exemplary embodiment, the plastic watermark is customized to avoid being confused by point-of-sale scanners. As noted above, such scanners have limited processing power and limited time to extract the watermark identifier. Several measures can be taken to help prevent point-of-sale scanners from attempting to read the plastic watermark - an effort that wastes valuable processing time and may prevent the scanner from decoding the product GTIN from a product label depicted in the same frame.

[0136] One measure to help avoid confusion by point-of-sale scanners is to use a reference signal in the plastic watermark that is unlikely to be mistaken for the reference signal used in the printed label watermark. Such a reference signal can be developed experimentally by randomly generating multiple candidate signals (e.g., by picking a set of random peak locations in the spatial frequency domain and assigning a random phase to each signal) and testing each candidate signal to assess the likelihood that a point-of-sale watermark reader will mistake such a signal for the printed label watermark reference signal. The candidate reference signal with the lowest likelihood of confusion is then used.

[0137] Another approach is similar, but involves mathematically computing a theoretical confusion (correlation) metric that indicates the similarity between different candidate random reference signals and the printed label reference signal, and picking the candidate with the lowest correlation.

[0138] The applicant prefers the first approach because, as a wise man once quipped, in theory there is no difference between theory and practice, but in practice there is a difference.

[0139] The process of finding candidate reference signals for plastic watermarks can be aided by applying different constraints during the signal generation or selection process. One is that, ideally, any peak in the plastic reference signal should not be identical to any peak in the printed label reference signal. Any randomly generated candidate plastic reference signal with this property can be discarded.

[0140] Figure 4 The peaks of the printed label watermark reference signal in the 2D Fourier amplitude domain are shown. The reference signal of the plastic watermark preferably has no common peak positions.

[0141] Relatedly, in the printed label reference signal, each frequency peak falls on a different radial line from the origin. Figure 4A The zoomed in view of shows some frequency peaks. Hopefully, no peak in the plastic reference signal lies on any of these radial lines. (Depending on the scale at which the watermarked object is viewed, the reference signal peaks move concentrically toward and away from the origin, traveling along these radial lines - if both reference signals have peaks on the same radial line, there is a risk of aliasing.)

[0142] In such an arrangement, it may be desirable to have no peaks in the plastic reference signal located on the vertical or horizontal axes 31, 32 of the spatial frequency plane. Many other features of the captured image may have signal energy concentrated along these axes, so it is desirable to avoid peaks along these axes.

[0143] The reference signal for a printed label is four-sided symmetrical - mirrored around the vertical and horizontal frequency axes, and for reasons of detector efficiency this arrangement may be used for a plastic reference signal. However, this is not required, and reference signals for plastic watermarks that do not exhibit this property may present a smaller risk of aliasing.

[0144] Although it is best to avoid peaks along the vertical and horizontal axes, it is usually desirable for the peaks of the plastic reference signal to be placed on radial lines at different angles. In each quadrant of the four-sided symmetrical reference signal, one-quarter to one-third of the peaks may be on different radial lines within 30 degrees of the horizontal axis, one-quarter to one-third may be on different radial lines within 30 degrees of the vertical axis, and one-third to one-half may be on different radial lines between these two ranges.

[0145] Also, it is desirable that the peaks of the plastic reference signal be at different distances from the origin. Low frequency points (e.g., below 20 or 25 cycles per block) are not preferred because scaling may cause them to move to a position where the watermark reading software is not looking for peaks (hence the Figure 4 The blank area in the center) and for high frequencies (e.g., more than 50 or 60 cycles per block), the inverse. But in the middle donut band ( Figure 4 ), the space budget for the allocation peak can be used, as described in the previous paragraph, to ensure a roughly even distribution.

[0146] Another measure to help avoid confusion with point-of-sale scanners is to use a reference signal in the plastic watermark that has fewer peaks than the reference signal in the printed label watermark. The fewer the number of peaks, the less likely they are to be mistaken for peaks in the printed label watermark.

[0147] A corollary benefit is that in a plastic watermark reference signal with fewer peaks, each peak can be encoded with more energy because the available signal energy budget is spread across fewer features. A plastic reference signal consisting of fewer strong peaks is less likely to cause aliasing than a reference signal consisting of more and weaker peaks.

[0148] A further method of preventing confusion between printed and plastic watermarks is to form the marks at different scales. As mentioned above, printed watermarks are typically formed at 75 or 150 watermark elements per inch (i.e., 1.7 or 0.85 square inch watermark blocks). Plastic watermarks can be formed at different resolutions, such as 200, 250 or 300 watermark elements per inch (i.e., 0.64, 0.51 and 0.43 square inches). Doing so will also help detect them from curved container surfaces, as the apparent edge distortion of the watermark pattern due to curvature is reduced with smaller watermark block sizes.

[0149] One algorithm for generating candidate plastic reference signals is to perform a Fourier magnitude plot of the label reference signal to be avoided and add two circumscribed circles (e.g. Figure 4 Then add radial lines extending from the center of the plot through each label reference signal peak to the outer circle, similar to Figure 4A Finally, triangulate within the annular space, using the label reference signal peaks as vertices to define the largest triangle that does not contain any other peaks. Then identify a point in the annulus that is farthest from the nearest straight line (i.e., radial line, triangulation line, and horizontal and vertical axis) and add it to the set of candidate points. Repeat until the desired number of points has been identified.

[0150] Different candidate plastic reference signals are tested for potential confusion with the tag reference signal by applying different random distortions, such as tilt, rotation, and scaling, as well as additive Gaussian noise, to each candidate signal and determining how often a point-of-sale watermark reader at the reference signal detection stage mistakes the distorted signal for the tag watermark's reference signal. After each candidate reference signal is tested with hundreds of different distortions, one candidate signal will typically outperform the others. (This signal may be examined by a human reviewer in the spatial domain to check that it does not have subjectively objectionable properties, but such review may be skipped.)

[0151] Several candidate plastic watermark reference signals are shown in Figure 2 through their Fourier amplitude maps. Figure 5A , 5B and 5C. Fig. 6A , 6B and 6C show their corresponding spatial domain representations.

[0152] As the spatial domain representation of the plastic reference signal becomes "flatter," confusion with the printed label watermark reference signal tends to decrease. Thus, according to another aspect of the present technique, each candidate reference signal for the plastic watermark is varied by trying different phase assignments for different peaks in the Fourier magnitude map to identify a set of phase assignments that minimize the standard deviation of the pixels in the spatial domain representation. This is a task well suited to computer automation, with hundreds of thousands or millions of different sets of phase assignments being tried to find the set that produces a spatial domain pattern with the smallest standard deviation.

[0153] Although Figures 5A-5C The reference signal patterns of (and 6A-6C) were generated experimentally, for example, using the techniques detailed above, but checking their correlation with the printed label watermark reference patterns revealed very little correlation. r , whose maximum value is 0.2> r >-0.2 (0.1 in some cases> r >-0.1), when tested for obfuscation using the print label reference pattern over the full range of affine transformations, i.e., in increments of 0.02 over the range between 0.5 and 2.0, in increments of 1 degree over the range of rotations between -90 and +90 degrees, and at every pixel where a translation is possible.

[0154] Two images f 1 and f 2 The correlation between the pixels (all P×P pixels) can be expressed as:

[0155]

[0156] It should be understood that the detailed reference signal consists of sinusoids of equal amplitude. In other embodiments, the sinusoids may have different amplitudes, thereby producing a more conspicuous "weave"-like pattern for its spatial domain representation.

[0157] As described above, the reference signal is one of the two elements of the watermark signal, the other being an encoded representation of the payload message. This representation can be generated by convolutionally encoding the symbols of the payload message to generate a longer string of symbols (e.g., 1024 bits), which can be called a signature. The signature can be randomized by XORing with a scrambling key of the same length. The chips that redundantly represent each scrambled signature bit are randomly spatially spread between positions in a square array, such as 128×128 (16,384) elements, to form a signature array.

[0158] A continuous tone watermark can be generated by scaling the spatial domain reference signal so its average pixel value is 128, and then adding or subtracting an offset value to each component pixel value, the offset value depending on whether the chip assigned to that position is 1 or 0.

[0159] Sparse watermarks can be generated by various methods, which generally involve producing an output pattern of spaced dots. Several methods are described in detail in the referenced documents and are discussed below in the section entitled "Review of Exemplary Watermark Methods".

[0160] As mentioned above and Figures 3A-3D As shown in , sparse patterns can be converted into various bitonal line-based representations. Another such pattern, called snake, is Figure 7 It is generated from a continuous tone watermark according to the following algorithm executed using Adobe Photoshop and Illustrator:

[0161] A monochrome (grayscale) block at 300 DPI with a white background was filled with 50% gray and encoded with a continuous tone watermark (reference and payload signal). The image was then Photoshop adjusted using the controls "Adjust → Exposure: Default / Exposure: 1.05 / Offset: -0.075 / Gamma Correction: 0.3". Next, filtering was applied using the Photoshop controls "Filter → Blur → Gaussian Blur: Radius: 3 pixels; then Filter → Stylize → Wind: Method = Wind / Direction: Right or Left (doesn't matter)". The image was then thresholded via the Photoshop controls "Image → Adjust → Threshold: Threshold Level: 140 (+ / - 5)". The resulting file was saved and then opened in Adobe Illustrator. Select the image you just edited from inside the layer. Click the "Image Trace" button in the main upper box of the Illustrator user interface and after the preview appears, click the "Image Trace Panel" next to the "Default" drop-down box that displays. From the top row of icons, click the "Outline" button. Once the preview is rendered, click the "Expand" button next to the drop-down box that says "Tracing Results". This renders the UI to make the stroke size in the pattern thicker or thinner. Some bolding is applied to produce something like Figure 7 mode.

[0162] It can be seen that such a pattern consists of multiple curved segments (many of the segments are compound curved, i.e. have multiple turns along their length) scattered over an area, with some segments intersecting with other segments while other segments are independent and do not intersect with other segments.

[0163] Turning now to larger systems, a recycling facility in accordance with one embodiment of the present technology employs one or more cameras and light sources to capture images of watermarked plastic containers traveling on a conveyor belt depicting a waste stream. Depending on the implementation, the area of ​​the conveyor belt imaged by the camera system (i.e., its field of view) may be as small as about 2×3 inches, or as large as about 20×30 inches or more, depending primarily on the camera sensor resolution and lens focal length. In some implementations, multiple imaging systems are employed to capture images that together span the width of the conveyor belt. (In bulk feed systems, the conveyor belt may be as wide as 5 feet or 2 meters. Single-feed systems (where one item is metered onto the conveyor belt at a time) are narrower, such as 12 inches or 50 centimeters wide. Conveyor belt speeds of 1-5 meters / second are common.)

[0164] Figure 8 A simple arrangement is shown in which the camera and light source are substantially co-located; that is, the illumination is from a position less than 10 degrees from the projection of the camera viewing axis onto the waste stream conveyor (i.e., the camera target). In another arrangement, the light source is positioned to illuminate the camera target obliquely, i.e., the light source is directed in a direction more than 50 degrees away from the orientation of the camera lens axis, such as Fig. 9 In yet another arrangement ( Fig.10 ), opposite illumination is used. That is, the axis orientation of the light source differs from the orientation of the camera lens by more than 140 degrees. In the latter arrangement, surface texturing can result in localized shadows on the plastic surface, for example, each plastic protrusion blocks light, causing adjacent areas to be imaged at a relatively lower brightness than the area where light is incident.

[0165] Figure 8-10 The positions of the cameras and light sources in each of the embodiments can be interchanged. In other embodiments, multiple light sources can be used. Of course, the exposure interval should be short enough to avoid motion blur. Stroboscopic light sources help avoid blur. The light sources can be as close to the conveyor belt as the size of the items passing underneath allows, or they can be spaced a greater distance, such as two feet or four feet.

[0166] Fig.11 An arrangement is shown where two different coloured light sources (red and blue) illuminate the camera target from opposite sides of the camera - in this example at oblique angles (>50 degrees). A green light source is co-located with the camera. Fig.11The camera in is an RGB camera, comprising a 2D CMOS sensor covered with a Bayer pattern of color filters. Its raw output includes red-filtered pixels, green-filtered pixels, and blue-filtered pixels. Thus, three different monochrome (grayscale) images are formed by corresponding pixels from the sensor array, one depicting the waste stream in the red part of the visible spectrum, one depicting the waste stream in the blue part of the spectrum, and one depicting the waste stream in the green part of the spectrum.

[0167] Fig.11 The arrangement shows light sources arranged along the direction of movement of the conveyor belt (waste stream). In alternative embodiments, the light sources are across the direction of movement of the conveyor belt rather than in a straight line with the direction of movement of the conveyor belt. In other embodiments, a first pair of red / blue light sources are positioned along the direction of movement of the conveyor belt (as shown in the figure), while a second pair is positioned across the direction of movement. The light source pairs are activated for alternating frames of images captured by the camera (which can be capture frames, for example, at 60 or 150 frames per second). One frame is illuminated by red / blue light sources arranged in a straight line, the next frame is illuminated by crossed red / blue light sources, etc. Each frame is illuminated by a green light source.

[0168] The watermark data of each resulting image frame is analyzed to look for printed label watermarks and plastic watermarks. In some embodiments, a fourth image frame is generated by calculating the difference between the red and blue pixel values ​​in each Bayer cell. The resulting difference can be divided in half and summed with an offset value of 128 to ensure that the elements of the difference image are in the range 0-255. The difference image is also processed to decode any printed label or plastic watermarks that are present. This arrangement is shown in FIG. Fig.13 shown.

[0169] Although Fig.13 Eight different watermark reading systems are shown, but some of the image processing can be merged and the results shared between the label and plastic reading watermark systems. For example, each red data frame can be FFT'd by a common FFT stage and the result used for label and plastic reading (synchronized). However, if the resolutions of the two watermarks are different (e.g., 150 and 250 WPI), completely independent processing paths may be preferred.

[0170] In one particular embodiment, the image is captured with an f / 8 lens with an exposure interval of 20 microseconds. The imaging distance is set so that each captured pixel corresponds to an area of ​​approximately 1 / 150 inch in the focal area located 3 inches above the conveyor. Therefore, each pixel corresponds to a single watermark element at 150 WPI. The camera gain (or distance from the light source to the conveyor) is adjusted so that a pure white item on the conveyor is depicted with a captured 8-bit pixel value of 250.

[0171] The effective dynamic range of an imaging system can be extended by employing zones of varying illumination intensity. A normal intensity zone can be illuminated as just described so that a white target produces a camera pixel value of 250. An adjacent high intensity zone can be illuminated at twice the intensity or more. Doing so will overexpose the bright areas but allow dark items to be distinguished by better brightness gradations (i.e., contrast enhancement). For example, a watermark pattern on a darkly printed label may appear as pixel values ​​in the 5-10 range under front lighting conditions, but may appear in an extended range, such as 10-20 (or even 50-100) under rear lighting conditions.

[0172] In one particular implementation, this illumination variation is a design parameter of the lens over a single light source. For example, a linear array of LEDs may be equipped with a linear lens that projects a pattern of varying intensities with a high intensity zone in the center flanked by normal intensity zones. As a conveyor moves the items past the projected light, each point on the item first passes through a normal intensity zone, then a high intensity zone, and then another normal intensity zone. Depending on the conveyor speed, frame rate, and illumination zone, each point on the item may be imaged once, twice, or more times as it passes through each of these zones.

[0173] In another arrangement, two or more different light sources may be employed to provide higher and lower intensity light zones with similar effects.

[0174] exist Fig.12 In yet another arrangement depicted in , a linear light source 120 (shown in side view) designed to output a substantially uniform intensity across its illuminated area is tilted relative to a conveyor belt 122 so that the length from the light to different areas of the conveyor belt is different. In this case, the falloff of illumination with distance produces a gradient effect, wherein areas 124 of the conveyor belt closest to the light are intensely illuminated, while areas 126 further away are gradually less illuminated.

[0175] In one particular embodiment, the captured image frames span areas of brighter and darker illumination on the conveyor. In a first single frame, the bright areas on the item will be overexposed, while the dark areas will be contrast enhanced. In another single frame, the bright areas will be properly exposed, while the dark areas will be relatively underexposed. The decoder will tend to ignore one or more overexposed areas because it has no changes in pixel values ​​that can serve as a clue to select patches to analyze, and therefore will not analyze such patches. The decoder will similarly tend to ignore one or more areas that are too dark because there is also a lack of pixel changes. Therefore, in a series of frames depicting a single item as it passes through variable illumination, a darker area will tend to be analyzed from one frame (when it falls in brighter illumination) but not in another frame (when it falls in darker illumination). Similarly, a brighter area will tend to be analyzed in one frame (when it falls in darker illumination) but not in another frame (when it falls in brighter illumination).

[0176] In another arrangement, red and white light sources are used. One or more red light sources and one or more white light sources can illuminate a common area, or they can illuminate overlapping or adjacent areas. All of these illuminated areas can be within the field of view of a common imaging camera.

[0177] In other arrangements, polarized light is used for illumination. Additionally or alternatively, one or more polarization filters may be used at the image sensor to attenuate light of orthogonal polarizations.

[0178] In many applications, glare (specular reflections of light from surfaces) is an impediment. In contrast, in certain embodiments of the present technology, such specular reflections may be important in signaling watermark information. In addition to filtering out glare, polarizing filters may also be used to emphasize signaling glare.

[0179] Some embodiments of the present technology employ novel image sensors with polarization filter arrays. One example is the Sony Polarsens image sensor. The pixel array is overlaid with a spatially corresponding polarizer array consisting of four polarizers at different angles (90°, 45°, 135°, and 0°). Image frames consisting only of data from the 90° polarization sensor can be analyzed for watermark data. Similarly, image frame analysis is performed for each of the other three polarization states. In addition, the difference between, for example, a 90° "image" and a 45° "image", etc., can be calculated, and such difference images can also be analyzed for watermark data.

[0180] This Sony sensor is available in several configurations. The IMX250MZR is illustrative. It is a monochrome CMOS sensor with 2464×2056 pixels. The color counterpart is the Sony IMX250MYR.

[0181] The sensitivity of human vision is particularly acute in the green spectrum, so it is unlikely that digital data can be encoded in the green channel if the goal is to be imperceptible. Even better would be a camera optimized to sense digital data using wavelengths away from green, such as blue and red (extending to ultraviolet and infrared in some cases).

[0182] Our patent 10,455,112 details a sensor optimized for digital watermark detection (in non-green visible wavelengths). One specific embodiment detailed in that patent uses a color filter array on a monochrome sensor with three magenta-filtered photocells for each green-filtered photocell.

[0183] Once the plastic items are identified, they can be transferred from the conveyor to appropriate containers or to other conveyors by known means, such as solenoid-operated plungers, stepper motor-controlled arms, forced air jets, etc. Exemplary separation and sorting mechanisms are known to the skilled person, for example, from patent publications US5,209,355, US5,485,964, US5,615,778, US20040044436, US20070158245, US20080257793, US20090152173, US20100282646, US20120168354 and US20170225199. These mechanisms are referred to herein as "sorting diverters", or simply "diverters", and their operation is controlled based on the type of plastic identified.

[0184] Fig.14 Some of the data involved are detailed in more detail.

[0185] In the depicted embodiment, each plastic watermark conveys a 32-bit payload. This payload can be segmented into different fields. One field identifies the plastic type by category (e.g., ABS, EPS, HDLPE, HDPE, HIPS, LDPE, PA, PC, PC / ABS, PE, PET, PETG, PLA, PMMA, POM, PP, PPO, PS, PVC, etc.). Another identifies the subtype of the plastic, for example by its average molecular weight, solution viscosity value, or recommended solvents, or by whether the plastic is used as a food container or a non-food container. A third field identifies the color of the plastic. (Color can sometimes be perceived optically. However, plastic consumer packaging increasingly has shrink sleeves printed on the container to hide its color.) A fourth field identifies the date the plastic was manufactured, for example by month and year. A fifth identifies the country of manufacture. A sixth identifies the manufacturing company. Naturally, more or fewer fields may be used. Other fields include whether it is packaged for food (vs. non-food), whether it is multi-layered (vs. single-layered), and whether it is compostable (vs. recyclable only). Some fields convey index or flag (yes / no) values. If necessary, each index value can be resolved into a literal text, date string, or a value (or range of values) by referencing a data structure (such as a table or database).

[0186] In one illustrative embodiment, the sorting diverter is responsive to the first three fields of data for separating plastics by type, subtype, and color. All decoded watermark data is recorded to provide statistics about the waste stream being processed.

[0187] Printed label payloads typically convey a longer payload, such as 48 or 96 bits. The contents may vary from item to item, but each typically begins with the GTIN and may be followed by one or more application identifier key-value pairs (e.g., indicating expiration date, batch number, item weight, etc.). In some arrangements, none of the payloads indicate the type of plastic used in the item's container.

[0188] To determine the plastic type, a data structure 121, such as a table or database, may be used. It is used to associate the item GTIN with corresponding information about the plastic used for the item container. That is, the data structure is queried using the GTIN identifier decoded from the printed label watermark payload, and the system thereby accesses previously stored data identifying the plastic type, subtype, and color (if available) of the product with that GTIN. This plastic material information is provided to the logic controlling the sorting diverter, just like the data of the plastic watermark.

[0189] From the foregoing, it will be appreciated that the technical problem of the prior art is to ensure reliable reading of a GTIN label watermark on a product package presented to a point of sale scanner within the limited time and processing constraints of such an environment. The technical effect of the detailed arrangement is to enable such a package to convey a second watermark to facilitate recycling without affecting the reliable reading of the GTIN label watermark by the point of sale scanner due to the different signaling protocols used in the two watermarks.

[0190] Another technical problem is to ensure reliable optical reading of watermark data from items in a fast moving waste stream. In some embodiments, reliability is enhanced by an imaging device used to capture a depiction of the items in the waste stream. In some embodiments, reliability is enhanced by the shape of a textured mark applied to the surface of a plastic container within the waste stream.

[0191] It will be appreciated that the present technology may be used in waste sorting systems sold by Pellenc ST, MSS, Bulk Handling Systems, National Recovery Technologies, Rofin Australia PTY, Green Machine Sales, Eagle Vizion, BT Wolfgang Binder, RTT Steinert, S+S Separation and Sorting Technology and Tomra Systems ASA. The optical sorting employed in such machines (e.g., based on near infrared or visible spectra, based on different absorption spectra of different plastics, etc.) may be replaced by the present technology, or the present technology may be used in combination with these other methods.

[0192] Block Analysis

[0193] In one illustrative embodiment, a conveyor belt is spanned by an array of cameras, each providing image frames at a rate of 150 per second. Each frame measures 1280×1024 pixels and spans a field of view measuring approximately 8×6 inches on the conveyor belt. Analysis blocks are arranged over each captured image, and each block is analyzed for watermark clues, such as a watermark reference signal. If a watermark reference signal is found, it is used to identify the pose of the watermarked object on the conveyor belt (e.g., using the techniques detailed in patents US9,959,587 and US10,242,434). Using the pose information, the image is resampled in the area where the reference signal was detected to extract the watermark element data, which is then provided to a decoder seeking to extract the watermark payload.

[0194] Although the present description generally refers to processing image blocks or patches measuring 128×128 pixels (or watermark elements), the applicant has found that detailed arrangements can often be better provided by processing smaller data sets, such as 96×96, 88×88, 80×80, 64×64, etc. (Due to the bending and crushing of items in the waste stream, there are not many flat surfaces. However, geometric synchronization is usually performed based on the assumption of planarity. This is believed to be the reason why processing small image patches can produce excellent results, namely, the non-planar effects of physical distortion are minimized.) Therefore, the reader should understand that references to 128×128 in connection with watermark reading operations are merely exemplary and smaller data sets are considered and generally preferred. (In contrast, watermark encoding can still be performed on the basis of the 128×128 block size, but decoding can extract the watermark payload from the analysis of the smaller image blocks. Alternatively, encoding can also be performed on a smaller block basis.)

[0195] The analysis blocks arranged across each image frame for watermark reading may be spaced evenly or randomly, or tiled edge-to-edge, or overlapped (eg, each block overlaps its immediate neighbors by 20%-80%). Fig.15 An exemplary block pattern is shown, where a 1280x1024 image frame is analyzed using 96x96 pixel blocks, with each block overlapping its neighbor by 25%. Some blocks are shown in bold because the tiling pattern makes the boundaries of individual blocks unclear.

[0196] In some embodiments, if a watermark reference signal or other clue is found in one of the analysis blocks (e.g., as described in detail below), a denser cluster of analysis blocks in the same neighboring block is checked for the reference signal, and if successful, the payload data is then analyzed. Fig.16 An example is shown. The original block is shown in bold in the center. More blocks are arranged around it with 75% overlap (block positions analyzed in the original block position array are omitted). Also, for clarity, some blocks are shown in bold dashed lines. Fig.17 exist Fig.15 This region of more densely placed blocks in context is shown in the frame of - located where a watermark reference signal or other clue is found in the initially examined block.

[0197] In some recycling systems, there are places where the conveyor belt is empty, without any items in the camera's field of view. Clues as to whether such gaps exist can be detected, allowing processing resources to be applied to more promising images. Similarly, watermarking of a captured image can be triggered only if a quick assessment of the image reveals clues that indicate plastic may be present (or that something other than a conveyor belt is depicted).

[0198] Plastics are often characterized by areas of specular reflection or glare, as the plastic surface reflects incident illumination specularly to the camera. This glare can be perceived and can serve as a cue to activate (trigger) the watermarking process. For example, multiple blocks in a sequence of input image frames (e.g., 150 frames per second) can each be analyzed for a 2×2 pixel area where the pixel size is within the top 5%, 10%, or 20% output of the sensor (or within a similar percentile of the historical sensed pixels of previous blocks depicting that area of ​​the conveyor belt). Any frame that meets this criteria is analyzed for watermark data. (Ideally, more than just the image segments near the glare are analyzed, as the plastic may extend well beyond that point.)

[0199] In one particular embodiment, no part of the frame is processed until a glare pixel is detected. When this event occurs, a full frame analysis is not triggered. Instead, a 7×7 array of overlapping pixel blocks is located based on the glare point, and each of these blocks is analyzed for the presence of a watermark reference signal. Blocks may overlap by 50% or more of their width, i.e., more than usual block overlap. Fig.18 An example is shown where the blocks overlap by 75% of their width. Glare spots are identified by a "+" sign in the middle of densely overlapping blocks. Also, some blocks are specifically identified by thick dashed lines because the boundaries of the component blocks are not clear.

[0200] Additionally or alternatively, metrics other than glare may be used to determine that an image may be worthy of watermarking.

[0201] One approach, called the block-triggered method, provides a clue that helps distinguish between empty and non-empty portions of the conveyor belt based on a comparison of incoming pixel values ​​to historical nominal values.

[0202] A specific block triggering algorithm compiles the sub-blocks within the analysis block (such as Fig.15 ). Each block can be 96×96 watermark elements (pixels, scaled to 1). The block is logically divided into a 4×4 array of sub-blocks, with 24 watermark elements on a side of each sub-block (i.e., 16 sub-blocks per block, each containing 24^2 or 576 pixels). The values ​​of the randomly selected but static 25 pixels from each sub-block are averaged together to produce one average pixel value per sub-block (i.e., having a value between 0 and 255 in 8-bit grayscale). A new such sub-block average pixel value is produced for each frame.

[0203] 256 of these average pixel values ​​for a particular sub-block are finally compiled into a histogram (i.e. over 256 frames). These values ​​show a tight peak corresponding to the average pixel value of an empty conveyor belt, at the conveyor belt position corresponding to that particular sub-block (and its particular illumination).

[0204] When a new frame is captured, the values ​​of the 16 sub-blocks within the block are calculated again. Each value is judged against the histogram of the block. If the new value is within a few digits (e.g., 1, 2, 3, or 4) of the pixel values ​​for which the histogram shows a peak, then this counts as a vote for the conclusion that the sub-block image depicts an empty conveyor belt. The resulting 16 votes are calculated for the 16 sub-blocks of the block. If a threshold number of votes (e.g., 11 out of 16) leads to the conclusion that the sub-block image depicts an empty conveyor belt, then the block is inferred to depict an empty conveyor belt. In this case, analysis of the block is skipped. Otherwise, the watermark data of the block is analyzed.

[0205] For each frame, all blocks within the camera view (e.g. Fig.15 This process is performed for all blocks shown in .

[0206] (If the image is captured at a finer resolution, i.e. more than 1 pixel per watermark element, the 25 values ​​for each sub-block may be determined by subsampling, e.g. by averaging the values ​​of four or nine finer pixels close to the 25 static positions. Alternatively, the value of a single fine pixel closest to each of the 25 static positions may be taken.)

[0207] Fig.19 An illustrative histogram of an exemplary sub-block after 219 frames have been processed is shown. The x-axis shows the average pixel value of the different frames, calculated for the sub-block. The y-axis shows the count of frames ("bin counts") for which the sub-block has different average pixel values. The histogram peaks at 20. In the relevant section, the relevant bin counts for different average pixel values ​​are as follows:

[0208]

[0209] When the next image frame is captured, if the average value calculated from the 25 static pixel positions in that sub-block is equal to 18, 19, 20, 21 or 22 (i.e. a peak value of 20, + / - 2), then the sub-block is considered to depict an empty conveyor belt. If another 10 of the 16 sub-blocks of the block agree, then this is taken as a clue that the block depicts an empty conveyor belt. Therefore, no watermarking is performed on the block. Alternatively, if there is no such agreement, then this serves as a clue that the block may depict a plastic item, and further processing is triggered.

[0210] Each histogram is kept fresh by periodically discarding data to make room for more data. For example, when the frame counter associated with the histogram indicates that 256 frames have been processed and 256 average values ​​of the sub-blocks fill the histogram, the histogram contents will be refined by half, to 128 values. This can be done by dividing the bin count for each average pixel value in the histogram by two (rounding down). The frame counter is reset accordingly, i.e., to 128 frames. The average pixel value count for 128 frames is then recorded in the histogram, at which point the refinement is repeated. With this arrangement, the importance of historical pixel values ​​decays exponentially, allowing the histogram to reflect recent data.

[0211] For each newly captured frame, the block triggering method provides clues for each block position in the frame as to whether to trigger a watermark reading operation. The average pixel value derived from the newly captured frame is used to update the corresponding histogram for evaluating blocks in subsequent image frames.

[0212] (It will be appreciated that if enough sub-blocks have average pixel values ​​above (brighter) and / or below (darker) the histogram peak (i.e., the nearest carousel brightness), then a watermark read may be triggered. That is, a plastic object may include areas of dark pixels as well as areas of light pixels. Both help inform the triggering decision.)

[0213] Related art techniques perform a similar process, but are based on the statistics of color distribution rather than brightness distribution.

[0214] In one particular implementation of the block triggering algorithm, if further analysis of a block is triggered and the analysis detects a watermark reference signal (or decodes a watermark payload) from the block, the average sub-block pixel value data for the block is Won't to their respective histograms (or, if added earlier, remove such counts). This way the histograms are not affected by known Won't Depicting the effect of image data of an empty conveyor belt.

[0215] Many recycling systems place a limit on the number of image blocks that can be analyzed during each frame of processing - the processing budget. For example, the limit may be 200 blocks. A small fraction of this total, such as 50-75 blocks, may be reserved for analyzing blocks that are densely placed around any block from which a watermark reference signal or other clue is detected (e.g., as described above in conjunction with Figure 15-17 If clues are detected from multiple blocks such that dense placement of further analysis blocks would exceed the 200 block limit, more blocks may be allocated based on the value of the clues (e.g., the strength of the detected watermark reference signal), where those blocks appear most promising for obtaining the largest allocation of neighboring analysis blocks.

[0216] If all 200 blocks aren't being analyzed every frame, power consumption drops, and so does the heat output of one or more computer processors (which normally must be offset by air conditioning).

[0217] In a variant block-triggered approach, the full processing budget (e.g., 150 block analyses) is spent per frame. Some block analyses are triggered as described above, i.e., because 11 of the 16 sub-blocks (or more generally, K of the L sub-blocks) have average pixel values ​​within a few digits of their respective histogram peaks (over 25, or more generally, N selected pixels). Any remaining analysis blocks are then allocated according to the difference between the indicated average sub-block pixel value and the peak of the respective histogram summed over all 16 sub-blocks of the block. Those blocks with the smallest aggregate difference are triggered for watermark analysis until the full budget of 150 analysis blocks is reached.

[0218] Some systems automatically or manually enable this variant when the material load on the conveyor exceeds a certain threshold. In extreme cases, the conveyor may be almost completely obscured - over an interval of hundreds of consecutive frames - by covering objects. In this case, the prominent peak associated with the background conveyor brightness will not appear from the histogram. But each histogram will still have a peak somewhere. This variant of the block triggering method will use the above process to allocate the entire budget of analysis blocks to the image frame. In practice, this will result in a large number of randomly selected blocks for analysis. But since the conveyor belt is obviously crowded with objects, this is not an unreasonable block selection strategy.

[0219] Other clues for identifying images that may be worthy of watermarking employ image statistics, such as the mean, standard deviation, and / or variance.

[0220] Fig. 20 The image frame field of view is shown across the conveyor belt by the large rectangle. The dashed rectangle shows the position of the linear LED light source also across the conveyor belt. Due to the orientation of the light source or its lens (or reflector), the illumination has a spatial intensity distribution, as shown in the graph immediately to the left, which shows a maximum intensity in the area of ​​the light, falling off sharply in one direction and less sharply in the other (on a scale of 0-100).

[0221] Arranged along the conveyor belt's travel direction are multiple image patches, each stripe is, for example, 128 x 128 pixels in size. Only two rows are shown in the figure, but similar stripes span the image frame. Due to the illumination distribution of the lamp, adjacent stripes are usually illuminated in different ways.

[0222] (Although Fig. 20 The blocks in are not adjacent and do not overlap, but this is for clarity of illustration. In actual practice, the blocks are usually adjacent or overlapping.)

[0223] A metric is derived from each of these blocks in each strip and used as a clue to determine the similarity of the image block to an image depicting an empty conveyor belt.

[0224] In one illustrative embodiment, a feature value f is calculated for each block and used to identify areas that are worthy of further watermark analysis. Typically, f(·) is a function of each pixel in the block. An initialization phase is performed while the conveyor is moving but empty, for example, when the sorting system is initially powered on. The feature value f is calculated for each block over multiple frames, and these values ​​are grouped by stripes. For each stripe, the overall mean and standard deviation are estimated from the corresponding set of sample feature values ​​obtained over multiple frames.

[0225] When a new image frame is captured thereafter, an eigenvalue is calculated for each block in the new frame. For each eigenvalue, a normalized eigenvalue is calculated using the previously estimated mean and standard deviation values ​​of the strip containing the block for which the eigenvalue was calculated. The normalized eigenvalue is calculated as

[0226]

[0227] where μ and σ are the estimated mean and standard deviation, respectively. If the eigenvalues ​​have a Gaussian or generalized Gaussian distribution for the region containing the carousel, then the normalized features will have a monotonically decreasing probability distribution; smaller values ​​are more likely than larger values. These normalized eigenvalues ​​produce a first measure that indicates how similar (or different) the new block is to the empty carousel data in the region. Smaller values ​​indicate more similarity to the empty carousel data. Larger values ​​indicate more dissimilarities. The largest values ​​indicate the blocks that are most suitable for watermark reading because they are the least similar to the empty carousel.

[0228] The normalized feature metric values ​​are sorted, from largest to smallest, to identify the blocks with the best foreground depicting the watermark object. This establishes the priority order for watermark reading. If the system processing budget allows 150 blocks to be analyzed per frame, data from the 150 blocks with the highest first metric are routed for watermark processing.

[0229] Different underlying features f may be used, producing different effects in the watermarking process. Illustrative embodiments include block mean and block standard deviation.

[0230] The effectiveness of a particular feature in distinguishing patches containing only carousel pixels from others depends on the conditional distribution of the features of these two types of patches. For some non-carousel patches, the feature f A may not be used to distinguish between blocks and conveyor blocks, and feature f B The blocks can be distinguished well. For other non-conveyor blocks, the situation may be the opposite, f A This leads to another class of embodiments using multiple features.

[0231] In a plurality of feature embodiments, an independent set of mean and standard deviation estimates are calculated for each feature during an initialization phase, and a corresponding normalized feature value is calculated for each feature of each block in a new image frame. The normalized feature values ​​are combined into a single metric value using a combining function. The resulting combined metric values ​​are sorted, and the sorted list of metric values ​​forms a priority list for watermark processing.

[0232] An example of a combining function is the sum of normalized eigenvalues. Other embodiments include more complex functions, derived, for example, from a statistical analysis of the normalized eigenvalue distributions for both carousel blocks and non-carousel blocks. In some implementations, a polynomial equation for combining the eigenvalues ​​is used. Additional embodiments may have a combining function that varies with image strips to take advantage of the fact that different image strips may produce different normalized eigenvalue distributions.

[0233] It should be appreciated that the arrangement just described always fully utilizes the entire system processing budget. If the system budget allows 150 blocks to be analyzed per frame, then 150 blocks are analyzed per frame. (As before, there may be a reserve budget for additional blocks that can be allocated based on the processing results of the first 150 blocks.)

[0234] The foregoing arrangement can be viewed as a kind of classifier, ie classifying whether an image is likely to depict a conveyor belt (or glare from plastic). Many other kinds of classifiers can be used to provide clues to control the watermarking process.

[0235] One such alternative employs a neural network that is trained to classify an image frame (or portion thereof) as (a) depicting only a conveyor belt, or (b) not just a conveyor belt, by training the network with a large number of labeled images that variously display one class or the other. Suitable networks and training methods are described in detail in patent publications 20160063359, 20170243085, and 20190019050, and in Krizhevsky et al., "Imagenet classification with deep convolutional neural networks," Advances in Neural Information Processing Systems 2012, pp. 1097-1105. Additional information is described in detail in co-pending application Ser. No. 15 / 726,290, filed Oct. 5, 2017.

[0236] If it is determined that an image or image patch likely depicts only a conveyor belt, no further analysis is performed on such images. (Instead, the freed-up processor cycles can be used to further process other images, e.g., by analyzing additional blocks, by attempting decoding using different candidate affine transforms, etc.)

[0237] The different arrangements provide clues for classifying images depicting an empty conveyor belt and distinguishing such images from other images by sensing characteristic conveyor belt markings. For example, conveyor belts often have scars, stains, and other stripe patterns that are elongated along the axis of conveyor belt travel (conveyor belt direction). Such markings detected in the images are primarily low frequency. The captured images can be low pass filtered to reduce high frequency noise, and the resulting images can then be analyzed to assess edge strength in different directions (e.g., by Canny or Sobel algorithms).

[0238] In one particular embodiment, a 128×128 image patch is low-pass filtered and then examined with a Canny edge detector to evaluate the strength of the gradient running along the conveyor belt direction, as well as the strength of the gradient running across the conveyor belt direction (e.g., by summing the gradient values ​​in the vertical and horizontal image directions). If the patch depicts a conveyor belt, the sum of the previous gradients will be significantly greater than the sum of the subsequent gradients. A logistic regressor is trained to respond to the two intensity values ​​by classifying the image patch as depicting a conveyor belt or not depicting a conveyor belt. If it depicts a conveyor belt, no further analysis is performed on such a patch; if it does not depict a conveyor belt, further watermark analysis can be initiated for that patch.

[0239] In other embodiments, a simple ratio between two accumulated gradient metrics is calculated and this value is compared to a threshold to determine whether the image patch depicts a conveyor belt.

[0240] Sorting machines are sometimes equipped with laser systems to detect the presence of objects. For example, a laser beam can be swept over a broad area of ​​a conveyor using a rotating mirror arrangement and used to trigger detection by elements of a linear photodetector array along the other side. As long as each photodetector detects the laser beam, it is known that the conveyor over the swept area is empty. This check can be used to suppress analysis of captured image blocks.

[0241] Another type of clue that can trigger further watermark analysis is based on a pepper-in-salt (or salt-in-pepper) pattern metric that indicates the likelihood that a block depicts a sparse point watermark. An exemplary algorithm for computing such a metric is described next.

[0242] If necessary, the input image patch is downsampled so that it has a scale of 1. That is, each watermark element is depicted with a size of one pixel. We are looking for dark pixels in the light field - pixel outliers. However, image contrast may be large or small, and the illumination across the patch may vary. Ideally, the computed metric should be robust to these variables. To this end, we compute a metric that examines the pixel neighborhood and also takes into account sensor acquisition noise.

[0243] The acquisition noise present in the captured image is a function of the pixel value, with higher pixel values ​​having higher noise values. A polynomial function or reference table may provide a noise standard deviation value between 0 and 255 for each pixel value. In order to identify pixels with outliers (e.g., the relatively darkest pixel in a region of relatively bright pixels), the metric sigma is calculated for a neighborhood of pixels around a subject pixel at coordinates (i, j) and having a value x by the following equation:

[0244]

[0245] Where the N-bar term is the average pixel value over the neighborhood, and the S term is the standard deviation of the neighborhood acquisition noise, based on a polynomial or reference table data. (The optimal neighborhood size can be determined heuristically. For a 128×128 block size, the neighborhood can be as small as 4×4, or as large as 64×64. An 8×8 neighborhood is an example. Alternatively, the neighborhood can be temporal, including the value of a single pixel at coordinate (i, j) in a series of previously captured frames (e.g., tens or hundreds)).

[0246] For pixels that are darker than the neighborhood average, the sigma value is negative. To treat them as sparsely marked points, we set the sigma value to satisfy, such that σ ij A darkness threshold of < -3 is used. We generate a filtered image patch that contains only those pixels whose corresponding sigma values ​​satisfy this test. All other pixels are removed (e.g., set to white, pixel value = 255).

[0247] So far, the process identifies the darkest points, but in doing so also includes pixels that form the darkest edges (e.g., including dark text). In order to focus our watermark extraction efforts only on sparsely marked points, we want to filter out pixels that are not separated from other pixels (a form of morphological filtering). A variety of techniques can be used for this task. A simple technique is to visit each dark pixel, check a 5×5 pixel region centered on that image location, and count the number of dark pixels in that region. If there are more than two dark pixels in the 5×5 region, then remove the center pixel (e.g., change it to white). The resulting processed block then consists entirely of isolated dark points.

[0248] Finally, this processed block is checked to count the number of dark spots remaining within the block boundaries. This count is used as a metric to indicate the likelihood that the block contains a sparse watermark.

[0249] This metric can be compared to see if it exceeds an empirically determined threshold K (e.g., K=500) to identify frames that may depict sparse watermark data. Alternatively, blocks within a frame can be sorted based on their associated sparsity metric, and then those blocks with the highest sparsity metric can be further analyzed for watermark data, up to the block processing budget limit.

[0250] Various simplifications and modifications can be made to this particular algorithm. For example, a simpler process simply identifies a set of the darkest pixels in a block. (For example, the darkest 10% or 30% of pixels in a block can be identified.) The process then applies the above-described morphological filtering and counting operations to produce a sparse metric.

[0251] Another variant arrangement distinguishes potential sparse points from non-sparse points by learning collected from previous image frames.

[0252] For a series of past frames, the exemplary learning process analyzes sampled pixel values ​​from, for example, 10 blocks in each frame. Each block is divided into sub-blocks, for example, 5×8 pixel sub-blocks. For each analyzed sub-block, the average pixel value and the minimum pixel value are determined.

[0253] In some sub-blocks, the minimum pixel value will be the pixel value of the dark sparse point. Other sub-blocks do not include sparse points, so the minimum pixel value will simply be the lowest value pixel among the non-sparse points in the image content (e.g., background image, items marked with continuous tone watermarks instead of sparse watermarks, etc.).

[0254] From these collected statistics, we identify the maximum value among the minimum pixel values ​​of each associated sub-block average (“max-min”). For example, if we consider all sub-blocks with an average pixel value of 151, we may find that the maximum value among the minimum pixel values ​​encountered in the series of frames analyzed is 145. In the sub-block with an average value of 151, any pixel value greater than 145 is definitely not a sparse marker point. Therefore, this value and other similar observed values ​​can help establish a threshold to distinguish potential sparse marker points (outliers) from non-potential sparse points.

[0255] In one particular embodiment, we describe the set of all these points by a best fit line characterized by a slope and an offset, such as:

[0256] τ Outlier =0.96*μ-1.6

[0257] Where μ is the average pixel value of the sub-block.

[0258] Thereafter, when a new image frame is received, we calculate the mean value for each 5×8 pixel subblock and determine the applicable anomaly threshold via the best fit line equation. Any pixel in the subblock having a value less than this threshold is identified as a candidate sparse point. (For example, if the mean pixel value of a subblock is 82, we consider all pixels in that subblock with pixel values ​​of 77 and below as candidate sparse points.) We then apply a morphological filter on the block to discard connected points, as previously described, and then count the number of points remaining in the block, producing a sparse metric. As before, this metric can be tested against a threshold to identify blocks worthy of watermarking. Alternatively, all blocks in the frame can be sorted according to this metric and selected for processing on this basis until the block processing budget is reached.

[0259] In an alternative embodiment, the detailed process described above can be altered to generate metrics based on light pixels in a dark field (ie, salt in pepper). One such alteration is to simply invert the image block, replacing light with dark, before executing one of the algorithms described above.

[0260] Some recycling systems may look for multiple clues to decide which blocks to watermark. For example, block trigger clues may be first obtained for all blocks in the image frame to identify blocks that depict only conveyor belts. Each remaining block may then be evaluated to determine a sparsity metric, as just described, to assess which non-conveyor belt depicting blocks are most promising for watermark analysis.

[0261] As items are moved by the recycling system conveyor, they pass one or more cameras in a linear fashion, entering the field of view from one side of the camera sensor and exiting from the other side. Once a clue indicating an image block depicting a non-empty conveyor is sensed in one frame, for example, as described above, the image can be analyzed not only in the current frame, but also in the following N frames depicting successively displaced areas in the camera's field of view. N is a function of the camera frame rate, the conveyor speed, and the span of the camera's field of view. For example, if the camera has a field of view of 15 inches and the conveyor moves at 10 feet per second, then any item on the conveyor should be visible within one-eighth of a second as it passes through the camera's field of view. If the camera captures 60 frames per second, then N can be set to 6 (i.e., analyzing corresponding blocks in a total of seven frames).

[0262] In one particular embodiment, an array of overlapping analysis blocks is positioned along the side of the camera's field of view where the object first enters, and each of these blocks is analyzed for a watermark reference signal in each frame. If a reference signal or other clue is found within any of these blocks, this detection results in further analysis of a cluster of overlapping blocks centered around the detection block as previously described. This cluster advances from frame to frame in the field of view, depending on the speed of the conveyor belt.

[0263] Figures 21A-21D This arrangement is illustrated. A row of blocks is analyzed on the input side of the article in each image frame. (Blocks inside the frame are not typically analyzed.) A watermark reference signal or other clue is discerned in one of these edge blocks (shown in bold), in which case a cluster of overlapping blocks in that vicinity may be analyzed for the watermark reference signal. If a watermark reference signal is detected, analysis continues with an attempt to recover the watermark payload from the watermark element data obtained using affine parameters extracted from the reference signal. In successive frames, corresponding clusters of blocks are analyzed at successive locations until the detected object is no longer within the camera's field of view.

[0264] If a block in the forward cluster detects a watermark reference signal or other clue (e.g. Fig. 21C If the original block is a bold block in the image, a complementary cluster of analysis blocks (shown in dashed lines) centered on the detection block can be generated. This complementary cluster of blocks can also advance in the field of view of the original cluster in synchronization with the conveyor belt movement. At the same time, the original block belt arranged on the entrance side of the camera field of view continues to check each new image frame for watermark reference signals or other clues.

[0265] optimization

[0266] As mentioned above, the conveyor belt used to mark / sort plastic items moves at a relatively high speed. In order to ensure sufficient lighting and depth of field, a small aperture and a long exposure time are required. This may cause motion blur.

[0267] Some embodiments of the technique apply deblurring to the captured image before attempting to read the watermark. Different techniques can be used, including deconvolution by inverse filtering, by Wiener filtering, or by the Richardson-Lucy algorithm. A suitable point spread function (PSF) can be estimated for 1D motion using classical methods. (The PSF essentially characterizes the amount of energy that light from a single point in the scene exposes each pixel of the camera during static exposure.)

[0268] More sophisticated approaches can be used, for example, using a trembling shutter technique to sample the scene at different times and intervals, and using the resulting images to derive a more accurate estimate of the unblurred scene. (See, for example, patent publication US20090277962.)

[0269] In one illustrative embodiment, deblurring is performed in the Fourier domain, where the Fourier transform of the image is divided by the Fourier transform of the blur kernel. In other embodiments, such operations may be performed in the spatial (pixel) domain.

[0270] In the convolutional decoding of the watermark payload, list decoding can be employed. Instead of outputting a single decoded payload, list decoding outputs a list of possibilities, of which one is correct. This allows handling of more errors than unique decoding would allow. The listed multiple payloads can then be evaluated using CRC data or constraints within the payload itself (e.g., the value of a field of known data is only drawn from a subset of possible values) to identify the correctly decoded payload.

[0271] As described above, rather than attempting to characterize the pose of a 128×128 patch of watermark elements, a smaller patch, such as 96×96 watermark elements, is analyzed. (If, as in the preferred embodiment, the camera sensor, lens, and imaging distance are selected so as to depict the watermarked object in a ratio where each pixel corresponds approximately to the area of ​​a single watermark element, then a 96×96 watermark element patch corresponds to a 96×96 pixel patch.) An FFT of size 128×128 is performed on this patch, either by zero padding or by processing the adjacent pixel image using a square or Gaussian window to focus on the central area. As described above, the methods detailed in patents US9,959,587 and US10,242,434 are used to characterize rotation and scale. The translation can then be determined using the phase deviation method of the '587 patent. In general, the rotation, scale, and translation (affine parameters) describe how the original watermark appears in the captured image.

[0272] The phase deviation method provides a metric indicating the strength of the reference signals detected within the analyzed image patch, i.e., the sum of the phase deviations between the measured and expected phases of each reference signal. If this phase deviation metric is below a threshold (the lower the metric, the better), it is concluded that the image patch contains a readable watermark. An interpolation operation is then performed - the image is sampled at points corresponding to the locations of the watermark elements, guided by the identified affine parameters, to generate data for payload decoding.

[0273] As described above, if an image patch is determined to contain a readable watermark, adjacent patches are checked to determine whether they also contain readable watermarks, for example, using the process just described. For each such patch, a corresponding set of affine parameters is determined. (Typically, each patch is characterized by a different set of affine parameters.) Interpolation is again performed to generate more watermark element data for payload decoding.

[0274] As before, adjacent patches can be edge-adjacent, or can overlap by any number of watermark elements.

[0275] If image patches smaller than 128×128 are analyzed (e.g., 96×96 or 64×64 watermark elements), not all 128×128 watermark element encoding positions may be depicted in each patch (depending on the scaling). Nevertheless, corresponding positions are identified between the analyzed patches (using affine parameters) and their sampled watermark element data are combined (e.g., averaged or summed). A combined count of watermark element data for some or all of the 128×128 encoding positions is thereby generated and provided to a Viterbi decoder for use in extracting the encoded watermark payload.

[0276] This is Fig. 22 Schematically shown in . A reference signal is detected in a small image patch 141 (shown here as only an 8×8 watermark element), which leads to searching and finding the reference signal in adjacent small image patches 142 and 143. Each has a different affine pose. The watermark signal block (not specifically shown) spans a larger area than any patch.

[0277] For some watermark elements in the watermark signal block, such as watermark element 144, interpolated data from a single image patch is provided to the decoder. For other watermark elements, such as watermark element 145, interpolated data may be obtained from each of two overlapping patches. The two interpolated values ​​are averaged (or summed) and provided to the decoder. For other watermark elements, such as watermark element 146, data from three patches are averaged (summed) and provided to the decoder. For other watermark elements, such as watermark element 147, no data is available to the decoder.

[0278] Sometimes the data for a particular watermark element may be obtained from two different (but usually adjacent) 128x128 watermark element watermark blocks. Fig.23 Two such blocks are shown in solid lines. Also shown are two 96×96 watermark element patches, shown in dashed lines, which are processed as described above. From the affine pose parameters determined for these patches, it is known that the watermark element represented by the circle in the left patch corresponds spatially to the watermark element represented by the circle in the right patch. Both convey the same signature information chip. In this case, the two watermark element values ​​are summed for presentation to the decoder.

[0279] The decoder processes any available data and generates an extracted payload (or a list of candidate payloads).

[0280] In some embodiments, the watermark element data contributed by each image patch is weighted according to a strength measure of an associated reference signal. In other embodiments, different measures may be employed, for example, as detailed in US Pat. No. 10,506,128 (where they are referred to as reference pattern strength and linear reference pattern strength). Alternatively, each watermark element data may be weighted according to a corresponding message strength factor, as detailed in US Pat. No. 7,286,685.

[0281] The just described accumulation of watermark element data from multiple patches in an image frame may be referred to as intra-frame signature combination. Additionally or alternatively, accumulation of watermark element data from the same or corresponding watermark element positions of patches depicted in different image frames may be used, which may be referred to as inter-frame signature combination.

[0282] Once the affine parameters of the patch (describing the appearance of the watermark in the patch) are known, the payload reading can be done by payload correlation techniques instead of Viterbi decoding. This is particularly useful if the number of different payloads is small, such as tens or hundreds. This may be the case if the only payload of interest is plastic type data, and there are a limited number of plastic types that may be encountered.

[0283] In one particular arrangement, a set of templates are generated, each expressing a watermark element encoding associated with a particular type of plastic. Watermark elements that are common to all plastic types (or a large portion, such as 30%) may be omitted from the templates to reduce the chance of confusion. Image data is correlated with the different templates to identify a pattern that most strongly corresponds to it. Since the image has been determined to include a reference signal (e.g., that of a plastic texture watermark), then one of a limited number of watermark element patterns should be present, making correlation a high reliability method of discerning the payload.

[0284] Plastic bottles are increasingly no longer printed directly, but are wrapped in a plastic sleeve that is printed and heat shrunk to conform to the shape of the bottle. This presents difficulties because heat shrink materials typically shrink primarily in one direction (the circumferential direction). Any watermark pattern printed on such a sleeve is then scaled differently by the heat shrink, thereby hindering watermark reading.

[0285] To address this problem, one or more "seed" linear transforms (described in detail in patents US9,959,587 and US10,242,434) that serve as starting points for an iterative search to determine the affine transform of the watermark are initialized to include a differential scale component. This allows the iterative process to more quickly reach a better estimate of the affine distortion when detecting the watermark from a heat-shrinkable plastic sleeve.

[0286] Sometimes the watermark will not be read from the marked item and it will travel on the conveyor without being identified as a type. Meanwhile other items will pop off the conveyor, for example into a bin of ABS, HDPE, PET, PETg, etc. leaving only the unidentified item.

[0287] These unidentified items may be collected into their own bin and reprocessed later. Read failures are unusual and are usually corrected by changing the pose in which the item is presented to the lighting and camera. By collecting these items and reprocessing them, they will be presented in a different pose the second time and will then likely be identified.

[0288] Alternatively, instead of being collected and re-handled, the item may tumble (e.g., fall from one conveyor onto another), or collide / drag (e.g., a conveyor may pass through a curtain of hanging obstacles) to change the item's posture, and a second camera / lighting system can then collect additional images for analysis.

[0289] In some embodiments, the captured image is a function of light transmission through the article, rather than simply light reflection from the article. Fig.24 Schematically illustrates an arrangement in which items briefly "fly" from one conveyor belt to another, presenting themselves to one or more light sources on the other side. Pass through Camera view of the item. This approach can be used with any of the camera / lighting systems described previously. "Clues" based on conveyor belt detection can also be based on zero background detection of such flyovers.

[0290] A direct least squares method for determining scale and rotation transforms to characterize the appearance of the watermark in the image operates by successively screening and refining a large number of candidate transforms until only one remains. This is followed by a referenced phase deviation process, which establishes the x and y translations of the watermark pattern in the image. In certain embodiments of the present technology, the direct least squares method does not reduce the candidate transforms to one; instead, two or more top candidates are output. The phase deviation process is applied to each candidate, resulting in a plurality of candidate affine poses. The best pose is selected that produces the minimum sum of phase deviations between the measured phase and the expected phase for each reference signal. With this arrangement, fewer items are not read the first time through the device, thereby minimizing the need for reprocessing.

[0291] Sometimes, the pattern of reflected light from patches of textured surfaces—particularly clear plastic bottles—can appear inverted: dark is light, and light is dark. Additionally, when reading a clear, textured surface from the underside, the pattern can appear flipped (mirrored). Therefore, after establishing scale and rotation (whether through direct least squares, correlation, or some other method), multiple versions of the image are submitted for analysis through a process that determines the x and y translations (whether through phase deviation, or correlation). One version is inverted—white is black (light is dark). Another version is mirrored (right is left). And another is the original image. Of these, only one version will be correctly synchronized with the known phase characteristics of the reference signal peak that determines the translation; no match will be found for the other version. Again, this measure helps maximize the number of plastic items that are read on the first pass through the equipment, thereby minimizing the need for reprocessing.

[0292] (In some embodiments, the decoder is configured to test the polarity of the message chips encoded in each sub-block (e.g., 32 by 32 watermark elements, 16 sub-blocks per block) to assess whether the message chips are inverted. Such a decoder performs the test by correlating the watermark signal of the sub-block to check whether it has a positive correlation peak or a negative correlation peak. A negative peak indicates that the signal is inverted, and the decoder inverts the chips from such inverted sub-blocks before aggregating them with the chips of the other sub-blocks. The correlation may be performed with a known or fixed portion of the watermark signal.)

[0293] The applicant has discovered that it is sometimes advantageous to capture frames with different imaging parameters - each imaging parameter depicting a common area of ​​the conveyor belt. For example, a single camera may alternate between shorter and longer exposure intervals, such as 20 and 100 microseconds, in consecutive frames. Or two cameras may capture images of a common area of ​​the conveyor belt, one camera having a relatively large aperture (e.g., f / 4) and the other camera having a relatively small aperture (e.g., f / 8). Or different exposure intervals may be used. The resulting variation in the captured images helps to ensure that small changes associated with the watermark encoding are easily detected despite the wide range of brightness reflected from the items being imaged.

[0294] When a plastic material is molded, a first surface of the material is typically adjacent to the forming mold surface, while an opposing second surface is not. This opposing surface may nonetheless be formed, such as when a vacuum draws the first surface of the material into the mold and the second surface follows. But the second surface is not as well defined physically; it lacks high-frequency detail. However, the second surface may be the surface that is imaged by the camera (for example, as may occur on a carbon black plastic tray for packaging meat, which may be presented to the camera either from the top down or from the bottom up). To address this issue, some or all of the captured frames (or segments) may be processed to highlight the high-frequency detail.

[0295] In one exemplary embodiment, if no reference signal is found in the analyzed block, the block is processed by an unsharp mask filter and the analysis is repeated, with the expectation that such processing will aid in detecting the watermark reference signal depicted from the back of the molded plastic. In another exemplary embodiment, if a reference signal is detected in the block, but payload extraction fails, the block is processed by an unsharp mask filter and the payload extraction operation is retried.

[0296] The exemplary lighting system is composed of circuit board modules 250, one of which is Fig.25 Each module is 10cm wide and is configured to hold 75 LEDs from the Cree XP-E2 series. The white LEDs in this series are rated to provide 220 to 280 lumens of light output at a drive current of 1A, so a module of 75 LEDs can produce an output flux of 16,000-21,000 lumens. These modules are designed with side-by-side use in mind. For example, to span a 1-meter wide conveyor belt, 10 or more such modules can be used, indicating a total light output of 160,000-200,000 lumens or more.

[0297] The circuit board module is configured to mount the LEDs in a triplet, for example, by three groups of closest pads 252a, 252b, 252c. Each such triplet LED is adapted to receive a lens assembly 254 to focus the light output onto the imaging area of ​​the conveyor belt. Ideally, the lens has an elliptical output, with the light spread more in one dimension than in the vertical dimension. A suitable lens assembly is Carclo Technical Plastics (UK) part number 10510, which focuses the output from the Cree LED into a half-height full-width beam having a 45 degree by 16 degree angle. The wide dimension is oriented along the width of the conveyor belt, while the narrow dimension is oriented along the length (direction of travel) of the conveyor belt. The latter measurement is typically selected based on the distance between the LED module and the belt and the extent of the imaging field of view along the length of the conveyor belt.

[0298] The more intense the light, the shorter the exposure interval (greater depth of field). If the exposure interval is 100 microseconds, and frames are captured at a rate of 150 frames per second, the camera sensor collects only 0.015 seconds of gathered light per second. If the lighting system is only running during the interval when the camera is capturing the exposure (strobing), the lighting system is running at a 1.5% duty cycle. In this case, the LEDs can be operated at drive currents much higher than the nominal specified value of 1A. For example, a drive current of 3A can be used. By doing so, the light output is further increased, for example to the order of 300,000 lumens per meter of bandwidth. (It should be recognized that lumens is a measure based on the sensitivity of the human visual system. Generally, it is more useful in machine vision to measure illuminance in watts. Lumens are used as a measure simply because it is more familiar to some people.)

[0299] The light output of LEDs decreases as temperature rises. Therefore, it is desirable to keep the LEDs relatively cool. To help achieve this goal, the circuit board module can have an aluminum or copper base plate, and the module can be thermally bonded to an aluminum or copper heat sink using a suitable thermally conductive paste. The heat sink can be finned to increase its surface area and increase passive heat transfer to the ambient air. Alternatively or additionally, the heat sink can be cooled by forced air or forced water flow.

[0300] In some embodiments, the LEDs are all white. In other cases, the LEDs are all red (e.g., Cree part number XPEBPR-L1-0000-00D01, which has a peak emission between 650 and 670 nm). In yet another embodiment, module 250 includes LEDs with different spectra. Control circuitry is then provided to drive the different color LEDs (sometimes different grades of LEDs) independently or in different combinations.

[0301] In one such embodiment, Fig.25 Each of the "triplets" shown in the figure includes a red LED, a green LED, and a blue LED. They are divided into three "levels" A, B, and C, as shown. The A-level red LEDs are switched in series, as are the A-level green LEDs and the A-level blue LEDs. Likewise, the same is true for levels B and C. Any set of LED colors and levels can be operated during the exposure interval - alone or in combination with any other set or sets of LED colors and one or more levels. This arrangement allows different image frames to be captured under different light spectra. For example, one frame can be captured using all-red illumination, while the next frame can be captured using green or red+green+blue (~white) illumination.

[0302] In another such embodiment, not all classes have the same type of LEDs. For example, class A and class C might have the red / green / blue LEDs just described, while class B might have only white LEDs, or only red LEDs.

[0303] In another arrangement, module 250 is equipped with up to nine different spectrums of LEDs. Class A can be occupied by LEDs with spectrum 1, spectrum 2, and spectrum 3. Class B can be occupied by LEDs with spectrum 4, spectrum 5, and spectrum 6. Class C can be occupied by LEDs with spectrum 7, spectrum 8, and spectrum 9. Among them, some of these spectra may be outside the visible light range, extending to ultraviolet or infrared wavelengths. This allows the acquisition of data that can identify an object by its spectral characteristics, as detailed in, for example, the applicant's "Spectrum ID" patent publication 20140293091 and the pending patent application 62 / 956,845 filed on January 2, 2020.

[0304] If the camera sensor is a color sensor, for example, a color filter array is overlaid on a monochrome sensor, then the different colored photodetectors can capture images at different wavelengths. If both the red and blue LEDs are powered during frame exposure, the red filtered photodetector will sense images near 660nm and the blue filtered photodetector will sense images near 465nm. Subtracting the blue image from the red produces an image in which certain encoded marks may be particularly easy to detect (e.g., due to color channels modulated in the printed label artwork to achieve the encoding). Likewise, in combination with other colors.

[0305] Because plastic surfaces can be glossy, specular reflections are not uncommon. That is, light from a given location may reflect primarily from a surface patch to a single location. Unless a camera is located at that location, the surface patch may be imaged as dark, making it difficult to analyze the encoded information. Therefore, it is desirable to illuminate the surface from multiple directions. An elongated light bar consisting of multiple modules 250 mounted side by side extending across the conveyor belt with a wide light dispersion (45 degrees, using the lens cited in the example above) across the conveyor belt helps achieve this spatial diversity. Diversity is further aided by having two or more such light bars illuminating the conveyor belt from different locations along its length.

[0306] Other embodiments employ optical diffuser arrangements, such as Fig.26A and 26B shown. Fig.26A A cross section of a generally cylindrical reflector 261 is shown, with its axis extending across the width of the conveyor belt. A linear array 262a of lighting modules, such as module 250 discussed above, extends along one edge of the reflector - pointing upward to illuminate the reflector surface. A similar array of lighting 262b also comes from the opposite edge of the reflector. Thus, for a one meter wide conveyor belt, illumination of over half a million lumens can be achieved.

[0307] The surface of the reflector 261 is typically white, but a mirrored or colored surface may also be used. A diffuser may be used on each lighting array 262a, 262b to disperse the illumination from the LEDs onto the reflector. Alternatively, a lens arrangement may be used, typically with a wider spread than the 16 degrees cited previously. For example, a spread of 90-120 degrees may be used to provide broad illumination of the reflector. Although the reflector 261 is shown in cross-section as a portion of a circle, a different shape may be used, customized to focus light from the two linear lighting arrays onto a light strip 263 extending across the conveyor belt to increase illumination of the area being imaged by the camera 264.

[0308] Fig.26B A cross section of an alternative arrangement is shown in which multiple linear arrays of LED modules 262c extend across the belt. These modules 262c differ from modules 250 in that they do not include lenses. Instead, the LEDs illuminate a plastic diffuser 266. A suitable diffuser is available from Curbell Plastics under the Optix brand. With four or more arrays of modules 262c extending across the conveyor belt, illumination of over 1 million lumens can be achieved for a one meter wide conveyor belt.

[0309] Additionally, or alternatively, the problem of specular reflections can be mitigated by using multiple cameras positioned not only across the width of the conveyor belt, but also at different locations along its length. Two or more such cameras can be oriented to capture images from a common focal area on the conveyor belt. Due to the different viewpoints, one camera system may successfully decode an identifier from an object on the conveyor belt while another camera that is imaging the same object fails.

[0310] In addition to capturing different views of an object to improve decoding reliability, using multiple cameras to view a common area can also extract 3D information about objects on the conveyor belt using familiar stereo principles. This provides further information that can identify the object.

[0311] Fig. 27 An arrangement using multiple light sources and multiple cameras along the length of a conveyor belt is shown. This is a schematic diagram. Ideally, a light source with a large aperture and a diffuser is used, as described above in conjunction with Fig.26B discussed and explained.

[0312] Fig.28A variant arrangement is shown in which a single camera is used but a portion (e.g., half) of its field of view is occupied by a different view of the conveyor, relayed by a system of mirrors (shown by the bold lines). The path length via the mirrors is twice the path length without the mirrors. Therefore, the resolution of the mirrored half of the field of view is half that of the direct view, and a higher resolution sensor is generally required. If the entire image is to be input to a common detection module, the direct view portion of the captured image can be downsampled to match the resolution of the mirrored view. Alternatively, the two halves of the image can be provided to two different detector modules, each optimized for a particular resolution of the half of the image it captures. In either case, if there are an odd number of mirrors in the path, it is desirable to watch out for mirrored reflections of the inverted image, or to analyze the reflected portion of the image in a manner that expects such reflections. (Again, the light sources described previously can be used - including dome reflectors and diffusers.)

[0313] Ideally, the illumination source is as close to the conveyor as possible to allow the shortest possible camera interval. However, sufficient clearance must be provided to allow items to pass under the conveyor. A suitable compromise is a distance of between 15-20cm. Depending on the type of items on the conveyor, a larger clearance may sometimes be required, up to 25cm.

[0314] As previously mentioned, specular reflections are sometimes an aid (e.g., sensing the texture encoding of black plastic) and sometimes a hindrance. One advantageous arrangement employs multiple individually operable light sources positioned relative to the camera in a configuration such that one (or more) is positioned to facilitate specular reflections and another (or more) is positioned to avoid specular reflections.

[0315] Fig.29 An exemplary embodiment is schematically illustrated in . Light source A is positioned and oriented so that specular reflections (arrows AA) from a horizontal surface 7 cm above the conveyor (e.g., the nominal position of the top surface of a plastic object) will be reflected to the lens of the camera (each angle of incidence = angle of reflection). In contrast, light source B is positioned and oriented so that specular reflections (arrows BB) from such a surface will not encounter the lens of the camera. Instead, the light from light source B sensed by the camera will be due to diffuse reflections. Light sources A and B are operated to illuminate different frame captures, thereby generating image frames optimized to depict specular reflections and diffuse reflections, respectively.

[0316] Ideally, light source B is positioned so that its specularly reflected light BB passes through a distance D of at least 10 cm, and preferably more than 15 or 20 cm, from the camera lens.

[0317] (Although Fig.29The specular reflection of light source A is shown entering the camera lens along its central axis, and therefore appears in the center of the captured image frame, but this is not required. All that is required is that the specular reflection from light source A is located somewhere within the camera's field of view. )

[0318] In another specific arrangement, light source A is at a 45 degree angle (eg Fig.29 As shown), while light source B is at a straight downward angle.

[0319] In some embodiments, the colors of light sources A and B are different. For example, the former may be any one of white, red, blue, ultraviolet and / or infrared, while the latter may be a different such color.

[0320] Accurate extraction of payload signature data from image patches is highly dependent on accurate spatial registration of the patches, i.e., accurate estimation of the affine pose of the patches so that the watermark element values ​​can be sampled from the exact originally encoded positions within the image. As noted elsewhere, the registration in the illustrative embodiments is performed using a reference (grid) signal consisting of a constellation of peaks in the spatial frequency (Fourier) domain.

[0321] As described above, registration accuracy can be assessed by a metric ("grid strength metric" or "linear reference pattern strength") that compares the Fourier amplitude of each expected grid signal frequency to the amplitudes of its 4 or 8 neighbors, for example by the ratio of the former to the average of the latter. The values ​​for all grid points can then be summed to produce a net grid strength metric.

[0322] To ensure the accuracy of the extracted signature data, the applicant uses a process identified elsewhere to characterize the affine pose of the image patch and then iterates one or more pose parameters while monitoring changes in the grid strength metric to optimize the affine pose. For example, the x translation parameter of the determined affine pose can be adjusted by one-tenth or one-quarter of the watermark element to determine whether the grid strength metric increases. If so, further such adjustments are made. If the metric instead decreases, adjustments are made in the opposite direction, and so on. The same process is followed for the y translation parameter until a local maximum is found in the grid strength parameter function.

[0323] The process can be based on image patches of size 32×32 watermark elements, with the pose of each such patch being optimized to maximize the value of the associated grid strength metric. In a particularly preferred embodiment, this analysis is performed on different 32×32 watermark element patches of an image overlapped by 16 watermark elements. Three such 32×32 overlapping patches, 281 (shown in bold), 282, and 283, are shown in FIG. Fig.30 In this overlapping arrangement, each watermark element is contained in four overlapping patches. An example is Fig.30watermark element 285; it is contained in patches 281, 282, 283, the fourth patch not being shown (to avoid confusing the illustration).

[0324] Overlaying in this manner enables four estimates of the value of watermark element 284 (and all other such watermark elements) to be obtained. Its value is sampled once according to the affine pose parameters of patch 281; it is sampled a second time according to the affine pose parameters of patch 282; it is sampled again according to the affine pose parameters of patch 283; and it is sampled a fourth time according to the affine pose of a fourth patch.

[0325] As described above, the value of each such watermark element data is preferably weighted according to the grid strength of the image patch in which it is located to accumulate a value for submission to the Viterbi decoder. Since each watermark element is found in four overlapping patches, the sum of the four weighted data is accumulated and provided to the decoder as a confidence-weighted estimate of the watermark element value.

[0326] The arrangement just mentioned has been found to produce a significant improvement in the percentage of images from which payload data is successfully extracted.

[0327] The percentage of images from which payload data is successfully extracted can be further improved by dark frame subtraction techniques. In nighttime astronomy and other long exposure or high ISO photography, it is common to determine fixed pattern sensor noise by capturing long image exposures while a lens cover blocks sensor illumination, and then subtracting the corresponding pattern residue from the later captured image. Applicants are not aware of any such techniques used in extremely short exposure, high illumination environments like the present technique. However, this method has been found to provide significant improvements in decoding performance.

[0328] In one particular approach, the applicants placed a cap over the camera lens and captured 100 images with the sensor "dark" and the exposure interval and analog gain set to the values ​​expected to be used during normal operation. The frames were averaged to mitigate thermal (shot) noise. This produced a matrix of residual noise values ​​(a combination of read noise and dark noise) that could be subtracted from image frames captured later during operation to reduce this fixed sensor noise. (Dark pixel values ​​in the 1 to 12 digit range were found through this approach - noise patterns that interfere with decoding in many edge cases.)

[0329] Additional information for characterizing and removing fixed pattern noise prior to watermark decoding is described in detail in our patent US9,544,516.

[0330] Of course, the larger the sensor, the more sensitive it is and the shorter the exposure time. Ideally, the sensor has pixels larger than 3.5 microns on a side, and preferably larger than 5 microns on a side. Ideally, a sensor with a pixel size of 10 or 15 microns would be used, but cost is a factor. (An example is the SOPHIA 2048B-152 from Princeton Instruments—a 2K×2K sensor with a pixel size of 15 microns.) An alternative is to use “binning” for higher resolution sensors, such as a 2.5K×2.5K sensor with 5-micron pixels, where adjacent 2×2 groups of pixels are binned together to produce performance similar to that of a 1.25K×1.25K sensor with 10-micron pixels. However, binning reduces sensor resolution, so it is preferable to use a sensor that has appropriate sensitivity at its native resolution.

[0331] As noted above, either monochrome or color sensors may be used. Some printed labels are encoded using a "chromatic" watermark, for example using a combination of cyan and magenta inks. These two inks have different spectral reflectance curves, and when illuminated by white (red-green-blue) illumination, the difference between the red and blue (and / or green) channel camera responses can be subtracted to produce an image in which the watermark signal is emphasized. (See, for example, US Pat. No. 9,245,308.) However, despite the signal increase achieved by this technique, the applicants have discovered that illuminating such a label with red light alone and sensing it with a monochrome sensor produces a stronger and less noisy recovered watermark signal. (In addition, red LEDs are more efficient than, for example, green and blue LEDs - sometimes by twice as much or more. This translates into less heat, which, as noted above, in turn produces a greater light flux output.)

[0332] In other embodiments, the printed label may be encoded with machine-readable data (eg, a sparse watermark pattern) formed from yellow ink to encode recycling-related data.

[0333] More about Plastic Forming

[0334] The following discussion further details techniques for encoding plastic containers and labels to carry machine-readable indicia, including details for overcoming certain signal distortions introduced in the design and production of plastic containers.

[0335] To briefly review, a watermark is an optical code that typically consists of a two-dimensional pattern of code signal elements, usually in a square, that can be tiled edge-to-edge with other blocks to span a surface. Each square array can be thought of as a "grid" of coded positions. In some embodiments, each position is labeled to represent one of two data, such as "-1" or "1". (In other embodiments, the two data can be "0" and "1".)

[0336] Document 20040156529, previously cited by the applicant, describes how a code signal can be applied by etching a mold with a pattern of data. After determining the pattern that carries the desired data, the pattern is used to texture the surface of the plastic by forming the plastic in the mold. For the injection molding process, the mold is etched by a computer-driven etching device. Each cell in the output grid (array) pattern corresponds to, for example, a 250×250 micron patch on the mold. If the value of the output grid pattern for a particular cell is "1", a depression will be formed in the corresponding patch on the mold surface. If the value of the output grid pattern in the cell is "-1" representing "0", no depression is formed, and the appropriate depth of the depression depends on aesthetic considerations. Typical depression depths are less than half a millimeter and can be the size of the patch (250 microns) or smaller. The resulting mold pitting pattern is a physical manifestation of the output grid pattern. When the mold is used to form the surface of a product container, a negative of this pattern is produced, and each pit will form a raised dot on the container.

[0337] The size of the textured area depends on the patch size and the number of rows / columns in the output grid pattern. The larger the textured area, the more "signal" is available for decoding and the less stringent the specifications of the reading equipment. A textured area of ​​about one centimeter on a side has been found to provide adequate signal. Smaller (or larger) textured areas can be used depending on the application requirements.

[0338] Technologies other than computer-controlled etching equipment can be used to shape the mold based on the output grid signal. Small computer-controlled milling machines can be used. The same is true for laser cutting equipment.

[0339] While the foregoing methods contemplate containers having a texture already formed thereon, in other embodiments the container may be formed with a flat surface and, assuming the packaging material is a thermoplastic, the texture is subsequently applied, such as by a heated die.

[0340] To enhance the "signal" conveyed by the texture, surface variations may be made corresponding to "1" and "-1" values ​​in the output pattern grid (rather than just corresponding to "1" values, as described above). Thus, raised areas are formed in patches corresponding to output pattern cells taking "1" values, and recesses are formed corresponding to output pattern cells taking "-1" values.

[0341] In other embodiments, texturing can also be applied by additional layers of material applied to the container after the container has been formed in the desired output pattern. For example, a viscous ink can be applied during a screen printing process. The screen has an opening where the corresponding cell of the output grid pattern has a value of "1" and otherwise has no opening. When the viscous ink is applied through the screen, small patches of ink are deposited at the poor openings of the screen and not elsewhere.

[0342] Patches larger than 250 microns may be employed in such an embodiment, depending on the resolution limitations of the screen printing process. The result is again a textured surface, with the pattern of raised areas conveying the binary data payload.

[0343] A variety of materials besides ink can be applied to create a textured layer on top of the container. Thermoplastics and epoxies are just two options.

[0344] In some such embodiments, the textured layer is applied to the container using a technique other than printing. For example, various photolithography techniques may be used. One technique employs a photoreactive polymer, which is applied to the surface and then optically exposed through a mask corresponding to the output grid pattern. The exposed polymer is developed, thereby removing the material patch.

[0345] In other embodiments, the output grid pattern is printed onto the container surface in two contrasting colors (e.g., black and white). Cells with "1" values ​​may be printed in one color and cells with "-1" values ​​may be printed in another color. In such embodiments, the binary payload is not discerned from the texture pattern, but rather from the contrasting color pattern.

[0346] As further discussed in priority application 62 / 814,567, other patent documents of the applicant identified herein detail other procedures for physically implementing 2D optical codes on articles.

[0347] To counteract signal distortions in the design and / or manufacturing of plastic containers, a number of approaches can be used.

[0348] In a first embodiment, a signal encoding is introduced into a container mold during 3D printing of the mold. The inner surface of the mold is in contact with the outer surface of the container and is printed to include a subtle texture, pattern, image or design. The texture, pattern, image or design carries the encoded signal. For example, an original sparse watermark signal is generated, for example, as detailed in the publications US20170024840, US20190139176 and US20190332840. We use the term "original" here to mean that the sparse watermark signal has not yet been combined with a host image or surface. The original sparse watermark is used as a template to guide the inner surface of the 3D printed mold. The surface of the mold includes various bumps and pits, which together (often redundantly) carry the original sparse watermark.

[0349] Let's look at the workflow. A 3D mold is designed in CAD software such as AutoCad, Photoshop, Solidworks, Materialise, or many others. The CAD software defines the geometry of the mold. For example, the mold can be shaped to produce a water bottle, yogurt cup, or other container. A 2D encoded signal (e.g., a sparse watermark) is generated. At this point, the 2D watermark signal needs to be mapped onto the 3D inner surface of the mold, preferably in a way that minimizes distortion of the encoded signal.

[0350] One approach to minimizing distortion utilizes a unidirectional pre-distortion based on the relative size of the intended container. We will use an hourglass-shaped container as an example. The radius in the middle of such a container is smaller than the radius at the top and bottom. If a 2D rectangular watermark tile is mapped to this container, the middle of the container may be scaled differently relative to the top and bottom. As a result, the watermark tile may be stretched more in one spatial dimension (x-axis) than in another spatial dimension (y-axis). This type of distortion is sometimes referred to as differential scaling or shearing. Consider the example where the original watermark tile is a square. Due to the differential scaling, the square may be distorted into a parallelogram with unequal sides. The differential scaling parameter defines the nature and extent of this stretching. Differential scaling can cause specific problems for watermarks and detectors. When looking at a square embedded tile with x and y coordinates and equal x and y sides, the length of the x dimension in the middle of the container decreases when applied, while the length of the y dimension generally remains the same. If the middle radius is about .75 relative to the top and bottom radius, the x coordinate will shrink by about .75*x when mapped to the surface, while the y coordinate will remain roughly the same (1*y). This results in a different scale of the x and y coordinates, which like creates an image capture angle of about 41 degrees, making the detection of sparse watermarks difficult.

[0351] On the encoding side, one goal of the solution is to create an encoded signal that, once presented to the mold surface, is in a range of orientations detectable by the decoder. For example, the signal is preferably within one scale, rotation, and translation state that the detector can detect. Differential scales are particularly difficult to re-register for data extraction. To address this differential scale issue, we strive to keep the x and y coordinates of the tile similar dimensions to each other after mapping to the 3D surface. Therefore, we pre-distort the tile in 1 direction before embedding. In particular, we pre-distort the tile in the y direction by a similar amount to be expected from any x-direction distortion. The result is that after pre-distortion and mapping, the x and y sides are of similar dimensions, albeit resulting in smaller embedded tiles. The y direction of different tiles placed on the surface can be determined individually by the relative size of the radius at each embedding location. The pre-distortion will vary across the mold based on where the tile is placed on the 3D surface. (The same distortion correction process can be used to apply a label to a container, such as when applying a shrink wrap label to a curved container. The y direction of the embedding file can be changed to contain the same expected scale as the x direction after heat shrinking.)

[0352] Another way to minimize distortion is to use so-called UV texturing (or mapping). UV texturing uses the polygons that make up the 3D object to texture with the surface properties of a 2D image (e.g., a "UV texture map"). The texture map has coordinates U, V, while the 3D object has coordinates X, Y, Z. UV mapping assigns pixels in the UV texture map to the surface mapping on the polygons. This can be achieved by copying the triangular portion of the UV texture map and pasting it onto the triangles on the 3D object. UV texturing is an alternative mapping system that only maps into the texture space of the object instead of the geometric space. The rendering calculation uses the UV texture coordinates to determine how to arrange the 3D surface. UV texturing can be used to convey a 2D sparse watermark (or other 2D encoded signal) onto the surface of the mold. Here, a sparse watermark is used as a UV texture map to texture the mold surface. Different gray levels in the UV texture map can be used to indicate the depth or height of the mold surface texture. The resulting textured surface of the mold carries a watermark signal.

[0353] Even further detection improvements can be achieved when using the above-described 1-D scaling or UV texture mapping prior to 3D printing or using laser texturing when used in combination with advanced decoding techniques within the detector, as described in assignee's patent documents US9,182,778 (e.g., including direct least squares for recovering geometric transformations in the detector), US9,959,587 (e.g., using direct least squares for perspective approximation and using phase estimates in coordinate updates and related metrics), and patent 10,373,299 (e.g., using direct least squares to improve perspective distortion (tilt) performance), and document US10,242,434 (e.g., detector using hybrid complementary methods for geometric registration, such as Log polar for low tilt angles / weak signals and direct least squares for higher tilt angles). Such detection techniques will help recover signals that were distorted during mold creation and / or during image capture of containers made using such molds. For example, mold creation may introduce first distortions associated with mapping a 2D surface to a 3D object, and image capture may introduce tilt, scale, and / or relationship relative to the container from the camera perspective.

[0354] In a further embodiment, we utilize the distortion resolution techniques discussed in the patent documents of the applicant to compensate for mapping the 2D signal into the 3D mold. See, for example, patent documents US6,122,403, US6,614,914, US6,947,571, US7,065,228, US8,412,577, US8,477,990, US9,033,238, US9,182,778, US9,349,153, US9,367,770, US9,864,919, US9,959,587, US10,242,434 and 10,373,299.

[0355] The surface texture of the inner surface of the mold is used to create an encoded signal in the plastic container. The texture is created by producing bumps and / or pits in the mold surface, which produces pits or bumps on the container. For example, for a sparse watermark tile, each embedding location corresponds to, for example, an n×m inch patch on the mold. If the value of the embedding location is "1", a depression is formed in the corresponding patch on the mold surface. If the value of the embedding location is "-1", no depression (or pit) is formed. Therefore, raised areas are formed in the container corresponding to embedding locations that take the value of "1", and unchanged areas (or pits) are formed corresponding to embedding locations that take the value of "-1". When the image of the marked container is analyzed, the bumps and pits will have different reflective characteristics. These differences can be analyzed to decode the encoded signal.

[0356] Going back to the workflow, changing the geometry of the mold that defines it, generating a 2D encoded signal (e.g., a sparse watermark), and mapping the watermark signal to the 3D inner surface of the mold, we generate the corresponding 3D printer file format (e.g., STL, OBJ, AMF, or 3MF) to control the 3D printer. The 3D printer prints the mold according to the encoded signal pattern, including the surface texture. Of course, the 3D printer needs to be able to print at a resolution that accommodates the encoded signal pattern. For example, if the encoded signal pattern corresponds to 50 or 100 dots per inch, the printer needs to be able to reproduce such a pattern.

[0357] In other embodiments, instead of using sparse watermarks to guide surface texturing, a signal generated by a neural network or a Voronoi, Delaunay, or stipple halftone based signal may be used. Such signals are discussed in patent publications WO / 2019 / 113471 and US20190378235.

[0358] In other embodiments, the coding is introduced into the mold surface by laser engraving, etching, embossing or ablation. When the container is formed, a machine-readable mark (carried by surface topological changes within the mold) is imparted to the plastic. Very fine texture patterns can be achieved using laser engraving and tool etching. Recently, laser texturing of molds has been developed to the point where different depths can be produced. Multiple different levels of depth can be used to carry different signal information. For example, in the signal value term, the first depth can represent "1", while the second depth can represent "0", and the third depth can represent "-1". As described above, UV texture mapping and / or 1-direction pre-distortion can be used to counteract 2D to 3D conversion.

[0359] Another consideration for 3D printed, laser engraved, and etched molds is that the container surface signal must be able to survive the forming process, not degrade the performance of the finished container (e.g., create areas that are too thin), and facilitate the release of the container from the mold (e.g., not get stuck in the mold). For example, if the mold produces bumps or raised areas on the container, the shape of the corresponding indentations in the mold should facilitate the release of the container from the mold. For example, if the mold includes sharp and deep dimples (corresponding to sharp and high bumps on the container), the container may not be able to be demolded from the mold. Mold dimple bumps can be shaped in a directionally oriented manner in the demolding direction, such as a teardrop shape (or Rochemontone shape). Alternatively, the shape of the dimples can match the tool release draft angle of the tool, material type, and / or part shape.

[0360] Similar considerations apply to sintered metal or ceramic parts with a watermarked surface texture. The watermarked textured block must be demolded without deformation before firing, and the watermarked texture will deform along with the watermarked textured block during sintering. The expected deformation can be corrected by predistortion of the watermark signal.

[0361] In yet another embodiment, the watermark texture may be formed by linear stripe ridges modulated by the stripe position. A mold comprising such linear ridges may be moved in space, and / or increased / decreased in size, to represent the watermark signal.

[0362] Detailed instructions for mold generation

[0363] One specific example takes a sparse watermark signal and places shapes or structures at the point locations. Instead of marking square points, the 3D surface topology is preferably formed with smooth divots, pits or peaks, for example, in the shape of a Gaussian or sinusoidal curve. Another example forms artistic lines or element features (such as circles, lines, ellipses, etc.) that coincide with the peaks and / or valleys of the continuous watermark signal. Another example forms a 3D surface pattern of a signal-rich artistic design described in patent disclosures WO / 2019 / 113471 and US20190378235, for example, including but not limited to Voronoi, stipple, Delaunay, and traveling salesman patterns. In such an example, the topology is formed so that the cross-section of the pattern of surface peaks or depressions is smoothed (for example, in the form of a sinusoidal or Gaussian cross-section). The achievable cross-section depends on the type of marking (CNC milling, laser marking, 3D printing) and should be designed to ensure proper release of the part from the mold. The smoothing of the contour should address the latter goal.

[0364] In the following example, the design goal of converting a two-dimensional data-bearing signal into a die is discussed.

[0365] Various factors come into play when selecting the signal type (e.g., continuous vs. binary vs. sparse), such as the type of plastic being molded (PET, PP, PE, HDPE, LDPE, etc.), the type of manufacturing process (e.g., blow molding, injection molding, thermoforming, etc.), the type of mold (metal, ceramic, etc.), the mold manufacturing process (etching, engraving, etc.), the aesthetics and properties of the camera / lighting used for inspection. In particular, continuous signals typically require higher resolution in both space and depth (embossing, debossing, etching, etc.). Binary signals typically include higher resolution in space, but lower resolution in depth. Sparse binary signals can be performed when both available spatial and depth resolution are lower, such as in thermoforming. (Blow molding and injection molding offer better accuracy than thermoforming.)

[0366] Another factor to consider is the ratio between the reference (synchronization) signal strength and the message signature strength. Ensuring that the message signal strength is sufficiently strong relative to the synchronization signal component enhances the reliability of digital payload recovery. For sparse and binary markings, the ratio of the synchronization signal to the message signal can be heuristically determined based on the specified watermark resolution, image resolution, dot size, number of dots per tile, and payload size. A different set of heuristics can be created for different types of plastics, molds, etc. For example, the properties of the plastic (e.g., the inherent viscosity of sheet-grade, bottle-grade, and film-grade PET) can determine the extent to which the embossed plastic is beneficial in carrying the spectral characteristics of the watermark signal (e.g., low frequencies versus high frequencies). Similar considerations apply to continuous and binary signaling.

[0367] Another factor to consider is the watermark signal resolution. The resolution of the watermark signal in each signal block (tile) should be high enough to achieve the desired aesthetics while being able to read the watermark payload from small tiles so that object curvature introduces less geometric deformation on each tile. In one example, the recommended resolution is 200 watermark units (watermark elements) per inch (WPI) or higher. With a tile size of 128×128 watermark elements, the tile size for a 200WPI tile is 0.64 inches by 0.64 inches.

[0368] In addition to improving the detection of objects with non-rectangular geometries, higher resolution watermarks can improve the detection of flattened, crushed, deformed, or shredded objects found in recycling streams.

[0369] Reducing the density of dots within each watermark tile has multiple benefits. For example, the signal pattern on the molded object is less visible, which means it interferes less with the visual quality and aesthetics of the object. For transparent containers, the signal pattern has less visual impact on the contents of the container (e.g., water in a clear plastic water bottle). In addition, since the dots are converted into bumps or pits / divots / notches in the object surface, fewer dots means smaller dot spacing, making it easier to form corresponding shapes in the mold. Techniques for making mold surface topology (e.g., bumps or pits / divots / notches) include, for example, laser engraving, etching, electrical discharge machining (e.g., so-called "spark erosion"). Computer numerically controlled ("CNC") milling or 3D printing. When using CNC milling cutters, care should be taken to ensure sufficient resolution. Marking equipment with a larger mark width can be used to remove surface material, leaving a contour bump with a diameter less than the drill width. The drill bit shape can be varied to achieve the desired dot representation, including, for example, conical drill bits, triangular drill bits, circular cross-sections, ball mills. In addition, the divots may not need to be very deep, but intensity variations can be used. Having fewer, wider bumps / divots spaced apart also makes it more feasible to smooth out the contours of the bumps and divots in the mold.

[0370] The dot density can be expressed as a percentage of dots relative to the tile with the maximum dot coverage percentage. The maximum percentage of dot coverage for each cell in the watermark signal tile containing a binary pattern of dots or no dots is 50%. This means that half of the cells (watermark elements) in the tile are marked with dots (e.g., dark values). The dot density should preferably be lower to reduce visibility, such as 10 to 35 (meaning 5% to 17.5% of the tile marked with dots).

[0371] The dot size was mentioned above. The dot size is a parameter that controls the size of the basic dot structure within the sparse signal. We express the dot size in terms of the specified image resolution in dots per inch (DPI), for example, 600 DPI means 600 pixels per inch. The dot size is an integer value that represents the dot size (in pixels) along an axis at a given image resolution. A dot size of 1 means that the dot is one pixel. A dot size of 2 means that the dot is two pixels (e.g., arranged in a row, column, or along a diagonal in a two-dimensional array of pixel coordinates). For example, a dot size of 1 or 2 at 600 DPI translates to a dot width of 42 or 84 microns, and depressions with this dot width can be made with a larger drill size (e.g., 257 microns) because the drill requires it to be only partially pressed into the surface of the aluminum mold.

[0372] Dots can be of different shapes. While square dots can be easily represented in an image as pixels, there may be more appropriate shapes and structures to encode signals in physical materials such as plastics or metals. Examples include circles, ovals, lines, etc. Smoother shapes may be easier to produce than shapes with sharp edges or corners (e.g., because molds can be easily made).

[0373] Different types of plastics, molds, and mold making allow for different depths of marking on the plastic surface, such as deeper or shallower. Typically, when deeper marking is possible, a lower dot density can be used, while when the mark is shallow, a higher dot density is typically used. Deeper marks are more likely to survive changes in the workflow, such as surface wear, flattening, and crushing.

[0374] The image signal representation of the watermark tile provided for conversion into the 3D surface topology of the mold can be a vector or rasterized image file, such as an SVG image format. The file can be generated using image editing tools such as Adobe Photoshop, design tools such as Adobe Illustrator, or signal processing software such as MATLAB from Math Works, for example, in combination with the watermark tool plug-in of the applicant Digimarc Corp.

[0375] In an electronic image file, dots can be different shapes. While square dots can be easily represented in an image in the form of pixels, different shapes and structures are often better suited for encoding signals in physical materials such as plastics or metals. Examples include circles, ovals, lines, etc. For example, smoother shapes are easier to reproduce than shapes with sharp edges or corners due to the ease of making molds. Vector representations enable the definition of dots based on their shape, which is beneficial for the aesthetics of the finished molded product and the performance of the mold. Performance considerations for molds are the taper, smoothness, or contour of the divot or bump to facilitate release of the molded part from the mold. In a simple example, the dots have a circular shape, which helps to be formed into the surface of an aluminum mold, for example, using CNC equipment. The 3D structure of the shape plays a role in ease of manufacturing (e.g., release) as well as the induced light intensity variation of the modulation required to carry the encoded watermark signal. Signal-rich art forms can be created by selectively placing objects of desired shapes at point locations and / or drawing vector art through the point locations so that the vector art is highly correlated with the watermark signal at the point locations (e.g., as discussed in US20190378235 and WO / 2019 / 113471).

[0376] The resolution of the tile image (e.g., in DPI) determines the granularity of modulation that can be made in the material. Using a higher resolution (e.g., 600 DPI) allows greater flexibility in designing features (e.g., dots or other structures) that can be embossed, debossed, etched, milled, corroded, etc. Using a higher resolution also provides greater flexibility in signal formation, for example, creating sparse binary marks by allowing more leeway in choosing forbidden areas, dot shapes, sizes, etc.

[0377] Review of Exemplary Watermarking Methods

[0378] In an exemplary watermarking method, a multi-symbol message payload (e.g., 48 binary bits, which together with 24 associated CRC bits can represent a product's Global Trade Identification Number (GTIN) or plastic recycling information) is applied to an error correction encoder. The encoder converts the symbols of the message payload into a longer array of encoded message elements (e.g., binary or multi-ary elements) using an error correction method. (Suitable encoding methods include block codes, BCH, Reed Solomon, convolutional codes, turbo codes, etc.) The encoder output may include hundreds or thousands of binary bits, such as 1024, which may be referred to as raw signature bits. These bits may be scrambled by XORing with a scrambling key of the same length to produce a scrambled signature.

[0379] Each of these scrambled signature bits modulates a pseudorandom noise modulation sequence of length 16 (the spread carrier), e.g., via XOR. Thus, each scrambled signature bit produces a modulated carrier sequence of 16 “chips,” resulting in an amplified scrambled payload sequence of 16,384 elements. This sequence is mapped to the elements of a square block of 128×128 embedding positions, based on the data in the scatter table, resulting in a 2D payload signature pattern. (The scatter table assigns 4 chips per scrambled signature bit to each of the four 64×64 quadrants in the 128×128 block.) Each position in the 128×128 block is associated with the values ​​0 and 1, or black and white, with approximately half of the positions having each state. This bimodal signal often maps to a larger bimodal signal centered around an 8-bit grayscale value of 128 (e.g., values ​​95 to 161). Each of these embedding locations may correspond to a small region of pixels, such as a 2x2 patch called a "bump," which results in a watermark message block having a size of 256x256 pixels.

[0380] As mentioned above, a synchronization component is often included in a digital watermark to help discern the parameters of any affine transformations that the watermark has been subjected to prior to decoding so that the payload can be correctly decoded. The particular synchronization component takes the form of a reference signal consisting of a dozen or more amplitude peaks of a pseudo-random phase sinusoid in the Fourier domain. This signal is transformed to the spatial domain (e.g., by an inverse fast Fourier transform) in a 256×256 block size corresponding to the 256×256 block to which the upscaled scrambled payload sequence is mapped. The spatial domain reference signal may include floating point values ​​between -1 and 1, which may be scaled to a range of -40 to 40 and combined with the 256×256 pixel payload block to produce the final watermark signal block, e.g., values ​​ranging from 55 (i.e., 95-40) to 201 (i.e., 161+40). This signal may then be summed with the host image, first scaled down to make it unobtrusive.

[0381] If such a watermark signal block is printed at a spatial resolution of 300 dots per inch (DPI), the printed block is approximately 0.85 square inches. (Since a 0.85 inch side dimension corresponds to 128 watermark elements, this works out to 150 watermark elements per inch.) These blocks can be tiled edge-to-edge to mark larger surfaces.

[0382] The watermark signal just described may be referred to as a "continuous tone" watermark signal. It is typically characterized by multi-valued data, i.e., not just on / off (or 1 / 0, or black / white), hence the "continuous" moniker. Each pixel of the host image (or region within a host image) is associated with one corresponding element of the watermark signal. Most pixels in the image (or image region) change value by combining with their corresponding watermark element. These changes are typically both positive and negative, e.g., changing the local brightness of the image upward in one location and downward in another. And the extent of the changes may vary, with some pixels changing by relatively small amounts and others by relatively large amounts. Typically, the amplitude of the watermark signal is low enough that its presence in the image would not be noticed by a casual viewer (i.e., it is steganographic).

[0383] (Due to the highly redundant nature of encoding, some embodiments may ignore pixel changes in one direction or the other. For example, one such embodiment changes pixel values ​​only in the positive direction. Pixels that would normally change in the negative direction remain unchanged.)

[0384] In variant continuous tone watermarks, the signal does not change the local brightness of the picture pixels, but their color. Such watermarks are called "chrominance" watermarks (as opposed to "luminance" watermarks). For example, an example is described in detail in patent US9,245,308.

[0385] "Sparse" or "binary" watermarks differ from continuous-tone watermarks. They do not alter the values ​​of most pixels in a host image (or image region). Instead, they are printed at a density (which may sometimes be set by the user) that results in marking approximately 5% to 45% of the pixel locations in the image. The adjustments are usually all in the same direction, e.g., reducing brightness. Sparse elements are usually two-tone, e.g., either white or black. Although sparse watermarks may be formed over other images, they are usually presented in areas of the picture that are blank or colored with a uniform tone, in which case the sparse mark may contrast with its background, making the mark visible to a casual viewer. Although sparse marks can take the form of a seemingly random field of dots, they can also take the form of a structure of lines, as detailed elsewhere. As with continuous-tone watermarks, sparse watermarks usually take the form of blocks of signal tiled across an image area.

[0386] Sparse watermarks can be generated from continuous-tone watermarks by thresholding, that is, the darkest elements of the summed reference / payload signal block are copied into the output signal block until the desired point density is achieved.

[0387] Patent publication US20170024840 details various other forms of sparse watermarking. In one embodiment, the signal generator starts with two 128×128 inputs. One is a payload signal block whose positions are filled with a binary (0 / 1, black / white) extended scrambled payload sequence, as described above. The other is a spatial domain reference signal block, with each position assigned a floating point number between -1 and 1. The darkest (most negative) "x"% of these reference signal positions are identified and set to black; the others are set to white. The spatially corresponding elements of the two blocks are ANDed to find the coincidence of black elements between the two blocks. These elements are set to black in the output block; the other elements remain white. The output signal block can be made darker or lighter by setting "x" higher or lower.

[0388] Publication US20190332840 details additional sparse coding embodiments. One embodiment uses a reference signal generated at a relatively high resolution (e.g., 384×384 pixels), and a payload signature spanning a relatively low resolution array (e.g., 128×128). The latter signal has only two values ​​(i.e., it is bitonal); the former signal has more values ​​(i.e., it is multi-level, such as binary grayscale or composed of floating point values). The payload signal is interpolated to the higher resolution of the reference signal and converted from bitonal form to multi-level form in the process. The two signals are combined at a higher resolution (e.g., summed by a weighted ratio), and a threshold operation is applied to the result to identify the locations of extreme (e.g., dark) values. These locations are marked as producing sparse blocks (e.g., 384×384). The threshold level determines the point density of the resulting sparse marking.

[0389] Different embodiments sort the samples in the reference signal block by value (darkness), thereby producing a ranked list of the darkest N positions (e.g., 1600 positions), each with one position (e.g., within a 128×128 element array). The darkest of these N positions is always marked in the output block (e.g., 400 positions or P positions) to ensure a strongly expressed reference signal. Whether the other positions in the N positions (i.e., NP or Q positions) are marked depends on the values ​​of the message signal data mapped to these positions (e.g., by a scattering table in the encoder). Positions in the sparse block that are not in the N darkest positions (i.e., not among the P or Q positions) are never marked, so they are definitely ignored by the decoder. By setting the number N larger or smaller, sparse markings with more or fewer points are produced. (This embodiment is referred to as the "fourth embodiment" in the earlier cited disclosure 20190332840.)

[0390] When generating sparse markers, spacing constraints may be applied to candidate marker locations to prevent clumping. Spacing constraints may take the form of circular, elliptical, or other (e.g., irregular) shaped forbidden regions. Forbidden regions may have two or more or fewer axes of symmetry (or none). Enforcement of spacing constraints adds dark markers to the output block, with corresponding data stored in a data structure identifying locations that are no longer available for possible markers due to spacing constraints.

[0391] In some embodiments, the reference signal may be adjusted to have a non-random appearance by varying the relative amplitudes of the spatial frequency peaks so that they are not all of equal amplitude. Such variations in the reference signal have a corresponding effect on the appearance of the sparse signal.

[0392] Sparse patterns can be presented in a variety of terms. The most straightforward is a seemingly random pattern of dots. But more artistic renderings are possible, including those discussed and illustrated above.

[0393] Other disclosed artistic modes for conveying watermark data are described in detail in patent publication US20190139176. In one detailed method, a designer creates a candidate picture design or selects one from a design library. In this method, vector art in the form of small, discrete printed structures of lines or desired shapes works well. The payload is input to a signal generator that generates a raw data signal in the form of two-dimensional data signal elements. The method then edits the picture at spatial locations based on the data signal elements at those locations. When a picture with the desired aesthetic quality and robustness is produced, it is applied to an object, such as by laser marking.

[0394] Other techniques for generating visible pictures with robust data signals are detailed in assignee's patent publication US20190213705 and pending application 62 / 841,084 filed on April 30, 2019. In some embodiments, a neural network is applied to an image including a machine-readable code to transform its appearance while maintaining its machine readability. A particular method uses a style image with various features to train a neural network. The trained network is then applied to an input pattern that encodes a multiple symbol payload. The network adjusts features from the style image to express the details of the input pattern, thereby producing an output image in which the features from the style image contribute to the encoding of the multiple symbol payload. The output image can then be used as a graphic component in product packaging, such as a background, border, or pattern fill. In some embodiments, the input pattern is a watermark pattern, while in other embodiments it is a host image that has been previously watermarked.

[0395] Other such techniques do not require a neural network. Instead, a watermark signal block (i.e., reference plus message signal) is decomposed into sub-blocks. The style image is then analyzed to find the sub-block with the highest correlation with each watermark signal sub-block. The sub-blocks from the style image are then stitched together to produce an output image that visually evokes the style image but has signal characteristics that mimic the watermark signal block.

[0396] In addition to the references cited elsewhere, details regarding watermark encoding and reading that may be included in implementations of the present technology are disclosed in the applicant's previous patent applications, including U.S. Patents Nos. 5,850,481, 6,122,403, 6,590,996, 6,614,914, 6,782,115, 6,947,571, 6,975,744, 6,985,600, 7,044,395, 7,065,228, 7,123,740, 7,130,087, 7,403,633, 7,763,179, 8,224,018, 8,300,274, 8,4 12,577, US8,477,990, US8,543,823, US9,033,238, US9,349,153, US9,367,770, US9,521,291, US9,600,754, US9,749,607, US9,754,341, US9,864,919, US10,113 ,910, US10,217,182 and US20160364623, as well as pending U.S. applications 16 / 270,500 filed on February 7, 2019, 62 / 814,567 filed on March 6, 2019, 62 / 820,755 filed on March 19, 2019, and 62 / 946,732 filed on December 11, 2019.

[0397] While the techniques just detailed are generally described in the context of printed watermarks, the same techniques can be used for 3D texture / shape based watermarks.The sparse point and line elements of a binary mark can be represented by protrusions (or depressions) in a 3D surface.

[0398] Similarly, positive and negative changes in pixel values ​​associated with a continuous tone watermark may be represented by spatial changes in the height of the 3D surface. In some arrangements, the surface changes in only one direction, such as by protrusions upward from the surface. In other arrangements, the surface may change in reciprocal directions by protrusions upward from the 3D surface and depressions (pits) downward into the 3D surface.

[0399] When the surface changes in only one direction, one implementation ignores negative changes in the watermark value; the surface changes only due to positive changes. Another implementation ignores positive changes in the watermark signal; the surface changes only due to negative changes. In both implementations, the surface changes can be in the positive protrusion direction, or in the negative concavity direction.

[0400] In other implementations, the most negative change (extreme value) of a continuous watermark signal is mapped as no surface change, while increasingly positive changes from this extreme value are mapped as increasingly larger surface changes (protrusions or depressions). In still other implementations, the most positive change of a continuous watermark signal is mapped as no surface change, while increasingly negative changes from this value are mapped as increasingly larger surface changes (again, protrusions or depressions).

[0401] When the surface varies in both directions, negative values ​​of the continuous tone watermark signal may be mapped as depressions in the surface (the depth depends on the negative signal value), while positive values ​​of the watermark signal may be mapped as protrusions in the surface (the height depends on the positive signal value). In other embodiments, the polarity may be switched, with positive values ​​of the watermark signal being mapped as depressions in the surface and vice versa. The depth of the deepest depression and the height of the tallest protrusion may be equal, but they need not be. The average depression depth and average protrusion height are the same as above. The depth / height may be asymmetric, as if a DC offset were applied to the continuous tone watermark signal.

[0402] Where the surface changes in two directions, both the depressions and protrusions desirably convey watermark payload information (unlike the arrangement of US20180345323 which teaches that only one or the other conveys payload information).

[0403] refer to Fig.32A and 32B A recycling system including individually addressable sorting bins (or categories) is discussed. The recycling system includes two stages, but the functions can be combined into one stage or multiple stages. One or more sorting units include one or more light sources, one or more image capture units, a watermark reader, and control logic. The sorting unit reads digital watermark information from image data depicting plastic objects in the waste stream. This information can indicate the type of plastic (e.g., polyethylene terephthalate, high-density polyethylene, low-density polyethylene, polypropylene, polycarbonate, etc.) and can also convey other information useful for recycling. Diverters and / or other mechanisms are controlled based on this digital watermark information to direct the plastic objects to the appropriate destination for recycling or reuse. In the first stage, such as Fig.32AAs shown on the left side of FIG, plastic objects (or other container materials) are initially classified in a binary manner, for example into coded and non-coded categories. The non-coded category includes plastics without a detectable digital watermark. This may include plastics that originally did not contain a watermark, as well as any plastics for which the original digital watermark has deteriorated to the point of being undetectable. The coded category includes plastics with a detectable digital watermark.

[0404] The coded plastic is further processed according to the second level, e.g. Fig.32A As shown on the right. Fig.32A Two separate processing stages are shown, but the functionality of the first (left side of the figure) and second (right side of the figure) stages may be combined into one or three or more stages.

[0405] refer to Fig.32A and 32B The coded plastic is funneled or otherwise conveyed under or through the sorting unit at a speed of, for example, 1-3 m / sec, and even more preferably at a speed of 5-9 m / sec, for example at a speed of 5 m / sec. In one implementation, the coded plastic is fed in single pieces; in other implementations, the coded plastic is fed in multiple pieces.

[0406] The sorting unit includes one or more light sources, one or more image capture units, one or more watermark readers, and control logic. For example, the one or more light sources may include one or more LEDs, and the image capture unit may include one or more cameras or image sensor arrays. The watermark reader operates to decode a watermark from an image frame representing the encoded plastic. The watermark reader provides the decoded watermark data to the control logic to control individually addressable sorting diverters along the stream sorter path.

[0407] Fig.33A The arrangement shows an implementation of a sorting unit that includes multiple light sources arranged along the direction of movement of the conveyor belt (waste flow). In an alternative embodiment, the light source crosses the direction of movement of the conveyor belt instead of being in a straight line with the direction of movement of the conveyor belt. In other embodiments, one or more light sources are positioned along the direction of movement of the conveyor belt (as shown in the figure) and one or more light sources are positioned across the direction of movement. Different light sources can be activated for alternating frames of image capture by the camera (which can be capture frames, for example, at 150, 300 or 500 frames per second). One frame is illuminated by one light source, the next frame is illuminated by another light source, and so on. Or if multiple image sensors are used, or an image sensor with two or more color filters is used, the light sources can be activated simultaneously, where the image corresponding to the first light source is captured by the first sensor, the image corresponding to the second light source is captured by the second sensor, and so on.

[0408] In one specific example, Fig.33A The three light sources in include a red LED (e.g., having a peak illumination between 620nm-700nm, referred to as "at or near 660nm"), a blue LED (e.g., having a peak illumination between 440nm-495nm, referred to as "at or near 450nm"), and an infrared (or far-red) LED (e.g., having a peak illumination between 700nm-790nm, referred to as "at or near 730nm"). In a more specific example, the red LED has a narrow-band center wavelength between 650nm-670nm, for example at 660nm, and the full width half maximum ("FWHM") of the emission is equal to or lower than 30nm; the blue LED has a narrow-band center wavelength between 440nm-460nm, for example at 450nm, and the FWHM emission is equal to or lower than 30nm; and the infrared (or far-red) LED has a narrow-band center wavelength between 720nm-740nm, for example at 730nm, and the FWHM of the emission is equal to or lower than 40nm.

[0409] exist Fig.33B In another specific example of the sorting unit shown, the two light sources include a blue LED (e.g., having a peak luminance between 440nm-495nm, referred to as "at or near 450nm") and an infrared LED (e.g., having a peak luminance between 700nm-790nm, referred to as "at or near 730nm"). In a more specific example, the blue LED has a peak luminance between 440nm-465nm, such as at 450nm; and the infrared LED has a peak luminance between 710nm-740nm, such as at 730nm.

[0410] In another specific example of a sorting unit, ambient light is used to illuminate the coded objects.

[0411] The one or more image capture units of the sorting unit include one or more cameras or image sensor arrays for collecting images or image frames corresponding to different LED illuminations. The cameras or image sensor arrays can be positioned relative to the LEDs at various positions (or vice versa), for example, as discussed in this patent document.

[0412] In an alternative implementation, reference Fig.33C and 33D, each point of the conveyor within the FoV is preferably illuminated by a diffuse light source of each light color from multiple directions. The LED light is preferably focused by a lens of at least 40 degrees FWHM perpendicular to the travel of the conveyor. Along the conveyor, the light preferably comes from at least 2 directions (e.g., 2 light bars) about 10-25 degrees from the camera axis. The lights can be placed at least 50cm from the conveyor to minimize the difference between the near field and the far field. Each light source can be focused to illuminate the field of view (FoV). In addition, light diffusion on top of the LED lens is recommended, and a depth of field (DoF) of at least about 10cm is preferred. It is not recommended to use camera gain (digital or analog) to maximize SNR. The captured image can be monochrome, for example, with 8-bit dynamic range without compression.

[0413] The image capture unit may include one or more monochrome cameras, for example, with a capture rate of up to 300-700 frames per second (FPS), 8-bit or higher dynamic range. The number of frames per second depends to some extent on the conveyor speed. For example, at least 300 FPS is preferred for a conveyor speed of 3 m / sec (e.g., allowing 150 FPS of red LED illumination frames + 150 FPS of blue LED illumination frames). But at least 500 FPS is preferred for a conveyor speed of 5 m / sec (e.g., allowing 250 FPS of red LED frames + 250 FPS of blue LED frames). The recommended maximum camera exposure time is approximately 60 μs for a 3 m / sec conveyor and approximately 40 μs for a 5 m / sec conveyor. A monochrome area scan camera with a global shutter can be used to minimize motion artifacts. In this alternative implementation, as Fig.33C As shown, the camera optical axis is perpendicular to the conveyor belt. The sampling resolution of the camera at a distance of 50cm from the conveyor belt can be measured in pixels, such as 150-600 pixels per inch, or 170-180PPI in one example. Preferably, one or more cameras are positioned so that the FoV captures the entire width of the conveyor belt; if multiple cameras are used, at least 2cm FoV overlap is preferred. As for the conveyor belt length FoV, it is preferred to capture at least 14cm of the conveyor belt along the direction of travel of the conveyor belt. We recommend using a lens aperture of f / 5.6 or smaller, such as f / 8.

[0414] The light source can be pulsed and synchronized with the camera and can cycle through a mix of different color LEDs, e.g. by first illuminating with a 730nm LED and then with a 450nm LED, producing 2 frames, or by first mixing 730nm and 450nm and then mixing with 660nm, producing 2 frames.

[0415] In one illustrative embodiment, the sorting unit includes a watermark reader or decoder to analyze the resulting image frames for watermark data, such as Fig.34 Look for printed label watermarks and plastic watermarks as shown; or, more generally, look for Fig.35 One or more watermarks carried by the plastic are shown. The watermark payload data can be used as one or more inputs to sorting unit control logic (e.g., diverter control logic). In one implementation, the watermark payload data is provided to a database that indexes relevant information, and the relevant information is provided to the control logic. The diverter control logic controls the sorting unit along the Fig.32B Various sorting diverters positioned in a path, for example, the path includes one or more conveyor belts, rollers or a free fall path.

[0416] refer to Fig.36A , a watermark reader (housed in Fig.32A and 32B The digital watermark is decoded from the captured image frame by a system (in or in communication with the sorting unit shown in ). In one example, the decoded digital watermark includes the GTIN (and possibly other data). In order to determine the BIN VALUE (bin value), a data structure 122 such as a table or database can be used. It is used to associate the item GTIN with corresponding information about the encoded plastic container. That is, the data structure is queried using the GTIN identifier decoded from the digital watermark, and the system thereby accesses previously stored data identification for the product having the GTIN, such as the BIN VALUE (and other information, such as plastic type, subtype and / or color). The BIN value information can be provided to the logic that controls the sorting diverter. This allows bins to be individually addressed along the recycling path.

[0417] Back to Fig.32B , the control logic uses the BIN VALUE information to activate one or more sorting diverters along the waste recycling path to sort the coded plastic items into a certain bin. Consider an example: Brand X produces three different types of plastic containers, including container A (encoded with GTIN A), container B (encoded with GTIN B), and container C (encoded with GTIN C). Brand X is very interested in recycling these containers to help minimize material costs. Data structure 122 is updated ( Fig.36B) to include bin locations associated with the GTINs of container A, container B, and container C. The sorting unit decodes a digital watermark including GTIN B from the plastic items on the conveyor. The sorting unit uses GTIN B to query data structure 122 to discover the associated BIN VALUE, in this case "Brand X, Bin B". The control logic uses the BIN VALUE to activate "Sort Diverter - Brand X, Bin B" to place the corresponding plastic items in Bin B of BrandX. The control logic can use other data related to the recycling system, such as conveyor belt speed and the physical location of Brand X Bin B along the path to determine the timing for activating "Sort Diverter - Brand X, Bin B". If no BIN VALUE is associated with a particular GTIN, the corresponding plastic container can be classified according to material type, subtype, or other information included in data structure 122. The decoded watermark data and related classification events can be recorded to provide statistical data about the waste stream being processed.

[0418] From the above, it should be recognized that the technical problem is a binary level sorting (e.g., coded or uncoded). However, using the currently disclosed technology, N-ary sorting (or individually addressable sorting) can be achieved. This detailed level of sorting can achieve container-specific recycling, which helps to ensure material purity and reduce the use of virgin raw materials.

[0419] Fig.37 Graphically summarizes the Fig.32A and 32B The ecosystem of recycling systems shown in Figure 1 clearly sets out the elements of the container life cycle, including sending well-sorted bales into specialized composite reprocessing, ultimately producing recycled-derived supply, ready to compete with or offset virgin material supply.

[0420] The illustrated recycling system can increase knowledge about how to design products made of composite or multi-layer materials for reuse and recycling ("design for circularity"). Another benefit is increased understanding of the overall environmental footprint of the container, including the net impact on greenhouse gas emissions, and improved sorting, separation and recycling of composite and multi-layer materials.

[0421] Conclusion

[0422] Having described and illustrated the principles of our inventive work with reference to illustrative examples, it should be recognized that the technology is not limited thereto.

[0423] For example, while the described embodiments relate to images captured using visible illumination, this is not required. Other forms of illumination, such as ultraviolet and infrared, may instead be used.

[0424] Although the plastic bottle is described as including both a printed watermark and a textured watermark, it should be recognized that certain current technologies constitute improvements over textured watermarks - regardless of the presence or absence of a printed watermark. Thus, for example, a plastic bottle encoded with recycling information using a pattern like that shown in the figure is an improvement over prior art markings of plastic containers with recycling data (which tend to be more conspicuous and detract from the aesthetics of the package). Similarly, other improvements are generally applicable to watermarks, such as detailed clues to distinguish between empty and non-empty conveyors.

[0425] Although the printed and textured watermarks in various detailed embodiments employ reference signals that include peaks at different spatial frequencies, this is not necessary to avoid aliasing. In other embodiments, both watermarks employ reference signals that include peaks at the same spatial frequencies, in which case other properties of their protocols may be used to distinguish the watermarks (e.g., by a POS terminal). For example, a version bit string encoded in the variable data may be used to distinguish between a printed label watermark and a textured plastic watermark. (In an exemplary signaling protocol, a 1024-bit message string is formed as (a) a 100-bit string indicating the protocol version, followed by (b) a 924-bit string based on 47 bits of payload data. These latter bits are formed by concatenating the 47 bits of payload data with 24 corresponding CRC bits. These 71 bits are then convolutionally encoded at a rate of 1 / 13 to produce 924 bits. Thus, the bit string indicating the protocol version represents approximately 10% of the signal energy.) Alternatively, the reference signal of one watermark may employ peaks at spatial frequencies that are a subset of the peaks used in the reference signal of the other watermark.

[0426] If the reference signals of two watermarks share some or all of the spatial frequency peaks, the peaks in one reference signal can be assigned a different phase than the peaks in the other reference signal to avoid confusion. If distinguishing the two watermarks by peak phase does not produce enough false positive behavior, additional tests can be performed. For example, the phase can be checked twice on two different counterparts of the captured image. These counterparts can be consecutive image frames, or they can be a single image frame that is processed to produce two images. For example, Gaussian noise can be added to produce the second image. Or the second image can be generated by discarding pixel data from even rows and columns from the first image. There are many such possibilities. The result is only credible if the two phase-based identifications of the watermark signals from the two counterpart images are consistent.

[0427] In still other embodiments, the two watermarks use different scrambling keys, or different expansion keys, or different scattering tables to avoid confusion.

[0428] In embodiments where the reference signals of the two watermarks share common spatial frequency peaks, the processing arrangement can be simplified. For example, the scale and rotation synchronization of the two watermarks can be performed by a common processing stage, since such synchronization involves a set of common reference signal peaks. For example, this is described in detail in patent application 62 / 834,260 filed on April 15, 2019 and patent application 62 / 834,657 filed on April 16, 2019.

[0429] In one particular detector, a "supergrid" reference signal construction is employed that includes all peaks from both reference signals. The scale and rotation of the input image is determined by synchronization with such a composite reference signal. Once such synchronization is achieved, it is straightforward to determine whether the input image includes one reference signal or the other, for example by examining a peak or phase that is unique to one of the two reference signals.

[0430] Although the described embodiments employ a reference signal consisting of peaks in the Fourier magnitude domain, it will be appreciated that the reference signal may exhibit peaks in a different transform domain.

[0431] Relatedly, the digital watermark signal does not have to include different reference signals for geometric synchronization purposes.Sometimes, the payload portion of the watermark signal itself has known aspects or structures that allow geometric synchronization without relying on separate reference signals.

[0432] The term "watermark" generally refers to a mark that escapes human notice, i.e., is steganographic. While steganographic watermarks may be advantageous, they are not essential. Watermarks that form an obvious, human-noticeable pattern may be employed in embodiments of the present technology.

[0433] Although from Fig.13 The GTIN payload data field of the label watermark in the embodiment is used to access information such as the corresponding plastic type from the database, but this is not required. Other fields of the label watermark can be used for this purpose, for example, to elaborate on the various data types referenced herein. In fact, it is not necessary to use a database in conjunction with the label watermark; the payload can directly convey the plastic data, such as in one of the application identifier key-value pairs.

[0434] Similarly, while GTIN information is typically encoded only in the label watermark, in some embodiments, the plastic texture watermark may also encode this information. In this case, information about the plastic part or destination bin may be obtained by using a data structure (such as table 121) that associates the GTIN with such other information.

[0435] Although this specification specifically describes a 2D image sensor, a 2D sensor is not required. Image sensing may instead be performed by a linear array sensor that captures line scan images at a suitably high rate.

[0436] Although some of the figures show surface shapes that are primarily straight lines, this is in part a cartographic expediency. Typically, surface texturing affects curved, tapered shapes.

[0437] Reference is made to processing patches of captured images of the dimensions specified in the watermark element. While the exact watermark element size of a patch cannot be determined prior to evaluating its scale (e.g., using the referenced direct least squares method), the encoded scale of each watermark that the system may encounter is known in advance, and the imaging distance is fixed, so the scale correspondence between captured pixels and encoded watermark elements is approximately known, which is sufficient for present purposes.

[0438] As described in the cited references, the image processing described herein is generally performed on data that has previously been "eight-axis" (or "criss-cross") filtered. In an illustrative embodiment, the eight-axis filtered data may have any integer value in the range of -8 to 8.

[0439] In some embodiments, after the affine parameters characterizing the pose of the object have been identified, an estimate of the reference signal is subtracted from the captured image, since the reference signal then serves only as noise. Eight-axis processing can then be applied to the remaining signal.

[0440] Although this specification repeatedly references plastic bottles, it will be appreciated that the technology may be used in conjunction with any article such as trays, pouches, cups, shipping containers, and the like.

[0441] Furthermore, while this specification focuses on recycling, it should be understood that the same technology can be used to sort plastic and other containers for reuse. For example, a beverage producer could serialize its bottles through texturing, with each bottle having a unique identifier. When a customer returns a bottle for reuse, a processing line that cleans and refills the bottles could also use this technology to sense the serialized identifier and increment a counter to track the number of times the bottle has been processed for reuse. When a bottle reaches an empirically determined useful life (e.g., after 30 uses), it can be turned over for recycling.

[0442] To achieve optimal diverter performance, the center of gravity of the detected plastic object is estimated and this location is used to control the operation of the diverter mechanism (e.g., the location is the target of a compressed air jet). Each detection of a watermark block is used as a data point for estimating the center of gravity. In one example, the center coordinates of each watermark block are determined in the coordinate system of the image of the detection block. These coordinates in the x and y directions are averaged separately to produce the center of gravity of the object within that image frame. For example, in Fig.31 The average of these coordinates indicates the location shown by the bullseye. The spatial relationship between the camera field of view and the diverter assembly is known, as is the speed of the conveyor, allowing the diverter to be activated instantaneously and aimed at a location calculated to best divert items from the conveyor.

[0443] (If the conveyor belt is crowded with objects, the watermark blocks can be checked for payload consistency before averaging the positions. If one watermark represents one type of plastic, and a nearby block represents a different type of plastic, then they are said to mark different items, and their coordinates should not be used in a common average.)

[0444] The object images may also be submitted to a convolutional neural network that has been trained to classify input images as depicting objects belonging to one of a limited number of categories, such as bottles or flat objects (e.g., padded plastic shipping envelopes). The pressure or direction of the air exhausted from the air jet diverter is preferably controlled based on this classification to help ensure that the object is properly diverted. For example, a flat object may be used as a sail to capture air, so that less air is used to divert the flat object than to divert a bottle (whose curved surface would typically divert air around the bottle).

[0445] There is a short time interval between the moment an item is imaged by one or more cameras and the moment the item is positioned for transfer from a conveyor. This interval is typically sufficient to enable cloud processing. For example, the captured images (or derivatives of such images) can be sent to a remote cloud computer service, such as Microsoft Azure, Google Cloud, Amazon AWS. One or more cloud processors can perform some or all of the processing detailed herein and return the resulting data to the waste treatment system, controlling the diverter accordingly and in a timely manner.

[0446] Likewise, where some items in the waste stream do contain a plastic recycling code in their payload, while the recycling code for other items must be obtained from a database (e.g., based on a lookup from a decoded GTIN identifier), the short time interval before the transfer allows time to consult the cloud database to learn the required recycling code for the following item.

[0447] It should be appreciated that a recycling system employing aspects of the present technology does not require a conveyor belt per se. For example, the article may be transported by a camera system to a diverting system, such as by rollers or by free fall. All such alternatives should be included in the term "conveyor belt".

[0448] While reference is made to processing a captured image using an unsharp mask filter, other filters (linear or non-linear) may similarly be used to emphasize high frequency components of an image (or similarly, de-emphasize low frequency components).

[0449] Although most detailed arrangements operate using grayscale images, the greater dimensionality of multi-channel images can achieve certain performance improvements (e.g., more reliable identification of empty conveyor belts and certain watermark decoding modes). As mentioned previously, an RGB sensor can be used. However, half of the pixels in an RGB sensor are typically green filtered (due to the prevalence of common Bayer color filters). Better results can be obtained using a sensor that outputs four (or more) different data channels (such as R / G / B / ultraviolet, or R / G / B / infrared, or R / G / B / polarized, or R / G / B / white).

[0450] Although described in the context of plastic articles, it will be appreciated that many aspects of the technology are applicable to other articles, such as articles made of glass or metal.

[0451] Similarly, while the technology is described in the context of digital watermarks, it should be appreciated that any other machine-readable markings may be used, such as DotCode and dot pin markings (although certain benefits, such as readability from different angles, may be compromised). Document US 8,727,220 teaches twenty different 2D codes that may be embossed or molded into the outer surface of a plastic container.

[0452] As noted, image blocks in some embodiments are analyzed for clues suggesting whether the block depicts a conveyor belt. If not, further analysis is performed, such as analyzing the block for a watermark reference signal. In other embodiments, the block is initially analyzed for the presence of a watermark reference signal, and the detection of such a reference signal is used as a clue. Such reference signal detection triggers further analysis, such as analysis of blocks for payload data, and / or analysis of neighboring blocks or spatially displaced blocks in subsequent image frames for reference signals. (Typically, the type of reference signal detected indicates whether the associated watermark is likely to be of the print or texture variety, so that a corresponding decoding algorithm can be applied.)

[0453] As described above, certain embodiments of the present technology contemplate two watermarks that differ in three aspects: form, payload, and signaling protocol. For the avoidance of doubt, it should be understood that each of these attributes is different. The form of the two watermarks may be different (printed vs. textured), but the signaling protocol and payload are the same. Similarly, the payload of the two watermarks may be different, but the form and signaling protocol are the same. Similarly, the two watermarks may differ in the signaling protocol, but be the same in form and payload. (The signaling protocol covers all aspects of the watermark except its form and payload, including, for example, reference signals, encoding algorithms, output data formats, payload length, syntax, etc.)

[0454] Although watermark tiles of square shape are often mentioned, it will be appreciated that a printed or textured surface may be tiled with watermark tiles of other shapes as well. For example, a hexagonal honeycomb shape may be composed of triangular watermark elements.

[0455] Although the technique is described with reference to detecting a watermark synchronization (reference) signal using direct least squares and phase deviation methods, other techniques may be used. One example is a coiled full-pose arrangement, as detailed in patent publication 20190266749. Another option is to use a pulse-matched filter approach (e.g., correlated with a template consisting of peaks), as detailed in patent documents US10,242,434 and US6,590,996.

[0456] It should be recognized that treating a surface to achieve a matte or frosted finish is a form of 3D surface shaping / texturing, albeit on a small scale. In general, any inkless treatment that changes the bidirectional reflectance distribution function (BDRF) of a surface is considered a 3D shaping / texturing operation herein.

[0457] Although for example Figures 1H-1L The surfaces shown in the drawings are sections of spheres, and / or the cross sections are symmetrical, but neither is required. Sections of ellipsoids and more complex (higher order) surfaces may be more generally employed. Some such surfaces, when sectioned by a plane perpendicular to the nominal surface of the plastic article, may have an asymmetrical shape. In fact, some such surfaces are characterized in that they have no cross sections, perpendicular to the symmetrical nominal plastic surface.

[0458] Particular attention is paid to the earlier cited application 62 / 956,845. That application details the work of a different team from the current assignee, but deals with the same subject matter of recycling, etc. That application details the features, methods, and arrangements that the present applicant intends to incorporate into embodiments of the present technology. (Likewise, the present applicant intends to incorporate the features, methods, and arrangements of the present technology into embodiments of the '845 technology.) Thus, for example, identifying objects using deterministic and probabilistic methods, triggering object-specific analysis routines (e.g., contamination analysis), etc. are described in detail in the cited application and are also found in embodiments of the present technology. That application should be read in conjunction with the present application for a fuller understanding of the subject technology. (The application is not reproduced in its entirety here only to comply with the patent law requirement that the description must be "concise.")

[0459] It should be understood that the methods and algorithms detailed above can be performed using a computer device that employs one or more processors, one or more memories (e.g., RAM), storage (e.g., disk or flash memory), a user interface (which may include, for example, a keyboard, a TFT LCD or OLED display, a touch or other gesture sensor, and software instructions for providing a graphical user interface), interconnections between these elements (e.g., a bus), and a wired or wireless interface for communicating with other devices.

[0460] The methods and algorithms detailed above can be implemented in a variety of different hardware processors, including microprocessors, ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays). A mixture of such arrangements can also be employed.

[0461] By microprocessor, the applicant refers to a specific structure, namely a multi-purpose, clock-driven integrated circuit that includes integer and floating point arithmetic logic units (ALUs), control logic, a set of registers, and scratchpad memory (aka cache memory), which are linked by a fixed bus interconnect. The control logic fetches instruction codes from external memory and initiates the sequence of operations required by the ALU to execute the instruction codes. The instruction codes are taken from a limited instruction vocabulary, which can be considered the native instruction set of the microprocessor.

[0462] A specific implementation of the above detailed process on a microprocessor, such as discerning affine pose parameters from a watermark reference signal in a captured image, or decoding watermark payload data, first involves defining a sequence of algorithmic operations in a high-level computer language, such as MatLab or C++ (sometimes referred to as source code), and then generating machine code (i.e., instructions in a native machine instruction set, sometimes referred to as object code) from the source code using a commercial compiler (such as the Intel C++ compiler). (Both source code and machine code are considered software instructions in this article.) The process is then executed by instructing the microprocessor to execute the compiled code.

[0463] Many microprocessors are now amalgamations of several simpler microprocessors, called "cores." This arrangement allows multiple operations to be performed in parallel. (Some elements, such as bus structures and caches, may be shared between cores.)

[0464] Examples of microprocessor architectures include the Intel Xeon, Atom, and Core-I series of devices, as well as various models from ARM and AMD. In many applications, they are attractive choices because they are off-the-shelf parts. Implementation does not require waiting for custom design / manufacturing.

[0465] Closely related to microprocessors are GPUs (graphics processing units). GPUs are similar to microprocessors because they include ALUs, control logic, registers, caches, and fixed bus interconnections. However, the native instruction set of GPUs is usually optimized for image / video processing tasks, such as moving large blocks of data in and out of memory, and performing the same operations on multiple groups of data simultaneously. Other special tasks are usually also supported, such as rotating and translating vertex data arrays to different coordinate systems and interpolation. Leading suppliers of GPU hardware include Nvidia, ATI / AMD, and Intel. As used herein, the applicant intends to mention that microprocessors also include GPUs.

[0466] Due to the nature of the data being processed and the opportunities for parallelism, GPUs are an attractive architectural choice for executing certain detailed algorithms.

[0467] While a microprocessor can be reprogrammed with suitable software to perform a variety of different algorithms, an ASIC cannot. While a particular Intel microprocessor can be programmed today to discern affine pose parameters from a watermark reference signal, and programmed tomorrow to prepare a user's tax return, the ASIC structure does not allow for this flexibility. Instead, an ASIC is designed and built to serve a specialized task. It is purpose-built.

[0468] An ASIC structure consists of a series of circuits that are custom designed to perform a specific function. There are two major categories: gate arrays (sometimes called semi-custom) and full-custom. In the former, the hardware consists of a regular array of millions of digital logic gates (e.g., XOR and / or AND gates) fabricated in diffusion layers and spread across a silicon substrate. Metallization layers that define custom interconnects are then applied, which permanently link certain gates in a fixed topology. (A consequence of this hardware structure is that many of the fabricated gates—often the majority—are typically unused.)

[0469] However, in a full-custom ASIC, the arrangement of gates is custom designed to meet the intended purpose (e.g., to execute a specified algorithm). Custom designs make more efficient use of available substrate space, allowing for shorter signal paths and higher speed performance. Full-custom ASICs can also be manufactured to include analog components and other circuits.

[0470] Generally speaking, ASIC-based implementations of watermark detectors and decoders provide higher performance and consume less power than microprocessor-based implementations. However, the disadvantage is that it takes a lot of time and money to design and manufacture circuits tailored for a specific application.

[0471] A particular implementation of any of the above processes using an ASIC (e.g., for discerning affine pose parameters or decoding watermark payload data from a watermark reference signal in a captured image) again begins with defining a sequence of operations in source code such as MatLab or C++. However, rather than compiling to the native instruction set of a multi-purpose microprocessor, the source code is compiled into a "hardware description language" such as VHDL (an IEEE standard) using a compiler such as HDL Coder (available from MathWorks). The VHDL output is then applied to a hardware synthesis program such as Synopsis' Design Compiler, Mentor Graphics' HDL Designer, or Cadence Design Systems' Encounter RTL Compiler. The hardware synthesis program provides output data that specifies a specific array of electronic logic gates that will implement the technology in hardware as a dedicated machine dedicated to this purpose. This output data is then provided to a semiconductor manufacturing contractor, which uses it to produce custom silicon parts. (Suitable contractors include TSMC, Global Foundries, and ON Semiconductors.)

[0472] A third hardware structure that can be used to execute the algorithm detailed above is the FPGA. The FPGA is a cousin of the semi-custom gate array discussed above. However, instead of using metallization layers to define fixed interconnections between generic gate arrays, the interconnections are defined by a network of switches that can be electrically configured (and reconfigured) to be on or off. The configuration data is stored in and read from external memory. With such an arrangement, the linkage of logic gates, and therefore the functionality of the circuit, can be changed at will by loading different configuration instructions from memory that reconfigure how these interconnect switches are set.

[0473] FPGAs also differ from semi-custom gate arrays in that they are not typically composed entirely of simple gates. Instead, FPGAs can include some logic elements configured to perform complex combinational functions. In addition, memory elements (e.g., flip-flops, but more typically full blocks of RAM memory) can be included. The same is true for A / D and D / A converters. Again, the reconfigurable interconnects that characterize FPGAs enable such additional elements to be incorporated where needed in the larger circuit.

[0474] Examples of FPGA architectures include Intel's Stratix FPGAs and Xilinx's Spartan FPGAs.

[0475] As with other hardware structures, implementing the detailed processes described above on an FPGA begins with describing the processes in a high-level language. And, as with ASIC implementations, the high-level language is then compiled into VHDL. However, the interconnect configuration instructions are generated from the VHDL by software tools specific to the FPGA family being used (e.g., Stratix / Spartan).

[0476] A mix of the foregoing structures can also be used to execute detailed algorithms. One employs a microprocessor integrated on a substrate as a component of an ASIC. This arrangement is known as a system on a chip (SOC). Similarly, the microprocessor can be one of the components that can be used to reconfigure interconnections with other components in an FPGA. This arrangement can be known as a system on a programmable chip (SORC).

[0477] Another type of processor hardware is neural network chips, such as the Intel Nervana NNP-T, NNP-I, and Loihi chips, the Google Edge TPU chip, and the Brainchip Akida neuromorphic SOC.

[0478] Software instructions for implementing detailed functions on the selected hardware can be written by a technician without undue experimentation based on the description provided in this article in combination with relevant data, for example, using C, C++, Visual Basic, Java, Python, Tcl, Perl, Scheme, Ruby, Caffe, TensorFlow, etc.

[0479] Software and hardware configuration data / instructions are typically stored as instructions in one or more data structures conveyed by a tangible medium such as a magnetic or optical disk, memory card, ROM, etc. accessible via a network. Some embodiments may be implemented as an embedded system, a special-purpose computer system in which the operating system software and application software are indistinguishable to the user (e.g., as is typically the case in a basic cellular telephone). The functions detailed in this specification may be implemented in operating system software, application software, and / or embedded system software.

[0480] Different functions may be implemented on different devices. Different tasks may be performed exclusively by one or another device, or execution may be distributed among devices. Likewise, descriptions of data stored on specific devices are exemplary; data may be stored anywhere: on a local device, on a remote device, in the cloud, distributed, etc.

[0481] Other recycling arrangements are taught in patent documents US4644151, US5965858, US6390368, US20060070928, US20140305851, US20140365381, US20170225199, US20180056336, US20180065155, US20180349864, and US20190030571. Alternative embodiments of the present technology employ features and arrangements from these references.

[0482] Various embodiments have been discussed in this specification. It should be understood that the methods, elements and concepts described in detail in conjunction with one embodiment can be combined with the methods, elements and concepts described in detail in conjunction with other embodiments. Although some such arrangements have been particularly described, many have not been described due to the number of permutations and combinations. The applicant similarly recognizes and intends to combine, replace and interchange the methods, elements and concepts of this specification, not only among and between them, but also with those combinations, replacements and interchanges known from the cited prior art. In addition, it should be recognized that detailed technology can be included with other technologies (current and forthcoming) to achieve favorable effects. According to the teachings provided in this disclosure, the implementation of this combination is direct to the technician.

[0483] Although this disclosure details a particular order of acts and particular combinations of elements, it should be recognized that other conceived methods may reorder the acts (perhaps omitting some and adding others), and that other conceived combinations may omit some elements and add others, and so on.

[0484] Although disclosed as a complete system, sub-combinations of detailed arrangements are also contemplated separately (eg, omitting various features of the complete system).

[0485] Although certain aspects of the present technology have been described with reference to illustrative methods, it should be appreciated that devices configured to perform the acts of these methods are also contemplated as part of the inventive work of the present applicant. Likewise, other aspects have been described with reference to illustrative devices, and methods performed by such devices are also within the scope of the present technology. Still further, tangible computer-readable media containing instructions for configuring a processor or other programmable system to perform such methods are also expressly contemplated.

[0486] In order to provide a comprehensive disclosure while meeting the patent law's requirement for brevity, the applicant incorporates by reference each document cited herein. (These materials are incorporated in their entirety, even if their specific teachings are cited above.) These references disclose the techniques and teachings that the applicant intends to incorporate into the arrangements detailed herein, and the techniques and teachings currently detailed are incorporated therein.

[0487] In view of the variety of embodiments to which the above principles and features may be applied, it is apparent that the embodiments described in detail are illustrative only and should not be taken as limiting the scope of the invention.

Claims

1. A method, comprising the following acts: In an electronic document, defining a data pattern, the data pattern comprising spaced binary elements at positions within a regular 2D position grid, the pattern defining a first fixed reference signal and a first variable data signal, the first fixed reference signal facilitating geometric registration and extraction of the first variable data signal by a decoder, the first variable data signal being presented as an image captured by a camera depicting a physical counterpart of the data pattern, the 2D position grid comprising an N×N position grid, each position grid having a grayscale or floating-point value of a first fixed reference signal corresponding thereto, the first variable data signal comprising an M×M position array, where M < N, each position in the M×M position array having a bistable value of a variable data signal corresponding thereto, and the defining the data pattern comprising: interpolating M×M first variable data signal values ​​to produce an N×N array of interpolated values, and in doing so converting the first variable data signal from a bitonal form to a grayscale or floating point form; summing corresponding values ​​of the first fixed reference signal and the interpolated first variable data signal at each of N×N positions with a weighted ratio to produce a summed array of N×N values; identifying a subset of the N×N positions in the summed array of values ​​having extreme values; as well as marking a subset of the positions so as to produce a data pattern of the binary elements; The method further includes shaping a 3D surface topological pattern of the mold according to a smooth counterpart of the data pattern, the topological pattern including peaks or valleys having a smooth cross-section to facilitate demolding of the molded part from the mold.

2. The method according to claim 1, wherein, The subset of the NxN positions includes 25% or less of the NxN positions.

3. The method according to claim 1, further comprising shaping a plastic container using a shaping die, wherein, The formed plastic of the container conveys the first variable data signal.

4. A plastic container shaped using the method according to claim 3.

5. A die produced by the method according to claim 1.

6. A method, comprising the following acts: In an electronic document, defining a data pattern, the data pattern comprising spaced binary elements at positions within a regular 2D position grid, the pattern defining a first fixed reference signal and a first variable data signal, the first fixed reference signal facilitating geometric registration and extraction of the first variable data signal by a decoder, the first variable data signal being presented as an image captured by a camera depicting a physical counterpart of the data pattern, wherein; The 2D position grid defines M positions, each position having a value of the first fixed reference signal corresponding thereto, each value representing a relative darkness of the reference signal at the position; The first variable data signal comprises binary symbols, each symbol being associated with a corresponding position within the regular 2D grid of positions; And the method includes generating the data pattern by: sorting the values ​​of the first fixed reference signal to produce a ranking of the N darkest locations; Identify the P darkest positions using the ranking of the N darkest positions, leaving Q other positions, and label each of these P positions with a binary element; as well as marking or unmarking each of the Q other positions with a binary element depending on whether a corresponding binary symbol of the variable data signal has a first value or a second value; The method further includes shaping a 3D surface topological pattern of the mold according to a smooth counterpart of the data pattern, the topological pattern including peaks or valleys having a smooth cross-section to facilitate demolding of the molded part from the mold.

7. A plastic container having a surface shaped with a smooth 3D counterpart to an N×N data pattern, the data pattern comprising a reference signal and a variable data signal, the reference signal facilitating geometric registration and extraction of the variable data signal by a decoder, the variable data signal being presented as an image captured by a camera depicting the surface, the data pattern comprising a selection of locations marked with 3D bumps or depressions within a regular 2D grid of N×N locations on the container surface, the selection of the locations of the markings corresponding to the locations of extreme values ​​in a weighted sum of the following signals: (a) a fixed N×N reference signal consisting of grayscale or floating point values, and (b) a binary M×M variable data signal that has been interpolated to produce an N×N array of grayscale or floating point values, wherein M <N。 8. The container according to claim 7, wherein: The selection of locations marked with 3D bumps or depressions includes 25% or less of the N×N locations.

9. The container according to claim 7, wherein: The other 75% or more of the N x N locations follow the nominal contour of the plastic container.

10. The plastic container according to claim 7, wherein: Within the boundaries of the data pattern, a large portion of the container surface area follows the nominal contour of the container.

11. A method of marking a container to convey a multi-symbol payload, comprising the following acts: generating a data pattern encoding the payload, the pattern comprising spaced apart elements at positions within a regular 2D grid of positions, the pattern defining a fixed reference signal and a variable data signal, the reference signal facilitating geometric registration and extraction of the variable data signal by a decoder, the variable data signal being presented as a camera captured image depicting a physical counterpart to the data pattern; as well as forming a physical counterpart of the pattern on the container by printing or texturing; in, The 2D grid of positions defines J positions, each position having a value of the reference signal corresponding thereto, each value representing a relative darkness of the reference signal at that position, and the variable data signal comprises binary symbols, each symbol being associated with a corresponding position within the regular 2D grid of positions; The generation behavior more specifically includes: sorting the values ​​of the reference signal to produce a ranking of K darkest locations; Using the ranking of the K darkest positions, identify the P darkest positions, leaving Q other positions, and label each of the P positions with a binary element; and Each of the Q other positions is marked or unmarked with a binary element depending on whether the corresponding binary symbol of the variable data signal has the first value or the second value.

12. A container produced by the method of claim 11, said container conveying said data pattern by texturing.

13. The method according to claim 11, wherein: The physical counterpart of the pattern formed on the container comprises a thermoformed plastic material.

14. The method according to claim 11, wherein: Forming a physical counterpart of the pattern on the container may include laser etching.

15. A plastic container having a surface, the surface comprising a plurality of raised bumps and / or recessed pit markings, the markings defining a plurality of first pattern blocks, the plurality of first pattern blocks being tiled edge to edge in a repeating array on the surface, each first pattern block comprising a plurality of first pattern blocks defining M=N 2 An N×N array of candidate positions, the bump and / or indentation marks can be located at M=N 2 candidate locations, where: The raised and / or recessed marks are located at 25% or less of the M candidate positions in each pattern block; and 75% or more of the M candidate positions in each pattern block follow the nominal contour of the plastic container and are smooth; Wherein, the plastic container also includes a printed label, wherein the raised bumps and / or recessed pit marks encode a first watermark using a first signaling protocol, and wherein the printing on the label encodes a second watermark using a second signaling protocol different from the first signaling protocol, and wherein both the first watermark and the second watermark are used to identify the plastic composition of the container.

16. The plastic container of claim 15, wherein the marks are located at 20% or less of the M candidate positions in each pattern block, and 80% or more of the M candidate positions in each pattern block follow the nominal contour of the plastic container and are smooth.

17. The plastic container of claim 15, wherein the marks are located at 10% or less of the M candidate positions in each pattern block, and 90% or more of the M candidate positions in each pattern block follow the nominal contour of the plastic container and are smooth.

18. The plastic container of claim 15, wherein the container comprises a beverage bottle.

19. The plastic container of claim 15, wherein each pattern block encodes a multi-symbol payload and a reference signal, the reference signal comprising a constellation of multiple peaks in the spatial frequency domain.

20. A mold having a surface pattern formed to produce the plastic container according to claim 15 by molding.

21. A plastic container having a surface, the surface comprising a plurality of light scattering texture marks, the marks defining a plurality of first pattern blocks, the plurality of first pattern blocks being tiled edge to edge in a repeating array on the surface, each first pattern block comprising a plurality of light scattering texture marks defining a plurality of first pattern blocks, each first pattern block defining ... 2 The light scattering texture mark can be located at M=N N×N position arrays of candidate positions. 2 At candidate positions, the light scattering texture mark does not interfere with the nominal shape of the container, wherein: The light scattering texture marks are located at 25% or less of the M candidate positions in each pattern block; and 75% or more of the M candidate positions in each pattern block follow the nominal contour of the plastic container and are smooth rather than having a light scattering texture; The plastic container also includes a printed label, wherein the light scattering texture mark encodes a first watermark using a first signaling protocol, and wherein the printing on the label encodes a second watermark using a second signaling protocol different from the first signaling protocol, and wherein the first watermark and the second watermark are both used to identify the plastic composition of the container.

Images