Reading optically readable codes

Abstract

The present invention relates to the technical field of marking an article with an optically readable code and decoding the code. Such a code is introduced on the surface of the article. The code is decoded based on a captured image of a transformed code. The transformed captured image is generated from at least one captured image of the code using a machine learning model. The model is trained to generate the transformed captured image from the at least one captured image, such that reading the optically readable code results in fewer decoding errors than reading the code in the at least one captured image. The present invention relates to a method for training a machine learning model, as well as a method, system, and computer program product for decoding a code using a trained machine learning model.

Description

[Technical field]

[0001] The present invention relates to the technical field of marking objects with optically readable codes and reading (decoding) the codes. [Background technology]

[0002] Tracking of goods and products plays an important role in many sectors of the economy. Products and goods or their packaging and / or containers are provided with unique identifiers (e.g., serial numbers) to enable them to be tracked along their supply chains and to allow machine capture of incoming and outgoing products (serialization) (see, e.g., U.S. Pat. No. 10,140,492, China Patent Publication No. 112434544).

[0003] Identifiers are often applied to products and goods or their packaging and / or containers in an optically readable form, for example in the form of a barcode (e.g. an EAN-8 barcode) or a matrix code (e.g. a QR Code®). The barcode or matrix code often conceals a serial number that typically provides information about the type and origin of the product or good.

[0004] For some medicines, even individual marking of the individual packs is required: according to Article 54a(1) of Directive 2001 / 83 / EC, as amended by the EU Fake Medicines Directive 2011 / 62 / EU (FMD), at least medicines that are the subject of a prescription should be marked with an individual recognition feature (called a unique identifier) ​​which makes it possible, in particular, to check the authenticity and to identify the individual packs.

[0005] Plant and animal products are usually only provided with a serial code. In the case of plant and animal products, the marking is usually applied or attached to the packaging and / or container, for example in the form of a sticker or stamp.

[0006] In some cases, identifiers are also applied directly to plant or animal products: for example, in the European Union, eggs are provided with a producer code from which the poultry farming industry, the country of origin of the eggs, and the company of origin of the eggs can be derived. The skins of fruit and vegetable units are also increasingly being marked (see, for example, E. Etxeberria et al., Anatomical and Morphological Characteristics of Laser Etching Depressions for Fruit Labeling, 2006, HortTechnology. 16, 10.21273 / HORTTECH.16.3.0527).

[0007] Consumers are becoming increasingly interested in the origin and supply chain of plant and animal products: they want to know, for example, where the respective product comes from, whether and how it has been treated (e.g. with crop protection compositions), how long the transport lasted, what conditions prevailed during transport, etc.

[0008] EP 3896629 proposes to provide plant and animal products with a unique identifier in the form of an optically readable code. The code can be read by the consumer, for example via the camera of his smartphone. Based on the read code, various information about the plant or animal product can be displayed to the consumer. EP 3896629 proposes to introduce an optically readable code into the surface of the plant or animal product, for example by means of a laser. An optical code introduced into the surface of the plant or animal product has a lower contrast with the surrounding tissue and is therefore more difficult to read than an optical code applied in black on a white background, as is typically the case, for example, for stickers or tags. Furthermore, optical codes on different objects have different appearances. For example, if an optically readable code is applied by a laser to a curved surface, as is the case for many fruit and vegetable varieties, distortions of the code may occur, preventing reading. If an optically readable code is applied to a smooth surface, reflections may occur at the surface during reading (for example as a result of ambient light), preventing reading. The surfaces of fruits and vegetables may be uneven. For example, apples can have bitter pits and spots, potatoes can have bumps, etc. Such irregularities and non-uniformity can prevent the code from being read. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] U.S. Pat. No. 10,140,492 [Patent Document 2] China Patent Application Publication No. 112434544 [Patent Document 3] European Patent No. 3896629 [Non-patent literature]

[0010] [Non-Patent Document 1] E. Etxeberria et al. Anatomical and Morphological Characteristics of Laser Etching Depressions for Fruit Labeling, 2006, HortTechnology.16, 10.21273 / HORTTECH.16.3.0527 Summary of the Invention [Problem to be solved by the invention]

[0011] The problem addressed is therefore to provide a means by which consumers can reliably read optically readable codes on many different objects, particularly many plant and animal products, using simple means such as, for example, a smartphone. [Means for solving the problem]

[0012] This problem is solved by the subject matter of the independent claims. Preferred embodiments can be found in the dependent claims, the description and the drawings.

[0013] A first subject of the present invention is a method for training a machine learning model, the method comprising the steps of: - providing training data, the training data for each object of the plurality of objects comprising i) at least one reference image record of an optical code to be introduced into a surface of the object, and ii) a transformed reference image record of the optical code; - providing a machine learning model, the machine learning model being configured to generate a second image record based on the at least one first image record and model parameters; - training a machine learning model, the training for each object of the plurality of objects comprising: inputting at least one reference image record into said machine learning model; receiving a predicted transformed reference image record from a machine learning model; - calculating the deviation between the transformed reference image record and the predicted transformed reference image record; modifying the model parameters in relation to reducing the deviation; - storing and / or outputting the trained machine learning model, and / or transmitting the trained machine learning model to a separate computer system, and / or using the trained machine learning model to generate a transformed image record of at least one new image record of the object.

[0014] A further subject of the invention is a computer-implemented method for decoding an optically readable code introduced into a surface of an object, the method comprising the steps of: - receiving an image recording of an optically readable code; - providing the image records to a trained machine learning model, the trained machine learning model having been trained using training data, the training data for each object of the plurality of objects comprising i) at least one reference image record of an optical code to be introduced onto a surface of the object, and ii) a transformed reference image record of the optical code, the step of decoding the optical code on the transformed reference image record producing fewer decoding errors than decoding the optical code on the reference image record, and the training for each object of the plurality of objects comprises: inputting at least one reference image record into a machine learning model; receiving a predicted transformed reference image record from a machine learning model; - calculating the deviation between the transformed reference image record and the predicted transformed reference image record; modifying the model parameters in relation to reducing the deviation; receiving a transformed image record from a machine learning model; - decoding the optically readable code imaged in the transformed image record.

[0015] A further subject of the invention is a system comprising at least one processor, the processor comprising: - receiving an image recording of an optically readable code introduced into a surface of an object; - providing the image records to a trained machine learning model, the trained machine learning model being trained using training data, the training data for each object of the plurality of objects comprising i) at least one reference image record of an optical code to be introduced onto a surface of the object, and ii) a transformed reference image record of the optical code, wherein the step of decoding the optical code on the transformed reference image record produces fewer decoding errors than decoding the optical code on the reference image record, and the training for each object of the plurality of objects comprises: inputting at least one reference image record into a machine learning model; receiving a predicted transformed reference image record from a machine learning model; - calculating the deviation between the transformed reference image record and the predicted transformed reference image record; modifying the model parameters with respect to reducing the deviation, receiving a transformed image record from the machine learning model; - configured to decode an optically readable code imaged in the transformed image record.

[0016] A further subject of the invention is a computer program product comprising a data carrier on which a computer program is stored, the computer program being loadable into the main memory of a computer and configured to: - receiving an image recording of an optically readable code; - providing the image records to a trained machine learning model, the trained machine learning model having been trained using training data, the training data for each object of the plurality of objects comprising i) at least one reference image record of an optical code to be introduced onto a surface of the object, and ii) a transformed reference image record of the optical code, the step of decoding the optical code on the transformed reference image record producing fewer decoding errors than decoding the optical code on the reference image record, and the training for each object of the plurality of objects comprises: inputting at least one reference image record into a machine learning model; receiving a predicted transformed reference image record from a machine learning model; - calculating the deviation between the transformed reference image record and the predicted transformed reference image record; modifying the model parameters in relation to reducing the deviation; receiving a transformed image record from a machine learning model; - decoding the optically readable code imaged in the transformed image record. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 illustrates, by way of example, a schematic of a method for training a machine learning model. [Diagram 2] FIG. 2 illustrates, diagrammatically and by way of example in the form of a flow chart, the reading of an optically readable code introduced onto the surface of an object. [Diagram 3] FIG. 3 shows a schematic diagram of a system according to the invention by way of example. [Figure 4] FIG. 4 shows a schematic diagram of a computer system. [Diagram 5] FIG. 5 illustrates, by way of example, a schematic representation of the generation of a training dataset for training a machine learning model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The invention is described in more detail below without distinguishing between the subject matter of the invention (training method, decoding method, system, computer program product). Instead, the following descriptions shall apply equally to all subject matter of the invention, regardless of the context in which they are provided (training method, decoding method, system, computer program product).

[0019] If steps are described in a sequence in the present specification or claims, this does not necessarily mean that the invention is limited to the sequence described. Instead, steps may be performed in a different sequence or in parallel with each other, unless it is absolutely necessary that one step is built on another step and that the step built on the previous step is subsequently performed (however, this will become clear in each individual case). The sequence described is therefore a preferred embodiment of the present invention.

[0020] The present invention provides a means for reading (decoding) optically readable codes. The terms "read" and "decode" are used interchangeably herein.

[0021] The term "optically readable code" is understood to mean a marking that can be captured, for example by means of a camera, and converted into an alphanumeric character.

[0022] Examples of optically readable codes are bar codes, stacked codes, composite codes, matrix codes and 3D codes. Alphanumeric characters that can be captured (interpreted, read) and digitized by automatic text recognition (called optical character recognition, abbreviated OCR) also belong to the term optically readable codes.

[0023] Optically readable codes belong to the class of machine-readable codes, i.e. codes that can be captured and processed by a machine. In the case of optically readable codes, such a "machine" is usually equipped with a camera.

[0024] A camera usually comprises an image sensor and optical elements. The image sensor is a device that records a two-dimensional image from light by electrical means. It usually contains a semiconductor-based image sensor, for example a CCD (CCD = Charge Coupled Device) or a CMOS sensor (CMOS = Complementary Metal Oxide Semiconductor). The optical elements (lenses, stops, etc.) provide maximum clarity of the imaging of the object, from which a digital image record is produced on the image sensor.

[0025] Optically readable codes have the advantage that they can be read by many consumers using simple means. In this regard, many consumers have, for example, a smartphone equipped with one or more cameras. By means of such a camera, it is possible to generate an image representation of the optically readable code on an image sensor. The image representation can be digitized and processed and / or stored by a computer program stored in the smartphone. Such a computer program can be configured to identify and interpret the optically readable code, i.e. to convert the optically readable code into different forms, for example, a numeric string, a character string, etc., depending on what information is present in the form of the optically readable code.

[0026] The optically readable code is introduced on the surface of an object. Such an object is a real, physical, tangible object. Such an object can be a natural object or an industrially manufactured object. Examples of objects in the sense of the present invention are tools, machine parts, circuit boards, chips, containers, packaging, jewelry, design objects, art objects, pharmaceuticals (e.g. in the form of tablets), pharmaceutical packaging, and plant and animal products.

[0027] In one preferred embodiment of the invention, the object is a pharmaceutical product (eg, in the form of a tablet or capsule) or pharmaceutical packaging.

[0028] In a further preferred embodiment of the invention, the object is an industrially manufactured product, such as a part for a machine.

[0029] In a further preferred embodiment of the invention, the object is a plant or animal product.

[0030] A plant product is an individual plant or part of a plant (e.g., a fruit) or a group of plants or a group of plant parts. An animal product is an animal or part of an animal, or a group of animals or a group of animal parts, or an object produced by an animal (e.g., an egg from a hen). Industrially processed products, such as cheese and sausage products, are also intended to be included in the term plant or animal product.

[0031] A plant or animal product is typically a part of a plant or animal suitable and / or intended for human or animal consumption.

[0032] The plant product is preferably at least a part of a crop plant. The term "crop plant" is understood to mean a plant that is cultivated especially as a useful plant by human intervention. In a preferred embodiment, the crop plant is a fruit plant or a vegetable plant. Although fungi are not biologically considered plants, fungi, in particular fungal fruiting bodies, are also intended to fall within the term plant product.

[0033] Preferably, the crop plant is one of the plants listed in the following encyclopedia: Christopher Cumo: Encyclopedia of Cultivated Plants: From Acacia to Zinnia, volumes 1-3, ABC-CLIO, 2013, ISBN 9781598847758.

[0034] The plant or animal product can be, for example, apple, pear, lemon, orange, tangerine, lime, grapefruit, kiwi, banana, peach, plum, mirabelle plum, apricot, tomato, cabbage (such as cauliflower, white cabbage, red cabbage, kale, Brussels sprouts), melon, pumpkin, cucumber, pepper, zucchini, eggplant, potato, sweet potato, leek, celery, kohlrabi, radish, carrot, parsnip, scozonera, asparagus, sugar beet, rhubarb, ginger root, coconut, Brazil nut, walnut, hazelnut, chestnut, egg, fish, meat pieces, cheese pieces, sausage, and the like.

[0035] The optically readable code is introduced into the surface of the object. This means that the optically readable code is not attached to the object in the form of a tag, nor is it applied to the object in the form of a sticker. Instead, the surface of the object is modified so that the surface itself carries an optically readable code.

[0036] In the case of plant products and eggs, the surface may be, for example, the skin / shell.

[0037] The optically readable code may be engraved, etched, burned, imprinted, and / or introduced into the surface of the object in some other way. Preferably, the optically readable code is introduced into the surface of the object (e.g., the skin in the case of fruits or vegetables) by a laser. In this case, the laser may modify (e.g., bleach and / or destroy) pigment molecules on the surface of the object and / or cause instances of localized burning and / or destruction and / or chemical and / or physical modification of the tissue (e.g., evaporation of water, denaturation of proteins, etc.), thus creating a contrast with the surrounding parts of the surface (parts not modified by the laser).

[0038] Optically readable codes can also be introduced into the surface of an object by a water jet or a stream of sand.

[0039] Optically readable codes can also be mechanically introduced into the surface of an object by scribing, piercing, splitting, rubbing, scraping, punching, and the like.

[0040] Further details concerning the marking of objects, in particular plant or animal products, can be gleaned from the prior art (see, for example, EP 2281468, WO 2015 / 117438, WO 2015 / 117541, WO 2016 / 118962, WO 2016 / 118973, DE 102005019008, WO 2007 / 130968, US 5660747, EP 1737306, US 10481589, US 20080124433).

[0041] Preferably, the optically readable code is introduced into the surface of the object by means of a carbon dioxide laser (CO2 laser).

[0042] Introducing an optically readable code on the surface of an object usually results in a marking with a lower contrast than an optically readable code, for example, in the form of a black or colored inscription on a white sticker. The contrast can be optimized by the individual design possibilities of the sticker. As an example, a black marking on a white background (for example, a black barcode or a black matrix code) has a very high contrast. Such a high contrast is not usually achieved by introducing an optically readable code on the surface of an object, especially in the case of plant or animal products, and in the case of pharmaceutical products. This can lead to difficulties in reading the code. Furthermore, an optically readable code introduced on the surface of an object, for example by a laser, has a different appearance for different objects. In other words, an optically readable code introduced on an apple, for example, usually has a different appearance than an optically readable code introduced on a banana, a tomato, a pumpkin, or a potato. The respective variety of fruit can also affect the appearance of the optically readable code, an optically readable code on a "Granny Smith" apple has a different appearance than an optically readable code on a "Pink Lady" apple. The structure of the surface of the plant or animal product can also affect the appearance of the optically readable code, the relatively rough surface structure of a kiwi can prevent the reading of the optically readable code in exactly the same way as the irregularities and scratches on the skin of potatoes and apples. Fruit and vegetable surfaces usually have a curvature. When an optical code is introduced into a curved surface, distortion of the code can occur, the code can have, for example, a pincushion- or barrel-shaped distortion. Such distortion can prevent the decoding of the code. The degree of distortion usually depends on the degree of curvature in this case. If an optically readable code with a size of 2 cm x 2 cm is introduced into an apple, the distortion will be greater than if the same code is introduced into a melon. Furthermore, the distortions that occur in approximately spherical products (e.g. apples, tomatoes, melons) are different from those in approximately cylindrical products (e.g. cucumbers).A smooth surface (such as in the case of an apple) produces more reflection (eg, as a result of ambient light) than a rough surface (such as in the case of a kiwi).

[0043] The above-mentioned characteristics of the object (such as unevenness, non-uniformity, scratches, curved surfaces, smooth (specular) surfaces, coloration, texture, surface roughness, etc.) are also referred to herein as disturbance factors.

[0044] Thus, according to the invention, an image record of an optically readable code introduced into the surface of an object is subjected to a transformation or transformations before the code is read (decoded). Such transformations may increase the contrast between the optically readable code and the surrounding parts of the surface (parts of the surface that do not carry the code), and / or reduce or eliminate distortions, and / or reduce or eliminate reflections, and / or reduce or eliminate other artifacts due to the particular object present.

[0045] A "transform" is a function or operator that takes an image as input and produces an image as output. The transform can ensure that an optically readable code imaged in the input image has a higher contrast with its surroundings in the output image than in the input image and / or has less distortion in the output image than in the input image. The transform can ensure that light reflections on the object's surface are reduced in the output image compared to the input image. The transform can ensure that unevenness and / or non-uniformity on the object's surface appear less clearly in the output image than in the input image. In general, the transform ensures that the output image produces fewer decoding errors during decoding of the imaged optically readable code than the input image. Such decoding errors occur, for example, when a white square in a QR code is interpreted as a black square by an image sensor.

[0046] First, an image recording of an optically readable code introduced into the surface of the object is generated. It is also conceivable to generate multiple image recordings of the optically readable code.

[0047] The term "image recording" is preferably understood to mean a two-dimensional image representation of an object or a part thereof. The image recording is usually a digital image recording. The term "digital" means that the image recording can be processed by a machine, typically a computer system. "Processing" is understood to mean known methods for electronic data processing (EDP).

[0048] The digital image records may be processed, edited, and reproduced, for example, by computer systems and software, and may also be converted into standardized data formats such as JPEG, Portable Network Graphics (PNG) or Scalable Vector Graphics (SVG). The digital image records may be visualized by a suitable display device, such as, for example, a computer monitor, a projector, and / or a printer.

[0049] In digital image recordings, image content is usually represented and stored as integers. In most cases, this includes two-dimensional images that can be binary coded and optionally compressed. Digital image recordings usually include raster graphics, where image information is stored in a uniform raster grid. Raster graphics consist of a raster arrangement of so-called image elements (pixels) in the case of two-dimensional representations, or volume elements (voxels) in the case of three-dimensional representations, which are assigned in each case a color or grayscale value. The main characteristics of 2D raster graphics are therefore the image size (width and height measured in pixels, also informally called image resolution) and the color depth. Colors are usually assigned to the pixels of a digital image file. The color coding used for the pixels is defined, among other things, in terms of a color space and a color depth. The simplest case is a binary image, where the pixels store black and white values. In the case of an image whose color is defined in terms of the so-called RGB color space (RGB stands for the primary colors red, green and blue), each pixel consists of three color values, one color value red, one color value green and one color value blue. The color of a pixel results from the superposition (additive mixture) of the three color values. The individual color values ​​are discretized into, for example, 256 distinguishable levels, which are called tone values ​​and usually range from 0 to 255. The color nuance "0" of each color channel is the darkest. If all three channels have a tone value of 0, the corresponding pixel appears black, and if all three channels have a tone value of 255, the corresponding pixel appears white. When implementing the invention, the digital image record undergoes certain operations (transformations). In this case, the operations mainly concern the pixels as so-called spatial operators, such as, for example, edge detectors, or the tone values ​​of the individual pixels, as in the case of, for example, color space transformations. There are a large number of possible digital image formats and color codings. For simplicity, in this description, it is assumed that the current image is an RGB raster graphic with a certain number of pixels. However, this assumption should in no way be understood as limiting: it will be clear to one skilled in the art of image processing how the teachings herein can be applied to image files that exist in other image formats and / or in which the color values ​​are coded differently.

[0050] The at least one image recording may also be one or more excerpts from a video sequence.

[0051] The at least one image recording is produced by means of one or more cameras, preferably by one or more cameras of a smartphone.

[0052] The use of multiple cameras looking at an object from different directions and producing image records from different viewing directions has the advantage that depth information is captured. As an example, information regarding the curvature of the imaged object present can be derived and / or gleaned from such depth information.

[0053] At least one image record shows an optically readable code that has been introduced into a surface of the object.

[0054] In a next step a transformed image record is generated from the image record. It is possible that several image records are involved in generating the transformed image record.

[0055] A transformed image record is an image record that has undergone one or more transformations. A transformed image record may be one or more image records and / or one or more transformed image records combined to form an image record.

[0056] In the transformed image record, the same optical code as in the one or more image records is imaged to serve to generate the transformed image record, however, in the transformed image record, the optical code is imaged clearer and / or more clearly and / or with less distortion and / or with less reflections and / or with fewer distinctive features that may lead to decoding errors than in the one or more untransformed image records.

[0057] According to the present invention, the transformed image record is generated using a trained machine learning model.

[0058] Such a model can be trained in a supervised learning manner to generate transformed image records from one or more image records.

[0059] A "machine learning model" can be understood as a computer-implemented data processing architecture. The model can receive input data and provide output data based on said input data and model parameters. The model can learn the relationship between the input data and the output data by training. During training, the model parameters can be adapted to provide a desired output for a particular input.

[0060] During training of such a model, the model is presented with training data from which it can learn. A trained machine learning model is the result of the training process. The training data includes the input data as well as the correct output data (target data) that the model attempts to generate based on the input data. Patterns that map the input data to the target data are recognized during training.

[0061] In the training process, the input data of the training data are fed into the model, which generates output data. The output data is compared with target data (so-called ground truth data). The model parameters are changed to reduce the deviation between the output data and the target data to a (defined) minimum value.

[0062] During training, a loss function can be used to evaluate the predictive quality of the model. The loss function can be selected to reward desirable relationships between the output data and the target data and / or to penalize undesirable relationships between the output data and the target data. Such relationships can be, for example, similarity, dissimilarity, or some other relationship.

[0063] A loss function can be used to calculate the loss for a given pair of output data and target data. The goal of the training process is to change (fit) the parameters of the machine learning model such that the loss for all pairs of the training dataset is reduced to a (defined) minimum value.

[0064] A loss function can, for example, quantify the deviation between the output data of a model and the target data for a particular input data. For example, if the output data and the target data are numbers, the loss function can be the absolute difference between these numbers. In this case, a high absolute value of the loss function can mean that one or more model parameters must be significantly changed.

[0065] For output data in vector form, one can choose as the loss function a difference metric between vectors, e.g., mean squared error, norm of difference vectors such as cosine distance, Euclidean distance, Chebyshev distance, Lp norm of difference vectors, weighted norm, or any other type of difference metric of two vectors.

[0066] In the case of high dimensional outputs, e.g. two-dimensional, three-dimensional or higher dimensional outputs, e.g. element-wise difference metrics can be used. Alternatively or additionally, the output data can be converted, e.g. into a one-dimensional vector, prior to the calculation of the loss value.

[0067] In this case, the machine learning model is a model configured to generate a second image record based on one or more first image records.

[0068] In this case, the machine learning model can be trained to generate a transformed image record from one or more image records. The training can be based on training data. The training data can include a number of reference image records and transformed reference image records. The term "reference" is used in this description to distinguish data used to train the machine learning model from data used during use of the trained machine model. However, this term is not intended to be limiting in any way.

[0069] The term "multiple" preferably means more than 100. The reference image records usually represent optically readable codes introduced on the surface of multiple different samples of the object. The transformed reference image records represent the same optically readable code as the untransformed reference image records. The transformed reference image records may have been generated by one or more experts in the field of optical image processing on the basis of the reference image records. The one or more experts may apply one or more transformations to the reference image records in order to make the optically readable code clearer and / or more distinct and / or have less distortions and / or less reflections and / or less characteristic features that may lead to decoding errors than in the case of the untransformed reference image records. The one or more experts may combine with each other multiple reference image records representing the same optically readable code on the surface of the same object to generate a transformed reference image record. The purpose of the transformation and / or combination is to generate a transformed reference image record that generates fewer decoding errors during reading of the imaged optical code than the untransformed reference image record.

[0070] The decoding error can be determined empirically. It can be, for example, the percentage of codes that could not be (correctly) decoded. Thus, for example, if 10 of 100 codes imaged in 100 reference image records could not be decoded or produced one or more reading errors, the percentage of codes that could not be correctly decoded is 10%. One or more image processing experts generate, for example, 100 transformed reference image records from the 100 reference image records. The codes imaged in the transformed reference image records are decoded. If the experts perform their work correctly, more than 90 of the 100 codes in the 100 transformed reference image records, ideally all 100, should be correctly decoded.

[0071] Many optically readable codes have a means for error correction. The decoding error can also indicate the number of corrected bits in the code.

[0072] One or more image processing experts can define reference image recording transformations that result in an increase in the contrast of the optically readable code relative to its surroundings, and / or result in the reduction / elimination of distortions, and / or result in the reduction / elimination of reflections, and / or result in the reduction / elimination of other features that may result in decoding errors. Each of the transformations can be tested. If they do not result in the desired success, they can be discarded, refined, modified, and / or extended with further transformations.

[0073] Examples of transformations are spatial low-pass filtering, spatial high-pass filtering, sharpening, blurring (e.g. Gaussian blur), unsharp masking, erosion, median filtering, maximum filtering, contrast range reduction, edge detection, color depth reduction, grayscale level conversion, negative creation, color correction (color balance, gamma correction, saturation), color replacement, Fourier transform, Fourier low-pass filtering, Fourier high-pass filtering, and inverse Fourier transform.

[0074] Some transformations can be performed by convolution of raster graphics using one or more convolution matrices (called convolution kernels). The latter are usually square matrices with odd dimensions that can have various sizes (for example, 3x3, 5x5, 9x9, etc.). Some transformations can be represented as linear systems, and discrete convolution, linear operations are applied. For a discrete two-dimensional function (digital image), the following formula for discrete convolution arises:

number

[0075] For example, for an apple, below are listed transformations and sequences of transformations that have the effect that the transformed reference image record produces fewer decoding errors than the untransformed reference image record. -Color conversion to intensity-linear RGB signal - color transformation of the linear RGB signal, for example into a reflection and / or illumination channel and at least two color channels for distinguishing between coded and non-coded surface parts, where for this purpose the intensity linear RGB color signals are linearly combined with each other to achieve the best possible distinction between coded and non-coded surface parts. -Reflection correction by subtracting the reflection channel from at least two color channels (additive correction) - Lighting correction by normalizing at least two color channels to the lighting channel (multiplicative correction) -Detection of disturbances on the apple surface and correction of disturbances by spatial interpolation of surrounding imperfections -Unsharp masking of illumination- and reflectance-corrected image segments using a filter mask results in that, in terms of extent, e.g. the spatial non-uniformity of the image brightness is sufficiently compensated by the curved shape of the apple and the image contrast between coded and non-coded surface parts is amplified and optimized by increasing the high-frequency image parts.

[0076] Further examples of transformations can be gleaned from numerous publications on the topic of digital image processing.

[0077] If a large number of transformed reference image records have been generated, the training data can be used to train a machine learning model. In this case, the (untransformed) reference image records are fed as input data to the model. The model is configured to generate an output image record from one or more reference image records. The output image record is compared to the transformed reference image record (target data). The deviation between the output image record and the transformed reference image record can be quantified with a loss function. The determined error value can be used to adapt model parameters of the machine learning model such that the error value is reduced. If the error value reaches a predefined minimum value, the model can be trained and used to generate new transformed image records based on the new image records. In this case, the term "new" means that the corresponding image record was not used during training of the model.

[0078] The machine learning model may, for example, be or include an artificial neural network.

[0079] An artificial neural network includes at least three layers of processing elements: a first layer having input neurons (nodes), an Nth layer having at least one output neuron (node), and N-2 inner layers, where N is a natural number greater than 2.

[0080] The input neurons are responsible for receiving one or more image records, and the output neurons are responsible for outputting transformed image records.

[0081] The processing elements of the hierarchy between the input neurons and the output neurons are connected in a predetermined pattern with predetermined connection weights.

[0082] The training of a neural network can be carried out, for example, by backpropagation. The objective for the network here is the maximum reliability of the mapping of given input data to given output data. The mapping quality is described by a loss function. The objective is to minimize the loss function. In backpropagation, the artificial neural network is taught by modifying the connection weights.

[0083] In the trained state, the connection weights between the processing elements contain information about the relationship between the image records and the transformed image records.

[0084] Cross-validation methods can be used to split the data into a training data set and a validation data set. The training data set is used in backpropagation training of the network weights. The validation data set is used to check the accuracy of predictions that the trained network can make when applied to unknown data.

[0085] In a particularly preferred embodiment, the machine learning model comprises a generative adversarial network (abbreviated as GAN). Details about these and other artificial neural networks can be gleaned from the prior art (see, for example, M.-Y. Liu et al., Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications, arXiv:2008.02793; J. Henry et al., Pix2Pix GAN for Image-to-Image Translation, DOI:10.13140 / RG.2.2.32286.66887).

[0086] Once the machine learning model is trained, it can be used to generate new transformed image records based on new image records.

[0087] An image record (or multiple image records) is fed into the trained model, and the model produces a transformed image record.

[0088] In the transformed image record, the optically readable code is more recognizable and readable than in the original (non-transformed) image record (or the original image records in the case of multiple image records).

[0089] The optically readable code in the transformed image record is read (decoded) in the next step. Depending on the code used, there are now existing methods for reading (decoding) the respective code.

[0090] The read (decoded) code can contain information about objects within the surface in which the code is introduced.

[0091] In a preferred embodiment, the read code comprises a (unique) identifier, based on which the consumer can retrieve further information about the object, for example from a database. The read code and / or the information linked to the read code can be output, i.e. displayed on a screen, printed on a printer and / or stored on a data storage medium.

[0092] Further information regarding such (unique) identifiers and the information that can be stored and displayed to the consumer regarding objects linked to the identifiers is described in European Patent Application No. 3896629, the contents of which are hereby incorporated by reference in their entirety.

[0093] The invention is explained in more detail below with reference to the drawings, without wishing to limit the invention to the features and combinations of features shown in the drawings.

[0094] FIG. 1 illustrates, by way of example, a schematic of a method for training a machine learning model.

[0095] Training of a machine learning model (MLM) is based on training data (TD), which typically consists of at least i) one reference image record (RI) and ii) one transformed reference image record (RI) for each object of a plurality of objects. t ) for one object. Only one data set for one object is shown in Fig. 1. The object is an apple in this example. An optical code is introduced on the surface of the object. In this example, a QR code is introduced on the skin of the apple. The transformed reference image record (RI t Both the transformed reference image record (RI) and the untransformed image record (RI) refer to an optically readable code that has been introduced into the surface of an object. t ) may have been generated by an expert based on an untransformed image record (RI). The expert may then generate a transformed reference image record (RI t In one embodiment, the reader may have been faced with the objective of determining, for an untransformed reference image record (RI), one or more transformations that ensure that reading of the optically readable code in a reference image record (RI) produces fewer decoding errors than reading of the optically readable code in an untransformed reference image record (RI).

[0096] The reference image record (RI) is provided to a machine learning model (MLM) (step 110). The machine learning model (MLM) is configured to generate an output image record (I*) based on the reference image record (RI) and model parameters (MP) (step 120). The output image record (I*) is a predicted transformed reference image record. The output image record (I*) is a predicted transformed reference image record based on the transformed reference image record (RI). t ) can be compared using the loss function (LF) to calculate the output image record (I*) and the transformed reference image record (RI t ) can be calculated (step 130), where the loss (L) is the loss between the output image record (I*) and the transformed reference image record (RI t) The loss (L) can be used to modify the model parameters (MP) with respect to reducing the deviation. This can be done with optimization methods, e.g. gradient methods.

[0097] A machine learning model (MLM) is trained based on a number of reference image records as input data and transformed reference image records as target data. As a result, the model learns transformations that result in fewer decoding errors during reading.

[0098] FIG. 2 illustrates, diagrammatically and by way of example in the form of a flow chart, the reading of an optically readable code introduced onto the surface of an object.

[0099] In a first step (210), a digital image record (I) of an optically readable code introduced on the surface of an object (O) is generated using a camera (C). In a second step (220), the digital image record (I) is trained using a trained machine learning model (MLM t ) Machine learning models (MLM t ) is constructed and trained to generate a transformed image record (I*) based on an image record (I). t ) can be performed as described in connection with FIG. 1. In a third step (230), a machine learning model (MLM t ) provides a transformed image record (I*), in which the optically readable code (in this case a QR code) introduced on the surface of the object (O) has a higher contrast with its surroundings and is therefore more clearly recognizable and easier to read than the optically readable code in the case of the non-transformed image record (I). In a fourth step (140), the optically readable code is read to provide a read code (OI). The read code (OI) can be displayed and / or information about the object (O) can be provided (e.g. communicated and displayed), for example read from a database, on the basis of the read code (OI).

[0100] FIG. 3 shows a schematic diagram of a system according to the invention by way of example.

[0101] The system (1) comprises a computer system (10), a camera (20) and one or more data storage media (30). An image recording of an object can be generated by means of the camera (20). The camera (20) is connected to the computer system (10) so that the generated image recording can be transmitted to the computer system (10). The camera (20) can be connected to the computer system (10) via a cable connection and / or a wireless connection. A connection via one or more networks is also conceivable. Furthermore, it is conceivable that the camera (20) is an integral part of the computer system (10), as is the case for example with today's smartphones and tablet computers.

[0102] The computer system (10) is configured (e.g., by a computer program) to receive one or more image records (from a camera or a data storage medium), generate a transformed image record, decode an optical code in the transformed image record, output the decoded code, and / or provide information linked to the decoded code.

[0103] The image records, models, model parameters, computer programs and / or other / further information can be stored on a data storage medium (30). The data storage medium (30) can be connected to the computer system (10) via a cable connection and / or a wireless connection. A connection via one or more networks is also conceivable. Furthermore, it is conceivable that the data storage medium (30) is an integral part of the computer system (10). Furthermore, it is conceivable that there are multiple data storage media.

[0104] 4 shows a schematic representation of a computer system (10). Such a computer system (10) may comprise one or more fixed or portable electronic devices. The computer system (10) may comprise one or more components, such as, for example, a processing unit (11) connected to a storage medium (15).

[0105] The processing unit (11) may comprise one or more processors, either alone or in combination with one or more storage media. The processing unit (11) may include conventional computer hardware capable of processing information, such as digital image records, computer programs and / or other digital information. The processing unit (11) typically consists of an arrangement of electronic circuits, some of which may be embodied as an integrated circuit or as multiple integrated circuits connected together (integrated circuits are sometimes called "chips"). The processing unit (11) may be configured to execute computer programs, which may be stored in a main memory of the processing unit (11) or in a storage medium (15) of the same or a different computer system.

[0106] The storage medium (15) may be any conventional computer hardware capable of temporarily and / or permanently storing information, such as digital image records, data, computer programs, and / or other digital information. The storage medium (15) may comprise volatile and / or non-volatile storage media and may be permanently embedded or removable. Examples of suitable storage media are RAM (random access memory), ROM (read only memory), hard disk, flash memory, interchangeable computer floppy disk, optical disk, magnetic tape, or combinations of the foregoing. Optical disks may include compact disks with read only memory (CD-ROM), compact disks with read / write capabilities (CD-R / W), DVDs, Blu-ray disks, and the like.

[0107] In addition to the storage medium (15), the processing unit (11) may also be connected to one or more interfaces (12, 13, 14, 17, 18) for displaying, transmitting, and / or receiving information. The interfaces may comprise one or more communication interfaces (17, 18) and / or one or more user interfaces (12, 13, 14). The one or more communication interfaces may be configured to transmit and / or receive information, for example, from a camera, another computer, a network, a data storage medium, etc. The one or more communication interfaces may be configured to transmit and / or receive information via a physical (wired) and / or wireless communication connection. The one or more communication interfaces may include one or more interfaces for connecting to a network, for example, using technologies such as cellular, Wi-Fi, satellite, cable, DSL, fiber optics, etc. In some examples, the one or more communication interfaces may comprise one or more short-range communication interfaces configured to connect devices with short-range communication technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA), etc.

[0108] The user interface (12, 13, 14) may comprise a display (14). The display (14) may be configured to display information to a user. Suitable examples are liquid crystal displays (LCD), light emitting diode displays (LED), plasma display panels (PDP), and the like. The user input interfaces (12, 13) may be wired or wireless and may be configured to receive information from a user to the computer system (10), for example, for processing, storage, and / or display. Suitable examples of user input interfaces are microphones, image or video recording devices (e.g., cameras), keyboards or keypads, joysticks, touch-sensitive surfaces (either separate from or integrated into a touchscreen), and the like. In some examples, the user interface may include automatic identification and data capture technology (AIDC) for machine-readable information. It may include bar codes, radio frequency identification (RFID), magnetic strips, optical character recognition (OCR), integrated circuit cards (ICC), and the like. The user interface may further comprise one or more interfaces for communicating with peripheral devices, such as printers.

[0109] One or more computer programs (16) can be stored in the storage medium (15) and executed by the processing unit (11) so as to be programmed to perform the functions described herein. The fetching, loading, and execution of instructions of the computer programs (16) can be performed sequentially as each instruction is fetched, loaded, and executed. However, the fetching, loading, and / or execution can also be performed in parallel.

[0110] The system according to the invention may be embodied as a laptop, notebook, netbook, tablet PC and / or handheld device (eg smartphone). Preferably, the system according to the invention comprises a camera.

[0111] 5 illustrates, by way of example, a schematic representation of the generation of a training data set for training a machine learning model. Based on the training data set, the machine learning model can be trained to perform one or more transformations of an image record.

[0112] The generation of the training data set can be a manual process performed by one or more experts. The starting point in this example is the reference image records RI1 to RI n where n is a number n of reference images, n being preferably an integer greater than 100. Each reference image record represents an optical code introduced into a surface of an object. Preferably, each reference image record represents an optically readable code introduced into different samples of the object. The object may for example be an apple. In that case, each reference image record preferably represents an optically readable code in different apple samples.

[0113] If the reference object is always the same, for example an apple, the training data set can be used to train a machine learning model to reduce and / or eliminate disturbances in the image recording of an optically readable code introduced into the apple.

[0114] For example, if there are different reference objects, such as different fruits (e.g., apples and pears), the training data set can be used to train the machine learning model to reduce and / or eliminate disturbances in the image recordings of optically readable codes introduced in the different fruits. The more different the reference objects imaged in the reference image recordings (the greater the number of variants), the more training data is required and the more versatile the trained machine learning model can be used. A machine learning model trained only on reference image recordings of apples of a defined variety, when used to read codes on bananas, achieves worse results than a model trained on reference image recordings of different varieties of apples and bananas. The optically readable codes imaged in the reference image recordings may similarly be identical or different in all reference image recordings.

[0115] In this embodiment (FIG. 5), at least one expert generates a transformed reference image record from each reference image record. As described herein, multiple reference image records can also be combined to form a transformed reference image record. The transformed reference image record is generated by subjecting the reference image record to one or more transformations. Which transformations are performed and in what order, in case of multiple transformations, the transformations are performed, are specified by the at least one expert based on his / her expertise. The purpose of the transformation is to generate a transformed reference image record from the at least one reference image record in which less disturbances are present and therefore the probability of occurrence of a decoding error is reduced.

[0116] Disturbance factors are features of the object and / or of an optically readable code introduced into the object's surface that cause disturbances in at least one image recording.

[0117] Such disturbances are, for example, distortions, light reflections, code elements having different colors and / or sizes, etc.

[0118] Disturbance factors and disturbances can have the effect that elements of the optically readable code as imaged in the at least one image recording are not recognized or are misinterpreted.

[0119] Disturbance factors and disturbances can have the effect that features of the object as imaged in at least one image recording are interpreted as features of the optical code despite not being part of the optically readable code.

[0120] One or more disturbances in at least one image recording are reduced and / or eliminated with the aid of the present invention by means of one or more transformations. [Explanation of symbols]

[0121] 1 System 10. Computer Systems 11 Processing Unit 12 User Input Interface 13 User Input Interface 14 User Interface, Display 15 Storage medium 16 Computer Programs 17 Communication Interface 18 Communication Interface 20 Camera 30 Data storage media

Claims

1. 1. A computer-implemented method for training a machine learning model, comprising: - providing training data, the training data for each object of a plurality of objects includes: i) at least one reference image record of an optically readable code to be introduced onto a surface of the object; and ii) a transformed reference image record of the optically readable code; decoding the optically readable code in the transformed reference image record produces fewer decoding errors than decoding the optically readable code in the reference image record; for each object, the transformed reference image record is generated by applying one or more transformations to the at least one reference image record; the one or more transformations increase the contrast between the optically readable code and the surrounding portion of the surface, and / or reduce or eliminate distortion, and / or reduce or eliminate reflections; the objects are different samples of plant or animal products; - providing a machine learning model, said machine learning model being configured to generate a second image record based on at least one first image record and model parameters, said second image record being the transformed first image record; training the machine learning model, wherein for each object of the plurality of objects, the training comprises: inputting said at least one reference image record into said machine learning model; receiving a predicted transformed reference image record from said machine learning model; - calculating the deviation between said transformed reference image record and said predicted transformed reference image record; modifying said model parameters in relation to reducing said deviation; - storing and / or outputting the trained machine learning model, and / or communicating the trained machine learning model to a separate computer system, and / or using the trained machine learning model to generate a transformed image record from at least one new image record of the object.

2. 2. The method of claim 1, wherein the one or more transformations are selected such that reading the optically readable code in the transformed reference image record results in fewer decoding errors than reading the optically readable code in the at least one reference image record.

3. The method of claim 1 , wherein the one or more transformations are determined empirically.

4. The method of claim 1 , wherein the machine learning model is or includes an artificial neural network.

5. The method of claim 1 , wherein the optically readable code is introduced into the surface of the object by a laser.

6. The method of claim 1 , wherein the optically readable code is a matrix code or a bar code.

7. The method of claim 1 , wherein the optically readable code includes alphanumeric characters.

8. 1. A computer-implemented method for decoding an optically readable code introduced onto a surface of an object, comprising: - receiving an image record from said optically readable code; - feeding said image recordings to a trained machine learning model, said trained machine learning model having been trained with the method of any one of claims 1 to 7; - receiving a transformed image record from said machine learning model; - decoding said optically readable code imaged in said transformed image record.

9. - further comprising the step of outputting said decoded optically readable code and / or information linked to said decoded code, The method of claim 8.

10. 1. A system comprising at least one processor, the processor comprising: - receiving an image record from an optically readable code, said optically readable code being introduced into the surface of an object; - feeding said image recordings to a trained machine learning model, said trained machine learning model having been trained with the method of any one of claims 1 to 7; - receiving a transformed image record from said machine learning model; - decoding the optically readable code imaged in the transformed image record.

11. The system of claim 10 , wherein the processor is configured to output the decoded code or information linked to the decoded code.

12. A computer program loadable into a main memory of a computer system, said computer system comprising: - receiving an image recording of an optically readable code, said optically readable code being introduced into the surface of an object; - feeding said image recordings to a trained machine learning model, said trained machine learning model having been trained with the method of any one of claims 1 to 7; - receiving transformed image records from the trained machine learning model; - decoding the optically readable code imaged in the transformed image record; - outputting said decoded optically readable code and / or information linked to said decoded code.

Images