Camera-based code reader and method for reading optical codes

Abstract

The application relates to a camera-based code reader and to a method for reading an optical code. The camera-based code reader (10) has an image sensor (24) for detecting image data with an optical code (20), a memory (34) for storing the image data, a first processing unit (30) for transferring the image data from the image sensor into the memory (34) in the form of an image data stream, and a second processing unit (32) which is designed for decoding (36), in which decoding the stored image data are evaluated in order to read out the code content of the optical code. The first processing unit (30) is designed for at least one auxiliary function, wherein the first processing unit reads back at least a portion of the image data from the memory (34), pre-processes it and writes the pre-processing result back into the memory, and the second processing unit (32) is designed to call the auxiliary function of the first processing unit (30) as required in order to support the decoding.

Description

[0001] The present invention relates to a camera-based code reader and a method for reading optical codes according to the preamble of claims 1 and 13.

[0002] Barcode readers are widely used in supermarket checkouts, automated parcel identification, mail sorting, airport baggage handling, and other logistics applications. Camera-based readers use image sensors to record images of objects with codes on them, and image evaluation software extracts the code information from these images. Camera-based readers can also easily handle code types other than one-dimensional barcodes, which, like matrix codes, are constructed in two dimensions and provide more information.

[0003] In a number of important applications, coded objects are conveyed through barcode readers. Image sensors repeatedly record image data, which overlaps more or less depending on the recording frequency and conveying speed. Barcode readers are often part of a more complex barcode reading system. Therefore, multiple readers are typically positioned at the reading channel to record the object from multiple or all sides. Additional sensors are provided here, such as encoders to determine the conveyor feed or conveying speed, or laser scanners to measure the geometry of the object.

[0004] With the further development of camera and image sensor technology, the images to be processed are becoming increasingly larger. This massive amount of data poses a challenge to the limited computing power and bandwidth available for real-time transmission and processing within the camera. Here, a heterogeneous computing architecture is used, combining a microprocessor (CPU) with, for example, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), or a so-called KI accelerator (NPU, Neural Processing Unit), or a tensor processing unit (TPU).

[0005] Here, the combination with an FPGA responsible for reading image data from the image sensor proves particularly effective. Furthermore, the FPGA is connected to the microprocessor's memory via high-speed interfaces (PCI (Peripheral Component Interconnect), PCIe, PCI Express, MIPI (Mobile Industry Processor Interface)). Therefore, image data is transferred from the FPGA to memory via DMA (Direct Memory Access). This read-and-store process is called streaming. Then, a decoder implemented by the CPU can access the stored image and read the code downstream. Other additional modules in the aforementioned heterogeneous architecture, which typically already require stored images, are less suitable for streaming, while the FPGA can directly read image data pixel-by-pixel and further process it.

[0006] Preferably, the image data has already been preprocessed directly and on-the-fly in the FPGA, and can be stored together with the image data for additional information or metadata of the microprocessor. Typical preprocessing steps involve binarization (where a grayscale image is converted to a black and white image) or segmentation (where regions of interest (ROIs) with candidate codes are found). Thus, EP 2 003 599 A1 describes a photoelectric sensor and a method for detecting codes, wherein a binarizer for converting a color or grayscale image into a binary image has been formed during reception and / or in real time, in such a way that the corresponding read-in portion is binarized while other portions are still being read in. EP 2 555 160 B1 finds regions of interest or candidate codes based on contrast quantities during preprocessing on the FPGA. In an unpublished European application with document number 20176228.3, the first layer of the neural network for segmentation has already been run on the FPGA during read-in. EP 1 365 577 A1 discloses a method for operating a photoelectric sensor, wherein the image has been compressed during reception.

[0007] Here, the FPGA's performance is fully utilized to perform many simple computational operations, such as matrix multiplication, in parallel and in real time. The decoder's sequential, more complex computational operations are reserved for the microprocessor, which utilizes the provided image data and possible preprocessing results. However, there are cases, such as extreme module sizes, where the decoder struggles and must perform computationally intensive operations on a very large number of pixels for a potentially successful read attempt. The available time window for decoding may no longer be sufficient before new image data with additional code arrives. On the other hand, the FPGA cannot simply perform appropriate processing steps preventively on all incoming image data during streaming. Firstly, this would introduce unnecessary computational work, as much code would be read even without additional processing steps. Secondly, appropriate processing steps must be selected on a case-by-case basis, and only the decoder possesses the necessary information, especially after the first unsuccessful read attempt, to determine the appropriate steps for the current specific situation.

[0008] Therefore, the purpose of this invention is to further improve code reading in heterogeneous architectures.

[0009] This objective is achieved by a camera-based reader and a method for reading optical codes according to claims 1 and 13. For example, the optical code is a barcode or QR code based on any standard. Image sensors record image data with multiple pixels, typically achieving high resolution using a line-scanning sensor (Zeilensensor) with several thousand pixels or a matrix sensor with several million pixels. In operation, the reader detects objects with optical codes, and thus the image data subsequently includes at least one area with the optical code.

[0010] The reader's first processing unit reads image data from the image sensor, preferably a high-resolution raw image, and transfers the image data to a memory. Reading and storage are performed as a stream of image data in groups of adjacent pixels, a process known as streaming. Depending on the implementation, the image data may be preprocessed directly during streaming, and the correspondingly processed image data or additional information or metadata obtained during preprocessing may be stored. The reader also includes a second processing unit for decoding, specifically for evaluating or processing the image data stored in the memory and possible preprocessing results to read the code content of the optical codes within the image data.

[0011] This invention is based on the following fundamental idea: a first processing unit is additionally used during decoding to support a second processing unit. The first processing unit thus acquires a dual function, acting as an additional coprocessor to extend the existing streaming capabilities (i.e., reading and storing image data). Therefore, the first processing unit is designed for at least one auxiliary function that reads back at least a portion of the image data already streamed to memory to perform preprocessing and re-store the preprocessing results. The auxiliary function is triggered by the second preprocessing unit only as needed when the decoder requires it to decode the current code. The first processing unit then operates in its secondary role as a coprocessor, supporting the second processing unit with the auxiliary function. The result of the auxiliary function can be processed image data and / or additional information or metadata. The second processing unit accesses these results in memory to continue decoding, reusing the auxiliary function of the first processing unit as necessary.

[0012] The advantage of this invention is that it reduces the burden on the second processing unit. This is particularly advantageous for pixel-based operations that require multiple iterations, which are performed significantly more efficiently by the first processing unit. Therefore, improved runtime, or processing within the available time window, becomes primarily possible. It can even expose algorithms to the decoder that would otherwise be completely disregarded by known procedures, as there are insufficient computational and time resources available to the second processing unit.

[0013] Preferably, the first processing unit has an FPGA (Field-Programmable Gate Array) and / or the second processing unit has a microprocessor (CPU). FPGAs are particularly suitable for implementing streaming and real-time processing of large amounts of data using relatively simple single computational operations. CPUs, on the other hand, are more flexible and capable of independently performing more complex computational steps, which is advantageous for decoding. If the CPU encounters problems requiring numerous simple mathematical operations during decoding, such as pixel-level operations over a large image area, it can revert to using the auxiliary functions of the FPGA.

[0014] Preferably, the first processing unit reads image rows or portions thereof from the image sensor and stores them in memory. Therefore, the image is read in line by line from the image sensor or streamed, and the processing is line-oriented. For preprocessing involving more than one image row, such as convolution with a filter kernel, the first processing unit may have an image buffer in which the corresponding number of image rows are cached in a rolling manner. In principle, image columns can also be processed instead of image rows in the same way. This is simply understood as a difference in coordinate definition and is not distinguished here.

[0015] Preferably, the first processing unit is designed to preprocess the image data read by the image sensor before transferring it to the memory, or even while the image sensor is reading additional image data. Therefore, the first processing unit is designed to perform preprocessing "on the fly" or already during streaming. This preprocessing must be clearly distinguished from auxiliary functions. Preprocessing occurs almost directly at the source, even before the decoder of the second processing unit has full access to the image data. Conversely, auxiliary functions are requested selectively by the decoder based on the image data already stored by the first processing unit.

[0016] Preferably, the first processing unit has a first partition for transferring image data from the image sensor to memory and a second partition for performing auxiliary functions. Therefore, within the first processing unit, the two functions of streaming and coprocessing, or transferring image data and performing auxiliary functions, are clearly separated. Specifically, separate planes or regions are provided on the FPGA for this purpose. Other partitions may exist, preferably a third partition with algorithms that may be required by both partitions.

[0017] Preferably, the first and second partitions operate at different clock frequencies. This is technically feasible due to the explicit separation. In this way, limited energy or thermal budgets can be allocated very specifically, for example. Preferably, the adjustment of the clock frequency involves auxiliary functions because streaming must be synchronized with a normally fixed frame rate, and is less context-sensitive and adaptable.

[0018] Preferably, the first processing unit is designed for segmentation, searching for regions of interest (ROIs) with candidate codes as an auxiliary function. This is an exemplary coprocessor function or auxiliary function. One or more such auxiliary functions can be implemented. In terms of content, some processing of the auxiliary function may have already been performed during streaming. However, this is not an auxiliary function in the sense understood herein, as the auxiliary function is requested or triggered by the decoder specifically for the already stored image data after streaming. As mentioned at the outset, in the prior art, segmentation is sometimes performed on-the-fly during streaming, in which image regions (ROIs) with inferred optical codes or candidate codes are found. Alternatively, the decoder may request segmentation first, or allow refined segmentation or segmentation performed under other parameters in addition to pre-segmentation already performed during streaming.

[0019] Preferably, the first processing unit is designed for resolution enhancement as an auxiliary function. This is another exemplary coprocessor function or auxiliary function, and the general comments in the foreword similarly apply to these and other auxiliary functions. Resolution enhancement corresponds to subsequent or digital scaling functions. Many methods are known in themselves, using terms such as upsampling and super-resolution. However, the second processing unit of the decoder typically lacks the necessary resources, so the corresponding auxiliary function significantly accelerates or enables resolution enhancement in the first place. Resolution enhancement can involve the entire image, a partial region, particularly a ROI with candidate codes, or only the code portion.

[0020] Preferably, the first processing unit is designed to rotate a portion of the image as an auxiliary function. This is another exemplary coprocessor function or auxiliary function. The formulas or algorithms used for rotation are well known. However, this is pixel-based processing, which a second processing unit typically cannot provide, thus support from the first processing unit is a significant improvement.

[0021] Preferably, the first processing unit is designed to utilize filters as an auxiliary function, specifically sharpening filters, edge filters, noise filters, or smoothing filters. Convolution of the image or image region with the filter kernel requires multiple separate operations, which are efficiently performed by the first processing unit, but could overload the second processing unit within a predetermined time budget. Therefore, the image can be sharpened to prepare for edge detection and thus module detection, or deblurring can be performed.

[0022] Preferably, the first processing unit is designed to perform template comparison with the code portion as an auxiliary function. Template matching allows for the direct identification or at least partial decoding of the code portion on the image data. Although the decoder typically operates using binarized black-and-white image data, template comparison can be performed on the underlying color or grayscale data.

[0023] Preferably, the first processing unit is designed to perform viewpoint correction or distortion correction as an auxiliary function. This allows for compensation for unfavorable viewpoints on the recorded code or distortion errors in the image sensor lens. This requires pixel-level operations, which are effectively implemented by the first processing unit as an auxiliary function.

[0024] The method according to the invention can be further developed in a similar manner and exhibits similar advantages. These advantageous features are described, exemplarily but not exhaustively, in the dependent claims which are subordinate to the independent claims. Attached Figure Description

[0025] Other features and advantages of the invention will now be described in more detail, based on exemplary embodiments and with reference to the accompanying drawings. In the drawings:

[0026] Figure 1 A schematic overview diagram shows an example of a camera-based barcode reader mounted above a conveyor belt; and

[0027] Figure 2 A schematic diagram of a heterogeneous architecture is shown, in which an FPGA acts as a stream processor and a CPU acts as a decoder, and the FPGA also acts as a coprocessor.

[0028] Figure 1 A camera-based barcode reader 10 is shown mounted above a conveyor belt 12, as indicated by arrow 16, which transports an object 14 through the reader 10's detection area 18. The object 14 carries a code 20 on its outer surface, which is detected and evaluated by the reader 10. The code 20 can only be recognized by the reader 10 if it is attached to the upper side or at least visible from above. Therefore, unlike... Figure 1 As illustrated in the diagram, multiple barcode readers 10 can be mounted from different directions to read the codes 22 attached to the side or bottom, enabling so-called omnidirectional reading. In practice, multiple barcode readers 10 are typically arranged into a reading system as a reading channel. It is also common practice to fix barcode readers to conveyor belts. However, this invention relates to the processing of image data or the barcode reader 10 itself, and therefore this example should not be interpreted restrictively.

[0029] The code reader 10 uses the image sensor 24 to detect the transported object 14 and the code 20, which are further processed by the control and evaluation unit 26 using image evaluation and decoding methods. See below for reference. Figure 2 The control and evaluation unit 26 is described in more detail. The reader 10 outputs information, such as the read code or image data, via interface 28.

[0030] Figure 2 The control and evaluation unit 26 and its connections are illustrated schematically. The control and evaluation unit 26 includes a first processing unit 30 and a second processing unit 32. The first processing unit is described below as an example of an FPGA (Field-Programmable Gate Array), and the second processing unit is described below as an example of a CPU (Central Processing Unit). The first processing unit 30 has at least two parts, wherein a first part 30a is designated as a streaming component and a second part 30b is designated as a coprocessor, and may include an optional common third part 30c. The second processing unit 32 has a decoder 36 for reading optical codes from image data.

[0031] The first processing unit 30 is connected to the image sensor 24 (Imager Interface) on one hand, and has a high-speed interface (PCI, PCIE, MIPI) toward the second processing unit 32 on the other. Of particular importance to this invention is that both processing units 30 and 32 access a memory 34 to acquire image data and processing results. Preferably, the corresponding read and write processes are performed using DMA (Direct Memory Access). The memory 34 can be understood at least functionally as part of the CPU 32, and structurally as well, depending on the implementation. Another connection between the two processing units 30 and 32 enables the invocation of auxiliary functions.

[0032] During operation, image sensor 24 records new images or new image portions, respectively. This can be a rectangular image from a matrix sensor, but it is also conceivable that it can be a single or multiple image rows from a line scanning sensor, which subsequently generate a complete image continuously during relative movement between reader 10 and object 14. A first processing unit (specifically its first part 30a) reads the image data from image sensor 24 and transmits or streams it to memory 34.

[0033] The decoder 36 of the second processing unit 32 accesses image data in memory 34 to read the contents of the optical code recorded along with the image data. In some cases, this requires pixel-intensive and computationally intensive operations, which the CPU is not particularly well-suited for, but which can be performed more efficiently and quickly in the first processing unit 30. In this case, the decoder 36 can be optionally supported by the first processing unit 30 as a coprocessor. Therefore, the first processing unit 30 acquires the dual function of being both a stream processor and a coprocessor.

[0034] The supporting function is referred to as the auxiliary function. Currently, the second part 30b of the first processing unit 30 is responsible for this. After a corresponding call from the second processing unit 32, the first processing unit 30 accesses the previously stored image data in the memory 34 again or reloads it back into the first processing unit 30. The first processing unit executes the auxiliary function and rewrites the result back into the memory 34. The decoder 36 of the second processing unit 32 can then continue reading code by accessing the memory 34.

[0035] Preferably, the auxiliary function is an operation on image data at the pixel level, but alternatively or additionally, it can also be evaluated and generate corresponding additional information or metadata, which is then written into memory 34. Decoder 36 can also parameterize the auxiliary function as desired through such metadata, or through a suitable interface (register interface) of the first processing unit 30.

[0036] During streaming, preprocessing can also be envisioned; therefore, the image data and potentially related metadata initially transmitted from the first portion 30a of the first processing unit 30 to the memory are already preprocessed results. However, this preprocessing step is always performed in the same manner for all image data. At this point, the decoder 36 cannot even intervene and evaluate specific image data. Instead, auxiliary functions are performed only as needed based on the current image data, and furthermore, the image data can be parameterized specifically.

[0037] An example of preprocessing that can be performed as an auxiliary function during and / or after streaming is segmentation, where regions of interest (ROIs) with candidate codes are sought. This operation is performed based on general standards during or on-the-fly during streaming, while as an auxiliary function, it can also be specifically targeted at only a portion of the image data, for example, particularly sophisticated methods can be employed for image regions that are difficult to interpret.

[0038] As already explained, the first processing unit 30 has different partitions or sections 30a-30c. Section 30a is responsible for streaming and, if necessary, preprocessing and generating metadata during streaming. Section 30b acts as a coprocessor responsible for performing auxiliary functions. An optional third section 30c is responsible for functions that are commonly used during streaming and during the execution of auxiliary functions. Preferably, sections 30a-30c can be understood not only as being functionally implemented separately, but also as being practically implemented separately through separation (especially in different regions or surface sections), thus forming true partitions. Sections 30a-30c can even operate at different clock frequencies as needed. This is particularly important when energy or thermal budgets are limited, in order to balance and optimize the overall performance of the various functions.

[0039] The following describes some examples of auxiliary functions that are not mentioned in detail. At least one auxiliary function may be provided, but it can also be multiple auxiliary functions in any combination. Similarly, the decoder of the second processing unit 32 may choose, as appropriate, whether to request an auxiliary function, and which individual auxiliary function or combination of auxiliary functions to request. Auxiliary functions may also be invoked multiple times. An example here could be a repeating neural network layer that is implemented by repeatedly invoking corresponding auxiliary functions with their own varying weights.

[0040] An example of an auxiliary function is resolution enhancement. This corresponds to a scaling function, but instead of optical magnification, it subsequently refines the resolution of the image data. For example, decoder 36 requires this auxiliary function when the module size is particularly small, i.e., when code elements are represented by too few pixels. Algorithms for upsampling via interpolation, such as bicubic, bilinear, linear, or by means of spline curves, are known in themselves. Image data (e.g., grayscale values) is transferred back from memory 34 to the first processing unit 30 or its second portion 30b, preferably along with the desired resolution enhancement factor and the desired interpolation method. For the interpolation method, factors 2, 4, ... are ideal. Other methods are described in the literature with super-resolution as a keyword, for which neural networks are specifically used.

[0041] Another example of an auxiliary function is the rotation or rotation of an image region or code. Preferably, for this purpose, the individual line segments are sequentially transmitted from memory 34 back to the first processing unit 30, where decoder 36 determines the desired rotation angle and transmits it to the first processing unit 30. Preferably, only the relevant line segments are transmitted and processed according to the rotation angle. Meta-information (such as the position of the code within the line segment) can be transmitted together in both directions. In the extension of planar rotation, viewpoint correction is also conceivable.

[0042] Another example of auxiliary functions is filtering. Convolution with a filter kernel is, in principle, a matrix operation, making FPGAs particularly well-suited for this. There are many possibilities related to noise, distortion and warp, motion blur, etc. For example, if the code structure exhibits a poor signal-to-noise ratio due to noise, a sharpness filter can be applied. Possible parameters here are the average gray value, the maximum and minimum gray values, and other statistical variables such as the standard deviation. Edge detection can be prepared and improved using appropriate filters (e.g., Sobel filters). Blur or motion blur can be at least partially compensated for by suitable filters (Lucy-Richardson (deconvolution), deblurring). Geometric correction (Geokorrektur), specifically used to compensate for warp errors, can also be understood as a filter. Besides filters or alternative filters, it is conceivable to determine specific feature variables, such as sharpness, and return them as meta-information. The decoder 36 can then evaluate whether the reading attempt was successful, or what other measures and possible auxiliary functions are needed beforehand. Similar effects to the filters described above can also be achieved using neural networks.

[0043] Another example of an assistive function is template comparison (template matching). This attempts to find and decode code segments directly in color or grayscale image data. In this case, parameters such as module size, templates, statistics, and code type are of interest. Neural networks are also suitable for this assistive function.

Claims

1. A camera-based barcode reader (10) comprising: an image sensor (24) for detecting image data having an optical code (20); a memory (34) for storing the image data; a first processing unit (30) for transmitting the image data from the image sensor (24) to the memory (34) as an image data stream; and a second processing unit (32) designed for decoding (36) in which the image data stored in the decoding (36) is evaluated to read out the code content of the optical code (20). Its features are, The first processing unit (30) is designed for at least one auxiliary function in which the first processing unit (30) reads back at least a portion of the image data from the memory (34), preprocesses the at least a portion of the image data, and writes the preprocessing result back to the memory (34), thereby enabling the first processing unit (30) to have a dual function: as a stream processor for transferring the image data from the image sensor (24) to the memory (34) and as a coprocessor for performing the auxiliary function; and The second processing unit (32) is designed to invoke the auxiliary function of the first processing unit (30) to support the decoding (36) when the current code needs to be decoded, and to access the result of the auxiliary function to continue decoding.

2. The code reader (10) according to claim 1, wherein, The first processing unit (30) has an FPGA and / or the second processing unit (32) has a microprocessor.

3. The code reader (10) according to claim 1 or 2, wherein, The first processing unit (30) reads an image row or a portion of an image row from the image sensor (24) and stores the image row or a portion of the image row in the memory (34).

4. The code reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) is designed to preprocess the image data before transferring the image data read by the image sensor (24) to the memory (34), or even while the image sensor is reading other image data.

5. The barcode reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) has a first partition (30a) for transferring image data from the image sensor (24) to the memory (34) and a second partition (30b) for performing auxiliary functions.

6. The code reader (10) according to claim 5, wherein, The first partition (30a) and the second partition (30b) operate at different clock frequencies.

7. The barcode reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) is designed to segment and find regions of interest with candidate codes as an auxiliary function.

8. The barcode reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) is designed to enhance resolution as an auxiliary function.

9. The barcode reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) is designed to rotate the image portion as an auxiliary function.

10. The barcode reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) is designed to use filters for processing as an auxiliary function.

11. The code reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) is designed to perform template comparison with the code section as an auxiliary function.

12. The code reader (10) according to any one of the preceding claims, wherein, The first processing unit (30) is designed for viewpoint correction or distortion correction as an auxiliary function.

13. The code reader (10) according to claim 10, wherein, The filter is a sharpness filter, an edge filter, or a smoothing filter.

14. A method for reading optical codes (20), wherein, An image sensor (24) detects image data with an optical code (20), a first processing unit (30) transmits the image data from the image sensor (24) to a memory (34) as an image data stream, and decodes (36) the data in a second processing unit (32), wherein the stored image data is evaluated to read out the code content of the optical code (20). Its features are, When the second processing unit needs to decode the current code, it calls at least one auxiliary function of the first processing unit (30) to support the decoding (36) and accesses the result of the at least one auxiliary function to continue decoding. When the first processing unit (30) executes the auxiliary function, it reads back at least a portion of the image data from the memory (34), preprocesses it, and writes the preprocessing result back to the memory (34), so that the first processing unit (30) has a dual function, namely, as a stream processor that transmits the image data from the image sensor (24) to the memory (34) and as a coprocessor that executes the at least one auxiliary function.

Images