Apparatus and method for preprocessing an image having an element of interest

Abstract

At least some embodiments of the present disclosure relate to a method for processing an input image having embedded elements, the method comprising the steps of: calculating one or more histograms of the input image; identifying a range of interest in one of the one or more histograms; and based on the range of interest, determining a threshold by the processor. In some embodiments, the computing device uses the threshold to process the input image to generate an output image.

Description

Technical Field

[0001] The present disclosure relates to preprocessing an image having an element of interest. Summary of the Invention

[0002] At least some embodiments of the present disclosure relate to a method for processing an input image having an embedded element, the method comprising the steps of: calculating, by a processor, one or more histograms of the input image, wherein each of the one or more histograms is calculated on a color plane in a color space; selecting a color plane in the color space; identifying a range of interest in one of the one or more histograms, calculating the one of the one or more histograms on the selected color plane; determining, by the processor, a threshold based on the range of interest; and processing, by the processor, the input image using the threshold to generate an output image, wherein the range of interest is identified at least in part based on a known color intensity of the embedded element.

[0003] At least some embodiments of the present disclosure relate to a device comprising: a processor; and a memory device coupled to the processor and having stored thereon a program for execution by the processor to perform operations comprising: calculating one or more histograms of an input image, wherein each of the one or more histograms is calculated on a color plane in a color space; selecting a color plane in the color space; identifying a range of interest in one of the one or more histograms, calculating the one of the one or more histograms on the selected color plane; determining a threshold based on the range of interest; and processing the input image using the threshold to generate an output image, wherein the range of interest is identified at least in part based on a known color intensity of the embedded element. Brief Description of the Drawings

[0004] The drawings are incorporated into and constitute a part of this specification, and the drawings, together with the description, explain the advantages and principles of the present invention. In the drawings,

[0005] Figure 1 a system diagram showing an example of an image preprocessing system is shown;

[0006] Figure 2A is an example of a flowchart of an image preprocessing system;

[0007] Figure 2B is another example of a flowchart of an image preprocessing system;

[0008] Figure 2C a flowchart showing an example of an image preprocessing system is shown;

[0009] Figures 3A to 3Iillustrates an exemplary process of an image preprocessing system; and

[0010] Figures 4A to 4I illustrates an example of the image preprocessing system preprocessing an image having an element of interest, where Figure 4A 、 Figure 4D and Figure 4G are each a single planar image of a workspace picture.

[0011] In the drawings, like reference numerals indicate like elements. Although the above drawings, which may not be drawn to scale, illustrate various embodiments of the present disclosure, other embodiments are also conceivable, as pointed out in the detailed description. In all cases, the present disclosure describes the presently disclosed disclosure in terms of representations of exemplary embodiments rather than by way of limitation. It should be understood that many other modifications and embodiments can be devised by those skilled in the art, which fall within the scope and spirit of the present disclosure. Detailed Description

[0012] In the world, single - dimensional and multi - dimensional codes embedded in labels are ubiquitous. Their applications in retail, shipping, and identification are everywhere. The range of hardware and software required to decode labels ranges from custom equipment (e.g., point - of - sale laser scanners) to smart phones with embedded cameras. The image - processing software required to decode labels must perform multiple tasks, including: 1) single or multiple label recognition in an image where clutter may be present; 2) label orientation (e.g., rotation and translation); and 3) decoding and error detection / correction of the embedded code.

[0013] The more clutter there is in a given image, the more computational resources are required to process the image. In many cases (e.g., portable battery-powered devices), computational resources are limited, and thus processing time becomes a major issue. This problem is further exacerbated when processing a video stream, where the processing time is constrained by the frame rate, or where computational resources are performing multiple simultaneous tasks (e.g., image processing and rendering). Some embodiments of the present disclosure describe systems and methods for processing an image with embedded elements to generate an output image such that the embedded elements in the output image can be extracted with less time and computational resource consumption. Some embodiments of the present disclosure describe a method of preprocessing an image containing one-dimensional or multi-dimensional encoded elements (e.g., 2D codes) such that the elements can be decoded more quickly and efficiently by decoding software. In some embodiments, the method includes the steps of: selecting a color channel from an input having one or more embedded elements, which input can be an image sequence or a static image; processing the color channel of the input to generate data distribution information to separate the embedded elements in the input from other elements; determining a threshold based on the processed data; and generating an output image based on the threshold. In some embodiments, the method includes the steps of: separating an image into its N component colors, selecting one of the N colors, processing the color component to maximize the embedded elements and minimize all other elements, and then generating an output image that will be provided to image processing software (e.g., decoding software) to extract information from the embedded elements.

[0014] In one embodiment, the functions, algorithms, and methods described herein can be implemented in software. The software can consist of computer-executable instructions stored on a computer-readable medium or a computer-readable storage device, such as one or more non-transitory memories or other types of hardware-based storage devices, local or networked. Additionally, such functions correspond to modules or processors, which can be software, hardware, firmware, or any combination thereof. Multiple functions can be executed in one or more modules or processors as needed, and the described embodiments are merely examples. The software can be executed on a digital signal processor, an ASIC, a microprocessor, or other types of processors running on a computer system, such as a personal computer, a server, or other computer systems, thereby transforming such computer systems into specifically programmed machines.

[0015] In some cases, the embedded element is a one - dimensional or two - dimensional element with distinguishable symbolic meaning. In some cases, the embedded element is a graphical element. In some embodiments, the embedded element is a code, which can be a one - dimensional or multi - dimensional code. In some cases, the embedded element is a masked code. A masked code refers to a code that is generally invisible to the human eye, but can be made visible to the human eye or read by a dedicated device using the dedicated device. Examples of masked codes are codes that are visible in wavelengths invisible to the human eye, for example, infrared codes, ultraviolet codes, or codes that rely on the manipulation of light polarization.

[0016] Figure 1 A system diagram showing an example of an image pre - processing system 100 is shown. In the example shown, the system 100 includes image data 110, an image capture device 130, a computing device 120, and optionally a lighting device 140. The image data includes an embedded element 112 and other elements 114. The image pre - processing system 100 will provide an output image to image - processing software 150 for further processing such as code extraction, decoding, etc.

[0017] In some embodiments, the image capture device 130 is a camera or other component configured to capture image data (such as a video stream, sequential images, or a static image). In some examples, the image capture device 130 can be a camera of a mobile device. In some cases, the image capture device 130 can include other components capable of capturing image data, such as a video recorder, an infrared camera, a CCD (charge - coupled device) or CMOS array, a laser scanner, etc. Additionally, the captured image data 110 can include at least one of an image, a video, an image sequence (i.e., multiple images acquired over a period of time and / or in a certain order), an image set, etc., and the term input image is used herein to refer to various exemplary types of image data.

[0018] In some embodiments, the lighting device 140 (e.g., a camera flash unit) is capable of emitting two or more simultaneous wavelengths and can be controlled by the computing device 120 and / or the image capture device 130. In some cases, assuming the image capture device 130 is sensitive to the selected wavelengths, the lighting device 140 can output wavelengths visible and invisible to the human eye (e.g., infrared or ultraviolet).

[0019] In some embodiments, at least one of the embedded element 112 and / or other elements 114 has known image attributes such that the computing device 120 can select a pre - processing method based on the known image attributes. Image attributes refer to attributes that can be recognized in an image, for example, color intensity, texture, etc.

[0020] The computing device 120 can be any device, server, or apparatus with computing capabilities, including but not limited to circuits, computers, processors, processing units, mobile devices, microprocessors, tablets, etc. In some cases, the computing device 120 can be implemented on a shared computing device. Alternatively, the components of the computing device 120 can be implemented on multiple computing devices. In some specific implementations, the various modules and components of the computing device 120 can be implemented as software, hardware, firmware, or a combination thereof.

[0021] In some embodiments, the preprocessing system 100 is designed to preprocess an image to generate an output image for standard decoding software for one-dimensional and multi-dimensional coded images (e.g., QR codes), the one-dimensional and multi-dimensional coded images utilizing highly visible materials, particularly those labels having retroreflective materials illuminated by the illumination device 140. Some embodiments of the present disclosure can be used in conjunction with decoding software that operates on hardware with limited resources, such as a smart phone or an embedded system having an embedded or connected color camera / imager equipped with a multi-wavelength (e.g., white or RGB light) illumination source. In some cases, the preprocessing system 100 can reduce clutter in the captured image by enhancing the embedded elements of interest in the image and suppressing other elements (including the background). In some cases, the preprocessing system 100 is optimized for speed rather than object recognition, which makes it useful for scanning video streams, sequential images (e.g., ticket scanning), or panoramic static images (e.g., labels on a wall).

[0022] Figure 2A is an example of a flowchart of an image preprocessing system. In some cases, the sequence of steps in the flowchart may not have an exact order. The image preprocessing system first receives an input image (step 210A), where the input image can be a video stream, an image sequence, or a static image. In some cases, the input image is captured by an image capture device. In some cases, the preprocessing system generates a single plane image (step 215A), where each single plane image corresponds to a color plane in the color space of the input image. Applicable color spaces include but are not limited to RGB (red, green, and blue), LAB (e.g., Hunter 1948 L, a, b color space, CIE 1976 (L*, a*, b*) color space), CMYK (cyan, magenta, yellow, and key (black)), HSV (hue, saturation, and value), HSL (hue, saturation, and lightness), HSI (hue, saturation, and intensity), sRGB (standard red, green, and blue) color spaces. In one example, a single plane image is generated by applying a Bayer filter to the input image such that each pixel is composed of a combination of three primary color (e.g., red, green, and blue) sub-pixels; the color planes are composed of the same color sub-pixels above a rectangular pixel array.

[0023] For a specific color plane, new images called P-images are generated using the color plane data. In one example, the P-image contains only the selected color plane data such that the P-image is then one-third the size of the original input image in a color space with three color planes. As another example, the P-image can be constructed by duplicating the selected color plane data in the remaining color planes such that the P-image size is equal to the original image size. As an example of an input image in the RGB color space, if the red plane is selected, the green and blue planes will be filled with the corresponding red data. The P-image will have one-third of the image resolution of the original image. As another example of constructing the P-image, functions F1(selected plane) and F2(selected plane) are applied to the selected single plane image data to generate the data for the remaining color planes; the P-image size will be equal to the original image size. Examples of functions can be, but are not limited to, linear functions (e.g., Fn(pixel) = K * pixel) or non-linear functions (e.g., Fn(pixel) = if (pixel <= K) then pixel = 0 else pixel = 1).

[0024] After generating the P-image, it is converted to a grayscale image, e.g., to simplify calculations and reduce the amount of time and / or resources for the calculations. The P-image can be converted to grayscale in several ways.

[0025] i. If the P-image uses only the selected single plane data, it is already multi-bit grayscale;

[0026] ii. If the P-image is in RGB format with the selected plane data, it can be converted to grayscale format by applying a conversion function GS(R,G,B) to generate a grayscale image. For each pixel with a value of R in the red channel, G in the green channel, and B in the blue channel, some exemplary functions include, but are not limited to:

[0027] GS(R,G,B) = [max(R,G,B) + min(R,G,B)] / 2(1)

[0028] GS(R,G,B) = (R+G+B) / 3(2)

[0029] GS(R,G,B) = 0.21*R + 0.72*G + 0.07*B(3)

[0030] The P-image can also be converted to 1-bit grayscale by thresholding, where the grayscale pixel value is 1 or 0 depending on whether the P-image pixel value is greater than a predetermined threshold. In some cases where multiple bits of data are required, the thresholded pixels can be assigned 0 or [(2^Res) – 1], where Res is the pixel resolution in bits. For example, if Res = 8 bits, the values are 0 and 255.

[0031] Next, the system calculates one or more histograms of the input image, each histogram corresponding to a single plane (220A) of the image. In some embodiments, the system selects color planes (step 225A) in the color space of the input image. In some cases, the color planes are selected based on the embedded elements of interest captured in the input image and / or other elements captured in the input image. In some cases, the color planes are selected based on the known color characteristics of the embedded elements of interest captured in the input image and / or the known color characteristics of other elements captured in the input image. For example, for an input image in the RGB color space with a monitor in the background, the green color plane or the red color plane will be selected. In some cases, the color planes can be selected based on knowledge of the image capture device, the lighting device, the embedded elements of interest, and the image background information, or alternatively, the color plane can be selected by applying a function (e.g., image average or standard deviation) to the single-plane image data and calculating the best candidate. Possible selection criteria can include but are not limited to data distribution (i.e., histogram), image noise, image contrast, image statistics, and dynamic range.

[0032] In some cases, step 220A is completed before step 225A. In some cases, step 225A is completed before step 220A. Next, the system identifies multiple peaks or ranges of interest (step 230A) in one of the one or more histograms, where one histogram corresponds to the selected color plane. In some cases, the histogram is filtered to remove high-frequency noise. In some cases, a convolutional matrix filter is used to smooth the high-frequency noise. In one example, the kernel (K) of the filter is a 3x3 matrix K[0:2,0:2], where K[1,1] = 0 and K[x<>1,y<>1] = 1, such that when the filter is applied, the calculation is simplified. In the above example, the kernel is selected for speed rather than fidelity. Other filters and kernels can be used for filtering. In some cases, the system selects the range of interest in the histogram of the selected color plane based on the known color intensity of the embedded elements and / or other elements of the input image. In some cases, the system selects two or more peaks in the histogram by determining local maxima, which is described in more detail below.

[0033] The system determines a threshold (240A) based on multiple peaks or values within a region of interest. In some examples, the embedded element is an element with high color intensity. For example, the embedded element is a retroreflective label. High-intensity elements are typically the brightest objects in the input image, e.g., elements with pixel values close to the maximum pixel image value. In some embodiments, the threshold is calculated as a function of multiple peaks. In some cases, the threshold is calculated as a function of the histogram values within the region of interest.

[0034] The system uses the threshold to further process the input image to generate an output image (250A). After setting the threshold, a thresholded image file T-image[N,M] = threshold(L-image[N,M], threshold) is generated. The thresholded image file T-image[N,M] can be passed to other image processing software, such as decoding software.

[0035] Figure 2B is another example of a flowchart of an image preprocessing system. First, the system illuminates a label, which is an example of an element of interest in a selected color (step 210B). In some cases, the selected color is a primary color. The system captures a single-plane image (step 215B). In some cases, the system can illuminate labels with other colors and capture other single-plane images. The system will calculate one or more histograms of the input image (step 220B). Other steps are the same as those in Figure 2A the steps described above.

[0036] Figure 2C shows a flowchart of an example of an image preprocessing system. Some of the steps in this process are optional, and some steps can be placed in a different order. In this example, the system captures an image or receives an image (step 210C). Next, the system separates the image into M component plane images (step 215C). For example, the system separates an RGB image into a red plane image, a green plane image, and a blue plane image. The system generates a histogram for each plane image (step 220C), and then selects plane P by applying the selection criteria described herein (step 225C). The system decides whether a filter should be applied by evaluating the histogram of plane image P and / or through configuration parameters (step 230C); if so, a noise filter will be applied (235C). Additionally, the system builds a peak array [1:N] from local peaks in histogram P (240C), and calculates the threshold TB as a function of the peak array [1:N] (step 245C).

[0037] The system evaluates whether the output image should be a binary image (step 250C), and if so, sets Min_Value = 0 and Max_Value = 1 (step 251C); and if not, sets Min_Value = 0 and Max_Value = Image_max, where for an image with R-bit pixels, Image_max = 2^R-1 (step 252C). The system will create a thresholded image T by thresholding all the pixels of plane P such that if the pixel <= TV, the pixel = Min_Value, otherwise the pixel = Max_Value (step 255C). The system receives an input on whether the output image should be a single-plane image (step 260C). The input can depend on the image processing software to receive the output image. If the output image is not a single-plane image, the system can calculate all the image planes [1:M] as a function of plane T (step 265C) and create the output image - T based on all the planes (step 285C). If the output image is a single plane, the output image - T = plane T (step 270C). Optionally, for the decoding software or another processing software, the system formats the image - T (step 275C). Additionally, the system sends or provides the image - T to the decoding software or another processing software (step 280C).

[0038] Figures 3A to 3I An exemplary process of an image preprocessing system is shown. Figure 3A An example of a histogram of an RGB image (top of the figure) that has been separated into its three component color planes (red, green, and blue shown from left to right below) is shown. In each color plane, a region of interest (ROI) outlining the range of possible values corresponding to the elements of interest is outlined. In some cases, the ROI is defined as a percentage of the maximum pixel value, e.g., the first 25% of the range of values (75% to the maximum value). The ROI of each color plane is examined, and one or two planes are selected based on the characteristics. In Figure 3A the example, the blue plane ROI is evenly filled with many samples, so it is not a good candidate. The red plane has a very large peak at the lower end of the ROI, which may make it more difficult to find the elements of interest in an example where the elements of interest are the brightest objects in the image. The green plane is more sparsely filled with less high-frequency noise, so it is the best candidate for this example.

[0039] When a plane has been selected, the system can choose whether the plane data or the histogram data needs to be filtered. Figure 3B Shows the green plane histogram selected from the Figure 3A exemplary RGB image. Figure 3D Shows the unfiltered histogram; Figure 3EShows a histogram filtered with a low-pass filter (LPF); and Figure 3F Shows a histogram generated from the green plane data filtered by the LPF. For this example, filtering the plane data has not improved the resulting histogram, while filtering the histogram data has reduced some noise, especially in the ROI.

[0040] Figure 3B Shows from Figure 3A An exemplary green plane histogram that has an ROI bounded by a dashed line. As previously mentioned, the ROI is the range of values that are most likely to contain the element image data of interest. Although thresholding the image is easier to decode, a method that can produce acceptable results, is computationally fast, and does not require thresholding is to establish a cut-off value corresponding to the lowest value in the ROI, and use this cut-off value to transform the green plane pixel data according to the following rule:

[0041] If (CPix[X,Y] < cut-off value)

[0042] Then TPix[X,Y] = 0

[0043] Otherwise TPix[X,Y] = CPix[X,Y]

[0044] Where CPix[X,Y] is the pixel value from the selected plane (e.g., the green plane), and TPix[X,Y] is the pixel value of the plane to be transmitted to the image processing software (e.g., the decoding software). The histogram of the transformed exemplary green plane is shown at the bottom of Figure 3C .

[0045] For cases where thresholding is required, Figures 3G to 3I Shows three examples of how the threshold ("Threshold") can be calculated. Figure 3G Shows Figure 3A The ROI of an exemplary green plane. In this histogram, the mean value of the ROI "Mean" is calculated in a conventional manner. In the histogram (i.e., [2^resolution - 1], where the resolution is the pixel resolution in bits), the mean value is subtracted from the maximum value (referred to as "Max"). This difference is divided by 2 and then added to the mean value to produce the threshold, such that

[0046] Threshold = (Max - Mean) / 2 + Mean(6)

[0047] Figure 3H Shows Figure 3A The ROI of an exemplary green plane. In this histogram, the mean value of the ROI and the standard deviation (referred to as "SD") are calculated again. The threshold is calculated as follows:

[0048] Threshold = Mean + N * SD(7)

[0049] Where N is a number between 1 and 2.

[0050] Figure 3I shows Figure 3A the ROI of an exemplary green plane. In this histogram, vertical lines indicating the peaks in the histogram have been added. In this example, inspection of the peaks shows that they cluster around three values (indicated by circles); in other cases, the peaks may be more diverse. In this case, to calculate the threshold, the difference between the minimum edge of the first peak (or the center if not clustered), Pk1, and the maximum edge of the second peak (or the center if not clustered), Pk2, will be calculated; this difference is named G. The threshold is calculated as follows:

[0051] G = Pk1 – Pk2

[0052] Threshold = Pk2 + M*G(8)

[0053] where M is a number between 0.25 and 0.75, and Pk1 > Pk2.

[0054] Once one or more of the selected planes have been processed (e.g., thresholded or transformed), the output image is assembled as previously described and then passed to software or a processing unit for further processing.

[0055] Figures 4A to 4I shows an example of an image preprocessing system preprocessing an image with an element of interest 400. The system calculates the histogram of each color plane image (as Figure 4A , Figure 4D and Figure 4G shown), where the histogram is shown in Figure 4B , Figure 4E , Figure 4H . Due to its relatively smooth data distribution and the large difference between the white peak (data clustered around 255) and the dark peaks (around 75 and 115), the system selects the red plane image. Examining the color plane images, the system finds that the blue plane has more high-brightness objects (e.g., monitor screens) than the red and green planes. This is marked with an underline in the histogram, where the blue histogram shows a much higher density (i.e., highest brightness values) in the rightmost quadrant of the histogram. The fourth quadrants of the green and red histograms have approximately equal densities.

[0056] The system uses one of the above-described embodiments to select a threshold and generate a thresholded image, as Figure 4C , Figure 4F and Figure 4IAs shown, where the element of interest 400 is magnified. The system evaluation thresholded color planes show that the element of interest in the red and green planes (but not the blue plane) can be easily separated, even though all three planes produce clear labeled images. Although the artifacts in the thresholded green plane are slightly fewer than those in the red plane, the red plane would be a better choice for thresholding because the first peak from the maximum value in the histogram (the right end of the red histogram) is better defined than in the green plane and is easier to locate in software (e.g., using differentiation and locating local maxima and minima).

[0057] The present invention should not be considered limited to the above specific examples and embodiments, as the detailed description of such embodiments is for the purpose of facilitating the illustration of various aspects of the present invention. On the contrary, the present invention should be understood to cover all aspects of the present invention, including various modifications, equivalent processes, and alternative devices that fall within the spirit and scope of the present invention as defined by the appended claims and their equivalents.

Claims

1. A method for processing an input image with embedded elements, the method comprising: The processor calculates a corresponding histogram associated with the input image for each respective color plane of one or more color planes in the color space of the input image; The processor selects a selected color plane from the one or more color planes in the color space; The processor identifies a region of interest in a first histogram among the corresponding histograms based on (i) a first known color characteristic of the embedded element and (ii) a second known color characteristic of an element represented in the input image and different from the embedded element, wherein the first histogram is calculated on the selected color plane; Based on the region of interest, the processor determines a threshold; and The processor processes the input image based on the threshold to generate an output image, wherein the output image is a binary image based on the threshold, wherein if the color intensity value of a pixel in the output image is greater than the threshold, the pixel has a maximum value, and wherein if the color intensity value of a pixel in the output image is less than or equal to the threshold, the pixel has a minimum value.

2. The method according to claim 1, wherein the input image is an image sequence or a static image.

3. The method according to claim 1, further comprising: The processor extracts a representation of the embedded element from the output image.

4. The method according to any one of claims 1 - 3, wherein the embedded element includes code, the method further comprising: The processor decodes encoded information from the embedded element, the encoded information representing the code.

5. The method according to claim 4, wherein the code is a two - dimensional code.

6. The method according to claim 4, wherein the code is a one - dimensional code.

7. The method according to any one of claims 1 - 3, further comprising: Before identifying the region of interest, the processor filters the first histogram with a filter.

8. The method according to claim 7, wherein the filter includes a low - pass filter.

9. The method according to claim 8, wherein the kernel of the low - pass filter is a 3x3 matrix K[0:2,0:2], where K[1,1]=0 and K[x<>1,y<>1]=1.

10. The method according to claim 1, wherein the maximum value is calculated by the formula 2^r – 1, where r is the pixel resolution of the output image, and wherein the minimum value is 0.

11. The method according to any one of claims 1 - 3, wherein the threshold is a function of the histogram values within the region of interest.

12. The method according to any one of claims 1 - 3, wherein the first known color characteristic of the embedded element is the color intensity corresponding to the highest peak in the first histogram.

13. A device for processing an input image with embedded elements, the device comprising: A memory device configured to store an input image having an embedded element; and A processor coupled to the memory device, the processor being configured to: Calculate a corresponding histogram associated with the input image for each respective color plane of one or more color planes in the color space of the input image stored in the memory device; Select a selected color plane from the one or more color planes in the color space; Identify a region of interest in a first histogram among the corresponding histograms based on (i) a first known color characteristic of the embedded element and (ii) a second known color characteristic of an element represented in the input image and different from the embedded element, wherein the first histogram is calculated on the selected color plane; Determine a threshold based on the region of interest; and Process the input image based on the threshold to generate an output image, wherein the output image is a binary image based on the threshold, wherein if the color intensity value of a pixel in the output image is greater than the threshold, the pixel has a maximum value, and wherein if the color intensity value of a pixel in the output image is less than or equal to the threshold, the pixel has a minimum value.

14. The apparatus according to claim 13, wherein the input image is an image sequence or a still image.

15. The apparatus according to any one of claims 13 - 14, wherein the processor is further configured to filter the first histogram with a filter before the region of interest is identified.

Images