System and method for decoding multiple optical codes

Abstract

To provide a system, device, and method for tracking a plurality of optical patterns such as a barcode.SOLUTION: A method disclosed herein comprises acquiring a plurality of frames from an image sensor. The plurality of frames includes respective codes for images. The plurality of frames comprises a first frame and a second frame, where the second frame is acquired after the first frame. The method also comprises identifying a code in the first frame, extracting features from the code, generating a filter, extracting features from the second frame, generating a response map based on the features extracted from the second frame, and locating the code based on the response map.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Non-Provisional Application No. 17 / 186,909, filed February 26, 2021, which claims priority to U.S. Provisional Application No. 63 / 149,009, filed February 12, 2021, which is incorporated herein by reference for all purposes. Application No. 17 / 186,909 is a continuation-in-part of U.S. Non-Provisional Application No. 17 / 139,529, filed December 31, 2020, which is a continuation-in-part of U.S. Non-Provisional Application No. 16 / 920,061, filed July 2, 2020, now U.S. Patent No. 10,963,658, issued March 30, 2021, which is a continuation-in-part of U.S. Non-Provisional Application No. 16 / 920,061, filed July 2, 2020, now U.S. Patent No. 10,963,658, issued March 30, 2021. No. 10,963,658 claims the benefit of priority to U.S. Provisional Application No. 63 / 017,493, filed April 29, 2020, U.S. Provisional Application No. 63 / 003,675, filed April 1, 2020, and U.S. Provisional Application No. 63 / 019,818, filed May 4, 2020, which are incorporated by reference herein for all purposes. Application No. 17 / 186,909 is a continuation-in-part of U.S. Patent Application No. 16 / 905,722, filed June 18, 2020, now U.S. Patent No. 10,846,561, issued November 24, 2020, which is incorporated by reference herein for all purposes. Application No. 17 / 186,909 claims priority to U.S. Provisional Application No. 63 / 025,850, filed May 15, 2020, which is incorporated by reference for all purposes. Application No. 17 / 186,909 is a continuation-in-part of U.S. Patent Application No. 17 / 105,082, filed November 25, 2020, which claims the benefit of priority to U.S. Provisional Application No. 63 / 044,635, filed June 26, 2020, which are incorporated by reference for all purposes. [Background technology]

[0002]

[0002] This disclosure relates generally to decoding codes, and more particularly, but not exclusively, to decoding barcodes in retail environments. Barcodes have traditionally been scanned using dedicated scanners. For example, barcode scanners with lasers are used to shine light onto the barcode, and reflected light from the barcode is detected and used to decode the barcode. As mobile devices with cameras (e.g., smartphones and tablets) become more common, mobile devices are being used to decode codes by capturing an image of the code and using image analysis to decode the code. An example of a method for using a smartphone to decode barcodes is provided in U.S. Patent No. 8,596,540, issued December 3, 2013. Summary of the Invention [Problem to be solved by the invention]

[0003]

[0003] This disclosure generally relates to tracking and decoding computer-readable codes (e.g., barcodes; QR codes). For example, a barcode may be a stock-keeping code (SKU) in a retail setting. However, the systems and / or processes described herein may also be used to track and / or identify other features, such as text through optical character recognition, in other settings (e.g., scanning a passport at an airport terminal). Furthermore, many applications are becoming web-based. However, web-based applications may have limited computational resources. As many operations become more computationally intensive, improvements are needed for capturing and / or visualizing data from images, particularly images captured by mobile devices. [Means for solving the problem]

[0004] In a particular aspect, a system for tracking an optical code includes a camera. The camera may be configured to acquire multiple frames. The system may include one or more processors. The one or more processors may be configured to acquire multiple frames from an image sensor, each including an image of a code, the multiple frames including a first frame and a second frame, and / or the second frame being acquired after the first frame, identify the code in the first frames, extract features from the code in the first frames, create a filter based on the extracted features of the code, extract features from the second frames, generate a response map based on the features extracted from the second frames combined with the filter, the response map providing a probability of the location of the code in the second frame, and / or confirm the location of the code in the second frame based on the response map without decoding the code. In some embodiments, the filter is a correlation filter, the code is a first code, and the one or more processors are configured to identify the second code in the second frame without decoding the second code, and the filter is created by transforming extracted features of the code into frequency space using a Fourier transform, and / or the response map is generated by transforming the second frame into frequency space using a Fourier transform to generate a frequency image and convolving the frequency image with the filter.

[0005]

[0005] In a particular embodiment, a method for tracking an optical code includes steps of acquiring a plurality of frames from an image sensor, each of the plurality of frames including an image of the code, the plurality of frames including a first frame and a second frame, and / or the second frame being acquired after the first frame; identifying the code in the first frame; extracting features from the code in the first frame; creating a filter based on the extracted features of the code; extracting features from the second frame; generating a response map based on the features extracted from the second frame combined with the filter; and locating the code in the second frame based on the response map without decoding the code. In some embodiments, the filter is a correlation filter, the code comprises a machine-readable barcode, the filter is created by transforming extracted features of the code into frequency space using a Fourier transform, the response map is created by transforming the second frame into frequency space using a Fourier transform to generate a frequency image and convolving the frequency image with the filter, the plurality of frames comprises a third frame, the third frame is acquired after the second frame, the plurality of frames comprises a fourth frame, the fourth frame is acquired after the third frame, the code is a first code, the plurality of frames comprises 10 frames taken within 1 second, and / or the method comprises: calculating a homography pose change of the image sensor from the first frame to the second frame based on identifying the position of the code; predicting a subarea of a third frame that includes an image of the code based on the position of the code in the second frame; decoding the code using the image of the code from the third frame; identifying the position of the second code in the third frame without decoding the second code; decoding the second code in a fourth frame; presenting the second frame on a display with a graphic on the display indicating the position of the code; and / or identifying the second code in the second frame without decoding the second code.

[0006] In certain embodiments, an apparatus for decoding an optical pattern includes a camera. The camera may include an image sensor. The camera may include a lens configured to focus light onto the image sensor. The lens may be a wide-angle lens. The apparatus may include one or more processors. The one or more processors may be configured to acquire multiple images using the camera, analyze the multiple images to detect the optical pattern in the multiple images, decode the optical pattern to acquire a data payload, and / or store the data payload in a memory device of the apparatus. The lens may be set to a fixed focus during acquisition of the multiple images. The exposure time for each of the multiple images may be 1 / 500 of a second or less. The multiple images may be acquired at a frame rate of the camera. The frame rate may be 30 frames per second or greater.

[0007]

[0007] In certain embodiments, a method for decoding an optical pattern includes acquiring a plurality of images using a camera, analyzing the plurality of images to detect an optical pattern in the plurality of images, decoding the optical pattern to acquire a data payload, and / or storing the data payload in a memory device, wherein the camera lens is set to a fixed focus during acquisition of the plurality of images, the camera lens is a wide-angle lens, the exposure time for each of the plurality of images is 0.500 seconds or less, and the plurality of images are acquired at a frame rate of the camera, and / or the frame rate is 30 frames per second or greater.

[0008] In certain embodiments, an apparatus includes an attachment for a mobile device having a torch and a camera with a first lens, the apparatus including a second lens and / or a third lens, the attachment configured to be removably coupled to the mobile device, the second lens configured to increase the angle of view of the camera of the mobile device while the attachment is coupled to the mobile device, and / or the third lens configured to focus the torch light while the attachment is coupled to the mobile device. In some embodiments, the third lens is configured to focus the torch light in a direction normal to the torch, the third lens is configured to focus the torch light into an elliptical shape, the elliptical shape being non-circular, the attachment is a case that covers at least a portion of two or more sides of the mobile device, and / or the attachment further includes a magnet.

[0009]

[0009] In certain embodiments, the method includes using a torch on the mobile device to aim the mobile device at the optical pattern.

[0010] Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for purposes of illustration only and are not intended to necessarily limit the scope of the present disclosure.

[0011]

[0011] The present invention will now be described in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1] 1 illustrates an embodiment of a tracking method. [Figure 2] 10 illustrates an embodiment in which the code in each frame is not tracked and the code is not decoded. [Figure 3] FIG. 1 shows a diagram of one embodiment for creating a correlation filter. [Figure 4]FIG. 1 shows a diagram of one embodiment for using a correlation filter to generate a response map for tracking chords. [Figure 5] 1 illustrates an embodiment for decoding codes across multiple frames. [Figure 6] 1 shows a flowchart of one embodiment of a process for tracking code. [Figure 7] An example of segmenting products on a shelf from an image. [Figure 8] 1 is an embodiment of a shelf diagram. [Figure 9] 10 shows a flowchart of one embodiment of a process for mapping objects onto a shelving unit. [Figure 10] Here is an example of blending two images of shelves. [Figure 11] 1 is an embodiment of a shelf diagram with a visual image of the shelf. [Figure 12] 1 shows a flowchart of one embodiment of a process for creating a visual image of a shelf. [Figure 13] FIG. 1 illustrates an embodiment of a mobile device for rapidly scanning an optical pattern. [Figure 14] 1 shows a chart illustrating scan ranges as a function of camera focus settings for scanning a first type of barcode. [Figure 15] 10 shows a chart illustrating scan range as a function of camera focus setting for scanning a second type of barcode. [Figure 16] 1 shows a flowchart of one embodiment of a process for decoding an optical pattern using a digital camera. [Figure 17] 1 is an embodiment of an attachment for a mobile device. [Figure 18] 10 is another embodiment of an attachment for a mobile device. [Figure 19] FIG. 1 is a block diagram of one embodiment of a computer system.

[0013]

[0031] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used herein, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE INVENTION

[0014]

[0032] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing the preferred exemplary embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0015]

[0033] Matrix Scan

[0034] Many applications are becoming web-based. However, web-based applications may have fewer computational resources than native applications. For example, a native application may be used to track a barcode based on decoding barcodes from multiple images. However, decoding the barcode is computationally intensive and can cause delays when moved to a web-based application. Therefore, in some embodiments, the barcode is tracked over several frames, but is only decoded periodically for the web-based application to be used to decode the barcode. In some embodiments, a frame is one of a series of separate photographs that make up a film or video.

[0016]

[0035] Referring first to FIG. 1 , one embodiment of a tracking scheme is shown. This tracking scheme can be used in native applications. In FIG. 1 , in a first frame 108-1 (e.g., initialization) at time T=1, a first code 104-1 and a second code 104-2 are decoded and their locations are confirmed. The first frame 108-1 corresponds to a first image acquired by an image sensor (e.g., from a camera in a mobile device). The code 104 is an optical pattern. The code 104 can be a machine-readable code, such as a one-dimensional barcode or a two-dimensional barcode (e.g., a QR code) having multiple horizontal lines, a symbol (e.g., a dollar sign, a triangle, etc.), a number, and / or a letter. For example, the code 104 can be a price, a VIN number for identifying a vehicle, a credit card number, a license plate number, a serial number, a tire code (TIN), or a date (e.g., an expiration date). The code 104 can be found in a variety of environments and / or applications. For example, the code 104 may be part of or on a shipping label, product label, passport, shipping invoice, driver's license, ID card, credit card, check, license plate, digital display (e.g., electronic price tag), utility meter, ID document with machine readable zone (MRZ), or retail receipt.

[0017]

[0036] At time T=2, a prediction 112 is calculated (e.g., assuming smooth motion of the mobile device), the code 104 is decoded in a second frame 108-2 (the second frame 108-2 corresponds to a second image acquired by the image sensor), the code 104 is matched against the prediction 112, and an updated position 116 of the code 104 is confirmed based on the code 104 decoded from the second frame 108-2.

[0018]

[0037] Some concerns with the above scheme include inaccurate and / or missed detections because the bounding box (e.g., updated position 116) based on decoding of the one-dimensional code may be inaccurate. If decoding is not performed every frame, detections may be missed and / or the detection track may be inconsistent. An example of a detection track is shown in updated position 116, where the dotted outline was the previous position. Also, decoding may be more difficult with faster movement and lower resolution.

[0019]

[0038] One possible solution is to use more data, such as predicting the position of the code 104 based on image history (e.g., to establish the trajectory of the code 104 within the field of view), using known locations of the code 104 and / or background structures (e.g., shelving units), and / or leveraging additional sensor data (e.g., inertial measurement unit (IMU) data from a gyroscope to predict attitude changes of a mobile device). However, using more data can cause delays when using web-based applications. Furthermore, not all devices have the computing power and / or equipment (e.g., an IMU) to use more data. Therefore, another possible solution, described in more detail below, is to decode the code in the first and third frames and track the position of the code 104 in the second frame between the first and third frames.

[0020]

[0039] 2 illustrates an embodiment in which the code in each frame is tracked and not decoded. In the first frame 108-1, at time T=1, an algorithm is used to search for locations that look like code 104. The locations that look like code 104 are decoded (or an attempt is made to decode the code). Tracking of code 104 is also initialized (e.g., as described in connection with FIG. 3 by creating a correlation filter).

[0021]

[0040] In the second frame 108-2, at T=2, the code 104 is tracked and not decoded. The tracking algorithm attempts to find a new position of the tracked code in the second frame 108-2. In the second frame 108-2, a first bounding box 204-1 is calculated relative to the second frame 108-2 for which the first code 104-1 is to be calculated (e.g., the bounding box 204 is simply calculated and / or overlaid on the second frame 108-2 in the user interface of the mobile device), and a second bounding box 204-2 is calculated relative to the second frame 108-2 for which the second code 104-2 is to be calculated. For example, as described in connection with FIG. 4, a correlation filter is used to create a response map and determine the position of the bounding box 204. Tracking in the second frame 108-2 does not rely on limitations that exist when decoding a code. For example, the code 104 may be blurred in the second frame 108-2. In one embodiment, the correlation filter is learned at tracking initialization and continuously updated during the tracking process to adapt, for example, to changes in viewpoint. In another embodiment, the correlation filter operates on a selected subset of image features, which may be highly specific to the barcode and may be extracted using a pre-trained neural net.

[0022]

[0041] In the third frame 108-3, at T=3, the code 104 is scanned and tracked. For example, an algorithm is used to search for locations that appear to be code 104. Locations that appear to be code 104 are decoded and / or the tracking algorithm identifies new locations of the code 104 within the third frame 108-3 for codes 104 that could not or were not decoded.

[0023]

[0042] While only one frame, the second frame 108-2, is shown between the first frame 108-1 and the third frame 108-3, it should be understood that the second frame 108-2 can be one of many frames between the first frame 108-1 and the third frame 108-3. It should also be understood that the third frame 108-3 action can be repeated (e.g., periodically or according to an event such as the location of the bounding box not moving more than a set distance, which may indicate there is likely little motion blur in the image) with the second frame 108-2 action occurring for one or more frames. Thus, a tracking algorithm can determine the location of the code between scanning the code and decoding the code. For example, a smartphone camera captures images at 30 frames per second. The third frame 108-3 action in FIG. 2 is configured to occur at a 2 Hz interval (sometimes referred to as the scan rate), and the second frame 108-2 action in FIG. 2 is configured to occur at frames captured by the camera during that interval. In some embodiments, the interval is at a frequency of 0.3, 0.5, 1, 2, or 5 Hz or greater, and at a frequency of 5, 10, or 20 Hz or less. In some configurations, the camera has a frame rate of 1, 5, 10, or 20 Hz or greater, and 10, 30, 60, or 120 Hz or less. There is a trade-off in the frame rate used between wanting a high frame rate so that objects do not move too much between frames, and wanting to discover new barcodes as they appear as quickly as possible. Applicant has found that 60 ms to 100 ms, e.g., 80 ms, between successive scans is a good trade-off, as it conveys a "fine-grained" experience to the user.

[0024]

[0043] In some embodiments, only detected new codes are decoded in subsequent frames. Thus, in some embodiments, barcode scanning and / or barcode decoding is performed only in the first frame 108-1. For example, a store employee scans several barcodes with his or her smartphone. In the first frame, the scan identifies two barcodes and decodes two barcodes. The two barcodes are then tracked, and a green dot is overlaid on the image displayed to the smartphone user, using the smartphone screen to provide the user with an augmented reality display of the decoded barcodes. The user can see a third barcode on the screen that is not covered by the green dot, so the user continues to move the smartphone and scans the third barcode. In the frame in which the image is searched for locations that appear to be barcodes, three locations are identified: two locations corresponding to the two previously identified barcodes and a new location. The two locations corresponding to the two previously identified barcodes are not decoded, while the new location reveals a third barcode. A third barcode is then tracked, and a green dot is overlaid on the screen where the third barcode is tracked. Thus, the user can see which code has been decoded without the application having to decode each barcode in each frame. Because decoding codes can consume more computing resources than tracking, tracking can improve the functionality of mobile devices, especially when the application is web-based and not running as a native application.

[0025]

[0044] In some embodiments, scanning a code can include both a search for locations that look like a code and a decoded code, and can be split to occur in different frames. For example, a search for locations that look like a code is performed in the first frame, and five locations are identified. In frames 2 through 10, five locations are tracked. In frame 11, the code in the first location is decoded. In frames 12 through 20, five locations are tracked. In frame 21, the code in the second location is decoded. In frames 22 through 30, five locations are tracked. In frame 31, a search for locations that look like a code is performed, and a sixth location is identified. The process continues searching for new locations and decoded codes.

[0026]

[0045] In some configurations, the tracking described in connection with FIGS. 2-6 can enable more stable tracking because the tracking does not rely on a visually decodable barcode (e.g., as described in FIG. 1). The tracking can provide a user with a smoother visualization due to increased frame rates and / or reduce energy consumption when performing "normal" barcode scanning (e.g., because codes already scanned are not rescanned). Decoding codes in each frame may not be useful in various situations. In one example, a user is scanning multiple codes for inventory control, the barcodes may be far away. If a user quickly moves their mobile phone to scan a barcode, the barcode may be blurred from motion blur. Therefore, tracking that relies on decoding codes may lose track of the barcode.

[0027]

[0046] In some embodiments, the method includes visually extracting the code, establishing correspondence between frames of decoding, predicting the location of the code, and re-scanning, where predicting is performed by independently predicting the location of multiple codes and / or by using only image data (e.g., not IMU data).

[0028]

[0047] In some embodiments, the tracking algorithms described in Figures 2-6 can be implemented with other algorithms, such as prediction algorithms. For example, the tracking algorithms can be used in combination with a homography prediction algorithm or when the homography prediction algorithm fails. In some embodiments, the barcodes can be tracked as a single rigid structure. Taking advantage of the fact that barcodes do not move relative to each other can help reduce computational complexity and improve fault tolerance / fault detection.

[0029]

[0048] FIG. 3 shows a diagram of one embodiment for creating a correlation filter 304. FIG. 3 is an example of initialization. Extracted features 308 are obtained from the code 104. The extracted features can be high-contrast areas (e.g., image gradients), lines, corners, etc. In many scenarios, the barcode is placed on a rigid object or surface, so the extracted features 308 can include geometric cues in addition to the code 104 itself (e.g., the edges of a shelf or dollar sign can be part of the extracted features 308). In some embodiments, the extracted features 308 are transformed from the visual domain to the frequency domain using a Fourier transform (FT) and combined with a target output 312 to form the correlation filter 304. The Fourier transform is efficient and reduces computation time. In another embodiment, the operation is performed in the spatial domain. In some embodiments, to "train" the correlation filter, the features are transformed in the frequency domain, and then a mask / filter is calculated. When convolved with the features, this yields the target output. The target output is a desired response map in the spatial domain (e.g., a probability map of object locations; essentially a matrix the size of the analyzed patch with a single peak at the center of the patch, indicating the object's central location), which is transformed to the frequency domain.

[0030]

[0049] FIG. 4 shows a diagram of one embodiment for using a correlation filter 304 to generate a response map 404 for tracking a code 104. In FIG. 4, extracted features 408 are obtained from an image 412. The image 412 is a portion of multiple image frames acquired by a camera. The image 412 includes a code 104. The extracted features 408 are transformed into frequency space using a Fourier transform and combined with a correlation filter 304 (e.g., convolution) to generate the response map 404. The extracted features 408 can be combined with a window 416 before combining with the correlation filter 304. The window 416 can be used to limit the search for the code 104 to an area smaller than the entire area of the image 412. The response map 404 provides a measure of the correlation. Thus, the whiter an area of the response map 404 is, the more likely that the code 104 is located in that area.

[0031]

[0050] The frame rate for tracking can be adjusted (e.g., dynamically) based on several criteria, including one or more of the speed of movement, the size of the window 416, and how far the cord 104 is from the camera (e.g., if Dad is farther away, a larger window 416 is selected and / or the frame rate of the image for tracking is increased).

[0032]

[0051] FIG. 5 illustrates an embodiment in which a code 104 is scanned over multiple frames 508. In some configurations, scanning a code (e.g., an identification code and / or a decoding code) can be computationally intensive. Identifying a code involves verifying the presence of the code in an image and / or the location of the code within the image. Decoding a code involves verifying what the code represents. For example, identifying a barcode involves verifying the presence of a barcode in a frame, and decoding a barcode involves interpreting the barcode's series of white and black lines as representing a string of numbers. By distributing the scan over multiple frames, computation can be distributed in time. FIG. 5 illustrates a first frame 508-1, a second frame 508-2, a third frame 508-3, and a fourth frame 508-4. In FIG. 5, scanning occurs over multiple frames. In the first frame 508-1, a first code 104-1 and a second code 104-2 are tracked. In the second frame 508-2, code scanning is performed only on the top half of the second frame 508-2. Thus, the first code 104-1 is identified and decoded, and the second code 104-2 is tracked without decoding. In the third frame 508-3, the code 104 is tracked. In the fourth frame 508-4, code scanning is performed only on the bottom half of the fourth frame 508-4. Thus, the second code 104-2 is identified and decoded, while the first code 104-1 is tracked without decoding. In some embodiments, the frame 508 is divided into quadrants, and new code (e.g., scanning) discovery is performed every fourth frame in each quadrant. In some embodiments, "attention" or "scene semantic" based decoding is used. In some configurations, this may mean that decoding may be limited to frames or portions of the observed scene that undergo dramatic visual changes, and / or that decoding may be limited to areas of special interest for a particular task (e.g., barcode labels, parts of a shelving unit, etc.).

[0033]

[0052] 6 illustrates a flowchart of one embodiment of a process 600 for tracking a code. The process 600 begins in step 604 by acquiring multiple frames from an image sensor, each including an image of a code, the multiple frames including a first frame and a second frame, the second frame being acquired after the first frame. In step 608, a code is identified in the first frame. In step 612, features are extracted from the code. For example, extracted features 308 are identified from the code 104 of FIG. 3. In step 616, a filter is created. For example, a correlation filter 304 is created based on the extracted features 308 of FIG. 3.

[0034]

[0053] In some embodiments, detecting and tracking a code can mean tracking multiple codes, such as a barcode and a line of text. For example, in a retail store, a price tag can be tracked, and in a second step, one or more codes on that price tag can be decoded, including one or more barcodes that identify the product, a series of numbers indicating the price, and / or a series of letters that describe the product name.

[0035]

[0054] In step 620, features are extracted from the second frame. For example, extracted features 408 are generated from image 412 of FIG. 4. In step 624, a response map is generated based on the features extracted from the second frame. For example, response map 404 is generated based on a convolution of extracted features 408 with correlation filter 304 of FIG. 4. In step 628, the location of the code is confirmed based on the response map. For example, areas of higher magnitude in the response map indicate likely locations within the response map of the code. Tracking the code as described in process 600 does not involve decoding the code (e.g., to conserve computational resources).

[0036]

[0055] A graphic indicating the location of the code, such as a box or the outline of a box, can be overlaid on an image containing the code to provide the user with an augmented reality output indicating the location of the code. The graphic can change, such as changing color, to indicate whether the code has been decoded. For example, if a code is identified as present but the camera is too far from the code to decode it, a red outline around the code may be displayed to the user. The user can then move the camera closer to the code. Once the code is decoded by an application (e.g., running on the device or web-based), the graphic changes to a green box, indicating to the user that the code has been successfully decoded.

[0037]

[0056] By tracking the code, the subarea of a subsequent frame can be predicted based on the location of the code in the second frame, so that when scanning is performed in subsequent frames, the scan of the code can be matched with the location of the code.

[0038]

[0057] In some embodiments, a system for decoding multiple optical codes includes a mobile data collector enabled with web browser software and a display, a camera module, two or more optical codes disposed with at least one object, and / or one or more processors that receive identification of two or more optical codes in one or more images captured by the camera module, decode the one or more optical codes, and / or visualize the decoded codes on a display of the mobile data collector. The system can include a mobile data collector enabled with web browser software and a display, a camera module, two or more optical codes disposed with at least one object, and / or one or more processors that detect the presence of two or more optical codes in images captured by the camera module, decode the two or more detected optical codes in the images, visualize the locations of the detected codes on a display of the mobile data collector, and / or visualize the decoded codes on a display of the mobile data collector. The system may include a mobile data collector enabled with a display and a camera module, one or more processors that detect the presence and location of a plurality of optical codes disposed with at least one object and / or the plurality of optical codes in a first image captured by the camera module, decode a subset of the detected optical codes from the first image, track the location of the plurality of optical codes from the first image to a second image, detect the presence of the plurality of optical codes in a second image captured by the camera module, and / or decode a subset of the detected optical codes from the second image, the subset consisting of codes not decoded from the first image.

[0039]

[0058] Code Mapping

[0059] Code mapping can be used to enable a user (e.g., a store employee) to capture the location of a code (e.g., SKU / product) within an environment (e.g., on a store shelf). Identifying and tracking codes can be useful to enable the creation of a virtual diagram of the physical structure. For example, a virtual diagram of a shelving unit can be used to help answer one or more of the following questions: Where on the shelf is the product, on which shelf, and / or at what height? Are the products in the correct location (e.g., does the product placement match the planogram)? What does a particular shelf look like visually? What was the condition of the shelf two weeks ago? Are the products priced correctly? Do the products correspond to their labels? Are there few products on the shelf (e.g., if the shelf is being restocked)?

[0040]

[0060] In some embodiments, cloud services for storage, APIs, dashboards, etc. are used to allow multiple users to contribute and / or search data. For example, in a retail setting, multiple employees with multiple mobile devices may contribute to data capture, and / or different parties may be able to view portions of the data in raw and / or aggregated versions. Cloud-based applications can also enable faster software development cycles.

[0041]

[0061] In some situations, the mapping is limited to a particular structure. For example, in a retail environment, a shelf is imaged as a two-dimensional structure, and multiple barcodes are mapped to lie on a vertically oriented plane (the normal to the vertical plane points to the horizontal). Products on the shelf unit can be identified by the barcodes on the shelf unit. In some embodiments, products on the shelf unit are identified by visual product recognition.

[0042]

[0062] Data can be stored remotely (e.g., in the cloud). Data can be agnostic from the capture and / or display device. For example, different mobile capture devices (e.g., different tablets and / or smartphones can be used) and captured data display can be in a mobile app or web-based dashboard. Capture runs can be per SKU, per shelf unit, and / or per store and can be flexibly combined from different capture devices.

[0043]

[0063] In some configurations, code mapping can be used for planogram compliance (e.g., comparing SKU locations to planned locations), planogram analysis (e.g., for sales numbers), in-store navigation to products, and / or AR marketing (e.g., instant sales when a customer is within a predetermined distance of a product).

[0044]

[0064] In certain embodiments, creating a code map includes identifying multiple SKUs (e.g., by reading shelf barcodes and / or by product recognition), segmenting SKU areas (e.g., areas of one type of product; done by image segmentation), determining two-dimensional layouts (e.g., which SKUs are next to other SKUs on a shelf unit and / or aisle), calculating inter-barcode distances (e.g., absolute distance in centimeters or relative distances), capturing three-dimensional layouts of barcodes (e.g., relationships between multiple aisles), and / or understanding the concept of a shelf (e.g., vertically divided units).

[0045]

[0065] SKUs can be identified by scanning barcodes on shelves. To identify SKUs through product recognition, a classifier is run on the image that can simultaneously label the product and / or segment its contours. The contours can be rectangular or pixel-wise image segmentation (e.g., using a convolutional neural network (CNN)-based classifier). The classifier can be pre-trained on a common data corpus (e.g., ImageNet) and fine-tuned / adapted for the product recognition use case. Preferably, new products can be added with several examples (e.g., one-shot learning). Examples of CNN-based object detection frameworks are YOLO, R-CNN, SSD, etc., although some can also be custom-built.

[0046]

[0066] FIG. 7 is an example of segmenting products on a shelf from an image. FIG. 7 shows frame 704, which is an image of product 708 arranged on shelf 712. Segmentation of SKU areas can be performed by image segmentation. If product 708 can be recognized directly from the image, a classifier can be used to segment the SKU area. If product 708 is recognized via a barcode 716 on shelf 712, this can be done based on image content, taking into account the barcode location. An exemplary approach involves determining whether product 708 corresponding to barcode 716 is above or below barcode 716. Determining whether barcode 716 is above or below (or to the side of) product 708 can be solved by a trained binary classifier, ad hoc rules, and / or human input. Image analysis is then used to expand regions (e.g., above or below the code) left or right based on similar image content. Within frame 704 of FIG. 7, product 708 corresponding to barcode 716 is above barcode 716. Similar image content can be measured using, for example, interest point matching, (cross-)correlation, color analysis, etc. Segmentation can be based on the assumption of using undistorted images and / or that similar products are horizontally aligned with shelf rows. Segment 720 (dashed line) of product 708 indicates product 708 that corresponds to barcode 716.

[0047]

[0067] FIG. 8 is one embodiment of a shelf diagram 800. The shelf diagram 800 provides information about the two-dimensional layout of SKUs (e.g., which SKUs are next to another SKU). In this embodiment, the SKUs are recognized by barcodes. When only one barcode is visible at a time, the user scans the barcodes in a defined order (e.g., a left-to-right sequence for each shelf; starting at the top of the shelf and moving down row by row). However, this only provides the layout, not the distance (relative or actual). As previously discussed with matrix scanning, when two or more barcodes are visible, the barcodes are mapped relative to each other. For example, starting from the top left of the shelf, previously referenced barcodes are kept in the current field of view, and newly recognized barcodes can be referenced (e.g., left, right, above, below) relative to one or more known barcodes. Some assumptions can simplify the mapping. For example, in some configurations, it can be assumed that the barcode capture view is aligned with the shelf row and there are no gaps. Approximate relative distances can be calculated using the spacing between barcodes in the image.

[0048]

[0068] If only one barcode is visible in the captured view to provide the relative distance between the barcodes, some embodiments can begin by capturing one or more overhead images of the shelving unit. In these overhead images, the barcodes are not (or cannot be) decoded (e.g., they are too small to identify the barcode location area). The overhead images are rectified, dewarped, and aligned with the shelf rows. If there are multiple views with different barcodes, they are stitched together or at least referenced to each other. Individual barcodes are then captured, including the image content surrounding each barcode. Individual barcode images can be registered to the overhead image using image content, with or without relying on the identified barcode area. Matching can be performed using image interest points, for example, using RANSAC (Random Sample Consensus) for homography or other image analysis for estimation. In some embodiments, registering individual barcode images to the overhead image includes identifying horizontal lines corresponding to the shelves and using the horizontal lines for horizontal image alignment and / or image compositing. This approach has the advantage of capturing the (relative) distance between barcodes with the help of an overhead image.

[0049]

[0069] In some embodiments, the absolute distance between barcodes is calculated (e.g., in cm). For example, matrix scanning can be used to simultaneously track two or more barcodes, with or without the barcodes being decoded. The algorithm can include the assumption that the shelf is planar (i.e., a two-dimensional problem in which the codes are located in the same plane), which can simplify some calculations. In a first option, the scale can be determined by the known width of a fiducial in the image. For example, if the height and / or width of a barcode is known, such as 1.2 cm, its scan can be used as a reference to convert relative measurements to absolute measurements. The same type of barcode can appear in multiple images, providing redundancy. In a second option, the scale can be determined using a known, calibrated camera and mobile device (e.g., using SLAM (simultaneous localization and mapping)). In a third option, a stereoscopic or depth camera can be used. The above three options are not mutually exclusive.

[0050]

[0070] A 3D model of the store can be generated. The 3D model can be created by combining multiple 2D models (e.g., shelving units) and / or using a single device by implementing a SLAM algorithm or a structure from motion (SfM) pipeline (Apple's ARKit or Google's ARCore offer such reconstruction capabilities on their respective mobile platforms). Barcodes can be decoded with a magnified scan and localized within the 3D store model. Another option is to use multiple cameras (stereo) and / or depth cameras. In a further option, 2D models of shelves are created and the user identifies the shelf locations within the store (e.g., using a store layout blueprint). In another embodiment, the store layout model is derived from an existing CAD model, resulting, for example, from first planning the retail store surface.

[0051]

[0071] The concept of a shelf (e.g., a vertical division unit). From an application perspective, a shelf (or module or shelving unit) is a descriptive unit between a store and its products. Products are placed on shelves, and every shelf in a store has a unique identifier. Therefore, it may be useful to record the physical boundaries of the shelf in a digital representation. When shelves are mapped one after the other, in the 2D case, the capture of shelf identifiers can be done by manual entry of identifiers or manual capture of dedicated barcodes (or different code types) placed on the shelves. Automatic entry may be preferred, but manual entry can be used if the shelves do not have identifiers. In some embodiments, the shelf identity can be inferred from the camera / device position. For example, in some embodiments, the phone's position can be determined using WiFi fingerprinting, Bluetooth beacons, GPS, SLAM (simultaneous localization and mapping), or IMU tracking. In a workflow, this is done, for example, before capturing the SKU map for that shelf. If the store is directly reconstructed in 3D, shelf segmentation (vertical division) can be performed automatically by image analysis, which searches for vertical division lines within a certain (parameterizable) distance from each other.

[0052]

[0072] The application used to scan shelves can be configured in a specific way, such as scanning each SKU sequentially. For example, a store partner can collect SKU data in an S-shaped pattern (e.g., from top left to bottom right), alternating direction for each shelf to minimize travel time. Data (e.g., for each shelf) is collected and sent to a backend. Data for each shelf can include: Store ID: store identifier; Shelf ID: shelf identifier; Time: acquisition start & end timestamps; and for each SKU: EAN / barcode, location (shelf row, product location), acquisition timestamp, price (read price, correct price), and / or product image; and / or smartphone identifier: to distinguish uploads from different phones.

[0053]

[0073] The data for a particular product is arranged in a vector format (e.g., an nx1 matrix). Table I below provides a sample vector for data mapping. The vector includes the code and the code's relative position on the shelf diagram (e.g., the vector is the "Value" column). [Table 1]

[0054]

[0074] 9 illustrates a flowchart of one embodiment of a process 900 for mapping objects on a shelving unit (e.g., creating a digital map). Process 900 begins in step 904 with receiving a plurality of images. For example, the images are captured by a camera from a mobile device. In step 608, a first item code is identified in the plurality of images. For example, the first item code is a barcode identifying a first SKU, and identifying the first item code includes decoding the first item code. In step 612, a second item code is identified in the plurality of images. For example, the second item code is a barcode identifying a second SKU, and identifying the second item code includes decoding the second item code.

[0055]

[0075] In step 916, the relative distance and direction between the first and second codes is calculated. The relative distance and direction can be based on placing the first and second item codes on a relative coordinate system (e.g., the coordinate system shown in FIG. 8; e.g., the second barcode is 4 units to the right and 3 units below the first barcode). In some embodiments, absolute distance is measured (e.g., the x position of the second barcode is equal to the x position of the first barcode plus 24 centimeters, and the y position of the second barcode is equal to the y position of the first barcode).

[0056]

[0076] In step 920, the relative distance and direction between the first and second item codes is calibrated to the shelf diagram. For example, multiple codes are combined with respect to the shelf diagram shown in Figure 8. The relative distance and direction between the first and second item codes is used to specify the positions of the first and second item codes on the shelf diagram (e.g., if the first item code is selected as the origin of the shelving unit, the first item code will have coordinates of 0,0 and the second item code will have coordinates of 4,-3).

[0057]

[0077] In step 924, a first vector is generated that includes the first item code and its relative location with respect to the shelf plan. For example, Table I provides a vector with the barcode ID and x / y coordinates of the barcode for shelf 21A7. Similarly, in step 928, a second vector is generated that includes the second item code and its relative location with respect to the shelf plan.

[0058]

[0078] In some embodiments, a system for mapping the location of an object on a display includes a mobile data collector that reads an optical code, a label having an optical code that is positioned with the object on the display, and / or one or more processors that receive identification information of two or more labels, calculate the relative distance and direction between the two or more labels, and / or place the locations of the two or more labels on a map.

[0059]

[0079] Shelf Visualization

[0080] Shelf visualization can be used to enable users (store employees, managers, etc.) to visualize the current and / or previous state of retail store shelves and / or display additional relevant information (e.g., via an overlay or additional window in a graphical user interface (GUI)). Some embodiments are limited to flat shelves, while other embodiments include almost any type of point-of-sale (PoS) product in a store. Shelf visualization can enable remote visual inspection of shelves, head to shopping experiences, street view-style strolls through a retail store from the couch, and / or augmented or virtual reality applications.

[0060]

[0081] FIG. 10 is an example of blending two images of a shelf. FIG. 10 shows a composite image 1000 of products 1008 on a shelf 1012. The composite image 1000 may appear somewhat blurry because the composite image 1000 may be formed by two or more images. Shelf visualization can be an image stitching problem. Although panoramic photographs are created using image stitching, the constraints of shelf visualization differ from traditional panoramic image stitching. For example, shelves (e.g., shelving units) may be close but do not fit into a single camera view. The shelf is captured from multiple different perspectives, and there may be significant camera shift (relative to the object) between the captured images (e.g., parallax). However, it can be assumed that there is no motion in the scene between the captured views. Visualization updates can come from multiple capture devices and multiple moments in time, even from a single SKU capture instance. Image blending often includes exposure compensation to make the visualization look appealing. As the visualization is updated, it is possible to sift through shelf images in time.

[0061]

[0082] In some configurations, analog-image stitching involves detecting keypoints and extracting invariant descriptors in the images, matching the descriptors between the two images, using RANSAC (as an example of some similar techniques) to estimate a homography matrix using matched image feature locations in both images, and combining the two images only if there are many matched features (e.g., using a threshold), applying a warping transformation using the resulting homography matrix, and blending the images (e.g., resizing, compensating for exposure, finding seams).

[0062]

[0083] In some configurations, the problem is formulated as a joint optimization problem (e.g., rather than a pairwise image matching problem). This can be an SfM problem. This can be based not only on points but also on other feature types such as lines (e.g., parts of shelving units). In some embodiments, the homography can be constrained to improve fault tolerance, for example, by assuming a near-frontal parallel view. In some embodiments, this problem can be addressed by formulating a global optimization problem. Considering the relationship between multiple images, jointly attempting to optimize / derive a rough estimate of the position at which the images were taken (derived visually and by correlating timestamps), can reduce errors. In some embodiments, high-quality image recording can be supported by low-level image descriptors or even planar estimates of the performed motion taken at much higher frequencies (e.g., estimated from gyros or through images; related to tracking), thereby allowing for better image correlation and optimizing the “stitched” panoramic image. In some embodiments, a sequence of images (or videos) is recorded to reduce parallax between captured frames and allow rendering with minimal visual artifacts. To improve stability, sensors (IMU / accelerometers) of the capturing mobile device can be used (and / or sensor fusion with cue-based images).

[0063]

[0084] While capturing SKUs, the user can also capture close-up images of SKUs where the barcode is recognizable. These close-up images can be incorporated into the overhead image to allow for enlargement and / or display of high-resolution images of the product and / or barcode. The stitching algorithm takes into account large parallax and scale changes. Image bending is performed carefully to minimize visual artifacts. In some embodiments, close-up images are available for all SKUs, and overview images are used to reference the close-up images relative to each other and, if necessary, fill in content between the close-up images. In some embodiments, the style of the close-up images is matched to the overhead image to minimize visual artifacts. This can include adapting contrast, brightness, saturation values, or composite exposure compensation (or similar techniques). In some embodiments, blending is applied for a smooth transition between the overhead image and the new close-up image, and content from both images is overlaid. In some embodiments, a "seam carving" algorithm is used for optimal seam location detection.

[0064]

[0085] 11 is an embodiment of a shelf diagram 1104 with a visual image 1108 of the shelf. Product information can be integrated into the visualization. For example, the visualization can be searchable by product, and searched products can be highlighted in the image visualization; the visualization can include overlays with product information such as price, stock levels, allergens, etc.; and specific products can be highlighted in the image (e.g., products on promotion, products that should soon be changed to promotion, or products with incorrect prices or incorrect locations in the parallelogram).

[0065]

[0086] FIG. 12 illustrates a flowchart of one embodiment of a process 1200 for creating a visual representation of an object on a display from multiple images. Process 1200 begins in step 1204 by acquiring multiple images, the multiple images including a first image and a second image. In step 1208, a code is detected in the first image. In step 1212, a code is detected in the second image. The code is decoded to receive an identification of the object in the first image (step 1216). A set of features is identified in the first image (step 1220). In step 1224, a set of features is identified in the second image. In step 1228, a position of the first image relative to a position of the second image is calculated based on comparing the position of the set of features in the first image to the position of the set of features in the second image. In step 1232, the first image is blended with the second image based on calculating the first image relative to the second image to create a blended image. The blended image is stored in step 1236 along with associated data identifying the object. In some embodiments, the metadata for the combined image includes a link or reference to a database that links the combined image with information that identifies the object.

[0066]

[0087] In some embodiments, the system comprises a mobile data collector that reads the optical code and acquires an image of an object on a display, a label having the optical code juxtaposed with the object on the display, and / or one or more processors that receive identification information of at least one object in one or more images, calculate the position of the two or more images relative to each other, and / or calculate a panoramic image representation from the one or more images.

[0067]

[0088] Simultaneous use of barcode decoding, OCR, and / or visual shelving

[0089] In some embodiments, the combined tools may enable an associated store to perform several tasks simultaneously. For example, a barcode can be scanned, a price label can be decoded using optical character recognition (OCR), a price can be associated with the product identified from scanning the barcode (e.g., by referencing a database), and the price associated with the product can be compared to the price label to verify a match. While price verification is being performed, images can be used to look for out-of-stock situations. Thus, work tasks can be performed in a single walk-through by an associated person or robot.

[0068]

[0090] Some decoding tasks, such as using OCR, can be computationally intensive. By decoding the code once and then tracing the code without decoding it, computational resources can be saved.

[0069]

[0091] A. Example of a user in a retail environment

[0092] i. Consultant - Retail

[0093] Susan is a sales associate who is approximately 30 years old. The store is a mid-sized grocery chain. Susan does not have a college degree. Although Susan knows how to use her mobile device, she would not be considered tech-savvy. Susan has changed retail employers three times in the past two years. She works different shifts depending on the week. As part of her job, her boss asks her to periodically walk through the store and perform specific data collection tasks, such as verifying price labels, recording shelf gaps, or verifying planogram compliance. Susan has access to a Zebra TC52 device. Susan's greatest concern is performing a store walkthrough as quickly and accurately as possible.

[0070]

[0094] Susan would like to run a session on her phone to quickly scan 1000 SKUs in seconds to collect and / or verify price label information (PLI). This is a daily challenge for her, and speed and accuracy are important. As she scans the SKUs for price label verification, she would also like to create a digital map of the SKUs on the shelf. Susan would prefer to run the session on her phone to collect data and then upload that data to a server in the cloud.

[0071]

[0095] In this example, Susan can scan the SKUs, view the individual date codes, and enter the code that expires earliest into the application. In other configurations, the individual date codes are decoded using an image of the individual date codes. In another variation, Susan enters all date codes she finds into the app and also specifies the quantity of product associated with each date code. Susan can also identify a gap on a shelf while working on another task. She can walk up to the gap and scan the SKU associated with the gap. In the user interface, she can optionally specify the number of products remaining on the shelf.

[0072]

[0096] ii. Paul Store Manager

[0097] Paul is about 45 years old, married, and has a couple of children. He typically works from 8:00 AM to 6:00 PM as a store manager for a large grocery chain. Paul has some college education and is more of a desktop user than a mobile device user. He spends most of his time at his desk in the management office behind his large computer screen. Paul has worked for the chain for almost 20 years. He prefers to do things the way they have always been done. Paul maintains a schedule for daily store walkthroughs and directly supervises store employees. Paul uses his iPhone® 8 at work.

[0073]

[0098] The web-based application allows Paul to create task lists listing compliance and operational issues, e.g., labels to be reprinted and replaced, gaps filled, etc., confirm on the portal that the issues have been resolved, review reports on in-store operational tasks / data, time frames, store associations for performing the tasks, type of event (PL non-compliance, gap scan, etc.), inspect summary statistics on the above, show enterprise trends over specified time frames / timelines, and / or zoom into ShelfView to review the latest visual data on the shelves.

[0074]

[0099] iii. Camilla – Nationwide store operations

[0100] Camilla is about 50 years old, married, and has two grown children. She typically works from 9 AM to 7 PM and is responsible for store operations for 200 stores. She holds an MBA degree. Camilla moves fluently between her mobile, iPad®, and Windows laptop. She is accustomed to monitoring operations from anywhere, anytime. Camilla has only been in her current role for two months, but she has already encountered resistance from the IT department when she tries to push out new software tools. Her biggest problem is the lack of real-time visibility into tasks taking place in the stores, and therefore, she is unsure whether the company is compliant with relevant regulations.

[0075]

[0101] In the web-based application, Camilla can review reports on operational tasks / data within the store, filter reports to a subset of stores, time frame, store association performing the task, type of event (PL non-compliance, gabb scan, etc.), inspect summary statistics on the above, show enterprise trends over a specified time frame / timeline, and / or zoom into ShelfView to review recent visual data for the shelves.

[0076]

[0102] B. Typical System Components

[0103] The web-based application can be configurable for multi-tenancy, where multiple retailers are hosted on the same system; multiple user roles with different access levels, such as store associate, store management, and enterprise management; multiple data elements that can include one or more of the following: date and time, session identifiers (user, date, store), decoded optical code (Uniform Product Code (UPC), International Article Number (EAN)), shelf / location identifier placeholder, high-resolution, raw image of detected barcode or other area to be decoded (OCR), full-field low-resolution image, and / or other fields (e.g., as described in the following applications); and individual data fields can be eligible for individual deletion schedules (depending on storage needs and privacy concerns), and customers can have the ability to schedule data deletion on numerous schedules, such as a) daily, b) after 48 hours, c) weekly, and / or deletion can be scheduled for a date and specified period after data collection.

[0077]

[0104] The web-based application can be configured to provide data reports with filtered data downloads in specific formats, such as CSV, XLS, or PDF, and the filtered data reports can be made available via a web API.

[0078]

[0105] C. Exemplary Applications and Workflows

[0106] The application can have a single SKU mode. The single SKU mode can be configured for walk-up gap scanning / PLV SKU scanning. In this mode, the store associate walks up to the gap on the shelf and scans the SKU code. The user interface guides the operator to ensure the product's display location is within the camera's field of view. The system uploads high-resolution images of the label and JPG images of the surrounding area. The user has the option to enter the following data into a manual interface: quantity remaining in stock, earliest date code, and / or general text input (free comments). In the backend, the data is processed as follows: if incorrect, a PLV is performed, added to the PL event list, added to the gap event list, and if available, other information (stock level, date code) is logged, and / or visual information of the shelf location is included in the shelf panoramic image.

[0079]

[0107] The single SKU mode can be configured for "walk-up data entry" SKU scanning. In this mode, store associates can walk up to the price tag or SKU and scan the code. The user interface can guide the associated store so that the product's display location is within the camera's field of view. The system uploads a high-resolution image of the label and a JPG image of the surrounding area. The user has the option to enter the following data into the manual interface: out of stock, remaining stock (including out of stock and "low stock"), earliest date code, and / or general text entry (free comment). In the backend, the data is processed as follows: if incorrect, a PLV is performed and added to the PL event list; if low / 0 stock, added to the gap event list, and / or record other information (stock level, date code) if available.

[0080]

[0108] The single SKU mode can be configured for a batch mode, such as systematic shelf scanning. In this mode, the store moves from left to right and top to bottom to systematically scan all products on a designated shelf. The user interface can be configured to guide the associated store so that the product's display location is within the camera's field of view. The system uploads a high-resolution image of the label and a JPG image of the surrounding area for each scan. The user (e.g., a store associate) has the option to enter the following data into the manual interface: inventory remaining, including "out of stock," earliest date code, and / or general text input (free comments). Additionally, the user has the option to scan or enter a shelf / module identifier (top left corner), erase the scan to correct an error, and / or resume data collection for the same shelf / module by starting the scan from any SKU (data following that SKU will be overwritten). While left-to-right scanning is illustrated, other scanning patterns (e.g., right-to-left, bottom-to-top-right, etc.) can also be used. In some embodiments, the scanning pattern is determined by the device orientation. For example, a phone held vertically can be used to scan downwards so that more SKUs are in the camera's field of view, while a phone held horizontally can scan right to left so that there is a higher chance of more than one SKU in the field of view. In the backend, the data is processed as follows: if incorrect, a PLV is performed and added to a PL event list, if low or out of stock, added to a gap event list, or other information (stock levels, date codes) and / or shelf position visual information, if available, is used to reconstruct a visual state / shelf panoramic image.

[0081]

[0109] D. Exemplary Performance Metrics

[0110] In some configurations, the application has the following performance metrics and / or tolerances: Speed: A moderately skilled operator spends 500ms or less per hour to complete the scan of a single code. SKU Mapping: The distance between SKUs is within + / - 10% of the true distance for 99% of the SKUs measured. SKU Mapping and Visualization: There are no obvious visual artifacts from the stitching process.

[0082]

[0111] E. Image Upload and Processing Examples

[0112] In some configurations, a high-resolution raw image of the detected code (e.g., barcode, price tag, etc.) is uploaded, and / or a JPEG-quality image of the entire image frame showing the label, the display area with the inventory, and / or the adjacent label. The barcode viewfinder is positioned to provide reasonable assurance that the inventory location is within the camera's field of view.

[0083]

[0113] For on-device PLV, a single SKU data capture PLV can be run directly on the device, and the results are shown to the user in an augmented reality (AR) overlay. In some embodiments, the results are presented to the user in no more than 500 ms.

[0084]

[0114] In some configurations of back-end PLV processing, false positives are below 10% (10% of errors detected). Incorrect price labels from a session / day can be accessible along with the image data for quality review by a human operator. The data can be presented to an online operator so that the operator can quickly review, image by image, and flag / correct any incorrectly decoded labels. After quality control, the data can be added to the PLV task list.

[0085]

[0115] In some configurations of SKU mapping and visualization, a map is constructed by maintaining the order of SKUs as the user scans one SKU after another. Data collection can include distances between SKUs while constructing the map. The map can include vertical distances between shelves.

[0086]

[0116] In some configurations using manual status entry, a user can count inventory levels and enter the information via a user interface (e.g., during a batch scan or individual SKU scan). The interface on the mobile device may be configurable by the enterprise user (e.g., to show only data entry options to be performed by store association). For example, the inventory interface may have a number input field for store association to enter certain products. The out-of-stock interface may include an out-of-stock button, a low-stock button, a refill button, and / or specific refill / reorder (number input field). The date code interface may include an oldest date code (date input field), multiple date codes, and / or an inventory number (e.g., a list of numbers and / or dates).

[0087]

[0117] F. Exemplary Portal Application

[0118] In some configurations of SKU, task, event, and / or date code reports, a user can request a report for all scan events within a specific time range, including scans that did not result in violations; shelf images within a specific time range; and / or specific SKUs that show violations within a specific time range. A user can request a report for SKUs and / or events filtered by one or more of the following criteria: tasks collected in a specific calendar range; tasks collected in a specific data collection session; tasks collected by a specific user; tasks within a specific product category; specific types of tasks (e.g., date code, out of stock, price label correction); resolved violations confirmed by user input (e.g., date code violation, out of stock, price label correction); and / or resolved violations confirmed by shelf scan (e.g., date code violation, out of stock, price label correction). A user can request a report on SKUs with date codes (e.g., a date code report) filtered by one or more of the following criteria: date codes within a specific calendar range; date codes within a specific product category; date codes collected in a time window; date code violations (date codes passed); and / or date code violations that have been resolved.

[0088]

[0119] In some configurations of shelf visualization, a backend system can stitch a representation of the shelf (shelf panorama) from individual SKU images (batch mode / SKU mapping). The visualization can be "clickable." For example, a user can click on an image or section of a SKU, and a metadata pop-up includes the date and time the image was taken, any user input from the time the image was taken, and / or status information. In some embodiments, the visualization is "zoomable," allowing a user to zoom in on a particular shelf location or SKU (e.g., enlarge the view of a particular shelf location or SKU). The visualization can be searchable, allowing a user to enter a SKU number or product description and "fly" to that product's location / image. As new scans / images of the SKU become available, the new scans / images replace the old images in the panorama.

[0089]

[0120] In some configurations, the dashboard or portal may include summary statistics and / or a timeline, which may include average time between scans (average across all SKUs), PLV violations by day / week, out-of-stock events by day of the week, total scans by day / week, and / or average time between scans (average across all SKUs).

[0090]

[0121] In some configurations of the backend, the database can store data from mobile devices (e.g., the data described above, including historical data) and can future-proof for new structured data (e.g., inventory counts). In some embodiments, the backend can be used to access a product database, such as a public database, a web database, a retailer's database, or a manufacturer's database. Metadata can be added to every shelf starting with the shelf category (e.g., beverages, cereal, etc.). Filtering / querying can also be permitted by parameters (e.g., shelf ID, store ID, user ID, etc.). In some embodiments, a backend process can be used to determine out-of-stock situations from collected images. For security and authentication purposes, the application (e.g., API and / or database) can be multi-tenant (3-10 pilot customers), and data uploaded by one customer can only be accessed / queried by that single customer. A unique configuration ID can be used to distinguish uploads. Also, in some embodiments, confidential company information (e.g., price, product location, etc.) is not stored in the cloud database (e.g., to ensure maximum privacy and data security).

[0091]

[0122] Some configurations of APIs or other interfaces include internal APIs for transmitting data to and from mobile devices and / or the cloud. For example, in-store data collected by a mobile device can be uploaded to the cloud and / or a product database and downloaded from the cloud by the mobile device, with the possibility of selecting only specific columns (e.g., price and SKU columns). External APIs for customers to query the backend database can include an API endpoint for querying the scan of a shelf in a specific store (e.g., this endpoint returns only the most recent scan and not historical data; / {store} / {shelf}), an API endpoint for updating the product database, a backend database to export in-store data to CSV (comma-separated values) format, a secure API, allowing customers to export in CSV format, and / or the possibility to query by historical data (e.g., by time period).

[0092]

[0123] The systems and / or methods disclosed herein can be used to establish and / or maintain a visual catalog, SKU-specific and SKU-independent out-of-stock detection, price label verification (PLV), simultaneous mapping and PLV, and / or OCR, and / or barcode decoding in a web browser. Images can be captured in several ways, including simultaneous image capture by mobile cameras on the shelf, drones, robots, fixed shelf cameras, and mobile barcode scanning.

[0093]

[0124] Some disclosed embodiments relate to methods and / or systems for operating an information system that aggregates pricing information from retail stores. More specifically, without limitation, some embodiments relate to acquiring imaging data of objects using imaging tools, acquiring pricing information from signage using imaging tools, acquiring product inventory information using imaging tools, acquiring information from receipts using imaging tools, using predictive algorithms to reconstruct pricing information, incorporating available pricing information from publicly available sources, incorporating pricing information from issuing recommendations regarding where to purchase products, presenting the information to one or more third parties, issuing recommendations regarding which products to purchase based on available choices, operating a retail store, and adjusting prices based on information regarding competitive prices. The system or method may include scanning products to image price labels on retail displays or shelves, using drones to scan products on shelves, using robots to scan products on shelves, scanning products and / or prices on displays using consumer devices, scanning and / or parsing shopping receipts to record pricing of purchased products, parsing digital receipts to record pricing of purchased products, and / or scanning and interpreting signage to infer pricing and special offers.

[0094]

[0125] In some embodiments, a system for detecting a state of a retail display includes a mobile data collector enabled with web browser software, a camera module, a label having an optical code disposed with an object on the display, and / or one or more processors that receive the identification of at least one object in one or more images captured by the camera module, receive an on-display price of the at least one object in the one or more images captured by the camera module, receive an intended price of the at least one object from a database, and / or compare the on-display price to the intended price and report the results. The system can include a mobile data collector enabled with web browser software, a camera module, a label having an optical code disposed with an object on the display, and / or one or more processors that receive an identification of the at least one object in one or more images captured by the camera module, decode information about the on-display object from the one or more images captured by the camera module, and / or calculate a map of the at least one object on the display from the one or more images captured by the camera module. The system may include a mobile data collector, a camera module, a label having an optical code disposed with an object on a display, and / or one or more processors that receive identification information of at least one object in one or more images captured by the camera module, receive image data regarding the display area from one or more images captured by the camera module, and / or detect the presence of at least one object in the display area from one or more images captured by the camera module.The system may include a mobile data collector, a camera module, a label having an optical code disposed with an object on a display, and / or one or more processors that receive identification information of at least one object in one or more images captured by the camera module, receive image data of the display area from the one or more images captured by the camera module, detect the presence of at least one object in the display area from the one or more images captured by the camera module, and / or store image data of the one or more objects along with identification information of the one or more objects in a database.

[0095]

[0126] Ultra-high speed scanning

[0127] Mobile devices (e.g., smartphones; tablets) can be used for ultra-fast scanning of optical codes. In some configurations, the mobile device uses a wide-angle lens (e.g., fixed focus—for close depth of field and better depth of field with no AF delay), high frames per second (e.g., for speed and / or redundancy), very short exposure times (e.g., to avoid motion blur), and a torch (e.g., for additional illumination and / or aiming).

[0096]

[0128] Figure 13 is an embodiment of a mobile device 1300 that rapidly scans an optical pattern 1304. The optical pattern 1304 can be the code 104 of Figure 1. In some embodiments, ultra-fast scanning or fast scanning refers to acquiring an image, detecting an optical code, and decoding the optical code in less than 0.25 seconds and / or more than 0.002 seconds.

[0097]

[0129] The mobile device 1300 includes a camera, a display, and one or more processors. The camera includes an image sensor and a lens. The lens is configured to focus light onto the image sensor. The one or more processors are configured to acquire a plurality of images, analyze the plurality of images to detect an optical pattern in the plurality of images, decode the optical pattern to obtain a data payload, and store the data payload in a memory device of the mobile device.

[0098]

[0130] A normal lens is a lens with a focal length approximately equal to the diagonal measurement of the image sensor. A normal lens has an angle of view of approximately 58° (e.g., measured diagonally), similar to the angle of view of the human eye. A wide-angle lens has a larger angle of view than a normal lens. The lens of the camera of the mobile device 1300 is a wide-angle lens. In some embodiments, the lens is an ultra-wide-angle lens (e.g., having an angle of view of 100, 120 degrees or more, or 125 degrees or more, and / or 128, 130, 150 degrees or less, or 180 degrees or less). For example, the lens has an angle of view of 128 degrees. The greater the depth of field of the camera, the greater the distance from the mobile device 1300 to the optical pattern 1304 can vary with the set focus. In some configurations, a camera with a 13 mm focal length (120 degree angle of view), a 2.4 aperture, and a sensor size of 1 / 3.6 inches is used.

[0099]

[0131] The camera lens is set to a fixed focus during the acquisition of multiple images. A lens set to a fixed focus may be a single focus lens, or the lens may be a variable focus lens with the autofocus function disabled (e.g., the focus is set to a specific predetermined value between 0 and 1). A single focus lens does not have variable focus. The camera lens is set to a fixed focus during image acquisition so that the application for decoding the optical pattern does not wait for the focus algorithm to run. Autofocus can cause scanning delays. However, some fixed focus settings have a small depth of field. Using a wider angle lens or a wider zoom setting can increase the depth of field.

[0100]

[0132] The exposure time for each of the multiple images can be 1 / 500 of a second or less. The exposure time is short to reduce motion blur. The ISO (sensitivity of the camera's sensor; sometimes called film speed or gain) can be set manually (e.g., auto-exposure is turned off) or controlled (e.g., adjusted) by the camera. For certain camera models, the inventors have found that a 1 / 750 second exposure is particularly useful for providing good exposure of the optical pattern 1304 under various conditions while using a light on a mobile device (e.g., a torch or flashlight). Automatic metering often uses an 18% gray standard. Metering can be stopped at one, two, three, or more stops to detect and decode the optical pattern 1304. The camera can automatically adjust the ISO depending on the amount of light available. In some embodiments, the exposure is 1 / 500 or greater and / or less than the camera's maximum shutter speed (e.g., 1 / 2000 or 1 / 8000). The choice of exposure can be important, as some phone models do not like to expose the noisiness of the image and therefore limit gain adjustment. By selectively choosing the exposure (e.g., 1 / 750 seconds) and / or using a torch, multiple images can be properly exposed across a variety of working conditions. In some embodiments, the torch is activated during the acquisition of multiple images, regardless of the camera's exposure measurement.

[0101]

[0133] The multiple images are captured at the frame rate of the camera. The frame rate is greater than 30 frames per second (fps). In some embodiments, the frame rate is equal to or greater than 30, 60, or 90 fps, and / or equal to or less than 120, 150, 180, or 240 fps. For example, the frame rate is set to 120 fps. At a high frame rate, many images of the optical code are captured. If the optical code is unreadable in one frame (e.g., due to glare caused by a torch on the plastic cover of the optical pattern 1304), there are other frames with the optical code that can be used to decode the optical code (i.e., having more image frames containing the optical pattern 1304 provides redundancy).

[0102]

[0134] Capturing multiple images at a high frame rate and processing the multiple images can be computationally intensive. To reduce computational intensity, a limited area of each image is scanned (e.g., processed) rather than the entire image. The limited area can be used based on the use case. For example, the limited area is located in the center of the camera frame, has a height equal to one-third of the height of the camera image frame, and a width equal to the width of the camera image frame. In some embodiments, the limited area is located in the center of the frame, has a height equal to or less than 60%, 50%, 40%, or 33%, has a height equal to or greater than 25% or 33% of the height of the image frame, and / or has a width equal to or greater than 50%, 66%, 75%, or 100% of the width of the image frame. Limiting the scan to a limited area in the center of the image can be beneficial for images captured with an ultra-wide-angle lens, as images captured with a wide-angle lens have greater distortion at the edges. In some embodiments, matrix scanning is used to track the code instead of attempting to decode the code within each frame. Optical code tracking can be used to reduce computation (eg, instead of trying to decode the optical code in each frame, track the code).

[0103]

[0135] In some embodiments, the mobile device 1300 comprises a torch. A torch is sometimes referred to as a flashlight. The torch is configured to emit light 1320. The torch may comprise a light emitting diode. A torch lens may be used to direct, diffuse, and / or shape the light beam emitted from the light emitting diode.

[0104]

[0136] Rapid scanning can be used to detect and decode many optical patterns 1304 in parallel (e.g., two or more optical patterns 1304 in one image frame) and / or sequentially (e.g., a user moving a mobile device 1300 along a shelf in a grocery store (e.g., as shown in FIG. 13 )). FIG. 13 shows a first optical pattern 1304-1 and a second optical pattern 1304-2. As the user moves the mobile device 1300 along the shelf, the camera of the mobile device 1300 captures multiple images. The multiple images are analyzed to detect and decode the first optical pattern 1304-1 to obtain a first data payload and the second optical pattern 1304-2 to obtain a second data payload. The first and second data payloads are stored in memory of the mobile device 1300 and / or transmitted to a remote hardware server (e.g., wirelessly and / or via the Internet).

[0105]

[0137] Capturing many images, processing many images, and using a torch can use batteries and be computationally intensive. Thus, in some embodiments, the torch is activated and / or multiple images are processed while receiving an indication of a user action. For example, while a user touches the display of a mobile device with their finger (e.g., the display is a touchscreen), the application enters a fast scan mode. In fast scan mode, the torch is turned on, the camera begins acquiring images, and the camera's frame rate is increased (e.g., to 120 fps) and / or the shutter speed is increased (e.g., to 1 / 750 sec). After the user removes their finger from the display, the application exits fast scan mode. After exiting fast scan mode, the torch is turned off, the frame rate is reduced, and / or the camera is turned off, and / or the shutter speed returns to normal (e.g., the camera returns to autoexposure with automatic shutter speed control). The torch value remains constant during the acquisition of multiple images (e.g., the torch does not flash as it would if the camera were attempting to take a picture in a dark environment). Reducing the amount of time an application is in fast scan mode reduces battery usage and / or computing resources.

[0106]

[0138] There are additional and / or other ways that an application can enter and / or exit fast scan mode. For example, a first double tap on the display triggers the application to enter fast scan mode, a single tap is used to select an optical code, and / or a second double tap is used to exit fast scan mode. A hardware button can be used to enter fast scan mode. For example, the hardware button may be on the mobile device (e.g., a volume button, a hidden button on the back of the phone, a home button, etc.) and / or integrated into the mobile device case. In some embodiments, a software button is used. For example, a rectangular icon with the word "SCAN" is displayed on the mobile device screen. As long as the user touches the screen where the rectangular icon is displayed, the application enters fast scan mode. In some embodiments, data from an inertial measurement unit (IMU) is used to enter and / or exit fast scan mode. For example, sensor data from the IMU corresponding to a shake or finger tap can be used by the application as an instruction to enter or exit fast scan mode.

[0107]

[0139] 14 shows a chart 1400 illustrating scan range as a function of camera focus setting for scanning a first type of barcode. The first type of barcode is an EAN 13 barcode (e.g., the thinnest element is 6 mils or 6 thousandths of an inch). Chart 1400 shows the scan range (in millimeters) of an ultra-wide-angle camera compared to a variable focus lens (the "main lens") for different focus values. For example, a mobile device may include two or more outward-facing cameras (e.g., cameras pointing in a direction not coincident with or 180 degrees from the normal direction of the mobile device's screen).

[0108]

[0140] The ultra-wide-angle camera has a 128-degree field of view. Its scanning range is 27 to 355 mm. The focal value of the main lens is between 0 and 1 (in tenths of a second), with 0 (or 0%) being the closest plane and focal value 1 (or 100%) being the farthest plane. The ultra-wide-angle camera has a wider scanning range than the varifocal lens, but the varifocal lens's focal value is between 0 and 0.5. At focal values of 0.6 and 0.7, the varifocal lens has a larger scanning range than the ultra-wide-angle camera: 65 to 520 mm and 105 to 720 mm, respectively. However, the mobile device scans the optical code farther away from the ultra-wide-angle lens (e.g., 65 and 105 mm for varifocal cameras with focal values of 0.6 and 0.7, compared to only 27 mm for the ultra-wide-angle lens). As shown in Figure 15, while the difference may not seem significant for the first type of barcode, the difference can be more pronounced when scanning different types of barcodes.

[0109]

[0141] 15 shows a chart 1500 illustrating the scan range as a function of camera focus setting for scanning a second type of barcode. The second type of barcode is a Code 128 barcode (e.g., the width of the thinnest element is 0.4 mm). Chart 1500 shows the scan range of an ultra-wide camera compared to a variable focus lens (the "main lens") for different focus values.

[0110]

[0142] The ultra-wide-angle camera has a scanning range of 66 to 380 mm. The scanning range is 184 mm to 417 mm with a main lens having a focus value of 0.6, and 242 mm to 562 mm with a main lens having a focus value of 0.7. Therefore, scanning with an ultra-wide-angle camera allows the mobile device to be much closer to the optical code than the main lens while still providing a wide scanning range. In some embodiments, an external lens (e.g., external to the mobile device) is used to provide an additional wide-angle lens (e.g., a lens with a field of view of 100, 120 degrees or more, or 125 degrees or more). In some embodiments, the camera is configured to be 0.025, 0.05, 0.06, or 0.1 meters or more away from the optical code and / or 0.5, 0.4, 0.3, or 0.2 meters or less away from the optical code to decode the optical code.

[0111]

[0143] FIG. 16 shows a flowchart of one embodiment of a process 1600 for decoding an optical pattern using a digital camera. Process 1600 begins in step 1604 with capturing multiple images. The multiple images are captured using a camera with a fixed focus, wide-angle lens, fast exposure, high frame rate, and / or torch activation. The camera lens is set to a fixed focus during capture of the multiple images. For example, a single focal length lens or a variable focus lens set to a fixed value. The lens is a wide-angle lens (e.g., having an angle of view of 60 degrees or 75 degrees or more) or an ultra-wide-angle lens (e.g., having an angle of view of 90, 100 degrees or more, or 120 degrees or more). The exposure is fast. For example, the exposure time for each of the multiple images is 1 / 500, 1 / 750, or 1 / 1500 of a second or less. The camera has a high frame rate during capture of the multiple images. For example, the frame rate is 30, 60, 90, or 120 frames per second or more.

[0112]

[0144] In step 1608, the multiple images are analyzed to detect optical patterns within the multiple images. For example, the multiple images are analyzed for one, two, three, or more optical patterns. The optical patterns may be a one-dimensional computer-readable optical code (e.g., a barcode), a two-dimensional computer-readable optical code (e.g., a QR code), and / or one or more symbols (e.g., letters, numbers, special characters, and / or unique symbols).

[0113]

[0145] In step 1612, one or more optical patterns are decoded to obtain a data payload or data payloads. The data payload is data obtained by scanning the optical pattern. The data payload may include one or more letters, numbers, special characters, or other data. For example, three barcodes may be decoded to obtain three numeric strings, one for each barcode. The three barcodes may be in one image frame and / or each barcode may be in a separate image frame. The data payload is then stored in memory in step 1616. The data payload may also be transmitted to a remote server.

[0114]

[0146] In some embodiments, a torch (e.g., a flashlight) on the mobile device is activated while capturing multiple images. For example, the torch is turned on when an image is captured by the camera, regardless of whether the exposure reading indicates there is enough light to properly expose the image without the torch on. The torch can be used as a sight. For example, the torch shines a light beam. When the mobile device is brought close to an object containing a light code, the user can move the light beam over the light code. Images of the light code can be captured, and the light code is decoded. A high frame rate is required because the light from the torch can cause glare (e.g., on plastic covering the light code). Therefore, a high frame rate increases the probability of capturing frames before or after the light from the torch shines directly onto the light code and can be used to decode the light code.

[0115]

[0147] In some embodiments, optical codes are tracked while performing very fast scanning (e.g., using matrix scanning). Optical codes can be tracked to speed up decoding and / or make decoding more accurate. This can be useful for scanning optical codes with similar or identical information. For example, some barcode readers use a duplication filter or manual input to reduce the likelihood of a barcode being decoded multiple times (e.g., more than once). The duplication filter can use a timer to prevent an optical code from being read multiple times. For example, the duplication filter's timer can be set to between 0.5 and 5 seconds (e.g., equal to 1, 2.1, or 3.5 seconds), depending on the application. In one example, when an optical code is decoded, the mobile device beeps and the screen momentarily turns green, indicating to the user that the optical code has been decoded. The duplication filter then sets a timer (e.g., 1 second) after decoding the optical code, during which the mobile device does not decode the same optical code again (in some embodiments, the optical scanner detects and / or decodes other optical codes during the duplication filter's time). Having overlapping filters helps prevent a mobile device from decoding the same optical code multiple times.

[0116]

[0148] Although duplication filters can help prevent optical codes from being decoded multiple times, they have several drawbacks. First, they cannot decode identical optical codes faster than the timer allows. Second, the duplication filter's timer can be difficult to set (type) for one or more applications and / or users (e.g., a time set on the timer that works well for one application or user may not work well for another application and / or user). Third, multiple barcodes may be present in a single image, and multiple barcodes may have the same information. For example, while performing an inventory on a stack of boxes, each box may contain a pair of shoes, each pair of shoes may be identical to each other, and each shoe box may have the same barcode attached to it. Second, using a one-second timer for a duplication filter not only slows down the scanning of inventory barcodes, but it may also result in incorrect counting of barcodes (e.g., undercounting due to the user moving too fast, or overcounting due to the user not moving fast enough). Some optical scanners track barcodes based on the assumption that each barcode is unique and cannot be used because the optical scanner will not count the number of barcodes correctly as the optical scanner will ignore duplicate barcodes.

[0117]

[0149] Some barcode scanners use manual input (e.g., software or hardware buttons, taps, etc.) to decode barcodes when there are multiple barcodes in the same image that are the same barcode. However, waiting to receive manual input before decoding a barcode can slow down decoding. Furthermore, using manual input can be tedious for the user. In some cases, there may be hundreds of optical codes to scan (e.g., in inventory for many products), which can involve hundreds of button presses by the user.

[0118]

[0150] In some configurations, optical code tracking (e.g., matrix scanning) is used instead of using overlapping filters (e.g., no timers between decoding optical codes, no timers between decoding identical codes), manual entry, and / or tracking of unique barcodes. After the optical code is decoded, an application tracks the optical code while it is within the field of view of the mobile device without decoding it again. In certain configurations, the application tracks the optical code without mapping the scene. For example, if the camera of the mobile device is moved so that the optical code is no longer within the field of view of the camera, and then the camera is moved back so that the optical code is within the field of view of the camera, the application decodes the optical code again.

[0119]

[0151] By using tracking, no overlap filters are used, the process can be automatic (e.g., no manual input), and / or the application can be used to decode multiple identical codes within an image or field of view. In some configurations, tracking the codes and / or not using overlap filters is performed without some or all of the features of ultra-fast scanning.

[0120]

[0152] Mobile device mounting with external lens

[0153] 17 is an embodiment of an attachment 1702 for a mobile device 1704. The attachment 1702 is an external attachment (e.g., the attachment 1702 is configured to be removably coupled to the mobile device 1704). The mobile device 1704 includes a camera. The camera includes a first lens 1708. The mobile device 1704 includes a torch 1712.

[0121]

[0154] The attachment 1702 includes a second lens 1716 and a third lens 1720. The second lens 1716 is configured to expand the angle of view of the camera of the mobile device 1704 while the attachment 1702 is coupled to the mobile device 1704. For example, the first lens 1708 may have a 60 degree angle of view, and the second lens 1716 increases the camera's angle of view to 90, 100, or 120 degrees.

[0122]

[0155] The third lens 1720 is configured to focus the light of the torch 1712 while the attachment 1702 is coupled to the mobile device 1704. The third lens 1720 is configured to focus the light of the torch 1712 in a direction normal to the torch (e.g., the third lens 1720 does not redirect the light beam from the torch to a different angle). The direction normal to the torch 1712 is parallel to the normal vector of the back surface 1724 of the mobile device 1704. In some embodiments, the attachment 1702 includes the third lens 1720 but does not include the second lens. In some configurations, the third lens 1720 is a Fresnel lens and / or is made of plastic.

[0123]

[0156] The torch 1712 can be used as a sight to scan the optical code. For example, the light from the torch 1712 indicates where the camera of the mobile device 1704 is pointed. By focusing the light from the torch 1712 (e.g., with a third lens 1720), the light from the torch 1712 is concentrated into a narrower spot.

[0124]

[0157] In some embodiments, the third lens 1720 is configured to focus the light of the torch 1712 into an oval shape (e.g., a rectangle or an ellipse, where an ellipse is non-circular). The mobile device 1704 is generally held by a user in a portrait orientation while scanning an optical code, such that the long side of the mobile device 1704 is substantially vertical, as shown as vertical direction 1728 in FIG. 17, and the short side of the mobile device is substantially horizontal, as shown as horizontal direction 1732 in FIG. The directional normal of the back surface 1724 is orthogonal to both the vertical direction 1728 and the horizontal direction 1732 of the mobile device. The elliptical shape is configured such that the major axis of the ellipse is generally parallel (e.g., plus or minus 1, 2, or 5 degrees) to the horizontal direction 1732 (e.g., in the direction of the shorter side of the mobile device 1704) and the minor axis is generally parallel (e.g., plus or minus 1, 2, or 5 degrees) to the vertical direction 1728 (e.g., in the direction of the longer side of the mobile device 1704). The light of the torch 1712 is configured to be shaped like an ellipse or rectangle (e.g., with the longer side of the rectangle substantially parallel to the horizontal direction 1732 and the shorter side of the rectangle substantially parallel to the vertical direction 1728) because optical codes are generally landscape-oriented and a user can hold the mobile device 1704 in a portrait orientation while scanning a barcode. By emitting light from the torch 1712 in a rectangular shape, it can be more intuitive for a user to use the torch as a pointer to scan an optical code. In some embodiments, the major axis of the ellipse is aligned with the vertical direction 1728. The mobile device 1704 can be held landscape or in other orientations. For example, if the mobile device 1704 is held in one hand and the code is scanned low to the ground, the hand can be turned to the side so that the code is oriented along the long axis of the screen. By using a torch as an aiming device, the mobile device can be held so that part of the hand blocks the screen instead of using the screen. This can be particularly useful for delivery drivers and transportation / logistics personnel who scan many items (e.g., in stacks).This can save them from the awkward maneuvering of scanning the mobile device 1704 in a vertical position (e.g., because they can look at the direction of the light from the torch instead of looking at the screen).

[0125]

[0158] The width of the beam from the torch 1712 on the surface containing the optical code can also indicate to the user how close the mobile device 1704 should be held to the surface containing the optical code. For example, if the beam traces a large ellipse on the surface with the optical code, the user should move the mobile device 1704 closer to the surface; if the beam traces a small ellipse on the surface, the user should move the mobile device 1704 further away from the surface so that the ellipse of light on the surface is approximately the same size as the optical code. In certain configurations, the beam is configured to be focused to a width (e.g., measured horizontally) that is more than half the width of the optical code and / or less than two or three times the width of the optical code while the code is within the camera's scanning range.

[0126]

[0159] In some configurations, the attachment 1702 is a case for the mobile device 1704. The case can include a second lens 1716 and / or a third lens 1720, and the case can cover at least a portion of two or more sides of the mobile device 1704 (e.g., for protection). For example, the case can cover 80% or more of the back surface 1724, as well as a portion of the long side(s) and / or short side(s) of the mobile device 1704.

[0127]

[0160] In some embodiments, attachment 1702 (e.g., illustratively) includes a magnet. The magnet (e.g.,) can be used to secure mobile device 1704 to a user's object (e.g., a metal cabinet) and / or a wearable device (e.g., a belt, armband, chest pack, leg band, backpack, fanny pack, etc.). Having a magnet allows mobile device 1704 to be positioned so that the user can use both hands for another task. For example, a delivery person wears an armband on their upper arm. Mobile device 1704 is secured (e.g., magnetically attached) to the armband. A woman delivers a package by carrying it with both hands to the doorway of a house and places it there. After placing the package on the doorstep, she removes mobile device 1704 from the armband and points the camera of mobile device 1704 at the label on the package. She touches the screen of the mobile device with her thumb. While touching the screen, torch 1712 lights up and the camera captures multiple images. The images are analyzed and the barcode on the package label is decoded. A green box is overlaid on the image presented on the screen to indicate to the user that the optical code has been decoded. The user then moves the mobile device back and an image of the doorstep with the package is saved for purchase verification. The user then removes their thumb from the screen. After the user removes their thumb, the torch turns off and the camera turns off or goes into standby mode. The user then places the mobile device on the armband and returns to their vehicle. The barcode and the image of the doorstep are sent from the mobile device 1704 to a remote server. Now that the barcode has been decoded, the image of the doorstep is automatically linked to the barcode and saved to a file identified by the barcode.

[0128]

[0161] 18 shows another embodiment of an attachment 1802 for a mobile device 1804. The attachment 1802 includes a lens 1816. The lens 1816 is an ultra-wide-angle lens (e.g., an angle of view of 120 degrees or more and / or 130, 150, 160 degrees or less, or 175 degrees). Some phone models do not have a wide-angle lens or a lens with too narrow an angle of view (not wide enough). The lens 1816 can be attached to increase the angle of view of the camera of the mobile device 1804.

[0129]

[0162] Sample Computing Device

[0163] FIG. 19 is a simplified block diagram of a computing device 1900. The computing device 1900 may implement some or all of the functions, behaviors, and / or capabilities described above using electronic storage or processing, as well as other functions, behaviors, or capabilities not explicitly described. The computing device 1900 includes a processing subsystem 1902, a storage subsystem 1904, a user interface 1906, and / or a communication interface 1908. The computing device 1900 may also include other components (not explicitly shown), such as a battery, a power controller, and other components operable to provide various enhanced functions. In various embodiments, the computing device 1900 may be implemented in a desktop or laptop computer, a mobile device (e.g., a tablet computer, a smartphone, a mobile phone), a wearable device, a media device, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, or any electronic unit designed to perform any of the functions or combinations of functions described above.

[0130]

[0164] The storage subsystem 1904 may be implemented using local and / or removable storage media, for example, using a disk, flash memory (e.g., a Secure Digital card, a Universal Serial Bus flash drive), or any other persistent storage medium or combination of media, and may include volatile and / or nonvolatile storage media. Local storage may include random access memory (RAM), including dynamic RAM (DRAM), static RAM (SRAM), or battery-backed RAM. In some embodiments, the storage subsystem 1904 may store one or more application and / or operating system programs to be executed by the processing subsystem 1902, including programs for performing some or all of the operations described above that would be performed using a computer. For example, the storage subsystem 1904 may store one or more code modules 1910 for performing one or more method steps described above.

[0131]

[0165] A firmware and / or software implementation may be implemented with modules (e.g., procedures, functions, etc.). Any machine-readable medium tangibly embodying instructions may be used in implementing the methodologies described herein. The code modules 1910 (e.g., instructions stored in a memory) may be implemented within the processor or external to the processor. As used herein, the term "memory" may refer to long-term, short-term, volatile, non-volatile, or other types of storage media and is not limited to a specific type or number of memories or the type of media on which the memory is stored.

[0132]

[0166] Additionally, the terms "storage medium" or "storage device" can refer to one or more memories for storing data, including read-only memory (ROM), RAM, magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, portable or non-removable storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing instructions and / or data.

[0133]

[0167] Furthermore, embodiments may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting languages, and / or microcode, the program code or code segments for performing tasks may be stored in a machine-readable medium such as a storage medium. A code segment (e.g., code module 1910) or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or a combination of instructions, data structures, and / or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted by any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0134]

[0168] The implementation of the techniques, blocks, steps, and means described above can be done in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. In the case of a hardware implementation, the processing unit may be implemented in one or more ASICs, DSPs, DSPDs, PLDs, FPGAs, processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions described above, and / or combinations thereof.

[0135]

[0169] Each code module 1910 may comprise a set of instructions (code) embodied on a computer-readable medium that instructs a processor of computing device 1900 to perform a corresponding operation. The instructions may be configured to be executed sequentially, in parallel (such as under different processing threads), or a combination thereof. After loading the code modules 1910 into a general-purpose computer system, the general-purpose computer is transformed into a special-purpose computer system.

[0136]

[0170] Computer programs incorporating various features described herein (e.g., in one or more code modules 1910) may be encoded and stored on various computer-readable storage media. Computer-readable media encoded with program code may be packaged with a compatible electronic device, or the program code may be provided separately from the electronic device (e.g., via Internet download or as a separately packaged computer-readable storage medium). Storage subsystem 1904 may also store information useful for establishing network connections using communications interface 1908.

[0137]

[0171] The user interface 1906 may include input devices (e.g., touchpad, touchscreen, scroll wheel, click wheel, dials, buttons, switches, keypad, microphone, etc.) and output devices (e.g., video screen, indicator lights, speakers, headphone jack, virtual or augmented reality display, etc.), along with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors, etc.). A user can manipulate the input devices of the user interface 1906 to invoke functions of the computing device 1900 and view and / or hear output from the computing device 1900 via the output devices of the user interface 1906. In some embodiments, the user interface 1906 may not be present (e.g., in the case of a process that uses an ASIC).

[0138]

[0172] The processing subsystem 1902 may be implemented as one or more processors (e.g., an integrated circuit, one or more single-core or multi-core microprocessors, a microcontroller, a central processing unit, a graphics processing unit, etc.). In operation, the processing subsystem 1902 may control the operation of the computing device 1900. In some embodiments, the processing subsystem 1902 may execute various programs in response to program code and may maintain multiple simultaneously executing programs or processes. At a given time, some or all of the program code to be executed may reside in the processing subsystem 1902 and / or in a storage medium such as the storage subsystem 1904. Through programming, the processing subsystem 1902 may provide various functions for the computing device 1900. The processing subsystem 1902 may also execute other programs to control other functions of the computing device 1900, including programs that may be stored in the storage subsystem 1904.

[0139]

[0173] Communications interface 1908 can provide voice and / or data communications capabilities to computing device 1900. In some embodiments, communications interface 1908 can include radio frequency (RF) transceiver components for accessing wireless data networks (e.g., Wi-Fi networks; 3G, 4G / LTE; etc.), mobile communications technologies, components for short-range wireless communications (e.g., using the Bluetooth communications standard, NFC, etc.), other components, or a combination of technologies. In some embodiments, communications interface 1908 can provide a wired connection (e.g., Universal Serial Bus, Ethernet, Universal Asynchronous Transmitter / Receiver, etc.) in addition to or instead of a wireless interface. Communications interface 1908 can be implemented using a combination of hardware (e.g., driver circuits, antennas, modulators / demodulators, encoders / decoders, and other analog and / or digital signal processing circuits) and software components. In some embodiments, communications interface 1908 can support multiple communications channels simultaneously. In some embodiments, communications interface 1908 is not used.

[0140]

[0174] It will be understood that computing device 1900 is illustrative and that variations and modifications are possible. The computing device may have various functions not specifically described (e.g., voice communication over a cellular network) and may include components appropriate for such functions.

[0141]

[0175] Furthermore, while computing device 1900 is described with reference to particular blocks, it should be understood that these blocks are defined for convenience of description and do not imply a particular physical arrangement of component parts. For example, processing subsystem 1902, storage subsystem, user interface 1906, and / or communication interface 1908 may be located within a single device or distributed across multiple devices.

[0142]

[0176] Furthermore, the blocks need not correspond to physically separate components. The blocks may be configured to perform various operations, for example, by programming a processor or by providing appropriate control circuitry, and various blocks may or may not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the present invention may be realized in a variety of apparatuses, including electronic devices implemented using a combination of circuitry and software. The electronic devices described herein may be implemented using computing device 1900.

[0143]

[0177] Various features described herein, e.g., methods, apparatus, computer-readable media, etc., can be implemented using a combination of dedicated components, programmable processors, and / or other programmable devices. The processes described herein can be performed on the same processor or different processors. Where components are described as being configured to perform certain operations, such configuration can be achieved, for example, by designing electronic circuitry to perform the operations, by programming a programmable electronic circuit (e.g., a microprocessor) to perform the operations, or a combination thereof. Furthermore, while the above-described embodiments may refer to specific hardware and software components, those skilled in the art will recognize that different combinations of hardware and / or software components can also be used, and that certain operations described as being implemented in hardware can be implemented in software, or vice versa.

[0144]

[0178] In the above description, specific details are set forth to provide an understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. In some instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0145]

[0179] While the principles of the present disclosure have been described above in connection with specific devices and methods, it should be understood that this description is made by way of example only and not as a limitation on the scope of the disclosure. The embodiments have been chosen and described in order to explain the principles and practical applications of the invention, and to enable those skilled in the art to utilize the invention in various embodiments and with various modifications, as may be suitable for the particular application contemplated. It is understood that the description is intended to cover modifications and equivalents.

[0146]

[0180] Also, it should be noted that the embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a series of processes, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0147]

[0181] The enumeration of "a," "an," or "the" is intended to mean "one or more" unless specifically indicated to the contrary. All patents, patent applications, publications, and descriptions referred to herein are incorporated by reference in their entirety for all purposes. None is admitted to be prior art.

[0148]

[0182] In some embodiments, a system for mapping objects onto a shelving unit using image analysis includes a camera having an image sensor configured to capture a plurality of image frames; and one or more processors configured to receive the plurality of images captured by the camera, identify a first item code within the plurality of images, identify a second item code within the plurality of images, calculate a distance and direction of the first item code relative to the second item code, calibrate the distance and direction to a shelf diagram, generate a first vector including the first item code and a relative position of the first item code relative to the shelf diagram, and / or generate a second vector including the second item code and a relative position of the second item code relative to the shelf diagram. The method may include receiving a plurality of images, identifying a first item code within the plurality of images, identifying a second item code within the plurality of images, calculating a distance and direction of the first item code relative to the second item code, calibrating the distance and direction relative to the shelf diagram, generating a first vector including the first item code and a relative position of the first item code relative to the shelf diagram, and / or generating a second vector including the second item code and a relative position of the second item code relative to the shelf diagram. In some embodiments, calculating the distance and direction of the first item code relative to the second item code is based on the first item code and the second item code being in a two-dimensional vertical plane; calculating the distance and direction of the first item code relative to the second item code is based on a width or height of the first item code; calculating the distance and direction of the first item code relative to the second item code is based on a width or height of the shelf label; calculating the distance and direction of the first item code relative to the second item code is based on a simultaneous position and mapping algorithm of the mobile device; the distance and direction of the first item code relative to the second item code is based on two image sensors of the mobile device used to acquire the multiple images and a known distance between the two image sensors; calculating the distance and direction of the first item code relative to the second item code is based on a depth sensor or a three-dimensional image sensor of the mobile device used to acquire the multiple images;the shelving unit is part of an aisle of a store, and the first vector includes an aisle indicator and a relative position of the first item code with respect to the aisle; and / or determining that the item identified by the first item code is in the correct aisle based on comparing the aisle indicator of the first vector with the item's planned aisle position from the parallelogram; ascertaining a first boundary of the object associated with the first item code, wherein the first vector includes a value indicative of the first boundary with respect to the shelf diagram; ascertaining a second boundary of the object associated with the second item code, wherein the second vector includes a value indicative of the second boundary with respect to the shelf diagram. using a value indicative of a first boundary from the first vector to compare the first boundary to a planned location of a first item in a planogram; determining that the first boundary for the shelf plan fits the planogram based on the first boundary for the shelf plan being similar to the planned location of the first item in the planogram; comparing the second boundary for the shelf plan to the planned location of a second item in the planogram using a value indicative of a second boundary from the second vector; and / or determining that the second boundary for the shelf plan fits the planogram based on the second boundary for the shelf plan being similar to the planned location of the second item in the planogram. In some embodiments, the method includes receiving a plurality of images, the plurality of images including an overhead image and a plurality of images having different fields of view, the overhead image including a plurality of barcodes, and / or the plurality of images each including a subset of the plurality of barcodes, and / or matching the different fields of view of the plurality of images with the overhead image to register locations of the barcodes in the overhead image. In some configurations, the barcodes cannot be decoded in the overhead image because they are too small to be decoded, the overhead image is aligned, rectified, and de-warped using rows in the overhead image, the matching of different views of the multiple images with the overhead image is performed using a random sample consensus algorithm for interest points and homographies, the matching of different views of the multiple images with the overhead image is based on the assumption that the barcodes are in a plane, and / or the method includes:The method includes tracking two or more barcodes simultaneously across multiple image frames without decoding the two or more barcodes, decoding the multiple barcodes using the multiple images, and / or calculating the scale of the overhead image based on a known height or width of one of the multiple barcodes.

[0149]

[0183] In some embodiments, 1. A method of image analysis for recognizing patterns in an image, comprising: receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns in an array; matching the array with a pattern template, the pattern template being a predetermined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array compared to the pattern template; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; A method comprising: 2. The method of claim 1, wherein the plurality of optical patterns is a barcode made up of parallel bars. 3. The method of claim 1, wherein the plurality of optical patterns are disposed on labels affixed to objects in the plurality of images, and the method further comprises the step of calculating physical dimensions of the objects. 4. The step of calculating the physical dimensions of an object is calculating relative dimensions of objects in the image to match relative dimensions of labels in the image; calculating a physical dimension of the object based on the relative dimensions of the object; 4. The method of claim 3, comprising: 5. Comparing the physical dimensions of the object to expected physical dimensions of an object associated with the object identifier; Matching the object with an object identifier; presenting a graphical element on the display indicating the match; The method of claim 3, further comprising: 6. The method of claim 3, wherein the physical dimensions of the object include at least one of the volume of the object or the weight of the object. 7. Steps to identify logistics standards based on physical dimensions The method of claim 6, further comprising: 8. The method of claim 7, wherein the logistics criteria includes at least one of a postage rate, an inventory order identifier, a packing order, or a warehouse location identifier. 9. Identifying a first identifier pattern in a first image of the plurality of images, the first identifier pattern being an identifier of a pattern template; identifying a second identifier pattern in a second image of the plurality of images; comparing the first identifier pattern to a second identifier pattern; If the first identifier pattern does not match the second identifier pattern, matching the second identifier pattern with a second pattern template, the second pattern template being a second predetermined arrangement of optical patterns; identifying a second selected optical pattern based on the position of the selected optical pattern within the array; decoding the second selected optical pattern to generate a second object identifier; storing the second object identifier in a memory device; The method of claim 1 further comprising: 10. Presenting a plurality of images including the pattern template on a display; presenting a visual representation of the pattern template on a display; presenting a graphic element overlaid on the selected optical pattern; The method of claim 1 further comprising: 11. A method of image analysis for recognizing a pattern in an image, comprising: receiving a target optical pattern type; receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns; matching a selected optical pattern of the plurality of optical patterns with a target optical pattern type; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; A method comprising: 12. The step of matching the selected optical pattern with a target optical pattern type comprises: receiving a first relative dimension of a target optical pattern type; determining a second relative dimension of the selected optical pattern; matching the second relative dimension to the first relative dimension; 12. The method of claim 11, comprising: 13. The step of matching the selected optical pattern with a target optical pattern type comprises: receiving a characteristic dimension of a target optical pattern type; determining a plurality of spatial dimensions of the plurality of optical patterns based on an arrangement of the plurality of optical patterns in an image of the plurality of images; Matching spatial dimensions of the selected optical pattern with characteristic dimensions of a target optical pattern type; 12. The method of claim 11, comprising: 14. The method of claim 11, wherein the target optical pattern type is a barcode consisting of parallel bars, a QR code, a MaxiCode, or an Aztec code. 15. The step of matching a selected optical pattern of the plurality of optical patterns with a target optical pattern type comprises: ascertaining a plurality of aspect ratios corresponding to the plurality of optical patterns or identifying a characteristic marking within one of the plurality of optical patterns; Matching the selected optical pattern to a target aspect ratio or characteristic marking of a target optical pattern type; 12. The method of claim 11, comprising: 16. A method of image analysis for recognizing a pattern in an image, comprising: receiving a first plurality of images captured by the camera; a first plurality of images including an image boundary and a partial optical pattern adjacent to the image boundary; determining the orientation of the partial optical pattern relative to an image boundary; generating a first prompt as a playback of a first recorded audio to move a camera in a first direction based on the direction; receiving a second plurality of images captured by a camera, the second plurality of images including a complete optical pattern corresponding to the partial optical pattern; decoding the complete optical pattern to generate an object identifier; providing a second prompt as a playback of a second recorded audio to indicate successful decoding of the complete optical pattern; storing the object identifier in a memory device; A method comprising: 17. The method of claim 16, wherein the complete optical pattern is a bar code made up of parallel bars. 18. Presenting a first plurality of images on a display; presenting a first graphic element on a display, the first graphic element showing a partial optical pattern; presenting a second plurality of images on the display; presenting a second graphic element on the display, the second graphic element showing the complete optical pattern; 17. The method of claim 16, further comprising: 19. Identifying the object information based on the object identifier; generating a third prompt including the object information as a playback of a third recorded audio; 17. The method of claim 16, further comprising: 20. An image analysis method for recognizing a pattern in an image containing multiple patterns, comprising: receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns; presenting a plurality of optical patterns on a display; Selecting a selected optical pattern from the plurality of optical patterns User actions, and a location of the selected optical pattern in one or more of the plurality of images; and identifying based on presenting a visual representation of the selected optical pattern on a display; decoding the selected optical pattern to generate an object identifier; indicating via a visual display that the selected optical pattern has been decoded; storing the object identifier in a memory device; A method comprising: 21. The method of claim 20, wherein the plurality of optical patterns is a bar code made up of parallel bars. 22. The plurality of optical patterns includes a first optical pattern and a second optical pattern; the first optical pattern is a selected optical pattern; The second optical pattern is not decoded. 21. The method of claim 20. 23. The method of claim 20, further comprising presenting, on a display, one or more graphical elements associated with processing the plurality of optical patterns. 24. The method of claim 23, wherein the one or more graphical elements include at least one of a reticle, a capture button, a bounding box, and a highlighted region of the display, and wherein at least the highlighted region is presented as an overlay on one optical pattern of the plurality of optical patterns. 25. The display presents a capture button and at least one of a reticle, a bounding box, or a highlighted area; The user action includes pressing the capture button. identifying the selected optical pattern based on the selected optical pattern located beneath the reticle, bounding box, or highlighted region; 25. The method of claim 24. 26. The method of claim 20, wherein the user action comprises a screen touch on the display. 27. The method of claim 20, wherein the user action comprises a screen touch on an area of the display corresponding to one of the plurality of optical patterns. 28. The display communicates with a smartphone that includes an inertial measurement unit; The user action is a tap on the exterior of the smartphone; The method is: receiving motion data from an inertial measurement unit of the smartphone; classifying the motion data as an impulse; Further comprising: and identifying the selected optical pattern based on the selected optical pattern being presented with the visual display upon receiving the impulse. 21. The method of claim 20. 29. The display communicates with an electronic device that includes a plurality of interactive buttons; The user action is a screen tap or an interaction with a capture button of a plurality of interactive buttons; The method is: presenting a graphical element on the display at a first position corresponding to a position of a first optical pattern of the plurality of optical patterns in one image of the plurality of images; repositioning the graphical element from a first position to a second position corresponding to a second optical pattern in the image in response to a user interaction with an interactive button of the plurality of interactive buttons; Further comprising: and identifying the selected optical pattern based on a correspondence between the graphic element and the selected optical pattern when coincident with a user action. 21. The method of claim 20. 30. Receiving a user interaction with a region of the display; magnifying an area of the plurality of images corresponding to the region by a predetermined magnification; 21. The method of claim 20, further comprising: 31. Presenting a plurality of images on a display in sequence according to a frame rate; receiving a second user action; presenting an image of the plurality of images on the display concurrently with a second user action; receiving a third user action; upon receiving a third user action, resuming presentation of the plurality of images on the display in sequence; 21. The method of claim 20, further comprising: 32. Identifying a first number of optical patterns of the plurality of optical patterns in a first image of the plurality of images; identifying a second number of the plurality of optical patterns in a second image of the plurality of images; determining the degree of change between the first number and the second number; comparing the degree of change to a threshold change criterion; presenting, on a display, an image of the plurality of images including a plurality of optical patterns if the degree of change meets a threshold change criterion; 21. The method of claim 20, further comprising: 33. Removing the image of the plurality of images from the display after a period of time has elapsed; presenting a plurality of images on a display; 33. The method of claim 32, further comprising: 34. The images are arranged in a sequence of frames corresponding to the frame rate; the second image corresponds to a second position in the sequence of frames subsequent to the first position in the sequence of frames corresponding to the first image; 33. The method of claim 32. 35. The method of claim 34, wherein the separation distance between the first position and the second position corresponds to a period according to the frame rate. 36. The method of claim 35, wherein the period is 300 milliseconds. 37. Identifying an application associated with the camera; determining a first position and a second position according to an application; 35. The method of claim 34, further comprising: 38. Identifying a device in communication with the camera and the display; determining a first location and a second location according to the device; 35. The method of claim 34, further comprising: 39. Ascertaining a plurality of device parameters describing at least one of a display, a camera, and a user; Determining the first location and the second location according to the device includes: calculating a period of time using a plurality of device parameters; setting the second position and the first position to correspond to the period according to the frame rate; 39. The method of claim 38, further comprising: 40. Decoding the plurality of optical patterns to generate a plurality of object identifiers; identifying a number of object identifiers corresponding to the plurality of object identifiers; According to the number of object identifiers that meet the threshold criteria, identifying a selected image of a plurality of images including a plurality of optical patterns; presenting the selected image on a display; 21. The method of claim 20, further comprising: 41. The method of claim 40, wherein the threshold criterion is a degree of change in the number of object identifiers, including a variation parameter in the number of object identifiers over a period of time. 42. Presenting a first object identifier of a plurality of object identifiers; presenting on a display first object information associated with the first object identifier; In response to a second user action, presenting a second object identifier of the plurality of object identifiers on the display and removing the first object identifier and the first object information from the display; presenting second object information associated with the second object identifier; 41. The method of claim 40, further comprising: 43. The method of claim 20, wherein the visual representation includes object information based on the object identifier. 44. The method of claim 20, wherein the visual display includes a dynamic graphic element that indicates the progress of the decoding as a progress bar. 45. The method of claim 20, wherein the visual display includes a dynamic graphic element that indicates time since decoding by gradually increasing the transparency of the visual display over a period of time.

[0150]

[0184] In some embodiments, 1. An image analysis method for recognizing patterns in an image, comprising: receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns in an array, the plurality of optical patterns disposed on a label affixed to an object in the plurality of images, one or more of the plurality of optical patterns encoding an object identifier; matching the array with a pattern template, the pattern template being a predetermined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; estimating relative dimensions of an object in one of the images compared to relative dimensions of a label in the image; estimating physical dimensions of the object based on the relative dimensions of the object; A method comprising: 2. The method of claim 1, wherein the plurality of optical patterns is a barcode made up of parallel bars. 3. Comparing the physical dimensions of the object to expected physical dimensions of an object associated with the object identifier; Matching the object with an object identifier; presenting a graphical element on the display indicating the match; The method of claim 1 further comprising: 4. The method of claim 1, wherein the physical dimensions of the object include at least one of the volume of the object or the weight of the object. 5. The method of claim 1, further comprising the step of verifying logistics criteria based on physical dimensions. 6. The method of claim 5, wherein the logistics criteria includes at least one of a postage rate, an inventory order identifier, a packing order, or a warehouse location identifier. 7. Presenting a plurality of images including the pattern template on a display; presenting a graphic element overlaid on the selected optical pattern; The method of claim 1 further comprising: 8. A camera having an image sensor configured to capture a plurality of images; one or more processors, receiving a plurality of images captured by an image sensor, the plurality of images including a plurality of optical patterns in an array, the plurality of optical patterns disposed on a label affixed to an object in the plurality of images, one or more of the plurality of optical patterns encoding an object identifier; matching the array with a pattern template, the pattern template being a predetermined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; Estimating a relative dimension of an object in one of the images compared to a relative dimension of a label in the image; Estimating the physical dimensions of an object based on the relative dimensions of the object; one or more processors configured to: A system comprising: 9. One or more processors may: 10. The system of claim 8, configured to validate logistics criteria based on physical dimensions, the logistics criteria including at least one of a postage rate, an inventory order identifier, a packing order, or a warehouse location identifier. 10. One or more processors may: comparing the physical dimensions of the object to expected physical dimensions of an object associated with the object identifier; Matching the object with an object identifier; presenting a graphical element on the display indicating the match; The system of claim 8 , further configured to: 11. One or more processors may: presenting a plurality of images including the pattern template on a display; presenting a graphic element overlaid on the selected optical pattern; The system of claim 8 , further configured to: 12. When executed, receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns in an array, the plurality of optical patterns disposed on a label affixed to an object in the plurality of images, one or more of the plurality of optical patterns encoding an object identifier; matching the array with a pattern template, the pattern template being a predetermined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; estimating relative dimensions of an object in one of the images compared to relative dimensions of a label in the image; estimating physical dimensions of the object based on the relative dimensions of the object; A memory device that contains instructions that cause one or more processors to perform 13. When executed, presenting a plurality of images including pattern templates on a display; presenting a graphic element overlaid on the selected optical pattern; and further comprising instructions to cause one or more processors to: Optionally, the plurality of optical patterns comprise a string of characters, the plurality of optical patterns comprise a two-dimensional optical code, the object is a first object, the method further comprises estimating physical dimensions of a second object in the plurality of images based on relative dimensions of the label in the images, the plurality of optical patterns comprise a string of characters, the plurality of optical patterns comprise a two-dimensional optical code, and / or the method further comprises presenting a visual representation of the pattern template on a display.

[0151]

[0185] In some embodiments, 1. A mobile device, The display and A camera and one or more processors in communication with the camera and / or the display; When executed by one or more processors, the mobile device using a camera to detect an optical pattern in a scene, the optical pattern encoding an object identifier; receiving a user action; performing a focus cycle of the camera after receiving a user action, the focus cycle changing a focus position of the camera from a first focus position to a second focus position, the second focus position corresponding to an optical pattern in focus within the scene; performing a focus cycle and then capturing an image of the scene using the camera at a second focus position; decoding the optical pattern in the image of the scene to generate an object identifier; one or more memory devices for storing instructions; A mobile device comprising: 2. The mobile device of claim 1, wherein the user action is received via the display after detecting an optical pattern in the scene. 3. The mobile device of claim 1, wherein the scene is a first scene, the user action includes an action of the mobile device, and the instructions, when executed, cause the one or more processors to detect a scene change from the first scene to a second scene. 4. Detecting scene changes Detecting behavior of the mobile device that exceeds a threshold behavior; or detecting a scene change based on motion blur in an image of the scene acquired by a camera; 4. The mobile device of claim 3, comprising: 5. The image of the scene is a second image, and detecting optical patterns in the scene is acquiring a first image of the scene using a camera, the focus position of the camera being at a first focus position; Detecting an optical pattern in the first image without decoding the optical pattern; confirming that the optical pattern is out of focus in the first image; The mobile device of claim 1 , comprising: 6. Identifying that the optical pattern is out of focus in the first image includes: Ascertaining the resolution of the optical pattern in the first image; and determining that the resolution of the optical pattern in the first image is less than a minimum resolution for decoding the optical pattern; The mobile device of claim 5, comprising: 7. The optical pattern is a first optical pattern, the object identifier is a first object identifier, and the instructions, when executed, further cause one or more processors to: detecting a second optical pattern in the first image prior to receiving the user action, the second optical pattern encoding a second object identifier; and decoding a second optical pattern in the first image to generate a second object identifier; The mobile device of claim 5. 8. Detecting, by a computer system, an optical pattern in the scene using a camera at a first focus position, the optical pattern encoding an object identifier; receiving, by the computer system, a user action; performing, by the computer system, a focus cycle of the camera after receiving a user action, the focus cycle changing a focus position of the camera from a first focus position to a second focus position, the second focus position corresponding to an optical pattern in focus in the scene; acquiring, by a computer system, an image of the scene using the camera at a second focus position; decoding, with a computer system, the optical pattern in the image of the scene to generate an object identifier; A method comprising: 9. Presenting, by the computer system, a visual indication on the display that the optical pattern cannot be decoded. The method of claim 8, further comprising: 10. The method of claim 8, wherein the scene is a first scene and the user action comprises an action of a computer system, the method further comprising detecting a scene change from the first scene to a second scene. 11. The step of detecting a scene change comprises: detecting behavior of the mobile device that exceeds a threshold behavior; or detecting a scene change based on motion blur in an image of the scene acquired by a camera; 11. The method of claim 10, comprising: 12. The image of the scene is a second image, and the step of detecting an optical pattern in the scene comprises: acquiring a first image of the scene using a camera, the focus position of the camera being at a first focus position; detecting an optical pattern in the first image without decoding the optical pattern; determining that the optical pattern is out of focus in the first image; 9. The method of claim 8, comprising: 13. The step of determining that the optical pattern is out of focus in the first image comprises: determining the resolution of the optical pattern in the first image; determining that a resolution of the optical pattern in the first image is less than a minimum resolution for decoding the optical pattern; 13. The method of claim 12, comprising: 14. The optical pattern is a first optical pattern, the object identifier is a first object identifier, and the method comprises: detecting a second optical pattern in the first image prior to receiving the user action, the second optical pattern encoding a second object identifier; decoding a second optical pattern in the first image to generate a second object identifier; 13. The method of claim 12, further comprising: 15. When executed by one or more processors of a computer system, the one or more processors: detecting an optical pattern in the scene using the camera at a first focus position, the optical pattern encoding an object identifier; receiving a user action; performing a focus cycle of the camera after receiving a user action, the focus cycle changing a focus position of the camera from a first focus position to a second focus position, the second focus position corresponding to an optical pattern in focus in the scene; capturing an image of the scene using the camera at a second focus position; decoding the optical pattern in the image of the scene to generate an object identifier; A memory device that stores instructions. 16. Disabling the camera's autofocus system, which controls the camera's focus position through a computer system. The memory device of claim 15 further comprising: 17. The memory device of claim 15, wherein the scene is a first scene, the user action comprises an action of a computer system, and the instructions, when executed, cause the one or more processors to detect a scene change from the first scene to a second scene. 18. Detecting scene changes Detecting computer system behavior that exceeds threshold behavior; or detecting a scene change based on motion blur in an image of the scene acquired by a camera; 20. The memory device of claim 17, comprising: 19. The image of the scene is a second image, and detecting an optical pattern in the scene is acquiring a first image of the scene using a camera, the focus position of the camera being at a first focus position; Detecting an optical pattern in the first image without decoding the optical pattern; confirming that the optical pattern is out of focus in the first image; The memory device of claim 15 , comprising: 20. Identifying that the optical pattern is out of focus in the first image includes: Ascertaining the resolution of the optical pattern in the first image; and determining that the resolution of the optical pattern in the first image is less than a minimum resolution for decoding the optical pattern; 20. The memory device of claim 19, comprising:

Claims

1. 1. A system for tracking an optical code, comprising: a camera having an image sensor configured to acquire a plurality of frames; one or more processors, acquiring the plurality of frames from the image sensor; each of the plurality of frames includes an image of a code; the plurality of frames includes a first frame and a second frame; the second frame is acquired after the first frame; Identifying the code in the first frame; extracting features from the code in the first frame; creating a filter based on the extracted features of the code; extracting features from the second frame; generating a response map based on features extracted from the second frame combined with the filter, the response map providing a probability of the location of the code within the second frame; determining a location of the code within the second frame based on the response map without decoding the code; one or more processors configured to: A system comprising:

2. The system of claim 1 , wherein the filter is a correlation filter.

3. the cord is a first cord, the one or more processors are configured to identify a second code in the second frame without decoding the second code. The system of claim 1 .

4. the filter is created by transforming the extracted features of the code into frequency space using a Fourier transform; The response map may be: transforming the second frame into frequency space using the Fourier transform to generate a frequency image; and The system of claim 1 , wherein the frequency image is generated by convolving the frequency image with the filter.

5. 1. A method for tracking an optical code, comprising: acquiring a plurality of frames from an image sensor; each of the plurality of frames includes an image of a code; the plurality of frames includes a first frame and a second frame; the second frame is acquired after the first frame; Steps and identifying the code in the first frame; extracting features from the code in the first frame; creating a filter based on the extracted features of the code; extracting features from the second frame; generating a response map based on features extracted from the second frame combined with the filter; determining the location of the code within the second frame based on the response map without decoding the code; A method comprising:

6. The method of claim 5 , wherein the filter is a correlation filter.

7. 6. The method of claim 5, further comprising calculating a homography pose change of the image sensor from the first frame to the second frame based on identifying the location of the code in the first frame and the second frame.

8. the plurality of frames includes a third frame; the third frame is acquired after the second frame; the method including predicting a sub-area of the third frame that includes the image of the code based on the position of the code in the second frame; The method of claim 5.

9. The method of claim 8 further comprising the step of decoding the code using the image of the code from the third frame.

10. the plurality of frames includes a fourth frame; the fourth frame is acquired after the third frame; the cord is a first cord, The method comprises: identifying a location of the second code within the third frame without decoding the second code; decoding the second code in the fourth frame; 10. The method of claim 9, comprising:

11. The method of claim 5 further comprising presenting the second frame on a display with a graphic on the display indicating the location of the code.

12. the cord is a first cord, the method further comprising identifying a second code in the second frame without decoding the second code. The method of claim 5.

13. The method of claim 5 , wherein the plurality of frames comprises 10 frames taken within one second.

14. the filter is created by transforming the extracted features of the code into frequency space using a Fourier transform; The response map may be: transforming the second frame into frequency space using the Fourier transform to generate a frequency image; and The method of claim 5 , wherein the frequency image is generated by convolving the frequency image with the filter.

15. The method of claim 5 , wherein the code comprises a machine-readable bar code.

16. The following steps are taken to trace the code as it is executed: acquiring a plurality of frames from an image sensor; each of the plurality of frames includes an image of the code; the plurality of frames includes a first frame and a second frame; the second frame is acquired after the first frame; Steps and identifying the code in the first frame; extracting features from the code in the first frame; creating a filter based on the extracted features of the code; extracting features from the second frame; generating a response map based on features extracted from the second frame combined with the filter; locating the code within the second frame based on the response map without decoding the code; A memory device containing instructions that cause one or more processors to perform the following:

17. the plurality of frames includes a third frame; the third frame is acquired after the second frame; the instructions are further configured to cause the one or more processors to predict a sub-area of the third frame that includes the image of the code based on the position of the code in the second frame.

17. The memory device of claim 16.

18. 20. The memory device of claim 17, wherein the instructions are configured to cause the one or more processors to decode the code using the image of the code from the third frame.

19. the cord is a first cord, the plurality of images includes a second code; the instructions are configured to cause the one or more processors to calculate a relative position between the first code and the second code based on information about the first code.

20. The memory device of claim 17.

20. 17. The memory device of claim 16, wherein the instructions are configured to be executed as part of a browser-based application.

21. 1. An apparatus for decoding an optical pattern, comprising: A camera, an image sensor; a lens configured to focus light onto the image sensor, the lens being a wide-angle lens; a camera comprising: one or more processors, acquiring a plurality of images using the camera; the lens is set to a fixed focus during acquisition of the plurality of images; the exposure time for each of the plurality of images is 1 / 500 of a second or less; the plurality of images are acquired at a frame rate of the camera; The frame rate is 30 frames / second or more, analyzing the plurality of images to detect optical patterns within the plurality of images; decoding the optical pattern to obtain a data payload; storing the data payload in a memory device of the device; one or more processors configured to: An apparatus comprising:

22. 22. The device of claim 21, wherein the lens is a single vision lens.

23. 22. The apparatus of claim 21, further comprising a torch, wherein the one or more processors are configured to activate the torch during the acquisition of the plurality of images.

24. 24. The device of claim 23, wherein the torch is configured to turn on while a user touches a screen of the device with a finger and to turn off upon the user removing their finger from the screen.

25. 24. The apparatus of claim 23, wherein the torch has a constant value during the acquisition of the plurality of images.

26. 22. The device of claim 21, wherein the plurality of images are configured to be captured while a user touches a screen of the device with a finger.

27. 27. The device of claim 26, wherein the camera stops capturing the images based on the user removing their finger from the screen.

28. 22. The apparatus of claim 21, wherein the camera has an angle of view of 100 degrees or greater.

29. 1. A method for decoding an optical pattern, comprising: acquiring a plurality of images using a camera; the camera lens is set to a fixed focus during acquisition of the plurality of images; the lens of the camera is a wide-angle lens, the exposure time for each of the plurality of images is 1 / 500 of a second or less; the plurality of images are acquired at a frame rate of the camera; The frame rate is 30 frames / second or more. Steps and analyzing the plurality of images to detect optical patterns within the plurality of images; decoding the optical pattern to obtain a data payload; storing the data payload in a memory device; A method comprising:

30. 30. The method of claim 29, wherein the plurality of images are captured while a user touches a screen of a device with a finger, the device comprising the camera.

31. 31. The method of claim 30, wherein the camera stops capturing the images based on the user removing their finger from the screen.

32. 32. The method of claim 31, wherein a torch is turned on while the user touches the screen on the device with a finger and is turned off based on the user removing their finger from the screen.

33. 30. The method of claim 29, wherein the lens is a variable focus lens in which autofocus is disabled during acquisition of the plurality of images.

34. 30. The method of claim 29, wherein the exposure time is set to a specific value and the camera adjusts its ISO during acquisition of the plurality of images.

35. 30. The method of claim 29, further comprising tracking the position of the optical pattern before decoding the optical pattern.

36. 30. The method of claim 29, further comprising tracking the position of the optical pattern after decoding the optical pattern without decoding the optical pattern again.

37. 37. The method of claim 36, wherein the optical pattern is a first optical pattern, the method further comprising automatically decoding a second optical pattern in an image that includes the first optical pattern.

38. 38. The method of claim 37, wherein the first optical pattern is the same as the second optical pattern.

39. 30. The method of claim 29, wherein after decoding the optical pattern, no replica filters with timers greater than 0.5 seconds are used.

40. the optical pattern is a first optical pattern, the data payload is a first data payload; The method comprises: analyzing the plurality of images for a second optical pattern; decoding the second optical pattern to obtain a second data payload; storing the second data payload in the memory device; 30. The method of claim 29, further comprising:

41. 30. The method of claim 29, wherein the exposure time is 1 / 650 seconds or more and 1 / 1000 seconds or less.

42. 30. The method of claim 29, wherein the frame rate is 120 fps or greater.

43. 30. The method of claim 29, wherein the plurality of images are acquired at a distance of at least 0.02 meters and at most 0.5 meters from the optical pattern.

44. When executed, the following steps are performed to decode the optical code: acquiring a plurality of images using a camera; the camera lens is set to a fixed focus during acquisition of the plurality of images; the lens of the camera is a wide-angle lens, the exposure time for each of the plurality of images is 1 / 500 of a second or less; the plurality of images are acquired at a frame rate of the camera; The frame rate is 30 frames / second or more. Steps and analyzing the plurality of images to detect optical patterns within the plurality of images; decoding the optical pattern to obtain a data payload; storing the data payload; A memory device containing instructions that cause one or more processors to perform the following:

45. 45. The memory device of claim 44, wherein the instructions, when executed, cause the one or more processors to transmit the data payload to a remote hardware server.

Images