Systems and methods for decoding multiple optical codes

Abstract

To provide a system, a method, a memory device, and an apparatus for acquiring and / or visualizing data from images acquired by a mobile device.SOLUTION: An image analysis method used for tracking a plurality of optical patterns such as a barcode, comprises: acquiring a plurality of frames from an image sensor; identifying a code, such as a barcode, in a first frame; acquiring the plurality of frames from the image sensor; extracting features from the code; extracting features based on the extracted information; generating a filter; extracting features from a second frame; generating a response map based on the extracted features; and locating the code based on the response map.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Nonprovisional Application No. 17 / 186,909, filed February 26, 2021, which claims priority to U.S. Provisional Application No. 63 / 149,009, filed February 12, 2021, which is incorporated by reference herein for all purposes. Application No. 17 / 186,909 is a continuation-in-part of U.S. Nonprovisional Application No. 17 / 139,529, filed December 31, 2020, which is a continuation-in-part of U.S. Nonprovisional Application No. 16 / 920,061, filed July 2, 2020, now U.S. Patent No. 10,963,658, which issued March 30, 2021, and is a continuation-in-part of U.S. Nonprovisional Application No. 16 / 920,061, filed July 2, 2020, now U.S. Patent No. 10,963,658, which issued March 30, 2021. No. 10,963,658 claims the benefit of priority to U.S. Provisional Application No. 63 / 017,493, filed April 29, 2020, U.S. Provisional Application No. 63 / 003,675, filed April 1, 2020, and U.S. Provisional Application No. 63 / 019,818, filed May 4, 2020, which are incorporated by reference herein for all purposes. Application No. 17 / 186,909 is a continuation-in-part of U.S. Patent Application No. 16 / 905,722, filed June 18, 2020, now U.S. Patent No. 10,846,561, issued November 24, 2020, which is incorporated by reference herein for all purposes. Application No. 17 / 186,909 claims priority to U.S. Provisional Application No. 63 / 025,850, filed May 15, 2020, which is incorporated by reference for all purposes. Application No. 17 / 186,909 is a continuation-in-part of U.S. Provisional Application No. 17 / 105,082, filed November 25, 2020, which claims the benefit of priority to U.S. Provisional Application No. 63 / 044,635, filed June 26, 2020, which are incorporated by reference for all purposes. [Background technology]

[0002]

[0002] The present disclosure relates generally to decoding codes, and more specifically, but not exclusively, to decoding barcodes in a retail environment. Barcodes have traditionally been scanned using dedicated scanners. For example, a barcode scanner with a laser is used to shine light on the barcode, and reflected light from the barcode is detected and used to decode the barcode. As mobile devices with cameras (e.g., smartphones and tablets) become more common, mobile devices are being used to decode codes by taking an image of the code and using image analysis to decode the code. One example of a method for using a smartphone to decode barcodes is provided in U.S. Patent No. 8,596,540, granted December 3, 2013. Summary of the Invention [Problem to be solved by the invention]

[0003]

[0003] This disclosure generally relates to tracking and decoding computer readable codes (e.g., barcodes; QR codes). For example, a barcode may be a stock keeping code (SKU) in a retail setting. However, the systems and / or processes described herein may also be used to track and / or identify other features, such as text by optical character recognition, in other settings (e.g., scanning a passport at an airport terminal). Furthermore, many applications are becoming web-based. However, web-based applications may have limited computational resources. As many operations become more computationally dependent, improvements are needed for acquiring and / or visualizing data from images, particularly images acquired by mobile devices. [Means for solving the problem]

[0004] In a particular embodiment, a system for tracking an optical code includes a camera. The camera may be configured to acquire a plurality of frames. The system may include one or more processors. The one or more processors may be configured to acquire a plurality of frames from an image sensor, each of the plurality of frames including an image of a code, the plurality of frames including a first frame and a second frame, and / or the second frame being acquired after the first frame, identify the code in the first frame, extract features from the code in the first frame, create a filter based on the extracted features of the code, extract features from the second frame, generate a response map based on the extracted features from the second frame combined with the filter, the response map providing a probability of a location of the code in the second frame, and / or confirm the location of the code in the second frame based on the response map without decoding the code. In some embodiments, the filter is a correlation filter, the code is a first code, and the one or more processors are configured to identify the second code in the second frame without decoding the second code, and the filter is created by transforming extracted features of the code into frequency space using a Fourier transform, and / or the response map is generated by transforming the second frame into frequency space using a Fourier transform to generate a frequency image and convolving the frequency image with the filter.

[0005]

[0005] In a particular aspect, a method for tracking an optical code includes acquiring a number of frames from an image sensor, each of the multiple frames including an image of the code, the multiple frames including a first frame and a second frame and / or the second frame being acquired after the first frame; identifying the code in the first frame; extracting features from the code in the first frame; creating a filter based on the extracted features of the code; extracting features from the second frame; generating a response map based on the features extracted from the second frame combined with the filter; and locating the code in the second frame based on the response map without decoding the code. In some embodiments, the filter is a correlation filter, the code comprises a machine-readable barcode, the filter is created by transforming extracted features of the code into frequency space using a Fourier transform, the response map is generated by transforming the second frame into frequency space using a Fourier transform to generate a frequency image and convolving the frequency image with the filter, the plurality of frames includes a third frame, the third frame is acquired after the second frame, the plurality of frames includes a fourth frame, the fourth frame is acquired after the third frame, the code is a first code, the plurality of frames includes 10 frames taken within 1 second, and / or the method comprises: the second code in a fourth frame; presenting the second frame on a display together with a graphic on the display indicating the location of the code; and / or identifying the second code in the second frame without decoding the second code.

[0006] In certain embodiments, an apparatus for decoding an optical pattern comprises a camera. The camera may comprise an image sensor. The camera may comprise a lens configured to focus light onto the image sensor. The lens may be a wide-angle lens. The apparatus may comprise one or more processors. The one or more processors may be configured to acquire a plurality of images using the camera, analyze the plurality of images to detect an optical pattern in the plurality of images, decode the optical pattern to acquire a data payload, and / or store the data payload in a memory device of the apparatus. The lens may be set to a fixed focus during acquisition of the plurality of images. An exposure time for each of the plurality of images may be 1 / 500th of a second or less. The plurality of images may be acquired at a frame rate of the camera. The frame rate may be 30 frames per second or greater.

[0007]

[0007] In certain embodiments, a method for decoding an optical pattern includes acquiring a plurality of images using a camera, analyzing the plurality of images to detect an optical pattern in the plurality of images, decoding the optical pattern to acquire a data payload, and / or storing the data payload in a memory device, wherein the camera lens is set to a fixed focus during acquisition of the plurality of images, the camera lens is a wide-angle lens, the exposure time for each of the plurality of images is 1 / 500 of a second or less, the plurality of images are acquired at a frame rate of the camera, and / or the frame rate is 30 frames per second or greater.

[0008] In certain embodiments, the apparatus comprises an attachment for a mobile device having a torch and a camera with a first lens, the apparatus comprises a second lens and / or a third lens, the attachment configured to be removably coupled to the mobile device, the second lens configured to increase the angle of view of the camera of the mobile device while the attachment is coupled to the mobile device, and / or the third lens configured to focus the light of the torch while the attachment is coupled to the mobile device. In some embodiments, the third lens is configured to focus the light of the torch in a normal direction from the torch, the third lens is configured to focus the light of the torch into an elliptical shape, the elliptical shape being non-circular, the attachment is a case covering at least a portion of two or more sides of the mobile device, and / or the attachment further comprises a magnet.

[0009]

[0009] In certain embodiments, the method includes using a torch on the mobile device to aim the mobile device at the optical pattern.

[0010] Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for purposes of illustration only and are not intended to necessarily limit the scope of the present disclosure.

[0011]

[0011] The present invention is described in conjunction with the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] 1 illustrates an embodiment of a tracking method. [Diagram 2] 1 illustrates an embodiment that does not track the code in each frame and decode the code. [Diagram 3] FIG. 1 shows a diagram of one embodiment for creating a correlation filter. [Figure 4]FIG. 1 illustrates an embodiment for using a correlation filter to generate a response map for tracking chords. [Diagram 5] 1 illustrates an embodiment for decoding codes across multiple frames. [Figure 6] 1 illustrates a flow chart of one embodiment of a process for tracking code. [Figure 7] An example of segmenting products on a shelf from an image. [Figure 8] 1 is an embodiment of a shelf diagram. [Figure 9] 1 shows a flow chart of one embodiment of a process for mapping objects onto a shelving unit. [Figure 10] Here is an example of blending two images of a shelf. [Figure 11] 1 is an embodiment of a shelf diagram with visual images of the shelves. [Figure 12] 1 shows a flow chart of one embodiment of a process for creating a visual image of a shelf. [Figure 13] FIG. 1 illustrates an embodiment of a mobile device for rapidly scanning an optical pattern. [Figure 14] 1 shows a chart illustrating scan range as a function of camera focus setting for scanning a first type of barcode. [Figure 15] 13 shows a chart illustrating scan range as a function of camera focus setting for scanning a second type of barcode. [Figure 16] 1 shows a flowchart of one embodiment of a process for decoding an optical pattern using a digital camera. [Figure 17] 1 is an embodiment of an attachment for a mobile device. [Figure 18] 1 is another embodiment of an attachment for a mobile device. [Figure 19] FIG. 1 is a block diagram of one embodiment of a computer system.

[0013]

[0031] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. If only a first reference label is used herein, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014]

[0032] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing the preferred exemplary embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0015]

[0033] Matrix Scan

[0034] Many applications are becoming web-based. However, web-based applications may have fewer computational resources than native applications. For example, a native application may be used to track a barcode based on decoding barcodes from multiple images. However, decoding the barcode is computationally intensive and may cause delays when moved to a web-based application. Thus, in some embodiments, the barcode is tracked for several frames, but is only decoded periodically for the web-based application to be used to decode the barcode. In some embodiments, a frame is one of a series of separate photographs that make up a film or video.

[0016]

[0035] Referring first to FIG. 1, one embodiment of a tracking scheme is shown. This tracking scheme can be used in native applications. In FIG. 1, in a first frame 108-1 (e.g., initialization) at time T=1, a first code 104-1 and a second code 104-2 are decoded and the locations of the first code 104-1 and the second code 104-2 are confirmed. The first frame 108-1 corresponds to a first image acquired by an image sensor (e.g., from a camera in a mobile device). The code 104 is an optical pattern. The code 104 can be a machine-readable code such as a one-dimensional barcode or a two-dimensional barcode (e.g., a QR code) having multiple horizontal lines, a symbol (e.g., a dollar sign, a triangle, etc.), a number, and / or a letter. For example, the code 104 can be a price, a VIN number for identifying a vehicle, a credit card number, a license plate number, a serial number, a tire code (TIN), or a date (e.g., an expiration date). The code 104 is in a variety of environments and / or applications. For example, the code 104 may be part of or on a shipping label, a product label, a passport, a shipping invoice, a driver's license, an ID card, a credit card, a check, a license plate, a digital display (e.g., an electronic price tag), a utility meter, an ID document with a machine readable zone (MRZ), or a retail store receipt.

[0017]

[0036] At time T=2, a prediction 112 is calculated (e.g., assuming smooth motion of the mobile device), the code 104 is decoded in a second frame 108-2 (the second frame 108-2 corresponds to a second image captured by the image sensor), the code 104 is matched against the prediction 112, and an updated position 116 of the code 104 is ascertained based on the code 104 decoded from the second frame 108-2.

[0018]

[0037] Some concerns with the above scheme include inaccurate detection and / or missed detections because the bounding box (e.g., updated position 116) based on the decoding of the 1D code may be inaccurate. If decoding is not performed on each frame, detections may be missed and / or the detection track may be inconsistent. An example of a detection track is shown in updated position 116, where the dotted outline was the previous position. Also, decoding may be more difficult with faster movement and lower resolution.

[0019]

[0038] One possible solution is to use more data, such as predicting the position of the code 104 based on the image history (e.g., to establish the trajectory of the code 104 in the field of view), using known locations of the code 104 and / or background structures (e.g., shelving units), and / or leveraging additional sensor data (e.g., Inertial Measurement Unit (IMU) data from a gyroscope to predict the attitude changes of the mobile device). However, using more data can cause delays when using web-based applications. Furthermore, not all devices have the computing power and / or equipment (e.g., IMU) to use more data. Therefore, another possible solution, described in more detail below, is to decode the code in the first and third frames and track the position of the code 104 in the second frame between the first and third frames.

[0020]

[0039] 2 illustrates an embodiment of tracking and not decoding the code in each frame. In the first frame 108-1, at time T=1, an algorithm is used to search for locations that look like code 104. The locations that look like code 104 are decoded (or an attempt is made to decode the code). Also, initialization of tracking of code 104 is performed (e.g., as described in connection with FIG. 3 by creating a correlation filter).

[0021]

[0040] In the second frame 108-2, at T=2, the code 104 is tracked and not decoded. The tracking algorithm attempts to find a new position of the tracked code in the second frame 108-2. In the second frame 108-2, a first bounding box 204-1 is calculated relative to the second frame 108-2 in which the first code 104-1 is to be calculated (e.g., the bounding box 204 is simply calculated and / or overlaid on the second frame 108-2 in the user interface of the mobile device) and a second bounding box 204-2 is calculated relative to the second frame 108-2 in which the second code 104-2 is to be calculated. For example, a correlation filter is used to create a response map and determine the position of the bounding box 204, as described in relation to FIG. 4. Tracking in the second frame 108-2 does not depend on the limitations that exist when decoding the code. For example, the code 104 may be blurred in the second frame 108-2. In one embodiment, the correlation filter is learned at tracking initialization and is continuously updated during the tracking process to adapt, for example, to changes in viewpoint. In another embodiment, the correlation filter operates on a selected subset of image features, which may be highly specific to the barcode and may be extracted using a pre-trained neural net.

[0022]

[0041] In the third frame 108-3, at T=3, the code 104 is scanned and tracked. For example, an algorithm is used to search for locations that appear to be code 104. Locations that appear to be code 104 are decoded and / or the tracking algorithm identifies new locations of code 104 in the third frame 108-3 for codes 104 that could not be or were not decoded.

[0023]

[0042] Although only one frame, the second frame 108-2, is shown between the first frame 108-1 and the third frame 108-3, it should be understood that the second frame 108-2 can be one of many frames between the first frame 108-1 and the third frame 108-3. It should also be understood that the third frame 108-3 action can be repeated (e.g., periodically or according to an event such as the location of the bounding box not moving beyond a set distance, which may indicate that there is likely to be less motion blur in the image) with the second frame 108-2 action occurring during one or more frames. Thus, the tracking algorithm can determine the location of the code between the scanning of the code and the decoded code. For example, a smartphone camera captures images at 30 frames per second. The third frame 108-3 action of FIG. 2 is set to occur at an interval (sometimes called the scan rate) of 2 Hz, and the second frame 108-2 action of FIG. 2 is set to occur at the frames captured by the camera during that interval. In some embodiments, the interval is at a frequency of 0.3, 0.5, 1, 2, or 5 Hz or more and at a frequency of 5, 10, or 20 Hz or less. In some configurations, the camera has a frame rate of 1, 5, 10, or 20 Hz or more and 10, 30, 60, or 120 Hz or less. There is a tradeoff in the frame rate used between wanting a high frame rate so that objects do not move too much between frames and wanting to find new barcodes as they appear as soon as possible. Applicant has found that 60 ms to 100 ms, e.g., 80 ms, between successive scans is a good tradeoff as it conveys a "fine-grained" experience to the user.

[0024]

[0043] In some embodiments, only new codes that are detected are decoded in subsequent frames. Thus, in some embodiments, barcode scanning and / or barcode decoding is performed only in the first frame 108-1. For example, a store employee scans several barcodes with his smartphone. In the first frame, the scan identifies two barcodes and decodes two barcodes. The two barcodes are then tracked and a green dot is overlaid on the image displayed to the smartphone user using the smartphone screen providing the user with an augmented reality display of the decoded barcodes. The user can see a third barcode on the screen that is not covered by the green dot, so the user continues to move the smartphone and scans the third barcode. In the frame where the image is searched for locations that appear to be barcodes, three locations are identified: two locations corresponding to the two previously identified barcodes and a new location. The two locations corresponding to the two previously identified barcodes are not decoded, while the new location reveals a third barcode. A third barcode is then tracked and a green dot is overlaid on the screen where the third barcode is tracked. Thus, the user can see which code has been decoded without the application having to decode each barcode in each frame. Because decoding a code can consume more computational resources than tracking, tracking can improve the functionality of a mobile device, especially when the application is a web-based application and not running as a native application.

[0025]

[0044] In some embodiments, scanning the code may include both a search for locations that look like a code and a decoded code, and may be split to occur in different frames. For example, a search for locations that look like a code is done in a first frame and five locations are identified. In frames 2 through 10, five locations are tracked. In frame 11, the code in the first location is decoded. In frames 12 through 20, five locations are tracked. In frame 21, the code in the second location is decoded. In frames 22 through 30, five locations are tracked. In frame 31, a search for locations that look like a code is performed and a sixth location is identified. The process continues searching for new locations and decoded codes.

[0026]

[0045] In some configurations, the tracking described in connection with FIGS. 2-6 may enable more stable tracking because the tracking does not rely on a visually decodable barcode (e.g., as described in FIG. 1). The tracking may provide a user with a smoother visualization due to increased frame rate and / or may reduce energy consumption when performing "normal" barcode scanning (e.g., because codes already scanned are not rescanned). Decoding codes in each frame may not be useful in various situations. In one example where a user is scanning multiple codes for inventory control, the barcodes may be at a long distance. If the user moves the mobile phone quickly to scan a barcode, the barcode may be blurred from motion blur. Thus, tracking that relies on decoding codes may lose track of the barcode.

[0027]

[0046] In some embodiments, the method includes visually extracting the code, establishing correspondence between frames of decoding, predicting the location of the code, and re-scanning, where predicting is performed by independently predicting the locations of multiple codes and / or by using only image data (e.g., not IMU data).

[0028]

[0047] In some embodiments, the tracking algorithms described in Figures 2-6 can be implemented with other algorithms, such as prediction algorithms. For example, the tracking algorithms can be used in combination with homography prediction algorithms or when homography prediction algorithms fail. In some embodiments, the barcodes can be tracked as a single rigid structure. Taking advantage of the fact that barcodes do not move relative to each other can help reduce computational complexity and improve fault tolerance / fault detection.

[0029]

[0048] FIG. 3 shows a diagram of one embodiment for creating the correlation filter 304. FIG. 3 is an example of initialization. Extracted features 308 are obtained from the code 104. The extracted features can be areas of high contrast (e.g., image gradient), lines, corners, etc. In many scenarios, the barcode is placed on a rigid object or surface, so the extracted features 308 can include geometric cues in addition to the code 100 itself (e.g., the edges of the shelf or dollar sign can be part of the extracted features 308). In some embodiments, the extracted features 308 are transformed from the visual domain to the frequency domain using a Fourier transform (FT) and combined with the target output 312 to form the correlation filter 304. The Fourier transform is efficient and reduces computation time. In another embodiment, the operation is performed in the spatial domain. In some embodiments, to "train" the correlation filter, the features are transformed in the frequency domain and then a mask / filter is calculated. When convolved with the features, this results in the target output. The target output is a desired response map in the spatial domain (e.g. a probability map of object locations; basically a matrix of the size of the analyzed patch with a single peak at the center of the patch, indicating the central location of the object) which is transformed to the frequency domain.

[0030]

[0049] FIG. 4 shows a diagram of one embodiment for using a correlation filter 304 to generate a response map 404 for tracking a code 104. In FIG. 4, extracted features 408 are obtained from an image 412. The image 412 is a portion of multiple image frames acquired by a camera. The image 412 includes a code 104. The extracted features 408 are transformed into frequency space using a Fourier transform and combined with a correlation filter 304 (e.g., convolution) to generate a response map 404. The extracted features 408 can be combined with a window 416 before combining with the correlation filter 304. The window 416 can be used to limit the search for the code 104 to an area smaller than the area of ​​the entire image 412. The response map 404 provides a measure of the correlation. Thus, the whiter an area of ​​the response map 404 is, the more likely that the code 104 is in that area.

[0031]

[0050] The frame rate for tracking can be adjusted (e.g., dynamically) based on one or more of a number of criteria including the speed of movement, the size of the window 416, and how far the cord 104 is from the camera (e.g., if Dad is farther away, a larger window 416 is selected and / or the image frame rate for tracking is increased).

[0032]

[0051] FIG. 5 illustrates an embodiment of scanning a code 104 over multiple frames 508. In some configurations, scanning a code (e.g., an identification code and / or a decoded code) can be computationally intensive. Identifying a code includes identifying the presence of a code in an image and / or the location of the code in an image. Decoding a code includes identifying what the code represents. For example, identifying a barcode includes identifying the presence of a barcode in a frame, and decoding a barcode includes interpreting a series of white and black lines in the barcode as representing a string of numbers. By distributing the scan over multiple frames, the computation can be distributed in time. FIG. 5 illustrates a first frame 508-1, a second frame 508-2, a third frame 508-3, and a fourth frame 508-4. In FIG. 5, the scan is performed over multiple frames. In the first frame 508-1, a first code 104-1 and a second code 104-2 are tracked. In the second frame 508-2, code scanning is performed only on the top half of the second frame 508-2. Thus, the first code 104-1 is identified and decoded, and the second code 104-2 is tracked without being decoded. In the third frame 508-3, the code 104 is tracked. In the fourth frame 508-4, code scanning is performed only on the bottom half of the fourth frame 508-4. Thus, the second code 104-2 is identified and decoded, while the first code 104-1 is tracked without being decoded. In some embodiments, the frame 508 is divided into quadrants, and discovery of new codes (e.g., scanning) is performed every fourth frame in each quadrant. In some embodiments, "attention" or "scene semantic" based decoding is used. In some configurations, this may mean that decoding may be limited to frames or portions of the observed scene that undergo dramatic visual changes, and / or that decoding may be limited to areas of special interest for a particular task (e.g., barcode labels, parts of a shelving unit, etc.).

[0033]

[0052] 6 shows a flow chart of one embodiment of a process 600 for tracking a code. The process 600 begins in step 604 by acquiring a number of frames from an image sensor, each of the frames including an image of a code, the number of frames including a first frame and a second frame, the second frame being acquired after the first frame. In step 608, a code is identified in the first frame. In step 612, features are extracted from the code. For example, extracted features 308 are identified from the code 104 of FIG. 3. In step 616, a filter is created. For example, a correlation filter 304 is created based on the extracted features 308 of FIG. 3.

[0034]

[0053] In some embodiments, detecting and tracking a code can mean tracking multiple codes, such as a bar code and a line of text. For example, in a retail store, a price tag can be tracked and in a second step, one or more codes on that price tag can be decoded, including one or more bar codes that identify a product, a series of numbers indicating a price, and / or a series of letters that describe the product name.

[0035]

[0054] In step 620, features are extracted from the second frame. For example, extracted features 408 are generated from image 412 of FIG. 4. In step 624, a response map is generated based on the features extracted from the second frame. For example, response map 404 is generated based on a convolution of extracted features 408 with correlation filter 304 of FIG. 4. In step 628, a location of the code is identified based on the response map. For example, areas of higher magnitude in the response map indicate likely locations of the code in the response map. Tracking the code as described in process 600 does not decode the code (e.g., to conserve computational resources).

[0036]

[0055] A graphic indicating the location of the code, such as a box or the outline of a box, can be overlaid on an image containing the code to provide the user with an augmented reality output indicating the location of the code. The graphic can change, such as changing color, to indicate whether the code has been decoded. For example, if a code is identified as being present but the camera was too far from the code to decode it, a red outline around the code may be displayed to the user. The user can then move the camera closer to the code. Once the code is decoded by an application (e.g., running on the device or web-based), the graphic changes to a green box, indicating to the user that the code has been successfully decoded.

[0037]

[0056] By tracking the code, the subarea of ​​a subsequent frame can be predicted based on the location of the code in the second frame, so that when a scan is performed in the subsequent frame, the scan of the code can be matched up with the location of the code.

[0038]

[0057] In some embodiments, a system for decoding multiple optical codes includes a mobile data collector enabled with web browser software and a display, a camera module, two or more optical codes disposed with at least one object, and / or one or more processors that receive identification information of two or more optical codes in one or more images captured by the camera module, decode the one or more optical codes, and / or visualize the decoded codes on a display of the mobile data collector. The system can include a mobile data collector enabled with web browser software and a display, a camera module, two or more optical codes disposed with at least one object, and / or one or more processors that detect the presence of two or more optical codes in images captured by the camera module, decode the two or more detected optical codes in the images, visualize locations of the detected codes on a display of the mobile data collector, and / or visualize the decoded codes on a display of the mobile data collector. The system may include a mobile data collector enabled with a display and a camera module, a plurality of optical codes disposed with at least one object, and / or one or more processors for detecting a presence and location of the plurality of optical codes in a first image captured by the camera module, decoding a subset of the detected optical codes from the first image, tracking the location of the plurality of optical codes from the first image to a second image, detecting a presence of the plurality of optical codes in a second image captured by the camera module, and / or decoding a subset of the detected optical codes from the second image, the subset consisting of codes not decoded from the first image.

[0039]

[0058] Code Mapping

[0059] Code mapping can be used to enable a user (e.g., a store employee) to capture the location of a code (e.g., SKU / product) within an environment (e.g., on a store shelf). Identifying and tracking codes can be helpful in enabling the creation of a virtual diagram of a physical structure. For example, a virtual diagram of a shelving unit can be used to help answer one or more of the following questions: Where on the shelf is the product, on which shelf, and / or at what height? Are the products in the right location (e.g., does the product placement match the planogram)? What does a particular shelf look like visually? What was the condition of the shelf two weeks ago? Is the product priced correctly? Do the products correspond to the label? Is there a low number of products on the shelf (e.g., if the shelf is being restocked)?

[0040]

[0060] In some embodiments, cloud services for storage, APIs, dashboards, etc. are used to allow multiple users to contribute and / or search data. For example, in a retail setting, multiple employees with multiple mobile devices may contribute to data capture and / or different parties may view portions of the data in raw and / or aggregated versions. Cloud-based applications can also enable faster software development cycles.

[0041]

[0061] In some situations, the mapping is limited to a particular structure. For example, in a retail environment, a shelf is imaged as a two-dimensional structure, and multiple barcodes are mapped to lie on a vertically oriented plane (the normal of the vertical plane points to the horizontal). Products on the shelf unit can be identified by the barcodes on the shelf unit. In some embodiments, products on the shelf unit are identified by visual product recognition.

[0042]

[0062] Data can be stored remotely (e.g., in the cloud). Data can be agnostic from the capture and / or display devices. For example, different mobile capture devices (e.g., different tablets and / or smartphones can be used) and captured data display can be in a mobile app or web-based dashboard. Capture runs can be per SKU, per shelf unit, and / or per store and can be flexibly combined from different capture devices.

[0043]

[0063] In some configurations, code mapping can be used for planogram compliance (e.g., comparing SKU locations to planned locations), planogram analysis (e.g., for sales numbers), in-store navigation to products, and / or AR marketing (e.g., instant sales when a customer is within a specified distance of a product).

[0044]

[0064] In certain embodiments, creating a code map includes identifying multiple SKUs (e.g., by reading shelf barcodes and / or by product recognition), segmenting SKU areas (e.g., areas of one type of product; done by image segmentation), ascertaining two-dimensional layouts (e.g., which SKUs are next to other SKUs on a shelf unit and / or aisle), calculating inter-barcode distances (e.g., absolute distance in centimeters or relative distances), capturing the three-dimensional layout of the barcodes (e.g., relationships between multiple aisles), and / or understanding the concept of a shelf (e.g., vertically divided units).

[0045]

[0065] SKUs can be identified by scanning the barcode on the shelf. To identify SKUs by product recognition, a classifier is run on the image that can label the product and / or simultaneously segment the contour. The contour can be a rectangular or pixel-wise image segmentation (e.g., using a Convolutional Neural Network (CNN) based classifier). The classifier can be pre-trained on a general data corpus (e.g., ImageNet) and fine-tuned / adapted for the product recognition use case. Preferably, new products can be added with some examples (e.g., one-shot learning). Examples of CNN-based object detection frameworks are YOLO, R-CNN, SSD, etc., but some can also be custom built.

[0046]

[0066] FIG. 7 is an example of segmenting products on a shelf from an image. FIG. 7 shows a frame 704 that is an image of a product 708 placed on a shelf 712. Segmentation of the SKU area can be performed by image segmentation. If the product 708 can be recognized directly from the image, a classifier can be used to segment the SKU area. If the product 708 is recognized via a barcode 716 on the shelf 712, this can be done based on the image content, taking into account the barcode location. An exemplary approach includes checking whether the product 708 corresponding to the barcode 716 is above or below the barcode 716. Determining whether the barcode 716 is above or below (or to the side of) the product 708 can be solved by trained binary classifiers, ad-hoc rules, and / or human input. Image analysis is then used to expand areas (e.g., above or below the code) to the left or right based on similar image content. In the frame 704 of FIG. 7, the product 708 corresponding to the barcode 716 is above the barcode 716. Similar image content can be measured using, for example, interest point matching, (cross) correlation, color analysis, etc. Segmentation can be based on the assumption that undistorted images are used and / or similar products are horizontally aligned with shelf rows. Segment 720 (dashed line) of product 708 indicates product 708 that corresponds to barcode 716.

[0047]

[0067] FIG. 8 is an embodiment of a shelf diagram 800. The shelf diagram 800 provides information about the two-dimensional layout of SKUs (e.g., which SKUs are next to another SKU). In this embodiment, the SKUs are recognized by their barcodes. If only one barcode is visible at a time, the user scans the barcodes in a defined order (e.g., a left-to-right sequence for each shelf; starting at the top of the shelf and moving down row by row). However, this only provides the layout, not the distance (relative or actual). As previously described in matrix scanning, if more than one barcode is visible, the barcodes are mapped relative to each other. For example, starting from the top left of the shelf, the previously referenced barcodes are kept in the current field of view, and the newly recognized barcode can be referenced (e.g., left, right, up, down) with respect to one or more known barcodes. Some assumptions can simplify the mapping. For example, in some configurations, it can be assumed that the barcode capture view is aligned with the shelf rows and there are no gaps. Approximate relative distances can be calculated using the spacing between the barcodes in the image.

[0048]

[0068] In cases where only one barcode is visible in the capture view to give the relative distance between the barcodes, some embodiments may start with the capture of one or more overhead images of the shelving unit. In these overhead images, the barcodes are not (or cannot be) decoded (e.g., they are too small to identify the barcode location area). The overhead images are rectified, dewarped, and aligned with the shelf rows. If there are multiple fields of view with different barcodes, they are stitched together or at least referenced to each other. Then, the individual barcodes are captured, including the image content surrounding each individual barcode. The image content can be utilized to register the individual barcode images to the overhead image, with or without relying on the identified barcode area. Matching can be done using image interest points, for example, using RANSAC (random sample consensus) for homography or other image analysis for estimation. In some embodiments, registering the individual barcode images to the overhead image includes identifying a horizontal line corresponding to the shelf and using the horizontal line for horizontal alignment of the images and / or compositing of the images. This approach has the advantage of capturing the (relative) distance between barcodes with the help of an overhead image.

[0049]

[0069] In some embodiments, the absolute distance between the barcodes is calculated (e.g., in cm). For example, matrix scanning can be used to track two or more barcodes simultaneously, with or without the barcodes being decoded. The algorithm can include the assumption that the shelf is planar (i.e., it is a two-dimensional problem where the codes are located in the same plane), which can simplify some calculations. In a first option, the scale can be determined by a known width of a fiducial in the image. For example, if the height and / or width of the barcode is known, such as 1.2 cm, its scan can be used as a reference to convert relative measurements to absolute measurements. The same type of barcode can appear in multiple images providing redundancy. In a second option, a known calibrated camera and a moving device (e.g., using SLAM (simultaneous localization and mapping)) can be used to determine the scale. In a third option, a stereoscopic or depth camera can be used. The above three options are not mutually exclusive.

[0050]

[0070] A 3D model of the store can be generated. The 3D model can be created by combining multiple 2D models (e.g., shelving units) and / or by using a single device by implementing a SLAM algorithm or a structure from motion (SfM) pipeline (Apple's ARKit or Google's ARCore provide such reconstruction capabilities on their respective mobile platforms). Barcodes can be decoded with a magnified scan and localized within the 3D store model. Another option is to use multiple cameras (stereo) and / or depth cameras. In a further option, a 2D model of the shelves is created and the user identifies the location of the shelves in the store (e.g., using a blueprint of the store layout). In another embodiment, the store layout model is derived from an existing CAD model, resulting for example from first planning the retail store surface.

[0051]

[0071] The concept of a shelf (e.g., a vertical division unit). From an application point of view, a shelf (or a module or a shelf unit) is a descriptive unit between the store and the products. The products are placed on the shelves, and every shelf in the store has a unique identifier. It may therefore be useful to record the physical boundaries of the shelves in a digital representation. If the shelves are mapped one after the other, in the 2D case, the capture of the shelf identifiers can be done by manual entry of the identifiers, or by manual capture of dedicated barcodes (or different code types) placed on the shelves. Although automatic entry may be preferred, manual entry can be used if the shelves do not have identifiers. In some embodiments, the shelf identity can be inferred from the camera / device position. For example, in some embodiments, the phone position can be determined using WiFi fingerprinting, Bluetooth beacons, GPS, SLAM (simultaneous localization and mapping), or IMU tracking. In a workflow, this is done, for example, before capturing the SKU map of that shelf. If the store is directly reconstructed in 3D, the shelf segmentation (vertical division) can be performed automatically by image analysis searching for vertical division lines within a certain (parameterizable) distance from each other.

[0052]

[0072] The application used to scan the shelves can be configured in a particular way, e.g., each SKU is scanned in sequence. For example, a store associate can collect SKU data in an S-shaped pattern going from top left to bottom right (for example), switching direction for each individual shelf to minimize travel time. Data (e.g., for each shelf) is collected and sent to the backend. The data for each shelf can include: Store ID: store identifier; Shelf ID: shelf identifier; Time: acquisition start & end timestamp; For each SKU: EAN / barcode, location (shelf row, product location), acquisition timestamp, price (read price, correct price), and / or product image; and / or Smartphone identifier: to distinguish uploads from different mobile phones.

[0053]

[0073] The data for a particular product is arranged in a vector format (e.g., an n x 1 matrix). Table I below provides a sample vector for data mapping. The vector includes a code and the code's relative position on the shelf map (e.g., the vector is the "Value" column). [Table 1]

[0054]

[0074] 9 shows a flow chart of one embodiment of a process 900 for mapping objects on a shelving unit (e.g., creating a digital map). The process 900 begins in step 904 with receiving a number of images. For example, the images are captured by a camera from a mobile device. 908 In step (a), a first item code is identified in the plurality of images. For example, the first item code is a barcode identifying a first SKU, and identifying the first item code includes decoding the first item code. 912 In the method, a second item code is identified in the plurality of images, for example, the second item code is a barcode identifying a second SKU, and identifying the second item code includes decoding the second item code.

[0055]

[0075] In step 916, the relative distance and orientation between the first and second codes is calculated. The relative distance and orientation can be based on placing the first and second item codes on a relative coordinate system (e.g., the coordinate system shown in FIG. 8; e.g., the second barcode is 4 units to the right and 3 units below the first barcode). In some embodiments, the absolute distance is measured (e.g., the x position of the second barcode is equal to the x position of the first barcode plus 24 centimeters, and the y position of the second barcode is equal to the y position of the first barcode).

[0056]

[0076] In step 920, the relative distance and direction between the first and second item codes is calibrated to the shelf diagram. For example, multiple codes are combined with respect to the shelf diagram shown in Figure 8. The relative distance and direction between the first and second item codes is used to specify the location of the first and second item codes on the shelf diagram (e.g., if the first item code is selected as the origin of the shelf unit, the first item code has coordinates of 0,0 and the second item code has coordinates of 4,-3).

[0057]

[0077] In step 924, a first vector is generated that includes the first item code and the relative location of the first item code with respect to the shelf diagram. For example, Table I provides a vector with the barcode ID and the x / y coordinates of the barcode for shelf 21A7. Similarly, in step 928, a second vector is generated that includes the second item code and the relative location of the second item code with respect to the shelf diagram.

[0058]

[0078] In some embodiments, a system for mapping the location of an object on a display includes a mobile data collector that reads an optical code, a label having an optical code that is positioned with the object on the display, and / or one or more processors that receive identification information of two or more labels, calculate the relative distance and direction between the two or more labels, and / or place the locations of the two or more labels on a map.

[0059]

[0079] Shelf Visualization

[0080] Shelf visualization can be used to allow users (store employees, managers, etc.) to visualize the current and / or previous state of retail store shelves and / or display additional relevant information (e.g., via overlays or additional windows in a Graphical User Interface (GUI)). Some embodiments are limited to planar shelves, while other embodiments include products at almost any type of point of sale (PoS) in a store. Shelf visualization can enable remote visual inspection of shelves, head to head shopping experiences, street view style strolls through a retail store from the couch, and / or augmented or virtual reality applications.

[0060]

[0081] FIG. 10 is an example of blending two images of a shelf. FIG. 10 shows a composite image 1000 of products 1008 on a shelf 1012. The composite image 1000 may appear somewhat blurry because the composite image 1000 may be formed by two or more images. Shelf visualization may be an image stitching problem. Although panoramic photographs are created using image stitching, the constraints of shelf visualization are different from traditional panoramic image stitching. For example, shelves (e.g., shelving units) may be close, but do not fit into one camera view. Shelves are captured from multiple different viewpoints, and there may be significant camera shifts (relative to objects) between the captured images (e.g., parallax). However, it can be assumed that there is no motion in the scene between the captured views. Visualization updates may come from multiple capture devices and from multiple moments in time, even from a single SKU capture instance. Image blending often includes exposure compensation to make the visualization look appealing. As the visualization is updated, it is possible to sift through the images of the shelf in time.

[0061]

[0082] In some configurations, analog-image stitching includes the steps of detecting keypoints and extracting invariant descriptors in the images, matching the descriptors between the two images, estimating a homography matrix using matching image feature locations in both images using RANSAC (as an example of some similar techniques), and combining the two images only if there are many matching features (e.g., using a threshold), applying a warping transformation using the resulting homography matrix, and blending the images (e.g., resizing, compensating for exposure, finding seams).

[0062]

[0083] In some configurations, the problem is formulated as a joint optimization problem (e.g., rather than a pair-wise image matching problem). This can be an SfM problem. This can be done based on points, but also other feature types such as lines (e.g., parts of a shelving unit). In some embodiments, the homography can be constrained to improve fault tolerance, for example, by assuming a parallel view from a near frontal perspective. In some embodiments, this problem can be addressed by formulating a global optimization problem. Considering the relationship between multiple images, jointly trying to optimize / derive a rough estimate of the position where the images were taken (derived visually and by correlating timestamps), can reduce errors. In some embodiments, high-quality image recording can be supported by low-level image descriptors, or even planar estimates of the performed movements taken at much higher frequencies (e.g., estimated from gyros or through the images; related to tracking), allowing the images to be better correlated and the "stitched" panoramic image to be optimized. In some embodiments, a sequence of images (or videos) is recorded to reduce parallax between captured frames and allow rendering with minimal visual artifacts. To improve stability, sensors (IMU / accelerometers) of the captured mobile device can be used (and / or sensor fusion with cue-based images).

[0063]

[0084] While capturing the SKUs, the user can also capture close-up images of the SKUs where the barcode is recognizable. These close-up images can be incorporated into the overhead image so that high-resolution images of the product and / or barcode can be enlarged and / or displayed. The stitching algorithm takes into account large parallax and scale changes. The image bending is done carefully to minimize visual artifacts. In some embodiments, close-up images are available for all SKUs and overview images are used to reference the close-up images to each other and fill in content between the close-up images, if necessary. In some embodiments, the style of the close-up images is matched with the overhead image to minimize visual artifacts. This can include adapting contrast, brightness, saturation values, or synthetic exposure compensation (or similar techniques). In some embodiments, blending is applied for a smooth transition between the overhead image and the new close-up image, and content from both images is overlaid. In some embodiments, a "seam carving" algorithm is used for optimal seam location detection.

[0064]

[0085] 11 is an embodiment of a shelf diagram 1104 with a visual image 1108 of the shelf. Product information can be integrated into the visualization. For example, the visualization can be searchable by product and the searched products can be highlighted in the image visualization, the visualization can include overlays with product information such as price, stock levels, allergens, etc., and specific products can be highlighted in the image (e.g., products on promotion or that should soon be changed to a promotion, or products that have an incorrect price or incorrect location in the parallelogram).

[0065]

[0086] FIG. 12 shows a flow chart of an embodiment of a process 1200 for creating a visual representation of an object on a display from a plurality of images. The process 1200 begins by acquiring a plurality of images, the plurality of images including a first image and a second image, at step 1204. At step 1208, a code is detected in the first image. At step 1212, a code is detected in the second image. The code is decoded to receive an identification of the object in the first image (step 1216). A set of features is identified in the first image (step 1220). At step 1224, a set of features is identified in the second image. At step 1228, a position of the first image is calculated relative to a position of the second image based on comparing a position of the set of features in the first image to a position of the set of features in the second image. At step 1232, the first image is blended with the second image based on calculating the first image relative to the second image to create a blended image. The blended image is stored at step 1236 along with associated data identifying the object. In some embodiments, the metadata of the combined image includes a link or reference to a database that links the combined image with information identifying the object.

[0066]

[0087] In some embodiments, the system includes a mobile data collector that reads the optical code and acquires images of the object on the display, a label having the optical code juxtaposed with the object on the display, and / or one or more processors that receive an identification of at least one object in the one or more images, calculate a position of the two or more images relative to each other, and / or calculate a panoramic image representation from the one or more images.

[0067]

[0088] Simultaneous use of barcode decoding, OCR and / or visual shelf

[0089] In some embodiments, the combined tools may enable an associated store to perform several tasks simultaneously. For example, a barcode can be scanned, a price label can be decoded using optical character recognition (OCR), a price can be associated with the product identified from scanning the barcode (e.g., by referencing a database), and the price associated with the product can be compared to the price label to verify a match. While price verification is being performed, images can be used to look for out-of-stock situations. Thus, work tasks can be performed in one walk-through by an associated person or robot.

[0068]

[0090] Some decoding tasks, such as using OCR, can be computationally intensive. By decoding the code once and then tracing the code without decoding the code, computational resources can be saved.

[0069]

[0091] A. Example of a user in a retail environment

[0092] i. Consultant - Retail

[0093] Susan is a store associate about 30 years old. The store is a mid-sized grocery chain. Susan does not have a college degree. Although Susan knows how to use her mobile device, she would not be considered tech-savvy. Susan has changed retail employers three times in the past two years. She works different shifts depending on the week. As part of her job, her boss asks her to periodically walk through the store and perform specific data collection tasks such as verifying price labels, recording shelf gaps, or verifying planogram compliance. Susan has access to a Zebra TC52 device. Susan's greatest concern is performing a walk-through of the store as quickly and accurately as possible.

[0070]

[0094] Susan would like to run a session on her phone to quickly scan 1000's of SKUs for collecting and / or verifying price label information (PLI). This is a daily challenge for her and speed and accuracy are important. She would also like to create a digital map of the SKUs on the shelf as she scans them for price label verification. Susan would prefer to run the session on her phone to collect data and then upload that data to a server in the cloud.

[0071]

[0095] In this example, Susan can scan the SKUs, see the individual date codes, and enter the code that will expire soonest into the application. In other configurations, the individual date codes are decoded using an image of the individual date codes. In another variation, Susan enters all date codes found into the app and also specifies the quantity of product associated with each date code. Susan can also identify a gap in a shelf while working on another task. She can walk up to the gap and scan the SKU associated with the gap. In the user interface, she can optionally specify the number of products remaining on the shelf.

[0072]

[0096] ii. Paul Store Manager

[0097] Paul is about 45 years old, married, and has a couple of children. He typically works from 8AM to 6PM as a store manager for a large grocery chain. Paul has some college and is more of a desktop user than a mobile device user. He spends most of his time at his desk in the administrative office behind his large computer screen. Paul has worked for the chain for almost 20 years. He prefers to do things the way they have always been done. Paul maintains a schedule for daily store walk-throughs and directly supervises store employees. Paul uses his iPhone® 8 at work.

[0073]

[0098] The web based application allows Paul to create task lists listing compliance and operational issues, e.g. labels to be reprinted and replaced, gaps filled, etc., confirm on the portal that the issues have been resolved, review reports on operational tasks / data within the store, time frame, store association for performing the task, type of event (PL non-compliance, gap scan, etc.), inspect summary statistics on the above, show enterprise trends over a specified time frame / timeline, and / or zoom to the ShelfView and review the latest visual data for the shelf.

[0074]

[0099] iii. Camilla – Nationwide store operations

[0100] Camilla is about 50 years old, married, and has two grown children. She generally works from 9am to 7pm and is responsible for store operations for 200 stores. She has an MBA degree. Camilla moves fluently between her mobile, iPad, and Windows laptop. She is used to monitoring operations from anywhere, anytime. Camilla has only been in her current role for two months, but has already encountered resistance from the IT department when she has tried to push new software tools. Her biggest problem is not having real-time visibility of tasks taking place in the stores, and therefore not knowing if the company is compliant with relevant regulations.

[0075]

[0101] In the web-based application, Camilla can review reports on operational tasks / data within the store, filter the reports to a subset of stores, time frame, store association for which the task is being performed, type of event (PL non-compliance, gabb scan, etc.), inspect summary statistics on the above, view enterprise trends over a specified time frame / timeline, and / or zoom into the ShelfView and review recent visual data for the shelves.

[0076]

[0102] B. Typical System Components

[0103] The web-based application can be configured for multi-tenancy, where multiple retailers are hosted on the same system; multiple user roles with different access levels such as store associate, store management, enterprise management, etc.; multiple data elements that can include one or more of date and time, session identifiers (user, date, store), decoded optical code (Uniform Product Code (UPC), International Article Number (EAN)), shelf / location identifier placeholder, high resolution, raw image of detected barcode or other area to be decoded (OCR), full field of view low resolution image, and / or other fields (e.g., as described in the following applications); individual data fields can be eligible for individual deletion schedules (storage needs and privacy concerns), and customers can have the ability to schedule data deletion on numerous schedules, such as a) daily, b) after 48 hours, c) weekly, etc., and / or deletion can be scheduled for a date and specified period after data collection.

[0077]

[0104] The web-based application can be configured to provide data reports including filtered data downloads in a particular format, such as CSV, XLS, or PDF, and the filtered data reports can be made available via a web API.

[0078]

[0105] C. Exemplary Applications and Workflows

[0106] The application can have a single SKU mode. The single SKU mode can be configured for walk-up gap scanning / PLV SKU scanning. In this mode, the store associate walks up to the gap in the shelf and scans the SKU code. The user interface guides the operator to ensure that the product display location is within the camera's field of view. The system uploads a high-resolution image of the label and a JPG image of the surrounding area. The user has the option to enter the following data into the manual interface: stock remaining, earliest date code, and / or general text input (free comment). In the backend, the data is processed as follows: if not correct, PLV is performed, added to PL event list, added to gap event list, if available, other information (stock level, date code) is logged, and / or visual information of the shelf location is included in the shelf panorama image.

[0079]

[0107] Single SKU mode can be configured for "walk-up data entry" SKU scanning. In this mode, store associates can walk up to the price tag or SKU and scan the code. The user interface can guide the associated store so that the product display location is within the camera's field of view. The system uploads a high-resolution image of the label and a JPG image of the surrounding area. The user has the option to enter the following data in the manual interface: out of stock, remaining stock (including out of stock and "low stock"), earliest date code, and / or general text entry (free comment). In the backend, the data is processed as follows: if incorrect, PLV is performed and added to PL event list, if low / 0 stock, added to gap event list, and / or record other information (stock level, date code) if available.

[0080]

[0108] The single SKU mode can be configured for a batch mode such as systematic shelf scanning. In this mode, the store moves from left to right, top to bottom and systematically scans all products on a given shelf. The user interface can be configured to guide the associated store so that the product's display location is within the camera's field of view. The system uploads a high-resolution image of the label and a JPG image of the surrounding area for each scan. The user (e.g., store associate) has the option to enter the following data into the manual interface: remaining stock quantity including "out of stock", earliest date code, and / or general text input (free comment). Additionally, the user has the option to do the following: scan or enter shelf / module identifier (top left corner), erase the scan to correct an error, and / or resume data collection for the same shelf / module by starting the scan from any SKU (data following that SKU will be overwritten). Although left to right scanning is given as an example, other scanning patterns (e.g., right to left, bottom to top to right, etc.) can also be used. In some embodiments, the scanning pattern is determined by the device orientation. For example, a phone held vertically can be used to scan downwards so that there are more SKUs in the camera's field of view, while a phone held horizontally can scan right to left so that there is a higher chance of more than one SKU in the field of view. In the backend, the data is processed as follows: if incorrect, a PLV is performed and added to a PL event list, if inventory is low or out of stock, added to a Gap event list, or other information if available (stock levels, date codes) and / or shelf position visual information is used to reconstruct a visual state / shelf panorama image.

[0081]

[0109] D. Exemplary Performance Metrics

[0110] In some configurations, the application has the following performance metrics and / or tolerances: Speed: A moderately skilled operator spends 500 ms or less per hour to complete the scan of a single code. SKU Mapping: The distance between SKUs is within + / - 10% of the true distance for 99% of the SKUs measured. SKU Mapping and Visualization: There are no obvious visual artifacts from the stitching process.

[0082]

[0111] E. Image Upload and Processing Examples

[0112] In some configurations, a high resolution raw image of the detected code (e.g., barcode, price tag, etc.) is uploaded and / or a JPEG quality image of the entire image frame showing the label, the display area with the inventory, and / or adjacent labels. The barcode viewfinder is positioned to provide reasonable assurance that the inventory location is within the field of view of the camera.

[0083]

[0113] For on-device PLV, a single SKU data capture PLV can be run directly on the device and the results are shown to the user in an augmented reality (AR) overlay. In some embodiments, the results are presented to the user in no more than 500 ms.

[0084]

[0114] In some configurations of back-end PLV processing, false positives are below 10% (10% of errors detected). Incorrect price labels from a session / day can be accessible along with the image data for quality review by a human operator. The data can be presented to an online operator so that the operator can quickly review image by image and flag / correct any incorrectly decoded labels. After quality control, the data can be added to the PLV task list.

[0085]

[0115] In some configurations of SKU mapping and visualization, a map is constructed by maintaining the order of SKUs as the user scans one SKU after another. Data collection can include distances between SKUs while constructing the map. The map can include vertical distances between shelves.

[0086]

[0116] In some configurations using manual status entry, a user can count the stock levels and enter the information via a user interface (e.g., during a batch scan or individual SKU scan). The interface on the mobile device can be configurable by the enterprise user (e.g., to show only data entry options to be performed by store association). For example, the inventory interface can have a number input field for the store to associate to enter a number of products. The out-of-stock interface can include an out-of-stock button, a low stock button, a refill button, and / or a specific refill / reorder (number input field). The date code interface can include an oldest date code (date input field), multiple date codes, and / or a stock number (e.g., a list of numbers and / or dates).

[0087]

[0117] F. Exemplary Portal Application

[0118] In some configurations of SKU, task, event, and / or date code reports, a user can request reports for all scan events in a particular time range, including scans that did not result in violations; shelf images in a particular time range; and / or specific SKUs that show violations in a particular time range. A user can request reports for SKUs and / or events filtered by one or more of the following criteria: tasks collected in a particular calendar range; tasks collected in a particular data collection session; tasks collected by a particular user; tasks in a particular product category; tasks of a particular type (e.g., date code, out of stock, price label correction); resolved violations identified by user input (e.g., date code violation, out of stock, price label correction); and / or resolved violations identified by shelf scan (e.g., date code violation, out of stock, price label correction). A user may request a report on SKUs with date codes (e.g., a date code report) filtered by one or more of the following criteria: date codes within a specific calendar range; date codes within a specific product category; date codes collected in a time window; date code violations (date codes passed); and / or date code violations resolved.

[0088]

[0119] In some configurations of shelf visualization, a backend system can stitch a representation of the shelf (shelf panorama) from individual SKU images (batch mode / SKU mapping). The visualization can be "clickable." For example, a user can click on an image or section of a SKU, and a metadata pop-up includes the date and time the image was taken, any user input from the time the image was taken, and / or status information. In some embodiments, the visualization is "zoomable," allowing a user to zoom in on a particular shelf location or SKU (e.g., enlarge the view of a particular shelf location or SKU). The visualization can be searchable, allowing a user to enter a SKU number or product description and "fly" to that product's location / image. As new scans / images of the SKU become available, the old images in the panorama are replaced with the new scans / images.

[0089]

[0120] In some configurations, the dashboard or portal can include summary statistics and / or a timeline, which can include average time between scans (average across all SKUs), PLV violations by day / week, out-of-stock events by day of the week, total scans by day / week, and / or average time between scans (average across all SKUs).

[0090]

[0121] In some configurations of the backend, the database can store data from the mobile device (e.g., the data listed above including historical data) and can be future proof for new structured data (e.g., inventory counts). In some embodiments, the backend can be used to access a product database, e.g., a public database, a web database, a retailer's database, or a manufacturer's database. Metadata can be added to every shelf starting with shelf category (e.g., beverages, cereals, etc.). Filtering / query can also be allowed by parameters (e.g., shelf ID, store ID, user ID, etc.). In some embodiments, a backend process can be used to determine out-of-stock situations from collected images. For security and authentication, the application (e.g., API and / or database) can be multi-tenant capable (3-10 pilot customers), where data uploaded by one customer is only accessible / queriable by the one customer. A unique configuration ID can be used to differentiate uploads. Also, in some embodiments, company-sensitive information (e.g., price, product location, etc.) is not stored in the cloud database (e.g., to ensure maximum privacy and data security).

[0091]

[0122] Some configurations of APIs or other interfaces include internal APIs for transmitting data to and from mobile devices and / or the cloud. For example, in-store data collected by a mobile device can be uploaded to the cloud and / or a product database and downloaded from the cloud by the mobile device, with the possibility to select only certain columns (e.g., price and SKU columns). External APIs for customers to query the backend database can include API endpoints for querying the scan of one shelf in a particular store (e.g., this endpoint returns only the most recent scan, not historical data; / {store} / {shelf}), API endpoints to update the product database, the possibility for the backend database to export in-store data to CSV (comma separated values) format, secure APIs, allow customers to export in CSV format, and / or allow querying by historical data (e.g., by time period).

[0092]

[0123] The systems and / or methods disclosed herein can be used to establish and / or maintain a visual catalog, SKU-specific and SKU-independent out-of-stock detection, price label verification (PLV), simultaneous mapping and PLV, and / or OCR, and / or barcode decoding in a web browser. Images can be captured in a number of ways, including simultaneous image capture by mobile cameras on the shelf, drones, robots, fixed shelf cameras, and mobile barcode scanning.

[0093]

[0124] Some disclosed embodiments relate to methods and / or systems for operating an information system that aggregates pricing information from retail stores. More specifically, but without limitation, some embodiments relate to acquiring imaging data of objects using imaging tools, acquiring pricing information from signage using imaging tools, acquiring product inventory information using imaging tools, acquiring information from receipts using imaging tools, using predictive algorithms to reconstruct pricing information, incorporating available pricing information from publicly available sources, incorporating pricing information from issuing recommendations on where to purchase products, presenting the information to one or more third parties, issuing recommendations on which products to purchase based on available choices, operating a retail store, and adjusting prices based on information regarding competitive prices. The system or method may include scanning products to image price labels on retail displays or shelves, using drones to scan products on shelves, using robots to scan products on shelves, scanning products and / or prices on displays using consumer devices, scanning and / or parsing shopping receipts to record pricing of products purchased, parsing digital receipts to record pricing of products purchased, and / or scanning and interpreting signage to infer pricing and special offers.

[0094]

[0125] In some embodiments, a system for detecting a state of a retail display includes a mobile data collector enabled with web browser software, a camera module, a label having an optical code disposed with an object on the display, and / or one or more processors that receive the identification of at least one object in one or more images captured by the camera module, receive a price on the display of the at least one object in one or more images captured by the camera module, receive an intended price of the at least one object from a database, and / or compare the price on the display to the intended price and report the results. The system can include a mobile data collector enabled with web browser software, a camera module, a label having an optical code disposed with an object on the display, and / or one or more processors that receive an identification of at least one object in one or more images captured by the camera module, decode information about the object on the display from the one or more images captured by the camera module, and / or calculate a map of the at least one object on the display from the one or more images captured by the camera module. The system may include a mobile data collector, a camera module, a label having an optical code disposed with an object on a display, and / or one or more processors that receive an identification of at least one object in one or more images captured by the camera module, receive image data regarding the display area from the one or more images captured by the camera module, and / or detect the presence of at least one object in the display area from the one or more images captured by the camera module.The system may include a mobile data collector, a camera module, a label having an optical code disposed with an object on a display, and / or one or more processors that receive identification information of at least one object in one or more images captured by the camera module, receive image data of the display area from the one or more images captured by the camera module, detect the presence of at least one object in the display area from the one or more images captured by the camera module, and / or store image data of the one or more objects together with identification information of the one or more objects in a database.

[0095]

[0126] Ultra-high speed scanning

[0127] Mobile devices (e.g., smartphones; tablets) can be used for ultra-fast scanning of optical codes. In some configurations, the mobile device uses a wide-angle lens (e.g., fixed focus - for close focus and better depth of field with no AF delay), high frames per second (e.g., for speed and / or redundancy), very short exposure times (e.g., to avoid motion blur), and a torch (e.g., for additional illumination and / or aiming).

[0096]

[0128] Figure 13 is an embodiment of a mobile device 1300 that rapidly scans an optical pattern 1304. The optical pattern 1304 can be the code 104 of Figure 1. In some embodiments, ultra-fast scanning or fast scanning refers to acquiring an image, detecting an optical code, and decoding the optical code in less than 0.25 seconds and / or more than 0.002 seconds.

[0097]

[0129] The mobile device 1300 comprises a camera, a display, and one or more processors. The camera comprises an image sensor and a lens. The lens is configured to focus light onto the image sensor. The one or more processors are configured to obtain a plurality of images, analyze the plurality of images to detect an optical pattern in the plurality of images, decode the optical pattern to obtain a data payload, and store the data payload in a memory device of the mobile device.

[0098]

[0130] A normal lens is a lens with a focal length approximately equal to the diagonal measurement of the image sensor. A normal lens has an angle of view of approximately 58° (e.g., measured diagonally), similar to the angle of view of the human eye. A wide-angle lens has a larger angle of view than a normal lens. The lens of the camera of the mobile device 1300 is a wide-angle lens. In some embodiments, the lens is an ultra-wide-angle lens (e.g., having an angle of view of 100, 120 degrees or more, or 125 degrees or more, and / or 128, 130, 150 degrees or less, or 180 degrees or less). For example, the lens has an angle of view of 128 degrees. The greater the depth of field of the camera, the greater the distance from the mobile device 1300 to the optical pattern 1304 may vary with the set focus. In some configurations, a camera with a focal length of 13 mm (120 degree angle of view), an aperture of 2.4, and a sensor size of 1 / 3.6 inches is used.

[0099]

[0131] The camera lens is set to a fixed focus during the acquisition of the images. The lens set to a fixed focus may be a prime lens, or the lens may be a variable focus lens with the autofocus function disabled (e.g., the focus is set to a certain predefined value between 0 and 1). A prime lens does not have a variable focus. The camera lens is set to a fixed focus during the image acquisition so that the application for decoding the optical pattern does not wait for the focus algorithm to be executed. Autofocus may cause scanning delays. However, some fixed focus settings have a small depth of field. Using a wider angle lens or a wider zoom setting can increase the depth of field.

[0100]

[0132] The exposure time for each of the multiple images may be 1 / 500th of a second or less. The exposure time is short to reduce motion blur. The ISO (sensitivity of the camera's sensor; sometimes called film speed or gain) can be set manually (e.g., auto exposure is turned off) or controlled (e.g., adjusted) by the camera. For certain camera models, the inventors have found that an exposure of 1 / 750th of a second is particularly beneficial to provide good exposure of the optical pattern 1304 under various conditions while using a light on a mobile device (e.g., a "torch" or "flashlight"). Auto-metering often uses an 18% gray standard. Metering can be stopped at one, two, three, or more stops to detect and decode the optical pattern 1304. The camera can automatically adjust the ISO depending on the amount of light available. In some embodiments, the exposure is 1 / 500 or more and / or less than the maximum shutter speed of the camera (e.g., 1 / 2000 or 1 / 8000). The choice of exposure can be important, as some phone models do not like to expose the noisiness of the image and therefore limit gain adjustment. By selectively choosing the exposure (e.g., 1 / 750 sec) and / or using a torch, the images can be properly exposed across a range of working conditions. In some embodiments, the torch is activated during the capture of the images, regardless of the camera's exposure measurement.

[0101]

[0133] The multiple images are captured at the frame rate of the camera. The frame rate is greater than 30 frames per second (fps). In some embodiments, the frame rate is greater than 30, 60, 90 fps and / or greater than or equal to 120, 150, 180, or 240 fps. For example, the frame rate is set to 120 fps. At a high frame rate, many images of the optical code are captured. If the optical code is unreadable in one frame (e.g., due to glare caused by a torch on the plastic cover of the optical pattern 1304), there are other frames with the optical code that can be used to decode the optical code (i.e., having more image frames that include the optical pattern 1304 provides redundancy).

[0102]

[0134] Acquiring multiple images at a high frame rate and processing multiple images can be computationally intensive. To reduce computational intensity, a restricted area of ​​each image is scanned (e.g., processed) instead of the entire image. The restricted area can be used based on the use case. For example, the restricted area is in the center of the camera frame, has a height equal to 1 / 3 of the height of the camera image frame, and a width equal to the width of the camera image frame. In some embodiments, the restricted area is in the center of the frame, has a height equal to or less than 60%, 50%, 40%, or 33%, has a height equal to or more than 25% or 33% of the height of the image frame, and / or has a width equal to or more than 50%, 66%, 75%, or 100% of the width of the image frame. Limiting the scan to a restricted area in the center of the image can be beneficial for images acquired by an ultra-wide angle lens, since images acquired by a wide angle lens have high distortion at the edges. In some embodiments, matrix scanning is used to track the code instead of trying to decode the code in each frame. Optical code tracking can be used to reduce computation (eg, instead of trying to decode the optical code in each frame, track the code).

[0103]

[0135] In some embodiments, the mobile device 1300 comprises a torch. A torch may also be referred to as a flashlight. The torch is configured to emit light 1320. The torch may comprise a light emitting diode. A torch lens may be used to direct, diffuse, and / or shape the light beam emitted from the light emitting diode.

[0104]

[0136] Rapid scanning can be used to detect and decode many optical patterns 1304 in parallel (e.g., two or more optical patterns 1304 in one image frame) and / or sequentially (e.g., a user moving a mobile device 1300 along a shelf in a grocery department (e.g., as shown in FIG. 13 )). In FIG. 13 , a first optical pattern 1304-1 and a second optical pattern 1304-2 are shown. As the user moves the mobile device 1300 along the shelf, the camera of the mobile device 1300 captures multiple images. The multiple images are analyzed to detect and decode the first optical pattern 1304-1 to obtain a first data payload and the second optical pattern 1304-2 to obtain a second data payload. The first data payload and the second data payload are stored in the memory of the mobile device 1300 and / or transmitted (e.g., wirelessly and / or via the Internet) to a remote hardware server.

[0105]

[0137] Acquiring many images, processing many images, and using a torch can use batteries and can be computationally intensive. Thus, in some embodiments, the torch is activated and / or multiple images are processed while receiving an indication of a user action. For example, while the user touches the display of the mobile device with a finger (e.g., the display is a touch screen), the application enters a fast scan mode. In the fast scan mode, the torch is turned on, the camera starts acquiring images, and the camera frame rate is increased (e.g., to 120 fps) and / or the shutter speed is increased (e.g., to 1 / 750 sec). After the user removes his / her finger from the display, the application exits the fast scan mode. After exiting the fast scan mode, the torch is turned off, the frame rate is reduced, and / or the camera is turned off, and / or the shutter speed returns to normal (e.g., the camera returns to autoexposure with automatic control of the shutter speed). The torch value is constant during the acquisition of multiple images (e.g., the torch does not flash as it would if the camera were trying to take a picture in a dark environment). Reducing the amount of time an application is in fast scan mode reduces battery usage and / or computing resources.

[0106]

[0138] There are additional and / or other ways that an application can enter and / or exit the fast scan mode. For example, a first double tap on the display triggers the application to enter the fast scan mode, a single tap is used to select the optical code, and / or a second double tap is used to exit the fast scan mode. A hardware button can be used to enter the fast scan mode. For example, the hardware button may be on the mobile device (e.g., volume buttons, hidden buttons on the back of the phone, home button, etc.) and / or integrated on the case of the mobile device. In some embodiments, a software button is used. For example, a rectangular icon with the word "SCAN" is displayed on the screen of the mobile device. As long as the user touches the screen where the rectangular icon is displayed, the application enters the fast scan mode. In some embodiments, data from an inertial measurement unit (IMU) is used to enter and / or exit the fast scan mode. For example, sensor data from the IMU corresponding to a shake or finger tap can be used by the application as an indication to enter or exit the fast scan mode.

[0107]

[0139] FIG. 14 shows a chart 1400 illustrating the scan range as a function of camera focus setting for scanning a first type of barcode. The first type of barcode is an EAN 13 barcode (e.g., the thinnest element is 6 mils or 6 thousandths of an inch). Chart 1400 shows the scan range (in millimeters) of an ultra-wide camera compared to a variable focus lens (the "main lens") for different focus values. For example, a mobile device may include two or more outward-facing cameras (e.g., cameras pointing in a direction not coincident with or 180 degrees from the normal direction of the mobile device's screen).

[0108]

[0140] The angle of view of the ultra-wide-angle camera is 128 degrees. The ultra-wide-angle camera has a scanning range of 27 to 355 mm. The focus value of the main lens is between 0 and 1 (in tenths of a second), with 0 (or 0%) being the closest plane and focus 1 (or 100%) being the furthest focal plane. The ultra-wide-angle camera has a wider scanning range than the varifocal lens, but the focus value of the varifocal lens is between 0 and 0.5. At focus values ​​of 0.6 and 0.7, the varifocal lens has a larger scanning range than the ultra-wide-angle camera, 65-520 mm and 105-720 mm, respectively. However, the mobile device is farther away from the optical code to scan than the ultra-wide-angle lens (e.g., 65 and 105 mm for the varifocal camera with focus values ​​of 0.6 and 0.7, compared to only 27 mm for the ultra-wide-angle lens). As shown in FIG. 15, the difference may not seem to be much for the first type of barcode, but the difference can be more noticeable when scanning different types of barcodes.

[0109]

[0141] 15 shows a chart 1500 illustrating the scan range as a function of camera focus setting for scanning a second type of barcode. The second type of barcode is a Code 128 barcode (e.g., the width of the thinnest element is 0.4 mm). Chart 1500 shows the scan range of an ultra-wide angle camera compared to a variable focus lens (the "main lens") for different focus values.

[0110]

[0142] The ultra-wide-angle camera has a scanning range of 66-380 mm. The scanning range is 184 mm-417 mm with a main lens with a focus value of 0.6, and 242 mm-562 mm with a main lens with a focus value of 0.7. Thus, scanning with an ultra-wide-angle camera allows the mobile device to be much closer to the optical code than the main lens while still providing a wide scanning range. In some embodiments, an external lens (e.g., external to the mobile device) is used to provide an additional wide-angle lens (e.g., a lens with a field of view of 100, 120 degrees or more, or 125 degrees or more). In some embodiments, the camera is configured to be 0.025, 0.05, 0.06, or 0.1 meters or more away from the optical code and / or 0.5, 0.4, 0.3, 0.2 meters or less away from the optical code to decode the optical code.

[0111]

[0143] FIG. 16 shows a flow chart of an embodiment of a process 1600 for decoding an optical pattern using a digital camera. The process 1600 begins with acquiring a plurality of images in step 1604. The plurality of images are acquired using a camera with a fixed focus, wide angle lens, fast exposure, high frame rate, and / or torch activation. The camera lens is set to a fixed focus during acquisition of the plurality of images, for example, a single focus lens or a variable focus lens set to a fixed value. The lens is a wide angle lens (e.g., having an angle of view of 60 degrees or 75 degrees or more) or an ultra-wide angle lens (e.g., having an angle of view of 90, 100 degrees or more, or 120 degrees or more). The exposure is fast. For example, the exposure time for each of the plurality of images is 1 / 500, 1 / 750, or 1 / 1500 of a second or less. The camera has a high frame rate during acquisition of the plurality of images. For example, the frame rate is 30, 60, 90, or 120 frames per second or more.

[0112]

[0144] In step 1608, the multiple images are analyzed to detect optical patterns in the multiple images. For example, the multiple images are analyzed for one, two, three or more optical patterns. The optical patterns may be a one-dimensional computer readable optical code (e.g., a barcode), a two-dimensional computer readable optical code (e.g., a QR code), and / or one or more symbols (e.g., letters, numbers, special characters, and / or unique symbols).

[0113]

[0145] In step 1612, one or more optical patterns are decoded to obtain a data payload or data payload. The data payload is data obtained by scanning the optical pattern. The data payload may include one or more letters, numbers, special characters, or other data. For example, three barcodes are decoded to obtain three numeric strings, one for each barcode. The three barcodes may be in one image frame and / or each barcode may be in a separate image frame. The data payload is then stored in memory in step 1616. The data payload may also be sent to a remote server.

[0114]

[0146] In some embodiments, a torch (e.g., a flashlight) on the mobile device is activated while acquiring multiple images. For example, the torch is turned on when an image is acquired by the camera, regardless of whether the exposure reading indicates there is enough light to properly expose the image without the torch on. The torch can be used as a sight. For example, the torch shines a light beam. When the mobile device is brought close to an object containing a light code, the user can move the light beam over the light code. Images of the light code can be acquired and the light code is decoded. A high number of frames per second is required because the light from the torch can cause glare (e.g., on plastic covering the light code). Thus, a high flow of frames increases the probability of acquiring frames before or after the light from the torch is shined directly onto the light code and can be used to decode the light code.

[0115]

[0147] In some embodiments, the optical code is tracked (e.g., using matrix scanning) while performing a very fast scan. The optical code can be tracked to speed up the decoding and / or to make the decoding more accurate. This can be useful for scanning optical codes that have similar or identical information. For example, some barcode readers use a duplication filter, or manual input, to reduce the chances that a barcode is decoded multiple times (e.g., more than once). The duplication filter can use a timer to ensure that the optical code is not read multiple times. For example, the timer of the duplication filter can be set to 0.5 seconds or more and 5 seconds or less (e.g., equal to 1, 2.1, or 3.5 seconds), depending on the application. In one example, when the optical code is decoded, the mobile device beeps and the screen turns green momentarily to indicate to the user that the optical code has been decoded. The duplication filter then sets a timer (e.g., for 1 second) after decoding the optical code, during which the mobile device does not decode the same optical code again (in some embodiments, the optical scanner detects and / or decodes other optical codes during the duplication filter time). Having overlapping filters helps prevent a mobile device from decoding the same optical code multiple times.

[0116]

[0148] Although a duplication filter can help prevent a photocode from being decoded multiple times, the duplication filter has several drawbacks. First, the duplication filter cannot decode identical photocodes faster than the timer allows. Second, the timer of the duplication filter can be difficult to set (type tune) for one or more applications and / or users (e.g., a time set on the timer that works well for one application or user may not work well for another application and / or user). Third, there may be multiple barcodes in one image, and multiple barcodes may have the same information. For example, while performing an inventory on a stack of boxes, each box may contain a pair of shoes, each pair of shoes may be identical to each other, and each shoe box may have the same barcode attached to it. Second, using a 1 second timer for the duplication filter may not only slow down the scanning of barcodes in the inventory, but may also result in miscounting of barcodes (e.g., undercounting due to the user moving too fast, or overcounting due to the user not moving fast enough). Some optical scanners track barcodes based on the assumption that each barcode is unique and cannot be used because the optical scanner will not count the number of barcodes correctly as the optical scanner will ignore duplicate barcodes.

[0117]

[0149] Some barcode scanners use manual input (e.g., software or hardware buttons, taps, etc.) to decode barcodes when there are multiple barcodes in the same image that are the same barcode. However, waiting to receive manual input before decoding the barcode can slow down the decoding. Furthermore, using manual input can be tedious for the user. In some cases, there may be hundreds of optical codes to scan (e.g., in inventory for many products), which can use hundreds of button presses by the user.

[0118]

[0150] In some configurations, optical code tracking (e.g., matrix scanning) is used instead of using overlap filters (e.g., no timers between decoding optical codes; no timers between decoding the same code), manual entry, and / or tracking of unique barcodes. After the optical code is decoded, the application tracks the optical code while it is within the field of view of the mobile device without decoding the optical code again. In certain configurations, the application tracks the optical code without mapping the scene. For example, if the camera of the mobile device is moved so that the optical code is no longer within the field of view of the camera, and then the camera is moved back so that the optical code is within the field of view of the camera, the application decodes the optical code again.

[0119]

[0151] By using tracking, no overlap filters are used, the process can be automatic (e.g., no manual input) and / or the application can be used to decode multiple identical codes within an image or field of view. In some configurations, tracking of the codes and / or not using overlap filters is performed without some or all of the features of ultra-fast scanning.

[0120]

[0152] Mobile device mounting with external lens

[0153] 17 is an embodiment of an attachment 1702 for a mobile device 1704. The attachment 1702 is an external attachment (e.g., the attachment 1702 is configured to be removably coupled to the mobile device 1704). The mobile device 1704 includes a camera. The camera includes a first lens 1708. The mobile device 1704 includes a torch 1712.

[0121]

[0154] The attachment 1702 comprises a second lens 1716 and a third lens 1720. The second lens 1716 is configured to increase the angle of view of the camera of the mobile device 1704 while the attachment 1702 is coupled to the mobile device 1704. For example, the first lens 1708 may have an angle of view of 60 degrees, and the second lens 1716 increases the camera's angle of view to 90, 100, or 120 degrees.

[0122]

[0155] The third lens 1720 is configured to focus the light of the torch 1712 while the attachment 1702 is coupled to the mobile device 1704. The third lens 1720 is configured to focus the light of the torch 1712 in a normal direction of the torch (e.g., the third lens 1720 does not redirect the light beam from the torch to a different angle). The direction normal of the torch 1712 is parallel to the normal vector of the back surface 1724 of the mobile device 1704. In some embodiments, the attachment 1702 includes the third lens 1720 and does not include the second lens. In some configurations, the third lens 1720 is a Fresnel lens and / or is made of plastic.

[0123]

[0156] The torch 1712 can be used as a sight to scan the light code. For example, the light from the torch 1712 indicates where the camera of the mobile device 1704 is pointed. By focusing the light from the torch 1712 (e.g., with a third lens 1720), the light from the torch 1712 is concentrated into a narrower spot.

[0124]

[0157] In some embodiments, the third lens 1720 is configured to focus the light of the torch 1712 into an oval shape (e.g., rectangular or elliptical, where an ellipse is non-circular). The mobile device 1704 is generally held by the user in a portrait orientation while scanning the optical code such that the long side of the mobile device 1704 is substantially vertical, as shown as vertical direction 1728 in FIG. 17, and the short side of the mobile device is substantially horizontal, as shown as horizontal direction 1732 in FIG. 17. The directional normal of the back surface 1724 is orthogonal to both the vertical direction 1728 and the horizontal direction 1732 of the mobile device. The elliptical shape is configured such that the major axis of the ellipse is generally parallel (e.g., plus or minus 1, 2, or 5 degrees) to the horizontal direction 1732 (e.g., in the direction of the short side of the mobile device 1704) and the minor axis is generally parallel (e.g., plus or minus 1, 2, or 5 degrees) to the vertical direction 1728 (e.g., in the direction of the long side of the mobile device 1704). The light of the torch 1712 is configured to be shaped into an ellipse or rectangle (e.g., the major side of the rectangle is substantially parallel to the horizontal direction 1732 and the minor side of the rectangle is substantially parallel to the vertical direction 1728) since optical codes are generally landscape-oriented and a user may hold the mobile device 1704 in a portrait orientation while scanning a barcode. By illuminating the torch 1712 in a rectangular shape, it may be more intuitive for a user to use the torch as a pointer to scan an optical code. In some embodiments, the major axis of the ellipse is aligned with the vertical direction 1728. The mobile device 1704 may be held landscape or in other orientations. For example, holding the mobile device 1704 in one hand and scanning a code low to the ground, the hand can be turned to the side so that the code is oriented along the long axis of the screen. By using a torch as an aiming device, instead of using the screen, the mobile device can be held so that part of the hand blocks the screen. This can be particularly useful for delivery drivers and transportation / logistics personnel who scan many items (e.g., in a stack).This can save them from the awkward maneuvering of scanning the mobile device 1704 in a vertical position (e.g., because they can look at the direction of the light from the torch instead of looking at a screen).

[0125]

[0158] The width of the beam from the torch 1712 on the surface containing the optical code can also indicate to the user how close the mobile device 1704 should be held to the surface containing the optical code. For example, if the beam traces a large ellipse on the surface with the optical code, the user should move the mobile device 1704 closer to the surface, and if the beam traces a small ellipse on the surface, the user should move the mobile device 1704 further away from the surface so that the ellipse of light on the surface is approximately the same size as the optical code. In certain configurations, the beam is configured to be focused to a width (e.g., measured horizontally) that is more than half the width of the optical code and / or less than or equal to two or three times the width of the optical code while the code is within the scanning range of the camera.

[0126]

[0159] In some configurations, the attachment 1702 is a case for the mobile device 1704. The case can include a second lens 1716 and / or a third lens 1720, and the case can cover at least a portion of two or more sides of the mobile device 1704 (e.g., for protection). For example, the case can cover 80% or more of the back surface 1724, as well as a portion of the long side(s) and / or short side(s) of the mobile device 1704.

[0127]

[0160] In some embodiments, the attachment 1702 (e.g., illustratively) includes a magnet. The magnet (e.g.,) can be used to attach the mobile device 1704 to a user's object (e.g., a metal cabinet) and / or a wearable device (e.g., a belt, armband, chest pack, leg band, backpack, fanny pack, etc.). By having a magnet, the mobile device 1704 can be positioned so that the user can use both hands for another task. For example, a delivery person wears an armband on their upper arm. The mobile device 1704 is secured (e.g., magnetically attached) to the armband. The woman delivers the package by carrying it with both hands to the doorway of the house and places the package at the doorway. After setting the package on the doorstep, she removes the mobile device 1704 from the armband and points the camera of the mobile device 1704 at the label on the package. She touches the screen of the mobile device with her thumb. While touching the screen, the torch 1712 is lit and the camera captures multiple images. The images are analyzed and the barcode on the label of the package is decoded. A green box is overlaid on the image presented on the screen to indicate to the user that the optical code has been decoded. The user then moves the mobile device backwards and an image of the doorstep with the package is saved for verification of the purchase. The user then removes his thumb from the screen. After the user releases his thumb, the torch turns off and the camera turns off or goes into standby mode. The user then places the mobile device on the armband and returns to his vehicle. The barcode and the image of the doorstep are transmitted from the mobile device 1704 to a remote server. Now that the barcode has been decoded, the image of the doorstep is automatically linked to the barcode and saved to a file identified by the barcode.

[0128]

[0161] 18 illustrates another embodiment of an attachment 1802 for a mobile device 1804. The attachment 1802 includes a lens 1816. The lens 1816 is an ultra-wide angle lens (e.g., an angle of view of 120 degrees or more and / or 130, 150, 160 degrees or less, or 175 degrees). Some phone models do not have a wide angle lens or a lens with too narrow an angle of view (not wide enough). The lens 1816 can be attached to increase the angle of view of the camera of the mobile device 1804.

[0129]

[0162] Sample Computing Device

[0163] FIG. 19 is a simplified block diagram of a computing device 1900. The computing device 1900 may implement some or all of the functions, behaviors, and / or capabilities described above that would result in the use of electronic storage or processing, as well as other functions, behaviors, or capabilities not explicitly described. The computing device 1900 includes a processing subsystem 1902, a storage subsystem 1904, a user interface 1906, and / or a communication interface 1908. The computing device 1900 may also include other components (not explicitly shown), such as a battery, a power controller, and other components operable to provide various enhanced functions. In various embodiments, the computing device 1900 may be implemented in a desktop or laptop computer, a mobile device (e.g., tablet computer, smartphone, mobile phone), a wearable device, a media device, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions or combinations of functions described above.

[0130]

[0164] The storage subsystem 1904 may be implemented using local and / or removable storage media, for example, using disks, flash memory (e.g., Secure Digital Cards, Universal Serial Bus Flash Drives), or any other persistent storage media, or combination of media, and may include volatile and / or non-volatile storage media. Local storage may include random access memory (RAM), including dynamic RAM (DRAM), static RAM (SRAM), or battery-backed RAM. In some embodiments, the storage subsystem 1904 may store one or more application and / or operating system programs to be executed by the processing subsystem 1902, including programs for performing some or all of the operations described above that may be performed using a computer. For example, the storage subsystem 1904 may store one or more code modules 1910 for performing one or more method steps described above.

[0131]

[0165] A firmware and / or software implementation may be implemented with modules (e.g., procedures, functions, etc.). Any machine-readable medium tangibly embodying instructions may be used in implementing the methodologies described herein. The code modules 1910 (e.g., instructions stored in a memory) may be implemented within the processor or external to the processor. As used herein, the term "memory" refers to long-term, short-term, volatile, non-volatile, or other types of storage media and is not limited to a particular type or number of memories or the type of media on which the memory is stored.

[0132]

[0166] Additionally, the term "storage medium" or "storage device" may refer to one or more memories for storing data, including read-only memory (ROM), RAM, magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, portable or non-removable storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing instructions and / or data.

[0133]

[0167] Further, embodiments may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting languages, and / or microcode, the program code or code segments for performing tasks may be stored in a machine-readable medium, such as a storage medium. A code segment (e.g., code module 1910) or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or a combination of instructions, data structures, and / or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted by any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0134]

[0168] The implementation of the techniques, blocks, steps, and means described above can be done in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. In the case of a hardware implementation, the processing unit may be implemented in one or more ASICs, DSPs, DSPDs, PLDs, FPGAs, processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions described above, and / or combinations thereof.

[0135]

[0169] Each code module 1910 may comprise a set of instructions (code) embodied on a computer-readable medium that instructs a processor of the computing device 1900 to perform a corresponding operation. The instructions may be configured to be executed sequentially, in parallel (such as under different processing threads), or a combination thereof. After loading the code modules 1910 into a general-purpose computer system, the general-purpose computer is transformed into a special-purpose computer system.

[0136]

[0170] Computer programs incorporating various features described herein (e.g., in one or more code modules 1910) may be encoded and stored on a variety of computer-readable storage media. Computer-readable media encoded with program code may be packaged with a compatible electronic device, or the program code may be provided separately from the electronic device (e.g., via Internet download or as a separately packaged computer-readable storage medium). Storage subsystem 1904 may also store information useful for establishing network connections using communications interface 1908.

[0137]

[0171] The user interface 1906 may include input devices (e.g., touch pad, touch screen, scroll wheel, click wheel, dials, buttons, switches, keypad, microphone, etc.) and output devices (e.g., video screen, indicator lights, speakers, headphone jack, virtual or augmented reality displays, etc.) along with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors, etc.). A user may manipulate the input devices of the user interface 1906 to invoke functions of the computing device 1900 and view and / or hear output from the computing device 1900 via the output devices of the user interface 1906. In some embodiments, the user interface 1906 may not be present (e.g., in the case of a process that uses an ASIC).

[0138]

[0172] The processing subsystem 1902 can be implemented as one or more processors (e.g., an integrated circuit, one or more single-core or multi-core microprocessors, a microcontroller, a central processing unit, a graphics processing unit, etc.). In operation, the processing subsystem 1902 can control the operation of the computing device 1900. In some embodiments, the processing subsystem 1902 can execute various programs in response to program code and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code to be executed can reside in the processing subsystem 1902 and / or in a storage medium such as the storage subsystem 1904. Through programming, the processing subsystem 1902 can provide various functions to the computing device 1900. The processing subsystem 1902 can also execute other programs for controlling other functions of the computing device 1900, including programs that can be stored in the storage subsystem 1904.

[0139]

[0173] The communication interface 1908 can provide voice and / or data communication capabilities to the computing device 1900. In some embodiments, the communication interface 1908 can include radio frequency (RF) transceiver components for accessing wireless data networks (e.g., Wi-Fi networks; 3G, 4G / LTE; etc.), mobile communication technologies, components for short-range wireless communication (e.g., using Bluetooth communication standards, NFC, etc.), other components, or combinations of technologies. In some embodiments, the communication interface 1908 can provide a wired connection (e.g., Universal Serial Bus, Ethernet, Universal Asynchronous Transmitter / Receiver, etc.) in addition to or instead of a wireless interface. The communication interface 1908 can be implemented using a combination of hardware (e.g., driver circuits, antennas, modulators / demodulators, encoders / decoders, and other analog and / or digital signal processing circuits) and software components. In some embodiments, the communication interface 1908 can support multiple communication channels simultaneously. In some embodiments, the communication interface 1908 is not used.

[0140]

[0174] It will be understood that computing device 1900 is illustrative and that variations and modifications are possible, and that the computing device may have various functions not specifically described (e.g., voice communication over a cellular network) and may include components appropriate for such functions.

[0141]

[0175] Additionally, while computing device 1900 has been described with reference to particular blocks, it should be understood that these blocks are defined for convenience of description and do not imply a particular physical arrangement of component parts. For example, processing subsystem 1902, storage subsystem, user interface 1906, and / or communication interface 1908 may be located within a single device or distributed across multiple devices.

[0142]

[0176] Furthermore, the blocks need not correspond to physically separate components. The blocks can be configured to perform various operations, for example, by programming a processor or by providing appropriate control circuitry, and the various blocks may or may not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the invention can be realized in a variety of apparatuses, including electronic devices implemented using a combination of circuitry and software. The electronic devices described herein can be implemented using a computing device 1900.

[0143]

[0177] Various features described herein, e.g., methods, apparatus, computer-readable media, etc., can be implemented using a combination of dedicated components, programmable processors, and / or other programmable devices. The processes described herein can be implemented on the same processor or different processors. Where components are described as being configured to perform certain operations, such configuration can be achieved, for example, by designing electronic circuitry to perform the operations, by programming a programmable electronic circuit (such as a microprocessor) to perform the operations, or by a combination thereof. Additionally, while the embodiments described above may refer to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and / or software components can also be used, and that certain operations described as being implemented in hardware can be implemented in software, or vice versa.

[0144]

[0178] In the above description, specific details are given to provide an understanding of the embodiments. However, it is understood that the embodiments may be practiced without these specific details. In some instances, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0145]

[0179] While the principles of the present disclosure have been described above in connection with specific devices and methods, it should be understood that this description is made by way of example only and not as a limitation on the scope of the present disclosure. The embodiments have been selected and described in order to explain the principles and practical applications of the invention, and to enable those skilled in the art to utilize the invention in various embodiments and various modifications as suited to the particular applications contemplated. It is understood that the description is intended to cover modifications and equivalents.

[0146]

[0180] It is also noted that the embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a series of processes, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0147]

[0181] The enumeration of "a," "an," or "the" is intended to mean "one or more" unless specifically indicated to the contrary. All patents, patent applications, publications, and descriptions referred to herein are incorporated by reference in their entirety for all purposes. None are admitted to be prior art.

[0148]

[0182] In some embodiments, a system for mapping objects onto a shelf unit using image analysis includes a camera having an image sensor configured to capture a plurality of image frames, and one or more processors configured to receive the plurality of images captured by the camera, identify a first item code in the plurality of images, identify a second item code in the plurality of images, calculate a distance and direction of the first item code relative to the second item code, calibrate the distance and direction to the shelf diagram, generate a first vector including the first item code and a relative position of the first item code relative to the shelf diagram, and / or generate a second vector including the second item code and a relative position of the second item code relative to the shelf diagram. The method may include receiving a plurality of images, identifying a first item code in the plurality of images, identifying a second item code in the plurality of images, calculating a distance and direction of the first item code relative to the second item code, calibrating the distance and direction to the shelf diagram, generating a first vector including the first item code and a relative position of the first item code relative to the shelf diagram, and / or generating a second vector including the second item code and a relative position of the second item code relative to the shelf diagram. In some embodiments, calculating the distance and direction of the first item code relative to the second item code is based on the first item code and the second item code being in a two dimensional vertical plane, calculating the distance and direction of the first item code relative to the second item code is based on a width or height of the first item code, calculating the distance and direction of the first item code relative to the second item code is based on a width or height of the shelf label, calculating the distance and direction of the first item code relative to the second item code is based on a simultaneous location and mapping algorithm of the mobile device, the distance and direction of the first item code relative to the second item code is based on two image sensors of the mobile device used to capture the multiple images and a known distance between the two image sensors, calculating the distance and direction of the first item code relative to the second item code is based on a depth sensor or a three dimensional image sensor of the mobile device used to capture the multiple images,the shelving unit being part of an aisle of the store, and the first vector including an aisle indicator and a relative position of the first item code with respect to the aisle, and / or determining that the item identified by the first item code is in the correct aisle based on comparing the aisle indicator of the first vector with the item's planned aisle position from the parallelogram; ascertaining a first boundary of the object associated with the first item code, the first vector including a value indicative of the first boundary with respect to the shelf diagram; ascertaining a second boundary of the object associated with the second item code, the second vector including a value indicative of the second boundary with respect to the shelf diagram. comparing the first boundary to the planned location of the first item in the planogram using a value indicative of a first boundary from the first vector, determining that the first boundary for the shelf plan fits the planogram based on the first boundary for the shelf plan being similar to the planned location of the first item in the planogram, comparing the second boundary to the planned location of the second item in the planogram using a value indicative of a second boundary from the second vector, and / or determining that the second boundary for the shelf plan fits the planogram based on the second boundary for the shelf plan being similar to the planned location of the second item in the planogram. In some embodiments, the method includes receiving a plurality of images, the plurality of images including an overhead image and a plurality of images having different fields of view, the overhead image including a plurality of barcodes, and / or the plurality of images each including a subset of the plurality of barcodes, and / or matching the different fields of view of the plurality of images with the overhead image to register locations of the barcodes in the overhead image. In some configurations, the barcodes cannot be decoded in the overhead image because the barcodes in the overhead image are too small to be decoded, the overhead image is aligned, rectified, and dewarped using a row of bars in the overhead image, the matching of the different views of the multiple images with the overhead image is performed using a random sample consensus algorithm for interest points and homography, the matching of the different views of the multiple images with the overhead image is based on an assumption that the barcodes are in a plane, and / or the method includes:The method includes tracking two or more barcodes simultaneously across multiple image frames without decoding the two or more barcodes, decoding the multiple barcodes using the multiple images, and / or calculating a scale of the overhead image based on a known height or width of one of the multiple barcodes.

[0149]

[0183] In some embodiments, 1. A method of image analysis for recognizing patterns in an image, comprising: receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns in an array; matching the array with a pattern template, the pattern template being a predefined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array compared to the pattern template; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; The method includes: 2. The method of claim 1, wherein the plurality of optical patterns is a bar code made up of parallel bars. 3. The method of claim 1, wherein the multiple optical patterns are disposed on a label affixed to the object in the multiple images, the method further comprising the step of calculating a physical dimension of the object. 4. The step of calculating the physical dimensions of an object is calculating relative dimensions of objects in the image to match relative dimensions of labels in the image; calculating a physical dimension of the object based on the relative dimensions of the object; 4. The method of claim 3, comprising: 5. Comparing the physical dimensions of the object to expected physical dimensions of an object associated with the object identifier; Matching the object with an object identifier; presenting a graphical element on the display indicating the match; The method of claim 3, further comprising: 6. The method of claim 3, wherein the physical dimensions of the object include at least one of the volume of the object or the weight of the object. 7. Steps to identify logistics criteria based on physical dimensions The method of claim 6, further comprising: 8. The method of claim 7, wherein the logistics criteria includes at least one of a postage rate, an inventory order identifier, a packing order, or a warehouse location identifier. 9. Identifying a first identifier pattern in a first image of the plurality of images, the first identifier pattern being an identifier of a pattern template; identifying a second identifier pattern in a second image of the plurality of images; comparing the first identifier pattern to a second identifier pattern; If the first identifier pattern does not match the second identifier pattern, matching the second identifier pattern against a second pattern template, the second pattern template being a second predetermined arrangement of optical patterns; identifying a second selected optical pattern based on a position of the selected optical pattern within the array; decoding the second selected optical pattern to generate a second object identifier; storing a second object identifier in a memory device; The method of claim 1, further comprising: 10. Presenting a plurality of images including the pattern template on a display; presenting a visual representation of the pattern template on a display; presenting a graphic element overlaid on the selected optical pattern; The method of claim 1, further comprising: 11. A method of image analysis for recognizing a pattern in an image, comprising: receiving a target optical pattern type; receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns; matching a selected optical pattern of the plurality of optical patterns with a target optical pattern type; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; The method includes: 12. The step of matching the selected optical pattern with a target optical pattern type comprises: receiving a first relative dimension of a target optical pattern type; determining a second relative dimension of the selected optical pattern; Matching the second relative dimension to the first relative dimension; 12. The method of claim 11, comprising: 13. The step of matching the selected optical pattern with a target optical pattern type comprises: receiving a characteristic dimension of a target optical pattern type; Ascertaining a plurality of spatial dimensions of the plurality of optical patterns based on an arrangement of the plurality of optical patterns in an image of the plurality of images; Matching spatial dimensions of the selected optical pattern to characteristic dimensions of a target optical pattern type; 12. The method of claim 11, comprising: 14. The method of claim 11, wherein the target optical pattern type is a barcode consisting of parallel bars, a QR code, a MaxiCode, or an Aztec code. 15. The step of matching the selected optical pattern of the plurality of optical patterns with a target optical pattern type comprises: ascertaining a plurality of aspect ratios corresponding to the plurality of optical patterns or identifying a characteristic marking within one of the plurality of optical patterns; Matching the selected optical pattern to a target aspect ratio or characteristic markings of a target optical pattern type; 12. The method of claim 11, comprising: 16. A method of image analysis for recognizing a pattern in an image, comprising: receiving a first plurality of images captured by a camera, a first plurality of images including an image boundary and a partial optical pattern adjacent to the image boundary; determining an orientation of the partial optical pattern relative to an image boundary; generating a first prompt as a playback of a first recorded sound to move a camera in a first direction based on the direction; receiving a second plurality of images captured by a camera, the second plurality of images including a complete optical pattern corresponding to the partial optical pattern; decoding the complete optical pattern to generate an object identifier; providing a second prompt as a playback of a second recorded audio to indicate successful decoding of the complete optical pattern; storing the object identifier in a memory device; The method includes: 17. The method of claim 16, wherein the complete optical pattern is a bar code made up of parallel bars. 18. Presenting a first plurality of images on a display; presenting a first graphical element on a display, the first graphical element showing a partial optical pattern; presenting a second plurality of images on the display; presenting a second graphic element on the display, the second graphic element showing a complete optical pattern; 17. The method of claim 16, further comprising: 19. Identifying object information based on the object identifier; generating a third prompt including the object information as a playback of a third recorded sound; 17. The method of claim 16, further comprising: 20. A method of image analysis for recognizing a pattern in an image containing a plurality of patterns, comprising: receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns; presenting a plurality of optical patterns on a display; Selecting a selected optical pattern from the plurality of optical patterns User actions; a position of the selected optical pattern in one or more of the plurality of images; and identifying based on presenting a visual representation of the selected optical pattern on a display; decoding the selected optical pattern to generate an object identifier; indicating via a visual indication that the selected optical pattern has been decoded; storing the object identifier in a memory device; The method includes: 21. The method of claim 20, wherein the plurality of optical patterns is a bar code made up of parallel bars. 22. The plurality of optical patterns includes a first optical pattern and a second optical pattern; the first optical pattern is a selected optical pattern; The second optical pattern is not decoded. The method of claim 20. 23. The method of claim 20, further comprising the step of presenting, on a display, one or more graphical elements related to processing the plurality of optical patterns. 24. The method of claim 23, wherein the one or more graphical elements include at least one of a reticle, a capture button, a bounding box, and a highlighted area of ​​the display, and at least the highlighted area is presented as an overlay on one optical pattern of the plurality of optical patterns. 25. The display presents a capture button and at least one of a reticle, a bounding box, or a highlighted area; The user action includes pressing the capture button. identifying the selected optical pattern based on the selected optical pattern located under the reticle, bounding box, or highlighted region; The method of claim 24. 26. The method of claim 20, wherein the user action comprises a screen touch on the display. 27. The method of claim 20, wherein the user action includes a screen touch onto an area of ​​the display that corresponds to one of the plurality of optical patterns. 28. The display communicates with a smartphone that includes an inertial measurement unit; The user action is a tap on the exterior surface of the smartphone; The method is: receiving motion data from an inertial measurement unit of the smartphone; classifying the motion data as an impulse; Further comprising: and identifying the selected optical pattern based on the selected optical pattern being presented with the visual display upon receipt of the impulse. The method of claim 20. 29. The display communicates with an electronic device that includes a plurality of interactive buttons; The user action is a screen tap or an interaction with a capture button of a plurality of interactive buttons; The method is: presenting a graphical element on the display at a first location corresponding to a location of a first optical pattern of the plurality of optical patterns in an image of the plurality of images; repositioning the graphical element from a first position to a second position corresponding to a second optical pattern in the image in response to a user interaction with an interactive button of the plurality of interactive buttons; Further comprising: and identifying the selected optical pattern based on a correspondence between the graphic element and the selected optical pattern when coincident with a user action. The method of claim 20. 30. Receiving a user interaction with a region of the display; magnifying an area of ​​the plurality of images corresponding to the region by a predetermined factor; 21. The method of claim 20, further comprising: 31. Presenting a plurality of images on a display in a sequence according to a frame rate; receiving a second user action; presenting an image of the plurality of images on the display concurrently with a second user action; receiving a third user action; upon receipt of a third user action, resuming presentation of the plurality of images on the display in accordance with the sequence; 21. The method of claim 20, further comprising: 32. Identifying a first number of the plurality of optical patterns in a first image of the plurality of images; identifying a second number of the plurality of optical patterns in a second image of the plurality of images; determining a degree of change between the first number and the second number; comparing the degree of change to a threshold change criterion; presenting, on a display, an image of the plurality of images including a plurality of optical patterns if the degree of change satisfies a threshold change criterion; 21. The method of claim 20, further comprising: 33. Removing said image of the plurality of images from the display after a period of time has elapsed; presenting a plurality of images on a display; 33. The method of claim 32, further comprising: 34. The images are arranged in a sequence of frames corresponding to a frame rate; the second image corresponds to a second position in the sequence of frames subsequent to the first position in the sequence of frames corresponding to the first image; 33. The method of claim 32. 35. The method of claim 34, wherein the separation between the first position and the second position corresponds to a period according to a frame rate. 36. The method of claim 35, wherein the period is 300 milliseconds. 37. Identifying an application associated with the camera; determining a first position and a second position according to an application; 35. The method of claim 34, further comprising: 38. Identifying a device in communication with the camera and the display; determining a first location and a second location according to the device; 35. The method of claim 34, further comprising: 39. Identifying a plurality of device parameters describing at least one of a display, a camera, and a user; Determining the first location and the second location according to the device includes: calculating a period of time using a plurality of device parameters; setting the second position and the first position to correspond to the period according to a frame rate; 39. The method of claim 38, further comprising: 40. Decoding the plurality of optical patterns to generate a plurality of object identifiers; identifying a number of object identifiers corresponding to the plurality of object identifiers; According to the number of object identifiers that meet the threshold criteria, identifying a selected image of a plurality of images including a plurality of optical patterns; presenting the selected image on a display; 21. The method of claim 20, further comprising: 41. The method of claim 40, wherein the threshold criterion is a degree of change in the number of object identifiers, including a variation parameter in the number of object identifiers over a period of time. 42. Presenting a first object identifier of a plurality of object identifiers; presenting on a display first object information associated with the first object identifier; in response to a second user action, presenting a second object identifier of the plurality of object identifiers on the display and removing the first object identifier and the first object information from the display; presenting second object information associated with a second object identifier; 41. The method of claim 40, further comprising: 43. The method of claim 20, wherein the visual representation includes object information based on the object identifier. 44. The method of claim 20, wherein the visual indication includes a dynamic graphical element that indicates the progress of the decoding as a progress bar. 45. The method of claim 20, wherein the visual representation includes a dynamic graphical element that indicates the time since decoding by progressively increasing the transparency of the visual representation over a period of time.

[0150]

[0184] In some embodiments, 1. A method of image analysis for recognizing a pattern in an image, comprising: receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns in an array, the plurality of optical patterns disposed on a label affixed to an object in the plurality of images, one or more of the plurality of optical patterns encoding an object identifier; matching the array with a pattern template, the pattern template being a predefined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; estimating a relative dimension of an object in one of the images compared to a relative dimension of a label in the image; estimating physical dimensions of the object based on the relative dimensions of the object; The method includes: 2. The method of claim 1, wherein the plurality of optical patterns is a bar code made up of parallel bars. 3. Comparing the physical dimensions of the object to expected physical dimensions of an object associated with the object identifier; Matching the object with an object identifier; presenting a graphical element on the display indicating the match; The method of claim 1, further comprising: 4. The method of claim 1, wherein the physical dimensions of the object include at least one of the volume of the object or the weight of the object. 5. The method of claim 1, further comprising the step of verifying logistics criteria based on physical dimensions. 6. The method of claim 5, wherein the logistics criteria includes at least one of a postage rate, an inventory order identifier, a packing order, or a warehouse location identifier. 7. Presenting a plurality of images including pattern templates on a display; presenting a graphic element overlaid on the selected optical pattern; The method of claim 1, further comprising: 8. A camera having an image sensor configured to capture a plurality of images; one or more processors, receiving a plurality of images captured by an image sensor, the plurality of images including a plurality of optical patterns in an array, the plurality of optical patterns disposed on a label affixed to an object in the plurality of images, one or more of the plurality of optical patterns encoding an object identifier; Matching the array to a pattern template, the pattern template being a predetermined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array; Decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; Estimating a relative dimension of an object in one of the images compared to a relative dimension of a label in the image; Estimating the physical dimensions of an object based on the relative dimensions of the object; one or more processors configured to A system comprising: 9. One or more processors may: 10. The system of claim 8, configured to validate logistics criteria based on physical dimensions, the logistics criteria including at least one of a postage rate, an inventory order identifier, a packing order, or a warehouse location identifier. 10. One or more processors may: Comparing the physical dimensions of the object to expected physical dimensions of an object associated with the object identifier; Matching the object to an object identifier; presenting a graphical element on the display indicating the match; The system of claim 8, further configured to: 11. One or more processors may: presenting a plurality of images including the pattern template on a display; presenting a graphic element overlaid on the selected optical pattern; The system of claim 8, further configured to: 12. When executed, receiving a plurality of images captured by a camera, the plurality of images including a plurality of optical patterns in an array, the plurality of optical patterns disposed on a label affixed to an object in the plurality of images, one or more of the plurality of optical patterns encoding an object identifier; matching the array with a pattern template, the pattern template being a predefined array of optical patterns; identifying one optical pattern of the plurality of optical patterns as a selected optical pattern based on a position of the selected optical pattern in the array; decoding the selected optical pattern to generate an object identifier; storing the object identifier in a memory device; estimating a relative dimension of an object in one of the images compared to a relative dimension of a label in the image; estimating physical dimensions of the object based on the relative dimensions of the object; A memory device that contains instructions that cause one or more processors to 13. When executed, presenting a plurality of images including pattern templates on a display; presenting a graphic element overlaid on the selected optical pattern; and further comprising instructions for causing one or more processors to: Optionally, the plurality of optical patterns comprises a character string, the plurality of optical patterns comprises a two-dimensional optical code, the object is a first object, the method further comprises estimating physical dimensions of a second object in the plurality of images based on relative dimensions of the label in the images, the plurality of optical patterns comprises a character string, the plurality of optical patterns comprises a two-dimensional optical code, and / or the method further comprises presenting a visual representation of the pattern template on a display.

[0151]

[0185] In some embodiments, 1. A mobile device, comprising: A display and A camera and one or more processors in communication with the camera and / or the display; When executed by one or more processors, the mobile device Using a camera to detect an optical pattern in a scene, the optical pattern encoding an object identifier; Receiving a user action; performing a focus cycle of the camera after receiving a user action, the focus cycle changing a focus position of the camera from a first focus position to a second focus position, the second focus position corresponding to an optical pattern that is in focus within the scene; performing a focus cycle and then acquiring an image of the scene using the camera at a second focus position; Decoding the optical pattern in the image of the scene to generate an object identifier; one or more memory devices for storing instructions; A mobile device comprising: 2. The mobile device of claim 1, wherein the user action is received via the display after detecting an optical pattern in a scene. 3. The mobile device of claim 1, wherein the scene is a first scene, the user action includes an action of the mobile device, and the instructions, when executed, cause the one or more processors to detect a scene change from the first scene to a second scene. 4. Detecting scene changes Detecting behavior of the mobile device that exceeds a threshold behavior; or Detecting a scene change based on motion blur in an image of the scene captured by a camera; The mobile device of claim 3, comprising: 5. An image of a scene is a second image, and detecting optical patterns in the scene is A first image of the scene is acquired using a camera, the focus position of the camera being at a first focus position; Detecting an optical pattern in the first image without decoding the optical pattern; determining that the optical pattern is out of focus in the first image; The mobile device of claim 1 , further comprising: 6. Identifying that the optical pattern is out of focus in the first image includes: Ascertaining the resolution of the optical pattern in the first image; and determining that a resolution of the optical pattern in the first image is less than a minimum resolution for decoding the optical pattern; The mobile device of claim 5, further comprising: 7. The optical pattern is a first optical pattern and the object identifier is a first object identifier, the instructions, when executed, further causing one or more processors to: detecting a second optical pattern in the first image prior to receiving the user action, the second optical pattern encoding a second object identifier; and decoding a second optical pattern in the first image to generate a second object identifier; 6. The mobile device of claim 5. 8. Detecting, by a computer system, an optical pattern in the scene using a camera at a first focus position, the optical pattern encoding an object identifier; receiving, by a computer system, a user action; performing, by the computer system, a focus cycle of the camera after receiving a user action, the focus cycle changing a focus position of the camera from a first focus position to a second focus position, the second focus position corresponding to an optical pattern in focus in the scene; acquiring, by a computer system, an image of the scene using the camera at a second focus position; decoding, by a computer system, the optical pattern in the image of the scene to generate an object identifier; The method includes: 9. Presenting, by the computer system, a visual indication on the display that the optical pattern was not decoded. The method of claim 8, further comprising: 10. The method of claim 8, wherein the scene is a first scene and the user action comprises an action of a computer system, the method further comprising detecting a scene change from the first scene to a second scene. 11. The step of detecting a scene change comprises: detecting motion of the mobile device that exceeds a threshold motion; or Detecting a scene change based on motion blur in an image of the scene captured by a camera; 11. The method of claim 10, comprising: 12. The image of the scene is a second image, and the step of detecting an optical pattern in the scene includes: acquiring a first image of the scene using a camera, the focus position of the camera being at a first focus position; detecting an optical pattern in the first image without decoding the optical pattern; determining that the optical pattern is out of focus in the first image; 9. The method of claim 8, comprising: 13. The step of determining that the optical pattern is out of focus in the first image includes: determining a resolution of the optical pattern in the first image; determining that a resolution of the optical pattern in the first image is less than a minimum resolution for decoding the optical pattern; 13. The method of claim 12, comprising: 14. The optical pattern is a first optical pattern and the object identifier is a first object identifier, and the method comprises: detecting a second optical pattern in the first image prior to receiving the user action, the second optical pattern encoding a second object identifier; decoding a second optical pattern in the first image to generate a second object identifier; The method of claim 12, further comprising: 15. When executed by one or more processors of a computer system, the one or more processors are detecting an optical pattern in the scene using the camera at a first focus position, the optical pattern encoding an object identifier; Receiving a user action; performing a focus cycle of the camera after receiving a user action, the focus cycle changing a focus position of the camera from a first focus position to a second focus position, the second focus position corresponding to an in-focus optical pattern in the scene; acquiring an image of the scene using the camera at a second focus position; Decoding the optical pattern in the image of the scene to generate an object identifier; A memory device that stores instructions. 16. Disabling the camera's autofocus system, which controls the camera's focus position through a computer system The memory device of claim 15 further comprising: 17. The memory device of claim 15, wherein the scene is a first scene, the user action includes an action of a computer system, and the instructions, when executed, cause the one or more processors to detect a scene change from the first scene to a second scene. 18. Detecting scene changes Detecting operation of a computer system that exceeds a threshold operation; or Detecting a scene change based on motion blur in an image of the scene captured by a camera; 20. The memory device of claim 17, comprising: 19. An image of a scene is a second image, and detecting an optical pattern in the scene is A first image of the scene is acquired using a camera, the focus position of the camera being at a first focus position; Detecting an optical pattern in the first image without decoding the optical pattern; determining that the optical pattern is out of focus in the first image; The memory device of claim 15 comprising: 20. Identifying that the optical pattern is out of focus in the first image includes: Ascertaining the resolution of the optical pattern in the first image; and determining that a resolution of the optical pattern in the first image is less than a minimum resolution for decoding the optical pattern; 20. The memory device of claim 19, comprising:

Claims

1. 1. A system for mapping objects on a shelving unit using image analysis, comprising: a camera including an image sensor configured to capture a plurality of image frames; One or more processors, receiving a plurality of images captured by the camera; Identifying a first item code within the plurality of images; Identifying a second item code within the plurality of images; tracking a location of the first item code in the plurality of images without decoding the first item code in at least one or more of the plurality of images; tracking a location of the second item code in the plurality of images without decoding the second item code in at least one or more of the plurality of images; calculating a distance and a direction of the first item code relative to the second item code using an image of the plurality of images including the first item code and the second item code; calibrating said distance and said direction to a shelf plan; generating a first vector including the first item code and a relative position of the first item code with respect to the shelf diagram; generating a second vector including the second item code and the relative position of the second item code with respect to the shelf diagram; one or more processors configured to A system comprising:

2. 1. A method for mapping objects onto a shelving unit using image analysis, comprising: receiving a plurality of images; identifying a first item code within the plurality of images; identifying a second item code within the plurality of images; tracking a location of the first item code in the plurality of images without decoding the first item code in at least one or more of the plurality of images; tracking a location of the second item code in the plurality of images without decoding the second item code in at least one or more of the plurality of images; calculating a distance and a direction of the first item code relative to the second item code using an image of the plurality of images including the first item code and the second item code; calibrating the distance and the direction to a shelf plan; generating a first vector including the first item code and a relative position of the first item code with respect to the shelf diagram; generating a second vector including the second item code and a relative position of the second item code with respect to the shelf diagram; The method includes:

3. 3. The method of claim 2, wherein the step of calculating the distance and the direction of the first item code relative to the second item code is based on the first item code and the second item code being in a two-dimensional perpendicular plane.

4. The method of claim 2 , wherein the step of calculating the distance and the orientation of the first item code relative to the second item code is based on a width or height of the first item code.

5. The method of claim 2 , wherein the step of calculating the distance and the direction of the first item code relative to the second item code is based on a width or height of a shelf label.

6. The method of claim 2 , wherein the step of calculating the distance and the direction of the first item code relative to the second item code is based on a simultaneous location and mapping algorithm of a mobile device.

7. 3. The method of claim 2, wherein the step of calculating the distance and the direction of the first item code relative to the second item code is based on two image sensors of a mobile device used to capture the multiple images and a known distance between the two image sensors.

8. 3. The method of claim 2, wherein the step of calculating the distance and the direction of the first item code relative to the second item code is based on a depth sensor or a three-dimensional image sensor of a mobile device used to capture the multiple images.

9. the shelving unit being part of an aisle of a store; The first vector is Aisle indicators; the relative position of said first item code to said passage; The method of claim 2 , comprising:

10. 10. The method of claim 9, further comprising determining that an item identified by the first item code is in a correct aisle based on comparing the aisle indicator of the first vector to a planned aisle location of the item from a planogram.

11. ascertaining a first boundary of an object associated with the first item code, the first vector including a value indicative of the first boundary relative to the shelf view; ascertaining a second boundary of an object associated with the second item code, the second vector including a value indicative of the second boundary relative to the shelf view; The method of claim 2 , further comprising:

12. using a value indicative of the first boundary from the first vector to compare the first boundary to a planned position of a first item in a planogram; determining that the first boundary for the shelf plan fits the planogram based on the first boundary for the shelf plan being similar to the planned location of the first item in the planogram; using a value indicative of the second boundary from the second vector to compare the second boundary to a planned position of a second item in the planogram; determining that the second boundary for the shelf plan fits the planogram based on the second boundary for the shelf plan being similar to the planned location of the second item in the planogram; The method of claim 11 further comprising:

13. 2. The system of claim 1, wherein the one or more processors are configured to calculate a distance and a direction of the first item code relative to the second item code based on the first item code and the second item code being in a two-dimensional vertical plane.

14. 2. The system of claim 1, wherein the one or more processors are configured to calculate the distance and the orientation of the first item code relative to the second item code based on a width or height of the first item code.

15. 2. The system of claim 1, wherein the one or more processors are configured to calculate the distance and the orientation of the first item code relative to the second item code based on a width or height of a shelf label.

16. 2. The system of claim 1, wherein the one or more processors are configured to calculate the distance and the direction of the first item code relative to the second item code based on a contemporaneous location of a mobile device and a mapping algorithm.

17. 2. The system of claim 1, wherein the one or more processors are configured to calculate the distance and the orientation of the first item code relative to the second item code based on two image sensors of a mobile device used to capture the multiple images and a known distance between the two image sensors.

18. 2. The system of claim 1, wherein the one or more processors are configured to calculate the distance and the direction of the first item code relative to the second item code based on a depth sensor or a three-dimensional image sensor of a mobile device used to acquire the plurality of images.

19. the shelving unit being part of an aisle of a store; The first vector is Aisle indicators; the relative position of said first item code to said passage; The system of claim 1 , comprising:

20. 20. The system of claim 19, wherein the one or more processors are further configured to determine that an item identified by the first item code is in a correct aisle based on comparing the aisle indicator of the first vector to a planned aisle location of the item from a planogram.

Images