4D Barcode Mapping for Moving Objects

Abstract

Systems and methods for tracking objects in space are disclosed. The systems and methods include capturing two-dimensional (2D) image data and capturing, generating, or otherwise accessing three-dimensional (3D) image data. A barcode is decoded from the two-dimensional (2D) image data. The object is identified from the three-dimensional (3D) image data. The 2D image of the barcode and the barcode data are combined with the 3D object to form reference 3D object data. The reference 3D object data is used for comparison with subsequently captured 3D and 2D image data. In some embodiments, a four-dimensional (4D) projection of the reference 3D object data is used for comparison and verification of subsequently captured 3D and 2D image data.

Description

[Background technology]

[0001] Machine vision systems are attractive for imaging and analyzing moving objects because they generally provide high-fidelity image analysis. For example, machine vision cameras are commonly deployed in industrial applications to track objects moving through a facility on conveyor belts or similar transportation systems. In some such applications, multiple machine vision cameras are used to image and scan objects on a scan area, and barcodes in the scan area are also scanned with the hope of associating the objects with their respective barcodes. While it is important to properly identify objects by tracking their respective barcodes, tracking errors can occur. In some cases, an object moves throughout the scan area on a conveyor belt but is not identified by its barcode. The object may be obscured by or confused as part of another object. In some cases, an object may be identified by one machine vision camera but not by a subsequent machine vision camera positioned downstream on the conveyor belt. In yet another example, an object may be identified by multiple machine vision cameras that may collectively fail to recognize it as the same object at different conveyor belt locations, resulting in duplicate identification of the same object.

[0002] Therefore, there is a need for a system and method for more accurately tracking moving objects using machine vision systems. Summary of the Invention

[0003]

[0004] In one embodiment, the invention is a system for tracking barcodes in space, comprising: a three-dimensional (3D) data acquisition subsystem, a two-dimensional (2D) imaging subsystem including one or more 2D imagers and one or more decoding processors; and a computing subsystem including one or more processors and a non-transitory computer-readable storage medium storing instructions, the 2D imaging subsystem oriented and configured to: capture, with the one or more 2D imagers, 2D image data representing a 2D image of an initial environment of a scan tunnel; and decode, with the one or more decoding processors, barcodes identified in the captured 2D image data, the instructions, when executed by the one or more processors, cause the computing system to access, from the 3D data acquisition subsystem, captured 3D image data corresponding to a 3D representation of the environment, identify objects in the environment based on the captured 3D image data, associate the barcodes identified in the captured 2D image data with the objects, and generate marked object data representing the objects in the environment.

[0005] In a variation on this embodiment, the non-transitory computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the computing system to access, from the 3D data acquisition subsystem, next captured 3D image data corresponding to a next 3D representation of a next environment in the scan tunnel downstream of the first environment; identify an object within the next environment from the next captured 3D image data; generate a successful scan indication in response to successfully associating a decoded barcode corresponding to the next environment with the object in the next environment; and generate a failed scan indication in response to not successfully associating a decoded barcode with the object in the next environment.

[0006] In a variation on this embodiment, the non-transitory computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the computing system to capture, with the one or more 2D imagers, next 2D image data representing a 2D image of the next environment, and cause the one or more decoding processors to attempt to identify a barcode within the next 2D image data, and in response, decode the barcode identified in the next 2D image data.

[0007] In a variation on this embodiment, the non-transitory computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the computing system to generate the failed scan indication in response to the one or more 2D imagers failing to capture next 2D image data representing a 2D image of the next environment, or in response to the one or more 2D imagers capturing the next 2D image data representing the 2D image of the next environment but failing to identify a barcode within the next 2D image data.

[0008] In a variation of this embodiment, the non-transitory computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the computing system to perform a four-dimensional (4D) projection of the marked object data within the scanning tunnel, the projection representing a predicted future position of the object within the scanning tunnel based on predicted or measured movement of the object.

[0009] In a variation on this embodiment, the 2D imaging subsystem is further configured to capture, with the one or more 2D imagers, next 2D image data representing a 2D image of a next environment in the scan tunnel downstream of the first environment, and decode, with the one or more decoding processors, a next barcode identified in the next 2D image data; and the non-transitory computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the computing system to access, from the 3D data acquisition subsystem, next captured 3D image data corresponding to a next 3D representation of the next environment, identify a next object in the next environment from the next captured 3D image data, associate the next barcode with the next object, generate next marked object data representing the next object, determine whether the next marked object data satisfies the 4D projection of the marked object data, and determine, when the next marked object data satisfies the 4D projection of the marked object data, the decoding of the next barcode or the decoding of a duplicate of the barcode.

[0010] In a variation of this embodiment, the instructions, when executed by the one or more processors, cause the computing system to determine whether the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment, and determine that the next marked object data satisfies the 4D projection of the marked object data if the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment, thereby determining whether the next marked object data satisfies the 4D projection of the marked object data.

[0011] In a variation of this embodiment, the instructions, when executed by the one or more processors, cause the computing system to determine that the next marked object data does not satisfy the 4D projection of the marked object data if the next marked object data does not at least partially overlap with the 4D projection of the marked object data in the next environment, and to determine that the next object is different from the object.

[0012] In a variation of this embodiment, the system further comprises a moving surface configured to move across the scanning tunnel, and the instructions, when executed by the one or more processors, cause the computing system to perform the 4D projection of the marked object data within the scanning tunnel based on a predicted movement of the moving surface on which the object resides.

[0013] In a variation on this embodiment, the moving surface is a conveyor belt that moves in a substantially linear manner through the environment.

[0014] In a variation of this embodiment, the instructions, when executed by the one or more processors, cause the computing system to associate the barcode with the object by identifying the position and orientation of the barcode relative to the object based on the position and orientation of the one or more 2D imagers.

[0015] In a variation of this embodiment, the instructions, when executed by the one or more processors, cause the computing system to associate the barcode with the object by accessing the captured 3D image data captured at a capture time associated with the capture time of the 2D image data.

[0016] In a variation of this embodiment, the 3D data acquisition subsystem includes a 3D camera, a time-of-flight 3D camera, a structured light 3D camera, or a machine learning model that processes one or more 2D images to create the 3D image data.

[0017] In a variation on this embodiment, the computing subsystem is communicatively coupled to one or more barcode scanner subsystems and / or the 3D data acquisition subsystem via a communications network.

[0018] In another embodiment, the invention is a method for tracking barcodes in space, comprising: capturing, with one or more two-dimensional (2D) imagers, 2D image data representing a 2D image of an initial environment of a scan tunnel; decoding, with one or more decode processors, barcodes identified in the captured 2D image data; accessing, from a three-dimensional (3D) data acquisition subsystem, captured 3D image data corresponding to a 3D representation of the environment; and identifying objects in the environment based on the captured 3D image data and associating the barcodes identified in the captured 2D image data with the objects to generate marked object data representing the objects in the environment.

[0019] A variation on this embodiment further includes the steps of: accessing from the 3D data acquisition subsystem next captured 3D image data corresponding to a next 3D representation of a next environment in the scan tunnel downstream from the first environment; identifying an object within the next environment from the next captured 3D image data; generating a successful scan indication in response to successfully associating a next decoded barcode corresponding to the next environment with the object in the next environment; and generating a failed scan indication in response to not successfully associating the next decoded barcode with the object in the next environment.

[0020] A variation on this embodiment further comprises capturing, with the one or more 2D imagers, next 2D image data representing a 2D image of the next environment; and attempting, with the one or more decode processors, to identify a barcode in the next 2D image data and, in response, decoding the barcode identified in the next 2D image data.

[0021] A variation on this embodiment further comprises generating the failed scan indication in response to the one or more 2D imagers failing to capture next 2D image data representing a 2D image of the next environment, or in response to the one or more 2D imagers capturing the next 2D image data representing the 2D image of the next environment but failing to identify a barcode within the next 2D image data.

[0022] A variation on this embodiment further comprises performing a four-dimensional (4D) projection of the marked object data within the scanning tunnel, the projection representing a predicted future position of the object within the scanning tunnel based on predicted or measured movement of the object.

[0023] A variation on this embodiment further includes capturing, with the one or more 2D imagers, next 2D image data representing a 2D image of a next environment in the scan tunnel downstream of the first environment; decoding, with the one or more decode processors, a next barcode identified in the next 2D image data; accessing, from the 3D data acquisition subsystem, next captured 3D image data corresponding to a next 3D representation of the next environment; identifying a next object in the next environment from the next captured 3D image data; associating the next barcode with the next object to generate next marked object data representing the next object; determining whether the next marked object data satisfies the 4D projection of the marked object data; and determining, when the next marked object data satisfies the 4D projection of the marked object data, that the decoding of the next barcode is either a decoding of a duplicate of the barcode.

[0024] A variation of this embodiment further includes the steps of determining whether the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment, and determining that the next marked object data satisfies the 4D projection of the marked object data if the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment.

[0025] A variation of this embodiment further includes a step of determining that the next marked object data does not satisfy the 4D projection of the marked object data and determining that the next object is different from the object if the next marked object data does not at least partially overlap with the 4D projection of the marked object data in the next environment.

[0026] A variation on this embodiment further comprises performing the 4D projection of the marked object data within the scanning tunnel based on the predicted movement of a moving surface on which the object resides.

[0027] A variation on this embodiment further comprises associating the barcode with the object by identifying a position and orientation of the barcode relative to the object based on the position and orientation of the one or more 2D imagers.

[0028] A variation on this embodiment further comprises associating the barcode with the object by accessing the captured 3D image data captured at a capture time associated with the capture time of the 2D image data.

[0029] The accompanying drawings, together with the following detailed description, which are incorporated in and form a part of this specification, serve to further explain embodiments of the concepts comprising the claimed invention(s) and to explain various principles and advantages of those embodiments, in which like reference numbers indicate identical or functionally similar elements throughout the different views. [Brief explanation of the drawings]

[0030] [Figure 1] 1 illustrates an exemplary environment in which a system / apparatus for barcode tracking according to embodiments described herein may be implemented, where a 3D object may be associated with 2D barcode data.

[0031] [Figure 2] FIG. 2 is a block diagram of example logic circuitry for implementing example methods and / or operations described herein.

[0032] [Figure 3A] FIG. 3A is a flowchart illustrating a method of object identification and barcode marking for use in tracking objects according to embodiments described herein.

[0033] [Figure 3B]FIG. 3B is a flowchart illustrating a method for associating a 3D object with a 2D image of a barcode, which may be implemented by the method of FIG. 3A, according to embodiments described herein.

[0034] [Figure 3C] FIG. 3C illustrates a virtual projection of an association of a barcode of a 3D object with a 2D image, which may be performed by the method of FIG. 3B, according to embodiments described herein.

[0035] [Figure 3D] FIG. 3D illustrates virtually marked object data that may be generated by the method of FIGS. 3A and 3B according to embodiments described herein.

[0036] [Figure 4A] 4A and 4B illustrate an exemplary environment in which a system / apparatus for tracking barcodes according to embodiments described herein may be implemented, showing a successful ( FIG. 4A ) and an unsuccessful ( FIG. 4B ) subsequent object scan. [Figure 4B] 4A and 4B illustrate an exemplary environment in which a system / apparatus for tracking barcodes according to embodiments described herein may be implemented, showing a successful ( FIG. 4A ) and an unsuccessful ( FIG. 4B ) subsequent object scan.

[0037] [Figure 5] FIG. 5 is a flowchart illustrating an object tracking method for identifying successful and unsuccessful tracking of an object on a scanning tunnel according to embodiments described herein.

[0038] [Figure 6] FIG. 6 is a flowchart illustrating a method for tracking an object on a scanning tunnel using 4D projections according to embodiments described herein.

[0039] [Figure 7]7 illustrates an exemplary environment in which a system / apparatus for barcode tracking according to embodiments described herein may be implemented. A 4D projection of an object is shown.

[0040] [Figure 8] 8 illustrates an exemplary environment in which a system / apparatus for barcode tracking according to embodiments described herein may be implemented. Successful tracking of an object using 4D projection is shown.

[0041] [Figure 9] 9 illustrates an exemplary environment in which a system / apparatus for barcode tracking according to embodiments described herein may be implemented. Unsuccessful tracking of an object using 4D projection is shown. DETAILED DESCRIPTION OF THE INVENTION

[0042] Those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present invention.

[0043] Components of the apparatus and methods are designated by conventional numerals in the drawings, where appropriate, and the drawings show only those specific details relevant to understanding the embodiments of the invention so as not to obscure the detailed disclosure, which disclosure will be readily apparent to those skilled in the art having the benefit of the descriptions herein.

[0044] As previously mentioned, machine vision systems can experience errors while tracking moving objects, particularly fast-moving objects transported through a facility on a conveyor belt or other transportation mechanism. In particular, machine vision systems may track an object but fail to track the barcode associated with that object. For example, in some cases, an object may move across the entire scan area on a conveyor belt but not be associated with or identified as a barcode. The object may be obscured by or confused as part of another object. In some cases, an object may be identified by a barcode by one machine vision camera but not by a subsequent machine vision camera positioned downstream on the conveyor belt. In yet another example, an object may be identified by a barcode by multiple machine vision cameras, but these cameras may fail to collectively recognize it as the same object at different conveyor belt locations and at different times, resulting in duplicate identifications of the same object.

[0045] An object of the present disclosure is to provide a system and method capable of determining whether an object is associated with one or more barcodes from a 2D image at the end of a scan field. If an object is associated with multiple barcodes from a 2D image, the duplicate associations are removed. If the object is not associated at all, the object is virtually marked so that follow-up action can be taken. More specifically, in some embodiments, an object of the present disclosure is to eliminate these and other problems in conventional machine vision systems by performing four-dimensional (4D) tracking of moving objects using two-dimensional (2D) barcode mapping and three-dimensional (3D) image data. Various machine vision cameras are positioned at different locations along a conveyor system, each capturing image data of a different portion of the conveyor system (possibly from different directions). The captured 2D image data can be used to identify a barcode or other indicia associated with the object from the 2D image data. The captured or generated 3D image data can be used to identify the location of the object in 3D space. As used herein, references to associating a barcode or other indicia with an object refer to associating an image of the barcode or other indicia obtained from 2D image data. In some embodiments, such association includes not only associating an image of the barcode or other indicia, but also associating decoded barcode (payload) data or decoded indicia data obtained therefrom. Using various data types, such as calibration data, position data, and orientation data of the various machine vision cameras, a geometric projection can be performed in both 3D space and time. The geometric projection of the initial 2D and / or 3D image data is then used to identify duplicate decode detection events, in which a barcode associated with the same object is erroneously scanned and decoded multiple times.Furthermore, the lack of sufficient geometric projections can be used to identify failed scan events, such as an object passing through a conveyor belt scan tunnel but not being scanned, or an object being tracked in only part of a conveyor belt scan tunnel but not the entire zone.

[0046] 1 illustrates an exemplary environment 100 in which a system / apparatus for tracking objects in space using multiple imaging devices according to embodiments described herein may be implemented. The exemplary environment 100 may generally be an industrial environment including different sets of imaging devices 102 / 104 positioned above or around a conveyor belt 106. The imaging devices 102 may each be machine vision cameras positioned at different locations along the conveyor belt 106 and having different orientations relative to the conveyor belt 106, configured to capture image data over a corresponding field of view. The imaging devices 102 may be 3D imaging devices, such as a 3D camera, that capture 3D image data of an environment. Collectively, the 3D imaging devices 102 form a three-dimensional (3D) data acquisition subsystem for the environment 100. For example, 3D imaging devices herein include time-of-flight 3D cameras in which the captured 3D image data is a map of the distance of an object to the camera, structured light 3D cameras in which a device projects a typically invisible pattern onto an object and an offset camera captures the pattern (with each point in the pattern shifted by an amount indicative of the object on which it will land), or virtual 3D cameras in which, for example, 2D image data is captured and passed to a trained neural network or other image processor to generate a 3D scene from the 2D image data.

[0047] The multiple imaging devices 104 may be 2D imagers, such as 2D color imagers or 2D grayscale imagers, each configured to capture 2D image data of a corresponding field of view. Typically, the imaging devices 104 are 2D imagers configured to identify barcodes within the 2D image data and decode the identified barcodes. Examples of such barcodes include 1D barcodes, 2D barcodes such as quick response (QR) codes, or other indicia identifiable within the 2D image data. Thus, in some embodiments, the multiple imaging devices 104 may collectively form a barcode scanner subsystem of the environment 100. A belt 106 may carry a target object 108 across an entry point 110 where an initial set of imaging devices 102, 104 is located. Captured images may be transmitted from the imaging devices to a server 112 for analysis, which facilitates tracking the object in space as it moves along the conveyor belt 106. A server 112 may be communicatively coupled to each of the imaging devices 102, 104, a target object 108 may move along the conveyor belt 106 past each of the sets of imaging devices 102, 104, and images captured from each of the imaging devices 102, 104 may be used by the server 112 to track and mark the object 108 at one or more different locations as it moves along the conveyor belt 106. The combination of the conveyor belt 106 and the sets of machine vision cameras 104 and barcode images 102 may be referred to herein as a "scan tunnel."

[0048] More specifically, the set of imaging devices 102, 104 may be organized in an array or other manner that allows for image capture along the entire working length, and may be arranged in a leader / follower configuration with a leader device (not shown) that may be configured to trigger the multiple machine vision cameras 104 to capture 3D image data of the target object 108, compile the image capture / inspection results of each machine vision camera, and transmit the results and / or captured images to the server 112. Each imager in the imaging devices 102, 104 stores an executable program (e.g., a "job") that includes information about the image capture parameters of the respective imager, such as focus, exposure, gain, details of the barcode type to be decoded, or a particular machine vision inspection procedure.

[0049] During operation, an object 108 on a conveyor belt 106 enters the scan tunnel 114 at an entry point 110, travels on the conveyor belt 106 at a conveyor speed Cs, and exits at an exit point 116. The object 108 is first imaged by the machine vision camera 102a and the 2D imaging device 104a, both of which are positioned to capture separate images of a first environment in the scan tunnel 114 that coincides with the entry point 110. In particular, the machine vision camera 102a captures 3D image data corresponding to a 3D representation of the environment surrounding the entry point 110. The imaging device 104a, which includes a 2D imager, is configured to capture the 2D images, identify barcodes present in the images, decode the barcodes, and transmit the 2D images and the decoded barcode data to the server 112. In the illustrated example, the object 108 includes a barcode 118 that is identified and decoded within the captured 2D image data from the imaging device 104a, which is transmitted to the server 112. Although one machine vision camera 102a and one 2D imaging device 104a are shown, it should be understood that any suitable number of devices may be used to capture all images of the target object 108, obtain multiple image captures of the target object 108, and / or otherwise capture sufficient image data of the target object 108 to enable the server 112 to accurately track and identify the target object 108.

[0050] As described in further detail below, the server 112 executes an object tracking application 120 to analyze the 3D image data received from the machine vision camera 102a and the 2D image data, including the barcode data, from the imaging device 104a, and perform an association between the two image data types to generate marked object data that is stored (saved) on the server 112. The marked object data thereby represents the 3D object data with a 2D image of the barcode projected onto the 3D object space, e.g., projected onto a 3D point cloud of the 3D object data. In some embodiments, the 2D image of the barcode is projected onto the 3D object space. In other embodiments, the 2D image of the barcode is projected, and the barcode data (payload) is further associated with the 3D object data, e.g., as metadata. Having generated the marked object data, the server 112 uses the marked object data to track the object 108 as it moves along the conveyor belt 106 to the exit point 116.

[0051] To facilitate tracking, in various embodiments, server 112 obtains the timestamp of the 2D image resulting in the decoded barcode and the location of the 2D imager 104a-104d that captured the 2D image. Additionally, server 112 accesses a series of 3D images (e.g., captured by one or more machine vision cameras 102a-102d) stored in its memory and their corresponding timestamps. Server 112 can then perform correlation between the barcode time and the 3D image(s) based on the known or measured speed of conveyor belt 106 and the locations of machine vision cameras 102a-102d, and derive the 3D scene corresponding to the viewpoint and timestamp of the 2D image resulting in the decode. The server 112 further receives the position of the decoded barcode in the 2D image via the object tracking application 120 and projects this position onto the object in the previously derived 3D scene via the imager's known viewpoint in 3D space. The server 112 then virtually marks the object as bearing the barcode whose content it received from the scanner, generating marked object data in virtual space. The marked object data can then be used to interrogate the 3D image of the object and the 2D image of the barcode captured downstream for further object tracking across the scanning tunnel through which the conveyor belt 106 passes.

[0052] Figure 2 is a block diagram illustrating an example logic circuit capable of implementing the example methods and / or operations described herein. As an example, the example logic circuit may be capable of implementing one or more components of Figures 1, 2, 4A, 4B, and 7-9. The example logic circuit of Figure 2 is a processing platform 220 capable of executing instructions to implement the operations of the example methods described herein, as may be represented, for example, by the flowcharts in the drawings accompanying this description. For example, other example logic circuits capable of implementing the operations of the example methods described herein include field programmable gate arrays (FPGAs) and application specific integrated circuits (ASICs). In one embodiment, processing platform 220 is implemented in server 112 of Figure 1.

[0053] The example processing platform 220 of FIG. 2 includes a processor 222, such as, for example, one or more microprocessors, controllers, and / or any suitable type of processor. The example processing platform 220 of FIG. 2 includes memory (e.g., volatile memory or non-volatile memory) 224 accessible by the processor 222 (e.g., via a memory controller). The example processor 222 interacts with the memory 224 to retrieve machine-readable instructions stored in the memory 224, for example, corresponding to the operations represented by the flowcharts of this disclosure. The memory 224 includes an object tracking application 224a having a 4D projection application 224b and projection data 224c, each of which is accessible by the example processor 222. The object tracking application 224a including the 4D projection application 224b may include rule-based instructions, artificial intelligence (AI) and / or machine learning-based models, and / or any other suitable algorithmic architecture or combination thereof configured to perform, for example, object tracking and 4D projection of object data. For example, the example processor 222 may access the memory 224 and execute the object tracking application 224a and the 4D projection application 224b when the 3D imager 230 and / or the 2D imager 240 capture and communicate sets of image data (via the imaging assemblies 239, 249) to the processing platform 220. Additionally or alternatively, machine-readable instructions corresponding to the example operations described herein may be stored on one or more removable media (e.g., compact discs, digital versatile discs, removable flash memory, etc.) that may be coupled to the processing platform 220 and provide access to the machine-readable instructions stored thereon.

[0054] 2 also includes a network interface 226 that enables communication with other machines, for example, via one or more networks. The example network interface 226 includes any suitable type of communication interface (e.g., wired and / or wireless interface) configured to operate according to any suitable protocol (e.g., Ethernet for wired communication and / or IEEE 802.11 for wireless communication).

[0055] 2 also includes an input / output (I / O) interface 228 that allows for receiving user input and communicating output data to a user. Such user input and communication may include, for example, any number of keyboards, mice, USB drives, optical drives, screens, touchscreens, etc.

[0056] The exemplary processing platform 220 is connected to a 3D imaging device 230 configured to capture 3D image data of a target object (e.g., the target object 108) and a 2D imaging device 240 configured to capture 2D image data of the target object (e.g., the target object 108), particularly 2D image data of a barcode on the target object. The imaging devices 230, 240 may be communicatively coupled to the platform 220 via a network 250.

[0057] The 3D imager 230 may be or include a machine vision camera 102a-20dd and may further include one or more processors 232, one or more memories 234, a network interface 236, an I / O interface 238, and an imaging assembly 239. The 3D imager 230 may optionally include an object tracking application 234a and a 4D projection application 224b.

[0058] 2D image capture device 240 may be or include image capture devices 104a-104d and may further include one or more processors 242, one or more memories 244, a network interface 246, an I / O interface 248, and an imaging assembly 249. 2D image capture device 240 may also optionally include object tracking application 234a and 4D projection application 224b.

[0059] Each of the imaging devices 230, 240 may include flash memory used to determine, store, or otherwise process imaging data / datasets and / or post-imaging data. The imaging devices 230, 240 may then receive, recognize, and / or otherwise interpret triggers that cause the imaging devices 230, 240 to capture images of a target object (e.g., the target object 108) according to configurations established via one or more job scripts. Once the images are captured and / or analyzed, the imaging devices 230, 240 may transmit the images and any associated data to the processing platform 220 via the network 250 for further analysis and / or storage according to the methods herein. In various embodiments, the imaging devices 230, 240 are “thin” camera devices that capture 3D and 2D image data, respectively, and offload the image data to the processing platform 220 for processing without further processing at the imaging device. In various other embodiments, the image capture devices 230, 240 may be “smart” cameras and / or may be configured to automatically perform image processing functions sufficient to implement all or part of the methods described herein.

[0060] Imaging assembly 239, 249 may include a digital camera and / or digital video camera for capturing or filming digital images and / or frames. Each digital image may include pixel data, which may be analyzed according to instructions executed by one or more processors 232, 234, as described herein. For example, the digital camera and / or digital video camera of imaging assembly 239 may be configured to film, capture, or otherwise generate 3D digital images, and, in at least some embodiments, may store such images in one or more memories 234. In some examples, imaging assembly 239 captures a series of 2D images that are processed to generate a 3D image. Such processing may occur in 3D imager 230 using an imaging processing application (not shown) or may occur in processing platform 220 within processing application 224d. Imaging assembly 249 may be configured to film, capture, or otherwise generate 2D digital images, which may be stored in one or more memories 244.

[0061] The imaging assembly 240 may include a photorealistic camera (not shown) or other 2D imager for capturing, sensing, or scanning 2D image data. The photorealistic camera may be an RGB (red, green, blue)-based camera for capturing 2D images having RGB-based pixel data. In various embodiments, the imaging assembly 230 includes a 3D camera (not shown) for capturing, sensing, or scanning 3D image data. The 3D camera may include an infrared (IR) projector and an associated IR camera for capturing, sensing, or scanning 3D image data / dataset.

[0062] Each of the one or more memories 224, 234, 244 may include one or more forms of volatile and / or non-volatile fixed and / or removable memory, such as read-only memory (ROM), electronically programmable read-only memory (EPROM), random access memory (RAM), erasable electronically programmable read-only memory (EEPROM), and / or other hard drives, flash memory, MicroSD cards, etc. Generally, computer programs or computer-based products, applications, or code (e.g., object tracking application 224a, 4D projection application 224b, image processing application 224d, and / or other computing instructions described herein) may be stored on a computer-usable storage medium or tangible, non-transitory computer-readable medium (e.g., standard random access memory (RAM), optical disk, universal serial bus (USB) drive, etc.) in which such computer-readable program code or computer instructions are embodied. wherein computer-readable program code or computer instructions may be installed for execution by one or more processors 222, 232, 242 (e.g., operating in conjunction with a respective operating system in one or more memories 224, 234, 244) or may otherwise be adapted to facilitate, implement, or execute machine-readable instructions, methods, processes, elements, or limitations, as shown, depicted, or described for purposes of the various flowcharts, illustrations, schematics, figures, and / or other disclosure herein. In this regard, program code may be implemented in any desired programming language, and may be implemented as machine code, assembly code, bytecode, interpretable source code, or the like (e.g., via Golang, Python, C, C++, C#, Objective-C, Java, Scala, ActionScript, JavaScript, HTML, CSS, XML, etc.).

[0063] The one or more memories 224, 234, 244 may store an operating system (OS) (e.g., Microsoft Windows, Linux, Unix, etc.) capable of facilitating functions, apps, methods, or other software as described herein. Additionally or alternatively, the object tracking application 224a, the 4D projection application 224b, and the image processing application 224d may be stored in an external database (not shown), which may be accessible or otherwise communicatively coupled to the processing platform 220 via the network 130. The one or more memories 224, 234, 244 may also store machine-readable instructions, including one or more applications, one or more software components, and / or one or more application programming interfaces (APIs), which may be implemented to facilitate or perform features, functions, or other disclosures described herein, such as methods, processes, elements, or limitations. As shown, depicted, or described in the various flowcharts, illustrations, schematics, figures, and / or other disclosures herein, for example, at least some of the applications, software components, or APIs may be, include, or be part of a machine-vision-based imaging application and may be configured to facilitate various functions described herein. One or more other applications may be envisioned and executed by one or more processors 122a, 124a, 126a.

[0064] One or more processors 222, 232, 242 may be connected to one or more memories 224, 234, 244 via a computer bus and may be responsible for transmitting electronic data, data packets, or other electronic signals between the one or more processors 222, 232, 242 and the one or more memories 224, 234, 244, and may implement or execute machine-readable instructions, methods, processes, elements, or limitations as shown, depicted, or described in the various flowcharts, illustrations, schematics, figures, and / or other disclosures herein.

[0065] The one or more processors 222, 232, 242 may interface with one or more memories 224, 234, 244 via a computer bus and may execute an operating system (OS). The one or more processors 222, 232, 242 may also interface with one or more memories 224, 234, 244 via the computer bus and may create, read, update, delete, or otherwise access or interact with data stored in the one or more memories 224, 234, 244 and / or in an external database (e.g., a relational database such as Oracle, DB2, MySQL, or a NoSQL-based database such as MongoDB). The data stored in the one or more memories 224, 234, 244 and / or in the external database may include all or any portion of the data or information described herein, including, for example, image data from images captured by the imaging assemblies 239, 249 and / or other suitable information.

[0066] The networking interfaces 226, 236, 246 may be configured to communicate (e.g., send and receive) data via one or more external / network ports to one or more networks, such as the network 250 described herein, or to local terminals. In some embodiments, the networking interfaces 226, 236, 246 may include client-server platform technologies, such as ASP.NET, Java J2EE, Ruby on Rails, Node.js, web services, or online APIs, and may be responsible for receiving and responding to electronic requests. The networking interfaces 226, 236, 246 may implement client-server platform technologies that may interact with one or more memories 224, 234, 244 (applications, components, APIs, data, etc., stored therein) via a computer bus and may implement or execute machine-readable instructions, methods, processes, elements, or limitations, as shown, depicted, or described in the various flowcharts, illustrations, schematics, drawings, and / or other disclosures herein.

[0067] According to some embodiments, networking interfaces 226, 236, 246 may include or interact with one or more transceivers (e.g., WWAN, WLAN, and / or WPAN transceivers) that function according to IEEE, 3GPP, or other standards, which may be used to transmit and receive data via an external / network port connected to network 250. In some embodiments, network 250 may include a private network or a local area network (LAN). Additionally or alternatively, network 250 may include a public network such as the Internet. In some embodiments, network 250 may include a router, wireless switch, or other such wireless connection point that communicates with processing platform 220 (via networking interface 226), 3D imager 230 (via networking interface 236), and 2D imager 240 (via networking interface 246) via wireless communications based on any one or more of a variety of wireless standards. Such wireless standards include, by way of non-limiting example, IEEE 802.11a / b / c / g (WIFI), BLUETOOTH standards, and the like.

[0068] I / O interfaces 228, 238, 248 may include or implement an operator interface configured to present information to and / or receive input from an administrator or operator. The operator interface may provide a display screen (not shown) that a user / operator may use to visualize any images, graphics, text, data, features, pixels, and / or other suitable visual images or information. For example, processing platform 220, 3D imager 230, and / or 2D imager 240 may, at least in part, include, implement, have access to, render, or otherwise expose a graphical user interface (GUI) for displaying images, graphics, text, data, features, pixels, and / or other suitable visual images or information on a display screen. I / O interfaces 228, 238, 248 may also include I / O components (e.g., ports, capacitive or resistive touch-sensitive input panels, keys, buttons, lights, LEDs, any number of keyboards, mice, USB drives, optical drives, screens, touchscreens, etc.) that may be directly / indirectly accessible through or directly / indirectly attached to processing platform 220, 3D imager 230, and / or 2D imager 240.

[0069] In general, the object tracking application 224a and the image processing application 224d may include and / or otherwise comprise executable instructions (e.g., via one or more processors 222) that allow a user to configure the machine vision job and / or imaging settings of the image capture devices 230, 240. For example, the applications 224a, 224d may render a graphical user interface (GUI) on a display (e.g., I / O interface 228) or on a connected device, with which a user may interact to change various settings and modify the machine vision job, input data, tracking parameters, image capture device position and orientation data, conveyor belt operating parameters, etc.

[0070] The object tracking application 224a can be configured to address the shortcomings of conventional systems by performing a number of object tracking operations, examples of which are described with reference to Figures 3A-6, in which methods are also shown.

[0071] For example, to facilitate object tracking along the scan tunnel 114, the object tracking application 224a may be executed by the processing platform 220, functioning as a computing subsystem, to perform an object identification and marking process, such as the embodiment illustrated in method 300 of FIG. 3A. At block 302, the method 300 captures 2D image data representing a 2D image of the environment along the scan tunnel with a 2D imaging device, such as the 2D imaging device 240 or imaging devices 104a-104d. In some embodiments, the 2D image data is captured and received only individually at block 302. At block 304, the 2D imaging device identifies and decodes a barcode (e.g., barcode 118 of the target object 108) within the captured 2D image data and appends a timestamp of the moment the image was captured. The timestamp may be embedded within the 2D image data from the 2D imaging device or may be determined by the computing subsystem. In the latter case, for example, in block 304, the computing subsystem may determine a timestamp from time data embedded within the 2D image data and perform a projection on the time data using calibration data (e.g., the positions of various associated 2D and 3D imaging devices) to determine a timestamp that matches the time frame of the captured 3D image data (e.g., the 3D imaging device is located downstream or upstream of the 2D imaging device that captures the 2D image data).

[0072] In block 306, the computing subsystem further accesses 3D image data corresponding to the same environment in which the 2D image data was captured. In some embodiments, in block 306, method 300 obtains a timestamp of the 2D image data (resulting in the barcode to be decoded) from block 304 and the location of the 2D imaging device (e.g., imaging device 104a-104d) that captured the 2D image data. From there, in block 306, method 300 can access a series of 3D image data from memory (e.g., captured by one or more of machine vision cameras 102a-102d) and a timestamp corresponding to each captured 3D image data. The 3D image data stored in memory can be a series of consecutively captured 3D image data or a video stream of 3D image data, from which method 300 identifies the desired 3D image data as corresponding to the 2D image data. Block 306 can identify the desired 3D image data by comparing the timestamp of the 2D image data with the timestamp of the 3D image data to identify a match. In some examples, the comparison is performed taking into account the known or measured speed of the conveyor belt 106 and the positions of the machine vision cameras 102a-102d to derive a 3D scene (environment) corresponding to the viewpoint and timestamp of the resulting decoded 2D image. That is, the 3D image data of the 3D scene corresponding to the 2D image may have a different (e.g., later) timestamp than that of the 2D image because the barcode is moving on the conveyor belt 106. In any case, the 3D image data accessed in block 306 corresponds to the 3D space of the environment in which the 2D image data of the barcode was captured. Furthermore, the 3D image data may be derived from the captured 3D image data in the sense that 3D image data from two or more consecutive times is used to construct an interpolated, extrapolated, or predicted 3D image. This may be important, for example, when one or more objects are observed moving relative to the conveyor belt, such as a box tipping over or one box falling on another.

[0073] In block 308, the computing subsystem identifies one or more objects (e.g., target object 108) within the 3D image data received from block 306. For example, block 308 may receive the location of a decoded barcode within the 2D image data and project the barcode location onto the object through the known viewpoint of a machine-vision camera (e.g., machine-vision cameras 102a-102d) in 3D space. An exemplary projection is illustrated in FIG. 3B. Block 308 virtually marks the object as having a barcode (i.e., having a 2D image of the barcode) whose content was received from the 2D imaging device, generating marked-object data in virtual space. The marked-object data is stored in block 310 for use in verifying the 3D image of the object and the 2D image of the barcode captured downstream to further track the object through the scanning tunnel through which the conveyor belt 106 passes. That is, in block 310, the computing subsystem may store marked object data for use in tracking (movement, presence, etc.) an object (e.g., target object 108) on a scan tunnel (e.g., scan tunnel 114). For example, the processing platform 220 may store the marked object data in memory 224 for access by the tracking application 224a or the 4D projection application 224b.

[0074] The processing of block 308 may be implemented in various manners. For example, method 300 may identify one or more objects within the 3D image data from block 306. From there, method 300 may perform an association of the object data for the object(s) with the 2D image data from block 304. For example, in block 308, method 300 may identify a particular object within the object data that corresponds to a barcode in the 2D image data, and generate virtual marked object data therefrom. The virtual marked object data is stored for use in tracking in block 312. In some embodiments, the object data is associated with the 2D image of the barcode by identifying an intersection or overlap between the 2D image of the barcode and the surface of the object in the object data.

[0075] FIG. 3B illustrates a method 330 that may be implemented by block 308 to perform object association and marked object data generation. In block 332, method 330 (e.g., implemented by a computing subsystem) obtains 2D image data and calibration data (e.g., position and orientation data for the 2D imager) for a corresponding 2D imaging device and projects a 2D image of a decoded barcode within a 3D cone. For example, the 2D image may be a 2D image of a 1D barcode, a QR code, or other decodable indicia (including decodable text, graphics, symbols, etc.). FIG. 3C illustrates an exemplary 3D imaging device 352 virtually projecting a 2D image 354 of a barcode within a 3D cone 350. In one embodiment, block 332 may obtain a barcode rectangle from the 2D image data and project the rectangle into the 3D cone in 3D space using calibration data. In block 334, a 3D scene corresponding to the timestamp of the decoded barcode, i.e., the time the barcode's 2D image data was captured, is constructed. In block 336, the 3D cone (e.g., cone 350) is virtually projected into the 3D scene, and an intersection of the 3D cone with the surface of an object in the 3D scene is identified (e.g., the intersection of 3D cone 350 with surface 356 of object 358 (partially shown)). Block 336 may identify the intersection using 3D image data corresponding to the 3D scene, the timestamp of the barcode decode event, and corresponding 3D imaging device calibration data. That is, block 336 (or block 334) may use 3D imaging device calibration data (e.g., position and orientation data of the 3D imaging device relative to position and orientation data of the 2D imaging device and / or conveyor belt) to find the object surface intersection. At block 338, method 330 associates objects that intersect the 3D cone with the decoded barcode of the 2D image data (i.e., the 2D image of the decoded barcode). In this way, block 338 can associate only the appropriate object with the barcode, even if the 3D image data corresponding to the 3D scene includes multiple objects.At block 338, the method 330 may further create virtual marked object data defining the association. Figure 3D shows an example of virtually generated marked object data 360 including a target object 362 and an associated barcode 364.

[0076] In some embodiments, the object tracking application 244a tracks an object along the scan tunnel and determines whether the object has properly exited the scan tunnel. FIGS. 4A and 4B illustrate an environment 450 similar to the environment 100 of FIG. 1. A target object 108 is moved to the next environment at the exit point 116 and is either successfully tracked ( FIG. 4A ) or not successfully tracked ( FIG. 4B ), resulting in a failed tracking event. FIG. 5 illustrates a flowchart of an example method 400 that may be implemented by the object tracking application 224a for identifying successful and unsuccessful tracking of an object along the scan tunnel. In block 402, a computing subsystem acquires marked object data (e.g., corresponding to the target object 108) generated by the process 300 of FIG. 3A. In block 404, the next (subsequent) captured 3D image data corresponding to the next (subsequent) environment along the scan tunnel is accessed. For example, 3D image data may be captured by machine vision camera 102d over the environment surrounding exit point 116, downstream from entry point 110. Of course, 3D image data may be captured from any machine vision camera 102b-102d downstream from initial machine vision camera 102a. In some embodiments, subsequent 3D image data may be captured from the same imaging device that captured the 3D image data (resulting in the marked object data obtained in block 402).

[0077] To trigger a determination of whether the tracking attempt was successful or unsuccessful, in block 406, method 400 identifies the object in the next (subsequent) 3D image data. In some embodiments, block 404 may capture 3D image data over a time window, and for each captured 3D image data, block 406 attempts to identify the object in the corresponding 3D image data. If block 406 does not identify the object after a predetermined scan window time has elapsed, a fault condition is determined (not shown), and a fault signal is sent to an operator and / or the fault signal data is stored in a computing subsystem. Assuming an object is identified in the next (subsequent) 3D image data, in block 408, method 400 attempts to mark the corresponding object data with a barcode. In particular, method 400 attempts to associate a next (subsequent) decoded barcode (i.e., a 2D image of a next (subsequent) decoded barcode) obtained from 2D image data captured in the same environment as the 3D image data (e.g., from 2D image data captured by imaging device 104d). If block 408 fails to associate the next (subsequent) barcode with the object from block 406, an indication of scan failure is generated to the user and / or stored in the computing subsystem. Alternatively, if block 408 successfully associates the next (subsequent) barcode with the next (subsequent) 3D image data, control passes to block 410. A failed attempt at block 408 can occur in a variety of ways. If none of the imaging devices (e.g., imaging devices 104a-104d) captures the next (subsequent) 2D image data of the next (subsequent) environment, the attempt at block 408 fails. If one or more 2D image data pieces are captured by one or more imaging devices (e.g., imaging devices 104a-104d), but block 408 does not identify a barcode on any of those devices, the attempt at block 408 fails. Figure 4B illustrates an example environment 450 in which 3D image data for an object 452 is captured by machine vision camera 102d, but imaging device 104d (or imaging devices 104c, 104b) fails to capture 2D image data that includes a barcode.

[0078] In block 410, the next (subsequent) object and associated barcode from block 408 are compared to the marked object data to determine whether the next (subsequent) object is the same as the originally scanned object used to generate the marked object data. In some embodiments, in block 410, method 400 performs a comparison of the 3D image data (e.g., point cloud data) and the barcode data (e.g., decoded barcode payload data) to determine whether a match exists. If either the point cloud data or the barcode data from the marked object data does not match the associated next (subsequent) object and next (subsequent) barcode (referred to as the next (subsequent) marked object data) (e.g., as shown in FIG. 4B ), block 410 determines that a match does not exist, and an indication of a failed scan is generated in block 412. Such a scenario may occur if the object is damaged or rotated, if the barcode is peeled or damaged, or if the object is otherwise affected. In such a scenario, the object may need to be provided special handling to prevent it from being passed on to the next stage down the conveyor belt. In some embodiments (not shown), objects that fail at block 410 may be indicated with a laser spot or laser symbol. That is, block 412 may send an indication signal to an external laser device to provide a visual indication on the object that failed to scan. In some embodiments, block 412 may send a signal to a conveyor belt offloader that removes the object from the conveyor belt after it passes the exit point. In any case, if the next (subsequent) object matches the marked object data (e.g., as shown in FIG. 4A ), block 414 displays an indication of successful scanning to the user.

[0079] In various embodiments, the object tracking application 224a includes a 4D projection application 224b that uses the projection data 224c to track an object through the scan tunnel and determine when the object has successfully passed through the scan tunnel. In particular, the 4D projection application is designed to receive marked object data and perform a 4D projection of the marked object data. The 4D projection represents the predicted future position of the object within the scan tunnel based on the object's predicted movement through the conveyor belt.

[0080] 6 shows a flowchart of a method 500 for tracking an object on a scanning tunnel using 4D projection. In block 502, a computing subsystem acquires previously stored marked object data (e.g., corresponding to target 550 and associated barcode 552 of environment 600 of FIG. 7). In block 503, 4D projection is performed on the marked object data using various types of acquired, sensed, and / or stored projection data. The 4D transformation converts the marked object data from a 3D position at time t1 to a downstream 3D position at time t2 {(x1, y1, z1, t1) → (x2, y2, z2, t2)}, resulting in projected marked object data 560 (with an associated projected barcode 562). To affect the projection, block 503 may use various types of projection data (e.g., projection data 224c) including the position and orientation in 3D space of each of the imaging devices (3D imaging device and 2D imaging device), taking into account their relative positions to each other, to the scan tunnel, and to the conveyor belt (e.g., conveyor belt 106). The projection data may include the speed C1 of the conveyor belt.

[0081] In block 504, the next (subsequent) captured 3D image data corresponding to the next (subsequent) environment along the scan tunnel is accessed. For example, the 3D image data may have been captured by machine vision camera 102d over the environment surrounding exit point 116, downstream from entry point 110. The 3D image data may be captured from any machine vision camera 102b-102d downstream from the initial machine vision camera 102a. In some embodiments, the next (subsequent) 3D image data may be captured from the same imaging device that captured the 3D image data (resulting in the marked object data obtained in block 502).

[0082] In block 506, method 500 identifies an object in the next (subsequent) 3D image data. Similar to blocks 404 and 406, in some embodiments, block 504 may capture 3D image data over a time window, and for each captured 3D image data, block 506 attempts to identify the object in the corresponding 3D image data. If block 506 does not identify an object after a predetermined scan window time has elapsed, a fault condition is determined (not shown), and a fault signal is sent to an operator and / or the fault signal data is stored in a computing subsystem.

[0083] Assuming an object is identified in the next (subsequent) 3D image data, in block 508, method 500 analyzes the object data to attempt to determine whether the object data corresponds to a 4D projection of the marked object data projected in block 503. An example is shown in FIG. 8. In block 504, 3D image data is acquired from machine vision camera 102d, and in block 506, an object 580 is identified in the 3D image data. In particular, block 506 identifies the object data in the 3D space into which the projection is made. Next, block 508 compares the object data defining the 3D position of object 580 with the 3D position of the projected marked object 560. If the 3D position of object 580 substantially overlaps the 3D position of projection 560, block 508 determines that the object is the same as the initial object 550. Block 508 may be configured to set the predetermined amount of substantial overlap as at least 50% overlap, at least 60% overlap, at least 70% overlap, at least 80% overlap, or at least 90% overlap. If block 508 determines that substantial overlap exists (see, e.g., FIG. 8 ) and therefore object 580 is identical to object 550, method 500 may generate an indication of successful scanning in block 510. Additionally, to prevent an object from being barcode scanned twice and counted as two different objects, block 510 discards all associated barcode data obtained from all subsequently captured 2D images. That is, if identification is successful in block 508, block 510 discards all decodings of barcode 582 captured within the 2D image data. In particular, in some embodiments, including the illustrated embodiment, this discarding occurs regardless of whether the barcode 582 is associated with object data along the same plane as the original marked object data 550 / 552.

[0084] In some embodiments, to determine the correspondence, block 508 uses both the object data from the next (subsequent) 3D image data and the barcode data from the next (subsequent) 2D image data. For example, in some instances, block 508 analyzes the next (subsequent) 2D image data with a computing subsystem to identify and decode a next (subsequent) barcode identified within the 2D image data. Block 508 may then perform an association between the barcode and the object data from block 506 to generate next (subsequent) marked object data (e.g., the association between object 580 and barcode 582 in FIG. 8 ). In block 508, method 500 compares the next (subsequent) marked object data with the projected marked object data to determine whether a match exists. Thus, in such embodiments, the location of the associated barcode may be taken into consideration. In some embodiments, if the barcode in the next (subsequent) marked object data is at a position or orientation that is different from the expected position or expected orientation of the barcode based on the projected marked object data, block 508 may determine that no match exists (e.g., object 580 does not meet projected object 560).

[0085] In response to no match being found at block 508 (see, e.g., FIG. 9 ), control passes to block 512. Block 512 may be configured in a variety of different ways. In the illustrated embodiment, at block 512, method 500 allows a next (subsequent) barcode associated with the object to be stored and / or indicated as a successful scan event, provided that the next (subsequent) identified object and associated barcode are different from the initial object and barcode. In some embodiments (not shown), block 512 may be configured to reject the next (subsequent) object (e.g., object 580) and generate a failed scan indication.

[0086] It should be understood that the operations of methods 300, 400, 500 may be performed in any suitable order and any suitable number of times to modify a program running on any of the 2D imaging devices, 3D imaging devices, and / or any other suitable devices described herein, or a combination thereof.

[0087] [Additional Considerations] The preceding description refers to block diagrams in the accompanying drawings. Alternative implementations of the embodiments represented by the block diagrams include one or more additional or alternative elements, processes, and / or devices. Additionally or alternatively, one or more of the illustrative blocks in the diagrams may be combined, divided, rearranged, or omitted. Components represented by blocks in the diagrams may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. In some examples, at least one of the components represented by blocks is implemented by logic circuitry. As used herein, the term “logic circuitry” is expressly defined as a physical device including at least one hardware component configured to control one or more machines and / or perform the operations of one or more machines (e.g., via operation according to a predetermined configuration and / or via execution of stored machine-readable instructions). Examples of logic circuits include one or more processors, one or more coprocessors, one or more microprocessors, one or more controllers, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more microcontroller units (MCUs), one or more hardware accelerators, one or more special-purpose (dedicated) computer chips, and one or more system-on-chip (SoC) devices. Some exemplary logic circuits, such as an ASIC or FPGA, are hardware specifically configured to perform operations (e.g., one or more operations described herein and, if present, represented by the flowcharts of this disclosure). Some exemplary logic circuits are hardware that executes machine-readable instructions (instructions) to perform operations (e.g., one or more operations described herein and, if present, represented by the flowcharts of this disclosure).Some example logic circuits include a combination of specially configured hardware and hardware that executes machine-readable instructions. The foregoing description refers to various operations described herein and to flowcharts that may accompany the description to illustrate the flow of those operations. Any such flowcharts represent example methods disclosed herein. In some examples, methods represented by flowcharts implement apparatuses represented by block diagrams. Alternative implementations of example methods disclosed herein may include additional or alternative operations. Furthermore, operations of alternative implementations of methods disclosed herein may be combined, divided, rearranged, or omitted. In some examples, operations described herein are implemented by machine-readable instructions (e.g., software and / or firmware) stored on a medium (e.g., a tangible machine-readable medium) for execution by one or more logic circuits (e.g., processors). In some examples, operations described herein are performed by one or more configurations of one or more specially designed logic circuits (e.g., ASICs). In some examples, the operations described herein are implemented by a combination of one or more logic circuits specially designed for execution by the one or more logic circuits and machine-readable instructions (instructions) stored on a medium (e.g., a tangible machine-readable medium).

[0088] As used herein, each of the terms “tangible machine-readable medium,” “non-transitory machine-readable medium,” and “machine-readable storage device” is expressly defined as a storage medium (e.g., a hard disk drive, a digital versatile disk (DVD), a compact disk (CD), flash memory, a read-only memory (ROM), a random access memory (RAM), etc.) on which machine-readable instructions (e.g., program code in the form of software and / or firmware) are stored for any suitable period of time (e.g., permanently, for a long period (e.g., while a program associated with the machine-readable instructions is being executed), and / or for a short period (e.g., while the machine-readable instructions are cached and / or during a buffering process)). Furthermore, as used herein, each of the terms “tangible machine-readable medium,” “non-transitory machine-readable medium,” and “machine-readable storage device” is expressly defined to exclude a propagating signal. That is, as used in the claims, none of the terms "tangible machine-readable medium," "non-transitory machine-readable medium," and "machine-readable storage device" may be read as being implemented by a propagating signal.

[0089] The foregoing specification describes specific embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present invention as set forth in the claims. Accordingly, the specification and drawings should be understood in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present teachings. Furthermore, the described embodiments / examples / implementations should not be construed as mutually exclusive, but instead should be understood as potentially combinable where such combination is in any way permissible. In other words, any feature disclosed in any of the foregoing embodiments / examples / implementations may be included in any of the other foregoing embodiments / examples / implementations.

[0090] Benefits, advantages, solutions to problems, and any elements that may cause or make more noticeable any benefit, advantage, or solution should not be construed as critical, necessary, or essential features or elements of any or all claims. The claimed invention is defined solely by the appended claims, including any amendments made during the pendency of this application and all equivalents of those claims as issued.

[0091] Furthermore, in this document, related terms such as first and second, above and below, etc. may be used only to distinguish one entity or operation from another and may not necessarily require or imply an actual relationship or ordering between such entities or operations. The terms "comprises," "comprising," "has," "having," "include," "including," "contains," "containing," or any other variations thereof, are intended to cover a non-exclusive inclusion. A description of a process, method, article, or apparatus as comprising, having, including, or containing a list of elements does not include only those elements, but may include other elements not expressly listed and other elements inherent in such process, method, article, or apparatus. An element preceded by "comprises ... a," "has ...," "includes ... a," or "contains ... a" does not, without further constraints, preclude the presence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, or contains the element. The terms "a" and "an" are defined as one or more, unless expressly stated otherwise. The terms "substantially," "essentially," "approximately," "about," and any other variations thereof are defined as close as understood by one of ordinary skill in the art, and in one non-limiting embodiment, such terms are defined as within 10%, in another embodiment within 5%, in another embodiment within 1%, and in another embodiment within 0.5%. The term "coupled," as used herein, is defined as connected, but not necessarily directly, and not necessarily mechanically. A device or structure "configured" in a certain way is configured in at least that way, but may also be configured in ways not listed.

[0092] This Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. It may also be noted that in the foregoing Detailed Description, various features are grouped together in various embodiments for the purpose of facilitating this disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. The following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

Claims

1. 1. A system for tracking barcodes in space, comprising: a three-dimensional (3D) data acquisition subsystem; a two-dimensional (2D) imaging subsystem including one or more 2D imagers and one or more decode processors; a computing subsystem including one or more processors and a non-transitory computer-readable storage medium storing instructions; Equipped with the 2D imaging subsystem: capturing, with the one or more 2D imagers, 2D image data representative of a 2D image of an initial environment of the scan tunnel; decoding, with the one or more decoding processors, barcodes identified within the captured 2D image data; and is oriented and configured as follows: The instructions, when executed by the one or more processors, cause the computing system to: accessing, from the 3D data acquisition subsystem, captured 3D image data corresponding to a 3D representation of the environment; Identifying objects within the environment based on the captured 3D image data and associating the barcodes identified in the captured 2D image data with the objects to generate marked object data representing the objects within the environment. A system characterized by:

2. The non-transitory computer-readable storage medium stores instructions: The instructions, when executed by the one or more processors, cause the computing system to: accessing from the 3D data acquisition subsystem next captured 3D image data corresponding to a next 3D representation of a next environment in the scan tunnel downstream from the first environment; identifying an object within the subsequent environment from the subsequent captured 3D image data; generating a successful scan indication in response to successfully associating a decoded bar code corresponding to the next environment with the object within the next environment; generating a failed scan indication in response to not successfully associating the decoded barcode with the object within the next environment; 2. The system of claim 1.

3. The non-transitory computer-readable storage medium stores instructions: The instructions, when executed by the one or more processors, cause the computing system to: causing the one or more 2D imagers to capture subsequent 2D image data representative of a 2D image of the subsequent environment; causing the one or more decoding processors to attempt to identify a barcode within the subsequent 2D image data, and in response, decoding the barcode identified within the subsequent 2D image data.

3. The system of claim 2.

4. The non-transitory computer-readable storage medium stores instructions: The instructions, when executed by the one or more processors, cause the computing system to: generating the failed scan indication in response to the one or more 2D imagers failing to capture next 2D image data representing a 2D image of the next environment, or in response to the one or more 2D imagers capturing the next 2D image data representing the 2D image of the next environment but failing to identify a barcode within the next 2D image data; 3. The system of claim 2.

5. The non-transitory computer-readable storage medium stores instructions: The instructions, when executed by the one or more processors, cause the computing system to: performing a four-dimensional (4D) projection of the marked object data within the scan tunnel; The projection represents the predicted future position of the object within the scanning tunnel based on the predicted or measured movement of the object.

2. The system of claim 1.

6. The 2D imaging subsystem further comprises: capturing, with the one or more 2D imagers, subsequent 2D image data representing a 2D image of a subsequent environment in the scan tunnel downstream from the first environment; and decoding, with the one or more decoding processors, a next barcode identified within the next 2D image data. It is structured as follows: The non-transitory computer-readable storage medium stores instructions: The instructions, when executed by the one or more processors, cause the computing system to: accessing, from the 3D data acquisition subsystem, next captured 3D image data corresponding to a next 3D representation of the next environment; identifying a next object within the next environment from the next captured 3D image data; associating the next barcode with the next object to generate next marked object data representing the next object; determining whether the next marked object data satisfies the 4D projection of the marked object data; determining whether the decoding of the next barcode or the decoding of a duplicate of the barcode is when the next marked object data satisfies the 4D projection of the marked object data; 6. The system of claim 5.

7. The instructions, when executed by the one or more processors, cause the computing system to: determining whether the next marked object data satisfies the 4D projection of the marked object data by determining whether the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment, and determining that the next marked object data satisfies the 4D projection of the marked object data if the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment; The system of claim 6 .

8. The instructions, when executed by the one or more processors, cause the computing system to: determining that the next marked object data does not satisfy the 4D projection of the marked object data when the next marked object data does not at least partially overlap with the 4D projection of the marked object data in the next environment, thereby determining that the next object is different from the object; 8. The system of claim 7.

9. The system further comprises a moving surface configured to move across the scan tunnel; The instructions, when executed by the one or more processors, cause the computing system to: performing the 4D projection of the marked object data within the scanning tunnel based on the predicted movement of the moving surface on which the object resides.

6. The system of claim 5.

10. The moving surface is a conveyor belt that moves substantially linearly through the environment.

10. The system of claim 9.

11. The instructions, when executed by the one or more processors, cause the computing system to: Associating the barcode with the object by identifying a position and orientation of the barcode relative to the object based on the position and orientation of the one or more 2D imagers.

2. The system of claim 1.

12. The instructions, when executed by the one or more processors, cause the computing system to: Associating the barcode with the object by accessing the captured 3D image data captured at a capture time associated with the capture time of the 2D image data.

2. The system of claim 1.

13. The 3D data acquisition subsystem includes a 3D camera, a time-of-flight 3D camera, a structured light 3D camera, or a machine learning model that processes one or more 2D images to create the 3D image data.

2. The system of claim 1.

14. The computing subsystem is communicatively coupled to one or more barcode scanner subsystems and / or the 3D data acquisition subsystem via a communications network.

2. The system of claim 1.

15. 1. A method for tracking barcodes in space, comprising: capturing, with one or more two-dimensional (2D) imagers, 2D image data representative of a 2D image of an initial environment of the scan tunnel; decoding, with one or more decoding processors, barcodes identified within the captured 2D image data; accessing captured three-dimensional (3D) image data corresponding to a 3D representation of the environment from a 3D data acquisition subsystem; identifying objects in the environment based on the captured 3D image data and associating the barcodes identified in the captured 2D image data with the objects to generate marked object data representing the objects in the environment; A method comprising:

16. accessing, from the 3D data acquisition subsystem, next captured 3D image data corresponding to a next 3D representation of a next environment in the scan tunnel downstream from the first environment; identifying an object within the subsequent environment from the subsequent captured 3D image data; generating a successful scan indication in response to successfully associating a next decoded barcode corresponding to the next environment with the object within the next environment; generating a failed scan indication in response to not successfully associating the next decoded barcode with the object within the next environment; 16. The method of claim 15 further comprising:

17. capturing, with the one or more 2D imagers, subsequent 2D image data representative of a 2D image of the subsequent environment; attempting with the one or more decoding processors to identify barcodes within the subsequent 2D image data and responsively decoding the barcodes identified within the subsequent 2D image data; 17. The method of claim 16, further comprising:

18. generating the failed scan indication in response to the one or more 2D imagers failing to capture next 2D image data representing a 2D image of the next environment, or in response to the one or more 2D imagers capturing the next 2D image data representing the 2D image of the next environment but failing to identify a barcode within the next 2D image data.

17. The method of claim 16, further comprising:

19. performing a four-dimensional (4D) projection of the marked object data within the scan tunnel. Further provided with The projection represents the predicted future position of the object within the scanning tunnel based on the predicted or measured movement of the object.

16. The method of claim 15.

20. capturing, with the one or more 2D imagers, subsequent 2D image data representing a 2D image of a subsequent environment in the scan tunnel downstream from the first environment; decoding, with the one or more decoding processors, a next barcode identified within the next 2D image data; accessing, from the 3D data acquisition subsystem, next captured 3D image data corresponding to a next 3D representation of the next environment; identifying a next object within the next environment from the next captured 3D image data; associating the next barcode with the next object to generate next marked object data representing the next object; determining whether the next marked object data satisfies the 4D projection of the marked object data; determining that the decoding of the next barcode or a duplicate of the barcode is when the next marked object data satisfies the 4D projection of the marked object data; 20. The method of claim 19 further comprising:

21. determining whether the next marked object data satisfies the 4D projection of the marked object data by determining whether the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment and determining that the next marked object data satisfies the 4D projection of the marked object data if the next marked object data at least partially overlaps with the 4D projection of the marked object data in the next environment; 21. The method of claim 20.

22. determining that the next marked object data does not satisfy the 4D projection of the marked object data when the next marked object data does not at least partially overlap with the 4D projection of the marked object data in the next environment, and determining that the next object is different from the object; 22. The method of claim 21 .

23. performing the 4D projection of the marked object data within the scanning tunnel based on the predicted movement of a moving surface on which the object resides.

20. The method of claim 19 further comprising:

24. associating the barcode with the object by identifying a position and orientation of the barcode relative to the object based on the position and orientation of the one or more 2D imagers.

16. The method of claim 15 further comprising:

25. associating the barcode with the object by accessing the captured 3D image data captured at a capture time associated with the capture time of the 2D image data.

16. The method of claim 15 further comprising:

26. The 3D data acquisition subsystem includes a 3D camera, a time-of-flight 3D camera, a structured light 3D camera, or a machine learning model that processes one or more 2D images to create the 3D image data.

16. The method of claim 15.

Images