System and method for assigning symbols to objects

Abstract

The present invention provides an imaging system including a machine vision system for acquiring and analyzing images of objects or symbols (e.g., barcodes), and a method for assigning symbols to objects in an image. [Solution] A method for assigning symbols to objects located in an image includes the steps of: receiving an image captured by an imaging device; receiving the three-dimensional (3D) positions of one or more points corresponding to pose information indicating the 3D orientation of an object in the image in a first coordinate system; mapping the 3D positions of one or more points of the object to 2D positions in the image; and assigning symbols to objects based on the relationship between the 2D positions of symbols in the image and the 2D positions of one or more points of the object in the image.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit and priority of U.S. Provisional Application No. 63 / 215,229, filed on June 25, 2021, entitled "Systems and Methods for Assigning Symbols to Objects", which is hereby incorporated by reference in its entirety for all purposes.

[0002] Statement Regarding Federally Sponsored Research Not Applicable

[0003] The present technology relates to an imaging system including a machine vision system configured to acquire and analyze images of objects or symbols (e.g., barcodes).

Background Art

[0004] Machine vision systems are generally configured to capture images of objects and symbols and analyze those images to identify objects or decode symbols. Thus, machine vision systems generally include one or more devices for image acquisition and image processing. In conventional applications, these devices can be used to acquire images and analyze the acquired images, for purposes such as decoding imaged symbols such as barcodes and text. Depending on the situation, machine vision and other imaging systems can be used to acquire images that are larger than the field of view (FOV) of the corresponding imaging device or alternatively, images of objects that may be moving relative to the imaging device.

Summary of the Invention

[0005] According to embodiments, a method for assigning symbols to objects in an image includes the step of receiving the image captured by an imaging device, wherein the symbols are located in the image. The method further includes the steps of: receiving the three-dimensional (3D) positions of one or more points corresponding to pose information indicating the 3D orientation of the object in the image in a first coordinate system; mapping the 3D positions of the one or more points of the object to 2D positions in the image; and assigning the symbols to the object based on the relationship between the 2D positions of the symbols in the image and the 2D positions of the one or more points of the object in the image. In some embodiments, the mapping is based on the 3D positions of the one or more points in the first coordinate space.

[0006] In some embodiments, the method may further include the steps of: determining the surface of the object based on the 2D positions of one or more points of the object in the image; and assigning the symbol to the surface of the object based on the relationship between the 2D position of the symbol in the image and the surface of the object. In some embodiments, the method may further include the steps of: determining that the symbol is associated with a plurality of images; aggregating the assignments of the symbol to each of the plurality of images; and determining whether at least one of the assignments of the symbol is different from the remaining assignments of the symbol. In some embodiments, the method may further include the step of determining the edges of the object in the image based on the image acquisition data of the image. In some embodiments, the method may further include the step of determining a confidence score for the symbol assignments. In some embodiments, the 3D positions of one or more points may be received from a 3D sensor.

[0007] In some embodiments, the method may further include the step of determining whether the plurality of objects overlap in the image. In some embodiments, the image includes an object having a first boundary with a margin and a second object having a second boundary with a second margin. The method may further include the step of determining whether the first boundary and the second boundary overlap in the image. In some embodiments, the 3D positions of the one or more points are acquired in a first time and the image is acquired in a second time. Mapping the 3D positions of the one or more points to the 2D positions in the image may include mapping the 3D positions of the one or more points from the first time to the second time. In some embodiments, the orientation information may include mapping the 3D positions of the one or more points to the 2D positions in the image, which includes mapping the 3D positions of the one or more points from the first time to the second time. In some embodiments, the orientation information may include point cloud data.

[0008] According to another embodiment, a system for assigning symbols to objects in an image comprises a calibrated imaging device configured to capture an image, and a processor device. The processor device is programmed to perform the steps of: receiving the image captured by the calibrated imaging device, wherein the symbols are located in the image; receiving the three-dimensional (3D) positions of one or more points corresponding to pose information indicating the 3D orientation of the object in the image in a first coordinate system; mapping the 3D positions of the one or more points of the object to 2D positions in the image; and assigning the symbols to the object based on the relationship between the 2D positions of the symbols in the image and the 2D positions of the one or more points of the object in the image. In some embodiments, the mapping is based on the 3D positions of the one or more points in the first coordinate space.

[0009] In some embodiments, the system further comprises a conveyor configured to support and transport the object, and a motion measuring device coupled to the conveyor and configured to measure the movement of the conveyor. In some embodiments, the system further comprises a 3D sensor configured to measure the 3D position of the one or more points. In some embodiments, the orientation information may include the angles of the object in the first coordinate space. In some embodiments, the orientation information may include point cloud data. In some embodiments, the processor device may further be programmed to perform the steps of: determining the surface of the object based on the 2D positions of one or more points of the object in the image; and assigning the symbol to the surface of the object based on the relationship between the 2D position of the symbol in the image and the surface of the object. In some embodiments, the one processor device may further be programmed to perform the steps of: determining that the symbol is associated with a plurality of images; aggregating the assignments of the symbol to each of the plurality of images; and determining whether at least one of the assignments of the symbol is different from the remaining assignments of the symbol.

[0010] In some embodiments, the image associated with the symbol may include a plurality of objects, and the processor device may further be programmed to determine whether the plurality of objects overlap in the image. In some embodiments, the image may include an object having a first boundary with a margin, and a second object having a second boundary with a second margin. The processor device is further programmed to determine whether the first boundary and the second boundary overlap in the image. In some embodiments, assigning the symbol to an object may include assigning the symbol to a surface.

[0011] According to another embodiment, a method for assigning symbols to objects in an image includes the step of receiving the image captured by an imaging device. The symbols are located in the image. The method includes the steps of: receiving the three-dimensional (3D) positions of one or more points corresponding to pose information indicating the 3D orientation of one or more objects in a first coordinate system; mapping the 3D positions of the one or more points of the object to 2D positions in the image in a second coordinate space; determining the surface of the object based on the 2D positions of the one or more points of the object in the image in the second coordinate space; and assigning the symbols to the surface based on the relationship between the 2D positions of the symbols in the image and the 2D positions of the one or more points of the object in the image. In some embodiments, assigning the symbols to the surface may include determining the intersection between the surface and the image in the second coordinate space. In some embodiments, the method may further include determining a confidence score for the symbol assignment. In some embodiments, the mapping is based on the 3D positions of the one or more points in the first coordinate space.

[0012] The various purposes, features, and advantages of the disclosed subject matter can be more fully understood by referring to the following detailed description of the disclosed subject matter, when considered in relation to the following drawings in which similar reference numbers identify similar elements. [Brief explanation of the drawing]

[0013] [Figure 1A] Figure 1A shows an example of a system according to one embodiment of this technology that captures multiple images of each face of an object and assigns symbols to the object. [Figure 1B] Figure 1B shows an example of a system according to one embodiment of this technology that captures multiple images of each face of an object and assigns symbols to the object. [Figure 2A] Figure 2A shows another example of a system according to one embodiment of the present technology that captures multiple images of each face of an object and assigns symbols to the object. [Figure 2B] Figure 2B shows an example of a set of images acquired from the bank of imaging devices in the system shown in Figure 2A, according to one embodiment of the present technology. [Figure 3] Figure 3 shows another example of a system according to one embodiment of the present technology that captures multiple images of each face of an object and assigns symbols to the object. [Figure 4] Figure 4 shows an example of a system for assigning symbols to objects, according to several embodiments of the disclosed subject matter. [Figure 5] Figure 5 shows an example of hardware that can be used to implement the image processing apparatus, server, and imaging apparatus shown in Figure 3, according to several embodiments of the disclosed subject matter. [Figure 6A] Figure 6A shows a method for assigning a symbol to an object using images of multiple faces of the object, according to one embodiment of the present technology. [Figure 6B] Figure 6B illustrates a method, according to one embodiment of the present technology, for resolving overlapping surfaces of multiple objects in an image in order to assign a symbol to one of the multiple objects. [Figure 6C] Figure 6C shows a method for aggregating the symbol assignment results for symbols according to one embodiment of this technology. [Figure 7] Figure 7 shows an example of an image containing two objects according to one embodiment of the present technology, wherein at least one object contains an assigned symbol. [Figure 8A-B] Figures 8A-B show an example of an image having two objects with overlapping surfaces according to one embodiment of the present technology, where at least one object includes an assigned symbol. [Figure 8C] Figure 8C shows an example of an image having two objects with overlapping surfaces according to one embodiment of the present technology, where at least one object includes an assigned symbol. [Figure 9] Figure 9 shows an example image illustrating the assignment of a symbol to one of two objects having overlapping surfaces, according to one embodiment of the present technology. [Figure 10]FIG. 10 shows an example of determining the assignment of symbols identified in an image including two objects having overlapping surfaces by using image data. [Figure 11] FIG. 11 shows a method of aggregating symbol assignment results for symbols according to an embodiment of the present technology. [Figure 12A] FIG. 12A shows an example of a factory calibration setup that can be used to find the conversion between an image coordinate space and a calibration target coordinate space. [Figure 12B] FIG. 12B shows an example of a coordinate space and other aspects for a calibration process including factory calibration and on-site calibration that includes capturing multiple images of each surface of an object and assigning symbols to the object according to an embodiment of the present technology. [Figure 12C] FIG. 12C shows an example of a field calibration process related to different positions of one or more calibration targets according to an embodiment of the present technology. [Figure 13A] FIG. 13A shows an example of the correspondence between the coordinates of an object in a 3D coordinate space associated with a system that captures multiple images of each surface of the object and the coordinates of the object in a 2D coordinate space associated with an imaging device. [Figure 13B] FIG. 13B is a diagram showing another example of the correspondence between the coordinates of an object in a three-dimensional coordinate space and the coordinates of the object in a two-dimensional coordinate space. [Figure 14] FIG. 14 shows an example of determining the visible surfaces of one or more objects in an image according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] As conveyor technology improves and objects are moved by conveyors with narrower gaps (i.e., spacing between objects) (e.g., belt conveyors) or other conveyor systems, imaging devices are increasingly capturing single images containing multiple objects. For example, a photoeye can control the trigger cycle of an imaging device, so that image acquisition of a particular object begins when the leading edge (or other boundary feature) of the object crosses the photoeye and ends when the trailing edge (or other boundary feature) of the object crosses the photoeye. If there are relatively small gaps between adjacent objects on the relevant conveyor, the imaging device may inadvertently capture multiple objects during a single trigger cycle. Furthermore, symbols placed on objects (such as barcodes) may need to be decoded using the captured image, for example, to guide appropriate further actions on the relevant object. Thus, while it is important to identify which symbol is associated with which object, it may be difficult to determine exactly which object corresponds to a particular symbol in the captured image.

[0015] A machine vision system may include multiple imaging devices. For example, in some embodiments, a machine vision system may be implemented in a tunnel arrangement (or system), which may include a structure in which each imaging device is positioned at a certain angle to the conveyor to obtain an angled field of view (FOV). Image data of a common scene may be acquired using an imaging device. In some embodiments, the common scene may include relatively small areas, such as on a table or individual sections of a conveyor belt. In some embodiments, the FOVs of partial imaging devices may overlap in a tunnel system. Although the following description refers to a tunnel system or device, it should be understood that the systems and methods for assigning symbols to objects in images described herein are also applicable to other types of machine vision system devices.

[0016] Figure 1A shows an example of a system 100 according to one embodiment of the present technology, which captures multiple images of each face of an object and assigns symbols to the object. In some embodiments, the system 100 includes assigning symbols to objects (e.g., objects 118a, 118b), The system can be configured to evaluate symbols (e.g., barcodes, two-dimensional (2D) codes, fiducials, hazardous materials codes, machine-readable codes, etc.) on objects (e.g., objects 118a, 118b) moving through the tunnel 102, such as symbol 120 on body 118a. In some embodiments, symbol 120 is a flat 2D barcode on the top surface of object 118a, and objects 118a and 118b are substantially rectangular boxes. Additionally or alternatively, in some embodiments, the object being imaged can be any suitable shape, and all kinds of symbols and the positions of symbols, including non-direct part mark (DPM) symbols and DPM symbols, on the top surface or other surfaces of the object can be imaged and evaluated.

[0017] In Figure 1A, objects 118a and 118b are placed on a conveyor 116 configured to move objects 118a and 118b horizontally through a tunnel 102 at a relatively predictable continuous rate or a variable rate measured by a device such as an encoder or other motion measuring device. Additionally or alternatively, objects can be moved through the tunnel 102 by other means (e.g., nonlinear movement). In some embodiments, the conveyor 116 may include a conveyor belt. In some embodiments, the conveyor 116 may also consist of other types of conveying systems.

[0018] In some embodiments, system 100 may include an imaging device 112 and an image processing device 132. For example, system 100 may include multiple imaging devices in a tunnel arrangement, typically shown via imaging devices 112a, 112b, 112c (e.g., implementing a portion of tunnel 102), each having a field of view ("FOV") typically shown via FOVs 114a, 114b, 114c, including a portion of conveyor 116. In some embodiments, each imaging device 112 may be positioned at an angle to the top or side of the conveyor (e.g., at an angle to the normal direction of the symbols on the surfaces of objects 118a and 118b, or to the direction of movement), resulting in an oblique FOV. Similarly, portions of the FOV may overlap with other FOVs (e.g., FOVs 114a and 114b). In such embodiments, the system 100 may be configured to acquire images of one or more of a plurality of faces of objects 118a and / or 118b as the objects 118a and / or 118b are moved by the conveyor 116. In some embodiments, the captured images can be used to identify symbols on each object (e.g., symbol 120) and / or to assign symbols to each object, and then (if necessary) decode or analyze. In some embodiments, a gap in the conveyor 116 (not shown) can be used to facilitate imaging of the bottom surface of the objects using an imaging device or array of imaging devices (not shown) located beneath the conveyor 116 (e.g., described in U.S. Patent Application Publication 2019 / 0333259 filed April 25, 2018, which is incorporated herein by reference in its entirety). In some embodiments, images captured from the bottom surface of the objects can also be used to identify symbols on the objects and to assign symbols to each object, and then (if necessary) decode.Although two arrays consisting of three imaging devices 112 are shown to image the top of objects 118a and 118b, and four arrays consisting of two imaging devices 112 are shown to image the faces of objects 118a and 118b, this is merely an example, and images of various faces of an object can be captured using any suitable number of imaging devices. For example, each array may contain four or more imaging devices. Furthermore, although the imaging devices 112 are generally shown to image objects 118a and 118b without mirrors to change the direction of the FOV, this is merely an example, and the direction of the FOV of one or more imaging devices can be changed using one or more fixed mirrors and / or steerable mirrors, as described below with respect to Figures 2A to 3, which may facilitate the reduction of the vertical or lateral distance between the imaging devices and the objects in the tunnel 102. For example, the imaging device 112a can be positioned so that its optical axis is parallel to the conveyor 116, and one or more mirrors can be placed above the tunnel 102 to redirect the field of view (FOV) from the imaging device 112a towards the front and top of objects inside the tunnel 102.

[0019] In some embodiments, the imaging device 112 may be implemented using any suitable type of imaging device. For example, the imaging device 112 may be implemented using a 2D imaging device (e.g., a 2D camera), such as an area scan camera and / or a line scan camera. In some embodiments, the imaging device 112 may be an integrated system including a lens assembly and an imaging device such as a CCD or CMOS sensor. In some embodiments, each imaging device 112 may include one or more image sensors, at least one lens configuration, and at least one control device (e.g., a processor device) configured to perform computational operations on the image sensors. Each of the imaging devices 112a, 112b, or 112c may selectively acquire image data from different fields of view (FOV), regions of interest ("ROI"), or combinations thereof. In some embodiments, the system 100 can be used to acquire multiple images of each face of an object, and one or more images may contain multiple objects. Using the multiple images of each face, as described below with respect to Figures 6A to 6C, symbols in the images can be assigned to objects in the images. The object 118 may be associated with one or more symbols, such as a barcode or a QR code (registered trademark). In some embodiments, the system 100 can be configured to facilitate imaging of the bottom surface of an object supported by the conveyor 116 (for example, the surface of the object 118a resting on the conveyor 116). For example, the conveyor 116 may be mounted with gaps (not shown).

[0020] In some embodiments, a gap 122 is provided between objects 118a and 118b. In various implementations, the size of the gap between objects varies. In some implementations, the gap between objects may be substantially the same across all sets of objects in the system, or it may represent a fixed minimum size across all sets of objects in the system. In some embodiments, a smaller gap size may be used to maximize the throughput of the system. However, in some implementations, the size of the gap (e.g., gap 122) and the dimensions of adjacent sets of objects (e.g., objects 118a and 118b) may affect the usefulness of the resulting images captured by the imaging device 112, including for the analysis of symbols of specific objects. In some configurations, the imaging device (e.g., imaging device 112) may capture images such that a first symbol located on a first object appears in the same image as a second symbol located on a second object. Furthermore, if the gap size is small, the first object may overlap with the second object in the image (i.e., occlusion). This can occur, for example, when the size of gap 122 is relatively small and the first object (e.g., object 118a) is relatively tall. When such overlap occurs, it becomes necessary to determine whether the detected symbol corresponds to a particular object (i.e., whether the symbol should be considered "on" or "off" for the object). This can sometimes be extremely difficult to judge.

[0021] In some embodiments, system 100 may include a three-dimensional (3D) sensor (not shown), also referred herein as a dimensioner or dimension sensing system, which can measure the dimensions of an object moving toward tunnel 102 on conveyor 116, such dimensions may be used (e.g., by image processing device 132) in the process of assigning symbols to objects in images captured as one or more objects move through tunnel 102. Furthermore, system 100 may include a device (e.g., an encoder or other motion measuring device, not shown) for tracking the physical movement of objects (e.g., objects 118a, 118b) moving through tunnel 102 on conveyor 116. Figure 1B shows an example of a system according to one embodiment of the Art that captures multiple images of each face of an object and assigns codes to the object. Figure 1B shows a simplified diagram of system 140 showing an example of the arrangement of a 3D sensor (or dimensioner) and motion measuring device (e.g., an encoder) relative to the tunnel. As described above, the system 140 may include a 3D sensor (or dimensioner) 150 and a motion measuring device 152. In the illustrated example, the conveyor 116 is configured to move objects 118d and 118e along the direction indicated by the arrow 154 to pass the 3D sensor 150 before they are imaged by one or more imaging devices 112. In the illustrated embodiment, a gap 156 is provided between objects 118d and 118e, and the image processing device 132 may be able to communicate with the imaging device 112, the 3D sensor 150, and the motion measuring device 152. The 3D sensor (or dimensioner) 150 may be configured to determine the dimensions and / or position of an object (e.g., object 118d or 118e) supported by the support structure 116 at a particular point in time. For example, the 3D sensor 150 can be configured to determine the distance from the 3D sensor 150 to the top surface of an object, and can be configured to determine the size and / or orientation of the surface facing the 3D sensor 150. In some embodiments, the 3D sensor 150 can be implemented using various techniques.For example, the 3D sensor 150 can be implemented using a 3D camera (e.g., a structured light 3D camera, a continuous time of flight 3D camera, etc.). As another example, the 3D sensor 150 can be implemented using a laser scanning system (e.g., a LiDAR system). In a particular example, the 3D sensor 150 can be implemented using the 3D-A1000 system available from Cognex Corporation. In some embodiments, the 3D sensor (or dimensioner) (e.g., calculated from a time-of-flight sensor or stereo) may be implemented in a single device or enclosure with an imaging device (e.g., a 2D camera), and in some embodiments, a processor (e.g., which may be used as an image processing device) may be implemented in a device with the 3D sensor and imaging device.

[0022] In some embodiments, the 3D sensor 150 can determine the 3D coordinates of each corner of an object in a coordinate space defined by referencing one or more parts of the system 140. For example, the 3D sensor 150 can determine the 3D coordinates of each of the eight corners of an object that is at least substantially rectangular in shape in a Cartesian coordinate space defined with the 3D sensor 150 as the origin. As another example, the 3D sensor 150 can determine the 3D coordinates of each of the eight corners of an object that is at least substantially rectangular in shape in a Cartesian coordinate space defined with respect to the conveyor 116 (for example, having an origin starting from the center of the conveyor 116). As yet another example, the 3D sensor 150 can determine the 3D coordinates of the bounding box (e.g., having eight corners) of an object that is not rectangular in shape in any suitable Cartesian coordinate space (e.g., defined with respect to the conveyor 116, defined with respect to the 3D sensor 150, etc.). For example, the 3D sensor 150 can identify bounding boxes around any suitable non-cuboidal shape, such as a plastic bag, a padded envelope (jiffy mailer), an envelope, a cylindrical shape (such as a cylinder), a triangular prism, a square prism other than a cuboid, a pentagonal prism, a hexagonal prism, a tire (or other shape that can be approximated as a toroid). In some embodiments, the 3D sensor 150 can be configured to classify objects as cuboidal or non-cuboidal shapes, identifying the corners of the object if it is a cuboidal shape, and identifying the corners of a cuboidal bounding box if it is a non-cuboidal shape. In some embodiments, the 3D sensor 150 can be configured to classify objects into a specific class within a group of common objects (e.g., cuboids, cylinders, triangular prisms, hexagonal prisms, padded envelopes, plastic bags, tires, etc.). In some such embodiments, the 3D sensor 150 can be configured to determine the bounding box based on the classified shape. In some embodiments, the 3D sensor 150 can determine the 3D coordinates of non-cuboidal shapes such as a soft-sided envelope, a pyramidal shape (e.g., with four corners), and other prisms (e.g., a triangular prism with six corners, a non-cuboidal square prism, a pentagonal prism with ten corners, a hexagonal prism with twelve corners, etc.).

[0023] In addition or alternatively, in some embodiments, the 3D sensor 150 can provide raw data (e.g., point cloud data, distance data, etc.) to a control device (e.g., an image processing device 132, one or more imaging devices, etc., described later) that can determine the 3D coordinates of one or more points of an object.

[0024] In some embodiments, a motion measuring device 152 (e.g., an encoder) is linked to the conveyor 116 and the imaging device 112, and can provide an electronic signal indicating the amount of movement of the conveyor 116 to the imaging device 112 and / or the image processing device 132, and objects 118d, 118e are supported on it for a known period of time. This may be useful, for example, to adjust the acquisition of images of a particular object (e.g., objects 118d, 118e) based on the calculated position of the object relative to the field of view of the associated imaging device (e.g., imaging device 112). In some embodiments, the motion measuring device 152 may be configured to generate pulse counts that can be used to identify the position of the conveyor 116 along the direction of the arrow 154. For example, the motion measuring device 152 may provide pulse counts to the image processing device 132 to identify and track the position of objects (e.g., objects 118d, 118e) on the conveyor 116. In some embodiments, the motion measuring device 152 can increment the pulse count each time the conveyor 116 moves a predetermined distance (pulse count distance) in the direction of the arrow 154. In some embodiments, the position of an object can be determined based on its initial position, the change in pulse count, and the pulse count distance.

[0025] Returning to Figure 1A, in some embodiments, each imaging device (e.g., imaging device 112) is calibrated (e.g., as described below in relation to Figures 12A to 12C) to facilitate the mapping of the 3D positions of each corner of an object supported by the conveyor 116 (e.g., object 118) to 2D positions in the image captured by the imaging device. In some embodiments including a steerable mirror, such calibration can be performed by orienting the steerable mirror in a particular direction.

[0026] In some embodiments, the image processing device 132 (or control device) can coordinate the operation of various components of the system 100. For example, the image processing device 132 can cause a 3D sensor (e.g., the 3D sensor (or dimensioner) 150 shown in Figure 1B) to acquire the dimensions of an object located on the conveyor 116, and cause the imaging device 112 to capture images of each surface. In some embodiments, the image processing device 132 can control the detailed operation of each imaging device, for example, by controlling a steerable mirror, or by providing a trigger signal (e.g., when an object is expected to be within the field of view of the imaging device) to cause the imaging device to capture an image at a specific time. Alternatively, in some embodiments, other devices (e.g., a processor included in each imaging device, a separate controller device, etc.) can control the detailed operation of each imaging device. For example, the image processing device 132 (and / or other suitable device) can provide trigger signals to each imaging device and / or 3D sensor (e.g., the 3D sensor (or dimensioner) 150 shown in Figure 1B), and the processor of each imaging device can be configured to execute a pre-specified image acquisition sequence over a given region of interest in response to the trigger. The system 100 may also include one or more light sources (not shown) to illuminate the surface of an object, and the operation of such light sources can also be coordinated by a central device (e.g., the image processing device 132) and / or the control can be decentralized (e.g., the imaging device can control the operation of one or more light sources, and processors associated with one or more light sources can control the operation of the light sources, etc.). For example, in some embodiments, the system 100 may be configured to acquire images of multiple faces of an object simultaneously (e.g., simultaneously or over a common time interval), such as as part of a single trigger event. For example, each imaging device 112 may be configured to acquire each set of one or more images over a common time interval. Additionally or alternatively, in some embodiments, the imaging device 112 may be configured to acquire an image based on a single trigger event.For example, based on a sensor (e.g., a contact sensor, presence sensor, or imaging device) that determines whether object 118 has entered the field of view of the imaging device 112, the imaging device 112 can simultaneously acquire images of each surface of object 118.

[0027] In some embodiments, each imaging device 112 can generate a set of images depicting a particular face or various faces of an object (e.g., object 118) supported by the conveyor 116. In some embodiments, (for example, as described below in relation to Figures 13A and 13B showing multiple boxes on the conveyor) an image processing device 132 can map the 3D positions of one or more corners of object 118 to 2D positions in each image within the set of images output by each imaging device. In some embodiments, the image processing device can generate a mask (e.g., a bitmask where 1 indicates the presence of a particular face and 0 indicates the absence of a particular face) based on the 2D position of each corner, identifying which part of the image is associated with each face. In some embodiments, the 3D positions of one or more corners of a target object (such as object 118a), the 3D positions of one or more corners of an object 118c (leading object) in front of the target object 118a on the conveyor 116, and / or the 3D positions of one or more corners of an object 118b (following object) behind the target object 118a on the conveyor 116 may be mapped to 2D positions in each image within the image set output by each imaging device. Thus, if an image captures multiple objects (118a, 118b, 118c), one or more corners of each object in the image can be mapped to a 2D image.

[0028] In some embodiments, the image processing device 132 can identify which object 118 in an image contains the symbol 120 based on the mapping of object corners from the 3D coordinate space of the image to the image coordinate space, or other suitable information that can represent a surface, such as a plurality of planes (e.g., each plane corresponds to a face, and the intersections of the plurality of planes represent edges and corners), the coordinates of a single corner associated with height, width, and depth, or a plurality of polygons. For example, if the symbol in the captured image is within the mapped 2D position of an object corner, the image processing device 132 can determine that the symbol in the image is on an object. As another example, the image processing device 132 can identify which surface of an object contains the symbol based on the 2D position of an object corner. In some embodiments, each surface visible from a particular imaging device FOV of a given image may be determined based on which surfaces intersect each other. In some embodiments, the image processing device 132 may be configured to identify when two or more objects (e.g., surfaces of objects) overlap (i.e., occlusion) in an image. In one example, overlapping objects (as further described below with respect to Figure 8A) may be determined based on whether the surfaces of the objects in the image intersect each other, or intersect if a predetermined margin is given. In some embodiments, if an image containing identified symbols contains two or more overlapping objects, the relative positions of the imaging device's FOV and / or 2D image data from the image can be used to resolve the overlapping surfaces of the objects. For example, image data from an image can be used to determine the edges (or surfaces of objects) of one or more objects in the image using an image processing method, and the determined edges can be used to determine whether a symbol is positioned on a particular object. In some embodiments, the image processing system 132 may also be configured to aggregate the symbol assignment results for each symbol in a set of identified symbols in a set of images captured by the imaging device 112 to determine whether there are any inconsistencies between the symbol assignment results for each identified symbol.For example, in the case of a symbol identified in multiple images, the symbol may be assigned differently in at least one of the images in which it appears. In some embodiments, for a symbol with conflicting assignment results, if the symbol assignment results include an image in which no objects overlap, the image processing system 132 may be configured to select the image in which no objects overlap. In some embodiments, a confidence level (or score) may be determined for the symbol assignment to a symbol in a particular image.

[0029] As described above, the field of view (FOV) of one or more imaging devices can be changed using one or more fixed mirrors and / or steerable mirrors. This may facilitate the reduction of the vertical or lateral distance between the imaging device and an object in the tunnel 102. Figure 2A shows another example of a system according to one embodiment of the art that captures multiple images of each face of an object and assigns a code to the object. The system 200 includes multiple banks of imaging devices 212, 214, 216, 218, 220, 222 and multiple mirrors 224, 226, 228, 230 within the tunnel configuration 202. For example, the banks of imaging devices shown in Figure 2A include a left trailing bank 212, a left leading bank 214, an upper trailing bank 216, an upper leading bank 218, a right trailing bank 220, and a right leading bank 222. In the illustrated embodiment, each bank 212, 214, 216, 218, 220, 222 includes four imaging devices configured to capture images of one or more faces of an object (e.g., object 208a) and various FOVs of one or more faces of the object. For example, the upper leading bank 216 and mirror 228 may be configured to capture images of the top and back faces of the object using imaging devices 234, 236, 238, and 240. In the illustrated embodiment, the banks of imaging devices 212, 214, 216, 218, 220, 222 and the mirrors 224, 226, 228, 230 can be mechanically coupled to a support structure 242 on the conveyor 204. While the illustrated mounting positions of the bank imaging devices 212, 214, 216, 218, 220, and 222 relative to each other may be advantageous, in some embodiments, imaging devices for imaging different faces of an object can be reoriented relative to the positions shown in Figure 2A (e.g., imaging devices can be offset, imaging devices can be positioned at corners instead of faces, etc.). Similarly, while there are advantages associated with using four imaging devices per bank, each configured to acquire image data from one or more faces of an object, in some embodiments, specific imaging devices can be configured to acquire images of multiple faces of an object using a different number or arrangement of imaging devices, different mirror arrangements (e.g., the use of steerable mirrors, the use of additional fixed mirrors, etc.).In some embodiments, the imaging device can be dedicated to acquiring images of multiple faces of an object, including acquisition regions that overlap with those of other imaging devices included in the same system.

[0030] In some embodiments, the system 200 also includes a 3D sensor (or dimensioner) 206 and an image processing device 232. As described above, a plurality of objects 208a, 208b, and 208c are supported within the conveyor 204 and can move through the tunnel 202 along the direction indicated by the arrow 210. In some embodiments, the banks of imaging devices 212, 214, 216, 218, 220, and 222 (and each imaging device within the banks) can generate a series of images depicting the FOV of a particular face of an object (e.g., object 208a) supported by the conveyor 204 or a variety of FOVs. Figure 2B shows an example of a set of images acquired from the banks of imaging devices in the system of Figure 2A according to one embodiment of the Art. In Figure 2B, an example of a set of images 260 captured using a series of imaging devices on an object (e.g., object 208a) on the conveyor is shown. In the illustrated example, the set of images is acquired by an upper successor bank of imaging device 216, which is configured to capture images of the top and back surfaces of an object (e.g., object 208a) using imaging devices 234, 236, 238, 240 and a mirror 228 (shown in Figure 2A). An example of the image set 260 is shown as a grid, where each column represents an image acquired using one of the imaging devices 234, 236, 238, and 240 of the imaging device bank. Each row represents an image acquired by imaging devices 234, 236, 238, and 240, respectively, at a specific point in time as a first object 262 (e.g., a preceding object), a second object 263 (e.g., a target object), and a third object 264 (e.g., a successor object) move through a tunnel (e.g., tunnel 202 shown in Figure 2A).For example, row 266 shows a first image acquired by each imaging device in the bank at a first time point, row 268 shows a second image acquired by each imaging device in the bank at a second time point, row 270 shows a third image acquired by each imaging device in the bank at a third time point, row 272 shows a fourth image acquired by each imaging device in the bank at a fourth time point, and row 272 shows a fifth image acquired by each imaging device in the bank at a fifth time point. In the illustrated example, based on the size of the gaps between the first object 262 and the second object 263 on the conveyor and between the second object 263 and the third object 264, the first object 262 appears in the first image acquired for the second (or target) object 263 in the fifth image 274, and the third object 264 begins to appear in the image acquired for the second (or target) object 263 in the fifth image 274.

[0031] In some embodiments, each imaging device (e.g., imaging devices in imaging device banks 212, 214, 216, 218, 220, 222) can be calibrated (for example, as described below in relation to Figures 12A to 12C) to facilitate the mapping of the 3D positions of each corner of an object (e.g., object 208) supported by the conveyor 204 to 2D positions in the images captured by the imaging devices.

[0032] However, although Figures 1A to 2A show movable dynamic support structures (e.g., conveyor 116, conveyor 204), in some embodiments, stationary support structures may be used to support an object imaged by one or more imaging devices. Figure 3 shows another example of a system according to one embodiment of the present art that captures multiple images of each face of an object and assigns symbols to the object. In some embodiments, system 300 may include a plurality of imaging devices 302, 304, 306, 308, 310, and 312, each including one or more image sensors, at least one lens configuration, and at least one control device (e.g., a processor device) configured to perform computational operations on the image sensor. In some embodiments, imaging devices 302, 304, 306, 308, 310, and / or 312 include and / or can be associated with a steerable mirror (for example, as described in U.S. Patent Application No. 17 / 071,636 filed October 13, 2020, which is incorporated herein by reference in its entirety). Each of the imaging devices 302, 304, 306, 308, 310, and / or 312 can selectively acquire image data from different fields of view (FOV) corresponding to different orientations of the associated steerable mirror. In some embodiments, the system 300 can be used to acquire multiple images of each face of an object.

[0033] In some embodiments, system 300 can be used to acquire images of multiple objects presented for image acquisition. For example, system 300 may include support structures supporting each of the imaging devices 302, 304, 306, 308, 310, and 312, and a platform 316 configured to support one or more objects 318, 334, 336 to be imaged (each object 318, 334, 336 may be associated with one or more symbols such as a barcode or QR code®). For example, a transport system (not shown) including one or more robotic arms (e.g., a robot bin picker) may be used to place the multiple objects (e.g., in bins or other containers) onto platform 316. In some embodiments, the support structures may be configured as cage-type support structures. However, this is merely an example, and the support structures can be realized in a variety of configurations. In some embodiments, the support platform 316 may be configured to facilitate imaging of the bottom surfaces of one or more objects supported by the support platform 316 (e.g., the surfaces of the objects (e.g., objects 318, 334, or 336) that are in contact with the platform 316). For example, the support platform 316 can be implemented using a transparent platform, a mesh or grid platform, an open center platform, or any other suitable configuration. Except for the presence of the support platform 316, acquiring images of the bottom surfaces can be substantially the same as acquiring images of other surfaces of the objects.

[0034] In some embodiments, imaging devices 302, 304, 306, 308, 310, and / or 312 can be oriented to acquire images of specific faces of an object placed on the support platform 316 using the FOV of the imaging devices, thereby allowing each face of an object (e.g., object 318) positioned on and supported by the support platform 316 to be imaged by the imaging devices 302, 304, 306, 308, 310, and / or 312. For example, the imaging device 302 can be mechanically coupled to a support structure on the support platform 316 and oriented toward the upper surface of the support platform 316; the imaging device 304 can be mechanically coupled to a support structure beneath the support platform 316; and the imaging devices 306, 308, 310, and / or 312 can each be mechanically coupled to the side of the support structure, so that the respective FOVs of the imaging devices 306, 308, 310, and / or 312 face the lateral side of the support platform 316.

[0035] In some embodiments, each imaging device may be configured to have an optical axis substantially parallel to one other imaging device (for example, when the controllable mirror is in the neutral position) and perpendicular to the other imaging device. For example, imaging devices 302 and 304 may be configured to face each other (for example, so that the imaging devices have substantially parallel optical axes), and the other imaging device may be configured to have an optical axis perpendicular to the optical axes of imaging devices 302 and 304.

[0036] While the illustrated mounting positions for each of the imaging devices 302, 304, 306, 308, 310, and 312 may be advantageous, in some embodiments, imaging devices for imaging different faces of an object can be reoriented relative to the positions shown in Figure 3 (e.g., the imaging devices can be offset, or the imaging devices can be positioned at corners rather than faces). Similarly, while there may be advantages associated with using six imaging devices, each configured to acquire image data from a different face of an object (e.g., six faces of object 118) (e.g., improved acquisition speed), in some embodiments, a particular imaging device can be configured to acquire images of multiple faces of an object using a different number or arrangement of imaging devices, or a different arrangement of mirrors (e.g., the use of fixed mirrors, the use of additional movable mirrors, etc.). For example, fixed mirrors positioned so that imaging devices 306 and 310 can capture images of the back side of object 318 can be used instead of imaging devices 308 and 312.

[0037] In some embodiments, the system 300 may be configured to image each of several objects 318, 334, and 336 on the platform 316. However, the presence of multiple objects (e.g., objects 318, 334, and 336) on the platform 316 while imaging one of the objects (e.g., object 318) may affect the usefulness of the resulting images acquired by the imaging devices 302, 304, 306, 308, 310, and / or 312, including the analysis of symbols of a particular object. For example, when imaging device 306 is used to capture an image of one or more surfaces of object 318, objects 334 and 336 (e.g., one or more surfaces of objects 334 and 336) may appear in the image and overlap with object 318 in the image captured by imaging device 306. Therefore, it may be difficult to determine whether the detected symbols correspond to a particular object (i.e., whether the symbols are on or off the object).

[0038] In some embodiments, the system 300 may include a 3D sensor (or dimensioner) 330. As described above with respect to Figures 1A, 1B, and 2A, the 3D sensor The 3D sensor 330 can be configured to determine the dimensions and / or position of an object (e.g., object 318, 334, or 336) supported by the support structure 316. As described above, in some embodiments, the 3D sensor 330 can determine the 3D coordinates of each corner of an object in a coordinate space defined by reference to one or more parts of the system 300. For example, the 3D sensor 330 can determine the 3D coordinates of each of the eight corners of an object that is at least a nearly rectangular prism in a Cartesian coordinate space defined with the 3D sensor 330 as the origin. As another example, the 3D sensor 330 can determine the 3D coordinates of each of the eight corners of an object that is at least a nearly rectangular prism in a Cartesian coordinate space defined with respect to the support platform 316 (e.g., with an origin originating from the center of the support platform 316). As yet another example, the 3D sensor 330 can determine the 3D coordinates of the bounding box of a non-cuboidal object (e.g., one with eight corners) in a suitable Cartesian coordinate space (e.g., one defined with respect to the support platform 316, one defined with respect to the 3D sensor 330, etc.). For example, the 3D sensor 330 can identify the bounding box around any suitable non-cuboidal shape, such as a plastic bag, a padded envelope (jiffy mailer), an envelope, a cylindrical shape (such as a cylinder), a triangular prism, a non-cuboidal square prism, a pentagonal prism, a hexagonal prism, a tire (or other shape that can be approximated as a toroid). In some embodiments, the 3D sensor 330 can be configured to classify objects as either cuboidal or non-cuboidal, and can identify the corners of the object if it is cuboidal, and the corners of the cuboidal bounding box if it is non-cuboidal. In some embodiments, the 3D sensor 330 can be configured to classify objects into a specific class within a group of common objects (e.g., cuboids, cylinders, triangular prisms, hexagonal prisms, padded envelopes, plastic bags, tires, etc.). In some such embodiments, the 3D sensor 330 can be configured to determine a bounding box based on the classified shape.In some embodiments, the 3D sensor 330 can determine the 3D coordinates of non-cuboidal shapes such as a soft-sided envelope (e.g., with four corners), a pyramidal shape, and other prisms (e.g., a triangular prism with six corners, a non-cuboidal square prism, a pentagonal prism with ten corners, a hexagonal prism with twelve corners, etc.).

[0039] In addition or alternatively, in some embodiments, the 3D sensor (or dimensioner) 330 may provide raw data (e.g., point cloud data, distance data, etc.) to a control device (e.g., a control device 332, one or more imaging devices, etc., described later) that can determine the 3D coordinates of one or more points of an object.

[0040] In some embodiments, each imaging device (e.g., imaging devices 302, 304, 306, 308, 310, and 312) can be calibrated (for example, as described below in relation to Figures 12A to 12C) to facilitate mapping the 3D position of each corner of an object (e.g., object 318) supported by the support platform 316 to a 2D position in an image captured by the mirror imaging device using maneuverability in a specific direction.

[0041] In some embodiments, the image processing apparatus 332 can coordinate the operation of the imaging devices 302, 304, 306, 308, 310, and / or 312, and / or perform image processing tasks as described above in relation to the image processing apparatus 132 in Figure 1A and / or the image processing apparatus 410 described below in relation to Figure 4. For example, the image processing apparatus 332 can identify which objects in an image contain a symbol, for example, based on the mapping of the 3D angles of objects from the 3D coordinate space of the image associated with the symbol to the 2D image coordinate space.

[0042] Figure 4 shows an example 400 of a system for generating images of multiple faces of an object according to one embodiment of the present technology. As shown in Figure 4, an image processing device 410 (e.g., an image processing device 132) can receive images and / or information about each image (e.g., a 2D position associated with the image) from a plurality of imaging devices 402 (e.g., imaging devices 112a, 112b, and 112c mentioned above in relation to Figure 1A, imaging devices in imaging device banks 212, 214, 216, 218, 220, 222 mentioned above in relation to Figures 2A to 2B, and / or imaging devices 302, 304, 306, 308, 310, 312 mentioned above in relation to Figure 3). Furthermore, the image processing device 410 can receive dimensional data relating to an object imaged by the imaging device 402 from a dimension sensing system 412 (e.g., 3D sensor (or dimensioner) 150, 3D sensor (or dimensioner) 206, 3D sensor (or dimensioner) 330), the dimension sensing system 412 may be locally connected to the image processing device 410 and / or connected via a network connection (e.g., via a communication network 408). The image processing device 410 may also receive input from any other suitable motion measuring device, such as an encoder (not shown) configured to output values ​​indicating the movement of a conveyor over a specific period of time (e.g., between the time the dimensions are determined and each image of the object is generated), which can be used to determine the distance the object has moved. The image processing device 410 may also coordinate the operation of one or more other devices, such as one or more light sources (not shown) (e.g., flash, floodlight, etc.) configured to illuminate the object.

[0043] In some embodiments, the image processing apparatus 410 may perform at least a portion of the symbol assignment system 404 to assign symbols to an object using a group of images associated with the surface of the object. Additionally or alternatively, the image processing apparatus 410 may perform at least a portion of the symbol decoding system 406 to identify and / or decode symbols (e.g., barcodes, QR codes®, text, etc.) associated with an object imaged by the imaging device 402 using any suitable technique or combination of techniques.

[0044] In some embodiments, the image processing apparatus 410 can perform at least a portion of the symbol assignment system 404 to more efficiently assign symbols to objects using the mechanisms described herein.

[0045] In some embodiments, the image processing device 410 can communicate image data (e.g., images received from the imaging device 402) and / or data received from the dimension sensing system 412 to the server 420 via the communication network 408, and the server 420 can run at least a portion of the image archiving system 424 and / or the model rendering system 426. In some embodiments, the server 420 can use the image archiving system 424 to store the image data received from the image processing device 410 (for example, for further analysis such as attempts to decode symbols that could not be read by the symbol decoding system 406 for search and inspection in the event that an object is reported to be damaged, or attempts to extract information from text associated with the object). Additionally or alternatively, in some embodiments, the server 420 can use the model rendering system 426 to generate a 3D model of the object to be presented to the user.

[0046] In some embodiments, the image processing apparatus 410 and / or server 420 can be any suitable computing device or combination of devices, such as a desktop computer, laptop computer, smartphone, tablet computer, wearable computer, server computer, or virtual machine running on a physical computing device.

[0047] In some embodiments, the imaging device 402 can be any suitable imaging device. For example, each includes at least one imaging sensor (e.g., a CCD image sensor, a CMOS image sensor, or other suitable sensor), at least one lens arrangement, and at least one control device (e.g., a processor device) configured to perform computational operations on the measuring sensor. In some embodiments, the lens arrangement may include a fixed-focus lens. Additionally or alternatively, the lens arrangement may include an adjustable-focus lens, such as a liquid lens or a known type of mechanically adjustable lens. Additionally, in some embodiments, the imaging device 302 may include a steerable mirror that can be used to adjust the direction of the imaging device's FOV. In some embodiments, one or more imaging devices 402 may include a light source (e.g., a flash, a high-intensity flash, a light source, etc., as described in U.S. Patent Application Publication No. 2019 / 0333259) configured to illuminate objects within the FOV.

[0048] In some embodiments, the dimension sensing system 412 can be any suitable dimension sensing system. For example, the dimension sensing system 412 may be a 3D camera (e.g., a structured light 3D camera, a continuous time of flight 3D camera). It can be implemented using a camera, etc. Another example is that the dimension sensing system 412 can be implemented using a laser scanning system (e.g., a LiDAR system). In some embodiments, the dimension sensing system 412 can generate dimensions and / or 3D positions in any suitable coordinate space.

[0049] In some embodiments, the imaging device 402 and / or the dimension sensing system 412 can be local to the image processing device 410. For example, the imaging device 402 can be connected to the image processing device 410 by cable, direct wireless link, etc. As another example, the dimension sensing system 412 can be connected to the image processing device 410 by cable, direct wireless link, etc. Additionally or alternatively, in some embodiments, the imaging device 402 and / or the dimension sensing system 412 can be located locally and / or remotely from the image processing device 410 and can communicate data (e.g., image data, dimensions, and / or location data) to the image processing device 410 (and / or server 420) via a communication network (e.g., communication network 408). In some embodiments, one or more imaging devices 402, dimension sensing systems 412, image processing device 410, and / or other suitable components can be integrated as a single device (e.g., in a common housing).

[0050] In some embodiments, the communication network 408 can be any suitable communication network or combination of communication networks. For example, the communication network 408 can include a Wi-Fi® network (which may include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth® network), a cellular network (e.g., a 3G network, 4G network, 5G network, etc., compliant with suitable standards such as CDMA, GSM, LTE®, LTE Advanced, NR, etc.), a wired network, etc. In some embodiments, the communication network 408 can be a local area network (LAN), a wide area network (WAN), a public network (e.g., the Internet), a private or semi-private network (e.g., an intranet of a company or university), another suitable type of network, or a suitable combination of networks. The communication links shown in Figure 4 can each be any suitable communication link or combination of communication links, such as a wired link, a fiber optic link, a Wi-Fi® link, a Bluetooth® link, a cellular link, etc.

[0051] Figure 5 shows an example of hardware 500 that can be used to implement the image processing apparatus, server, and imaging apparatus shown in Figure 4, according to several embodiments of the disclosed subject matter. Figure 5 shows an example of hardware 500 that can be used to implement the image processing apparatus 410, server 420, and / or imaging apparatus 402, according to several embodiments of the disclosed subject matter. As shown in Figure 5, in some embodiments the image processing apparatus 410 may include a processor 502, a display 504, one or more inputs 506, one or more communication systems 508, and / or memory 510. In some embodiments the processor 502 may be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). In some embodiments the display 504 may include any suitable display device, such as a computer monitor, a touchscreen, or a television. In some embodiments the display 504 is optional. In some embodiments, input 506 may include any suitable input device and / or sensor that can be used to receive user input, such as a keyboard, mouse, touchscreen, or microphone. In some embodiments, input 506 is optional.

[0052] In some embodiments, the communication system 508 may include any suitable hardware, firmware, and / or software for communicating information over the communication network 408 and / or any other suitable communication network. For example, the communication system 508 may include one or more transceivers, one or more communication chips and / or chipsets, etc. In more specific examples, the communication network 408 may include hardware, firmware, and / or software that can be used to establish Wi-Fi® connections, Bluetooth® connections, cellular connections, Ethernet® connections, etc.

[0053] In some embodiments, memory 510 may include any suitable storage device that can be used to store instructions, values, etc., which can be used, for example, by the processor 502 to perform computer vision tasks, to present content using the display 504, to communicate with the server 420 and / or imaging device 402 via a communication system 508, etc. Memory 510 may include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 510 may include random access memory (RAM), read-only memory (ROM), electronically erasable programmable read-only memory (EEPROM), one or more flash drives, one or more hard disks, one or more solid-state drives, one or more optical drives, etc. In some embodiments, memory 510 may encode computer programs for controlling the operation of the image processing device 410. For example, in such embodiments, the processor 502 may execute at least a portion of the computer program to generate a composite image depicting the surface of an object, send the image data to the server 420, decode one or more symbols, etc. As another example, processor 502 may implement a symbol assignment system 404 and / or a symbol decoding system 406 by executing at least a portion of a computer program. As yet another example, processor 502 may execute at least a portion of processes 600, 630, and / or 660, which are described below in relation to Figures 6A, 6B, and / or 6C.

[0054] In some embodiments, the server 420 may include a processor 512, a display 514, one or more inputs 516, one or more communication systems 518, and / or memory 520. In some embodiments, the processor 512 may be any suitable hardware processor or combination of processors, such as a CPU, GPU, ASIC, FPGA, etc. In some embodiments, the display 514 may include any suitable display device, such as a computer monitor, touchscreen, television, etc. In some embodiments, the display 514 is optional. In some embodiments, the inputs 516 may include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, mouse, touchscreen, microphone, etc. In some embodiments, the inputs 516 are optional.

[0055] In some embodiments, the communication system 518 may include any suitable hardware, firmware, and / or software for communicating information over the communication network 408 and / or any other suitable communication network. For example, the communication system 518 may include one or more transceivers, one or more communication chips, and / or chipsets. In more specific examples, the communication system 518 may include hardware, firmware, and / or software that can be used to establish Wi-Fi® connections, Bluetooth® connections, cellular connections, Ethernet® connections, and the like.

[0056] In some embodiments, memory 520 may include any suitable storage device that can be used to store instructions, values, etc., which can be used, for example, by the processor 512 to present content using the display 514, or to communicate with one or more image processing devices 410. Memory 520 may include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 520 may include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid-state drives, one or more optical drives, etc. In some embodiments, memory 520 may encode and hold server programs for controlling the operation of server 420. For example, in such embodiments, the processor 512 may receive data from image processing devices 410 (e.g., values ​​decoded from symbols associated with objects), image device 402, and / or dimension sensing system 412, and / or store symbol assignments. In another example, the processor 512 may execute at least a portion of a computer program to implement an image archiving system 424 and / or a model rendering system 426. As yet another example, processor 512 may execute at least portions of processes 600, 630, and / or 660, which are described below in relation to Figures 6A, 6B, and / or 6C. Although not shown in Figure 5, server 420 may implement a symbol assignment system 404 and / or a symbol decoding system 406 in addition to, or instead of, such a system implemented using image processing device 410.

[0057] Figure 6A shows a process 600 according to one embodiment of the present technology for assigning symbols to an object using images of multiple faces of the object. In block 602, process 600 can receive a set of identified symbols from a set of one or more images. For example, as described above, images acquired from one or more objects using, for example, systems 100, 200, or 300 can be analyzed to identify any symbol in each image, and the identified symbols can be decoded. In some embodiments, a set of identified symbols (such as a list) may be generated, and each identified symbol may be associated with the image in which the symbol was identified, the imaging device used to capture the image, and the 2D position of the symbol in the image in which the symbol was identified. In block 604, for each symbol in the identified set of symbols, process 600 can receive the 3D position of a point corresponding to the object's 3D orientation (e.g., corresponding to a corner) in the tunnel coordinate space, and / or the physical space, defined based on the tunnel coordinate space associated with the device used to determine the 3D position, and / or the physical space (e.g., a conveyor such as conveyor 116 in Figure 1A, conveyor 204 in Figure 2A, or support platform 316 in Figure 3). For example, as described above in relation to Figures 1B, 2A, and 3, a 3D sensor (e.g., 3D sensors (or dimensions) 150, 206, 330) can determine the position of an object's corner at a specific point in time when a corresponding image is captured and / or when the object is at a specific location (e.g., a location associated with the 3D sensor). Thus, in some embodiments, the 3D position of a corner is associated with a specific point in time or a specific location where the image was captured by the imaging device. As another example, a 3D sensor (e.g., 3D sensor (or dimensioner) 150) can generate data indicating the 3D orientation of an object and provide this data (e.g., point cloud data, object height, object width, etc.) to process 600, which can then determine the 3D position of one or more points on the object. In some embodiments, the 3D positions of points corresponding to the corners of an object can be information indicating the 3D orientation of the object.For example, the 3D position of an object in coordinate space can be determined based on the 3D positions of points in that coordinate space corresponding to the corners of the object. The following explanation of Figures 6A-611 refers to the positions of points corresponding to the corners of a Buddha statue, but it should be noted that other information indicating the 3D orientation (e.g., point cloud data, object height, etc.) may also be used.

[0058] In some embodiments, the 3D position may be a position in a coordinate space associated with the device that measured the 3D position. For example, as described above in relation to Figures 1B to 2A, the 3D position can be defined in a coordinate space associated with the 3D sensor (e.g., the origin is located at the 3D sensor (or dimensioner)). As another example, as described above in relation to Figures 1B to 2A, the 3D position can be defined in a coordinate space associated with a dynamic support structure (e.g., conveyors such as conveyors 116, 204). In such an example, the 3D position measured by the 3D sensor can be associated with a specific time when the measurement was taken and / or a specific position along the dynamic support structure. In some embodiments, the 3D position of an object's corner when an image of the object is captured can be derived based on the initial 3D position, the elapsed time since the measurement was taken, and the velocity of the object during the elapsed time. Additionally or alternatively, the 3D position of an object's corner when an image of the object is captured can be derived based on the initial 3D position and the distance the object has moved since the measurement was taken (for example, recorded using a motion measuring device such as the motion measuring device 152 shown in Figure 1B, which directly measures the movement of a conveyor).

[0059] In some embodiments, process 600 can receive raw data indicating the 3D orientation of an object (e.g., point cloud data, object height, object width, etc.) and use the raw data to determine the 3D orientation of the object and the location of one or more features of the object (e.g., corners, edges, surfaces, etc.). For example, process 600 can use the technology described in U.S. Patent No. 11,335,021, issued on 17 May 2022, which is incorporated herein by reference in its entirety, to determine the 3D pose of an object and / or the location of one or more features of the object (e.g., for a rectangular prism, a plastic bag, an envelope, a padded envelope (jiffy mailer), and an object that can be approximated as a rectangular prism) from raw data indicating the 3D orientation of the object. As another example, process 600 utilizes the technique described in U.S. Patent Application Publication No. 2022 / 0148153, published on 12 May 2022 and incorporated herein by reference in its entirety, to determine the 3D orientation of an object (for example, for cylindrical and spherical objects) and / or the position of one or more features of the object from raw data indicating the 3D orientation of the object.

[0060] In block 606, for each object in the image associated with a symbol, process 600 can map each 3D position of a point corresponding to the object's 3D orientation in tunnel coordinate space (e.g., each 3D position of a corner) to a 2D position (and / or FOV angle) in the image coordinate space of the imaging device associated with the image. For example, as described below in relation to Figures 12A to 13B, the 3D position of each corner can be mapped to a 2D position in an image captured by the imaging device at a specific time and FOV. As previously mentioned, each imaging device is calibrated (e.g., as described below in relation to Figures 12A to 12C) to facilitate mapping the 3D position of each corner of an object to a 2D position in the image associated with the symbol. Note that in many images (e.g., as shown in image set 260), each corner may be outside the image, while in other images, one or more corners may be outside the image and one or more corners may be inside the image. In some embodiments, for an image containing multiple objects, the 3D angles of each object in the image (e.g., a target object and one or more preceding and succeeding objects) may be mapped to a 2D position in the image coordinate space of the image using the process described in blocks 604-606. In some embodiments, the dimensional data of each object (e.g., preceding object, target object, succeeding object) is stored, for example, in memory.

[0061] In block 608, process 600 can associate each portion of an image with a surface of an object (e.g., without analyzing the image content) based on the 2D position of points corresponding to the corners of an object. For example, process 600 can identify a particular portion of an image as corresponding to a first face of an object (e.g., the top of the object), and another portion of a particular image as corresponding to a second face of an object (e.g., the front of the object). In some embodiments, process 600 can use any suitable technique or combination of techniques to identify which portion of an image (e.g., which pixels) corresponds to a particular face of an object. For example, process 600 can draw lines (e.g., polylines) between the 2D positions associated with the corners of an object and group pixels that fall within the range of the lines (e.g., polylines) associated with a particular face of an object. In some embodiments, portions of an image may be associated with the surfaces of each object in the image. In some embodiments, the 2D positions of the corners of each object and the determined surfaces can be used to identify when two or more objects overlap (i.e., occlusion) in an image. For example, which surfaces are visible from a particular imaging device's FOV for a given image may be determined based on which of the determined surfaces intersect with each other. Figure 14 shows an example of determining the visible surfaces of one or more objects in an image according to one embodiment. For each surface of an object 1402 placed on a support structure 1404 (e.g., a conveyor or platform), the surface normal and its corresponding 3D point, as well as the optical axis and FOV 1408 of the imaging device 1406, may be used to identify the surfaces of the object 1402 that may be visible from the imaging device 1406. In the example shown in Figure 14, as the object 1402 moves along the direction of movement 1432, the back surface 1410 and left surface 1412 of the object 1402 may become visible from the following left imaging device 1406 (e.g., within the FOV 1408 of the imaging device 1406).For each of the surfaces 1410 and 1412 that may be visible from the imaging device 1406, the polylines created by the vertices of all surfaces in World 3D may be mapped to a 2D image using the calibration of the imaging device 1406 (for example, as described above with respect to block 606). For example, for the back surface 1410 of object 1402, the polyline 1414 created by the vertices of all surfaces in World 3D may be mapped to a 2D image, and for the left surface 1412 of object 1402, the polyline 1416 created by the vertices of all surfaces in World 3D may be mapped to a 2D image. The polylines resulting from the vertices of all surfaces in the 2D image may be, for example, entirely inside the 2D image, partially inside the 2D image, or entirely outside the 2D image. For example, in Figure 14, the polyline 1418 of the entire back surface 1410 in the 2D image is partially inside the 2D image, and the polyline 1420 of the entire left surface 1412 in the 2D image is partially inside the 2D image. The intersection of the polyline 1418 of the entire back surface 1410 and the 2D image may be determined to identify the visible surface region 1424 in the 2D image of the back surface 1401, and the intersection of the polyline 1420 of the entire left surface 1412 and the 2D image may be determined to identify the visible surface region 1426 in the 2D image of the left surface 1412. The visible surface regions in images 2D 1424 and 1426 are portions of the 2D image corresponding to each visible surface, such as the back surface 1410 and the left surface 1412, respectively. In some embodiments, the 2D visible surface region 1424 of the back surface 1410 and the 2D visible surface region 1426 of the left surface 1412 may be remapped to world 3D to determine the visible surface region of 3D 1428 of the back surface 1410 and the visible surface region of 3D 1430 of the left surface 1412, as needed. For example, a visible surface area identified in 2D may be remapped to world 3D (or box 3D or the object's coordinate space) to perform additional analysis, such as determining whether the placement of symbols on the object is correct, or to perform metric measurements of the symbol's position relative to the object's surface. This is important to verify whether the vendor is complying with specifications regarding where to place symbols or labels on the object.

[0062] To address errors in one or more mapped edges of one or more objects or surfaces of objects within an image (e.g., boundaries determined from the mapping of 3D positions of points corresponding to the corners of an object), in some embodiments, one or more mapped edges may be determined and refined using the image data content of the image and the image processing techniques used to generate the image. Errors in mapped edges are, for example, This may be caused by irregular movement of an object (e.g., an object swaying when moving on a conveyor belt) or errors in 3D sensor data (or dimensional data, calibration errors, etc.). In some embodiments, an image of an object can be analyzed to further refine edges based on the proximity of symbols to the edges. Thus, the image data associated with the edges can be used to determine where the edges should be placed.

[0063] For each image, blocks 610 and 612 allow the identification of symbols within the image to be assigned to objects and / or surfaces of objects within the image. Although blocks 610 and 612 are shown in a specific order, in some embodiments, blocks 610 and 612 may be executed in a different order than that shown in Figure 6A, or they may be bypassed. In some embodiments, in block 610, for each image, the identification of symbols within the image may be assigned to objects within the image based on the 2D position of a point corresponding to, for example, the corners of one or more objects within the image. Thus, a symbol can be associated with a specific object within the image, for example, an object within the image to which the symbol is attached. For example, it can be determined whether the position of the symbol (e.g., the 2D position of the symbol in the relevant image) is inside or outside a boundary defined by the 2D position of the corner of the object. For example, if the position of the symbol is within the boundary defined by the 2D position of the corner of the object, the identified symbol can be assigned (or associated) with the object.

[0064] In some embodiments, block 612 may assign a symbol identified in the image to a surface of an object in the image, for example, based on the 2D position of a point corresponding to a corner of the object and one or more surfaces of the object determined in block 610. Thus, the symbol can be associated with a specific surface of an object in the image, for example, the surface of an object to which the code is attached. For example, it can be determined whether the position of the symbol is inside or outside a boundary defined by the surface of the object. For example, if the position of the symbol is within a boundary defined by one of the determined surfaces of the object, the identified symbol can be assigned (or associated) with the surface of the object. In some embodiments, the assignment of a symbol to a surface in block 612 may be performed after the symbol has been assigned to an object in block 610. In other words, the symbol may first be assigned to an object in block 610 and then assigned to the surface of the assigned object in block 612. In some embodiments, block 612 may directly assign a symbol to a surface without first assigning the symbol to an object. In such embodiments, the object to which the symbol is attached can be determined based on the assigned surface.

[0065] In some embodiments, the image associated with the identified symbol may include two or more non-overlapping objects (or surfaces of objects). Figure 7 shows one embodiment of the art. An example of an image containing two objects is shown, where at least one object contains a symbol to which it is assigned. In Figure 7, an example image 702 of two objects 704 and 706 is shown with the corresponding 3D coordinate space 716 (associated with a support structure such as a conveyor 718, for example) and the FOV 714 of an imaging device (not shown) used to capture the image 802. As described above, the 3D positions of the corners of the first object 704 and the second object 706 may be mapped to the 3D image coordinate space and may be used to determine the boundary 708 (e.g., a polyline) of the first object 704 and the boundary 710 (e.g., a polyline) of the second object 706. In Figure 7, the 2D position of symbol 712 falls within the boundary 710 of the surface of object 706. Thus, symbol 712 may be assigned to object 706.

[0066] In some embodiments, the image associated with an identified symbol may include two or more overlapping objects (or surfaces of objects). Figures 8A–8C show an example of an image having two objects with overlapping surfaces according to one embodiment of the Art, where at least one object contains the assigned symbol. In some embodiments, if at least one surface of each of the two objects in the image intersects, the intersecting surfaces are determined to be overlapping (e.g., as shown in Figures 8B and 8C). In some embodiments, if there is no actual overlap between the surfaces of the two objects (i.e., the surfaces do not intersect (e.g., as shown in Figure 8A)), the boundaries of the two objects (e.g., polylines) are still close enough that errors in identifying or mapping the object boundaries may result in the assignment of an inaccurate symbol. In such embodiments, a margin may be provided around the mapped edge of each object to represent uncertainty in the boundary position due to errors (e.g., irregular movement of objects, errors in dimensional data, calibration errors, etc.). If the object boundaries, including the margin, intersect, the objects (or surfaces of objects) can be defined as overlapping. In Figure 8A, an example image 802 of two objects 804 and 806 is shown along with the corresponding 3D coordinate space 820 (associated with a support structure such as a conveyor 822, for example) and the FOV 818 of an imaging device (not shown) used to capture the image 802. In Figure 8A, the first object 804 (or the surface of the first object) in the image 802 has a boundary 808, and the second object 806 (or the surface of the second object) in the image 802 has a boundary 810. The margin around the boundary 808 of the first object 804 is defined between lines 813 and 815. The margin around the boundary 810 of the second object 806 is defined between lines 812 and 814. Boundaries 806 and 810 are close but do not overlap. However, the margins of the first object 804 (defined by points 813 and 815) and the margins of the second object 806 (defined by lines 812 and 814) overlap (or intersect). Therefore, it may be determined that the first object 804 and the second object 806 overlap.In some embodiments, the identified symbol 816 (for example, in blocks 610 and 612 of Figure 6A) may be initially assigned to a surface of object 806, but overlapping surfaces may be further resolved using additional techniques (for example, using the process of blocks 614 and 616 of Figures 6A and 6B).

[0067] In another example, Figure 8B shows an example image 830 of two overlapping objects 832 and 834 (e.g., two overlapping surfaces of the objects) along with the corresponding 3D coordinate space 833 (associated with a support structure, e.g., a conveyor 835) and the FOV 831 of an imaging device (not shown) used to capture the image 830. As described above, the 3D positions of the corners of the first object 832 and the second object 824 may be mapped to the 2D image coordinate space and used to determine the boundary 836 (e.g., a polyline) of the first object 832 and the boundary 838 (e.g., a polyline) of the second object 834. In Figure 1, and in Figure 8B, the boundary 836 of the first object 832 and the boundary 838 of the second object 834 overlap in the overlapping region 840. The 2D position of the identified symbol 842 is within the boundary 838 of the surface of object 834. Furthermore, the 2D position of symbol 842 lies within the margin (not shown) of the boundary 836 of the surface of object 832. However, (for example, due to errors in boundary mapping or localization), symbol 842 does not fall within the overlapping region 840. Therefore, symbol 842 may be initially assigned to object 834 (for example, in blocks 610 and 612 of Figure 6A), but the ambiguity and overlapping surfaces that may arise due to potential limiting errors can be further resolved using additional techniques (for example, using the processes of blocks 614 and 616 of Figures 6A and 6B).

[0068] In another example, in Figure 8C, an image example 850 of two overlapping objects 852 and 854 (e.g., two overlapping surfaces of objects) is shown together with the corresponding 3D coordinate space 864 (associated with a support structure, e.g., a conveyor 866) and the FOV 862 of an imaging device (not shown) used to capture the image 850. As described above, the 3D positions of the corners of the first object 852 and the second object 854 may be mapped to the 3D image coordinate space and used to determine the boundary 858 (e.g., a polyline) of the first object 852 and the boundary 860 (e.g., a polyline) of the second object 854. In Figure 8C, the surface of the first object 852 and the surface of the second object 854 overlap. The 2D position of the identified symbol 856 lies within the boundary of the overlapping surfaces (e.g., overlapping region 855) of surfaces 852 and 854 and may initially be assigned to surface 854 (e.g., in blocks 610 and 612 in Figure 6A), but the overlapping surfaces can be further resolved using additional techniques (e.g., using the process in blocks 614 and 616 in Figures 6A and 6B).

[0069] Returning to Figure 6A, in block 614, if the image associated with the symbol contains overlapping surfaces between two or more objects in the image, the overlapping surfaces may be resolved to identify or confirm the initial symbol assignment, for example, using a process further described below in relation to Figure 6B. If, in block 614, the image associated with the decoded symbol does not contain overlapping surfaces between two or more objects in the image, then in block 617, a confidence level (or score) for the symbol assignment may be determined. In some embodiments, the confidence level (or score) may fall within a range of values, such as between 0 and 1, or between 0% and 100%. For example, a 40% confidence level may indicate that there is a 40% probability that the symbol is attached to an object and / or surface, and a 60% probability that the symbol is not attached to an object and / or surface. In some embodiments, the confidence level may be a normalized measure indicating how far the 2D position of the symbol is from a boundary defined by the 2D position of a point on an object (e.g., corresponding to a corner of an object) or from a boundary defined by the surface of an object. For example, the confidence level may be higher if the 2D position of a symbol in the image is far from a boundary defined by a boundary or surface defined by a point's 2D position, and lower if the 2D position of a symbol is very close to a boundary defined by a point's 2D position or surface. In some embodiments, the confidence level may be based on one or more additional factors, one or more of which include, but are not limited to, whether the 2D position of a symbol is inside or outside a boundary defined by a point (e.g., corresponding to a corner) of an object or by a surface of an object, the ratio of the distance from the boundaries of various objects in the FOV to the 2D position of the symbol, whether there are overlapping objects in the image (as will be further discussed with respect to Figure 6B), and the reliability of the image processing techniques used to refine one or more edges of an object or surfaces of an object, and the techniques for finding the correct edge positions based on the content of the image.

[0070] Once the confidence level is determined in block 617 or the overlapping surfaces are resolved (block 616), block 618 determines whether the symbol is the last symbol in the set of identified symbols. If block 618 determines that the symbol is not the last symbol in the set of identified symbols, process 600 returns to block 604. If, in block 618, the symbol is the last symbol in the set of identified symbols, process 600 can identify any identified symbol that appears multiple times in the set of identified symbols (for example, a symbol is identified in multiple images). For each symbol that appears multiple times in the set of identified symbols (for example, for each symbol associated with multiple related images), the symbol assignment results for the symbol are aggregated in block 620. In some embodiments, the aggregation can be used to determine if there are inconsistencies between the symbol assignment results for each symbol (for example, for a symbol associated with two images, the symbol assignment results for each image will be different) and to resolve conflicts. An example of the aggregation process is described further below with reference to Figure 6C. Different symbol assignment results between different images associated with a particular symbol may be caused, for example, by irregular movement (e.g., an object sways when moving on a conveyor belt), errors in dimensional data, calibration, etc. In some embodiments, the aggregation of symbol assignment results may be performed in 2D image space. In some embodiments, the aggregation of symbol assignment results may be performed in 3D space. In block 622, the symbol assignment results may be stored, for example, in memory.

[0071] As described above, if the image associated with an identified symbol includes overlapping surfaces between two or more objects in the image, the overlapping surfaces can be resolved to identify or confirm the assignment of the symbol. Figure 6B shows a method for resolving overlapping surfaces of multiple objects in an image to assign a symbol to one of multiple objects, according to one embodiment of the art. In block 632, process 630 compares the 2D position of the symbol with the 2D boundaries and surfaces (e.g., polylines) of each object in the overlapping region. In block 634, the position of each overlapping object (or surface of each object) in the image relative to the imaging device's field of view (FOV) used to capture the image containing the symbol may be identified. For example, it may be determined which objects (or object surfaces) are in front of the imaging device's FOV and which are behind the imaging device's FOV. In block 636, the object causing the overlap (or occlusion) may be determined based on the position of the overlapping object (or object surface) relative to the imaging device's field of view. For example, an object in front of the imaging device's FOV A body (or object surface) may be identified as an occluding object (or object surface). In block 638, a symbol can be assigned to the occluding object (or object surface).

[0072] Once a symbol is assigned to an occluding object (or object surface), process 640 can determine whether further analysis or refinement of the symbol assignment can be performed. In some embodiments, after a symbol is assigned to an occluding object (or object surface), further analysis may be performed on each symbol assignment in the image where the objects overlap. In other embodiments, after a symbol is assigned to an occluding object or object surface, further analysis may not be performed on the symbol assignment in the image where the objects overlap. In some embodiments, if one or more parameters of the object (or object surface) and / or the position of the symbol meet a predetermined criterion, further analysis may be performed on the symbol assignment in the image where the objects overlap after the symbol has been assigned to the occluding object or object surface. For example, in Figure 6B, if in block 640 the 2D position of the symbol is within a predetermined threshold at the edge of the overlapping region, further analysis may be performed in blocks 642 through 646. In some embodiments, as shown in Figure 9, the predetermined threshold may be a neighborhood of the edge or boundary of the overlapping region between objects (or object surfaces) for one or more boundaries of the symbol (defined, for example, by the 2D position of the symbol). In the example image 902 of Figure 9, symbol 912 is closer to the edge 910 of the overlapping region 908 between the first object 904 and the second object 906 than a predetermined threshold (e.g., a predetermined number of pixels or mm). Therefore, further analysis can be performed with respect to symbol assignment. In the example image 914 of Figure 9, symbol 924 is further from the edge 922 of the overlapping region 920 between the first object 916 and the second object 918 than a predetermined threshold. Therefore, further analysis with respect to symbol assignment may not be necessary.

[0073] In some embodiments, when further analysis is performed, block 642 retrieves image data associated with the symbol and information indicating the 3D orientation of the object, such as the 3D angles of the object in the image. For example, 2D image data associated with one or more boundaries or edges of one or more overlapping objects can be obtained. In block 644, the content of the image data and image processing techniques may be used to refine one or more edges of one or more overlapping objects (e.g., boundaries determined from the mapping of the object's 3D corners). For example, as shown in Figure 10, an image 1002 having two overlapping objects (or object surfaces) 1004 and 1006 may be analyzed to further refine the edge 1016 of the first object 1004 based on the proximity of the symbol 1014 to the edge of the overlapping region 1012. Thus, the image data associated with the edge 1016 may be used to determine where the edge 1016 should be positioned. As described above, the error in the position of edge 1016 may result from an error in the mapping of one or more of the 3D corners of object 1004 (e.g., 3D corners 1024, 1026, 1028, 1030), for example, 3D corners 1028 and 1030. Figure 10 also shows the corresponding 3D coordinate space 1020 and FOV 1018 (related to a support structure, e.g., a conveyor belt 1022) of an imaging device (not shown) used to capture image 1002. Once the positions of one or more edges of one or more overlapping objects in block 646 are adjusted, a symbol may be assigned to an object in the image if, for example, the position of the symbol is within the boundary defined by the 2D position of the corner of the object. Furthermore, in some embodiments, a confidence level (or score) is determined and assigned to the symbol assignment. In some embodiments, the confidence level (or score) may fall within a range of values ​​such as between 0 and 1, or between 0% and 100%. For example, a 40% confidence level may indicate that there is a 40% probability that the symbol will be attached to the object and / or surface, and a 60% probability that the symbol will not be attached to the object and / or surface.In some embodiments, the confidence level may be a normalized measure indicating how far the 2D position of a symbol is from the boundary of an overlapping region. For example, the confidence level may be higher if the 2D position of a symbol in the image is farther from the boundary (or edge) of an overlapping region, and lower if the 2D position of a symbol is very close to the boundary (or edge). In some embodiments, the confidence level may be determined based on an image processing technique used to refine the position of one or more edges of overlapping objects and the confidence level of that technique, finding the correct edge position based on the content of the image. In some embodiments, the confidence level may be based on one or more additional factors, one or more of which include, but are not limited to, whether the 2D position of a symbol is inside or outside a boundary defined by the 2D position of a point of an object (e.g., corresponding to a corner), whether the 2D position of a symbol is inside or outside the boundary of an overlapping region, the ratio of the distance from the boundary of various objects in the FOV to the 2D position of the symbol, and whether there are overlapping objects in the image. In block 650, the symbol assignments and confidence levels may be stored, for example, in memory.

[0074] In block 640 of Figure 6B, if the 2D position of a symbol is not within a predetermined threshold of the edge of the overlapping region, a confidence level (or score) may be assigned in block 648 to the symbol assignment from block 632. As described above, the confidence level (or score) may fall within a range of values ​​such as between 0 and 1, or between 0% and 100%. In some embodiments, the confidence level may be a normalized measure indicating how far the 2D position of a symbol is from the boundary of the overlapping region; for example, the confidence level may be higher if the 2D position of a symbol in the image is farther from the boundary (or edge) of the overlapping region, and the confidence level may be higher if the 2D position of a symbol is very close to the boundary (or edge) of the overlapping region. As described above, in some embodiments, the confidence level may be based on one or more additional elements, one or more of which are the overlapping objects The reliability of the image processing techniques used to adjust the position of one or more edges, the techniques for finding the correct edge positions based on the image content, whether the 2D position of a symbol is inside or outside a boundary defined by the 2D position of a point or object (e.g., corresponding to a corner), whether the 2D position of a symbol is inside or outside the boundary of an overlapping region, the ratio of the distance from the boundary of various objects in the FOV to the 2D position of the symbol, and whether there are overlapping objects in the image. In block 650, the symbol assignment and reliability level may be stored, for example, in memory.

[0075] As described above with respect to block 620 in Figure 6A, the symbol assignment results for any identified symbol that appears multiple times within a set of identified symbols (for example, a symbol is identified in multiple images) may be aggregated, for example, to determine if there are inconsistencies between the symbol assignment results for a particular symbol and to resolve any conflicts. Figure 6C shows a method for aggregating symbol assignment results for symbols according to one embodiment of the present technology. In block 662, process 660 identifies each of any repeating symbols within a set of identified symbols, for example, a symbol having multiple associated images. In block 664, process 660 may identify each associated image in which the symbol appears for each repeating symbol. In block 666, process 660 may compare the symbol assignment results for each image associated with the repeating symbol. In block 668, if all the symbol assignment results are the same for each image associated with the repeating symbol, block 678 may store the common symbol assignment results for the images associated with the repeating symbol in memory, for example.

[0076] If, in block 668, there is at least one different symbol assignment result among the image assignment results associated with a repeating symbol, process 660 may decide in block 670 if there is at least one assignment result associated with an image where objects do not overlap (see, for example, image 702 in Figure 7): If there is at least one symbol assignment result associated with an image where objects do not overlap, process 660 may, in block 672, select a symbol assignment result for the image without overlapping objects, and the selected symbol assignment result may be stored in memory, for example, in block 678.

[0077] If, in block 670, there is no symbol assignment result associated with an image without overlapping objects, process 660 may, in block 674, compare the confidence levels (or scores) of symbol assignment results for images associated with the repeated symbols. In some embodiments, process 660 may not include blocks 670 and 672, and the confidence levels of all aggregated assignment results for the repeated symbols (both images with and without overlapping objects) may be compared. In block 676, in some embodiments, process 660 may select the assignment that has the highest confidence level as a symbol assignment for the repeated symbols. In block 678, the selected symbol assignment may be stored, for example, in memory.

[0078] Figure 11 shows a method for aggregating symbol assignment results for symbols according to one embodiment of the present technology. In Figure 11, Image Example 1102 shows the FOV 1116 of a first image device (not shown) capturing a first object 1106, a second object 1108, and a symbol 1112 on the second object 1108. Image Example 1104 shows the FOV 1114 of a second image device (not shown) capturing a first object 1106, a second object 1108, and a symbol 1112 on the second object 1108. In Image 1104, the first object 1106 and the second object 1108 do not overlap, while in Image 1102, the first object 1106 and the second object 1108 overlap. Therefore, Images 1102 and 1104 may yield different symbol assignment results. For example, boundary (or edge) 1113 may represent the correct boundary of the first object 1106, but an error in the top surface of object 1106 may cause the edge to be mapped to edge 1115. As a result, symbol 1112 may be assigned to the wrong object, i.e., the first object 1106. In contrast, in this example, the separation between the surfaces of the first object 1106 and the second object 1108 ensures that errors in boundary mapping do not cause ambiguity in symbol assignment in image 1104. As mentioned above with respect to Figure 6C, since image 1104 does not contain overlapping objects, the system can select a symbol assignment for symbol 1112 from image 1104. Figure 11 also shows the corresponding 3D coordinate space 1118 (associated with a support structure, e.g., a conveyor belt 1120) for example images 1102 and 1104. In another example in Figure 11, Image Example 1130 shows the FOV 1140 of a first imaging device (not shown) capturing a first object 1134, a second object 1136, and symbols on the second object 1136. Image Example 1132 shows the FOV 1138 of a second imaging device (not shown) capturing a first object 1134, a second object 1136, and symbols on the second object 1136. In Image 1132, the first object 1134 and the second object 1136 do not overlap, while in Image 1132, the first object 1134 and the second object 1136 overlap. Therefore, Images 1130 and 1132 may result in different symbol assignment results.As mentioned above with respect to Figure 6C, since image 1132 does not contain overlapping objects, the system may select a symbol assignment for a symbol from image 1132. Figure 11 also shows the corresponding 3D coordinate space 1142 (associated with a support structure such as a conveyor belt 1144, for example) for image examples 1130 and 1132.

[0079] Figure 12A shows an example of a factory calibration setup that can be used to find the transformation between the image coordinate space and the calibration target coordinate space. As shown, the imaging device 1202 can generate an image (e.g., image 808) that projects points in the 3D factory coordinate space (Xf, Yf, Zf) onto the 2D image coordinate space (xi, yi). The 3D factory coordinate space can be defined based on a support structure 804 (sometimes called a fixture) that supports a calibration target used to find the transformation between the factory coordinate space and the image coordinate space.

[0080] Generally, the overall goal of calibrating an imaging device 1202 (e.g., a camera) is to find the transformation between the physical 3D coordinate space (e.g., in millimeters) and the 2D coordinate space of the image (e.g., in pixels). The transformation in Figure 12A is an example of such a transformation using a simple pinhole camera model. The transformation may include other nonlinear components (e.g., to represent lens distortion). The transformation can be divided into external and internal parameters. External parameters may depend on the mounting position and orientation of the imaging device relative to the physical 3D coordinate space. Internal parameters may depend on internal imaging device parameters (such as sensor and lens parameters). The goal of the calibration process is to find the values ​​of these internal and external parameters. In some embodiments, the calibration process can be divided into two parts: one part is performed in factory calibration and the other part is performed in the field.

[0081] Figure 12B shows a calibration process that includes factory calibration and field calibration, which involves capturing multiple images of one or more faces of an object, according to an embodiment of the technology. Examples of coordinate spaces and other embodiments for a tunnel are shown. As shown in Figure 12B, the tunnel 3D coordinate space (e.g., the tunnel 3D coordinate space shown in Figure 12B having axes Xt, Yt, and Zt) can be defined based on a support structure 1222. For example, in Figure 12B, a conveyor (e.g., as described above in relation to Figures 1A, 1B, and 2A) is used to define the tunnel coordinate space, with the origin 1224 located at a specific position along the conveyor (e.g., if Yt=0 is defined at a specific point along the conveyor, for example, at a point defined based on the position of the photo eye described in U.S. Patent Application Publication No. 2021 / 0125373, Xt=0 is defined on one side of the conveyor, and Zt=0 is defined on the surface of the conveyor). In other examples, the tunnel coordinate space can be defined based on a stationary support structure (e.g., as described above in relation to Figure 3). Alternatively, in some embodiments, the tunnel coordinate space can be defined based on a 3D sensor (or dimensioner) used to measure the position of an object.

[0082] Additionally, in some embodiments, during a calibration process (e.g., a field calibration process), an object coordinate space (Xb, Yb, Zb) can be defined based on the object (e.g., object 1226) used to perform the calibration. For example, as shown in Figure 12B, symbols can be placed on the object (e.g., object 1226), and each symbol is associated with a specific position in the object coordinate space.

[0083] Figure 12C shows an example of a field calibration process 1230 for generating an imaging device model that can be used to convert the coordinates of an object in 3D coordinate space associated with a system that captures multiple images of each side of an object, according to one embodiment of the Art, to coordinates in 2D coordinate space associated with the imaging device. In some embodiments, the imaging device (e.g., imaging device 1202) can be calibrated before being installed in the field (e.g., a factory calibration can be performed). Such calibration can be used to generate an initial camera model that can be used to map points in 3D factory coordinate space to 2D points in image coordinate space. For example, as shown in Figure 12C, a factory calibration process can be performed to generate extrinsic parameters that can be used together with intrinsic parameters to map points in 3D factory coordinate space to 2D points in image coordinate space. Intrinsic parameters can represent parameters that relate pixels of the imaging device's image sensor to the image plane of the imaging device, such as focal length, image sensor format, and principal point. Extrinsic parameters can represent parameters that relate points in 3D tunnel coordinates (e.g., having an origin defined by a target used during factory calibration) to 3D camera coordinates (e.g., having a camera center defined as the origin).

[0084] A 3D sensor (or dimensioner) can measure a calibration object (e.g., a box with a code that defines the position of each code in object coordinate space, such as object 1226) in tunnel coordinate space, and the position in tunnel coordinate space can be correlated with the position of the calibration object in image coordinate space (e.g., associating the coordinates (Xt,Yt,Zt) with (xi,yi)). Using this correspondence, the camera model can be updated to account for the transformation between the factory coordinate space and the tunnel coordinate space (e.g., by deriving a field calibration external parameter matrix, which can be defined using a 3D rigid body transformation that associates one 3D coordinate space, such as tunnel coordinate space, with another 3D coordinate space, such as factory coordinate space). The field calibration external parameter matrix can be used in conjunction with the camera model derived during factory calibration to associate a point (Xt,Yt,Zt) in tunnel coordinate space with a point (xi,yi) in image coordinate space. Using this transformation, 3D points of an object measured by the 3D sensor can be mapped to an image of the object, and the portion of the image corresponding to a specific surface can be determined without analyzing the content of the image. Note that the models shown in Figures 12A, 12B, and 12C are simplified models (e.g., pinhole camera) that can be used to correct distortions caused by projection, in order to avoid overly complex explanations. More sophisticated models (e.g., including lens distortion) can be used in conjunction with the mechanisms described herein.

[0085] Note that this is merely an example, and other techniques can be used to define the transformation between tunnel coordinate space and image coordinate space. For example, instead of performing factory calibration and field calibration, field calibration can be used to derive a model that relates tunnel coordinates to image coordinates. However, using a new imaging device requires performing the entire calibration. Therefore, replacing the imaging device may become more complicated. In some embodiments, calibrating the imaging device using a calibration target to detect the transformation between 3D factory coordinate space and image coordinates, and calibrating the imaging device in the field to find transformations that facilitate mapping between tunnel coordinates (e.g., associated with a conveyor, support platform, or 3D sensor) can facilitate the replacement of the imaging device without repeating field calibration.

[0086] Figures 13A and 13B illustrate an example of a field calibration process relating to different locations of a calibration target (or multiple targets) according to one embodiment of the present technology. As shown in Figure 13A, the mechanism described herein relates to the shape of a conveyor (or other transport system) Objects related to the tunnel coordinate space defined based on the tunnel coordinate space specified by the tunnel coordinate space (and / or support structure) The assigned 3D points (e.g., corners of an object) can be mapped to points in image coordinate space based on a model generated based on factory and field calibrations. For example, as shown in Figure 13A, the mechanism described herein can map 3D points associated with a box 1328 defined in tunnel coordinates (Xt, Yt, Zt) with respect to a support structure 1322 to points in image coordinate space associated with an imaging device (e.g., imaging device 1202 shown in Figures 12A and 12B). In some embodiments, the 2D points of each corner in image space can be used, along with knowledge of the orientation of the imaging device, to associate each pixel in the image with a specific surface (or face) of an object (or determine that a pixel is not associated with an object) without analyzing the image content.

[0087] For example, as shown in Figure 13A, the imaging device (e.g., imaging device 1202 shown in Figures 12A and 12B) is configured to capture an image (e.g., image 1334) from a front-top angle with respect to tunnel coordinates (e.g., in field of view 1332). In such an example, the 2D positions of the box corners can be used to automatically associate the first part of the image with the "left side" of the box, the second part with the "front" side of the box, and the third part with the "top" side of the box. As shown in Figure 13A, only two of the corners are located within image 1332, while the other six corners are outside the image. Based on the knowledge that the imaging device is configured to capture an image from above an object and / or based on camera calibration (e.g., facilitating the determination of the imaging device's position relative to tunnel coordinates and the optical axis of the imaging device), the system can determine that both the leading lower-left corner and the leading upper-left corner are visible within the image.

[0088] As shown in Figure 13B, a second image 1336 is shown, taken after box 1328 has moved a distance ΔYt along the conveyor. As described above, a motion measuring device (e.g., an encoder) can be used to measure the distance box 1328 has moved between the time the first image 1334 shown in Figure 13A was captured and the time the second image 1336 was captured. The operation of such a motion measuring device (e.g., implemented as an encoder) is described in U.S. Patent No. 9,305,231 and U.S. Patent Application Publication No. 2021 / 0125373, filed on October 26, 2020, which are incorporated herein by reference in their entirety. Using the distance traveled and 3D coordinates, 2D points in the second image 1336 corresponding to the corners of box 1328 can be determined. Based on the knowledge that the imaging device is configured to capture images from above an object, the system can determine that the upper right corner of the rear is visible in image 1336, but the lower right corner of the rear is obscured by the top of the box. Additionally, the top of the box is visible, but the back and right side are not.

[0089] In some embodiments, any suitable computer-readable medium can be used to store instructions for performing the functions and / or processes described herein. For example, in some embodiments, the computer-readable medium may be transient or non-transient. For example, the computer-readable medium may include magnetic media (hard disks, floppy disks, etc.), optical media (compact disks, digital video disks, Blu-ray disks, etc.), semiconductor media (RAM, flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.), suitable media that are non-transient or persistent during transmission, and / or suitable tangible media. As another example, transient computer-readable media may include signals on networks, wires, conductors, optical fibers, circuits, or other suitable media that are transient and non-persistent during transmission, and / or suitable intangible media.

[0090] Furthermore, as used herein, the term "mechanism" may encompass hardware, software, firmware, or any appropriate combination thereof.

[0091] It should be understood that the steps described above in the process of Figure 6 are not limited to the order and sequence shown and explained in the figure, but can be performed in any order or sequence. Furthermore, some of the steps described above in the process of Figure 6 can be performed substantially simultaneously or in parallel as needed to reduce waiting and processing times.

[0092] Although the present invention has been described and illustrated in the exemplary embodiments described above, it should be understood that this disclosure is illustrative only and many modifications can be made to the details of the implementation of the invention without departing from the spirit and scope of the invention, which are limited only by the following claims. The features of the disclosed embodiments can be combined and rearranged in various ways.

Claims

1. A method for assigning symbols to objects in an image, A step of receiving the image captured by the imaging device, wherein the symbol is located within the image; The steps include receiving the three-dimensional (3D) positions of one or more points corresponding to pose information indicating the 3D orientation of the object in the image in a first coordinate system, The steps include mapping the 3D positions of one or more points of the object to 2D positions in the image, The steps include assigning the symbol to the object based on the relationship between the 2D position of the symbol in the image and the 2D positions of one or more points of the object in the image, A method characterized by including the following.

2. A step of determining the surface of the object based on the 2D positions of one or more points of the object in the image, A step of assigning the symbol to the surface of the object based on the relationship between the 2D position of the symbol in the image and the surface of the object, The method according to claim 1, further comprising:

3. The steps include determining that the symbol is associated with multiple images, A step of aggregating the assignment of the symbols to each of the multiple images, A step of determining whether at least one of the assignments of the symbol is different from the remaining assignments of the symbol, The method according to claim 1, further comprising:

4. The step of determining the edges of the object in the image based on the image acquisition data of the image. The method according to claim 1, further comprising:

5. Steps to determine the confidence score for the aforementioned symbol assignment. The method according to claim 1, further comprising:

6. The 3D positions of one or more points are received from the 3D sensor. The method according to feature 1.

7. The aforementioned image includes multiple objects, The aforementioned method, A step to determine whether the plurality of objects overlap in the image. The method according to claim 1, further comprising:

8. The aforementioned image includes an object having a first boundary with a margin, and a second object having a second boundary with a second margin, The aforementioned method, A step of determining whether the first boundary and the second boundary overlap in the image. The method according to claim 1, further comprising:

9. The 3D positions of one or more points are acquired in the first time, and the image is acquired in the second time. Mapping the 3D positions of one or more points to the 2D positions in the image includes mapping the 3D positions of one or more points from the first time to the second time. The method according to feature 1.

10. The orientation information includes the angles of the object in the first coordinate space. The method according to feature 1.

11. The aforementioned attitude information includes point cloud data. The method according to feature 1.

12. A system for assigning symbols to objects in an image, A calibrated imaging device configured to capture images, Processor devices and Equipped with, The aforementioned processor device, A step of receiving the image captured by the calibrated imaging device, wherein the symbol is located within the image; The steps include receiving the three-dimensional (3D) positions of one or more points corresponding to pose information indicating the 3D orientation of the object in the image in a first coordinate system, The steps include mapping the 3D positions of one or more points of the object to 2D positions in the image, The steps include assigning the symbol to the object based on the relationship between the 2D position of the symbol in the image and the 2D positions of one or more points of the object in the image, A system characterized by being programmed to perform the following actions.

13. A conveyor configured to support and transport the aforementioned object, A motion measuring device coupled to the conveyor and configured to measure the movement of the conveyor, The system according to claim 12, further comprising the following:

14. A 3D sensor configured to measure the 3D position of one or more points. The system according to claim 12, further comprising the following:

15. The orientation information includes the angles of the object in the first coordinate space. The system according to feature 12.

16. The aforementioned attitude information includes point cloud data. The system according to feature 12.

17. The aforementioned processor device further, A step of determining the surface of the object based on the 2D positions of one or more points of the object in the image, A step of assigning the symbol to the surface of the object based on the relationship between the 2D position of the symbol in the image and the surface of the object, The system according to claim 12, characterized in that it is programmed to perform the following.

18. The aforementioned at least one processor device further includes: The steps include determining that the symbol is associated with multiple images, A step of aggregating the assignment of the symbols to each of the multiple images, A step of determining whether at least one of the assignments of the symbol is different from the remaining assignments of the symbol, The system according to claim 12, characterized in that it is programmed to perform the following.

19. The image associated with the symbol includes multiple objects, The aforementioned processor device further, A step to determine whether the plurality of objects overlap in the image. The system according to claim 12, characterized in that it is programmed to perform the following.

20. The aforementioned image includes an object having a first boundary with a margin, and a second object having a second boundary with a second margin, The aforementioned processor device further, A step of determining whether the first boundary and the second boundary overlap in the image. The system according to claim 12, characterized in that it is programmed to perform the following.

21. Assigning the symbol to the object includes assigning the symbol to a surface. The system according to feature 12.

22. A method for assigning symbols to objects in an image, A step of receiving the image captured by the imaging device, wherein the symbol is located within the image; The steps include receiving the three-dimensional (3D) positions of one or more points corresponding to orientation information indicating the 3D orientation of one or more objects in a first coordinate system, The steps include mapping the 3D positions of one or more points of the object to 2D positions in the image in a second coordinate space, A step of determining the surface of the object based on the 2D positions of one or more points of the object in the image in the second coordinate space, The steps of assigning the symbol to the surface based on the relationship between the 2D position of the symbol in the image and the 2D position of one or more points of the object in the image, A method characterized by including the following.

23. Assigning the symbols to the surface includes determining the intersection between the surface and the image in the second coordinate space. The method according to the feature of 22.

24. Steps to determine the confidence score for the aforementioned symbol assignment. The method according to 22, characterized by including the following:

Images