System and method for three-dimensional data acquisition and processing under timing constraints

Through the coordinated acquisition and data combination of multiple depth camera groups, combined with buffer management and quality degradation strategies, the problem of low efficiency in obtaining three-dimensional data on the production line is solved, and efficient three-dimensional data acquisition and defect detection are achieved.

CN112424823BActive Publication Date: 2024-11-01PACKSIZE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980045032.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-05-04
Filing Date
2019-05-06
Publication Date
2024-11-01
Estimated Expiration
2039-05-06

AI Technical Summary

Technical Problem

The existing three-dimensional scanning system is difficult to efficiently obtain the complete three-dimensional data of the object under timing constraints, especially in the production line environment, where the object movement speed is uneven and the physical constraints are complex, resulting in low data acquisition efficiency and insufficient throughput.

Method used

Multiple depth camera groups are used to coordinate multiple views of the acquisition object, and combine data collected by different camera systems through the coordination server to generate a three-dimensional model of the object. The data processing process is optimized using buffer management and quality degradation strategies to adapt to uneven Object arrival rate and buffer storage limit.

Benefits of technology

It realizes efficient acquisition of three-dimensional data of objects under timing constraints, improves the throughput and data processing efficiency of the three-dimensional scanning system on the production line, and ensures real-time and accuracy of defect detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112424823B_ABST
    Figure CN112424823B_ABST
Patent Text Reader

Abstract

A system for acquiring a three-dimensional (3-D) model of an object includes a first camera group, the first camera group including: a first plurality of depth cameras having overlapping fields of view; a first processor; and a first memory storing instructions, the instructions, when executed by the first processor, causing the first processor to: control the first depth camera to simultaneously acquire a first set of images of a first portion of a first object; calculate a partial 3-D model representing the first portion of the first object; and detect defects in the first object based on the partial 3-D model representing the first portion of the first object.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the priority of U.S. Provisional Patent Application No. 62 / 666,942 filed in the U.S. Patent and Trademark Office on May 4, 2018, and its invention name is “SYSTEMS AND METHODS FOR THREE-DIMENSIONAL DATA ACQUISITION AND PROCESSING UNDER TIMING CONSTRAINTS”, and all the disclosed contents of which are incorporated herein by reference. Technical Field

[0003] Aspects of embodiments of the present invention relate to three-dimensional (3-D) scanning of an object, including acquiring 3-D data of the object, the 3-D data including the 3-D shape and surface texture of the object. Background Art

[0004] Three-dimensional (3-D) scanning systems can be used to capture 3-D data about an object. Conventional cameras capture a single two-dimensional (2-D) image of an object at a time. In contrast, a 3-D camera system can capture 3-D data about an object, which 3-D data includes information about the 3-D shape of the object. 3-D data can be represented as, for example, a "point cloud" (e.g., a collection of three-dimensional coordinates representing locations on the surface of an object) and a 3-D mesh model (e.g., a collection of polygons (e.g., triangles) arranged in 3-D space, where the polygons represent the surface of the object). Examples of 3-D camera systems (also called depth camera systems) and / or 3-D scanning systems include stereo camera systems and time-of-flight (ToF) cameras. See, for example, Hartley, Richard and Andrew Zisserman, et al., “Multiple View Geometry In Computer Vision,” Cambridge University Press, 2003; R. Szeliski, “Computer Vision: Algorithms and Applications,” Springer, 2010, pp. 467 et seq.; and P. Zanuttigh, et al., “Time-of-Flight and Structured Light Depth Camera,” Springer, 2015.

[0005] 3-D data acquired by a three-dimensional scanning system conveys more information about an object than a 2-D image of the object. 3-D data may sometimes be referred to as a depth map or a 3-D model (including point clouds and 3-D mesh models). For example, while a 2-D image typically provides only a single static view of an object, a user can typically manipulate the view of the 3-D data (e.g., by rotating, repositioning, and scaling the 3-D view or by changing the position of a virtual camera), thereby allowing the user to view the model to better understand the shape of the object represented by the 3-D data. Similarly, some techniques for automatically analyzing a scanned object (such as calculating the size and shape of the object) are more easily performed on a 3-D model of the object rather than on a separate 2-D image of the object. Summary of the invention

[0006] Various aspects of embodiments of the present invention relate to systems and methods for acquiring three-dimensional (3-D) data about an object, such as a 3-D scan of the object's physical shape and surface texture (e.g., color), which can be used to generate a 3-D model of the object.

[0007] Aspects of embodiments of the present invention also relate to systems and methods for coordinating multiple camera systems to acquire multiple views of an object and combining data acquired by the different camera systems to construct a 3-D model of the object.

[0008] According to one embodiment of the present invention, a system for acquiring a three-dimensional (3-D) model of an object includes a first camera group, wherein the first camera group includes: a first plurality of depth cameras with overlapping fields of view; a first processor; and a first memory storing instructions, wherein the instructions, when executed by the first processor, cause the first processor to: control the first depth camera to simultaneously capture a first set of images of a first portion of a first object; calculate a partial 3-D model representing the first portion of the first object; and detect defects in the first object based on the partial 3-D model representing the first portion of the first object.

[0009] The first camera group may further include a first start trigger configured to detect arrival of the object when the object enters an overlapping field of view of the first depth camera, and wherein the first processor may be configured to control the first depth camera of the first camera group to capture an image of the object in response to receiving a trigger signal from the first start trigger.

[0010] The first camera group may further include a first stop trigger configured to detect departure of the object from the overlapping field of view of the first depth camera, and the first processor may be configured to control the first depth camera of the first camera group to stop capturing images of the object in response to receiving a trigger signal from the first stop trigger.

[0011] The first camera group may further include a first prepare trigger configured to detect the presence of the object before the object enters the overlapping field of view of the first depth camera, and wherein the first processor may be configured to control the first depth camera of the first camera group to prepare to acquire an image of the object in response to receiving a trigger signal from the first prepare trigger.

[0012] The overlapping fields of view of the first depth cameras may be directed to a portion of a conveyor system configured to convey a plurality of objects, and the conveyor system may be configured to convey the objects one at a time into the overlapping fields of view of the first depth cameras.

[0013] The conveyor system can move at a non-uniform speed and the objects can arrive within the overlapping field of view of the first camera group at multiple different rates, the different rates including a maximum burst rate and an associated maximum burst time, and the first memory of the first camera group can include a buffer having a size sufficient to store images of the objects arriving at the maximum burst rate during the associated maximum burst time, the size being a function of at least a resolution of the first depth camera and a frame rate of the first depth camera.

[0014] The conveyor system may move at a non-uniform speed and the object may arrive within the overlapping field of view of the first camera group at multiple different rates, the first memory of the first camera group may include a buffer configured to store images captured by the first depth camera, and the first memory may further store instructions that, when executed by the first processor, cause the first processor to: determine a current buffer occupancy of the buffer; determine whether the current buffer occupancy exceeds a threshold; in response to determining that the current buffer occupancy does not exceed the threshold, set a configuration parameter of the first camera group to a nominal capture quality; and in response to determining that the current buffer occupancy exceeds the threshold: determine a new quality level based on multiple configuration settings stored in the first memory, the current buffer occupancy, and the current rate of the multiple different rates; and set the configuration parameter of the first camera group to the new quality level.

[0015] The system may further include a second camera group, comprising: a second plurality of depth cameras having overlapping fields of view, the second depth cameras being spaced apart from the first depth camera; a second processor; and a second memory storing instructions, which, when executed by the second processor, cause the second processor to: control the second depth camera to simultaneously capture a second set of images of a second portion of the first object; calculate a partial 3-D model representing the second portion of the first object; and detect defects in the first object based on the partial 3-D model representing the second portion of the first object.

[0016] The system may also include a coordination server, which includes a third processor and a third memory storing instructions, which when executed by the third processor causes the third processor to: receive the partial 3-D model representing the first part of the first object from the first camera group; receive the partial 3-D model representing the second part of the first object from the first camera group; and combine data from the partial 3-D model representing the first part of the first object and the partial 3-D model representing the second part of the first object.

[0017] The third memory may further store: a first buffer configured to store data from the first camera group; a second buffer configured to store data from the second camera group; and instructions that, when executed by the third processor, cause the third processor to: detect when both the first buffer and the second buffer store data corresponding to the first object; and combine data from the first camera group representing a first portion of the first object with data from the second camera group representing a second portion of the first object.

[0018] The instructions to combine data from the first camera group representing a first portion of the first object with data from the second camera group representing a second portion of the first object may include merging the partial 3-D model representing the first portion of the first object with the partial 3-D model representing the second portion of the first object.

[0019] The 3-D model may be a point cloud.

[0020] The 3-D model may be a mesh model.

[0021] Each depth camera may include: a first invisible light two-dimensional (2-D) camera having a first optical axis and a first field of view; a second invisible light 2-D camera having a second optical axis substantially parallel to the first optical axis of the first invisible light 2-D camera and having a second field of view overlapping the first field of view of the first invisible light 2-D camera; a color 2-D camera having a third optical axis substantially parallel to the first optical axis of the first invisible light 2-D camera and having a third field of view overlapping the first field of view of the first invisible light 2-D camera; and a projection source configured to emit invisible light in a portion of the electromagnetic spectrum detectable by the first invisible light 2-D camera and the second invisible light 2-D camera.

[0022] According to one embodiment of the present invention, a method for acquiring a three-dimensional (3-D) model of an object includes: controlling a first camera group including a first plurality of depth cameras with overlapping fields of view by a processor to simultaneously capture a first set of images of a first portion of a first object; calculating a partial 3-D model representing the first portion of the first object by the processor; and detecting defects in the first object based on the partial 3-D model representing the first portion of the first object.

[0023] The first camera group may further include a first start trigger configured to detect arrival of an object when the object enters an overlapping field of view of the first depth camera, and the method may also include controlling a first depth camera of the first camera group to capture an image of the object in response to a trigger signal received from the first start trigger.

[0024] The first camera group may further include a first stop trigger configured to detect departure of the object from the overlapping field of view of the first depth camera, and the method may further include controlling the first depth camera of the first camera group to stop capturing images of the object in response to a trigger signal received from the first stop trigger.

[0025] The first camera group may further include a first prepare trigger configured to detect the presence of the object before the object enters the overlapping field of view of the first depth camera, and the method may further include controlling the first depth camera of the first camera group to prepare to capture an image of the object in response to a trigger signal received from the first prepare trigger.

[0026] The overlapping fields of view of the first depth cameras may be directed to a portion of a conveyor system configured to convey a plurality of objects, and the conveyor system may be configured to convey the objects one at a time into the overlapping fields of view of the first depth cameras.

[0027] The conveyor system can move at a non-uniform speed and the objects can arrive within the overlapping field of view of the first camera group at multiple different rates, the different rates including a maximum burst rate and an associated maximum burst time, and the first camera group can include a memory, the memory including a buffer having a size sufficient to store images of the objects arriving at the maximum burst rate during the associated maximum burst time, the size being a function of at least a resolution of the first depth camera and a frame rate of the first depth camera.

[0028] The conveyor system may move at a non-uniform speed and the object may arrive within the overlapping field of view of the first camera group at multiple different rates, the first camera group may include a memory, the memory including a buffer configured to store images captured by the first depth camera, and the method may also include: determining a current buffer occupancy of the buffer; determining whether the current buffer occupancy exceeds a threshold; in response to determining that the current buffer occupancy does not exceed the threshold, setting the configuration parameters of the first camera group to a nominal capture quality; and in response to determining that the current buffer occupancy exceeds the threshold: determining a new quality level based on multiple configuration settings stored in the first memory, the current buffer occupancy and the current rate of the multiple different rates; setting the configuration parameters of the first camera group to the new quality level.

[0029] The method may also include: controlling, by the second processor, a second camera group including a second plurality of depth cameras having overlapping fields of view to simultaneously capture a second set of images of a second portion of the first object, the second depth cameras being separated from the first depth cameras; calculating, by the second processor, a partial 3-D model representing the second portion of the first object; and detecting defects in the first object based on the partial 3-D model representing the second portion of the first object.

[0030] The method may also include: receiving the partial 3-D model representing the first part of the first object from the first camera group through a coordination server; receiving the partial 3-D model representing the second part of the first object from the first camera group by the coordination server; and combining data from the partial 3-D model representing the first part of the first object and the partial 3-D model representing the second part of the first object through the coordination server.

[0031] The coordination server may include: a first buffer configured to store data from the first camera group; and a second buffer configured to store data from the second camera group, and the method may further include: detecting, by the coordination server, when both the first buffer and the second buffer store data corresponding to the first object; and combining, by the coordination server, data representing a first portion of the first object from the first camera group with data representing a second portion of the first object from the second camera group.

[0032] Combining data from the first camera group representing a first portion of the first object with data from the second camera group representing a second portion of the first object may include merging the partial 3-D model representing the first portion of the first object with the partial 3-D model representing the second portion of the first object.

[0033] The 3-D model may be a point cloud.

[0034] The 3-D model may be a mesh model.

[0035] Each depth camera may include: a first invisible light two-dimensional (2-D) camera having a first optical axis and a first field of view; a second invisible light 2-D camera having a second optical axis substantially parallel to the first optical axis of the first invisible light 2-D camera and having a second field of view overlapping the first field of view of the first invisible light 2-D camera; a color 2-D camera having a third optical axis substantially parallel to the first optical axis of the first invisible light 2-D camera and having a third field of view overlapping the first field of view of the first invisible light 2-D camera; and a projection source configured to emit invisible light in a portion of the electromagnetic spectrum detectable by the first invisible light 2-D camera and the second invisible light 2-D camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Various aspects of the embodiments of the present disclosure will become more apparent by reference to the following detailed description when considered in conjunction with the following drawings. In the drawings, similar reference numerals are used throughout the drawings to reference similar features and components. The drawings are not necessarily drawn to scale.

[0037] 1A is a schematic diagram of objects (depicted as handbags) traveling on a conveyor belt, utilizing multiple (five) cameras for simultaneous imaging, in accordance with one embodiment of the present invention.

[0038] 1B is a schematic diagram of an object (depicted as a handbag) traveling on a conveyor belt having two sections, wherein a first section moves the object in a first direction and a second section moves the object in a second direction orthogonal to the first direction, according to one embodiment of the present invention.

[0039] FIG. 2A is a schematic diagram of a camera group according to one embodiment of the present invention.

[0040] 2B is a schematic diagram of a depth camera suitable for use in a camera set according to one embodiment of the present invention.

[0041] 3A is a flow chart illustrating some stages of synthesizing a 3-D model according to one embodiment of the invention.

[0042] 3B is a flow chart illustrating a method for reducing the quality of a scanning process that continues scanning an object at the current throughput of the system.

[0043] 4A is a schematic diagram of multiple camera groups communicating with a coordination server according to one embodiment of the present invention.

[0044] 4B is a schematic diagram of correlation of data collected by multiple camera groups in communication with a coordination server according to one embodiment of the present invention.

[0045] FIG. 5 is a schematic diagram of a camera group with three triggers according to one embodiment of the present invention. DETAILED DESCRIPTION

[0046] In the following detailed description, only certain exemplary embodiments of the present invention are shown and described by way of illustration. As those skilled in the art will appreciate, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0047] Aspects of embodiments of the present invention relate to systems and methods for acquiring three-dimensional (3-D) data about an object, such as a 3-D scan of the physical shape and surface texture (e.g., color) of the object, which can be used to generate a 3-D model of the object. Aspects of embodiments of the present invention also relate to systems and methods for coordinating multiple camera systems to acquire multiple views of an object and combining data acquired by different camera systems to construct a 3-D model of the object. The acquired 3-D data can be used for visual analysis of the object, such as classifying the object and detecting defects in the object.

[0048] In order to capture a complete view of an object, whether using a two-dimensional camera system or a three-dimensional camera system, the camera system generally needs to capture all external visible surfaces of the object. This can generally be achieved by holding the camera in place while rotating the object in front of the camera, moving the camera around the object, or a combination of the foregoing. See, for example, U.S. Patent No. 9,912,862, "SYSTEM AND METHOD FOR ASSISTED 3D SCANNING", issued on March 6, 2018, the entire disclosure of which is incorporated herein by reference. In some cases, views from multiple different camera systems can be combined to capture sufficient views of the object. See, for example, U.S. Patent Application No. 15 / 866,217, "SYSTEMS AND METHODS FOR DEFECT DETECTION", filed with the U.S. Patent and Trademark Office on January 9, 2018, the entire disclosure of which is incorporated herein by reference. Techniques for synthesizing depth images from multiple images have been disclosed in, for example, Hartley, Richard and Andrew Zisserman, et al., Multiple View Geometry In Computer Vision, Cambridge University Press, 2003, and R. Szeliski, Computer Vision: Algorithms and Applications, Springer, 2010, pp. 467 et seq. For convenience, the discussion of acquiring a full view of an object excludes a detailed discussion of acquiring the bottom surface of the object, which is typically obscured by the surface supporting the object. However, embodiments of the present invention are not limited thereto and may encompass scanning systems that acquire the bottom surface of an object.

[0049] In some environments, physical and time constraints may limit the manner in which objects may be scanned. For example, in the case of a factory or other manufacturing process, objects may be moved along a conveyor system, where the objects (products) are modified or processed by people working at workstations and / or automated machinery. The arrangement of existing workstations and existing machinery may impose physical constraints on the placement of cameras to image the objects unobstructed without interfering with the manufacturing process. In addition, to avoid slowing down the existing process, the scanning system may acquire scans of the objects at a rate imposed by the flow rate through the production line.

[0050] For convenience, embodiments of the present invention will be described in the context of manufacturing products in a factory. In addition, various aspects of embodiments of the present invention are described in the context of using a captured 3-D model of the object to detect defects in the object. However, embodiments of the present invention are not limited to this and may also be applied to other situations and used to perform other types of analysis. These situations may involve similar physical constraints and time constraints, which lead to situations where embodiments of the present invention can solve data acquisition problems caused by these constraints. Examples include scanning products in a food processing plant to classify the products (e.g., assign grades) and scanning packages in a distribution or transportation center (e.g., identifying and routing the packages).

[0051] FIG. 1A is a schematic diagram of an object 10 (depicted as a handbag) traveling on a conveyor belt 12, wherein the object 10 is imaged simultaneously using multiple (five) depth cameras 100 (labeled 100a, 100b, 100c, 100d, and 100e) according to one embodiment of the present invention. For convenience, the depth cameras 100 will be referred to herein as "cameras" 100. The fields of view 101 of the cameras (labeled 101a, 101b, 101c, 101d, and 101e) are depicted as triangles with different shading, and show different views (e.g., surfaces) of the object 10 acquired by the cameras 100. For ease of depiction, the fields of view are represented as triangles, and in the general case of depth cameras, such fields of view can also be approximately characterized as pyramid shapes. The camera 100 may include color and infrared (IR) imaging units to capture geometric and texture characteristics of an object (e.g., the camera may be a stereo depth camera, such as the camera described in U.S. Patent Application Serial No. 15 / 147,879, filed with the U.S. Patent and Trademark Office on May 5, 2016, “Depth Perceptive Trinocular Camera System”, issued on June 6, 2017 as U.S. Patent No. 9,674,504). Each individual depth camera 100 may include at least two image sensors configured to focus light onto their respective image sensors and a corresponding optical system. The optical axes of the optical system may be substantially parallel so that the two image sensors capture images of a “stereo pair”, (e.g., two images of the same scene taken from slightly different viewpoints, where the viewpoints are separated by a baseline distance). Each camera may include one or more computing units, such as, but not limited to, a computing processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC). One example of the depth camera 100 will be described in more detail below with reference to FIG. 2B .

[0052] The cameras 100 can be arranged around the conveyor belt 12 so that they do not hinder the movement of the object 10 as the object moves along the conveyor belt 12. The camera arrangement can be generated as a result of a configuration process or an optimization process, in which a model of the target object and a model of movement on the conveyor belt are taken into account in order to obtain at least a minimum level of accuracy and completeness in the resulting 3-D model. For example, examples of coverage can be seen, for example, in U.S. Patent Application No. 15 / 866,217, filed with the U.S. Patent and Trademark Office on January 9, 2018, "SYSTEMS AND METHODS FOR DEFECT DETECTION", the entire disclosure of which is incorporated herein by reference. Some factors include the desired resolution of the resulting scan, the size of the smallest surface feature of the object that needs to be detected (e.g., the size of the smallest defect), the resolution of the individual depth cameras 100, the focal length of the camera, the light available in the environment, and the speed of movement of the object.

[0053] The cameras may be stationary and configured to capture images when at least a portion of the object 10 enters their respective fields of view (FOV) 101. The cameras 100 may be arranged so that as the object 10 moves along the conveyor belt 12, the combined camera FOVs 101 cover all critical (e.g., visible) surfaces of the object 10, and at a resolution suitable for the purpose of the captured 3-D model (e.g., to have more detail around the stitching attaching the handle to the bag). The captured images may be used to synthesize a 3-D model of the object 10.

[0054] As described above, in some cases, physical constraints hinder or prevent the installation of a camera system (or camera group) that can cover all critical surfaces of the object 10 at a single location. For example, in a production line, existing equipment or space allocated for human employees to work on the object 10 may occupy a large amount of space around the conveyor belt, leaving only a few places available for the camera to be placed. In addition, these remaining places may not allow the camera to completely surround the object, thereby hindering the ability to capture all critical surfaces of the object at one location (e.g., one point in the production line) at substantially the same time.

[0055] For convenience, the term "critical surface" will be used to refer to all surfaces of the object that are of interest for scanning. In more detail, for the purposes of defect detection, critical surfaces may include specific portions of an object that are susceptible to failure, such as the seams of a handbag, where the quality of the stitching and alignment of the fabric may be of particular interest, or the stitching associated with the attachment of a zipper or handle.

[0056] Therefore, some embodiments of the present invention are directed to a 3-D data acquisition system in which multiple camera groups are spaced apart from each other along a production line. Each camera group may include multiple depth cameras (e.g., stereo depth cameras), and the depth cameras of each group may be controlled together to capture images of a portion (e.g., a subset) of the key surfaces of an object 10 substantially simultaneously. In some embodiments, the camera groups are arranged so that the combined field of view of the cameras of all camera groups substantially captures all key surfaces of the object. In some embodiments, the fields of view of different camera groups do not overlap or do not overlap with each other.

[0057] According to one embodiment of the present invention, as an example of a camera arrangement, FIG. 1B is a schematic diagram of an object 10 (depicted as a handbag) traveling on a conveyor belt 12 having two sections, wherein a first section 12a of the conveyor belt 12 moves the object 10 along a first direction (-y direction), and a second section 12b of the conveyor belt 12 moves the object 10 along a second direction (+x direction) orthogonal to the first direction. When the object 10 travels along the first section 12a of the conveyor belt 12, at a first position, a first camera 100a images the top (+z) surface of the object 10 from above, while a second camera 100b images the +x side of the object. The first camera 100a and the second camera 100b may constitute a first camera group 130ab. At a second position on the first section 12a of the conveyor belt, a third camera 100c and a fourth camera 100d of a second camera group 130cd image the -x side of the object 10.

[0058] In this arrangement, it may be difficult to image the end of the object 10, since doing so would require placing the camera along the direction of movement of the conveyor and thus may hinder the movement of the object 10. Therefore, the object 10 can be transitioned (without rotation) to the second portion 12b of the conveyor 12, wherein, after the transition, the -y side of the object 10 is now visible relative to the cameras 100e and 100f of the third camera group 130ef at the third position. In the fourth position, the cameras 100g and 100h of the fourth camera group 130gh image the +y side of the object. Assuming that the cameras 100 of the camera group 130 are substantially stationary and the object moves along the conveyor system without rotation, each camera can capture multiple images of the object as it passes.

[0059] 1B shows an example of an arrangement of camera groups 130 that allows coverage of the entire visible surface of object 10. In some embodiments of the present invention, data acquired by cameras of different camera groups 130 are combined to synthesize a single 3-D model of the object (e.g., a global model of the entire object 10). In some embodiments, the data acquired by each camera group 130 is processed separately (e.g., several separate 3-D models are generated), without combining all acquired data into a single "global" 3-D model.

[0060] Although FIG. 1B depicts camera groups adjacent to each other, in many cases, the camera groups may be separated by significant distances, such as one or more workstations located between the camera groups. Manufacturing equipment and / or people may perform tasks on objects located in each of these workstations. Thus, different camera groups may capture images of products in different states (e.g., different stages of assembly). In some embodiments, the camera groups are arranged so that they capture only portions of an object that are expected to be completed or ready at a specific location (for a specific manufacturing stage). Thus, the flow of objects along the production line may be uneven. For example, in some processes, objects may be processed serially (e.g., one at a time) before serial operation is resumed, and batch operations may be performed on objects periodically (e.g., multiple objects may be grouped together and at the same time), potentially resulting in "pulses" of objects. As another example, a problem may occur at a workstation that may temporarily hold multiple objects while the problem is being solved, and then release all held objects at once, resulting in an accidental burst of objects. Similarly, a temporary hold at a workstation may result in a corresponding drop in the arrival rate of objects in downstream workstations of the production line.

[0061] Thus, a significant amount of time may elapse between the acquisition of data by an earlier camera group and the acquisition of data by a later camera group, thereby waiting time to synthesize a 3-D model of an object from the acquired data. In addition to the waiting time between the time a particular object is seen by one camera group and the time the same object is seen by another camera group, the throughput of the production line may also mean that an earlier camera group will acquire data for many additional objects before the particular object reaches a later camera group. Thus, data acquired by an earlier camera group may need to be buffered in order to be combined with later acquired data for the same object.

[0062] Camera Group

[0063] FIG2A is a schematic diagram of a camera group 130ijk according to one embodiment of the present invention. As shown in FIG2A , three cameras CAM1, CAM2, and CAM3 (respectively labeled 100i, 100j, and 100k) are configured to capture overlapping images of different portions of an object 10 on a conveyor system 12. The capture of the images may be triggered by a trigger system, which may include a start trigger 28 that detects when an object 10 enters the field of view of the cameras 100i, 100j, and 100k. The start trigger 28 of the trigger system may include a laser transmitter that is configured to send a trigger signal to a controller 24 (e.g., a computer or microcontroller) when the laser signal is interrupted by the presence of the object 10. The controller 24 may then control the cameras 100i, 100j, and 100k to begin capturing images of the object. In some embodiments of the present invention, the camera group 130ijk may include multiple triggers (see FIG. 5 ), such as an additional trigger (stop trigger) for detecting when an object has left the field of view of the cameras 100i, 100j, and 100k, and / or a trigger (prepare trigger) for detecting when the object 10 approaches the camera group 130ijk, thereby allowing the camera group 130ijk to perform setup or initialization operations before the object arrives. The cameras 100i, 100j, and 100k may be connected to the controller 24 via a peripheral interface base such as a universal serial bus (USB). In some other embodiments, the trigger may be obtained directly from imaging information acquired by one or more cameras in the group, such as by processing the acquired image data at a lower resolution and / or by utilizing a proximity sensor or an additional illumination source.

[0064] The controller 24 may also be connected to a network 26 (e.g., an Ethernet 802.3 network or a wireless LAN 802.11 network) to communicate with other devices, such as a coordination server computer 30 and / or other camera groups 130. For example, data collected by cameras 100i, 100j, and 100k may be transmitted to the coordination server 30 via the network 2626.

[0065] The various computing devices described herein (including the controller 24 and the coordination server 30) may include one or more processors (e.g., central processing units, graphics processing units, field programmable gate arrays, and application specific integrated circuits) coupled to a memory (e.g., dynamic memory and / or persistent memory) that stores instructions that configure the computing device to perform the specific specialized functions described herein. The one or more processors may communicate with other devices (such as the camera 100) through peripheral input / output devices, such as a network adapter and a universal serial bus (USB) controller.

[0066] Depth Camera Hardware

[0067] In some embodiments of the present invention, the depth camera 100 (also referred to as a range camera) may include at least two standard two-dimensional cameras with overlapping fields of view. In more detail, these two-dimensional (2-D) cameras may each include a digital image sensor, such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor, and an optical system (e.g., one or more lenses) configured to focus light onto the image sensor. The optical axes of the optical systems of the 2-D cameras may be substantially parallel so that the two cameras image substantially the same scene, albeit from slightly different perspectives. Thus, due to parallax, portions of the scene that are farther from the cameras will appear in substantially the same position in the images captured by the two cameras, while portions of the scene that are closer to the cameras will appear in different positions.

[0068] Using a geometrically calibrated depth camera, the 3-D positions of all visible points on the surface of an object can be identified relative to a reference coordinate system (e.g., a coordinate system having its origin at the depth camera). Thus, a range image or depth image acquired by a range camera 100 can be represented as a cloud of 3-D points that can be used to describe portions of the surface of an object (as well as other surfaces within the field of view of the depth camera).

[0069] 2B is a block diagram of a stereo depth camera system according to one embodiment of the present invention. The depth camera system 100 shown in FIG. 2B includes a first camera 102, a second camera 104, a projection source 106 (or an illumination source or an active projection system), and a host processor 108 and a memory 110, wherein the host processor may be, for example, a graphics processing unit (GPU), a more general processor (CPU), a suitably configured field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The first camera 102 and the second camera 104 may be rigidly attached to a frame, for example, so that their relative positions and orientations are substantially fixed. The first camera 102 and the second camera 104 may be collectively referred to as a “depth camera”. The first camera 102 and the second camera 104 include respective image sensors 102a and 104a, and may also include respective image signal processors (ISPs) 102b and 104b. The various components may communicate with each other via a system bus 112. The depth camera system 100 may include additional components, such as a network adapter 116 for communicating with other devices, an inertial measurement unit (IMU) 118 such as a gyroscope to detect acceleration of the depth camera 100 (e.g., to detect the direction of gravity to confirm orientation), and persistent memory 120, such as NAND flash memory for storing data collected and processed by the depth camera system 100. The IMU 118 may be of the type commonly found in many modern smartphones. The image acquisition system may also include other communication components, such as a universal serial bus (USB) interface controller. In some embodiments, the depth camera system 100 also includes a display device 122 and one or more user input devices 124 (e.g., a touch-sensitive panel and / or one or more physical buttons or triggers of the display device 122).

[0070] Although the block diagram shown in FIG2B depicts a depth camera 100 that includes two cameras 102 and 104 coupled to a host processor 108, a memory 110, a network adapter 116, an IMU 118, and a persistent memory 120, embodiments of the present invention are not limited thereto. For example, the three depth cameras 100 shown in FIG2A (described in more detail below) may each include only cameras 102 and 104, a projection source 106, and a communication component (e.g., a USB connection or a network adapter 116), and processing of the two-dimensional images captured by the cameras 102 and 104 of the three depth cameras 100 may be performed by a shared processor or a collection of shared processors that communicate with the depth cameras 100 using their respective communication components or network adapters 116. For example, the controller 24 of FIG2A may be used to process 2-D images received from cameras 100i, 100j, and 100k to generate three separate depth images corresponding to the views captured by the cameras 100i, 100j, and 100k.

[0071] In some embodiments, the image sensors 102a and 104a of the cameras 102 and 104 are RGB-IR image sensors. An image sensor capable of detecting visible light (e.g., red-green-blue or RGB) and invisible light (e.g., infrared light or IR) information may be, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor. Typically, a conventional RGB camera sensor includes pixels arranged in a "Bayer layout" or "RGBG layout," which is 50% green, 25% red, and 25% blue. According to the Bayer layout, a bandpass filter (or "microfilter") is placed in front of each photodiode (e.g., between the photodiode and the optics associated with the camera) for each of the green, red, and blue wavelengths. Typically, a conventional RGB camera sensor also includes an infrared (IR) filter or IR cut filter (e.g., formed as part of a lens or as a coating on the entire image sensor chip) that further blocks signals in the IR portion of the electromagnetic spectrum.

[0072] RGB-IR sensors are substantially similar to conventional RGB sensors, but may include different color filters. For example, in an RGB-IR sensor, one of the green light filters in each group of four photodiodes is replaced by an IR bandpass filter (or microfilter) to produce a layout of 25% green light, 25% red light, 25% blue light, and 25% infrared light, where infrared light pixels are mixed between visible light pixels. In addition, the RGB-IR sensor may omit the IR cutoff filter, the IR cutoff filter may be located only on pixels that detect red light, green light, and blue light, or the IR filter may be designed to pass visible light as well as light in a specific wavelength range (e.g., 840-860nm). An image sensor capable of collecting light in multiple parts or bands or spectral bands of the electromagnetic spectrum (e.g., red light, blue light, green light, and infrared light) will be referred to herein as a "multi-channel" image sensor.

[0073] In some embodiments of the present invention, image sensors 102a and 104a are conventional visible light sensors (e.g., RGB sensors). In some embodiments of the present invention, the system includes one or more visible light cameras (e.g., RGB cameras) and separately includes one or more invisible light cameras (e.g., infrared cameras, in which IR bandpass filters are located on all pixels). In other embodiments of the present invention, image sensors 102a and 104a are infrared (IR) light sensors. In some embodiments of the present invention, image sensors 102a and 104a are infrared (IR) light sensors. In some embodiments (such as those in which image sensors 102a and 104a are IR sensors), the depth camera 100 may include a third camera 105, which includes a color image sensor 105a (e.g., an image sensor configured to detect visible light of red, green, and blue wavelengths, such as an image sensor arranged in a Bayer layout or an RGBG layout) and an image signal processor 105b.

[0074] In some embodiments where the depth camera 100 includes a color image sensor (e.g., an RGB sensor or an RGB-IR sensor), the color image data collected by the depth camera 100 can supplement the color image data collected by the color camera 150. In addition, in some embodiments where the depth camera 100 includes a color image sensor (e.g., an RGB sensor or an RGB-IR sensor), the color camera 150 can be omitted from the system.

[0075] In general, a stereo depth camera system includes at least two cameras spaced apart from each other and rigidly mounted to a shared structure such as a rigid frame. The cameras are oriented in substantially the same direction (e.g., the optical axes of the cameras may be substantially parallel) and have overlapping fields of view. These individual cameras may be implemented using, for example, a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, with an optical system (e.g., including one or more lenses) configured to direct or focus light onto the image sensor. The optical system may determine the field of view of the camera, for example, based on whether the optical system implements a "wide angle" lens, a "telephoto" lens, or something in between.

[0076] In the following discussion, the image acquisition system of the depth camera system may be referred to as having at least two cameras, which may be referred to as a "master" camera and one or more "slave" cameras. In general, the estimated depth or disparity map is calculated from the viewpoint of the master camera, but any camera can be used as a master camera. As used herein, unless otherwise specified, terms such as master / slave, left / right, up / down, and first / second are used interchangeably. In other words, any one of the cameras can be a master camera or a slave camera, and the consideration of the camera on the left relative to the camera on its right can also be applied to other directions by symmetry. In addition, although the considerations given below may be valid for various numbers of cameras, for convenience, they will generally be described in the case of a system including two cameras. For example, the depth camera system may include three cameras. In such a system, two cameras may be invisible light (infrared) cameras, and the third camera may be a visible light (e.g., red / blue / green color camera) camera. All three cameras can be optically registered (e.g., calibrated) relative to each other. An example of a depth camera system including three cameras is disclosed in U.S. Pat. No. 9,674,504, issued on June 6, 2017, the entire disclosure of which is incorporated herein by reference. Such a three-camera system may also include an infrared light illuminator configured to emit light in a wavelength range detectable by the infrared light camera (e.g., 840-860 nm).

[0077] To detect the depth of features in a scene imaged by the camera, the depth camera system determines the pixel position of the feature in each image captured by the camera. The distance between features in two images is called disparity, which is inversely proportional to the distance or depth of the object. (This is the effect of comparing how much an object "shifts" when viewed with one eye at a time - the size of the shift depends on how far the object is from the viewer's eye, with closer objects producing larger shifts and farther objects producing smaller shifts and distant objects may have little to no detectable shift). For example, R. Szeliski, "Computer Vision: Algorithms and Applications", Springer, 2010, pp. 467 et seq.

[0078] The amount of parallax between the master and slave cameras depends on the physical properties of the depth camera system, such as the pixel resolution of the cameras, the distance between the cameras, and the field of view of the cameras. Therefore, in order to generate accurate depth measurements, the depth camera (or depth-aware depth camera system) is calibrated based on these physical properties.

[0079] In some depth camera systems, the cameras may be arranged so that horizontal rows of pixels of the camera's image sensor are substantially parallel. Image correction techniques may be used to accommodate distortion of the image due to variations in the shape of the camera's lens and the orientation of the camera.

[0080] In more detail, the camera calibration information can provide information to correct the input image so that the epipolar lines of the equivalent camera system are aligned with the scan lines of the corrected image. In this case, the 3-D points in the scene are projected onto the same scan line index in the master and slave images. Let u m and u s are the coordinates on the scan lines of the images of the same 3-D point p in the master and slave equivalent cameras, respectively, where in each camera these coordinates refer to an axis system centered at the principal point (intersection of the optical axis with the focal plane) with the horizontal axis parallel to the scan lines of the rectified image. The difference us-um is called the parallax and is denoted by d; it is inversely proportional to the orthogonal distance of the 3-D point relative to the rectified cameras (i.e., the length of the orthogonal projection of the point onto the optical axis of either camera).

[0081] Stereo algorithms take advantage of this property of disparity. These algorithms achieve 3-D reconstruction by matching points (or features) detected in the left and right views, which is equivalent to estimating disparity. Block matching (BM) is a commonly used stereo algorithm. Given a pixel in the main camera image, the algorithm calculates the cost to match the pixel with any other pixel in the slave camera image. The cost function is defined as the dissimilarity between the image content within a small window around the pixel in the main image and the pixel in the slave image. Finally, the optimal disparity at the point is estimated as the independent variable of the minimum matching cost. This process is usually called Winner-Take-All (WTA). These techniques are described in more detail in, for example, R. Szeliski, "Computer Vision: Algorithms and Applications", Springer, 2010. Since stereo algorithms like BM rely on appearance similarity, if more than one pixel in the slave image has the same local appearance, disparity calculation becomes challenging because all these pixels may be similar to the same pixel in the main image, resulting in blurred disparity estimates. A typical situation where this might happen is when presenting a scene with constant brightness, such as a flat wall.

[0082] There are methods of providing additional illumination by projecting patterns designed to improve or optimize the performance of block matching algorithms, which can pick up small 3-D details, such as the details described in U.S. Patent No. 9,392,262, issued on July 12, 2016, "System and Method for 3-D Reconstruction Using Multiple Multi-Channel Cameras," the entire disclosure of which is incorporated herein by reference. Another method projects a pattern that is used purely to provide texture to the scene, and specifically improves depth estimation in textureless areas by disambiguating parts of the scene that otherwise appear the same.

[0083] Projection source 106 according to embodiments of the present invention may be configured to emit visible light (e.g., light within a spectrum visible to humans and / or other animals) or invisible light (e.g., infrared light) toward a scene imaged by cameras 102 and 104. In other words, the projection source may have an optical axis substantially parallel to the optical axes of cameras 102 and 104, and may be configured to emit light in the direction of the field of view of cameras 102 and 104, wherein the emitted light is in a portion of the electromagnetic spectrum detectable by cameras 102 and 104 (e.g., when cameras 102 and 104 are invisible light or infrared light cameras, projection source 106 projects light in the invisible light or infrared light portion of the electromagnetic spectrum). An arrangement in which two cameras 102 and 104 are arranged with projection source 106 in this manner is sometimes referred to as "active stereo". In some embodiments, projection source 106 may include a plurality of separate illuminators, each illuminator having an optical axis (or multiple optical axes) spaced apart from the optical axes of the other illuminators (or multiple illuminators), and spaced apart from the optical axes of cameras 102 and 104.

[0084] Invisible light projection sources may be better suited for situations where the subject is a person (such as in a video conferencing system) because the invisible light does not interfere with the subject's visual abilities, whereas visible light projection sources may shine uncomfortably into the subject's eyes or may undesirably affect the experience by adding patterns to the scene. Examples of systems including invisible light projection sources are described, for example, in U.S. Patent No. 9,516,295, issued December 6, 2016, "Systems and Methods for Multi-Channel Imaging Based on Multiple Exposure Settings," the entire disclosure of which is incorporated herein by reference.

[0085] Active projection sources can also be classified as projecting static patterns, e.g., patterns that do not change over time, and dynamic patterns, e.g., patterns that change over time. In both cases, one aspect of the pattern is the illumination level of the projected pattern. This can be relevant because it can affect the depth dynamic range of the depth camera system. For example, if the optical illumination is at a high level, depth measurements of distant objects can be made under bright ambient light conditions (e.g., to overcome the decrease in optical illumination at the distance to the object by a factor proportional to the inverse of the square of the distance). However, high optical illumination levels can result in saturation of close-up portions of the scene. On the other hand, low optical illumination levels can allow measurement of close objects, but not distant objects.

[0086] Depth calculations may fail in some areas due to multiple factors, including: the mechanism used to calculate depth (triangulation, with or without active illuminators or time of flight); the geometry of the scene (such as the angle between each surface element and the associated line of sight, or the presence of partial occlusions that may obstruct the view of either sensor in a stereo system); and the reflectivity properties of the surface (e.g., the presence of a specular component that may hinder stereo matching or reflect light away from a projector, or very low albedo that results in insufficient light being reflected by the surface). For those pixels where the depth calculation fails or the depth image is unreliable, only color information is available.

[0087] Although embodiments of the present invention are described herein with respect to stereo depth camera systems, embodiments of the present invention are not limited thereto and may also be used with other depth camera systems (eg, structured light time of flight cameras and LIDAR cameras).

[0088] Depending on the choice of camera, different techniques can be used to generate the 3-D model. For example, Dense Tracking and Mapping in Real Time (DTAM) uses color cues to scan, and Simultaneous Localization and Mapping (SLAM) uses depth data (or a combination of depth and color data) to generate a 3D model.

[0089] Typically, the process of computing a 3-D model from images acquired by a camera includes several processing stages. FIG. 3A is a flow chart illustrating some stages of synthesizing a 3-D model according to one embodiment of the present invention. Referring to FIG. 3A , in operation 310, the cameras 100 of the camera group 130 are controlled to acquire images of the object 10 (e.g., 2-D images acquired by individual 2-D cameras 102 , 104 , and 105 of the depth camera 100 ). In some embodiments of the present invention, the controller 24 associated with the camera group 130 controls the cameras 100 of the camera group to acquire images substantially simultaneously (e.g., based on detecting the presence of the object 10 using the trigger system 28 , and in some embodiments, using a synchronization signal sent by the controller 24 to synchronize the acquisition of a single frame by the camera 100 ).

[0090] The data collected by each depth camera 100 is used to generate a partial point cloud by a point cloud generation module in operation 320. For example, each depth camera 100 may include at least one stereo pair of cameras (e.g., 2-D cameras 102, 104, and 105). Block matching can be used to match features in an image pair collected by the stereo pair, and the matching blocks can be used to calculate a disparity map, which is used to calculate a point cloud. In some embodiments, each depth camera 100 includes a point cloud generation module. For example, the point cloud generation module can be implemented in software stored in the memory 110 and / or persistent storage 120 of the depth camera 100 and executed by the host processor 108 of the depth camera 100, so that each depth camera 100 calculates a partial point cloud corresponding to the view of the object collected by its corresponding 2-D camera 102, 104, and 105.

[0091] In one embodiment, the point cloud merging module merges the separate partial point clouds in operation 330 to generate a merged point cloud for the entire object 10. Assuming that the poses of the cameras 100 within the camera group 130 relative to each other are known (the cameras are "calibrated"), the cameras are synchronized (e.g., controlled to acquire images substantially simultaneously), and there are accurate timestamps for the images acquired by the cameras, the necessary rigid transformations to map the point clouds acquired by the separate depth cameras 100 into a consistent coordinate system are known and straightforward. (In the case of an uncalibrated system, iterative closest point or ICP can be used to align the point clouds.)

[0092] In some embodiments, in operation 340, a 3-D mesh model generation module generates a 3-D mesh model from the merged point cloud. In some embodiments, the 3-D model is a 3-D mesh model. Examples of techniques for converting a point cloud into a 3-D mesh model include Delaunay triangulation and α-shapes for connecting adjacent points of a point cloud using the edges of a triangle. In some embodiments, a MeshLab software package is used to convert a point cloud into a 3-D mesh model (see, e.g., P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, G. Ranzuglia MeshLab: an Open-Source Mesh Processing Tool Sixth Eurographics Italian Chapter Conference, pp. 129-136, 2008. In some embodiments, operation 340 is omitted and the merged point cloud is considered to be a generated 3-D model.

[0093] In operation 350, defect detection may be performed on the merged point cloud, or, if operation 340 is performed, on the resulting 3-D model. In some embodiments, the calculated 3-D model may be compared to a reference 3-D model of the object. In some embodiments, the 3-D model may also include additional reflectivity information, such as a bidirectional reflectance function (BDRF). For additional details on defect detection, see, for example, U.S. Patent Application No. 15 / 866,217, filed with the U.S. Patent and Trademark Office on January 9, 2018, "SYSTEMS AND METHODS FOR DEFECT DETECTION". Additional examples of techniques include supplying the acquired 3-D model to a neural network trained to detect defects. Such a neural network may be, for example, a deep neural network including one or more convolutional layers. Providing the acquired 3-D model to the neural network may include rendering multiple views of the 3-D model (e.g., rendering 2-D views) and pooling feature vectors calculated from separate views (e.g., using max pooling).

[0094] Each operation shown in FIG3A may incur a certain degree of processing time or processing latency. For example, for one combination of computing hardware, the number of cameras, and the resolution of camera 100, acquiring an image of an object using a depth camera in operation 310 may take approximately 3 seconds and computing a point cloud in operation 320, merging the point clouds in operation 330, and detecting a defect in operation 350 may each take approximately 10 seconds. Thus, the processing latency for detecting a defect in a single object using such a system may be approximately thirty-three seconds. In addition, transmitting the raw acquisition data and the computed 3-D model and / or merged point cloud over a network may take approximately 10 seconds. These specific numbers are given as representative examples based on current technology.

[0095] Given this thirty-three second processing time, and assuming that the operations are performed locally by the controller 24 of the camera group 130 and that these operations are pipelined across multiple cores of a multi-core processor, the camera group 130 is limited to a throughput of slightly less than two objects scanned per minute. (In other words, the arrival rate of objects 10 within the field of view of the system will need to be less than two objects per minute).

[0096] However, as described above, in many manufacturing settings, the conveyor system of the production line may move at inconsistent speeds because the line may slow down or stop based on problems at a particular workstation, shift changes, worker breaks, etc., and the line may be temporarily accelerated to make up for lost time and / or clear a buffer of objects. The average arrival rate of objects at the camera group, as well as the variance and maximum of the arrival rate, impose timing constraints on the data acquisition and processing system.

[0097] While these processing times apply to one particular choice of image acquisition resolution, the same overall latency issues will generally apply to other choices of hardware. A particular combination of hardware may be selected based on the timing constraints of a particular application (e.g., the average arrival rate for a particular production line). For example, acquisition times may vary based on the frame rate of the camera (e.g., acquiring images of a sufficient number of objects) and the speed of the conveyor system (e.g., the amount of time an object is in the camera's field of view), and processing speeds may increase or decrease with less powerful or more powerful computers (e.g., faster processors, more specialized processors, multiple processing cores, etc.) and with higher or lower resolution cameras. Additionally, different combinations and configurations of pipelines may be used to accommodate higher average throughputs.

[0098] To handle temporary bursts of objects, the controller 24 of the camera group 130 according to some embodiments of the present invention includes a buffer memory for storing or queuing data during bursts of high object throughput. In some embodiments, the size of the buffer memory of the controller 24 can be set based on the maximum burst rate of the manufacturing line (and the associated amount of burst time that the maximum burst rate can be maintained) and the size of the data acquired by the camera and generated by the controller 24 (e.g., the size of the point cloud). These buffer memories store 2D image data acquired, for example, by a 2D camera of the depth camera 100, while calculations related to earlier acquisitions (e.g., images of previous objects) are analyzed according to Figure 3A (e.g., to calculate point clouds, merge point clouds, perform defect detection, etc.). After the processing time is faster than the time between objects again, the scanning system can recover by processing the data queued in the buffer.

[0099] In some embodiments of the present invention, when the buffer memory is insufficient to store all of the collected data during a burst, a best effort attempt may be made to retain at least some of the data. Situations where the buffer memory may not have sufficient space include the following: the network connection is offline and data cannot be transmitted from the controller 24; objects arrive at a higher rate than expected; hardware failure (e.g., storage module), etc. Best efforts to reduce the amount of data (e.g., reduce the quality of the output) may include: reducing the acquisition frame rate of the camera, compressing images acquired by the camera (with lossless or lossy compression), reducing the resolution of the point cloud (e.g., removing points), etc.

[0100] Different forms of quality degradation may result in different maximum throughputs of the scanning system. For example, compressing the image may reduce transmission time and therefore increase throughput, but the maximum throughput may still be limited due to the additional processing (e.g., CPU) overhead associated with compression performed by the controller 24. On the other hand, reducing the camera's acquisition frame rate may reduce the overall processing load, but reducing the acquisition rate too much may result in the inability to acquire some objects (e.g., objects that pass between frames) or the inability to acquire some surfaces of the object (e.g., if in a typical acquisition mode the camera acquires images of an object at three points in time with slightly different poses relative to the object, then data from all three poses will be available, whereas if the acquisition rate is reduced to a single point in time for each object, then data from other poses will not be available).

[0101] In addition, different specific applications may have different quality requirements for the scanning process. For example, detecting defects in certain parts of the object may have a higher importance (high priority) than detecting defects in other low priority parts. Therefore, the quality of the images acquired for the low priority surfaces of the object may be reduced before reducing the quality of the images of the high priority surfaces.

[0102] 3B is a flow chart illustrating a method 360 for reducing the quality of the scanning process to continue scanning objects at the current throughput of the system. For convenience, the term "buffer occupancy" will be used herein to refer to the amount of data stored in a buffer of the system ("buffer"), where the amount may be expressed in, for example, a number of bytes, a number of objects scanned, a percentage or fraction of the total buffer size, etc. In operation 362, the controller 24 determines the current buffer occupancy, and in operation 364, the controller 24 (or the coordination server computer 30) determines whether the buffer occupancy of the buffer exceeds a threshold level.

[0103] For example, where buffer occupancy is tracked as the number of scans of an object, a threshold may be determined based on the rate at which current scans are processed and removed from the buffer and the rate at which new scans are captured and it is determined that the buffer may soon overflow (e.g., a write to the buffer will fail because the buffer has reached full capacity or full occupancy) based on current conditions. As a specific working example, if the buffer currently stores scans of four different objects and has room for two more, and each scan takes 30 seconds to be processed, and objects arrive at a rate of one object every 15 seconds, the buffer may overflow in less than a minute. In some embodiments, the arrival rate is estimated from the rate at which objects are processed by camera group 130 earlier in the line (e.g., "upstream").

[0104] As another example, when the threshold is set as a percentage, the threshold can be set to a specific percentage of the total buffer size, such as 60% of the buffer size. In some embodiments, the threshold is set according to the design variance of the arrival rate of objects in the line (production line), such as by setting the threshold to accommodate two standard deviations in the rate of the line.

[0105] In response to determining that the threshold has not been exceeded, the controller 24 returns to normal or nominal acquisition quality in operation 366 and returns to checking the current buffer occupancy of the buffer in operation 362 .

[0106] In response to determining that the threshold has been exceeded, in operation 368, the controller 24 determines a new quality level for acquiring and / or processing scans of the object 10 in order to maintain throughput. The new quality level may be determined based on factors including current buffer occupancy, a current arrival rate of objects, and an estimated future arrival rate of objects (e.g., estimated based on arrival rates measured at the upstream camera group 130). For example, reducing the resolution of the acquired images reduces the amount of space consumed by each scan in the buffer (thereby allowing more scans of the object to be buffered), and also reduces processing time due to less data that needs to be processed to generate a partial point cloud.

[0107] In some embodiments of the present invention, the selection of a particular manner to reduce the quality level of the scan is controlled by the specific needs of the current application. For example, a configuration setting (e.g., a configuration file stored in memory or loaded from a configuration file) can specify a mapping from input parameters including the current object arrival rate and the current buffer occupancy level to a set of configuration parameters for the camera group. As described above, the configuration parameters control the quality of the scans acquired by the camera group 130, and can include, for example, the resolution of the 2-D images acquired by the individual 2-D cameras of at least some of the depth cameras 100 of the camera group 130, the frame rate (e.g., the number of image frames acquired by each depth camera 100 when the object is within the field of view of the camera group 130), the number of points and / or the density of points in the partial point cloud, whether to generate a mesh model from the point cloud, etc. When configuring the system based on the quality requirements or constraints set by the specific application of the scanning system (e.g., in a defect detection system), the specific characteristics of the quality reduction can be specified, and the reduced quality may be more acceptable for low-cost and easily replaceable object parts than for high-cost or difficult to replace parts, and thus the quality of the scans for low-cost and easily replaceable components can be selectively reduced.

[0108] In operation 370 , the controller 24 reconfigures the camera group 130 based on the configuration settings of the determined quality level (eg, setting the acquisition resolution or frame rate of each depth camera 100 ), and the process returns to operation 362 to continue monitoring the buffer occupancy.

[0109] In some embodiments of the invention, in some cases, the controller 24 addresses or mitigates high buffer occupancy conditions by decimating scans already in the buffer (e.g., reducing the amount of data, such as removing points from a portion of a point cloud or removing vertices from a mesh model). For example, in some embodiments, the controller reduces the resolution of data corresponding to a previously acquired scan stored in the buffer, and in some embodiments, lossless or lossy compression may be applied to the data stored in the buffer to reduce the size.

[0110] In some cases, given the expected rate at which objects 10 arrive at the camera array along the conveyor, the network connection will not provide sufficient bandwidth to send all the data at a high enough rate. (For example, in some manufacturing environments, an object may pass the camera array every 4 to 5 seconds. In other cases, an object may pass the camera array once a minute on average.)

[0111] In some embodiments of the present invention, the multiple cameras 100 of the camera group 130 are synchronized with respect to a clock common to the entire system or each camera group. In some embodiments, this synchronization is performed using an ad-hoc wired connection using F-sync or similar protocols, using wireless remote shutter technology, using broadband transmission, or using a networked synchronization protocol such as the Precision Time Protocol (PTP).

[0112] In some embodiments of the present invention, the coordination server 30 controls the communication with multiple camera groups 130a. FIG. 4A is a schematic diagram of multiple camera groups 130 in communication with the coordination server 30 according to one embodiment of the present invention. As shown in FIG. 4A, the first camera group 130abc ​​can capture images of a first side (e.g., the front side) of the object 10, the second camera group 130def can capture the second side (e.g., the left side of the object 10), and the third camera group 130ijk can capture the third side (e.g., the right side) of the object 10. In various other embodiments, additional camera groups 130 can be used to capture additional sides of the object (e.g., the back and top sides of the object).

[0113] In some embodiments of the invention, data acquired by separate camera groups 130 is combined at coordination server 30 to generate a 3-D model (e.g., a point cloud or mesh model) of a larger portion of an object than data acquired by any one of camera groups 130. In some embodiments, a complete ("global") 3-D model of the entire object is synthesized at coordination server 30. For example, each camera group 130 may send to coordination server 30: a 3-D mesh model; a calculated point cloud; and / or an originally acquired 2-D image of a portion of object 10. Based on the type or types of data received from camera groups 130, coordination server 30 may combine a 3-D mesh model or 3-D point cloud (e.g., a partial point cloud or a partial mesh model) created from separate partial 3-D models. In one embodiment, a “chunk assembly” process is used to combine separate partial 3-D models, which includes obtaining a rough alignment or registration of the partial 3-D models based on the known poses of the cameras 100 of the different camera groups 130 relative to the object 10 (as determined by an initial calibration process, see, for example, FIG. 1B , where, due to the arrangement of the camera groups, camera group 130ab is known to image the +x and +z sides of the object 10, camera group 130cd is known to image the -x and +z / -x surfaces of the object 10, and camera groups 130ef and 130gh image the +y and -y surfaces of the object), and refining the alignment or registration using techniques such as iterative closest points). For example, other techniques for aligning or registering separate blocks or partial 3-D models representing portions of a single object are described in U.S. patent application Ser. No. 15 / 630,715, “SYSTEM AND METHODS FOR SCANNING THREE-DIMENSIONAL OBJECTS,” filed with the U.S. Patent and Trademark Office on June 22, 2017, U.S. patent application publication No. 2017 / 0372527, the entire disclosure of which is incorporated herein by reference.

[0114] The coordination server 30 may also provide a user interface 31 that may be directly connected to the server or may be provided to a remote user terminal. For example, the user interface 31 may include a display panel and user input controls for displaying information to a user, or the user interface 31 may provide a web server for a remote device such as a laptop or mobile phone or tablet computer to connect to the web server to retrieve, display, and in some embodiments, modify or enhance information on the remote device. In some embodiments, the user interface 31 displays a 3-D model or a 3-D model generated from a point cloud acquired by the camera group 130. The user interface 31 may also report defects to the user by highlighting defects detected in the object by a defect detector (e.g., operation 350).

[0115] Aspects of embodiments of the present invention relate to systems and methods for coordinating the collection of data about objects on a production line, combining corresponding data sets collected by camera groups to synthesize a 3-D model. FIG. 4B is a schematic diagram of the correlation of data collected by multiple camera groups in communication with a coordination server according to one embodiment of the present invention. For example, if handbags 10A, 10B, 10C, 10D, and 10E move along a production line, the image of handbag 10A collected by the first camera group 130abc ​​should be combined with the image of handbag 10A collected by the second group 130def. Any images or other data related to handbag 10B should not be combined with images or other data of, for example, handbag 10A or handbag 10C.

[0116] In order to match blocks acquired by different camera groups 130 for the same object 10, in some embodiments of the present invention, it is assumed that the production line operates in a first-in-first-out (FIFO) manner, where the objects are kept in the same order between the first camera group and the last camera group. For example, if objects A, B, and C pass through the first camera group 130abc, it is assumed that the objects will pass through the second camera group 130def and the third camera group 130ijk in the same order as A, B, and C. Therefore, the image acquired by the first camera group 130abc ​​(e.g., object A) can be stored (e.g., buffered) in the system (e.g., in the controller 24 of the first camera group or at the coordination server 30) until additional data about the same object is also received from all other related camera groups (e.g., other camera groups directed to this particular production line) so that all data can be combined.

[0117] Referring to FIG. 4B , in one embodiment of the present invention, the coordination server 30 includes buffers 32abc, 32def, and 32ijk corresponding to the respective camera groups 130abc, 130def, and 130ijk. In the state shown in FIG. 4B , objects 10a, 10B, 10C, 10D, and 10E move along a production line represented by a block arrow pointing to the right, and camera group 130abc ​​is in the process of scanning object 10E. Objects 10A, 10B, 10C, and 10D previously scanned by camera group 130abc, and therefore data corresponding to those objects are stored in four corresponding locations in buffer 32abc (with one empty slot in buffer 32abc). Object 10D is between camera groups 130abc ​​and 130def and is not currently being scanned. Camera group 130def is scanning object 10C as well as previously scanned objects 10A and 10B, and therefore corresponding data of objects 10A and 10B acquired by camera group 130def is stored in corresponding buffer 32def of coordination server 30. Object 10B is between camera groups 130def and 130ijk and is not scanned. Object 10A leaves the field of view of camera group 130ijk, which has completed its scan and stores data corresponding to object 10A in corresponding buffer 32ijk.

[0118] Due to the potentially high latency between the time an object is scanned by the first camera group 130abc ​​and the time an object is scanned by the last camera group (e.g., the third camera group 130ijk), and due to the number of objects that may be scanned during this period, the controller 24 and / or the coordination server 30 may require considerable storage space to hold data awaiting combination with later scans. The specific amount of storage space required may depend on, for example, the image resolution, the point cloud resolution, the throughput of the manufacturing line, the number of scans that need to be stored in the time between the first scan of an object and the last scan of the same object, etc. In some cases, a burst of a large number of objects upstream in a row or upstream of a stop or slowdown in a row may cause data to accumulate in a buffer corresponding to an upstream camera group. Therefore, some embodiments of the present invention are directed to reducing the size of the data (e.g., by lossless or lossy compression, data decimation, etc.) to avoid overflow of the buffer.

[0119] In the example shown in FIG. 4B , the coordination server 30 aggregates data from only three different camera groups 130abc, 130def, and 130ijk. Because data from all three camera groups regarding object 10a is now buffered, further processing of the group data can proceed (as indicated by the thick black rectangle 34 around the portion of the three buffers storing data associated with object 10a) by the coordination server's group data processor 36. In some embodiments of the present invention, the group data processor 36 generates a combined 3-D model (e.g., a 3-D mesh model or a 3-D point cloud). In some embodiments, the group data processor 36 generates a cumulative report of defects detected in the three separate partial 3-D models acquired by the separate camera groups 130abc, 130def, and 130ijk, and the report can be used to control various aspects of the production line (e.g., controlling actuators to divert defective objects out of the production flow and updating production tracking records to note the defective objects) and / or the report can be displayed to a user via a user interface.

[0120] In some embodiments of the invention, the coordination server 30 may also perform defect detection on individual "chunks" of the object received from the camera group 130, or may also perform global defect detection across different camera groups using the group data processor 36 as described above. For example, an individual camera group may not be able to detect defects in individual images of different sides of a bag, but the assembled global 3-D model of the bag may show that different parts of the bag that were assumed to be made of the same color material are actually made of different materials (e.g., a black leather handle attached to a brown leather bag). Therefore, in some embodiments, the coordination server 30 may perform additional defect detection.

[0121] In some embodiments of the present invention, detected defects (e.g., as detected by individual camera groups 130 or as detected by a coordination server) are displayed to a user (e.g., a human manager of a production line) via a user interface 32. As described above, the user interface can appear on a display device attached to the coordination server 30, or can be on a remote device such as a mobile phone, tablet computer, laptop computer, or desktop computer. In some cases, a web browser running on the remote device is used to connect to the user interface 32 (e.g., the user interface 32 can be a web server). In some cases, an application (or "app") executed on the remote device connects to the user interface 32 (e.g., a user interface server) via an application programming interface (API) to obtain data about the object 10 scanned by the camera group 130.

[0122] In various embodiments of the present invention, defects in the object 10 scanned by the camera group 130 and detected by the defect detection algorithm (e.g., running on the controller 24 of the camera group 130 and / or on the coordination server 30) are displayed to the user. For example, in some embodiments of the present invention, a 3-D model representing the object 10 (e.g., a partial 3-D model acquired by the camera group 130 or a complete 3-D model of the entire object) may be displayed to the user, wherein the portion detected as defective is highlighted (e.g., with a colored outline or with a transparent shadow). Thus, embodiments of the present invention support quality assurance efforts in production facilities.

[0123] In some embodiments of the invention, the coordination server 30 controls the conveyor system 12 (e.g., controls an actuator in the conveyor system of a door) to divert an object 10 detected as having a defect so that the diverted object can be manually inspected to confirm the presence of a defect or to remove the defective product from the output stream. In some embodiments, the coordination server 30 has access to the state of the conveyor system 12 in order to dynamically estimate the state of the assembly line.

[0124] Diagnose camera group acquisition system problems

[0125] Some aspects of the present invention relate to the analysis of faults within the camera 100, the camera group 130, and the trigger system 28. FIG. 5 is a schematic diagram of a camera group with three triggers according to one embodiment of the present invention. As shown in FIG. 5, in some embodiments of the present invention, the camera group 130 may include a prepare trigger 28-prepare, a start trigger 28-start, and a stop trigger 28-stop. As previously discussed, the prepare trigger 28-prepare can be configured to detect when an object 10 is approaching the camera group 130ijk (e.g., before the object 10 arrives at the field of view of the camera 100), thereby allowing the camera group 130ijk to perform setup or initialization operations before the object 10 arrives. The start trigger 28-start can control the camera 100 to start collecting data, and the stop trigger 28-stop can control the camera 100 to stop collecting data. In some embodiments of the present invention, the start trigger 28-start and the stop trigger 28-stop can be used to identify when a particular object enters and leaves the field of view of the camera 100, for example, if the objects are spaced closely enough on the conveyor system 12, so that multiple objects 10 appear in a single frame. In some embodiments, only the ready trigger is a physical system and the start and stop triggers are virtual triggers whose signals are calculated as the wait time from the ready trigger signal, taking into account conveyor speed, geometric properties of the object being framed, and the physical arrangement of the camera.

[0126] The timing of the trigger signals provided by the trigger 28 provides information about the failure of various parts of the camera group 130. For example, if no data is acquired by the camera during a particular time period, but a trigger signal is received from the start trigger 28-start at the beginning of the particular time period, and a trigger signal is received from the stop trigger 28-stop at the end of the particular time period, it is possible that some aspect of the camera 100 has failed (e.g., a failure of the USB port connected to the camera, a software crash on the camera, no more storage space for images on the controller 24, etc.). On the other hand, if, for example, the start trigger 28-start is detected, but the arm trigger 28-arm is not detected, this may indicate a failure of the arm trigger 28-arm itself.

[0127] Thus, some aspects of embodiments of the present invention relate to using controller 24 to collect data regarding activation of triggers 28 of camera group 130 and a timestamped log of data collected by camera group 130. In some embodiments, controller 24 automatically detects and diagnoses faults in camera group 130 based on logging data received from triggers 28 and cameras 100. In some embodiments, the camera group automatically generates and transmits fault notifications to coordination server 30 to notify a user of the fault via user interface 31.

[0128] Accordingly, aspects of embodiments of the present invention are directed to systems and methods for three-dimensional data acquisition and processing under timing constraints.

[0129] While the invention has been described in conjunction with certain exemplary embodiments, it should be understood that the invention is not limited to the disclosed embodiments, but on the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims and their equivalents.

Claims

1. A system for obtaining a three-dimensional (3-D) model of an object, comprising: A first camera group, the first camera group comprising: a plurality of first depth cameras having overlapping fields of view within the plurality of first depth cameras, wherein each of the plurality of first depth cameras is a stereo depth camera having at least two image sensors; a first processor; a first memory storing instructions that, when executed by the first processor, cause the first processor to: controlling the first depth camera to simultaneously capture a first set of images of a first portion of a first object, computing a partial 3-D model representing a first portion of the first object based on the first set of images, and detecting defects in the first object based on the partial 3-D model representing a first portion of the first object; A second camera group, the second camera group comprising: a plurality of second depth cameras having overlapping fields of view within the plurality of second depth cameras, wherein each of the plurality of second depth cameras is a stereo depth camera having at least two image sensors, the second depth cameras being spaced apart from the first depth camera and including a field of view that does not overlap with the plurality of first depth cameras; a second processor; and a second memory storing instructions that, when executed by the second processor, cause the second processor to: controlling the second depth camera to simultaneously capture a second set of images of a second portion of the first object; calculating a partial 3-D model representing a second portion of the first object; and detecting a defect in the first object based on the partial 3-D model representing a second portion of the first object; and A coordination server comprising a third processor and a third memory storing instructions, the instructions, when executed by the third processor, causing the third processor to: receiving the partial 3-D model representing a first portion of the first object from the first depth camera; receiving the partial 3-D model representing a second portion of the first object from the second depth camera; and combining data from the partial 3-D model representing a first portion of the first object and the partial 3-D model representing a second portion of the first object; The overlapping fields of view of the first depth cameras are directed to a portion of a conveyor system configured to convey a plurality of objects, and wherein the conveyor system is configured to convey the objects one at a time into an overlapping field of view of the first depth camera; the conveyor system moves at a non-uniform speed and the objects arrive within the overlapping field of view of the first camera group at a plurality of different rates, wherein the first memory of the first camera group includes a buffer configured to store images acquired by the first depth camera, and The first memory further stores instructions which, when executed by the first processor, cause the first processor to: determining a current buffer occupancy of the buffer; Determining whether the current buffer occupancy exceeds a threshold; In response to determining that the current buffer occupancy does not exceed the threshold, setting the configuration parameters of the first camera group to a nominal acquisition quality; and In response to determining that the current buffer occupancy exceeds the threshold: determining a new quality level based on a plurality of configuration settings stored in the first memory, the current buffer occupancy, and a current rate of the plurality of different rates; and The configuration parameters of the first camera group are set to the new quality level.

2. The system according to claim 1, characterized in that: The first camera group further includes a first start trigger configured to detect arrival of the object when the object enters an overlapping field of view of the first depth camera, and The first processor is configured to control the first depth camera of the first camera group to capture an image of the object in response to receiving a trigger signal from the first start trigger.

3. The system according to claim 2, characterized in that: The first camera group also includes a first stop trigger configured to detect a departure of the object from the overlapping field of view of the first depth camera, and the first processor is configured to control the first depth camera of the first camera group to stop capturing images of the object in response to receiving a trigger signal from the first stop trigger.

4. The system according to claim 2, characterized in that: The first camera group further includes a first arming trigger configured to detect the presence of the object before the object enters the overlapping field of view of the first depth camera, and The first processor is configured to control the first depth camera of the first camera group to prepare to acquire an image of the object in response to receiving a trigger signal from the first preparation trigger.

5. The system according to claim 1, characterized in that: The conveyor system moves at a non-uniform speed, and the objects arrive within the overlapping field of view of the first camera group at a plurality of different rates, the different rates including a maximum burst rate and an associated maximum burst time, and Wherein the first memory of the first camera group includes a buffer having a size sufficient to store images of the object reaching a maximum burst rate during the associated maximum burst time, the size being a function of at least a resolution of the first depth camera and a frame rate of the first depth camera.

6. The system according to claim 1, characterized in that: The third memory further stores: a first buffer configured to store data from the first camera set; a second buffer configured to store data from the second camera set; as well as instructions which, when executed by the third processor, cause the third processor to: detecting when both the first buffer and the second buffer store data corresponding to the first object; as well as Data from the first camera group representing a first portion of the first object is combined with data from the second camera group representing a second portion of the first object.

7. The system according to claim 6, characterized in that: The instructions for combining data from the first camera group representing a first portion of the first object with data from the second camera group representing a second portion of the first object include merging the partial 3-D model representing the first portion of the first object with the partial 3-D model representing the second portion of the first object.

8. The system according to claim 1, characterized in that: The 3-D model is a point cloud.

9. The system according to claim 1, characterized in that: The 3-D model is a mesh model.

10. The system according to claim 1, characterized in that: Each of the first depth camera and the second depth camera comprises: a first invisible light two-dimensional camera having a first optical axis and a first field of view; a second invisible light two-dimensional camera having a second optical axis substantially parallel to the first optical axis of the first invisible light two-dimensional camera and having a second field of view overlapping the first field of view of the first invisible light two-dimensional camera; a color two-dimensional camera having a third optical axis substantially parallel to the first optical axis of the first invisible light two-dimensional camera and having a third field of view overlapping the first field of view of the first invisible light two-dimensional camera; and A projection source is configured to emit invisible light in a portion of the electromagnetic spectrum detectable by the first invisible light two-dimensional camera and the second invisible light two-dimensional camera.

11. A method for obtaining a three-dimensional (3-D) model of an object, characterized in that: The method includes: controlling, by a processor, a first camera group to simultaneously capture a first set of images of a first portion of a first object, the first camera group comprising a plurality of first depth cameras having overlapping fields of view within the plurality of first depth cameras, wherein each of the plurality of first depth cameras is a stereo depth camera having at least two image sensors; calculating, by a processor, a partial 3-D model representing a first portion of the first object based on the first set of images; detecting defects in the first object based on the partial 3-D model representing a first portion of the first object; controlling, by the processor, a second camera group to simultaneously capture a second set of images of a second portion of the first object, the second camera group comprising a plurality of second depth cameras, the plurality of second depth cameras being spaced apart from the first depth camera and comprising a field of view that does not overlap with the plurality of first depth cameras, wherein each of the plurality of second depth cameras is a stereo depth camera having at least two image sensors; calculating a partial 3-D model representing a second portion of the first object; detecting a defect in the first object based on the partial 3-D model representing a second portion of the first object; receiving, at a coordination server, from the first camera group, the partial 3-D model representing a first portion of the first object; receiving the partial 3-D model representing a second portion of the first object from the second camera group; and combining data from the partial 3-D model representing a first portion of the first object and the partial 3-D model representing a second portion of the first object; The overlapping fields of view of the first depth cameras are directed to a portion of a conveyor system configured to convey a plurality of objects, and The conveyor system is configured to convey the objects one at a time into an overlapping field of view of a first depth camera; the conveyor system moves at a non-uniform speed and the objects arrive within the overlapping field of view of the first camera group at a plurality of different rates, wherein the first camera group includes a memory, the memory including a buffer configured to store images acquired by the first depth camera, and The method further comprises: determining a current buffer occupancy of the buffer; Determining whether the current buffer occupancy exceeds a threshold; In response to determining that the current buffer occupancy does not exceed the threshold, setting the configuration parameters of the first camera group to a nominal acquisition quality; and In response to determining that the current buffer occupancy exceeds the threshold: determining a new quality level based on a plurality of configuration settings stored in the memory, the current buffer occupancy, and a current rate of the plurality of different rates; and The configuration parameters of the first camera group are set to the new quality level.

12. The method according to claim 11, characterized in that: The first camera group further includes a first start trigger configured to detect arrival of an object when the object enters an overlapping field of view of the first depth camera, and The method further includes controlling a first depth camera of the first camera group to capture an image of the object in response to a trigger signal received from the first start trigger.

13. The method according to claim 12, characterized in that: The first camera group further includes a first stop trigger configured to detect departure of the object from the overlapping field of view of the first depth camera, and The method further includes controlling a first depth camera of the first camera group to stop acquiring an image of the object in response to a trigger signal received from the first stop trigger.

14. The method according to claim 12, characterized in that: The first camera group further includes a first arming trigger configured to detect the presence of the object before the object enters the overlapping field of view of the first depth camera, and The method further comprises controlling a first depth camera of the first camera group to prepare to acquire an image of the object in response to a trigger signal received from the first preparation trigger.

15. The method according to claim 11, characterized in that: The conveyor system moves at a non-uniform speed, and the objects arrive within the overlapping field of view of the first camera group at a plurality of different rates, the different rates including a maximum burst rate and an associated maximum burst time, and Wherein the first camera group includes a memory, the memory including a buffer having a size sufficient to store images of the object reaching a maximum burst rate during the associated maximum burst time, the size being a function of at least a resolution of the first depth camera and a frame rate of the first depth camera.

16. The method according to claim 11, characterized in that: The coordination server comprises: a first buffer configured to store data from the first camera set; and a second buffer configured to store data from the second camera set, Wherein the method further comprises: detecting, by the coordination server, when both the first buffer and the second buffer store data corresponding to the first object; and Data representing a first portion of the first object from the first camera group is combined with data representing a second portion of the first object from the second camera group, by the coordination server.

17. The method according to claim 16, characterized in that: Combining data from the first camera group representing a first portion of the first object with data from the second camera group representing a second portion of the first object includes merging the partial 3-D model representing the first portion of the first object with the partial 3-D model representing the second portion of the first object.

18. The method according to claim 11, characterized in that: The 3-D model is a point cloud.

19. The method according to claim 11, characterized in that: The 3-D model is a mesh model.

20. The method according to claim 11, characterized in that: Each of the first depth camera and the second depth camera comprises: a first invisible light two-dimensional camera having a first optical axis and a first field of view; a second invisible light two-dimensional camera having a second optical axis substantially parallel to the first optical axis of the first invisible light two-dimensional camera and having a second field of view overlapping the first field of view of the first invisible light two-dimensional camera; a color two-dimensional camera having a third optical axis substantially parallel to the first optical axis of the first invisible light two-dimensional camera and having a third field of view overlapping the first field of view of the first invisible light two-dimensional camera; and A projection source is configured to emit invisible light in a portion of the electromagnetic spectrum detectable by the first invisible light two-dimensional camera and the second invisible light two-dimensional camera.