Method and apparatus for using range data to predict object characteristics

Abstract

A method, an apparatus, and a storage medium for predicting height information of an object before the object is completely within the field of view of a distance sensing device. [Solution] A method for determining a distance measurement model from distance sensing measurements includes determining first distance data at a first time when an object is at a first position that is only partially within the field of view, determining second distance data at a subsequent second time when the object is at a second, different position that is only partially within the field of view, determining a distance measurement model that models physical parameters of the object in the field of view based on the first distance data and the second distance data, determining third distance data based on the first distance data, the second distance data and the distance measurement model that represents an estimated distance to the object before the object is completely within the field of view of the distance sensing device, and determining data representing the height of the object based on the third distance data.

Description

[Technical Field]

[0001] The technology described herein generally relates to methods and apparatus for predicting object characteristics, including predicting object height, using range data. [Background technology]

[0002] Vision systems can be used in a wide range of applications to perform tasks such as measuring objects, inspecting objects, aligning objects, and / or decoding codes (e.g., barcodes). Such systems often include an image sensor that captures images of a subject or object, and one or more processors (e.g., on-board and / or interconnected with the image sensor) that process the captured images. Image information can be provided as an array of image pixels, each having different colors and / or intensities. Vision systems can be configured to generate desired outputs based on the processed images. For example, in the case of a barcode reader, an imaging device can capture an image of an object, which may include one or more barcodes. The system then processes the image to identify the barcodes, which the system can decode using a barcode decoding process. Summary of the Invention

[0003] The technology described herein, in some embodiments, relates to using a distance-sensing device, such as a time-of-flight sensor, to measure range data for predicting characteristics of a moving object before such characteristics can be determined using existing technology. In some embodiments, the technology uses the range data to predict the height of the moving object and focus a lens assembly so that an imaging device (viewing the scene through the lens) can image the moving object in sufficient focus to process the object's characteristics based on the image. For example, whereas a time-of-flight sensor may only be able to accurately measure the distance to an object if the object is entirely within the time-of-flight sensor's field of view, this technology can predict the distance to an object well before the object is entirely within the time-of-flight sensor's field of view. This technology enables machine vision systems to process physical characteristics of objects (e.g., object dimensions, object area, spacing between objects, etc.), barcodes on objects, object inspection, and / or similar features.

[0004] Some embodiments relate to a computerized method, including accessing first distance data determined by a distance sensing device at a first time point, where the distance sensing device determines the first distance data using an object at a first position within a field of view of the distance sensing device, the object being only partially within the field of view at the first position. The method includes accessing second distance data determined by the distance sensing device at a second time point occurring after the first time point, where the distance sensing device determines the second distance data using an object at a second position within the field of view of the distance sensing device, the first position being different from the second position and the object being only partially within the field of view at the second position. The method includes determining a distance measurement model for the object based on the first distance data and the second distance data, where the distance measurement model is configured to model physical parameters of the object within the field of view of the distance sensing device over time. The method includes determining third distance data representing an estimated distance to the object before the object completely enters the field of view of the distance sensing device based on the first distance data, the second distance data, and the distance measurement model. The method includes determining data representing the height of the object based on the third distance data.

[0005] Some embodiments relate to an apparatus, including a processor in communication with a memory. The processor executes instructions stored in the memory such that the processor accesses first distance data determined by a distance sensing device at a first time point, where the distance sensing device determines the first distance data using an object at a first position within a field of view of the distance sensing device, the object being configured to be only partially within the field of view at the first position. The processor executes instructions stored in the memory such that the processor accesses second distance data determined by the distance sensing device at a second time point, occurring after the first time point, where the distance sensing device determines the second distance data using an object at a second position within a field of view of the distance sensing device, the first position being different from the second position and the object being configured to be only partially within the field of view at the second position. The processor executes instructions stored in the memory such that the processor determines a distance measurement model for the object based on the first distance data and the second distance data, where the distance measurement model is configured to model physical parameters of the object within the field of view of the distance sensing device over time. The processor executes instructions stored in the memory such that the processor is configured to determine third distance data representing an estimated distance to the object before the object is completely within the field of view of the distance sensing device based on the first distance data, the second distance data, and the distance measurement model. The processor executes instructions stored in the memory such that the processor is configured to determine data representing a height of the object based on the third distance data.

[0006] Some embodiments relate to at least one non-transitory computer-readable storage medium having processor-executable instructions stored thereon that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to access first distance data determined by a distance sensing device at a first time point, where the distance sensing device determined the first distance data using an object at a first position within a field of view of the distance sensing device, the object being only partially within the field of view at the first position. The non-transitory computer-readable storage medium has processor-executable instructions stored thereon that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to access second distance data determined by the distance sensing device at a second time point, occurring after the first time point, where the distance sensing device determined the second distance data using an object at a second position within a field of view of the distance sensing device, the first position being different from the second position and the object being only partially within the field of view at the second position. The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to determine a distance measurement model for the object based on the first distance data and the second distance data, where the distance measurement model is configured to model physical parameters of the object within a field of view of the distance sensing device over time. The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to determine, based on the first distance data, the second distance data, and the distance measurement model, third distance data representing an estimated distance to the object before the object is completely within the field of view of the distance sensing device.The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to determine data representing a height of the object based on the third distance data.

[0007] The foregoing has outlined, rather broadly, the features of the disclosed subject matter in order that they may be better understood in the detailed description that follows, and in order that the present contribution to the art may be better appreciated. There are, of course, additional features of the disclosed subject matter that will be described below and that will form the subject of the claims appended hereto. It is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.

[0008] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures is represented by the same reference numeral. For clarity, not every component is shown in each drawing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an exemplary machine vision system application in which an object moves on a conveyor belt toward the field of view of an image sensing device, according to some embodiments of the technology described herein. [Figure 2] 2 illustrates the object of FIG. 1 further moved so that the object is partially within the field of view of the image sensing device, in accordance with some embodiments of the techniques described herein. [Figure 3] 3 illustrates the object of FIG. 1 further moved so that the object is more within the field of view of the image sensing device compared to FIG. 2, in accordance with some embodiments of the techniques described herein. [Figure 4A]FIG. 1 illustrates an example method for determining and updating a distance measurement model using range measurements, in accordance with some embodiments of the techniques described herein. [Figure 4B] FIG. 1 illustrates an example method for determining and updating a distance measurement model using distance sensing measurements, in accordance with some embodiments of the techniques described herein. [Figure 5A] FIG. 1 illustrates an example method for determining and updating a distance measurement model using distance measurements and parameter estimates, in accordance with some embodiments of the techniques described herein. [Figure 5B] FIG. 1 illustrates an exemplary detailed method for determining and updating a distance measurement model using distance measurements and parameter estimates, according to some embodiments of the techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0010] As described above, the techniques described herein can be used to predict object features in various machine vision applications. This technique can be used in logistics machine vision applications, including, for example, conveyor-based applications, sorting applications, and / or the like. Some logistics applications use fixed-focus lenses with a large depth of field and a small aperture. As a result of the small aperture, only limited light can pass through the lens, potentially resulting in an under-illuminated image. Alternatively, autofocus technology can be used instead of a fixed-focus lens. Autofocus technology allows the use of lenses with a shallow depth of field, thereby capturing more light and thereby improving image quality. Fast-focus lenses, such as liquid lenses, are available that can change focus in 1 to 3 milliseconds. Distance-sensing devices, such as time-of-flight sensors, can be used to detect the height of an object, which can then drive the autofocus of the lens. For example, time-of-flight sensors can operate at speeds of 3 to 5 kHz.

[0011] While autofocus technology can improve image quality, the inventors have discovered and understood that such technology may not be able to focus quickly enough to allow the imaging device to capture a sufficient number of high-quality images of an object. For example, high-speed logistics applications can use conveyor belts that move objects at very high speeds, such as 600 to 800 feet per minute (or 10 to 13.5 feet per second). Thus, an object may be within the field of view of the imaging device for less than one second, and in some cases, less than 100 to 500 milliseconds. Image sensors typically operate at 50 to 60 Hz, which is approximately 15 to 20 milliseconds for each image. In high-speed logistics applications, the imaging device may only be able to capture a few useful images of an object within the field of view of the imaging sensor (e.g., one to two images of the object's complete barcode). This problem can be exacerbated for tall objects that are closer to the image sensor, which can further limit the opportunity to capture sufficient images of the object (e.g., the field of view is more limited for tall objects, so the object remains in view for a shorter period of time than a lower object). The system may be further limited by the distance sensing device. For example, while time-of-flight sensors can operate at high speeds (e.g., a time-of-flight sensor can take 10-20 time-of-flight readings between each image capture of a 50-60 Hz imaging device), they can only provide accurate height measurements if the entire object is within the field of view of the time-of-flight sensor. Thus, in many machine vision systems, such as systems where the time-of-flight sensor is located adjacent to or near the image sensor, the time-of-flight sensor typically does not provide height measurements that are fast enough to drive autofocus applications.

[0012] The present inventors have therefore recognized and appreciated that it would be desirable to provide high-speed object height detection to address these and other problems in existing machine vision systems. For example, high-speed object height detection can be used to focus a lens assembly before and / or the moment an object (or a relevant portion of an object) moves into the field of view of an imaging device, so that the imaging device has maximum time to capture an image of the object when it is in the field of view of the imaging device. The present inventors have developed a technique for predicting the height of a moving object before the entire object is within the field of view of a distance sensing device. Such a technique allows a machine vision system to include both an imaging device and a distance sensing device in a single package, yet still enable the machine vision system to autofocus the lens assembly faster than was possible with previous distance sensing devices. For example, if a distance measuring device captures N images of an object as it enters the field of view of the distance measuring device before the object is completely within the field of view, the technique of the present invention can estimate object parameters 50% faster (e.g., 1 / 2 × N fewer captures). Such a technique can additionally or alternatively provide greater accuracy than would otherwise be achievable using incomplete data. This technique can, for example, perform autofocus with a small number of noisy distance measurements (e.g., a small number of noisy distance measurements taken before the object is fully within the system's field of view), which allows autofocus to be performed at much faster conveyor speeds than was previously possible.

[0013] In the following description, numerous specific details are set forth regarding the systems and methods of the disclosed subject matter, as well as the environments in which such systems and methods may operate, in order to provide a thorough understanding of the disclosed subject matter. In addition, it will be understood that the examples provided below are exemplary, and that other systems and methods are contemplated to exist that are within the scope of the disclosed subject matter.

[0014] FIG. 1 is a diagram 100 illustrating an exemplary machine vision system application in which an object 102 moves on a conveyor belt 104 toward a field of view S106 of an image sensing device 108, in accordance with some embodiments of the technology described herein. In this example, the object 102 has a height H110, and the object 102 is moving on the conveyor belt 104 in a direction 112. The image sensing device 108 is at a distance D114 from the conveyor belt 104. FIG. 2 is a diagram illustrating the object 102 of FIG. 1 moving further along the direction 112 so that the object 102 is partially within the field of view S106 of the image sensing device 108, in accordance with some embodiments of the technology described herein. As shown in FIG. 2, a portion Sk1 200 of the top surface of the object 102 is within the field of view S106. FIG. 3 is a diagram illustrating the object 102 of FIG. 1 moving further along the direction 112 so that the object 102 is further within the field of view S106 of the image sensing device 108 compared to FIG. 1, in accordance with some embodiments of the technology described herein. 3, a portion Sk2 300 of the top surface of object 102 is within field of view S106. Portion Sk2 300 is larger than portion Sk1 200 in FIG.

[0015] 1-3 show a single object 102, it should be understood that many objects may move on the conveyor belt, and the objects may not be uniform in shape, spacing, and / or type. Thus, in many vision system applications, such as code decoding in logistics operations (e.g., tracking barcodes on packages as they move along a conveyor line), the number, height, length, overall dimensions, and spacing between objects may vary widely.

[0016] In some embodiments, the distance sensing device is a time-of-flight sensor, such as an integrated single-point or multi-point time-of-flight sensor that predicts distance information. The time-of-flight sensor may include an emitter configured to emit a beam, such as a laser beam or a light pulse (e.g., an infrared light pulse), and a receiver configured to receive the reflected beam. The time-of-flight sensor may modulate the intensity of the beam at a high frequency so that there is a phase shift between the emitted beam and the reflected beam. The time-of-flight sensor may include circuitry that measures the degree of phase shift by comparing the phase at the emitter and the phase at the receiver.

[0017] Although not shown, the distance sensing device 108 can be located near the imaging device, e.g., integrated into a single package with the imaging device and / or mounted near the imaging device (e.g., attached to the lens assembly of the imaging device). An exemplary configuration may include an imaging device integrated with a time-of-flight sensor and / or an illumination device, such as a high-power integrated illuminator (HPIT) offered by Cognex Corporation, the assignee of the present application. Another exemplary configuration may include an imaging device and a time-of-flight sensor mounted at a different location from the imaging device. In such an example, the imaging device and time-of-flight sensor can be mounted at the same height, with one or more configuration constraints, such as an optical axis perpendicular to the conveyor, with the time-of-flight sensor mounted upstream of the conveyor belt from the imaging device, allowing the time-of-flight sensor to perform measurements before an object reaches the imaging device and / or other system configuration constraints. In some embodiments, the imaging device may include one or more internal processors (e.g., FPGA, DSP, ARM processor, and / or the like) and other components that enable it to function as a stand-alone unit, thereby providing desired output data (e.g., decoded code information) to downstream processes such as inventory tracking computer systems, logistics applications, etc. In some embodiments, the machine vision system may include external processing functionality, such as a personal computer, laptop, and / or the like, configured to provide the machine vision system processing functionality.

[0018] In some embodiments, the machine vision system can be an image-based code reader. The reader can include an imaging device with an image sensor and optics, and a vision system processor arranged to find and decode codes (e.g., barcodes) in images acquired by the image sensor. A distance sensing device 108 can be integrated with the imaging device and predicts distance information as described herein (e.g., in connection with FIGS. 4A-5B) for objects 102 within the field of view of the image sensor. A time-of-flight sensor can be operatively connected to at least one of the vision system processor and / or imager controller.

[0019] In some embodiments, the imaging device includes a lens assembly and / or can be in optical communication with a lens assembly. The choice of lens configuration can depend on various factors, such as illumination / illumination, field of view, focal length, the relative angle between the camera axis and the imaging plane, and / or the size of the details of the imaging plane. In some examples, the cost of the lens and / or the space available for mounting the vision system can also determine the choice of lens. An exemplary lens configuration that may be desirable in certain vision system applications is an autofocus assembly. By way of example, an autofocus lens can be easily realized by a so-called liquid lens assembly.

[0020] The use of liquid lenses simplifies installation, setup, and maintenance of vision systems by eliminating the need to manually touch or adjust the lens. Liquid lenses can have a faster response time than other autofocus mechanisms. Liquid lenses can also be used in applications where the reading distance varies between objects (surfaces) or when switching from reading one object to another, such as scanning a moving conveyor carrying objects of different dimensions / heights (e.g., shipping boxes). While the example shown in Figures 1-3 illustrates a conveyor belt-type machine vision application, it should be understood that rapid focusing for imaging may be desirable in many different vision system applications, and thus the technology described herein is not limited to such exemplary embodiments.

[0021] Liquid lenses can be implemented in a variety of ways. One exemplary liquid lens embodiment uses water and oil, essentially using an applied voltage to shape the oil into a lens. Varying the voltage across the lens with a surrounding lens changes the curvature of the oil-water interface, which in turn changes the focal length of the lens. Another exemplary liquid lens embodiment uses a movable membrane covering a liquid reservoir to change its focal length. A bobbin can apply pressure to change the shape of the membrane, thereby changing the focus of the lens. The input current can be changed, or the bobbin can be moved within a preset range. Varying the current level can change the focal length of the liquid lens.

[0022] As described herein, range / distance information from the distance sensing device can be processed to autofocus a variable (e.g., liquid) lens during runtime operation based on objects of a particular size / shape within and in front of the field of view. The object is completely within the field of view of the time-of-flight sensor. For example, the predicted distance information can be used to set the focal length of the lens of the imaging device before the object is partially and / or completely within the field of view of the imaging device. In some embodiments, the system is configured such that the distance measurement device is faster than the imaging sensor. For example, a 50 Hz image sensor can be used with a 4 kHz time-of-flight sensor.

[0023] FIG. 4A illustrates an exemplary method 400 for determining and updating a distance measurement model using range measurements, in accordance with some embodiments of the techniques described herein. A machine vision system can perform method 400 to predict the characteristics (e.g., height, area, etc.) of an object before the object is completely within the field of view of a distance sensing device. For example, with reference to FIGS. 2 and 3 , method 400 can be used to predict the height H110 of an object 102 before the object is completely within the field of view S of the distance sensing device 108. In step 402, the machine vision system determines a characteristic measurement model, e.g., a distance measurement model, that models the distance sensing device's distance measurements to the object over time as the object enters the field of view of the distance sensing device to predict the object's height before it is completely within the field of view. As described herein, the machine vision system can determine the distance measurement model based on predetermined parameters and / or based on initial distance measurements of the object as it begins to enter the field of view.

[0024] The machine vision system continues to acquire distance data over time and updates the distance measurement model accordingly. At step 404, the machine vision system acquires and / or processes new distance measurement data. At step 406, the machine vision system updates the state 406 of the distance measurement model based on the distance measurement data acquired at step 404. The method returns to step 404 to process and / or acquire new distance measurement data. The method performs steps 404 and 406 until the distance measurement model has sufficiently converged to the distance measurement data. For example, as described further herein, the machine vision system can determine that the distance measurement model has converged to the time-of-flight measurement data when the distance measurement model and / or the estimate of the object height determined at step 406 is below a predetermined threshold.

[0025] 4B illustrates an exemplary detailed method 450 for determining and updating a distance measurement model using distance sensing measurements, in accordance with some embodiments of the techniques described herein. In step 452, the machine vision system accesses first distance data determined by a distance sensing device (e.g., a time-of-flight sensor) at a first time. The distance sensing device captured and / or determined a first distance using an object at a first position that is only partially within the field of view of the distance sensing device. For example, referring to FIG. 2, the distance sensing device 108 captured first distance data to an object 102 that is only partially within the field of view S106, as indicated by Sk1 200.

[0026] In step 454, the machine vision system accesses second distance data determined by the distance sensing device at a second time point subsequent to the first time point. The distance sensing device captured and / or determined the second distance data at a second position where the object was only partially within the field of view of the distance sensing device. Because the distance data is determined over time as the object moves, the first position associated with the first distance data differs from the second position associated with the second distance data. For example, referring to FIG. 3, the distance sensing device 108 captured the second distance data with the object 102 only partially within the field of view S106, as indicated by Sk2 300. Compared to FIG. 2, the object 102 is only partially within the field of view S106 in both FIGS. 2 and 3, but the object 102 is more within the field of view S106 in FIG. 3 than in FIG. 2 (thus Sk2 300 is larger than Sk1 200).

[0027] As described herein, the distance sensing device can be a time-of-flight sensor that is part of a machine vision system that also includes internal and / or external processing hardware and associated software for performing machine vision tasks. Referring to steps 452 and 454, accessing the first and second range data can include processing hardware (e.g., that is part of the imaging device) receiving the first and second time-of-flight measurements from the time-of-flight sensor. As another example, the processing hardware can access the time-of-flight measurements from a memory shared with the time-of-flight sensor.

[0028] In step 456, the machine vision system determines a distance measurement model of the object based on the first distance data and the second distance data. The distance measurement model is configured to model physical parameters of the object in the field of view of the distance sensing device over time, such as the object's height, the object's surface area, and / or the like. In step 458, the machine vision system determines distance data representing an estimated distance to the object before the object was completely within the field of view of the distance sensing device based on the previous distance data (e.g., the first distance data and the second distance data for the first run) and the distance measurement model.

[0029] In step 460, the machine vision system determines whether the distance measurement model converges to the distance measurement data. In some embodiments, the machine vision system can determine that the change in the distance measurement model is below a predetermined threshold. In some embodiments, the machine vision system can determine that the change between the distance measurement data and the estimated object height (e.g., the data determined in step 458) is less than a predetermined threshold. For example, as further described in connection with Equations 1 and 2, the object height can be modeled as a parameter that is part of an observation matrix, and the observation matrix can be updated with each iteration. If, after a predetermined number of iterations, the predicted height is close to the observation (e.g., within a threshold), the system can determine that the model used in the current iteration is sufficient and the observation matrix can be used to determine the object height (e.g., the distance measurement model is stable in this state). If the machine vision system determines that the distance measurement model has not converged to the distance measurement data, method 450 proceeds to step 462, where the distance measurement model is updated with one or more additional distance measurements, and returns to step 458.

[0030] If the machine vision system determines that the distance measurement model converges to the distance measurement data, method 450 proceeds to step 464, where the machine vision system determines data representing the height of the object based on the distance data determined in step 458. For example, the machine vision system may use the last determined data from step 458 as representing the height of the object.

[0031] As described herein, a machine vision system can use data representing the height of an object to perform various machine vision system tasks. For example, the machine vision system can determine data representing a focus adjustment for a lens of an imaging device associated with the distance sensing device based on the distance data. The machine vision system can transmit one or more signals to change the focus of the lens based on the data representing the focus adjustment. In some embodiments, the machine vision system uses the focus adjustment to change the focus of the liquid lens. The machine vision system can capture an image of the object after transmitting one or more signals to change the focus of the lens. In some embodiments, the machine vision system can be configured to wait a predetermined time after transmitting one or more signals to change the focus of the lens before capturing an image of the object. In some embodiments, the system can receive feedback data from the liquid lens assembly indicating that the focus adjustment is complete, and the machine vision system can capture an image in response to receiving the feedback data.

[0032] In some embodiments, the machine vision system can determine data representing a brightness adjustment for the lighting module of the imaging device based on the estimated distance data. For example, if the distance data indicates a low object, the machine vision system can be configured to use a higher brightness setting for the lighting module compared to when the distance data indicates a high object. Thus, the machine vision system can be configured to use brighter lighting for objects farther from the imaging device and softer lighting for objects closer to the imaging device. In some embodiments, the techniques of the present invention can adjust the brightness of an image by adjusting the exposure time of the imaging device (e.g., without adjusting the lighting module) to adjust the lighting for the object. For example, the system can reduce the exposure time for objects closer to the camera.

[0033] In some embodiments, the machine vision system may have access to data representing at least one constraint of the machine vision application. For example, the machine vision system may have access to data representing object motion parameters associated with the movement of a box through the field of view of the time-of-flight sensor. The machine vision system may determine distance data using the data representing the motion parameters. For example, with reference to FIG. 4B , in step 458, the machine vision system may determine the distance data based on the first distance data, the second distance data, the distance measurement model, and one or more motion parameters. In some embodiments, the motion parameters include a velocity of the object and / or an acceleration of the object as the object moves through the field of view of the distance sensing device. For example, the machine vision system may determine the distance data based on the first distance data, the second distance data, the distance measurement model, and the velocity of the object.

[0034] In some embodiments, the machine vision system can use a Kalman filter to model the distance to an object as it enters the field of view of the distance sensing device. For example, the machine vision system can use the Kalman filter to estimate the area of ​​the object over time. Referring to step 458 of FIG. 4B , for example, the machine vision system can use a distance measurement model to determine an estimate of the area of ​​the object based on the acquired distance data (e.g., acquired in steps 452, 454, and / or 462). For example, the machine vision system can determine a first object area estimate for an object (that is only partially within the field of view of the distance sensing device) based on the first distance data. The machine vision system can use the Kalman filter to determine subsequent object area estimates. For example, the machine vision system can use the distance measurement model to determine a second object area estimate for an object (that is still within the field of view of the distance sensing device) based on the first object area estimate. The machine vision system can determine a height estimate for the object based on the second object area estimate.

[0035] The machine vision system may be configured to use one or more equations to measure a state (e.g., the area of ​​an object in the field of view) and / or to perform updates to the measurements of a distance sensing device (e.g., to update the estimated distance to an object). The following example equation 1 may be used to perform state updates: S K =S K-1 +S Δ +w k (Formula 1) where: S k is the area of ​​the top surface of the object in the field of view of the distance sensing device at the current time instant k, S k-1 is the previous time point k-1 is the area of ​​the top surface of the object in the field of view of the distance sensing device at S Δ is the difference between SK and S K-1 is the difference between w k is a model for system noise, random fluctuations, and / or measurement noise / inaccuracy.

[0036] As shown in Equation 1, the state can be the area of ​​the object within the field of view of the distance sensing device. The state can be modeled based on parameters of the machine vision system. For example, if the object is moving at a near-constant velocity (such as on a conveyor belt) and the frame rate of the time-of-flight sensor is high enough, the state update can be modeled as a linear function where the object area is constant per unit time. In some embodiments, the first two measurements of the object area can be initial values ​​determined based on the machine vision application specifications of box size range, conveyor belt width, object movement speed, and / or the like. It should be understood that the initial measurements of the object area do not need to be exact. For example, w k can be used to model noise from the physical world, such as the non-constant speed of a conveyor belt. k (and for example v k ) can be modeled in terms of the system noise and the observation covariance matrix. k can be assigned an initial value based on the application (e.g., based on expected changes in conveyor speed, accuracy of TOF readings, etc.). The initial value will not necessarily be accurate, but can be updated with each iteration using a Kalman filter to improve accuracy. From the third state (and beyond), the area and / or height of the object can be obtained and determined from the model using real-time distance measurements, and the model can be updated with each new distance measurement.

[0037] Equation 2 shows an example equation for performing distance measurement. TOF k =-(h / S)S k +d+v k (Formula 2) where: TOF kis the time-of-flight measurement at time k, h is the height of the object, S is the field of view of the time-of-flight sensor, S k is the area of ​​the top surface of the object in the field of view of the distance sensing device at the current time instant k, d is the distance between the time-of-flight sensor and the conveyor belt, v k is a model for observation noise in distance measurements (e.g., noise due to inaccuracies in the TOF readings).

[0038] Referring to Equation 2, the time-of-flight sensor reading can be determined based on a weighted average of the object's height h and distance d.

[0039] It should be understood that Equations 1 and 2 are provided for illustrative purposes only, and other models can be used for particular machine vision applications. Other formats of Kalman filter models can be used, for example, depending on whether the object's movement is governed by acceleration, whether the object's area changes, and / or other variables. The time-of-flight sensor measurements can gradually decrease over time (e.g., from measuring the distance to the conveyor belt to measuring the distance to the top of the object) as the object enters the time-of-flight sensor's field of view. The estimated box area determined by the model can correspondingly gradually increase over time inversely proportional to the time-of-flight measurements, such that the estimated box area increases over time.

[0040] In some embodiments, the machine vision system can update the parameters of the model using parameter estimation techniques (e.g., expectation maximization (EM), maximum a posteriori (MAP), etc.). For example, as shown in exemplary Equation 2, the object height can be part of the observation matrix. At each iteration, the box height can be updated using a parameter estimation algorithm. FIG. 5A illustrates an exemplary method 500 for determining and updating a distance measurement model using distance measurements and parameter estimation, in accordance with some embodiments of the techniques described herein. At step 502, the machine vision system determines a distance measurement model using the techniques described herein. Each time the system obtains a new distance observation at step 504, the system updates the state of the distance measurement model using a model (e.g., a Kalman filter) at step 506, and the machine vision system updates the system observation matrix using the parameter estimation at step 508.

[0041] 5B illustrates an exemplary detailed method 550 for determining and updating a distance measurement model using distance measurements and parameter estimation, in accordance with some embodiments of the techniques described herein. At step 552, the machine vision system accesses first distance data determined at a first time point using an object at a first position that is only partially within the field of view of the distance sensing device (e.g., as shown in FIG. 2). At step 554, the machine vision system accesses second distance data determined at a subsequent second time point using an object at a second position that is still only partially within the field of view of the distance sensing device (e.g., as shown in FIG. 3). At step 556, the machine vision system determines a distance measurement model of the object based on the first distance data and the second distance data.

[0042] In step 558, the machine vision system determines distance data representing an estimated distance to the object before the object is completely within the field of view of the distance sensing device based on previous distance data (e.g., the first distance data and the second distance data for the first run) and the distance measurement model. In step 560, the machine vision system determines whether the distance measurement model converges to the distance measurement data. If the machine vision system determines that the distance measurement model does not converge to the distance measurement data, method 550 proceeds to step 562, where it updates the parameters using parameter estimation (e.g., using EM, MAP, and / or another parameter estimation technique). The method proceeds to step 564, where it updates the distance measurement model with one or more additional distance measurements, and returns to step 558.

[0043] If the machine vision system determines that the distance measurement model converges to the distance measurement data, method 550 proceeds from step 560 to step 564, where the machine vision system determines data representing the height of the object based on the distance data determined in step 558. For example, the machine vision system may use the last determined data from step 558 as representing the height of the object.

[0044] As described herein, the time-of-flight sensor measurements can gradually decrease (e.g., from measuring the distance to the conveyor belt to measuring the distance to the top of the object) as the object comes within the field of view of the time-of-flight sensor. The estimated box area determined using the distance measurement model can correspondingly gradually increase over time inversely proportional to the time-of-flight measurements, such that the estimated box area increases over time. In some embodiments, parameter estimation can be used to fine-tune the relationship between the time-of-flight measurements and the box area measurements.

[0045] The predicted distance information generated by the distance sensing device can also be used to perform other aspects of the machine vision system. In some embodiments, the machine vision system can use the predicted distance information to automatically trigger the vision system to acquire an image of an object. For example, the system can trigger image capture when the system determines a change in the object's distance (e.g., a change in height from a supporting base / conveyor belt). In some embodiments, the machine vision system can use the predicted distance information to perform dimensioning of an object (e.g., a box). During calibration, the machine vision system can measure the distance D114 between the conveyor belt and the vision system and store the information. At runtime, the machine vision system can capture an image before the object is fully within the image to predict the distance to the object and / or measure the distance to the object in the center of the image / field of view. Once the system detects a rectangular shape, it can determine the dimensions of the rectangle based on the predicted and / or measured distance (and known optical properties of the imager, such as image sensor dimensions, lens focal length, etc.). 1-3, the height 110 of the object 102 may essentially represent the difference between the height of the imaging device above the conveyor belt 104 and the shortest predicted and / or measured distance between the imaging device and the object 102. In some embodiments, the imaging device is fixedly attached to the time-of-flight sensor, so the height between the imaging device and the object 102 may be represented by a distance D114.

[0046] In some embodiments, a machine vision system can be used to detect and analyze defects in an object. For example, after a machine vision system detects a (e.g., rectangular) top surface as described herein, the machine vision system can measure deviations from the top surface shape to detect damaged objects (e.g., a damaged box). In some embodiments, the system can perform region of interest (RoI) detection. For example, the field of view of a camera-based code reader can be projected onto the detected area of ​​a multi-point (e.g., n x 1 or n x m) time-of-flight sensor array. The sensor array can measure a 3D height map to narrow the region of interest for decoding the code. Once it is determined in which part of the image the object is present, symbol candidate features can be searched from the narrowed region of interest in the entire acquired image, thereby reducing decoding time.

[0047] In some embodiments, the machine vision system can detect gaps between objects in the field of view, which can help associate symbol codes with the appropriate imaged object. For example, in logistics applications where more than one object may enter and / or be present in the field of view at the same time, time-of-flight measurements can identify object edges and help determine which symbol is associated with each object in the field of view.

[0048] In some embodiments, the machine vision system can use the distance predicted and / or measured by the time-of-flight sensor to limit the read range of the vision system to prevent unintended reads. For example, if the distance to the object is within a defined range, the machine vision system (a) captures an image, (b) initiates a symbol decoding process, and (c) transmits the decoded data for further processing.

[0049] Techniques operating according to the principles described herein may be implemented in any suitable manner. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided on a single computer or distributed across multiple computers. Such processors may be implemented as integrated circuits, which include commercially available integrated circuit components with one or more processors within an integrated circuit component, known by such names as CPU chips, GPU chips, FPGA chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processor may be implemented in a custom circuit, such as an ASIC, or a semi-custom circuit resulting from constructing a programmable logic device. As yet another alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, with one or a subset of those cores being capable of constituting a processor. However, the processor may be implemented using any suitable form of circuitry.

[0050] Furthermore, it should be understood that the computer may be embodied in any of several forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, the computer may be embedded in devices that are not generally considered computers but have suitable processing capabilities, including personal digital assistants (PDAs), smartphones, or other suitable portable or fixed electronic devices.

[0051] A computer may also include one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for visual presentation of output and speakers, or other sound-generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, pointing devices such as mice and touchpads, and digitizing tablets. As another example, a computer can receive input information via voice recognition or other audible formats. In the illustrated embodiment, the input / output devices are illustrated as being physically separate from the computing device. However, in some embodiments, the input and / or output devices may be physically integrated into the same unit as the processor or other elements of the computing device. For example, a keyboard may be implemented as a soft keyboard on a touchscreen. Alternatively, the input / output devices may be completely separate from the computing device and functionally integrated via a wireless connection.

[0052] Such computers may be interconnected by one or more networks of any suitable form, including a local area network, or a wide area network such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0053] Additionally, the various methods or processes outlined herein may be coded as software executable on one or more processors using any one of a variety of operating systems or platforms. Alternatively, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine code or intermediate code that runs on a framework or virtual machine.

[0054] In this regard, the present invention may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memory, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed by one or more computers or other processors, perform methods for implementing the various embodiments of the present invention described above. As is evident from the foregoing examples, a computer-readable storage medium may retain information in a non-transitory form for a time sufficient to provide computer-executable instructions. Such computer-readable storage medium or media may be portable, allowing one or more programs stored thereon to be loaded into one or more different computers or other processors to implement various aspects of the present application, as described above. As used herein, the term "computer-readable storage medium" encompasses only computer-readable media that can be considered to be a manufacture (i.e., an article of manufacture) or machine. Alternatively, or additionally, the present invention may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagated signal.

[0055] The terms "code," "program," or "software" are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to implement various aspects of the present application, as described above. Furthermore, in accordance with one aspect of this embodiment, one or more computer programs that, when executed, perform the methods of the present application need not reside on a single computer or processor to implement various aspects of the present application, but may be distributed in a modular manner across multiple computers or processors.

[0056] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0057] Additionally, data structures may be stored in any suitable format on a computer-readable medium. For ease of explanation, data structures may be depicted as having fields that are related by location within the data structure. Such relationships may in turn be achieved by assigning storage for the fields to locations within the computer-readable medium that convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information within fields of a data structure, including the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0058] Various aspects of the present application may be used alone, in combination, or in various configurations not specifically discussed in the above embodiments, and therefore are not limited in their application to the details and arrangements of components set forth in the above description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0059] The present invention may also be embodied as a method, of which examples are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, although shown in the exemplary embodiments as a sequential series of acts, embodiments may be configured to perform acts in an order different from that described, which may include performing some acts simultaneously.

[0060] The indefinite articles "a" and "an," as used in the specification and claims, unless expressly stated to the contrary, should be understood to mean "at least one."

[0061] The phrase "and / or," as used in the specification and claims, should be understood to mean "either or both" of the elements so conjoined, i.e., elements present conjunctively in some cases and disjunctively in other cases. Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" of the elements so conjoined. Elements other than the elements specifically identified by the "and / or" clause may optionally be present, whether related or unrelated to the elements specifically identified. Thus, as a non-limiting example, a reference to "A and / or B," when used in conjunction with open-ended language such as "comprising," can refer in one embodiment to A only (optionally including elements other than B), in another embodiment to B only (optionally including elements other than A), in yet another embodiment to both A and B (optionally including other elements), and so forth.

[0062] The phrase "at least one," when used in this specification and claims when referring to a list of one or more elements, should be understood to mean at least one element selected from one or more elements listed in the list of elements, but not necessarily including at least one individual element specifically listed in the list of elements, and not excluding combinations of elements included in the list of elements. This definition also indicates that elements other than those specifically identified in the list of elements to which the phrase "at least one" refers may optionally be present, whether related or unrelated to the elements specifically identified. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B," or, equivalently, "at least one of A and / or B") can in one embodiment refer to at least one A, optionally including two or more As, and no Bs (and optionally including elements other than B); in another embodiment refer to at least one B, optionally including two or more Bs, and no As (and optionally including elements other than A); in yet another embodiment refer to at least one A, optionally including two or more As, and at least one B, optionally including two or more Bs (and optionally including other elements); etc.

[0063] The use of ordinal numbers such as "first," "second," "third," etc. to vary claim elements in the claims does not, in itself, imply a priority, precedence, or ranking of one claim element over another, or a chronological order in which method actions are performed, but is merely used as a label to distinguish a claim element with a particular name from another element with the same name (except for the use of ordinal numbers) to distinguish the claim elements.

[0064] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "having," "having," "including," "involving," and variations thereof herein is meant to encompass the items listed thereafter, and equivalents thereof, as well as additional items.

Claims

1. 1. A system for determining settings for an imaging device having a field of view: The system includes an illumination device configured to illuminate an object; The system further includes a distance sensing device having a field of view and configured to determine a distance of the object; The system further comprises one or more processors in communication with the lighting device and the distance sensing device: The processor is configured to receive first distance data corresponding to a first time when the object is located within a first portion of the field of view of the distance sensing device; The processor is further configured to receive second distance data corresponding to a second time when the object is located within a second portion of the field of view of the distance sensing device; The processor is further configured to access a distance measurement model configured to model and predict physical parameters of the object as it moves within a field of view of the distance sensing device; the processor is further configured to determine third distance data based on the distance measurement model and the first and second distance data, wherein the third distance data indicates an estimated distance to the object at a future time when the object will be within a field of view of the imaging device; and The processor is further configured to determine and transmit a signal to change a focal length of a lens of the imaging device based on the third distance data to the object; The above system.

2. The system of claim 1 , wherein said determining and transmitting results in changing the focal length before the object comes within a field of view of the imaging device.

3. The system of claim 1 , wherein the one or more processors are further configured to direct image capture of the object after transmitting the signal to change the focal length of the lens.

4. The system of claim 1 , wherein the one or more processors are further configured to direct image capture in response to a change in the estimated distance.

5. The system of claim 1 , wherein the distance sensing device comprises a time-of-flight sensor.

6. The system of claim 1 , wherein the physical parameters of the object include an area of ​​the object, a height of the object, or a combination thereof.

7. The system of claim 1 , wherein the one or more processors are further configured to determine data representing a brightness adjustment of the lighting device based on the third distance data.

8. The system of claim 7 , wherein the brightness adjustment includes setting an exposure time based on the third distance data.

9. The system of claim 7 , wherein the brightness adjustment comprises setting the brightness of the lighting device.

10. the one or more processors are further configured to access data indicative of motion parameters of the object; and determining the third distance data based on the distance measurement model, the first and second distance data, and the motion parameters; The system of claim 1 .

11. The system of claim 10 , wherein the motion parameters include a velocity of the object as it moves through the field of view of the distance sensing device.

12. The system of claim 10 , wherein the motion parameters include an acceleration of the object as it moves through the field of view of the distance sensing device.

13. 1. A system for determining settings for an imaging device having a field of view: The system includes a distance sensing device having a field of view and configured to determine a distance of an object; The system further comprises one or more processors in communication with the distance sensing device: The processor is configured to receive first distance data corresponding to a first time when the object is located within a first portion of the field of view of the distance sensing device; The processor is further configured to receive second distance data corresponding to a second time when the object is located within a second portion of the field of view of the distance sensing device; The processor is further configured to access a distance measurement model configured to model and predict physical parameters of the object as it moves within a field of view of the distance sensing device; the processor is further configured to determine third distance data based on the distance measurement model and the first and second distance data, wherein the third distance data indicates an estimated distance to the object at a future time when the object will be within a field of view of the imaging device; and The processor is further configured to determine and transmit a signal to change a focal length of a lens of the imaging device based on the third distance data to the object; The above system.

14. The system further comprises an illumination device configured to illuminate the object; The system of claim 13 , wherein the one or more processors are further configured to determine data representing a brightness adjustment of the lighting device based on the third distance data.

15. 1. A computerized method comprising: The method includes receiving first distance data determined by a distance sensing device having a field of view, the first distance data corresponding to a first time at which an object is located within a first portion of a field of view of the distance sensing device; The method further includes receiving second distance data determined by the distance sensing device, the second distance data corresponding to a second time when an object is located within a second portion of a field of view of the distance sensing device; The method further includes accessing a distance measurement model configured to model and predict physical parameters of the object as the object moves within a field of view of the distance sensing device; The method further includes determining third distance data based on the distance measurement model and the first and second distance data, wherein the third distance data indicates an estimated distance to the object at a future time when the object will be within a field of view of an imaging device; and The method further comprises determining and transmitting a signal to change a focal length of a lens of the imaging device based on the third distance data to the object; The computerized method.

16. The method of claim 15 , wherein said determining and transmitting results in changing the focal length before the object comes within a field of view of the imaging device.

17. 16. The method of claim 15, further comprising directing image capture of the object after transmitting the signal to change the focal length of the lens.

18. The method of claim 15 , further comprising directing image capture in response to a change in the estimated distance.

19. The method of claim 15 , further comprising determining data representing a brightness adjustment of a lighting device configured to illuminate the object based on the third distance data.

20. The method further comprises accessing data indicative of motion parameters of the object; and determining the third distance data based on the distance measurement model, the first and second distance data, and the motion parameters; 16. The method of claim 15.

Images