Method and apparatus for predicting object features using distance data
The distance data is measured by the time-of-flight sensor, and an object height prediction model is established, which solves the problem of insufficient object height detection speed in high-speed logistics applications in the prior art, and achieves fast and accurate automatic focus and image acquisition.
Patent Information
- Application Number
- CN202110192802.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-02-20
- Filing Date
- 2021-02-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-02-20
AI Technical Summary
Existing machine vision systems are difficult to detect object height quickly and accurately in high-speed logistics applications, resulting in imaging devices not being able to acquire high-quality images within sufficient time, especially when objects move rapidly.
By measuring distance data using time-of-flight sensors, an object distance measurement model is established to predict the object's height before it fully enters the field of view, thereby quickly focusing the lens assembly and improving the automatic focusing speed and image acquisition efficiency of the imaging device.
It realizes rapid detection of the object before its height is fully entered into the field of view, reduces the number of focus acquisitions, improves the time and accuracy of image acquisition, and is suitable for high-speed logistics applications, allowing automatic focus to be performed at faster conveyor belt speeds.
Smart Images

Figure CN113281762B_ABST
Abstract
Description
Technical Field
[0001] The technology described herein generally relates to methods and apparatus for using range data to predict characteristics of an object, including predicting the height of an object. Background Art
[0002] Vision systems are widely used to perform tasks such as measuring objects, inspecting objects, aligning objects, and / or decoding symbol information (e.g., barcodes). Such systems typically include an image sensor that captures an image of a subject or object, and one or more processors (e.g., onboard and / or interconnected to the image sensor) that process the captured image. The image information can be provided in the form of an array of image pixels, where each image pixel has a different color and / or intensity. The vision system can be configured to generate a desired output based on the processed image. For example, for a barcode reader, an imaging device can image an object that may contain one or more barcodes. The system can then process the image to identify the barcode and decode the barcode through a barcode decoding process. Summary of the invention
[0003] In some embodiments, the techniques described herein involve using a distance sensing device (e.g., a time-of-flight sensor) to measure distance data to predict the characteristics of a moving object before the characteristics of the object can be determined using existing techniques. In some embodiments, these techniques use the distance data to predict the height of the moving object to focus the lens assembly so that the imaging device (viewing the scene through the lens) can image the moving object at a sufficient focal length so that the characteristics of the object can be processed based on the image. For example, while a time-of-flight sensor may only be able to accurately measure the distance to an object when the object is completely within the field of view of the time-of-flight sensor, these techniques can predict the distance to the object well before the object is completely within the field of view of the time-of-flight sensor. These techniques can allow machine vision systems to process physical characteristics such as objects (e.g., object size, object area, spacing between objects, etc.) and barcodes on objects, perform object inspections, etc.
[0004] Some embodiments relate to a computerized method. The method includes accessing first distance data determined by a distance sensing device at a first time, wherein the distance sensing device determines the first distance data when an object is located at a first position within a field of view of the distance sensing device, wherein the object is only partially located at the first position within the field of view. The method includes accessing second distance data determined by the distance sensing device at a second time after the first time, wherein the distance sensing device determines the second distance data when the object is located at a second position within the field of view of the distance sensing device, wherein the first position is different from the second position and the object is only partially located at the second position within the field of view. The method includes determining a distance measurement model for the object based on the first distance data and the second distance data, wherein the distance measurement model is configured to model changes in a physical parameter of the object within the field of view of the distance sensing device over time. The method includes determining third distance data based on the first distance data, the second distance data, and the distance measurement model, the third distance data indicating an estimated distance to the object before the object is completely within the field of view of the distance sensing device. The method includes determining data indicating a height of the object based on the third distance data.
[0005] Some embodiments relate to an apparatus. The apparatus includes a processor in communication with a memory. The processor is configured to execute instructions stored in the memory, the instructions causing the processor to access first distance data determined by a distance sensing device at a first time, wherein the distance sensing device determines the first distance data when the object is located at a first position within the field of view of the distance sensing device, wherein the object is only partially located at the first position within the field of view. The processor is configured to execute instructions stored in the memory, the instructions causing the processor to access second distance data determined by the distance sensing device at a second time after the first time, wherein the distance sensing device determines the second distance data when the object is located at a second position within the field of view of the distance sensing device, wherein the first position is different from the second position, and the object is only partially located at the second position within the field of view. The processor is configured to execute instructions stored in the memory, the instructions causing the processor to determine a distance measurement model of the object based on the first distance data and the second distance data, wherein the distance measurement model is configured to model the change of a physical parameter of the object within the field of view of the distance sensing device over time. The processor is configured to execute instructions stored in the memory, the instructions causing the processor to determine third distance data based on the first distance data, the second distance data, and the distance measurement model, the third distance data indicating an estimated distance to the object before the object is completely within the field of view of the distance sensing device. The processor is configured to execute instructions stored in the memory, the instructions causing the processor to determine data indicating a height of the object based on the third distance data.
[0006] Some embodiments relate to at least one non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to access first distance data determined by a distance sensing device at a first time, wherein the distance sensing device determines the first distance data when an object is located at a first position within a field of view of the distance sensing device, wherein the object is only partially located at the first position within the field of view. The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to access second distance data determined by the distance sensing device at a second time occurring after the first time, wherein the distance sensing device determines the second distance data when the object is located at a second position within the field of view of the distance sensing device, wherein the first position is different from the second position and the object is only partially located at the second position within the field of view. The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to determine a distance measurement model for an object based on first distance data and second distance data, wherein the distance measurement model is configured to model changes over time in a physical parameter of an object within a field of view of a distance sensing device. The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to determine third distance data based on the first distance data, the second distance data, and the distance measurement model, the third distance data indicating an estimated distance to the object before the object is completely within the field of view of the distance sensing device. The non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to determine data indicating a height of the object based on the third distance data.
[0007] Thus, this document has outlined rather broadly the features of the disclosed subject matter so that the detailed description that follows may be better understood, and so that the present invention's contribution to the art may be better appreciated. Of course, additional features of the disclosed subject matter will be described hereinafter, and these additional features will also form the subject of the claims. It should be understood that the phraseology and terminology employed herein are for descriptive purposes only and should not be regarded as limiting. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The drawings are not drawn to scale. In the drawings, each identical or nearly identical component shown in various figures is represented by a like numeral. For clarity, not every component is labeled in every drawing. In the drawings:
[0009] Figure 1An exemplary machine vision system application according to some embodiments of the technology described herein is shown, wherein an object moves on a conveyor belt towards the field of view of an image sensing device;
[0010] Figure 2 Some embodiments according to the technology described herein are shown, wherein: Figure 1 further moving the object in the image sensing device so that the object is partially within the field of view of the image sensing device;
[0011] Figure 3 Some embodiments according to the technology described herein are shown, wherein: Figure 1 The object in has moved further, making Figure 2 The object is located further in the field of view of the image sensing device than;
[0012] Figure 4A An exemplary method of using distance measurements to determine and update a distance measurement model according to some embodiments of the techniques described herein is shown;
[0013] Figure 4B An exemplary detailed method for determining and updating a distance measurement model using distance sensing measurements according to some embodiments of the techniques described herein is shown;
[0014] Figure 5A An exemplary method of determining and updating a distance measurement model using distance measurements and parameter estimates according to some embodiments of the techniques described herein is shown; and
[0015] Figure 5B An exemplary detailed method for determining and updating a distance measurement model using distance measurements and parameter estimates according to some embodiments of the techniques described herein is shown. DETAILED DESCRIPTION
[0016] As described above, the technology described herein can be used to predict object features in various machine vision applications. For example, the technology can be used in logistics machine vision applications, including conveyor-based applications, sorting applications, and / or similar applications. Some logistics applications use fixed-focus lenses with deep depth of field and small apertures. Due to the use of a small aperture, the image may not be well illuminated because only limited light can pass through the lens. Optionally, autofocus technology can be used instead of a fixed-focus lens. Since autofocus technology can use a lens with a shallow depth of field, more light can be collected, thereby improving image quality. A fast-focusing lens can be used, such as a liquid lens that can change focus within 1-3 milliseconds. A distance sensing device such as a time-of-flight sensor can be used to detect the height of an object, and the height of the object can be used to drive the lens to automatically focus. The time-of-flight sensor can operate at a rate of, for example, 3-5 kHz.
[0017] Although autofocus technology can improve image quality, the inventors have discovered and realized that such technology may not be able to focus quickly enough to allow the imaging device to collect a sufficient number of high-quality images of the object. For example, high-speed logistics applications may use conveyor belts to move objects at very high speeds, such as 600-800 feet per minute (or 10-13.5 feet per second). Therefore, the object may only stay in the field of view of the imaging device for less than a second, or even less than 100-500 milliseconds. Imaging sensors typically operate at a frequency of 50-60 Hz, and each image is about 15-20 milliseconds. For high-speed logistics applications, the imaging device may only be able to collect some useful images of the object when it is within the field of view of the imaging sensor (for example, only 1-2 images of a complete barcode on the object). This problem may be further compounded for tall objects that are closer to the imaging sensor, which may further limit the opportunity to collect enough images of the object (for example, because the field of view is more limited for taller objects, the object is in the field of view less time than shorter objects). The system can be further limited by the distance sensing device. For example, while a time-of-flight sensor can operate at high speeds (e.g., such that a time-of-flight sensor can obtain 10-20 time-of-flight readings between each image acquisition of a 50-60 Hz imaging device), a time-of-flight sensor can only provide accurate height measurements when the entire object is within the field of view of the time-of-flight sensor. Thus, for many machine vision systems, such as systems that place the time-of-flight sensor at or near the image sensor, the time-of-flight sensor typically cannot provide fast enough height measurements to drive autofocus applications.
[0018] Accordingly, the inventors have discovered and appreciated that there is a need to provide rapid object height detection to address these and other problems with existing machine vision systems. For example, rapid detection of object height may be used to focus a lens assembly before and / or while an object (or a relevant portion of an object) moves into the field of view of an imaging device, such that the imaging device has a maximum amount of time to acquire an image of the object while the object is within the field of view of the imaging device. The inventors have developed techniques for predicting the height of a moving object before the entire object is within the field of view of a distance sensing device. Such techniques may allow a machine vision system to include both an imaging device and a distance sensing device within a single package and still allow the machine vision system to automatically focus the lens assembly faster than prior distance sensing devices. For example, if a distance measuring device performs N object acquisitions as the object enters the field of view of the distance measuring device until the object fully enters the field of view, the technique may estimate object parameters twice as fast (e.g., reducing the time required to acquire an object). 1 / 2×N acquisitions). Additionally or alternatively, such techniques may provide greater accuracy than using imperfect data. For example, these techniques may perform autofocus with only a few noisy distance measurements (e.g., only a few noisy distance measurements are taken before the object is completely within the field of view of the system), which may enable autofocus to be performed at faster conveyor speeds than before.
[0019] In the following description, many specific details about the systems and methods of the disclosed subject matter and the environments in which such systems and methods can operate are set forth to facilitate a thorough understanding of the disclosed subject matter. In addition, it should be understood that the examples provided below are exemplary, and it is contemplated that there are other systems and methods within the scope of the disclosed subject matter.
[0020] Figure 1 1 is a diagram 100 illustrating an exemplary machine vision system application according to some embodiments of the techniques described herein, wherein an object 102 moves on a conveyor belt 104 toward a field of view 106 of an image sensing device 108. In this example, the object 102 has a height H 110, and the object 102 moves on the conveyor belt 104 in a direction 112. The image sensing device 108 is at a distance D 114 from the conveyor belt 104. Figure 2 Some embodiments according to the technology described herein are shown, in which Figure 1 The object 102 is further moved along the direction 112 so that the object 102 is partially located in the field of view 106 of the image sensing device 108. Figure 2 As shown, a portion Sk1 200 of the top surface of the object 102 is within the field of view S 106 . Figure 3 Some embodiments according to the technology described herein are shown, in which Figure 1 The object 102 has moved further in the direction 112, so that Figure 2 In comparison, object 102 is further within field of view 106 of image sensing device 108. Figure 3 As shown, a portion Sk2300 of the top surface of the object 102 is within the field of view S 106. The portion Sk2300 is larger than Figure 2 The Sk1 200 part.
[0021] Although Figure 1-Figure 3 Only one object 102 is shown, but it should be understood that many objects may travel on a conveyor and that the objects may not be uniform in shape, spacing, etc. Thus, in many vision system applications, such as symbol information decoding in logistics operations (e.g., tracking a barcode on a package as it passes along a conveyor line), the number, height, length, overall size, and spacing between objects may vary greatly.
[0022] In some embodiments, the distance sensing device is a time-of-flight sensor, such as an integrated single-point or multi-point time-of-flight sensor that predicts distance information. The time-of-flight sensor may include a transmitter configured to emit a light beam such as a laser beam or a light pulse (e.g., an infrared (IR)-based light pulse), and a receiver configured to receive a reflected light beam. The time-of-flight sensor can modulate the intensity of the light beam at a high frequency so that there is a phase shift between the emitted light beam and the reflected light beam. The time-of-flight sensor may include a circuit that measures the degree of phase shift by comparing the phase of the transmitter and the receiver. The measured phase shift is then converted into a distance measurement based on calibration parameters in the sensor and / or external electronic devices.
[0023] Although not shown, the distance sensing device 108 can be disposed near the imaging device, such as being combined with the imaging device into a single package and / or mounted near the imaging device (e.g., mounted on a lens assembly of the imaging device). An exemplary configuration may include an imaging device integrated with a time-of-flight (ToF) sensor and / or an illumination device, such as a high-power integrated torch-shaped light source (HPIT) provided by Cognex, the assignee of the present application. Another exemplary configuration may include an imaging device and a ToF sensor mounted at a different position from the imaging device. In this instance, the imaging device and the ToF sensor may be mounted with one or more configuration constraints and / or other system configuration constraints, such as an optical axis perpendicular to a conveyor belt mounted at the same height, the ToF sensor being upstream of the conveyor belt of the imaging device, so that the ToF sensor can perform measurements before the object reaches the imaging device. In some embodiments, the imaging device may include one or more internal processors (e.g., FPGA, DSP, ARM processor, etc.) and other components that allow it to act as an independent unit, thereby providing desired output data (e.g., decoded symbol information) to a downstream process (e.g., an inventory tracking computer system or a logistics application). In some embodiments, the machine vision system may include external processing functionality, such as a personal computer, laptop computer, etc., configured to provide the processing functionality of the machine vision system.
[0024] In some embodiments, the machine vision system may be an image-based symbol information reader. The reader may include an imaging device having an image sensor and optics and a vision system processor, the imaging device being configured to find and decode symbols (e.g., bar codes) in images captured by the image sensor. The distance sensing device 108 may be integrated with the imaging device as described herein (e.g., in conjunction with a Figure 4A-Figure 5B ) predicts distance information of an object 102 in the field of view of the image sensor. The time-of-flight sensor may be operably connected to at least one of the vision system processor and / or the imager controller.
[0025] In some embodiments, the imaging device may include a lens assembly and / or be in optical communication with a lens assembly. The selection of a lens configuration may be based on a variety of factors, such as lighting / illumination, field of view, focal length, relative angles between the camera axis and the imaging surface, and / or the size of details on the imaging surface. In some instances, the selection of a drive lens may also be based on the cost of the lens and / or the available space for mounting the vision system. An exemplary lens configuration that may be required in certain vision system applications is an autofocus assembly. For example, an autofocus lens may be facilitated by a so-called liquid lens assembly.
[0026] When using liquid lenses, there is no need to manually touch or adjust the lens, which can simplify vision system installation, setup, and maintenance. Liquid lenses can have fast response times relative to other autofocus mechanisms. Liquid lenses can also be used in applications where the read distance varies depending on the object (surface), or when reading from one object to another, such as scanning a moving conveyor containing objects of varying sizes / heights (e.g., shipping boxes). Although Figure 1-Figure 3 The example shown in shows a conveyor machine vision application, but it should be understood that in many different vision system applications, fast focusing for imaging may be desirable, and thus the techniques described herein are not limited to this exemplary embodiment.
[0027] Liquid lenses can be implemented in different ways. One exemplary liquid lens embodiment uses water and oil, and the oil can be shaped essentially into a lens using an applied voltage. A change in voltage across the lens from a surrounding circuit causes a change in the curvature of the oil-water interface, which in turn causes a change in the focal length of the lens. Another exemplary liquid lens embodiment uses a movable membrane covering a liquid container to change its focal length. A bobbin can apply pressure to change the shape of the membrane, thereby changing the lens focus. The bobbin can be moved by varying the input current within a preset range. Different current intensities can provide different focal lengths for the liquid lens.
[0028] As described herein, distance information from a distance sensing device can be processed to automatically focus a variable (e.g., liquid) lens during run-time operation based on an object of a particular size / shape within the field of view and before the object is completely within the field of view of a time-of-flight sensor. For example, the predicted distance information can be used to set the focal length of a lens of an imaging device before the object is partially and / or completely within the field of view of the imaging device. In some embodiments, the system is configured such that the speed of the distance measuring device is higher than that of the imaging sensor. For example, a 50 Hz imaging sensor can be used with a 4 kHz time-of-flight sensor.
[0029] Figure 4AAn exemplary method 400 for determining and updating a distance measurement model using distance measurements according to some embodiments of the techniques described herein is shown. A machine vision system can perform method 400 to predict features (e.g., height, area, etc.) of an object before the object is completely within the field of view of a distance sensing device. For example, referring to Figure 2 and Figure 3 , method 400 can be used to predict the height H110 of object 102 before object 102 is completely within the field of view S of distance sensing device 108. In step 402, the machine vision system determines a feature measurement model, such as a distance measurement model, that models the change in the distance measurement from the distance sensing device to the object over time as the object enters the field of view of the distance sensing device to predict the height of the object before the object is completely within the field of view. As described herein, when the object begins to enter the field of view, the machine vision system can determine the distance measurement model based on predetermined parameters and / or based on an initial distance measurement of the object.
[0030] The machine vision system continuously collects distance data that changes over time and updates the distance measurement model accordingly. In step 404, the machine vision system collects and / or processes new distance measurement data. In step 406, the machine vision system updates the distance measurement model state 406 based on the distance measurement data collected in step 404. The method returns to step 404 and processes and / or collects new distance measurement data. The method performs steps 404 and 406 until the distance measurement model fully converges with the distance measurement data. For example, as further described herein, when the change in the distance measurement model and / or the estimated height of the object determined in step 406 is below a predetermined threshold, the machine vision system can determine that the distance measurement model converges with the time-of-flight measurement data.
[0031] Figure 4B An exemplary detailed method 450 for determining and updating a distance measurement model using distance sensing measurements according to some embodiments of the techniques described herein is shown. In step 452, a machine vision system accesses first distance data determined by a distance sensing device (e.g., a time-of-flight sensor) at a first time. The distance sensing device acquires and / or determines the first distance data when an object is located at a first location that is only partially within a field of view of the distance sensing device. For example, with reference to Figure 2 When the object 102 is only partially located within the field of view S 106 , the distance sensing device 108 collects first distance data, as shown in Sk1 200 .
[0032] In step 454, the machine vision system accesses second distance data determined by the distance sensing device at a subsequent second time after the first time. The distance sensing device acquires and / or determines the second distance data when the object is at a second location that is still only partially within the field of view of the distance sensing device. Because the distance data is determined over time as the object moves, the first location associated with the first distance data is different from the second location associated with the second distance data. For example, referring to Figure 3 When the object 102 is only partially within the field of view S 106, the distance sensing device 108 collects second distance data, as shown in Sk2 300. Figure 2 In comparison, although Figure 2-Figure 3 In the example, object 102 is only partially within field of view 106, but object 102 is Figure 3 In Figure 2 is further within the field of view 106 (hence Sk2 300 is greater than Sk1 200).
[0033] As described herein, the distance sensing device can be a time-of-flight sensor that is part of a machine vision system that also includes internal and / or external processing hardware and associated software for performing machine vision tasks. Referring to steps 452 and 454, accessing the first distance data and the second distance data can include processing hardware (e.g., hardware that is part of an imaging device) receiving the first time-of-flight measurement and the second time-of-flight measurement from the time-of-flight sensor. As another example, the processing hardware can access the time-of-flight measurement from a memory shared with the time-of-flight sensor.
[0034] In step 456, the machine vision system determines a distance measurement model for the object based on the first distance data and the second distance data. The distance measurement model is configured to model changes over time in a physical parameter of an object within the field of view of the distance sensing device, such as the height of the object, the surface area of the object, etc. In step 458, the machine vision system determines distance data indicating an estimated distance to the object before the object is completely within the field of view of the distance sensing device based on previous distance data (e.g., the first distance data and the second distance data for the first execution) and the distance measurement model.
[0035] In step 460, the machine vision system determines whether the distance measurement model converges with the distance measurement data. In some embodiments, the machine vision system can determine that the change in the distance measurement model is below a predetermined threshold. In some embodiments, the machine vision system can determine that the change between the distance measurement data and the estimated height of the object (e.g., the data determined in step 458) is below a predetermined threshold. For example, as further described in conjunction with equations 1-2, the height of the object can be modeled as a parameter as part of an observation matrix, and the observation matrix can be updated at each iteration. If after multiple iterations, the predicted height is close to the observed value (e.g., within a threshold), the system can determine that the model used in the current iteration is sufficient and the observation matrix can be used to determine the height of the object (e.g., so that the distance measurement model is stable in this state). If the machine vision system determines that the distance measurement model has not converged with the distance measurement data, the method 450 proceeds to step 462 and updates the distance measurement model with one or more additional distance measurements, and then returns to step 458.
[0036] If the machine vision system determines that the distance measurement model has converged with the distance measurement data, method 450 proceeds to step 464 and the machine vision system determines data indicative of the height of the object based on the distance data determined in step 458. For example, the machine vision system may use the last determined data from step 458 as an indication of the height of the object.
[0037] As described herein, a machine vision system can use data indicating the height of an object to perform various tasks of the machine vision system. For example, the machine vision system can determine data indicating a focus adjustment of a lens of an imaging device associated with a distance sensing device based on the distance data. The machine vision system can send one or more signals to change the focus of the lens based on the data indicating the focus adjustment. In some embodiments, the machine vision system uses the focus adjustment to change the focus of the liquid lens. The machine vision system can capture an image of the object after transmitting one or more signals to change the focus of the lens. In some embodiments, the machine vision system can be configured to wait a predetermined amount of time after sending one or more signals to change the focus of the lens before capturing an image of the object. In some embodiments, the system can receive feedback data from the liquid lens assembly indicating that the focus adjustment is complete, and the machine vision system can capture an image in response to receiving the feedback data.
[0038] In some embodiments, the machine vision system may determine data indicating brightness adjustments for a lighting module of the imaging device based on the estimated distance data. For example, if the distance data indicates a short object, the machine vision system may be configured to use a higher brightness setting for the lighting module (compared to when the distance data indicates a taller object). Thus, the machine vision system may be configured to use brighter lighting for objects that are farther from the imaging device and softer lighting for objects that are closer to the imaging device. In some embodiments, to adjust the lighting of the object, the technology may adjust the exposure time of the imaging device to adjust the image brightness (e.g., without adjusting the lighting module). For example, the system may reduce the exposure time for objects that are closer to the camera.
[0039] In some embodiments, the machine vision system can access data indicating at least one constraint of the machine vision application. For example, the machine vision system can access data indicating motion parameters of an object associated with the motion of the box through the field of view of the time-of-flight sensor. The machine vision system can use the data indicating the motion parameters to determine the distance data. For example, referring to Figure 4B In step 458, the machine vision system may determine the distance data based on the first distance data, the second distance data, the distance measurement model, and one or more motion parameters. In some embodiments, the motion parameters include the speed of the object and / or the acceleration of the object when the object passes through the field of view of the distance sensing device. For example, the machine vision system may determine the distance data based on the first distance data, the second distance data, the distance measurement model, and the speed of the object.
[0040] In some embodiments, when an object enters the field of view of the distance sensing device, the machine vision system can use a Kalman filter to model the distance to the object. For example, the machine vision system can use a Kalman filter to estimate the change of objects in the area over time. Figure 4B In step 458 of the method, for example, the machine vision system may use the distance measurement model to determine an estimate of the area of the object based on the collected distance data (e.g., collected in steps 452, 454, and / or 462). For example, the machine vision system may determine a first object area estimate for an object based on the first distance data (the object is only partially within the field of view of the distance sensing device). The machine vision system may use a Kalman filter to determine subsequent object area estimates. For example, the machine vision system may use the distance measurement model to determine a second object area estimate for an object based on the first object area estimate and the second distance data (the object is still only within the field of view of the distance sensing device). The machine vision system may determine a height estimate for the object based on the second object area estimate.
[0041] The machine vision system can be configured to use one or more equations to measure state (e.g., the area of an object within the field of view) and / or perform measurement updates of a distance sensing device (e.g., updating the estimated distance to an object). The following exemplary equation 1 can be used to perform state updates:
[0042] S k =S k-1 +S Δ +w k Equation 1
[0043] in:
[0044] S k is the area of the top surface of the object within the field of view of the distance sensing device at the current time k;
[0045] S k-1 is the area of the top surface of the object in the field of view of the distance sensing device at the previous time k-1;
[0046] S Δ YesS k and S k-1 and
[0047] w k is a model for system noise, random variation, and / or measurement noise / error.
[0048] As shown in Equation 1, the state can be the area of the object in the field of view of the distance determining device. These states can be modeled based on the parameters of the machine vision system. For example, if the object moves at a near constant speed (such as on a conveyor belt), and the frame rate of the time of flight (TOF) sensor is high enough, the state update can be modeled with a linear function, where the change in the object area is constant per unit time. In some embodiments, the first two object area measurements can be initial values determined based on the machine vision application specifications, such as box size range, conveyor belt width, object movement speed, etc. It should be understood that the initial object area measurements do not need to be accurate. For example, w k It can be used to model noise from the physical world, such as inconsistent speed of conveyor belts. w can be calculated based on the system noise and the observation covariance matrix. k (and, for example, the following v k ) is used to model the value of . k Initial values may be assigned based on the application (e.g., based on expected changes in conveyor speed, accuracy of TOF readings, etc.). The initial values do not have to be exact, but may be updated in each iteration to improve accuracy using the Kalman filter. From the third state (and beyond), the object area and / or height is obtained and determined from the model using real-time distance measurements, and the model may be updated for each new distance measurement.
[0049] Equation 2 provides an exemplary equation for performing distance measurements:
[0050] TOF k =-(h / S)S k +d+v k Equation 2
[0051] in:
[0052] TOF k is the flight time measurement at time k;
[0053] h is the height of the object;
[0054] S is the field of view of the time-of-flight sensor;
[0055] S k is the area of the top surface of the object within the field of view of the distance sensing device at the current time k;
[0056] d is the distance between the time-of-flight sensor and the conveyor belt; and
[0057] v k is a model for the observation noise in the distance measurement (e.g., noise from TOF reading errors).
[0058] Referring to Equation 2, the TOF sensor reading may be determined based on a weighted average of the height h and the distance d of the object.
[0059] It should be understood that Equations 1 and 2 are for illustrative purposes only, and that other models may be used for specific machine vision applications. For example, other forms of the Kalman filter model may be used, depending on whether the moving object is subject to acceleration, whether the area of the object changes, and / or other changing factors. As an object enters the field of view of the time-of-flight sensor, the measurement of the time-of-flight sensor may slowly decrease over time (e.g., from measuring the distance to the conveyor belt to measuring the distance to the top of the object). The estimated box area that can be determined by the model may correspondingly increase slowly over time and inversely proportional to the time-of-flight measurement, such that the estimated box area increases over time.
[0060] In some embodiments, the machine vision system can use parameter estimation techniques (e.g., expectation maximization (EM), maximum a posteriori probability (MAP), etc.) to update the parameters of the model. For example, the height of the object can be part of the observation matrix as shown in exemplary equation 2. The parameter estimation algorithm can be used to update the box height in each iteration. Figure 5A5 is an exemplary method 500 for determining and updating a distance measurement model using distance measurements and parameter estimates according to some embodiments of the techniques described herein. In step 502, a machine vision system determines a distance measurement model using the techniques described herein. Whenever the system obtains a new distance observation in step 504, in step 506, the system uses a model (e.g., a Kalman filter) to update the state of the distance measurement model, and in step 508, the machine vision system uses the parameter estimates to update the system observation matrix.
[0061] Figure 5B An exemplary detailed method 550 for determining and updating a distance measurement model using distance measurements and parameter estimates according to some embodiments of the techniques described herein is shown. In step 552, a machine vision system accesses first distance data determined at a first time, wherein an object is located at a first position that is only partially within a field of view of a distance sensing device (e.g., as shown in FIG. 5A ). Figure 2 In step 554, the machine vision system accesses second distance data determined at a subsequent second time when the object is at a second location, wherein the second location is still only partially within the field of view of the distance sensing device (e.g., as shown in FIG. 1 ). Figure 3 In step 556, the machine vision system determines a distance measurement model of the object based on the first distance data and the second distance data.
[0062] In step 558, the machine vision system determines distance data based on previous distance data (e.g., the first distance data and the second distance data for the first execution) and the distance measurement model, which indicates an estimated distance to the object before the object is completely within the field of view of the distance sensing device. In step 560, the machine vision system determines whether the distance measurement model converges with the distance measurement data. If the machine vision system determines that the distance measurement model does not converge with the distance measurement data, the method 550 proceeds to step 562 and updates the parameters using parameter estimation (e.g., using EM, MAP and / or other parameter estimation techniques). The method proceeds to step 564, updates the distance measurement model with one or more additional distance measurements, and returns to step 558.
[0063] If the machine vision system determines that the distance measurement model has converged with the distance measurement data, method 550 proceeds from step 560 to step 564, and the machine vision system determines data indicative of the height of the object based on the distance data determined at step 558. For example, the machine vision system may use the last determined data from step 558 as an indication of the height of the object.
[0064] As described herein, as an object enters the field of view of the time-of-flight sensor, the measurement of the time-of-flight sensor can slowly decrease (e.g., from measuring the distance to the conveyor belt to measuring the distance to the top of the object). The estimated box area determined using the distance measurement model can correspondingly increase slowly over time to be inversely proportional to the time-of-flight measurement, such that the estimated box area increases over time. In some embodiments, the relationship between the time-of-flight measurement and the box area measurement can be fine-tuned using parameter estimation.
[0065] The predicted distance information generated by the distance sensing device may also be used to perform other aspects of the machine vision system. In some embodiments, the machine vision system may use the predicted distance information to automatically trigger the vision system to capture an image of an object. For example, when the system determines a change in the distance of an object (e.g., a change in height from a support base / conveyor), the system may trigger image capture. In some embodiments, the machine vision system may use the predicted distance information to perform object (e.g., box) dimensioning. During the calibration process, the machine vision system may measure the distance D 114 between the conveyor and the vision system and store the information. At run time, the machine vision system may capture images and may predict the distance to the object before the object is fully within the image and / or measure the distance to the object to the center of the image / field of view. If the system detects a rectangular shape, the size of the rectangle may be determined based on the predicted and / or measured distances (e.g., and based on known optical properties of the imager, such as image sensor size, lens focal length, etc.). As Figure 1-Figure 3 As shown, the height 110 of the object 102 may substantially represent the difference between the height of the imaging device from the conveyor belt 104 and the predicted and / or measured shortest distance between the imaging device and the object 102. In some embodiments, since the imaging device is fixedly mounted to the time-of-flight sensor, the height between the imaging device and the object 102 may be represented by the distance D 114.
[0066] In some embodiments, a machine vision system can be used to detect and analyze object defects. For example, after the machine vision system detects a (e.g., rectangular) top surface as described herein, the machine vision system can measure deviations from the top surface shape to detect damaged objects (e.g., damaged boxes). In some embodiments, the system can perform region of interest (RoI) detection. For example, the field of view of a camera-based symbol information reader can be imaged onto the sensing area of a multi-point (e.g., n×1 or n×m) time-of-flight sensor array. The sensor array can measure a 3D height map to narrow the area of interest for symbol information decoding. Determining which part of the image an object resides in can reduce decoding time because symbol candidate features can be searched from a narrowed area of interest in the entire acquired image.
[0067] In some embodiments, the machine vision system can detect gaps between objects in the field of view, which can help link symbol codes to the appropriate imaged objects. For example, in logistics applications, there may be more than one object entering and / or within the field of view at a time, and time-of-flight measurements can help locate the edges of the objects and determine what symbol is associated with each object in the field of view.
[0068] In some embodiments, the machine vision system can use the distance predicted and / or measured by the time-of-flight sensor to limit the reading range of the vision system to prevent unintentional reads. For example, if the distance to the object is within a defined range, the machine vision system (a) acquires an image, (b) initiates the symbol decoding process, and / or (c) transmits the decoded data for further processing.
[0069] The technology operated according to the principles described herein can be implemented in any suitable manner. For example, the embodiment can be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or processor set, whether provided in a single computer or distributed in multiple computers. Such a processor can be implemented as an integrated circuit, with one or more processors in an integrated circuit assembly, including commercially available integrated circuit assemblies known in the art, such as CPU chips, GPU chips, FPGA chips, microprocessors, microcontrollers or coprocessors. Alternatively, the processor can be implemented in a custom circuit, such as an application specific integrated circuit (ASIC), or a semi-custom circuit generated by configuring a programmable logic device. As another option, the processor can be part of a larger circuit or semiconductor device, whether commercially available, semi-custom or custom. As a specific example, some commercially available microprocessors have multiple cores, so that one or a subset of these cores can constitute a processor. However, the processor can be implemented using any suitable form of circuit.
[0070] Furthermore, it should be understood that the computer may be implemented in any of a variety of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. In addition, the computer may be embedded in a device that is not generally considered a computer but has appropriate processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other appropriate portable electronic device or fixed electronic device.
[0071] In addition, the computer may have one or more input and output devices. These devices may be used to present a user interface, etc. Examples of output devices that may be used to provide a user interface include a printer or display screen for visually presenting output, and a speaker or other sound generating device for auditory presenting output. Examples of input devices that may be used for a user interface include a keyboard and a pointing device, such as a mouse, a touch pad, and a digitized tablet computer. As another example, a computer may receive input information through speech recognition or other auditory forms. In the illustrated embodiment, the input / output device is shown as being physically separated from the computing device. However, in some embodiments, the input and / or output device may be physically integrated into the same unit as other elements of a processor or computing device. For example, a keyboard may be implemented as a soft keyboard on a touch screen. Alternatively, the input / output device may be completely disconnected from the computing device and functionally integrated via a wireless connection.
[0072] Such computers may be interconnected in any suitable form via one or more networks, including a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.
[0073] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine.
[0074] In this regard, the present invention may be embodied as a computer-readable storage medium (or multiple computer-readable media) encoded with one or more programs (e.g., computer memory, one or more floppy disks, CDs, optical disks, DVDs, tapes, flash memory, circuit configurations in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media), which, when executed on one or more computers or other processors, perform methods for implementing various embodiments of the present invention described above. It is apparent from the foregoing examples that computer-readable storage media can retain information for a sufficient period of time to provide computer-executable instructions in a non-temporary form. Such computer-readable storage media may be transportable so that one or more programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present application as described above. The term "computer-readable storage medium" as used herein only covers computer-readable media that can be considered as a product (i.e., article of manufacture) or a machine. Alternatively or additionally, the present invention may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagation signal.
[0075] The terms "code", "program" or "software" are used in a general sense herein to refer to any type of computer code or computer executable instruction set that can be used to program a computer or other processor to implement various aspects of the present application as described above. In addition, it should be understood that, according to one aspect of this embodiment, when executing one or more computer programs, the method of implementing the present application does not require these computer programs to reside on a single computer or processor, but can be distributed in a modular manner among multiple different computers or processors to implement various aspects of the present application.
[0076] Computer executable instructions can be in various forms, such as program modules, that are executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, in various embodiments, the functionality of program modules can be combined or distributed as needed.
[0077] In addition, the data structure can be stored in a computer-readable medium in any suitable form. To simplify the description, the data structure can be shown as having fields that are related by position in the data structure. This relationship can also be achieved by assigning storage locations to the fields in a computer-readable medium that conveys the relationship between the fields. However, any suitable mechanism can be used to establish the relationship between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms that establish relationships between data elements.
[0078] The various aspects of the present application can be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore its application is not limited to the details and arrangements of the components set forth in the foregoing description or shown in the drawings. For example, aspects described in one embodiment can be combined in any manner with aspects described in other embodiments.
[0079] In addition, the present invention can be embodied as a method, examples of which have been provided. The actions performed as part of the method can be ordered in any suitable manner. Thus, embodiments can be constructed in which the actions are performed in a different order than shown, which can include performing some actions simultaneously, even though they are shown as sequential actions in the illustrative embodiments.
[0080] The expressions "a" and "an" used in the specification and claims should be understood to mean "at least one" unless there is a clear indication to the contrary.
[0081] The expression "and / or" used in the specification and claims should be understood as "one or both" of the combined elements, that is, the elements exist in combination in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same way, that is, "one or more" of the combined elements. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether or not these other elements are related to those specifically identified elements. Therefore, as a non-limiting example, in one embodiment, when used in conjunction with open language such as "comprises", a reference to "A and / or B" may refer only to A (optionally including elements other than B); in another embodiment, only to B (optionally including elements other than A); in yet another embodiment, to A and B (optionally including other elements); and so on.
[0082] As used in the specification and claims, with respect to a list of one or more elements, the phrase "at least one" is understood to mean at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of all elements specifically listed in the list of elements, and does not exclude any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether or not they are related to those specifically identified elements. Thus, as a non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently, "at least one of A and / or B") may refer to at least one (optionally including more than one) A in one embodiment, and no B (and optionally including elements other than B); in another embodiment, may refer to at least one (optionally including more than one) B, and no A (and optionally including elements other than A); in yet another embodiment, may refer to at least one (optionally including more than one) A, and at least one (optionally including more than one) B (and optionally including other elements); and so on.
[0083] When ordinal terms such as "first", "second", "third", etc. are used to modify the claims, they themselves do not imply any priority, precedence or order of one claim element relative to another claim element, or the temporal order of performance of the actions of the method, but are merely used as labels to distinguish one claim element with a specific name from another claim with the same name (but using ordinal numbers) to distinguish the claim elements.
[0084] Likewise, the words and terms used herein are for descriptive purposes and should not be considered limiting. "Including," "comprising," or "having," "covering," "involving," and variations thereof used herein are meant to encompass the items listed thereafter and their equivalents as well as other items.
Claims
1. A computerized method comprising: accessing, at a first time, first distance data determined by a distance sensing device, wherein the distance sensing device determines the first distance data when an object is located at a first position within a field of view of the distance sensing device, wherein the object is only partially located at the first position within the field of view; accessing second distance data determined by the distance sensing device at a second time occurring after the first time, wherein the distance sensing device determines the second distance data when the object is located at a second position within the field of view of the distance sensing device, wherein the first position is different from the second position and the object is only partially located at the second position within the field of view; determining a distance measurement model for the object based on the first distance data and the second distance data, wherein the distance measurement model is configured to model changes in a physical parameter of the object within the field of view of the distance sensing device over time; determining, based on the first distance data, the second distance data, and the distance measurement model, third distance data indicating an estimated distance to the object before the object is completely within the field of view of the distance sensing device; and Based on the third distance data, data indicative of the height of the object is determined.
2. The computerized method of claim 1, wherein: The physical parameters of the object include the area of the object, the height of the object, or some combination of the area and the height of the object.
3. The computerized method of claim 1 , further comprising: Based on the third distance data, data indicative of a focus adjustment of a lens of an imaging device associated with the distance sensing device is determined.
4. The computerized method of claim 3, further comprising sending one or more signals to change the focus of the lens based on the data indicative of the focus adjustment.
5. The computerized method of claim 4, further comprising acquiring an image of the object after transmitting the one or more signals to change the focus of the lens.
6. The computerized method of claim 1, further comprising: Based on the third distance data, data indicative of a brightness adjustment of an illumination module of an imaging device associated with the distance sensing device is determined.
7. The computerized method of claim 1, further comprising: accessing data indicative of motion parameters of the object; as well as Determining the third distance data includes determining the third distance data based on the first distance data, the second distance data, the distance measurement model, and the motion parameters.
8. The computerized method of claim 7, wherein: The motion parameters include a velocity of the object as the object passes through the field of view of the distance sensing device.
9. The computerized method of claim 8, wherein: The motion parameters include acceleration of the object as the object passes through the field of view of the distance sensing device.
10. The computerized method of claim 8, wherein: Determining the third distance data includes determining the third distance data based on the first distance data, the second distance data, the distance measurement model, and the speed of the object.
11. The computerized method of claim 1, wherein: The distance sensing device includes a time-of-flight (ToF) sensor, and accessing the first distance data and the second distance data includes receiving a first time-of-flight measurement and a second time-of-flight measurement from the time-of-flight (ToF) sensor.
12. The computerized method of claim 1, wherein: Determining the third distance data based on the first distance data, the second distance data, and the distance measurement model includes: determining, based on the first distance data, a first object area estimate of a first area of the object within the field of view of the distance sensing device using the distance measurement model; and A second object area estimate of a second area of the object within the field of view of the distance sensing device is determined based on the first object area estimate and the second distance data using the distance measurement model.
13. The computerized method of claim 12, wherein: The distance measurement model is a Kalman filter.
14. The computerized method of claim 12, further comprising determining a height estimate for the object based on the second object area estimate.
15. The computerized method of claim 14, wherein: Determining the height estimate for the object includes determining the height estimate based on the second object area estimate using a parameter estimation algorithm.
16. The computerized method of claim 15, wherein: The parameter estimation algorithm is an expectation-maximization EM algorithm, a maximum a posteriori probability MAP algorithm or some combination thereof.
17. An apparatus for determining settings of an imaging device, comprising a processor in communication with a memory, the processor configured to execute instructions stored in the memory, the instructions causing the processor to: accessing, at a first time, first distance data determined by a distance sensing device, wherein: The distance sensing device determines the first distance data when an object is located at a first position within a field of view of the distance sensing device, wherein the object is only partially located at the first position within the field of view; accessing second distance data determined by the distance sensing device at a second time occurring after the first time, wherein the distance sensing device determines the second distance data when the object is located at a second position within the field of view of the distance sensing device, wherein the first position is different from the second position and the object is only partially located at the second position within the field of view; determining a distance measurement model for the object based on the first distance data and the second distance data, wherein the distance measurement model is configured to model changes in a physical parameter of the object within the field of view of the distance sensing device over time; determining, based on the first distance data, the second distance data, and the distance measurement model, third distance data indicating an estimated distance to the object before the object is completely within the field of view of the distance sensing device; and Based on the third distance data, data indicative of the height of the object is determined.
18. The device according to claim 17, wherein: The processor is further configured to execute instructions stored in the memory, the instructions causing the processor to: accessing data indicative of motion parameters of the object; and Determining the third distance data includes determining the third distance data based on the first distance data, the second distance data, the distance measurement model, and the motion parameter.
19. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the following actions: accessing, at a first time, first distance data determined by a distance sensing device, wherein: The distance sensing device determines the first distance data when an object is located at a first position within a field of view of the distance sensing device, wherein the object is only partially located at the first position within the field of view; accessing second distance data determined by the distance sensing device at a second time occurring after the first time, wherein the distance sensing device determines the second distance data when the object is located at a second position within the field of view of the distance sensing device, wherein the first position is different from the second position and the object is only partially located at the second position within the field of view; determining a distance measurement model for the object based on the first distance data and the second distance data, wherein the distance measurement model is configured to model changes in a physical parameter of the object within the field of view of the distance sensing device over time; determining, based on the first distance data, the second distance data, and the distance measurement model, third distance data indicating an estimated distance to the object before the object is completely within the field of view of the distance sensing device; and Based on the third distance data, data indicative of the height of the object is determined.
20. The computer-readable storage medium of claim 19, wherein: The processor-executable instructions further cause the at least one computer hardware processor to perform the following actions: accessing data indicative of motion parameters of the object; and Determining the third distance data includes determining the third distance data based on the first distance data, the second distance data, the distance measurement model, and the motion parameters.
