Method for unsupervised measurement of the dimensions of an object using a single camera view

Description

[0001] This disclosure relates to an unsupervised object sizing method for single camera viewing without using a reference object.BACKGROUND

[0002] Unsupervised object size estimation using single camera view and no reference object is a very complex and uncommon task for computer vision systems. Despite the complexity of the problem, the information provided by the object sizing is useful in a variety of domains, such as video surveillance systems, medical images, and other kind of systems. Object profiling, object identification or object state recognition are examples of higher-level tasks that may rely on the object size information. There is a number of problems arising from the decision of using a single camera-based approach. However, it also presents a number of advantages.

[0003] In real world solutions, it is not always possible to have reference objects with known size.

[0004] Supervised methods require continuous interaction between the user and the system. In contrast, the advantage of an unsupervised method is to provide output without user intervention. Thus, there is a need for systems that provide an accurate and non-reference-based object sizing information without the use of a reference object.

[0005] Common state of the art solutions size objects by means of some external sensors or the interaction of multiple cameras. Current solutions that provide object size measurement are sometimes inaccurate or require a guide object with a known size to be precise.

[0006] Additionally, object size measurement is a complex task, and there are some issues that generally appear, for instance, the irregular shape of some objects, the occlusion that can suffer when the objects of interest are in realistic environments, lack of knowledge of the object's scale or the distance of the camera sensor to the object.

[0007] One approach is described in US Patent Publication No. 2007 / 0098251 which is a case of a single camera, no extra sensor and no reference algorithm. In this method, movement of the camera is essential to obtain the object distance which is a drawback because there are many scenarios where movement of the camera is not possible.

[0008] Another approach based on a single camera without external sensor and no reference object is described in U.S. Patent Publication No. 2012 / 0274762. This method introduces a focus swept so as to keep away from the required movement of the camera to acquire two pictures to compute the size, but selecting two points of the image by the user is needed to measure the distance. Supervised sizing systems are generally at a disadvantage as compared to unsupervised systems.

[0009] U.S. Patent Publication No. 2018 / 0336733 introduces an approximation that perform object detection combined with view detection and 3D reconstruction, however this proposal uses two cameras.

[0010] This section provides background information related to the present disclosure which is not necessarily prior art.

[0011] D1 WO2016 / 184152 discloses a measuring method, comprising performing image collection on a measured object, and determining the size of a collected objective image of the measured object; and determining, according to the size of the objective image of the measured object, a position of the objective image and a distance between the measured object and an image collection device, a first actual length and / or a second actual length of the measured object.

[0012] D2 US2016 / 225150 discloses a method for determining an object's size based on calibration data is disclosed. The calibration data is measured by capturing images with an image sensor and a lens module, having at least one objective of the capsule camera at a plurality of object distances and / or back focal distances and deriving from the images characterizing a focus of each objective for at least one-color plane. Images of lumen walls of gastrointestinal (GI) tract are captured using the capsule camera. Object distance for at least one region in the current image is estimated based on the camera calibration data and relative sharpness of the current image in at least two-color planes. The size of the object is estimated based on the object distance estimated for one or more regions overlapping with an object image of the object and the size of the object image.

[0013] D3 JP2007327882 discloses a three-dimensional shape measuring method comprises projecting slit light on a measuring object 10D and receiving the reflection light; repeating operation acquiring two-dimensional images in the measuring object 10D in prescribed times by changing a focal distance; calculating the contrast of an image face in a plurality of areas A-G; extracting a high contrast area exceeding a prescribed threshold by the contrast of the image face at each two-dimensional image acquired by each focal distance; acquiring distance information of each area by performing triangulation in the high contrast area; and performing matching adjustment of a measurement range 20E so as to be included in the measurement range 20E having prescribed measurement depth(a) by each area based on the distance information.SUMMARY

[0014] This section provides a general summary of the disclosure and is not a comprehensive disclosure of its full scope or all of its features. The invention is defined by the appended claims.

[0015] An unsupervised method is presented for determining the size of an object captured in an image. The method includes: capturing a set of images with a camera, where each image in the set of images is captured with the camera set at a different f-number; identifying a given image in the set of images; calculating a hyperfocal distance using parameters of the camera when taking the given image; and quantifying one or more dimensions of the object as a function of the hyperfocal distance.

[0016] The method further includes: capturing a triggering image of a scene and identifying presence of an object in the triggering image. The presence of an object in the image may be identified using a background subtraction method.

[0017] In one embodiment, for each image in the set of images, a quality metric for the image is computed, where the quality metric quantifies the focus of an image based on the contrast.

[0018] The given image in the set of images can be identified by identifying a subset of images from the set of images, where each image in the subset of images has a contrast metric with a value greater than a threshold, and selecting the given image from the subset of images, where the given image has the lowest f-number amongst the images in the subset of images.

[0019] In some embodiments, the one or more dimensions of the object may be quantified using the triangle similarity theorem and the contrast metric is further defined as fuzzy entropy.

[0020] In other embodiments, the method further includes: identifying the object in the triggering image using a neural network; extracting a region of interest from each image in the set of images, where the region of interest contains the identified object; and for each image in the set of images, computing the contrast metric for the image using only the region of interest extracted from the image.DRAWINGS

[0021] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure. FIG. 1 is a flowchart showing a general computer vision system. FIG. 2 is a block diagram depicting a system for object sizing. FIG. 3 is a flow diagram for an example embodiment of an object sizing method. FIG. 4 is a diagram showing the camera disposition for a known object dataset. FIGs. 5A and 5B are a frontal view and a perspective view of Rubik's Cube, respectively; and FIGs. 5C and 5D are a frontal view and a perspective view of watering can, respectively.

[0022] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0024] Figure 1 depicts a general computer vision system 10 which employs a standard architecture for designing computer vision applications. The main goal of the system is to provide a complete object characterization. Also, the defined architecture facilitates the integration of the multiple algorithms required to perform object profiling in the same architecture.

[0025] Generality object detector 11, from now on "generality detector, view classifier 12 and 3D reconstruction 13 are common to all the tasks and are described in detail later in the document. Depending on the application, the design follows the Categorizer branch 14 or the Identifier branch 16. The final step in the Categorizer branch is determining the object state, for example if the object is broken or not, as well as if it has missing parts or not. The methods set forth in this disclosure lie in the Identifier branch. The Identifier branch has itself two possible subbranches: sizing related applications 17 and features description applications 18. More specifically, this disclosure pertains to object sizing 17; whereas, the features branch 18 returns object attributes like the color, the material, etc.

[0026] Figure 2 depicts a system 20 for determining the size of an object captured in an image. The system 20 is comprised generally of: a camera 21, an image preprocessor 23, an object detector 24 and an object size calculator 25. The image preprocessor 23 is interfaced with the camera 21 and interacts with the camera 21 to capture a set of images 22, where each image in the set of images 22 is captured with the camera set at a different f-number. The object detector 24 is configured to receive the set of images 22. For each image in the set of images, the object detector 24 processes a given image with the highest quality metric, i.e. the highest contrast / f-number ratio, amongst the images in the set of images. The object size calculator 25 in turn receives the given image from the object detector 24. The object size calculator 25 uses the hyperfocal distance, calculated using parameters of the camera, and quantifies one or more dimensions of the object as a function of the hyperfocal distance.

[0027] Figure 3 further illustrates an example embodiment for determining the size of an object captured in an image. At step 31, a camera with a convex lens captures a frame. A frame is defined as an image of the observed scene. In this embodiment, objects to be sized appears in the frame.

[0028] At step 32, the presence of objects in the frame is evaluated. A background subtraction algorithm can be used to detect foreground objects in the scene. In one embodiment, the selected algorithm is Background Subtraction using Local Singular Valued Decomposition 25 Binary Pattern (BSLSVDBP) combined with objectness algorithm based on Binarized Normed Gradient as described by M. Cheng et al in "Binarized Normed Gradients for Objectness Estimation at 300 fps", 2014 IEEE Conference on Computer Vision and Pattern Recognition (2014). This BSLSVDBP algorithm is single-parameter dependent. The parameter is the learning rate. It is selected depending on the scenario. It is possible to use other background segmentation methods or region proposal methods to detect an object presence in the scene. This is a key step for the embodiment, as it addresses the unsupervision challenge.

[0029] If objects are detected in step 32, a focus swept is performed at step 33. The focus is defined as the camera aperture value to change the hyperfocal distance with a focal length and circle of confusion fixed values. The camera focus range is swept. For each focus value, a frame-focus value pair is saved. The number of saved frames is set as the maximun value between a minimum value (30) and the total discrete values in the camera focus range. The resulting frame set is forwarded at 34 to the next step. Object identification is performed at step 35. The analysis is performed only in one of the frames contained in the set. The analysis output may be: known object together with its associated category, or unknown object. In one embodiment, object identification is carried out by a neural network algorithm developed particularly for this duty. For example, a ResNet-101 neural network pre-trained in Image-Net may be used as described by K. He et al in "Deep Residual Learning For Image Recognition", 2016 IEEE Conference on Computer Vision and Pattern Recognition (2016) although other neural networks designed to carry out object recognition can be used in place. The model has been trained with a specific dataset composed of pre-defined (known) objects, where the objects are selected depending of the scenario where the system is implemented. For example, in an airport scenario, the selected objects are handbags, backpacks, trolleys, etc. The set of known objects may be changed by defining a new dataset to train.

[0030] Depending upon whether the object was identified, the method proceeds along one of two different paths. In the case the object is unknown, a region of interest (ROI) is extracted at 37 from each image in the frame set, where the region of interest contains the unknown object. For example, the region of interest is defined by the object bounding box and its mask, for example output in the step 32. Note that the same area is isolated in all the frames of the frame set. The resulting set of isolated areas is used as an input for step 38.

[0031] Next, the method identifies a given image from the set of isolated areas using a quality metric define as the ratio: focus metric / f - number, at step 38. In one embodiment, the set of isolated areas is scanned in search of the subset of images that surpass a particular focus metric threshold. The threshold is determined empirically. The quality is analyzed in terms of focus, specifically using a contrast metric. A contrast metric is computed for each image. In one embodiment, the contrast metric is computed with the fuzzy entropy at a pixel level on its neighbor window of size w x w pixels: By default, w = 5. This value can be modified depending on the size of the regions of interest. The full region of interest contrast is computed as the sum of all pixel contrast measurements. From the regions of interest with a contrast value that exceeds the threshold, we will select the one with the lowest aperture value (i.e., f-number). This selected image, along with its associated focus value, is forwarded to the next steps.

[0032] A hyperfocal distance (H) is calculated at step 39 using parameters of the camera when taking the image. In one embodiment, the hyperfocal distance is computed with the following equation: H = f 2 Nc + f where f is the focal distance, N is the F-number (i.e., f / D where D is the diameter of the aperture) and c is the circle of confusion limit. f and c are parameters associated to the camera, and N is also received from the camera when the image was captured.

[0033] One or more dimensions of the object are quantified at 41 as a function of the hyperfocal distance. In the example, the proper object sizing (real object size - ROS) is computed using the triangle similarity theorem. For example, the object size is computed as follow: ROS = H ∗ SOS f where SOS is the object size in the sensor, H is the hyperfocal distance and f is the focal distance.

[0034] Lastly, the size of the object is presented at step 42, for example on a display of the system. For unknown objects, two dimensions are determined from the image: height of the object and width of the object.

[0035] In the case the object is known, the pixels that contain the object are isolated at step 43. The process is performed in the complete set of frames. In the example embodiment, the segmentation algorithm is based on a neural network that segment the object from the background, and the output is the area of pixels that contains the object. For example, the segmentation is performed using a U-Net method as described by O. Ronnenberger et al in "Convolutional Networks for Biomedical Image Segmentation", Medical Image Computing and Computer-Assisted Invention, 2015, Cham, 2015 which is incorporated in its entirety herein. The U-Net method is pre-trained with the ImageNet dataset and refined with the known object dataset. It is applied over that frame to obtain the segmentation, however others algorithms that perform segmentation can be applied. The U-Net returns masks with the segmentation of frames. The masks are applied over the frames and the object of interest is isolated in all the frames of the frame set.

[0036] Next, the method identifies a given image, from the set of isolated areas, having the lowest f-number of those images considered "in focus" based on the focus metric threshold, at step 42. This step is the same as described in relation to step 38. Likewise, a hyperfocal distance (H) is calculated at step 45 using the parameters of the camera in the same manner as described in relation to step 39.

[0037] In the example embodiment, view classification of the object is executed at step 47. A neural network designed specifically for this task performs the object view classification. This network presents a VGG16 architecture with similar parameters to those described by K. Simonyan et al in "Very Deep Convolutional Networks For Large-Scale Image Recognition" (2014) although other algorithms designed for the same task or for object pose estimation can be used. The neural network was trained with a specific dataset composed of 108 views of each of the known objects. Although the number of views can vary, this number of views allows to sample the object every 10 degrees in the superior, central and inferior position (see FIG. 4) covering the object completely.

[0038] A three-dimensional reconstruction of the object is carried out in step 48. In one embodiment, a 3D reconstruction of the object is carried out by a neural network ResNet-34 refined to identify the 3D reconstruction (see FIG. 5) of the object and produce the spatial information based on the pose information obtained in the previous step. The neural network is trained with a dataset including 108 views for all the known objects and the 3D reconstruction models for each object. Once the 3D model is retrieved (meaning a 3D model of a known object has been identified), the 3D reconstruction is adjusted to the view of the object that has been isolated in the image selected by creating a correspondence of the points of the original object view contained in the dataset and the isolated object. For example, if the isolated area is classified as a middle-frontal view of a Rubik's Cube (see FIG. 5A), through this information the algorithm retrieves the respective 3D model of the Rubik's Cube, and adjust the dimensions to that of the object that has been isolated. In another example, the isolated area is classified as a middle-lateral view of a watering can (see FIG. 5C and 5D), the associated 3D model of the watering can is identified, and the water can dimensions are computed by adjusting them to that of the isolated object.

[0039] One or more dimensions of the object are then quantified at 49. In one embodiment, the object sizing is calculated in a similar way as in step 41. If some part of the object is occluded, the dimensions are predicted with the model and the known dimensions of the visible parts. For example, if the identified object is a Rubik's Cube and the view is a middle-frontal view, the depth dimension is not visible as can been seen in FIG. 5A. In that case, the depth is inferred by the model and the known visible dimensions. In the case of the middle-lateral watering can view, the occluded dimension is the width, the algorithm infers it. In the case of a known object, three dimensions of the object are quantified: height, width and depth of the object.

[0040] The unsupervised method for determining the size of an object is suitable for different applications. For example, the method can be used to automate the hand-luggage size control in an airport, where a camera in an unsupervised manner can automatically determine the size of hand luggage. In this case, the disclosure would facilitate a reduction of time in the boarding process.

[0041] Another application is to automatically measure the size of trucks to authorize or not the access of large vehicles to restricted roads that large vehicles cannot use.

[0042] A third example of application is the measurement of postal, UPS or Fedex packages to calculate automatically shipping costs.

[0043] The techniques described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium. The computer programs may also include stored data. Non-limiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.

[0044] Some portions of the above description present the techniques described herein in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, it has also proven convenient at times to refer to these arrangements of operations as modules or by functional names, without loss of generality.

[0045] Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0046] Certain aspects of the described techniques include process steps and instructions described herein in the form of an algorithm. It should be noted that the described process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

[0047] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a tangible computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Claims

1. An unsupervised method for determining size or dimension of an object captured in an image, comprising: capturing, by a camera (21), a triggering image of a scene, identifying, by a computer processor, presence of an object in the triggering image, capturing, by the camera (21), a set of images (22), each image in the set of images (22) is captured with the camera (21) set at a different f-number and the set of images (22) is captured in response to the identification of presence of the object in the image, wherein the method comprising furthermore identifying, by the computer processor, a given image in the set of images (22) by identifying a subset of images from the set of images (22), where each image in the subset of images has a contrast metric with a value greater than a threshold, and selecting the given image from the subset of images, where the given image has lowest f-number amongst the images in the subset of images, calculating, by the computer processor, a hyperfocal distance using parameters of the camera (21) when taking the given image, identifying, by the computer processor, the object in the triggering image, retrieving, by the computer processor, a three-dimensional model for the identified object, and quantifying, by the computer processor, three dimensions of the object using a function of the hyperfocal distance and the retrieved three-dimensional model.

2. The method of claim 1 further comprises identifying presence of an object in the image using background subtraction method.

3. The method of claim 1 further comprises for each image in the set of images, computing, by the computer processor, a quality metric for the image, where the quality metric is the ratio: contrast metric / f - number; and identifying, by the computer processor, a given image in the set of images (22) using the relation between the contrast metric and the f-number amongst the images in the set of images (22).

4. The method of claim 3 wherein computing a contrast metric further comprises computing fuzzy entropy.

5. The method of claim 1 further comprises quantifying one or more dimensions of the object using a triangle similarity theorem.

6. The method of claim 1 further comprises quantifying at least one dimension of the object with ROS = H * SOS / f where ROS is size of the object, H is hyperfocal distance, SOS is size of object in camera (21) sensor and f is focal distance.

7. The method of claim 1 further comprises identifying the object in the triggering image using a neural network, extracting a region of interest from each image in the set of images (22), where the region of interest contains the identified object; and for each image in the set of images (22), computing the contrast metric for the image using only the region of interest extracted from the image.

8. The method of claim 7 further comprises extracting a region of interest from each image in the set of images (22) using a neural network.

9. The method of claim 1 wherein quantifying three dimensions of the identified object further comprises determining a view of the object in the given image, determining whether one or more of the three dimensions is occluded in the given image, determining the one or more occluded dimensions using the three-dimensional model in response to the determining that one or more of the three dimensions is occluded in the given image, quantifying, from the given image, the real object size using the triangle similarity theorem and the three-dimensional model determined; and quantifying, from the given image, the real object size using ROS = H * SOS / f where ROS is real size of the object, H is hyperfocal, SOS is size of object in camera (21) sensor and f is focal distance.

10. A data processing system for determining the dimensions of an object captured in an image, comprising: a camera (21) configured to capture a triggering image of a scene, a computer processor configured to identify presence of an object in the triggering image an image preprocessor interfaced with the camera (21) that cooperatively interacts with the camera (21) to capture a set of images (22) of the scene, where each image in the set of images (22) is captured at a different f-number, an object detector (24) configured to receive the set of images (22) and identify presence of an object in the image, wherein a computer processor configured to identifying a given image in the set of images (22) by identifying a subset of images from the set of images (22), where each image in the subset of images has a contrast metric with a value greater than a threshold, and selecting the given image from the subset of images, where the given image has lowest f-number amongst the images in the subset of images, the computer processor configured to calculate a hyperfocal distance using parameters of the camera (21) when taking the given image, the computer processor configured to identify the object in the triggering image, the computer processor configured to retrieve a three-dimensional model for the identified object and quantifying three dimensions of the object using a function of the hyperfocal distance and the retrieved three-dimensional model.

11. The system of claim 10 wherein the object detector (24) identifies presence of an object in the image using a background subtraction method.

12. The system of claim 10 wherein the contrast metric is further defined as fuzzy entropy.

13. The system of claim 10 wherein an object size calculator (25) quantifies one or more dimensions of the object using a triangle similarity theorem.