Determining real world size (s) of three-dimensional space

Abstract

A method (1000) of determining a dimension value indicative of a physical dimension of a three-dimensional (3D) space is provided. The method (1000) includes obtaining (s1002) a first image, where the first image is generated using a first lens of a camera, identifying (s1004) a first set of one or more keypoints included in the first image, and obtaining (s1006) a second image, where the second image is generated using a second lens of the camera. The method (1000) further includes identifying (s1008) a second set of one or more keypoints included in the second image, and determining (s1010) a set of one or more 3D points associated with the first set of keypoints and the second set of keypoints, where the set of one or more 3D points includes the first 3D point. The method (1000) further includes calculating (s1012) a first distance value indicative of a distance between the camera and a real-world point corresponding to the first 3D point, and determining (s1014) a dimension value based at least on the calculated first distance value.

Description

Technical Field

[0001] Embodiments related to methods and apparatus for determining real-world dimension(s) (also referred to as physical dimension(s)) of a three-dimensional (3D) space are disclosed. Background Art

[0002] Nowadays, 3D reconstruction of space is widely used in various fields. For example, for home renovation, one or more 360-degree cameras can be used to capture multiple photos of the kitchen to be renovated, and the kitchen can be reconstructed in a 3D virtual space using the captured multiple images. The generated 3D reconstruction of the kitchen can be displayed on a screen and manipulated by the user to help the user visualize how to renovate the kitchen. Summary of the invention

[0003] However, there are certain challenges. For example, it is generally not possible to determine the real-world dimension(s) of the reconstructed 3D space using a 360 camera alone. Multiple photos from a 360 camera(s) can be used to estimate the scene geometry of the reconstructed 3D space, but the dimensions of the reconstructed 3D space measured by the camera(s) may be in arbitrary scale. Knowing only the dimension(s) in arbitrary scale (also referred to as "relative dimension(s)") may prevent the estimated scene geometry from being used for measurement purposes, and may complicate the comparison and embedding of multiple independent reconstructions. Therefore, a way is needed to measure the real-world dimension(s) of the 3D space (also referred to as "absolute dimension(s)").

[0004] Therefore, in one aspect of some embodiments of the present disclosure, a method for determining a size value indicating a physical size of a three-dimensional 3D space is provided. The method includes: obtaining a first image, wherein the first image is generated using a first lens of a camera, identifying a first group of one or more key points included in the first image, and obtaining a second image, wherein the second image is generated using a second lens of the camera. The method also includes: identifying a second group of one or more key points included in the second image, and determining a group of one or more 3D points associated with the first group of key points and the second group of key points, wherein the group of one or more 3D points includes a first 3D point. The method also includes: calculating a first distance value, the first distance value indicating the distance between the camera and a real-world point corresponding to the first 3D point, and determining a size value based at least on the calculated first distance value.

[0005] In another aspect, there is provided a computer program comprising instructions which, when executed by a processing circuit, cause the processing circuit to perform a method according to any one of the above embodiments.

[0006] In different aspects, a device for determining a size value indicating a physical size of a three-dimensional 3D space is provided. The device is configured to obtain a first image, wherein the first image is generated using a first lens of a camera, identify a first group of one or more key points included in the first image, and obtain a second image, wherein the second image is generated using a second lens of the camera. The device is also configured to identify a second group of one or more key points included in the second image, and determine a group of one or more 3D points associated with the first group of key points and the second group of key points, wherein the group of one or more 3D points includes a first 3D point. The device is also configured to calculate a first distance value indicating a distance between the camera and a real-world point corresponding to the first 3D point, and determine a size value based at least on the calculated first distance value.

[0007] In a different aspect, there is provided a device comprising a processing circuit and a memory containing instructions executable by the processing circuit, whereby the device is operable to perform the method of any one of the above embodiments.

[0008] Embodiments of the present disclosure allow for determination of real-world dimension(s) of a reconstructed 3D space without directly measuring the real-world dimension(s) using a depth sensor such as a Light Detection and Ranging (LiDAR) sensor, a stereo camera, or a laser rangefinder.

[0009] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate various embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An exemplary situation in which an embodiment of the present disclosure is implemented is shown.

[0011] Figure 2 A diagram showing an exemplary real-world environment.

[0012] Figure 3 An exemplary reconstructed 3D space is shown.

[0013] Figure 4 A process according to some embodiments is shown.

[0014] Figure 5A and 5B Key points according to some embodiments are shown.

[0015] Figure 6 3D points are shown in accordance with some embodiments.

[0016] Figure 7 A method of aligning two different rotation spaces is shown.

[0017] Fig. 8AA direction vector is shown in accordance with some embodiments.

[0018] Figure 8B A method of determining the physical size of a reconstructed 3D space according to some embodiments is shown.

[0019] Fig. 9 Shown is a top view of a 360 degree camera in accordance with some embodiments.

[0020] Fig.10 A process according to some embodiments is shown.

[0021] Fig.11 An apparatus according to some embodiments is shown. DETAILED DESCRIPTION

[0022] Figure 1 An exemplary scenario 100 in which an embodiment of the present disclosure is implemented is shown. In scenario 100, a 360-degree camera (hereinafter referred to as a "360 camera") 102 is used to capture a 360-degree view of a kitchen 180. In kitchen 180, an oven 182, a refrigerator 184, a photo frame 186, and a wall clock 188 are placed. In the present disclosure, a 360 camera is defined as any camera capable of capturing a 360-degree view of a scene.

[0023] like Figure 1 As shown in , the oven 182 is placed against the first wall 190, the photo frame 186 is placed against the second wall 192, the refrigerator 184 is placed against the second wall 192 and the third wall 194, and the wall clock 188 is placed against the fourth wall 196. Figure 2 Shown from viewpoint 178 (at Figure 1 180 degree view of the kitchen as seen from the left).

[0024] The camera 102 may include a first fisheye lens 104 and a second fisheye lens 106 . Figure 1 The number of fisheye lenses shown in is provided for illustration purposes only and does not limit the embodiments of the present disclosure in any way.

[0025] like Figure 3 As shown in , the captured 360-degree view of the kitchen 180 can be at least partially displayed on a display 304 (e.g., a liquid crystal display, an organic light emitting diode display, etc.) of an electronic device 302 (e.g., a tablet computer, a mobile phone, a laptop computer, etc.). Note that although Figure 3 Only a partial view of the kitchen 180 is shown on the display 304, but in some embodiments, the entire 360-degree view of the kitchen 180 may be displayed. For simplicity purposes, the curvature of the 360-degree view is also not shown. Figure 3 Shown in.

[0026] In some cases, it may be desirable to display the real-world length of the virtual dimension (e.g., “L0”) on the display 304 (note that Figure 3 is the length of the dimension in arbitrary scale). For example, to help a user determine whether a particular kitchen sink will fit in the space between first wall 190 and the left side of refrigerator 184, it may be desirable to show the real-world length of virtual dimension L0 on display 304. However, as described above, the real-world length of the dimension of the reconstructed 3D space cannot be measured or determined solely by the 360 ​​camera(s).

[0027] Therefore, in some embodiments of the present disclosure, executing Figure 4 4. The process 400 shown in 402 may be performed to determine the real-world dimension(s) of a reconstructed 3D space (eg, kitchen 180). The process 400 may start at step s402.

[0028] Step s402 includes: using the first fisheye lens 104 and the second fisheye lens 106 to capture the real world environment (also referred to as a "scene"), thereby obtaining a first fisheye image I F1 and the second fisheye image I F2 As described above, the number of cameras used to capture the real-world environment is not limited to two, but can be any number. Similarly, the number of fisheye images captured by camera 102 and / or the number of fisheye lenses included in camera 102 can be any number.

[0029] Step s404 comprises: using a set (T) of one or more lens distortion parameters to transform the first and second fisheye images I F1 and I F2 More specifically, in step s404, the first fisheye image I is transformed using the group T F1 Transformed into a first undistorted image, for example, a first equidistant image I FC1 , and use the group T to transform the second fisheye image I F2 Transformed into a second undistorted image, for example, a second equidistant image I FC2 Isometric image is a well-known term in the field of computer vision and thus is not explained in detail in this disclosure.

[0030] A description of the isometric image can be found at the following link: https: / / wiki.panotools.org / Fisheye_Projection.

[0031] Step s406 includes: converting the first undistorted image (eg, the first equidistant image I FC1 ) is transformed into the first equirectangular image I EQN1 , and the second undistorted image (eg, the second equidistant image IFC2 ) is transformed into a second equirectangular image I EQN2 An equirectangular image is an image with a specific format for 360-degree imaging, where the position of each pixel of the image reflects the longitude and latitude (also called azimuth and inclination) angle relative to a reference point. By definition, an equirectangular image covers 360 degrees (horizontally) by 180 degrees (vertically) in a single image. Like isometric images, equirectangular images are well-known terms in the field of computer vision and are therefore not explained in detail in this disclosure.

[0032] In some embodiments, instead of converting the first and second fisheye images into equirectangular images, the first and second fisheye images may be converted into perspective images. PE Alternatively, the equirectangular image obtained from the first and second fisheye images may be converted into a perspective image. PE It is like a "normal camera image" where straight lines in recorded space remain straight in the image. However, this also means that I PE You simply can’t cover more than 179.9 degrees in a single image, otherwise there will be severe distortion and the basic rule that straight lines must remain straight will be violated. In a 360-degree camera, I PE Usually not a “standard” result, however many computer vision solutions are designed for “standard” cameras and therefore work well with I PE Type of image.

[0033] On the contrary, I EQN is an "equirectangular image", a specific format used for 360-degree imaging where the position of each pixel actually reflects the longitude and latitude angle. By definition, I EQN Covers 360 degrees (horizontally) by 180 degrees (vertically) in a single image. In some 360-degree cameras, the I EQN is the "default" output format. EQN Can be converted into a set of several I PE , for example, using a cube map layout (as described in https: / / en.wikipedia.org / wiki / Cube_mapping).

[0034] Return to reference Figure 4 Step s408 comprises: from the first equirectangular image I EQN1 Identify the first set of key points (K1) and from the second equirectangular image I EQN2 In the present disclosure, key points are defined as points in a two-dimensional (2D) image plane that can help identify the geometry of a scene or the geometry of an object included in the scene. The key points correspond to points in at least one image (e.g., a first equirectangular image IEQN1 ) captured in real-world points.

[0035] Figure 5A The first equirectangular image I is shown as EQN1 Examples of key points in , and Figure 5B The second equirectangular image I is shown as EQN2 Examples of key points in . Note that for simplicity of illustration, Figure 5A and 5B In the figure, not all parts of the real environment captured in the first and second equirectangular images are shown, and the curvature of the lines included in the figure is omitted.

[0036] like Figure 5A As shown in FIG. 1 , the first equirectangular image I EQN1 A first set of one or more key points 502 (block circles shown in the figure) is included. Similarly, Figure 5B As shown in FIG. 1 , the second equirectangular image I EQN2 A second set of one or more key points 504 (block circles shown in the figure) is included. Figure 5A and 5B In the example of the first and second equirectangular images shown in , key points 502 and 504 identify the corners of oven 182, the corners of refrigerator 184, the corners of picture frame 186, and / or the corners of walls 190-196. Figure 5A and 5B As shown in , each of the key points 502 and 504 can be defined by pixel coordinates within each image. For example, if the lower left corner of each image is defined as the origin in the xy coordinate system, each of the key points 502 and 504 can be defined by pixel coordinates (x, y).

[0037] Return to reference Figure 4 After executing step s408 , execute step s410 . Step s410 includes: identifying a first group of matching key points ( K1*) 512 from the first group of key points ( K1 ) 502 , and identifying a second group of matching key points ( K2*) 514 from the second group of key points ( K2 ) 504 .

[0038] In the present disclosure, a matching keypoint is one of the keypoints identified in step s408 and is defined as a point in a two-dimensional (2D) image plane that corresponds to a real-world point captured in at least two different images. In the present disclosure, a real-world point is any point in a real-world environment (e.g., kitchen 180) that corresponds to a point on a physical feature of the real-world environment itself (e.g., a corner) or a physical feature of an object included in the real environment (e.g., a housing). For example, in Figure 5A and 5B In the embodiment, the four corners of the photo frame 186 are in the first and second equirectangular images IEQN1 and I EQN2 Therefore, the key points 502 and 504 corresponding to the four corners of the photo frame 186 are matched key points 512 and 514.

[0039] Similarly, since the two left corners of the upper door of the refrigerator 184 are in the first and second equirectangular images I EQN1 and I EQN2 Both are captured, so the key points 502 and 504 corresponding to the two left corners of the upper door of the refrigerator 184 are matching key points 512 and 514. Therefore, the key points 502 and 504 corresponding to the two left corners of the upper door of the refrigerator 184 are matching key points 512 and 514.

[0040] In summary, matching keypoints are keypoints that correspond to real-world points "observed" from multiple images captured, where the multiple images are captured from the same camera position.

[0041] Step s412 includes identifying a set of three-dimensional (3D) points (also referred to as a "sparse point cloud") corresponding to each set of key points described with respect to step s408. In the present disclosure, a 3D point is defined as a point in a 3D virtual space that corresponds to a key point described with respect to step s408. Here, the 3D point and the key point to which the 3D point corresponds identify the same real-world point.

[0042] Figure 6 3D points are shown. As shown in the figure, 3D point 602 corresponds to the same real-world point that key points 502 and 504 correspond to. More specifically, like key points 502 and 504, 3D point 602 identifies the corner of oven 182, the corner of refrigerator 184, the corner of picture frame 186, and / or the corner of walls 190-196. The main difference between key points and 3D points is that they are defined in different coordinate systems. While key points are defined on the image plane in a 2D coordinate system, 3D points are defined in virtual space in a 3D coordinate system. Therefore, as Figure 6 As shown in , the origin of the 3D coordinate system defining the 3D point is a point in the 3D virtual space. An example of the origin of the 3D coordinate system is a position in the virtual space corresponding to the real world position where the camera 102 is located when capturing the real environment.

[0043] Return to reference Figure 4, after executing step s412, step s414 may be executed. Step s414 includes: selecting a set of matching 3D points from the 3D points identified in step s412. In the present disclosure, a matching 3D point is one of the 3D points identified in step s412 and is defined as a point in the 3D virtual space that corresponds to a real-world point captured in the two different images. In summary, a 3D point (e.g., X1) is a 3D version of a key point (e.g., K1), and a matching 3D point (e.g., X1*) is a 3D version of a matching key point (e.g., K1*).

[0044] although Figure 4 Steps s408, s410, s412 and s414 are shown to be performed sequentially, but in some embodiments, these steps may be performed in a different order. In addition, in other embodiments, at least some of these steps may be performed simultaneously.

[0045] In some embodiments, steps s408, s410, s412 and s414 may be performed by running a Structure from Motion (SfM) technique on the equirectangular image obtained in step s406, such as COLMAP (described in https: / / colmap.github.io) and OpenMVG (described in https: / / github.com / openMVG / openMVG).

[0046] COLMAP only works with perspective images (also called "normal camera images"), so if you want to use COLMAP, I FC Need to be converted to I PE , and since OpenMVG is suitable for equirectangular images, if you want to use OpenMVG, I FC Instead, it needs to be converted to I EQN The decision to use COLMAP or OpenMVG can be based on various factors, such as performance, cost, accuracy, licensing, and preference.

[0047] Using the SfM technique, in addition to K, K*, X, and X*, additional data such as camera pose can be obtained.

[0048] Return to reference Figure 4 After identifying the first and second groups of matching key points (K1*, K2*) 512 and 514, step s416 may be performed. Step s416 includes: EQN1 and I EQN2104 and 106. In particular, since the first and second lenses 104 and 106 of the camera 102 point in different directions, the first and second equirectangular images I EQN1 and I EQN2 in a different rotational space, and therefore step s412 is required.

[0049] The first and second equirectangular images I EQN1 and I EQN2 One way to place it in the same rotation space is to place the second equirectangular image I EQN2 (or the first equirectangular image I EQN1 ) is placed on the first equirectangular image I EQN1 (or the second equirectangular image I EQN2 ). For example, Figure 7 The top three figures show the rotation space of the first equirectangular image, while Figure 7 The three figures on the bottom left of show the rotation space of the second equirectangular image. Figure 7 As shown in the bottom rightmost figure of , one way to place the images into the same rotation space is by changing the rotation space of the second equirectangular image so that the two images are in the same rotation space. More specifically, in one example, in step s416, the axes of the second 3D rotation space can be rotated to align with the axes of the first 3D rotation space so that the axes of the first and second 3D rotation spaces are now aligned.

[0050] Step s418 includes calculating a first direction vector (eg, Fig. 8A The V FC1 _X 802 ) and a second direction vector (e.g., V shown in FIG. 8 ) from the reference point of the second lens 106 to the second matching key point K2* (e.g., 862 as one of the matching key points 514). FC2 _X 802 As explained above, the first matching keypoint K1* (eg, 852) and the second matching keypoint K2* (eg, 862) correspond to the same real-world point (eg, Figure 1 ).

[0051] Step s420 includes performing triangulation in 3D space using the first and second direction vectors to identify real-world points corresponding to the first matching keypoint 852 and the second matching keypoint 862. In the present disclosure, triangulation is a mathematical operation for finding the intersection of two rays (e.g., vectors). In the field of computer vision, triangulation is a well-understood concept, and therefore a detailed explanation of how to perform triangulation is omitted in the present disclosure.

[0052] Fig. 8A FIG. 4 shows how step s420 may be performed. Fig. 8A In the example, the first direction vector V is determined FC1 _X 802 and the second direction vector V FC2 _X 802 The first direction vector V FC1 _X 802 is a vector from the first lens 104 toward the first matching key point 852, and the second direction vector V FC2 _X 802 is a vector from the second lens 106 toward the second matching key point 862. Fig. 8A As shown in , the first matching key point 852 and the second matching key point 862 correspond to the same real-world physical point (ie, the upper right corner of the photo frame 186).

[0053] Through step s420, the first direction vector V is determined FC1 _X 802 and the second direction vector V FC2 _X 802 The intersection of Fig. 8A , the intersection corresponds to point 802. Here, point 802 corresponds to a real-world physical location (eg, the physical location of the upper right corner of photo frame 186), which corresponds to the first and second matching key points 852 and 862.

[0054] Return to reference Figure 4 After finding the intersection (ie, point 802), step s422 is performed. Step s422 includes: calculating a first distance between the first lens 104 and point 802 (eg, Figure 8B ) and a second distance between the second lens 106 and the point 802 (eg, Figure 8B D2 shown in FIG.

[0055] Step s424 includes calculating the camera center (eg, Fig. 9 ) and the actual physical distance between the real-world physical point 802 (e.g., Fig. 9Any known mathematical operation can be used to calculate D0 using the first and second distances.

[0056] Since the SfM technique used to generate the 3D points X, X* generates an estimate of P (for each capture position) relative to some arbitrarily chosen 0 point (i.e., the center of the coordinate system), there is an offset between the position of the camera 102 and the arbitrarily chosen 0 point. To correct the scale correctly, the offset needs to be removed by re-centering the coordinate system around the camera position.

[0057] Therefore, step s426 may be performed. Step s426 includes: moving the origin of the coordinate system of the 3D reconstruction space from the reference point 650 to the position P of the camera 102 (eg Figure 1 ), will match 3D points (e.g., Figure 6 The initial coordinates X* (x*, y*, z*) of each of the coordinates 602) shown in FIG. 6 are converted into the corrected coordinates X* o (x* o , y* o , z* o For example, if the initial coordinates of the matching 3D point 602 in the coordinate system with the reference point 650 as the origin are (x*, y*, z*), then the transformed coordinates of the matching 3D point 602 in the coordinate system with the position P as the origin are (x* o , y* o , z* o ).

[0058] In determining the corrected coordinates X* of each of the matching 3D points 602 o (x* o , y* o , z* o ) Then, in step s428, the corrected coordinates X* of each of the matching 3D points are calculated. o (x* o , y* o , z* o ) and the new origin of the coordinate system (ie, the position P of the camera 102) is the distance D0 (X* o ). In other words, D0(X* o ) can be equal to or can be based on

[0059] Step s430 includes calculating a local scale factor S*, which indicates the ratio of the (one or more) virtual dimensions of the reconstructed 3D space to the (one or more) real world dimensions of the reconstructed 3D space. In some embodiments, the local scale factor S* may be based on For example,

[0060] As mentioned above, there is more than one matching 3D point (e.g. Figure 6 602 shown in FIG. 1 ). For example, Figure 6 As shown in , there are at least six matching 3D points of the photo frame 186. Therefore, in some embodiments, steps s418-s430 may be calculated for each of the plurality of matching 3D points. In those embodiments, as a result of performing step s430, a plurality of local scale factors S* may be obtained.

[0061] In such an embodiment, a step s432 may be provided. In step s432, a single overall scale factor (also referred to as an "absolute scale factor" or "global scale factor") for all matching 3D points may be calculated based on the obtained multiple local scale factors. For example, the absolute scale factor may be calculated based on an average of the multiple local scale factors, or the absolute scale factor may be equal to the average of the multiple local scale factors. The average may be a non-weighted average (where Where N is the local scale factor S i *) or weighted average.

[0062] If the average is a weighted average of multiple local scale factors, the weight of each local scale factor associated with each 3D point can be determined based on the confidence value of each 3D point. This confidence value for each 3D point can be generated by SfM techniques and can indicate or estimate the level of certainty that the 3D point is in the correct position in the 3D virtual space.

[0063] In one embodiment, the confidence value of a 3D point may indicate the number of key points that identify the 3D point. Figure 1 180 is captured by the camera 102 at two different locations, then two different images are generated - a first image with a first set of key points and a second image with a second set of key points. Let us assume a situation where (i) the first set of key points includes a first key point identifying the upper left corner of the photo frame 186 and a second key point identifying the lower right corner of the refrigerator 184, and (ii) the second set of key points includes a third key point identifying the upper left corner of the photo frame 186, but does not include any key point identifying the lower right corner of the refrigerator 184. In this case, because the upper left corner of the photo frame 186 is identified by two key points, while the lower right corner of the refrigerator 184 is identified by only one key point, the confidence value of the 3D point corresponding to the upper left corner of the photo frame 186 will be higher than the confidence value of the 3D point corresponding to the lower left corner of the refrigerator 184.

[0064] The absolute scale factor S obtained in step s432 can be used to bring absolute scale to the 3D build space. For example, as described above, there may be a situation where the user wants to measure the real-world length L between the first wall 190 and the left side of the refrigerator 184 (e.g., Figure 1Using the absolute scale factor S, the length L can be calculated. For example, the length L can be equal to L0 / S or can be calculated based on L0 / S, where L0 (as shown in Figure 3 ) is the length between the first wall 190 and the left side of the refrigerator 184 in the 3D reconstructed space.

[0065] Fig.10 A process 1000 for determining a dimension value indicating a physical dimension of a three-dimensional 3D space is shown according to some embodiments. The process 1000 may start at step 1002. Step s1002 includes obtaining a first image, wherein the first image is generated using a first lens of a camera. Step s1004 includes identifying a first set of one or more key points included in the first image. Step s1006 includes obtaining a second image, wherein the second image is generated using a second lens of a camera. Step s1008 includes identifying a second set of one or more key points included in the second image. Step s1010 includes determining a set of one or more 3D points associated with the first set of key points and the second set of key points, wherein the set of one or more 3D points includes a first 3D point. Step s1012 includes calculating a first distance value indicating a distance between the camera and a real-world point corresponding to the first 3D point. Step s1014 includes determining a dimension value based at least on the calculated first distance value.

[0066] In some embodiments, generating a first image includes: capturing a first fisheye image using a first lens of a camera, and converting the captured first fisheye image into a first image; generating a second image includes: capturing a second fisheye image using a second lens of the camera, and converting the captured second fisheye image into a second image; and each of the first image and the second image is an equidistant image or an equirectangular image.

[0067] In some embodiments, the method further includes: identifying a first subset of one or more key points from the first set of key points, and identifying a second subset of one or more key points from the second set of key points, wherein a first key point included in the first subset matches a second key point included in the second subset, and the first 3D point is mapped to the first key point and the second key point.

[0068] In some embodiments, the method also includes: determining a first direction vector having a first direction, wherein the first direction is from a first reference point of a first lens of the camera to a key point included in the first set of key points; and determining a second direction vector having a second direction, wherein the second direction is from a second reference point of a second lens of the camera to a key point included in the second set of key points.

[0069] In some embodiments, the method further comprises performing a triangulation process using the first direction vector, the second direction vector, and a baseline between the first reference point and the second reference point to determine an intersection of the first direction vector and the second direction vector, wherein the real world point is the intersection.

[0070] In some embodiments, the method further includes calculating a second distance value indicating a distance between the real-world point and the first lens of the camera; and calculating a third distance value indicating a distance between the real-world point and the second lens of the camera, wherein the first distance value is calculated using the second distance value and the third distance value.

[0071] In some embodiments, the distance between the real-world point and the camera is the distance between the real-world point and a reference point in the camera, and the reference point is located between the position of the first lens and the position of the second lens.

[0072] In some embodiments, the method further includes: converting the original coordinates of the one or more 3D points into transformed coordinates, wherein the original coordinates are in a first coordinate system, the transformed coordinates are in a second coordinate system, the center of the first coordinate system is not the reference point of the camera, and the center of the second coordinate system is the reference point of the camera.

[0073] In some embodiments, the transformed coordinates of the one or more 3D points include first transformed coordinates of a first 3D point, and the method further comprises: calculating a first reference distance value indicating a distance between a reference point of the camera and the first transformed coordinates of the first 3D point.

[0074] In some embodiments, the method further comprises determining a scale factor value based on a ratio of the first distance value and the first reference distance value, wherein the size value is determined based on the scale factor value.

[0075] In some embodiments, the method further includes: i) calculating a distance value indicating a distance between a reference point of the camera and a real-world point mapped to each of the original coordinates of the one or more 3D points; ii) calculating a reference distance value indicating a distance between the reference point of the camera and each of the transformed coordinates of the one or more 3D points; iii) determining a scale factor value for each of the one or more 3D points based on a ratio of the distance value obtained in step i) and the reference distance value obtained in step ii); and iv) calculating an average of the scale factors of the one or more 3D points.

[0076] In some embodiments, the size value is determined based on an average of the scale factors.

[0077] In some embodiments, the method further comprises: displaying at least a portion of the 3D space, wherein the indicator indicates the physical size.

[0078] Fig.11 It is shown that the process 400 can be performed Figure 4 All steps or processes included in 400 (shown in Figure 4 Device 1100 for performing at least some of the steps included in the steps (shown in FIG. 1 ). Device 1100 may be any computing device. Examples of device 1100 include, but are not limited to, a server, a laptop, a desktop, a tablet computer, a mobile phone, etc. Fig.11 As shown in , the device may include: processing circuit (PC) 1102, which may include one or more processors (P) 1155 (e.g., one or more general-purpose microprocessors and / or one or more other processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and the like); communication circuit 1148, which is coupled to an antenna arrangement 1149 including one or more antennas, and which includes a transmitter (Tx) 1145 and a receiver (Rx) 1147 for enabling the device to transmit data and receive data (e.g., wirelessly transmit / receive data); and a local storage unit (also referred to as a "data storage system") 1108, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In some embodiments, the device may not include the antenna arrangement 1149, but may instead include a connection arrangement required to send and / or receive data using a wired connection. In embodiments in which the PC 1102 includes a programmable processor, a computer program product (CPP) 1141 may be provided. The CPP 1141 includes a computer readable medium (CRM) 1142 storing a computer program (CP) 1143 including computer readable instructions (CRI) 1144. The CRM 1142 may be a non-transitory computer readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a memory device (e.g., a random access memory, a flash memory), and the like. In some embodiments, the CRI 1144 of the computer program 1143 is configured such that, when executed by the PC 1102, the CRI causes the device to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, the device may be configured to perform the steps described herein without requiring code. That is, for example, the PC 1102 may consist solely of one or more ASICs. Thus, the features of the embodiments described herein may be implemented in hardware and / or software.

Claims

1. A method (1000) for determining a dimension value indicating a physical dimension of a three-dimensional (3D) space, the method comprising: obtaining (s1002) a first image, wherein the first image is generated using a first lens of a camera; identifying (s1004) a first set of one or more key points included in said first image; obtaining (s1006) a second image, wherein the second image is generated using a second lens of the camera; identifying (s1008) a second set of one or more key points included in the second image; determining (s1010) a set of one or more 3D points associated with the first set of keypoints and the second set of keypoints, wherein the set of one or more 3D points includes the first 3D point; calculating (s1012) a first distance value indicating a distance between the camera and a real-world point corresponding to the first 3D point; as well as Based at least on the calculated first distance value, the size value is determined (s1014).

2. The method according to claim 1, wherein Generating the first image includes: capturing a first fisheye image using the first lens of the camera, and converting the captured first fisheye image into the first image, Generating the second image includes: capturing a second fisheye image using the second lens of the camera, and converting the captured second fisheye image into the second image, and Each of the first image and the second image is an equidistant image or an equirectangular image.

3. The method according to claim 1 or 2, further comprising: identifying a first subset of one or more keypoints from the first set of keypoints; as well as A second subset of one or more keypoints is identified from the second set of keypoints, wherein A first keypoint included in the first subset matches a second keypoint included in the second subset, and The first 3D point is mapped to the first key point and the second key point.

4. The method according to any one of claims 1 to 3, comprising: determining a first direction vector having a first direction, wherein the first direction is from a first reference point of the first lens of the camera to a key point included in the first set of key points; as well as A second direction vector having a second direction is determined, wherein the second direction is from a second reference point of the second lens of the camera to a keypoint included in the second set of keypoints.

5. The method according to claim 4, comprising: A triangulation process is performed using the first direction vector, the second direction vector, and a baseline between the first reference point and the second reference point to determine an intersection point of the first direction vector and the second direction vector, wherein The real world point is the intersection point.

6. The method according to any one of claims 1 to 5, further comprising: calculating a second distance value indicating a distance between the real-world point and the first lens of the camera; as well as calculating a third distance value indicating a distance between the real-world point and the second lens of the camera, wherein The first distance value is calculated using the second distance value and the third distance value.

7. The method according to any one of claims 1 to 6, wherein The distance between the real-world point and the camera is the distance between the real-world point and a reference point in the camera, and The reference point is located between the position of the first lens and the position of the second lens.

8. The method according to any one of claims 1 to 7, further comprising: The original coordinates of the one or more 3D points are converted into transformed coordinates, wherein The original coordinates are in the first coordinate system, The transformed coordinates are in the second coordinate system, The center of the first coordinate system is not the reference point of the camera, and The center of the second coordinate system is the reference point of the camera.

9. The method according to claim 8, wherein The transformed coordinates of the one or more 3D points include first transformed coordinates of the first 3D point, and The method further comprises: A first reference distance value is calculated, the first reference distance value indicating a distance between the reference point of the camera and the first transformed coordinates of the first 3D point.

10. The method according to claim 9, further comprising: A scaling factor value is determined based on a ratio of the first distance value to the first reference distance value, wherein The size value is determined based on the scale factor value.

11. The method according to claim 10, comprising: i) calculating a distance value indicating a distance between the reference point of the camera and a real-world point mapped to each of the original coordinates of the one or more 3D points; ii) calculating a reference distance value indicating a distance between the reference point of the camera and each of the transformed coordinates of the one or more 3D points; iii) determining a scale factor value for each of the one or more 3D points based on a ratio of the distance value obtained in step i) and the reference distance value obtained in step ii); as well as iv) calculating an average of said scale factors for said one or more 3D points.

12. The method according to claim 11, wherein: The size value is determined based on the average value of the scale factors.

13. The method according to any one of claims 1 to 12, further comprising: At least a portion of the 3D space is displayed, wherein an indicator indicates the physical size.

14. A computer program (1143) comprising instructions (1144) which, when executed by a processing circuit (1102), cause the processing circuit to perform the method of any one of claims 1-13.

15. A carrier comprising the computer program of claim 14, wherein the carrier is one of the following: an electronic signal, an optical signal, a radio signal and a computer-readable storage medium.

16. A device (1100) for determining a dimension value indicating a physical dimension of a three-dimensional (3D) space, the device being configured to: obtaining (s1002) a first image, wherein the first image is generated using a first lens of a camera; identifying (s1004) a first set of one or more key points included in said first image; obtaining (s1006) a second image, wherein the second image is generated using a second lens of the camera; identifying (s1008) a second set of one or more key points included in the second image; determining (s1010) a set of one or more 3D points associated with the first set of keypoints and the second set of keypoints, wherein the set of one or more 3D points includes the first 3D point; calculating (s1012) a first distance value indicating a distance between the camera and a real-world point corresponding to the first 3D point; as well as Based at least on the calculated first distance value, the size value is determined (s1014).

17. The apparatus of claim 16, wherein Generating the first image includes: capturing a first fisheye image using the first lens of the camera, and converting the captured first fisheye image into the first image, Generating the second image includes: capturing a second fisheye image using the second lens of the camera, and converting the captured second fisheye image into the second image, and Each of the first image and the second image is an equidistant image or an equirectangular image.

18. The device according to claim 16 or 17, the device being configured to: identifying a first subset of one or more keypoints from the first set of keypoints; and A second subset of one or more keypoints is identified from the second set of keypoints, wherein A first keypoint included in the first subset matches a second keypoint included in the second subset, and The first 3D point is mapped to the first key point and the second key point.

19. The device according to any one of claims 16 to 18, wherein the device is configured to: determining a first direction vector having a first direction, wherein the first direction is from a first reference point of the first lens of the camera to a key point included in the first set of key points; and A second direction vector having a second direction is determined, wherein the second direction is from a second reference point of the second lens of the camera to a keypoint included in the second set of keypoints.

20. The device according to claim 19, the device being configured to: A triangulation process is performed using the first direction vector, the second direction vector, and a baseline between the first reference point and the second reference point to determine an intersection point of the first direction vector and the second direction vector, wherein The real world point is the intersection point.

21. The device according to any one of claims 16 to 20, wherein the device is configured to: calculating a second distance value indicating a distance between the real-world point and the first lens of the camera; and calculating a third distance value indicating a distance between the real-world point and the second lens of the camera, wherein The first distance value is calculated using the second distance value and the third distance value.

22. The apparatus according to any one of claims 16 to 21, wherein The distance between the real-world point and the camera is the distance between the real-world point and a reference point in the camera, and The reference point is located between the position of the first lens and the position of the second lens.

23. The device according to any one of claims 16 to 22, wherein the device is configured to: The original coordinates of the one or more 3D points are converted into transformed coordinates, wherein The original coordinates are in the first coordinate system, The transformed coordinates are in the second coordinate system, The center of the first coordinate system is not the reference point of the camera, and The center of the second coordinate system is the reference point of the camera.

24. The apparatus of claim 23, wherein The transformed coordinates of the one or more 3D points include first transformed coordinates of the first 3D point, and The device is further configured to calculate a first reference distance value indicating a distance between the reference point of the camera and the first transformed coordinates of the first 3D point.

25. The device according to claim 24, the device being configured to: A scaling factor value is determined based on a ratio of the first distance value to the first reference distance value, wherein The size value is determined based on the scale factor value.

26. The device according to claim 25, the device being configured to: i) calculating a distance value indicating a distance between the reference point of the camera and a real-world point mapped to each of the original coordinates of the one or more 3D points; ii) calculating a reference distance value indicating a distance between the reference point of the camera and each of the transformed coordinates of the one or more 3D points; iii) determining a scale factor value for each of the one or more 3D points based on a ratio of the distance value obtained in step i) and the reference distance value obtained in step ii); as well as iv) calculating an average of said scale factors for said one or more 3D points.

27. The apparatus of claim 26, wherein: The size value is determined based on the average value of the scale factors.

28. The device according to any one of claims 16-27, wherein the device is configured to: At least a portion of the 3D space is displayed, wherein an indicator indicates the physical size.

29. An apparatus (1100), comprising: Processing circuit (1102); and A memory (1141) containing instructions executable by the processing circuit, whereby the apparatus is operable to perform the method of any one of claims 1-13.

Images