Indoor space size calculation method and device, equipment and storage medium

Abstract

The invention discloses an indoor space size calculation method, device and equipment and a storage medium, and the method comprises the steps: obtaining an indoor image shot by a monocular camera and internal parameters of the monocular camera, calculating a depth map of the indoor image, determining a target region with a depth value greater than a target value from the depth map, and calculating the indoor space size of the target region; calculating a first dimension size of the indoor space based on the depth value of each pixel in the target area, constructing a circumscribed rectangle of the target area, determining image coordinates of opposite angles of the circumscribed rectangle, and determining the depth value of the indoor space based on the image coordinates of the opposite angles of the circumscribed rectangle, the internal parameters of the monocular camera and the first dimension size of the indoor space. And calculating a second dimension size and a third dimension size of the indoor space. According to the method, the three-dimensional size of the indoor space is obtained directly based on the indoor image shot by the monocular camera in combination with the image processing method and the visual geometry method, a depth sensor is not needed, and the hardware cost is reduced.

Description

Technical Field

[0001] The present invention relates to a three-dimensional spatial reconstruction technology, and in particular to an indoor space dimension calculation method, device, equipment and storage medium. Background Art

[0002] Obtaining the three-dimensional size of space is very important for augmented reality, virtual reality, and some tasks that require the use of the three-dimensional size of space, such as depth-based camera autofocus, spatial perception-based microphone array initial parameter estimation, etc. The image captured by a general color camera is a two-dimensional signal obtained by perspective projection of the three-dimensional space, so the three-dimensional size of the space cannot be directly obtained from it.

[0003] The current calculation scheme for the three-dimensional size of indoor spaces mainly uses ToF (Time of flight) sensors, structured light sensors and other depth sensors to generate spatial depth information, while using color cameras to generate color information. The above two pieces of information are calibrated and registered, and finally the depth information is back-projected based on the internal parameters of the camera to generate a point cloud of the three-dimensional structure of the space and ultimately determine the three-dimensional size of the space.

[0004] Existing technologies require specific depth sensors to determine the three-dimensional size of a space, which is relatively expensive. Summary of the invention

[0005] The present invention provides an indoor space dimension calculation method, device, equipment and storage medium, which do not require a depth sensor and reduce hardware costs.

[0006] In a first aspect, the present invention provides a method for calculating indoor space dimensions, comprising:

[0007] Acquire an indoor image captured by a monocular camera and internal parameters of the monocular camera;

[0008] Calculating a depth map of the indoor image;

[0009] Determine a target area whose depth value is greater than a target value from the depth map;

[0010] Calculating a first dimension of the indoor space based on a depth value of each pixel in the target area;

[0011] Constructing a bounding rectangle of the target area and determining the image coordinates of the diagonal corners of the bounding rectangle;

[0012] Based on the image coordinates of the diagonals of the circumscribed rectangle, the internal parameters of the monocular camera, and the first dimension of the indoor space, the second dimension and the third dimension of the indoor space are calculated.

[0013] In a second aspect, the present invention further provides a device for calculating indoor space dimensions, comprising:

[0014] An acquisition module, used to acquire an indoor image taken by a monocular camera and internal parameters of the monocular camera;

[0015] A depth map calculation module, used to calculate the depth map of the indoor image;

[0016] A target area determination module, used to determine a target area having a depth value greater than a target value from the depth map;

[0017] A first size calculation module, configured to calculate a first dimension of the indoor space based on a depth value of each pixel in the target area;

[0018] A circumscribed rectangle construction module, used to construct a circumscribed rectangle of the target area and determine the image coordinates of the diagonal corners of the circumscribed rectangle;

[0019] The second size calculation module is used to calculate the second dimension and the third dimension of the indoor space based on the image coordinates of the diagonals of the circumscribed rectangle, the internal parameters of the monocular camera and the first dimension of the indoor space.

[0020] In a third aspect, the present invention further provides an electronic device, comprising:

[0021] one or more processors;

[0022] A memory for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the indoor space size calculation method provided in the first aspect of the present invention.

[0024] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the indoor space dimension calculation method provided in the first aspect of the present invention.

[0025] The method for calculating the size of an indoor space provided by the present invention comprises: obtaining an indoor image taken by a monocular camera and internal parameters of the monocular camera, calculating a depth map of the indoor image, determining a target area whose depth value is greater than a target value from the depth map, calculating a first dimension of the indoor space based on the depth value of each pixel in the target area, constructing a circumscribed rectangle of the target area, and determining the image coordinates of the diagonal corners of the circumscribed rectangle, and calculating the second dimension and the third dimension of the indoor space based on the image coordinates of the diagonal corners of the circumscribed rectangle, the internal parameters of the monocular camera and the first dimension of the indoor space. The present invention directly obtains the three-dimensional size of the indoor space based on the indoor image taken by the monocular camera, and combines the image processing method and the visual geometry method, without the need for a depth sensor, thereby reducing the hardware cost.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flow chart of a method for calculating indoor space dimensions provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a process of determining a target area from an indoor image and constructing a bounding rectangle according to an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the structure of an indoor space dimension calculation device provided by an embodiment of the present invention;

[0031] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0032] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Figure 1 This is a flow chart of a method for calculating indoor space size provided by an embodiment of the present invention. This embodiment is applicable to the case where indoor three-dimensional space size is calculated based on indoor images taken by a monocular camera. The method can be executed by an indoor space size calculation device provided by an embodiment of the present invention. The device can be implemented by software and / or hardware, and is usually configured in an electronic device, such as Figure 1 As shown, the indoor space size calculation method includes the following steps:

[0036] S101. Acquire an indoor image captured by a monocular camera and internal parameters of the monocular camera.

[0037] A monocular camera is a camera with only one lens. It converts the three-dimensional world into two dimensions. It leaves a projection of the captured scene on the imaging plane of the camera. It is impossible to determine the depth information through a monocular camera in a static state. The internal parameters of a monocular camera may include (fx, fy, cx, cy), where fx is the pixel representation of the focal length in the x direction in the image coordinate system, fy is the pixel representation of the focal length in the y direction in the image coordinate system, cx is the offset of the optical center in the x direction in the image coordinate system, and cy is the offset of the optical center in the y direction in the image coordinate system.

[0038] In an embodiment of the present invention, an indoor image of an indoor space captured by a monocular camera is obtained, and internal parameters of the monocular camera are obtained. Exemplarily, the electronic device can be connected to the monocular camera to read the indoor image captured and stored by the monocular camera and the internal parameters of the monocular camera.

[0039] S102: Calculate a depth map of the indoor image.

[0040] After acquiring the indoor image taken by the monocular camera, a preset algorithm is used to calculate the depth map of the indoor image. A depth map is an image or image channel containing information about the distance to the surface of the scene object from the viewpoint. Among them, the depth map is similar to a grayscale image, except that each pixel value (that is, the depth value) is the actual distance from the monocular camera to the object. The larger the pixel value, the greater the actual distance from the monocular camera to the object. Usually, the depth map of the indoor image is registered, so there is a one-to-one correspondence between the pixels of the indoor image and the depth map.

[0041] In some embodiments of the present invention, a monocular depth cue method can be used to calculate the depth map of an indoor image. For example, a method based on light and shade and illumination, which is based on the light and shade information in the image, uses a reflected illumination model to reconstruct the three-dimensional shape by restoring the normal vector of the object surface. Under ideal conditions, this algorithm can restore a more accurate three-dimensional model. For another example, a focusing method is used to restore the depth information of an object by analyzing the relationship between the focal length aperture of a monocular camera and the image clarity, and then obtain a three-dimensional model. The principle is that a clear image will only be projected when the object is at the focal length of the monocular camera. Therefore, by establishing a relationship between the distance from the object to the projection center and the image clarity, the depth information can be restored. The focusing method can use a small amount of images to calculate the dense depth information of the object surface, and the requirements for light source conditions are relatively loose, and the reconstruction effect is relatively fine.

[0042] In other embodiments of the present invention, the depth map of the indoor image can be calculated based on a machine learning algorithm. This type of algorithm mainly uses a machine learning method to input a large number of representative training image sets and corresponding depth sets into a defined model for supervised learning. After the training is completed, the actual input image is input into the trained model for depth calculation. The machine learning algorithm can also include parameter learning method and non-parameter learning method. The parameter learning method refers to a method containing unknown parameters in the energy function. The training process is to solve these parameters. This method requires artificial assumptions that the relationship between RGB images and depth satisfies a certain parameter model, and the assumed model is difficult to simulate the mapping relationship of the real world. For example, in the framework of maximizing the posterior probability, superpixels are used as units, and the mapping relationship between input image features and output depth is learned using Markov random fields, thereby realizing the estimation of depth. The non-parameter learning method is a data-driven algorithm that uses an existing data set to perform similarity retrieval to infer depth. It does not require parameters obtained by learning. Given a test image, this type of method obtains the depth of the test image by fusing the depth of images similar to its three-dimensional scene content in the database.

[0043] In other embodiments of the present invention, the depth map of the indoor image can be calculated based on a deep learning algorithm. The deep learning algorithm includes a supervised deep learning method and an unsupervised deep learning method. The supervised deep learning method needs to rely on the real depth when training the model, and also needs to rely on a large amount of data to train the network model. The data set generally includes a monocular image and the corresponding depth truth value. The unsupervised deep learning method reconstructs the corresponding image according to the geometric relationship between the image pairs, thereby supervising the training of the network through the image reconstruction loss, and does not rely on the depth truth value during network training.

[0044] S103: Determine a target area whose depth value is greater than a target value from the depth map.

[0045] After calculating the depth map of the indoor image, determine the target area whose depth value is greater than the target value from the depth map. The target value is used to characterize the depth value of the wall farthest from the monocular camera. In an embodiment of the present invention, the target value may be set in advance based on prior knowledge, or may be determined based on the statistical results of the depth value of each pixel in the depth map, which is not limited in the embodiment of the present invention.

[0046] In some embodiments of the present invention, the target value is determined based on the statistical results of the depth value of each pixel in the depth map. Exemplarily, the above step S103 includes the following sub-steps:

[0047] S1031 . Count the number of pixels corresponding to each depth value in the depth map.

[0048] In an embodiment of the present invention, the number of pixels corresponding to each depth value in the depth map is counted to obtain a set of discrete data. In order to avoid counting pixels with a depth value of zero, in an embodiment of the present invention, pixels with a depth value of zero are not counted, that is, when a pixel with a depth value of zero is encountered, the pixel is ignored. Pixels with a depth value of zero usually correspond to objects that cannot be recognized in the image, but their actual depth value is not zero. In order to avoid the influence of such pixels, the embodiment of the present invention does not count pixels with a depth value of zero, thereby improving the accuracy of subsequent target areas and further improving the accuracy of indoor space dimensions.

[0049] S1032: Calculate a first quantile based on the number of pixels corresponding to each depth value.

[0050] After counting the number of pixels corresponding to each depth value in the depth map and obtaining a set of discrete data, calculate the first quantile of the set of data. Exemplarily, in an embodiment of the present invention, the first quantile is the 92% quantile. Quantile refers to a point in a continuous distribution function, and the probability of this point corresponding to the quantile is the quantile. Exemplarily, in an embodiment of the present invention, the 92% quantile is the depth value corresponding to the probability value (quantile) of 92% in the continuous distribution function of the number of pixels corresponding to each depth value.

[0051] In some embodiments of the present invention, after counting the number of pixels corresponding to each depth value in the depth map to obtain a set of discrete data, the cumulative distribution function of the number of pixels corresponding to each depth value is calculated, and the value when the cumulative distribution function is greater than the first quantile (exemplarily, 92%) is taken as the first quantile (92% quantile).

[0052] Exemplarily, in an embodiment of the present invention, the depth map is processed based on the cross-platform computer vision library (OpenCV), the depth map and the set quantile (for example, 92%) are received, and OpenCV outputs the corresponding quantile (for example, 92%). The input depth map is a floating-point matrix, so it needs to be converted into an unsigned 16-bit integer format. This step is implemented by calling the convertTo function in OpenCV. Next, the number of pixels that appear for each depth value (that is, pixel value) in the depth map is counted, and these statistical results are saved in an array of length 65536. In order to avoid counting pixels with a depth value of zero, an if conditional judgment can be added to ignore pixels with a depth value of zero. Then, the cumulative distribution function of the array is calculated, and the value when the cumulative distribution is greater than the set quantile (for example, 92%) is obtained, and it is returned as the quantile result of the depth map (for example, 92% quantile).

[0053] S1033: Use an area enclosed by pixels corresponding to the first quantile as a target area whose depth value is greater than a target value.

[0054] In the embodiment of the present invention, the first quantile is used as the target value, and the area enclosed by the pixels corresponding to the first quantile is used as the target area with a depth value greater than the target value. That is, the 92% quantile is used as the target value, and the area enclosed by the pixels corresponding to the first quantile is used as the target area with a depth value greater than the target value.

[0055] It should be noted that, in other embodiments of the present invention, the first quantile may also take other values, for example, 90%, 95%, 97%, etc. The embodiment of the present invention is not limited here, as long as it can ensure that the extracted area covers most of the pixels with larger depth values.

[0056] In the actual processing process, there may be more than one area with a depth value greater than the target value, resulting in inaccurate construction of the enclosing rectangle, which in turn affects the calculation of the indoor space size. To address this problem, in an embodiment of the present invention, the number of areas with depth values ​​greater than the target value is determined. When the number of areas is one, the area is directly used as the target area; when the number of areas is greater than one, the area of ​​each area is calculated, and the area with the largest area is determined as the target area. When there are more than one areas with depth values ​​greater than the target value, the present invention uses the area with the largest area as the target area, thereby improving the accuracy of the indoor space size.

[0057] S104: Calculate a first dimension of the indoor space based on the depth value of each pixel in the target area.

[0058] After determining the target area whose depth value is greater than the target value, the first dimension size of the indoor space is calculated based on the depth value of each pixel in the target area.

[0059] Figure 2 FIG. 1 is a flow chart of determining a target area from an indoor image and constructing a bounding rectangle according to an embodiment of the present invention. Figure 2As shown, the direction facing the monocular camera is set as the first dimension Z, the direction perpendicular to the ground is set as the second dimension Y, and the direction perpendicular to the first dimension Z and the second dimension Y is set as the third dimension X. The target area is roughly the area where the window is located directly in front of the monocular camera, which is an irregular shape. The grayscale value of the pixel in the target area represents the depth value. It can be seen that the depth value of each pixel in the target area is not completely consistent. There may be a few pixels with large deviations. In order to avoid the influence of these pixels and improve the accuracy of the first dimension, in an embodiment of the present invention, the number of pixels corresponding to each depth value in the target area is counted, and the second quantile is calculated based on the number of pixels corresponding to each depth value as the first dimension of the indoor space. The process of calculating the second quantile can refer to the process of calculating the first quantile in the aforementioned embodiment, and the embodiment of the present invention will not be repeated here. Exemplarily, the second quantile is the 50% quantile, that is, the median of the distribution function of the number of pixels corresponding to each depth value in the target area is selected as the first dimension of the indoor space.

[0060] S105 , constructing a bounding rectangle of the target area, and determining the image coordinates of the diagonal corners of the bounding rectangle.

[0061] In the embodiment of the present invention, the minimum bounding rectangle algorithm of irregular graphics can be used to construct the bounding rectangle of the target area, and the image coordinates of the diagonal corners of the bounding rectangle can be determined. Figure 2 As shown, the image coordinates of the upper left corner of the circumscribed rectangle are (x1, y1), and the image coordinates of the lower right corner are (x2, y2). It should be noted that in other embodiments, the image coordinates of the lower left corner and the upper right corner may also be taken, and the embodiment of the present invention is not limited here.

[0062] S106 , calculating a second dimension and a third dimension of the indoor space based on the image coordinates of the diagonals of the circumscribed rectangle, the internal parameters of the monocular camera, and the first dimension of the indoor space.

[0063] After determining the image coordinates of the diagonal corners of the circumscribed rectangle, the image coordinates are converted into world coordinates based on the image coordinates of the diagonal corners of the circumscribed rectangle, the internal parameters of the monocular camera and the first dimension of the indoor space, so that the second and third dimensions of the indoor space can be calculated.

[0064] Exemplarily, in the embodiment of the present invention, the above step S106 includes the following sub-steps:

[0065] S1061. Based on the internal parameters of the monocular camera and the first dimension of the indoor space, convert the image coordinates of the diagonal corners of the circumscribed rectangle into world coordinates.

[0066] In the embodiment of the present invention, based on the internal parameters of the monocular camera and the first dimension of the indoor space, and the relationship between the world coordinates and the image coordinates, the image coordinates of the diagonal corners of the circumscribed rectangle are converted into world coordinates.

[0067] Exemplarily, the conversion formula between world coordinates and image coordinates is as follows:

[0068]

[0069]

[0070] Among them, (fx, fy, cx, cy) are the internal parameters of the monocular camera, where fx is the pixel representation of the focal length in the x direction in the image coordinate system, fy is the pixel representation of the focal length in the y direction in the image coordinate system, cx is the offset of the optical center in the x direction in the image coordinate system, cy is the offset of the optical center in the y direction in the image coordinate system, Z is the first dimension of the indoor space, that is, the first dimension coordinate of the point on the circumscribed rectangle in the world coordinate system, X is the second dimension coordinate of the point on the circumscribed rectangle in the world coordinate system, and Y is the third dimension coordinate of the point on the circumscribed rectangle in the world coordinate system.

[0071] Substitute the image coordinates of the upper left corner of the circumscribed rectangle (x1, y1) and the image coordinates of the lower right corner (x2, y2) obtained in the above step S105 into the above conversion formula between world coordinates and image coordinates to obtain the world coordinates of the upper left corner of the circumscribed rectangle (X1, Y1, Z) and the world coordinates of the lower right corner of the circumscribed rectangle (X2, Y2, Z).

[0072] S1062. Calculate the absolute value of the difference between the second dimension coordinate values ​​of the diagonal corners of the circumscribed rectangle in the world coordinates as the second dimension size of the indoor space.

[0073] Exemplarily, the absolute value of the difference between the second dimension coordinate values ​​of the upper left corner and the lower right corner of the circumscribed rectangle in the world coordinates is used as the second dimension size of the indoor space, that is, the second dimension size of the indoor space is |X1-X2|.

[0074] S1063. Calculate the absolute value of the difference between the third-dimensional coordinate values ​​of the diagonal corners of the circumscribed rectangle in the world coordinates as the third-dimensional size of the indoor space.

[0075] Exemplarily, the absolute value of the difference between the third-dimensional coordinate values ​​of the upper left corner and the lower right corner of the circumscribed rectangle in the world coordinates is taken as the third-dimensional size of the indoor space, that is, the third-dimensional size of the indoor space is |Y1-Y2|.

[0076] In the indoor image actually taken, the third dimension direction (height direction) may be blocked, resulting in a deviation between the calculated indoor height and the actual indoor height. In some embodiments of the present invention, in order to make the calculated indoor height (i.e., the third dimension size) consistent with the actual height, the calculated third dimension size can be corrected. Exemplarily, after calculating the third dimension size of the indoor space, it also includes:

[0077] 1. Determine whether the third dimension of the indoor space is within a priori value range formed by a first priori value and a second priori value, wherein the first priori value is smaller than the second priori value.

[0078] According to the architectural design specification, the indoor height is usually in the range of 2.9 meters to 3.5 meters. Therefore, in the embodiment of the present invention, 2.9 meters is used as the first prior value and 3.5 meters is used as the second prior value. It is determined whether the third dimension of the indoor space is within the prior value range formed by the first prior value and the second prior value.

[0079] 2. If the third dimension of the indoor space is within the range of the prior value, output the third dimension.

[0080] If the third dimension of the indoor space is within the range of the prior value, it means that the calculated indoor height is within the normal range. In this case, there is no need to correct the third dimension and the third dimension can be directly output.

[0081] 3. If the third dimension of the indoor space is smaller than the first a priori value, the third dimension is corrected to the first a priori value.

[0082] If the third dimension of the indoor space is smaller than the first a priori value, it means that there is a deviation in the calculated indoor height, and the third dimension is corrected to the first a priori value (ie, 2.9 meters).

[0083] 4. If the third dimension of the indoor space is larger than the second prior value, the third dimension is corrected to the second prior value.

[0084] If the third dimension of the indoor space is larger than the second a priori value, it means that there is a deviation in the calculated indoor height, and the third dimension is corrected to the second a priori value (ie, 3.5 meters).

[0085] The indoor space size calculation method provided by the embodiment of the present invention includes: obtaining an indoor image taken by a monocular camera and the internal parameters of the monocular camera, calculating a depth map of the indoor image, determining a target area whose depth value is greater than a target value from the depth map, calculating a first dimension of the indoor space based on the depth value of each pixel in the target area, constructing a bounding rectangle of the target area, and determining the image coordinates of the diagonal corners of the bounding rectangle, and calculating the second dimension and the third dimension of the indoor space based on the image coordinates of the diagonal corners of the bounding rectangle, the internal parameters of the monocular camera and the first dimension of the indoor space. The present invention directly obtains the three-dimensional size of the indoor space based on the indoor image taken by the monocular camera, and combines the image processing method and the visual geometry method, without the need for a depth sensor, thereby reducing the hardware cost.

[0086] The present invention also provides a device for calculating indoor space dimensions. Figure 3 A schematic diagram of the structure of an indoor space size calculation device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the indoor space size calculation device includes:

[0087] An acquisition module 201 is used to acquire an indoor image captured by a monocular camera and internal parameters of the monocular camera;

[0088] A depth map calculation module 202, configured to calculate a depth map of the indoor image;

[0089] A target region determination module 203, configured to determine a target region having a depth value greater than a target value from the depth map;

[0090] A first size calculation module 204, configured to calculate a first dimension of the indoor space based on a depth value of each pixel in the target area;

[0091] A circumscribed rectangle construction module 205 is used to construct a circumscribed rectangle of the target area and determine the image coordinates of the diagonal corners of the circumscribed rectangle;

[0092] The second dimension calculation module 206 is used to calculate the second dimension and the third dimension of the indoor space based on the image coordinates of the diagonals of the circumscribed rectangle, the internal parameters of the monocular camera and the first dimension of the indoor space.

[0093] In some embodiments of the present invention, the depth map calculation module 202 includes:

[0094] The depth map calculation submodule is used to calculate the depth map of the indoor image using a monocular depth cue method, a machine learning algorithm or a deep learning algorithm.

[0095] In some embodiments of the present invention, the target area determination module 203 includes:

[0096] A first statistical submodule, used to count the number of pixels corresponding to each depth value in the depth map;

[0097] A first quantile calculation submodule, used to calculate a first quantile based on the number of pixels corresponding to each depth value;

[0098] The first target region determination submodule is used to take the region enclosed by pixels corresponding to the first quantile as the target region whose depth value is greater than the target value.

[0099] In some embodiments of the present invention, the first statistics submodule includes:

[0100] Ignore unit, used to not count pixels with depth value zero.

[0101] In some embodiments of the present invention, the first quantile calculation submodule includes:

[0102] A cumulative distribution function calculation unit, used to calculate the cumulative distribution function of the number of pixels corresponding to each depth value;

[0103] The first quantile determining unit is used to take the value of the cumulative distribution function when it is greater than the first quantile as the first quantile.

[0104] In some embodiments of the present invention, the target area determination module 203 further includes:

[0105] A region number determination submodule is used to determine the number of regions where the depth value is greater than the target value;

[0106] A region area calculation submodule, used for calculating the area of ​​each region when the number of the regions is greater than one;

[0107] The second target region determination submodule is used to determine the region with the largest area as the target region.

[0108] In some embodiments of the present invention, the first size calculation module 204 includes:

[0109] A second statistical submodule, used to count the number of pixels corresponding to each depth value in the target area;

[0110] The first size calculation submodule is used to calculate a second quantile as a first dimension size of the indoor space based on the number of pixels corresponding to each depth value.

[0111] In some embodiments of the present invention, the second size calculation module 206 includes:

[0112] A coordinate conversion submodule, configured to convert the image coordinates of the diagonal corners of the circumscribed rectangle into world coordinates based on the internal parameters of the monocular camera and the first dimension of the indoor space;

[0113] A first calculation submodule, configured to calculate an absolute value of a difference between coordinate values ​​of a second dimension of the diagonal corners of the circumscribed rectangle in the world coordinates as a second dimension size of the indoor space;

[0114] The second calculation submodule is used to calculate the absolute value of the difference between the third dimension coordinate values ​​of the diagonal corners of the circumscribed rectangle in the world coordinates as the third dimension size of the indoor space.

[0115] In some embodiments of the present invention, the conversion formula of the coordinate conversion submodule is as follows:

[0116]

[0117]

[0118] Among them, (fx, fy, cx, cy) are the internal parameters of the monocular camera, wherein fx is the pixel representation of the focal length in the x direction in the image coordinate system, fy is the pixel representation of the focal length in the y direction in the image coordinate system, cx is the offset of the position of the optical center in the x direction in the image coordinate system, cy is the offset of the position of the optical center in the y direction in the image coordinate system, Z is the first dimension size of the indoor space, that is, the first dimension coordinate of the target angle of the circumscribed rectangle in the world coordinate system, X is the second dimension coordinate of the target angle of the circumscribed rectangle in the world coordinate system, Y is the third dimension coordinate of the target angle of the circumscribed rectangle in the world coordinate system, and the target angle is any one of the diagonals of the circumscribed rectangle.

[0119] In some embodiments of the present invention, the indoor space size calculation device further includes:

[0120] a judgment module, configured to judge, after calculating the third dimension of the indoor space, whether the third dimension of the indoor space falls within a priori value range formed by a first priori value and a second priori value, wherein the first priori value is smaller than the second priori value;

[0121] an output module, configured to output the third dimension if the third dimension of the indoor space is within the priori value range;

[0122] a first correction module, configured to correct the third dimension size of the indoor space to the first priori value if the third dimension size of the indoor space is smaller than the first priori value;

[0123] The second correction module is used to correct the third dimension size of the indoor space to the second priori value if the third dimension size of the indoor space is larger than the second priori value.

[0124] The indoor space size calculation device can execute the indoor space size calculation method provided by the aforementioned embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the indoor space size calculation method.

[0125] Figure 4 A schematic diagram of the structure of an electronic device provided for an embodiment of the present invention, the electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0126] like Figure 4 As shown, the electronic device includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0127] A number of components in the electronic device are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the indoor space size calculation method.

[0129] In some embodiments, the indoor space size calculation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the indoor space size calculation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the indoor space size calculation method in any other appropriate manner (for example, by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0135] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0136] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the indoor space size calculation method provided in any embodiment of the present application.

[0137] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for calculating indoor space dimensions, characterized in that: include: Acquire an indoor image captured by a monocular camera and internal parameters of the monocular camera; Calculating a depth map of the indoor image; Determine a target area whose depth value is greater than a target value from the depth map; Calculating a first dimension of the indoor space based on a depth value of each pixel in the target area; Constructing a bounding rectangle of the target area and determining the image coordinates of the diagonal corners of the bounding rectangle; Based on the image coordinates of the diagonals of the circumscribed rectangle, the internal parameters of the monocular camera, and the first dimension of the indoor space, the second dimension and the third dimension of the indoor space are calculated.

2. The indoor space size calculation method according to claim 1, characterized in that: Calculating a depth map of the indoor image, comprising: A monocular depth cue method, a machine learning algorithm or a deep learning algorithm is used to calculate a depth map of the indoor image.

3. The indoor space size calculation method according to claim 1, characterized in that: Determining a target area whose depth value is greater than a target value from the depth map includes: Counting the number of pixels corresponding to each depth value in the depth map; Calculate a first quantile based on the number of pixels corresponding to each depth value; The area enclosed by the pixels corresponding to the first quantile is taken as the target area whose depth value is greater than the target value.

4. The indoor space dimension calculation method according to claim 3, characterized in that: When counting the number of pixels corresponding to each depth value in the depth map, it also includes: Pixels with a depth value of zero are not counted.

5. The indoor space size calculation method according to claim 3, characterized in that: The first quantile is calculated based on the number of pixels corresponding to each depth value, including: Calculate the cumulative distribution function of the number of pixels corresponding to each depth value; The value when the cumulative distribution function is greater than the first quantile is taken as the first quantile.

6. The indoor space dimension calculation method according to any one of claims 1 to 5, characterized in that: Determining a target area whose depth value is greater than a target value from the depth map includes: Determine the number of regions where the depth value is greater than the target value; When the number of the regions is greater than one, calculating the area of ​​each of the regions; The region with the largest area is determined as the target region.

7. The indoor space dimension calculation method according to any one of claims 1 to 5, characterized in that: Calculating a first dimension of the indoor space based on the depth value of each pixel in the target area includes: Counting the number of pixels corresponding to each depth value in the target area; A second quantile is calculated based on the number of pixels corresponding to each depth value as the first dimension size of the indoor space.

8. The indoor space dimension calculation method according to any one of claims 1 to 5, characterized in that: Calculating the second dimension and the third dimension of the indoor space based on the image coordinates of the diagonal corners of the circumscribed rectangle, the internal parameters of the monocular camera, and the first dimension of the indoor space includes: Based on the internal parameters of the monocular camera and the first dimension of the indoor space, converting the image coordinates of the diagonal corners of the circumscribed rectangle into world coordinates; Calculate the absolute value of the difference between the second dimension coordinate values ​​of the diagonal corners of the circumscribed rectangle in the world coordinates as the second dimension size of the indoor space; The absolute value of the difference between the third-dimensional coordinate values ​​of the diagonal corners of the circumscribed rectangle in the world coordinates is calculated as the third-dimensional size of the indoor space.

9. The indoor space dimension calculation method according to claim 8, characterized in that: Based on the internal parameters of the monocular camera and the first dimension of the indoor space, the image coordinates of the diagonals of the circumscribed rectangle are converted into world coordinates, including: Among them, (fx, fy, cx, cy) are the internal parameters of the monocular camera, wherein fx is the pixel representation of the focal length in the x direction in the image coordinate system, fy is the pixel representation of the focal length in the y direction in the image coordinate system, cx is the offset of the position of the optical center in the x direction in the image coordinate system, cy is the offset of the position of the optical center in the y direction in the image coordinate system, Z is the first dimension size of the indoor space, that is, the first dimension coordinate of the target angle of the circumscribed rectangle in the world coordinate system, X is the second dimension coordinate of the target angle of the circumscribed rectangle in the world coordinate system, Y is the third dimension coordinate of the target angle of the circumscribed rectangle in the world coordinate system, and the target angle is any one of the diagonals of the circumscribed rectangle.

10. The indoor space dimension calculation method according to any one of claims 1 to 5, characterized in that: After calculating the third dimension of the indoor space, it also includes: Determine whether the third dimension of the indoor space is within a priori value range formed by a first priori value and a second priori value, wherein the first priori value is smaller than the second priori value; If the third dimension size of the indoor space is within the priori value range, outputting the third dimension size; If the third dimension of the indoor space is smaller than the first a priori value, correcting the third dimension to the first a priori value; If the third dimension of the indoor space is larger than the second a priori value, the third dimension is corrected to the second a priori value.

11. A device for calculating indoor space dimensions, characterized in that: include: An acquisition module, used to acquire an indoor image taken by a monocular camera and internal parameters of the monocular camera; A depth map calculation module, used to calculate the depth map of the indoor image; A target area determination module, used to determine a target area having a depth value greater than a target value from the depth map; A first size calculation module, configured to calculate a first dimension of the indoor space based on a depth value of each pixel in the target area; A circumscribed rectangle construction module, used to construct a circumscribed rectangle of the target area and determine the image coordinates of the diagonal corners of the circumscribed rectangle; The second size calculation module is used to calculate the second dimension and the third dimension of the indoor space based on the image coordinates of the diagonals of the circumscribed rectangle, the internal parameters of the monocular camera and the first dimension of the indoor space.

12. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the indoor space size calculation method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the indoor space size calculation method as described in any one of claims 1-10 is implemented.

Images