Spatial three-dimensional structure information recovery method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and provides a spatial three-dimensional structure information recovery method and device, equipment and a storage medium, and the method comprises the steps: obtaining a target scene image shot by a color camera and internal parameters of the color camera; inputting the target scene image into a depth estimation model, and estimating scale-free depth information of the target scene image through the depth estimation model to obtain a scale-free depth map; performing scale recovery on the scale-free depth map to obtain a scale depth map; performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image; according to the pixel coordinate position, associating the target scene image, the scale depth map and the instance segmentation map to obtain an association map; based on the internal parameters, back-projecting the association graph to the point cloud space to obtain preliminary spatial three-dimensional structure information; the three-dimensional structure information can be recovered by adopting a scheme of combining a common color camera with an algorithm, and the cost is low.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for recovering spatial three-dimensional structure information. Background Art

[0002] The three-dimensional structure information of space is very important for augmented reality (AR), virtual reality (VR) and some other tasks that require the use of the three-dimensional structure information of space, such as depth-based camera autofocus and spatial perception-based microphone array initial parameter estimation. The image captured by a color camera is a two-dimensional signal obtained by perspective projection of the three-dimensional space, so the three-dimensional structure information of space cannot be directly obtained from it.

[0003] At present, the three-dimensional structure restoration scheme of indoor space mainly uses depth sensors such as TOF (Time-of-Fl igh) and structured light to generate the depth information of the space, and uses color cameras to generate RGB color information, and calibrates and aligns the above two information. Finally, the depth information is back-projected in combination with the internal parameters of the camera to generate a point cloud of the three-dimensional structure of the space. However, the cost of equipment equipped with depth sensors is relatively high, and in industrial scenarios, the cost of components is often very sensitive. Therefore, there is an urgent need for a solution that can achieve three-dimensional structure restoration of indoor space and is relatively low-cost. Summary of the invention

[0004] In response to the above technical problems, the purpose of this application is to provide a method, device, equipment and storage medium for recovering three-dimensional structure information of space, aiming to solve the technical problem that the existing three-dimensional structure recovery of indoor space requires the use of equipment equipped with depth sensors, resulting in relatively high costs.

[0005] In a first aspect, an embodiment of the present application provides a method for restoring spatial three-dimensional structure information, comprising:

[0006] Acquire a target scene image captured by a color camera and internal parameters of the color camera;

[0007] Inputting the target scene image into a depth estimation model, estimating scale-free depth information of the target scene image through the depth estimation model, and obtaining a scale-free depth map;

[0008] Performing scale restoration on the scale-free depth map to obtain a scaled depth map;

[0009] Performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image;

[0010] Associating the target scene image, the scaled depth map, and the instance segmentation map according to pixel coordinate positions to obtain an association map;

[0011] Based on the internal parameters, the association map is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information.

[0012] Furthermore, after the step of back-projecting the association graph into the point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information, the method further includes:

[0013] According to the instance labels in the instance segmentation map, a point cloud plane fitting method is used to perform plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information to obtain optimized spatial three-dimensional structure information.

[0014] Furthermore, the performing scale restoration on the scale-free depth map includes:

[0015] The scale-free depth map is scaled using a priori calibration objects.

[0016] Furthermore, the step of using a priori calibration objects to restore the scale of the scale-free depth map includes:

[0017] Obtaining the actual height of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0018] Calculating the actual depth of the prior calibration object according to the actual height of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the vertical focal length of the camera;

[0019] Calculating a scale factor according to the actual depth and the corresponding scale-free depth information;

[0020] The scale-free depth map is scaled according to the scale factor.

[0021] Furthermore, the step of using a priori calibration objects to restore the scale of the scale-free depth map includes:

[0022] Obtaining the actual width of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0023] Calculating the actual depth of the prior calibration object according to the actual width of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the horizontal focal length of the camera;

[0024] Calculating a scale factor according to the actual depth and the corresponding scale-free depth information;

[0025] The scale-free depth map is scaled according to the scale factor.

[0026] Furthermore, based on the internal parameters, the association graph is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information, including:

[0027] Using the internal parameters, the correlation graph is subjected to distortion correction to obtain a distortion-corrected correlation graph;

[0028] Based on the pixel coordinates of the distortion-corrected association map, the internal parameters and the depth information of the scaled depth map, the distortion-corrected association map is back-projected into the point cloud space using a back-projection formula to obtain preliminary spatial three-dimensional structure information.

[0029] Furthermore, the camera is a fisheye camera or a pinhole camera.

[0030] In a second aspect, an embodiment of the present application provides a spatial three-dimensional structure information recovery device, comprising:

[0031] An acquisition module is used to acquire a target scene image captured by a color camera and internal parameters of the color camera;

[0032] An input module, used to input the target scene image into a depth estimation model, estimate the scale-free depth information of the target scene image through the depth estimation model, and obtain a scale-free depth map;

[0033] A scale restoration module, used for performing scale restoration on the scale-free depth map to obtain a scaled depth map;

[0034] An instance segmentation module is used to perform instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image;

[0035] An association module, used to associate the target scene image, the scaled depth map and the instance segmentation map according to pixel coordinate positions to obtain an association map;

[0036] The back-projection module is used to back-project the association map into the point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information.

[0037] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the methods described above are implemented.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0039] The embodiment of the present application provides a method for restoring spatial three-dimensional structure information, including: obtaining a target scene image obtained by taking a color camera and the internal parameters of the color camera; inputting the target scene image into a depth estimation model, estimating the scale-free depth information of the target scene image through the depth estimation model, and obtaining a scale-free depth map; performing scale restoration on the scale-free depth map to obtain a scaled depth map; performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image; according to the pixel coordinate position, the target scene image, the scaled depth map and the instance segmentation map are associated to obtain an association map; based on the internal parameters, the association map is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information. It can be seen that the embodiment of the present application realizes the restoration of three-dimensional structure information by adopting a solution of combining an ordinary color camera with an algorithm, which greatly reduces the cost compared to the restoration of spatial three-dimensional structure information based on hardware such as TOF and structured light, and has great advantages. In addition, by associating the target scene image, the scaled depth map and the instance segmentation map, the efficiency and accuracy of point cloud coloring are greatly improved, and instance-level point clouds can also be efficiently obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a flowchart of a method for restoring spatial three-dimensional structure information provided in an embodiment of the present application;

[0042] Figure 2 is another flowchart of the method for restoring spatial three-dimensional structure information provided by an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of the structure of a spatial three-dimensional structure information recovery device provided in an embodiment of the present application;

[0044] Figure 4 Another structural diagram of the spatial three-dimensional structure information recovery device provided in the embodiment of the present application is

[0045] Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may also be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0048] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0049] Embodiment 1:

[0050] like Figure 1 As shown, the embodiment of the present application provides a method for recovering spatial three-dimensional structure information, including: Step S1-Step S6:

[0051] S1, obtaining a target scene image captured by a color camera and internal parameters of the color camera;

[0052] S2, inputting the target scene image into a depth estimation model, estimating scale-free depth information of the target scene image through the depth estimation model, and obtaining a scale-free depth map;

[0053] S3, performing scale restoration on the scale-free depth map to obtain a scaled depth map;

[0054] S4, performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image;

[0055] S5. Associating the target scene image, the scaled depth map, and the instance segmentation map according to pixel coordinate positions to obtain an association map;

[0056] S6. Based on the internal parameters, back-project the association graph into the point cloud space to obtain preliminary spatial three-dimensional structure information.

[0057] As in step S1 above, the image captured by the color camera is a color image, which means that each pixel is composed of R (red), G (blue), and B (green) components. Generally, the embodiment of the present application is used for indoor space three-dimensional structure recovery. Therefore, the target scene image is an indoor target scene image. The color camera can be a common pinhole camera, a fisheye camera, or other cameras, which is not limited in the embodiment of the present application. If the color camera is a pinhole camera, the internal parameters K include f x , f y , c x , c y , k1, k2, k3, p1 and p2; among them, f x Indicates the horizontal focal length of the camera, f y Indicates the vertical focal length of the camera, c x represents the horizontal center of the camera, c y represents the vertical center of the camera, k1, k2, k3 represent the radial distortion coefficients of the lens, p1, p2 represent the tangential distortion coefficients of the lens. If the color camera is a fisheye camera, K is f x , f y , c x , c y , f1, f2, f3, f4; among them, f x Indicates the horizontal focal length of the camera, f y Indicates the vertical focal length of the camera, c x represents the horizontal center of the camera, c y represents the vertical center of the camera, and f1, f2, f3, and f4 all represent the distortion coefficients of the fisheye lens. In addition, the camera is a monocular camera.

[0058] As in step S2 above, the depth estimation model can be any model that can predict the scale-free depth information of the target scene image, including but not limited to a model based on statistical methods and a model based on deep learning methods. For example, the structure of the depth estimation model includes an encoder and a decoder, the encoder is used to extract features of the target scene image, and the decoder is used to estimate the scale-free depth information of the target scene image from the extracted features, that is, to estimate the scale-free depth information of the scene, and its value range is [0,1]. When the camera is a monocular camera, the depth estimation model is a monocular depth estimation model. In addition, the scale-free depth map represents the scale-free depth information of the target scene image.

[0059] As in step S3 above, the scale-free depth map is scaled to obtain a scaled depth map. The scaled depth map represents the scale information of the target scene image. It should be understood that by performing scale restoration on the scale-free depth information corresponding to each pixel of the target scene image, the scale information of the corresponding pixel is obtained, and then the scale information of the target scene image, that is, the scaled depth map corresponding to the target scene image, can be obtained. The pixel value of the depth map represents the depth, and the depth represents the distance between a point in the scene and the camera.

[0060] As in step S4 above, according to the different segmentation granularities, image segmentation is divided into semantic segmentation and instance segmentation. Semantic segmentation refers to classification at the pixel level, and pixels of the same category are divided into the same category. Compared with target detection using box annotation, semantic segmentation is more refined. Instance segmentation is more detailed than semantic segmentation, and objects of the same category but belonging to different individuals can be distinguished. The instance segmentation map is composed of mask labels at the instance level of each thing, and the mask labels at the instance level of things are usually referred to as instance labels. It should be noted that the embodiment of the present application does not limit the order of steps without logical relationship, such as step S4 can be executed before steps S2 and S3, or after steps S2 and S3 are executed, step S4 is executed. In addition, steps S2-S3 as a large step can also be executed in parallel with step S4.

[0061] As in step S5 above, let the target scene image be I, the scaled depth map be D, the instance segmentation map be C, and the association map be P. Then, according to the pixel coordinate position, the target scene image I, the scaled depth map D and the instance segmentation map C are associated. Specifically, according to the pixel coordinate position (x, y), I(x, y), D(x, y), and C(x, y) corresponding to each coordinate position are associated to form P(x, y). Then the association map P is the set of all P(x, y). Preferably, the target scene image I, the scaled depth map D, and the instance segmentation map C are all matrices with a scale of W*H, and each number in the matrix is ​​a floating point number, P = {I, D, C}, where {} is a series connection symbol, and the matrix dimension of P is W*H*3. The embodiment of the present application greatly improves the efficiency and accuracy of point cloud coloring by connecting {I, D, C} in series at the two-dimensional matrix level, which is different from the existing hardware solution that requires the depth information obtained by the color camera and the depth sensor to be calibrated and registered. In addition, the embodiment of the present application concatenates instance labels together to efficiently obtain instance-level point clouds.

[0062] As in step S6 above, the three-dimensional structure information of the space is composed of point cloud information. The point cloud is a collection of points. Each point in the point cloud represents a set of x, y, z coordinates and other information. Since the embodiment of the present application associates the target scene image, the scaled depth map, and the instance segmentation map, and the target scene image is a color map, the spatial three-dimensional structure information obtained after projecting the association map into the point cloud space includes color information and semantic information. Described in mathematical language, the spatial three-dimensional structure information J(i) = (x d ,y d ,z d ,r,g,b,c); where (x d ,y d ,z d ) represents the three-dimensional coordinates, r, g, b are the three color values ​​at the same pixel coordinates of the color image, c is the instance label at the same pixel coordinates, and i represents the number of the point cloud.

[0063] The embodiment of the present application realizes the restoration of three-dimensional structural information by using a common color camera combined with an algorithm. Compared with the restoration of spatial three-dimensional structural information based on hardware such as TOF and structured light, the cost is greatly reduced and it has great advantages. In addition, by associating the target scene image, the scaled depth map and the instance segmentation map, the efficiency and accuracy of point cloud coloring are greatly improved, and instance-level point clouds can also be efficiently obtained.

[0064] like Figure 2 As shown, in one embodiment, after the step of back-projecting the association graph into the point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information, step S7 is also included:

[0065] S7. According to the instance labels in the instance segmentation map, a point cloud plane fitting method is used to perform plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information to obtain optimized spatial three-dimensional structure information.

[0066] In the embodiment of the present application, specifically, according to the instance labels of the objects in the instance segmentation map, the point cloud plane fitting method such as the least square method can be used to perform plane fitting on the point clouds of plane categories such as walls, desktops, ceilings, floors, etc. in the preliminary spatial three-dimensional structure information, so as to obtain spatial three-dimensional structure information that is more in line with the actual situation. It should be noted that since the indoor space contains a large number of weak texture areas, such as walls, desktops, ceilings, and floors, it is generally difficult for the depth estimation model to obtain accurate depth information, resulting in a certain degree of error in the preliminary spatial three-dimensional structure information obtained above. The embodiment of the present application determines the weak texture area based on the instance label information in the preliminary spatial three-dimensional structure information, and performs plane fitting on it at the point cloud level, thereby improving the accuracy of the spatial three-dimensional structure information corresponding to the weak texture area.

[0067] In one embodiment, the step of restoring the scale of the scale-free depth map comprises:

[0068] The scale-free depth map is scaled using a priori calibration objects.

[0069] In the embodiment of the present application, the prior calibration object may be some planar objects in the room, such as a wall or a table. Specifically, the width or height of the prior calibration object may be used to restore the scale of the scale-free information of the scale-free depth map.

[0070] The embodiment of the present application uses a priori calibration objects to restore the scale of the scale-free depth information. Since the method is simple, the scale restoration of the scale-free depth information can be achieved efficiently.

[0071] In one embodiment, the step of using a priori calibration objects to restore the scale of the scale-free depth map includes steps S31 to S34:

[0072] S31, obtaining the actual height of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0073] S32, calculating the actual depth of the prior calibration object according to the actual height of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the vertical focal length of the camera;

[0074] S33, calculating a scale factor according to the actual depth and corresponding scale-free depth information;

[0075] S34: restore the scale of the scale-free depth map according to the scale factor.

[0076] As in the above steps S31-S34, taking the center coordinates of the prior calibration object as an example, the actual height of the (center coordinates (x, y)) at the center of the prior calibration object and the number of pixels of the height at the center coordinates of the prior calibration object in the scale-free depth map are obtained, and then the actual depth at the center coordinates of the prior calibration object is calculated according to the actual height at the center of the prior calibration object, the number of pixels of the height at the center coordinates of the prior calibration object in the scale-free depth map, and the vertical focal length of the camera, and then the scale-free depth information corresponding to the center coordinates of the prior calibration object is obtained, and then the scale factor is calculated according to the actual depth at the center coordinates of the prior calibration object and the scale-free depth information corresponding to the center coordinates of the prior calibration object, and the scale-free depth map is scaled according to the scale factor. Specifically, the actual depth is calculated using the formula d=(h / p)*fy, where h represents the actual height corresponding to the prior calibration object in the physical world, p represents the number of pixels of the height of the prior calibration object in the scale-free depth map, fy represents the vertical focal length of the camera, and d represents the actual depth of the prior calibration object. The scale factor is calculated using factor = d / D'(x, y), where D'(x, y) represents the scale-free depth information of the calibration object (it should be noted that the scale-free depth information of a certain point of the prior calibration object, such as the center point of the prior calibration object, is used as the scale-free depth information of the prior calibration object). The depth map is scaled using D = D'*factor. Where D' represents the scale-free depth map and D represents the scaled depth map.

[0077] The scale restoration method of the embodiment of the present application is simple and can efficiently realize scale restoration without scale depth information.

[0078] In one embodiment, the step of using a priori calibration objects to restore the scale of the scale-free depth map includes steps S35-S38:

[0079] S35, obtaining the actual width of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0080] S36, calculating the actual depth of the prior calibration object according to the actual width of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the horizontal focal length of the camera;

[0081] S37, calculating a scale factor according to the actual depth and the corresponding scale-free depth information;

[0082] S38: Restoring the scale of the scale-free depth map according to the scale factor.

[0083] As in the above steps S35-S38, taking the center coordinates of the prior calibration object as an example, the actual width of the (center coordinates (x, y)) at the center of the prior calibration object and the number of pixels at the height of the center coordinates of the prior calibration object in the scale-free depth map are obtained, and then the actual depth at the center coordinates of the prior calibration object is calculated according to the actual width at the center of the prior calibration object, the number of pixels at the height of the center coordinates of the prior calibration object in the scale-free depth map, and the horizontal focal length of the camera, and then the scale-free depth information corresponding to the center coordinates of the prior calibration object is obtained, and then the scale factor is calculated according to the actual depth at the center coordinates of the prior calibration object and the scale-free depth information corresponding to the center coordinates of the prior calibration object, and the scale-free depth map is restored according to the scale factor. Specifically, the actual depth is calculated using the formula d=(w / p)*fy, where w represents the actual width corresponding to the prior calibration object in the physical world, p represents the number of pixels at the height of the prior calibration object in the scale-free depth map, fy represents the horizontal focal length of the camera, and d represents the actual depth of the prior calibration object. The scale factor is calculated using factor = d / D'(x, y), where D'(x, y) represents the scale-free depth information of the calibration object (it should be noted that the scale-free depth information of a certain point of the prior calibration object, such as the center point of the prior calibration object, is used as the scale-free depth information of the prior calibration object). The depth map is scaled using D = D'*factor. Where D' represents the scale-free depth map and D represents the scaled depth map.

[0084] The scale restoration method of the embodiment of the present application is simple and can efficiently restore scale-free depth information.

[0085] It should be noted that when performing monocular scale restoration, for the sake of simplicity, only the derivation process of a single point (x, y) is discussed. In fact, multiple points can be sampled on the calibration object, and a more accurate scale factor can be calculated through the least squares method to obtain more accurate depth information.

[0086] In one embodiment, performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image includes:

[0087] The target scene image is input into an instance segmentation model, and instance segmentation is performed on the target scene image through the instance segmentation model to obtain an instance segmentation map corresponding to the target scene image.

[0088] In the embodiment of the present application, the instance labels of the objects in the image can be directly obtained through the instance segmentation model, and then the instance segmentation map can be obtained. It should be understood that the instance segmentation map is composed of the instance labels of each object. It should be noted that the instance segmentation model can be any model that can perform instance segmentation on the target scene image, and the embodiment of the present application does not limit this.

[0089] In one embodiment, the step of performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image includes steps S41-S43:

[0090] S41, predicting the semantic category of each pixel in the target scene image through a semantic segmentation model to obtain a semantic segmentation map;

[0091] S42, according to the semantic category, using a connected domain search algorithm to search for connected regions in the semantic segmentation graph;

[0092] S43: taking each connected region as an instance label to obtain an instance segmentation map corresponding to the target scene image.

[0093] As in step S41 above, the semantic segmentation model has a similar structure to the depth estimation model, including an encoder and a decoder, the encoder is used to extract features from the target scene image, and the decoder is used to classify the semantic information of the scene from the extracted features. It should be noted that the semantic segmentation model can be any model with such functions, including but not limited to those based on statistical methods and deep learning methods.

[0094] As in step S42 above, since the semantic categories of pixels are distinguished by labels, the mask map of each type of semantic label can be obtained by obtaining the semantic category of each pixel. According to the semantic category, the specific steps of using the connected domain search algorithm to find the connected area in the semantic segmentation map are: for the mask map of each type of semantic label (semantic segmentation map and mask map of each type of label), first set the pixel value with label in the mask map to 255, and set the pixel value without label to zero to obtain a binary map. Input the binary map into the connected domain search algorithm, extract the connected domain therein, and regard each type of connected domain as an instance label. It should be understood that the connected domain search algorithm can be any algorithm that can realize the connected domain search, for example, a seed filling algorithm can be used or a two-pass scanning method can be used. If the seed filling algorithm is used, first select a pixel point in the foreground as a seed, and gradually expand outward according to the connection conditions until all the pixels in the area are traversed, and search other areas in the same way until all areas on the image have been found. The condition for connectivity may be: if the eight neighboring values ​​of the 3*3 area around the current pixel position are all the same as the central pixel value, then the 3*3 area is determined to be connected. The condition for connectivity may also be other conditions, which can be set according to actual needs, and the embodiment of the present application does not limit this.

[0095] In one embodiment, the back-projecting of the association graph into a point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information includes:

[0096] Using the internal parameters, the correlation graph is subjected to distortion correction to obtain a distortion-corrected correlation graph;

[0097] Based on the pixel coordinates of the distortion-corrected association map, the internal parameters and the depth information of the scaled depth map, the distortion-corrected association map is back-projected into the point cloud space using a back-projection formula to obtain preliminary spatial three-dimensional structure information.

[0098] In the embodiment of the present application, the distortion correction of the correlation map can be performed using public functions such as opencv, such as using the undistort function to correct image distortion. Suppose the correlation map after distortion correction is P', then P' = cv2.undistort(P, np.array([k1, k2, k3, p1, p2]), np.array(K)); np.array means matrixing the array, and K is the internal parameter of the camera.

[0099] In the embodiments of the present application, the back-projection process varies depending on the camera.

[0100] When a pinhole camera is used, the back-projection formula (1) is used

[0101] The distortion-corrected correlation map is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information. Specifically, the coordinates in the distortion-corrected correlation map P are substituted into the back-projection formula (1) where u, v, u, v represent the horizontal and vertical coordinates of the pixels in the image, respectively, the corresponding depth information is substituted into D(u, v), and the internal parameters are substituted into K, then x can be calculated. d ,y d ,z d , where (x d ,y d ,z d) The pixel points representing the u,v positions are transformed into coordinates in the three-dimensional space. Since the target scene image I, the scaled depth map D, and the instance segmentation map C are associated, the preliminary spatial three-dimensional structure information includes color information and semantic information. The preliminary spatial three-dimensional structure information is J(i) = (x d ,y d ,z d ,r,g,b,c); where r,g,b are the three color values ​​at the same pixel coordinates of the color image, c is the instance label at the same pixel coordinates, and i represents the number of the point cloud. It should be understood that because P has associated the target scene image I (i.e., the color image), the scaled depth image D, and the instance segmentation image C, the color and instance labels at the same pixel coordinates are also associated when back-projected into the three-dimensional space.

[0102] When a fisheye camera is used, the distortion-corrected correlation map is back-projected into the point cloud space using the back-projection formula (2) to obtain preliminary spatial three-dimensional structure information.

[0103]

[0104]

[0105]

[0106]

[0107] θ * =d -1 (dθ)

[0108] d(θ)=θ+f1θ 3 +f2θ 5 +f3θ 7 +f4θ 9

[0109] Specifically, substitute the coordinates in the distortion-corrected correlation map P' into u and v in the back-projection formula (2), where u and v represent the horizontal and vertical coordinates of the pixels in the image, respectively. Substitute the corresponding depth information into D(u, v), and substitute the horizontal focal length of the camera f x , the vertical focal length of the camera f y , the horizontal center of the camera c x , the vertical center of the camera c y , and the fisheye lens distortion coefficients f1, f2, f3, f4 are substituted into the above formula to obtain x d ,y d ,z d , x d ,y d ,z d The pixel representing the position of u,v is transformed into the coordinates in three-dimensional space. x , m y Respectively represent the normalized coordinates of the pixels in the matrix in the horizontal and vertical coordinates, which are intermediate variables. In addition, the camera is a monocular camera. It should be noted that since the target scene image I, the scaled depth map D and the instance segmentation map C are associated, the preliminary spatial three-dimensional structure information contains color information and semantic information. The preliminary spatial three-dimensional structure information is J(i)=(x d ,y d ,z d,r,g,b,c); where r,g,b are the three color values ​​at the same pixel coordinates of the color image, c is the instance label at the same pixel coordinates, and i represents the number of the point cloud. It should be understood that because P has associated the target scene image I (i.e., the color image), the scaled depth image D, and the instance segmentation image C, the color and instance labels at the same pixel coordinates are also associated when back-projected into the three-dimensional space.

[0110] The embodiments of the present application are adaptable to various types of color cameras, including pinhole cameras and fisheye cameras, and are widely used.

[0111] Embodiment 2:

[0112] like Figure 3 As shown, the embodiment of the present application provides a spatial three-dimensional structure information recovery device, comprising:

[0113] Acquisition module 1, used to acquire the target scene image captured by the color camera and the internal parameters of the color camera;

[0114] An input module 2 is used to input the target scene image into a depth estimation model, and estimate the scale-free depth information of the target scene image through the depth estimation model to obtain a scale-free depth map;

[0115] A scale restoration module 3 is used to restore the scale of the scale-free depth map to obtain a scaled depth map;

[0116] An instance segmentation module 4 is used to perform instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image;

[0117] An association module 5 is used to associate the target scene image, the scaled depth map and the instance segmentation map according to pixel coordinate positions to obtain an association map;

[0118] The back-projection module 6 is used to back-project the association map into the point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information.

[0119] like Figure 4 As shown, in one embodiment, the spatial three-dimensional structure information recovery device further includes:

[0120] The optimization module 7 is used to perform plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information according to the instance labels in the instance segmentation map by using a point cloud plane fitting method to obtain optimized spatial three-dimensional structure information.

[0121] In one embodiment, the scale restoration module 3 is specifically configured to restore the scale of the scale-free depth map using a priori calibration objects.

[0122] In one embodiment, the scale recovery module 3 includes:

[0123] A first acquisition unit is used to acquire the actual height of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0124] A first actual depth calculation unit, configured to calculate the actual depth of the prior calibration object according to the actual height of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the vertical focal length of the camera;

[0125] A first scale factor calculation unit, configured to calculate a scale factor according to the actual depth and corresponding scale-free depth information;

[0126] The first scale restoration unit is configured to restore the scale of the scale-free depth map according to the scale factor.

[0127] In one embodiment, the scale recovery module 3 includes:

[0128] A second acquisition unit is configured to acquire an actual width of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0129] A second actual depth calculation unit, configured to calculate the actual depth of the prior calibration object according to the actual width of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the horizontal focal length of the camera;

[0130] A second scale factor calculation unit, configured to calculate a scale factor according to the actual depth and corresponding scale-free depth information;

[0131] The second scale restoration unit is configured to restore the scale of the scale-free depth map according to the scale factor.

[0132] In one embodiment, the back projection module comprises:

[0133] A distortion correction unit, configured to perform distortion correction on the correlation graph using the internal parameters to obtain a distortion-corrected correlation graph;

[0134] The back-projection unit is used to back-project the distortion-corrected association map to the point cloud space using a back-projection formula based on the pixel coordinates of the distortion-corrected association map, the internal parameters and the depth information of the scaled depth map to obtain preliminary spatial three-dimensional structure information.

[0135] In one embodiment, the camera is a fisheye camera or a pinhole camera.

[0136] It should be noted that the spatial three-dimensional structure information recovery device provided in the embodiment of the present application corresponds to the spatial three-dimensional structure information recovery method, so the embodiment of the present application will not be described in detail.

[0137] Embodiment three:

[0138] Reference Figure 5 The present application also provides a computer device, the internal structure of which can be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a method for recovering spatial three-dimensional structure information. The network interface of the computer device is used to communicate with an external terminal through a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen, etc. When the above computer program is executed by a processor, a method for recovering spatial three-dimensional structure information is implemented. The method for recovering spatial three-dimensional structure information includes the following steps: obtaining a target scene image captured by a color camera and the internal parameters of the color camera; inputting the target scene image into a depth estimation model, estimating the scale-free depth information of the target scene image through the depth estimation model, and obtaining a scale-free depth map; performing scale restoration on the scale-free depth map to obtain a scaled depth map; performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image; associating the target scene image, the scaled depth map, and the instance segmentation map according to the pixel coordinate position to obtain an association map; and back-projecting the association map into the point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information. Those skilled in the art can understand that Figure 5 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0139] Embodiment 4:

[0140] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a method for recovering spatial three-dimensional structure information is implemented. The method for recovering spatial three-dimensional structure information includes the following steps: obtaining a target scene image captured by a color camera and internal parameters of the color camera; inputting the target scene image into a depth estimation model, estimating the scale-free depth information of the target scene image through the depth estimation model, and obtaining a scale-free depth map; performing scale restoration on the scale-free depth map to obtain a scaled depth map; performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image; associating the target scene image, the scaled depth map and the instance segmentation map according to pixel coordinate positions to obtain an association map; and back-projecting the association map into a point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information.

[0141] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0142] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0143] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for restoring spatial three-dimensional structure information, characterized in that: include: Acquire a target scene image captured by a color camera and internal parameters of the color camera; Inputting the target scene image into a depth estimation model, estimating scale-free depth information of the target scene image through the depth estimation model, and obtaining a scale-free depth map; Performing scale restoration on the scale-free depth map to obtain a scaled depth map; Performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image; Associating the target scene image, the scaled depth map, and the instance segmentation map according to pixel coordinate positions to obtain an association map; Based on the internal parameters, the association map is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information.

2. The method for restoring spatial three-dimensional structure information according to claim 1, characterized in that: After the step of back-projecting the association graph into the point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information, the method further includes: According to the instance labels in the instance segmentation map, a point cloud plane fitting method is used to perform plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information to obtain optimized spatial three-dimensional structure information.

3. The method for restoring spatial three-dimensional structure information according to claim 1, characterized in that: The step of restoring the scale of the scale-free depth map comprises: The scale-free depth map is scaled using a priori calibration objects.

4. The method for restoring spatial three-dimensional structure information according to claim 3, characterized in that: The step of using a priori calibration objects to restore the scale of the scale-free depth map comprises: Obtaining the actual height of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map; Calculating the actual depth of the prior calibration object according to the actual height of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the vertical focal length of the camera; Calculating a scale factor according to the actual depth and the corresponding scale-free depth information; The scale-free depth map is scaled according to the scale factor.

5. The method for restoring spatial three-dimensional structure information according to claim 3, characterized in that: The step of using a priori calibration objects to restore the scale of the scale-free depth map comprises: Obtaining the actual width of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map; Calculating the actual depth of the prior calibration object according to the actual width of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the horizontal focal length of the camera; Calculating a scale factor according to the actual depth and the corresponding scale-free depth information; The scale-free depth map is scaled according to the scale factor.

6. The method for restoring spatial three-dimensional structure information according to claim 1, characterized in that: Based on the internal parameters, the association graph is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information, including: Using the internal parameters, the correlation graph is subjected to distortion correction to obtain a distortion-corrected correlation graph; Based on the pixel coordinates of the distortion-corrected association map, the internal parameters and the depth information of the scaled depth map, the distortion-corrected association map is back-projected into the point cloud space using a back-projection formula to obtain preliminary spatial three-dimensional structure information.

7. The method for restoring spatial three-dimensional structure information according to claim 1, characterized in that: The camera is a fisheye camera or a pinhole camera.

8. A spatial three-dimensional structure information recovery device, characterized in that: include: An acquisition module is used to acquire a target scene image captured by a color camera and internal parameters of the color camera; An input module, used to input the target scene image into a depth estimation model, estimate the scale-free depth information of the target scene image through the depth estimation model, and obtain a scale-free depth map; A scale restoration module, used for performing scale restoration on the scale-free depth map to obtain a scaled depth map; An instance segmentation module is used to perform instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image; An association module, used to associate the target scene image, the scaled depth map and the instance segmentation map according to pixel coordinate positions to obtain an association map; The back-projection module is used to back-project the association map into the point cloud space based on the internal parameters to obtain preliminary spatial three-dimensional structure information.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images