Point cloud correction method and three-dimensional model generation method, device and equipment

Abstract

The present disclosure provides a point cloud correction method and a three-dimensional model generation method, device, equipment and medium, relates to the field of artificial intelligence, and in particular to the technical field of virtual reality, augmented reality, metaverse, computer vision and deep learning. The point cloud correction method comprises: performing plane fitting according to first point cloud data of a first object to obtain a plane model; correcting the first point cloud data according to the plane model to obtain corrected point cloud data; determining target point cloud data corresponding to a target boundary line in a target image in the corrected point cloud data, the target boundary line being a boundary line between the first object and a second object; determining a depth value corresponding to a second pixel point of the second object according to the target point cloud data based on a positional relationship between the first pixel point corresponding to the target point cloud data in the target image and the second pixel point corresponding to the second object in the target image; and correcting second point cloud data of the second object according to the depth value.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the technical fields of virtual reality, augmented reality, metaverse, computer vision and deep learning, and more particularly to a point cloud correction method and a three-dimensional model generation method, device, equipment and medium. BACKGROUND

[0002] With the development of computer technology and network technology, various emerging technologies have emerged. For example, three-dimensional reconstruction technology for generating three-dimensional models of scenes has developed rapidly. Generally, in three-dimensional reconstruction technology, the generation of a plane has a high accuracy requirement for point cloud data, and there are often situations where the plane is distorted and the three-dimensional model is low in accuracy due to inaccurate point cloud data. SUMMARY

[0003] The present disclosure aims to provide a point cloud correction method and a three-dimensional model generation method, device, equipment and medium to improve the accuracy of point cloud data and three-dimensional models and avoid the distortion of a plane.

[0004] According to one aspect of the present disclosure, a point cloud correction method is provided, comprising: performing plane fitting according to first point cloud data of a first object to obtain a plane model representing a plane where the first object is located; correcting the first point cloud data according to the plane model to obtain corrected point cloud data; determining target point cloud data in the corrected point cloud data corresponding to a target boundary line in a target image, the target boundary line being a boundary line between the first object and a second object in the target image; determining a depth value corresponding to a second pixel point of the second object according to the target point cloud data based on a positional relationship between a first pixel point corresponding to the target point cloud data in the target image and the second pixel point; and correcting second point cloud data of the second object according to the depth value.

[0005] According to another aspect of the present disclosure, a three-dimensional model generation method is provided, comprising: generating initial point cloud data corresponding to a target image including a first object and a second object according to the target image; the initial point cloud data including first point cloud data of the first object and second point cloud data of the second object; correcting the first point cloud data and the second point cloud data using the point cloud correction method provided by the present disclosure to obtain corrected point cloud data; and generating a three-dimensional model according to the corrected point cloud data.

[0006] According to another aspect of the present disclosure, a device for correcting point cloud is provided, comprising: a plane fitting module configured to perform plane fitting on first point cloud data of a first object to obtain a plane model representing a plane where the first object is located; a first correction module configured to correct the first point cloud data according to the plane model to obtain corrected point cloud data; a target point cloud data determination module configured to determine target point cloud data corresponding to a target boundary line in a target image from the corrected point cloud data, the target boundary line being a boundary line between the first object and a second object in the target image; a depth value determination module configured to determine a depth value corresponding to a second pixel in the target image according to the target point cloud data based on a positional relationship between the first pixel corresponding to the target point cloud data and the second pixel corresponding to the second object in the target image; and a second correction module configured to correct second point cloud data of the second object according to the depth value.

[0007] According to another aspect of the present disclosure, a device for generating a three-dimensional model is provided, comprising: a point cloud generation module configured to generate initial point cloud data corresponding to a target image including a first object and a second object according to the target image; the initial point cloud data including first point cloud data of the first object and second point cloud data of the second object; a point cloud correction module configured to correct the first point cloud data and the second point cloud data using the device for correcting point cloud provided by the present disclosure to obtain corrected point cloud data; and a model generation module configured to generate a three-dimensional model according to the corrected point cloud data.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for correcting point cloud and / or the method for generating a three-dimensional model provided by the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the method for correcting point cloud and / or the method for generating a three-dimensional model provided by the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising computer programs / instructions, which, when executed by a processor, implement the method for correcting point cloud and / or the method for generating a three-dimensional model provided by the present disclosure.

[0011] It should be understood that the content described in this section is not intended to identify key or important features of embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0013] Figure 1 is an application scenario diagram of the point cloud correction method and the three-dimensional model generation method and the device according to the embodiments of the present disclosure;

[0014] Figure 2 is a flow diagram of the point cloud correction method according to the embodiments of the present disclosure;

[0015] Figure 3 is a schematic diagram of the implementation principle of the point cloud correction method according to the embodiments of the present disclosure;

[0016] Figure 4 is a schematic diagram of the principle of determining the depth value corresponding to the second pixel point according to the embodiments of the present disclosure;

[0017] Figure 5 is a flow diagram of the three-dimensional model generation method according to the embodiments of the present disclosure;

[0018] Figure 6 is an effect diagram of the three-dimensional model generated by the three-dimensional model generation method according to the embodiments of the present disclosure;

[0019] Figure 7 is a structural block diagram of the point cloud data correction device according to the embodiments of the present disclosure;

[0020] Figure 8 is a structural block diagram of the three-dimensional model generation device according to the embodiments of the present disclosure; and

[0021] Figure 9 is a block diagram of an electronic device for implementing the point cloud data correction method and / or the three-dimensional model generation method according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0023] In a three-dimensional reconstruction technique, a three-dimensional model of an object is usually obtained based on point cloud data of the object generated from images. The point cloud data is generated based on depth estimation results obtained for individual images. In the process of implementing the present disclosure, the inventors have found that the accuracy of the point cloud data is difficult to guarantee due to influences of light or depth estimation accuracy, etc. Due to the limitation of the accuracy of the point cloud data, there is usually a case that a generated three-dimensional model is distorted in a ground plane and / or a wall plane, and there is a gap between two intersecting planes.

[0024] To solve the technical problem, the present disclosure provides a point cloud correction method and a three-dimensional model generation method, which corrects the point cloud of an object to improve the accuracy of a reconstructed three-dimensional model.

[0025] The following first combines Figure 1 The application scenario of the method and device provided by the present disclosure is described.

[0026] Figure 1 The application scenario of the point cloud correction method and the three-dimensional model generation method, device according to the embodiments of the present disclosure is shown in the figure.

[0027] As Figure 1 shown, the application scenario 100 of the embodiments can include an electronic device 110, which can be various electronic devices with processing functions, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, servers, etc.

[0028] The electronic device 110 can process the input image 120, for example, which can first generate point cloud data of an object according to the image 120, and then correct the point cloud data by a plane fitting algorithm, etc. to obtain corrected point cloud data.

[0029] In an embodiment, the object whose point cloud data needs to be corrected can include two adjacent objects with an intersection line, for example. The electronic device can first correct the point cloud data of one of the two objects, and then correct the point cloud data of the other object according to the correction result, so as to ensure that there is no gap between the two objects in the three-dimensional model generated according to the corrected point cloud data, and both of the two objects are planar structures.

[0030] In an embodiment, the electronic device 110 can also generate a three-dimensional model 130 of the scene described by the image 120 according to the corrected point cloud data. For example, the three-dimensional model 130 can be represented by the corrected point cloud data.

[0031] In an embodiment, the application scenario 100 can further include a server 150. The electronic device 110 can be communicatively connected with the server 150 through a network. The network can include wired or wireless communication links.

[0032] Exemplarily, the electronic device 110 can send the image 120 to the server 150, and the server 150 can process the image 120, so as to correct the point cloud data of the planar structure object in the image. The server 150 can send the corrected point cloud data to the electronic device 110, so that the electronic device 110 generates the three-dimensional model 130 according to the corrected point cloud data. Alternatively, the image 120 can be processed by the electronic device 110 to obtain the corrected point cloud data 140, and the electronic device 110 can send the corrected point cloud data to the server 150, so that the server 150 generates the three-dimensional model 130 according to the corrected point cloud data. Alternatively, the server 150 can directly generate the three-dimensional model 130 according to the corrected point cloud data 140 after correcting the point cloud data.

[0033] It can be understood that the server 150 can be a background management server supporting the running of a client application in the electronic device 110, or a virtual server, and the present disclosure does not limit the same.

[0034] It should be noted that the point cloud correction method and / or the three-dimensional model generation method provided by the present disclosure can be executed by the electronic device 110 or the server 150. Accordingly, the point cloud correction device and / or the three-dimensional model generation device provided by the present disclosure can be arranged in the electronic device 110 or the server 150.

[0035] It should be understood that Figure 1 The number and type of the electronic device 110 and the server 150 in the application scenario 100 are only illustrative. According to the implementation needs, there can be any number and type of electronic device 110 and server 150.

[0036] The point cloud correction method provided by the present disclosure will be described in detail below. Figures 2-4 The point cloud correction method provided by the present disclosure will be described in detail below.

[0037] Figure 2 is a flowchart of the point cloud correction method according to an embodiment of the present disclosure.

[0038] As shown in Figure 2 The point cloud correction method 200 of the embodiment can include operation S210 to operation S250.

[0039] In operation S210, a plane model representing a plane where the first object is located is obtained by performing plane fitting on the first point cloud data of the first object.

[0040] According to an embodiment of the present disclosure, the first point cloud data of the first object can be generated in advance according to an image including the first object. For example, a deep learning model can be employed to process the image including the first object, so as to obtain the point cloud data of the first object. The deep learning model can be a 3D point cloud generation network (3D-point-cloud generation) that uses 2D convolution operation to predict 3D structure from multiple viewpoints and jointly applies geometric reasoning and 2D projection optimization. The deep learning model can also be a Virtual 3D Scanner that is used to simulate a real-world 3D scanner in a virtual environment.

[0041] According to an embodiment of the present disclosure, a plane detection algorithm can be employed to perform plane fitting on the first point cloud data. The plane detection algorithm can include, for example, a Random Sample Consensus (RANSAC) or a plane detection algorithm based on Hough transform.

[0042] Taking RANSAC as an example, the step of obtaining the plane model can include: first randomly selecting three points in the spatial points represented by the first point cloud data, and calculating a plane equation Ax+By+Zz+D=0 corresponding to the three points. Subsequently, the algebraic distance of all spatial points represented by the first point cloud data to the plane is calculated, for example, for the ith spatial point, the algebraic distance d i i i to the plane is |Ax i i +By i i +Zz i +D|. Subsequently, a threshold value of the algebraic distance is selected. If the algebraic distance of a spatial point to the plane is less than or equal to the threshold value, the spatial point is determined to be an inlier sample point of the plane model, otherwise the spatial point is determined to be an outlier sample point of the plane model. The above steps are repeated to select the best fitting parameters, i.e., the model parameters corresponding to the plane model with the largest number of inlier sample points. The plane model representing the plane where the first object is located can be represented by the best fitting parameters.

[0043] In operation S220, the first point cloud data is corrected according to the plane model, to obtain corrected point cloud data.

[0044] This embodiment can determine the projection point of the spatial point represented by the first point cloud data on the plane represented by the plane model, replace the coordinate value representing the spatial position of the spatial point in the first point cloud data with the coordinate value of the projection point, and correct the first point cloud data.

[0045] In operation S230, target point cloud data corresponding to a target boundary line in the target image is determined from the corrected point cloud data.

[0046] According to an embodiment of the present disclosure, the target boundary line is a boundary line between the first object and the second object in the target image. The first point cloud data can be understood as point cloud data corresponding to a pixel point where the first object is located in the target image. Based on the target image, second point cloud data of the second object can also be obtained.

[0047] According to an embodiment of the present disclosure, the target boundary line can be obtained by prior labeling. According to the correspondence between the pixel point where the target boundary line is located and the first point cloud data, the target point cloud data can be selected from the point cloud data of the first object.

[0048] According to an embodiment of the present disclosure, the target boundary line can also be obtained by using a HohoNet network framework. The HohoNet network framework is a multi-functional and efficient network framework for comprehensively understanding indoor 360-degree panoramas by using low-level ground panoramas. In this embodiment, the target image is input into the network framework, and the network framework outputs the boundary line between each object. For example, if the first object is a wall and the second object is the ground, the HohoNet network framework can be used to detect the boundary line between the wall and the ground, which is the target boundary line. After obtaining the target boundary line, the target point cloud data corresponding to the target boundary line in the first point cloud data can be determined according to the correspondence between the pixels in the target image and the point cloud data.

[0049] In operation S240, the depth value corresponding to the second pixel point is determined according to the target point cloud data based on the positional relationship between the first pixel point corresponding to the target point cloud data in the target image and the second pixel point corresponding to the second object in the target image.

[0050] According to an embodiment of the present disclosure, the first pixel point corresponding to the target point cloud data is the pixel point where the target boundary line is located.

[0051] According to an embodiment of the present disclosure, the center position of the lower edge line of the target image is the projection point of the center point of the image acquisition device (such as a camera) on the image plane. For each second pixel point, the embodiment can first determine the pixel point at the same height as the each second pixel point in the image from the first pixel points as a reference pixel point. Then, the embodiment can determine the pixel distance between the reference pixel point and the each second pixel point. According to the pixel distance, the depth value of the spatial point corresponding to the reference pixel point, and the projection point of the camera on the image plane, the depth value of the each second pixel point is inferred by using the principle of triangular measurement. It can be understood that this method is based on the premise that the vertical distance between the multiple pixel points at the same height in the image and the center point of the camera is equal.

[0052] In an embodiment, the depth value corresponding to the second pixel point can also be determined by the principle described below, which is not described here.

[0053] In operation S250, the second point cloud data of the second object is corrected according to the depth value.

[0054] It can be understood that the second point cloud data of the second object can be generated in advance according to the target image. The second point cloud data can be generated in advance or in real time, which is not limited by the present disclosure. It can be understood that the generation principle of the second point cloud data can be similar to the generation principle of the first point cloud data described above, which is not described here.

[0055] The embodiment can use the depth value corresponding to the second pixel point obtained in operation S240 to correct the depth value included in the second point cloud data corresponding to the second pixel point. Specifically, the depth value obtained in operation S240 can be used to replace the depth value included in the corresponding second point cloud data.

[0056] The embodiment of the present disclosure can first perform plane fitting on the point cloud data of the first object, adjust the point cloud data of the first object according to the plane fitting result, infer the depth value of the pixel point corresponding to the second object according to the adjusted point cloud data of the first object, and adjust the point cloud data of the second object according to the depth value. The adjusted first point cloud data can accurately express the first object with a planar structure, and can compensate for the misalignment of the point cloud data between the first object and the second object expressed by the first point cloud data due to the adjustment of the first point cloud data. In this way, it can be ensured that there is no gap between the first object and the second object in the three-dimensional model generated based on the adjusted point cloud data, and the object is a planar structure. Therefore, the accuracy of the scene expressed by the corrected point cloud data can be improved, and the authenticity of the generated three-dimensional model can be improved.

[0057] In an embodiment, before the point cloud data is corrected, the point cloud data to be corrected can be generated according to the image, for example, so as to facilitate the implementation of the point cloud data correction process. The following will be described in combination with Figure 3 The implementation principle of the point cloud correction method of the embodiment of the present disclosure is described.

[0058] Figure 3 is a diagram of the implementation principle of the point cloud correction method according to the embodiment of the present disclosure.

[0059] As Figure 3As shown in the embodiment 300, before the point cloud data is corrected, a target image 310 including a first object and a second object can be obtained. The first object and the second object are two objects adjacent in an actual scene. The included angle between the two objects can be, for example, approximately 90°. For example, the first object can be a wall, and the second object can be a ground; or the first object can be a ground, and the second object can be a wall. It can be understood that the first object and the second object are only examples for facilitating understanding of the present disclosure, and the first object and the second object can be any two objects having the same boundary line in an actual scene, which is not limited in the present disclosure.

[0060] As shown in the embodiment 300, before the point cloud data is corrected, a target image 310 including a first object and a second object can be obtained. The first object and the second object are two objects adjacent in an actual scene. The included angle between the two objects can be, for example, approximately 90°. For example, the first object can be a wall, and the second object can be a ground; or the first object can be a ground, and the second object can be a wall. It can be understood that the first object and the second object are only examples for facilitating understanding of the present disclosure, and the first object and the second object can be any two objects having the same boundary line in an actual scene, which is not limited in the present disclosure. Figure 3 As shown in the embodiment 300, after the target image 310 is obtained, the embodiment can generate point cloud data 320 for the target image according to the target image 310. For example, the embodiment can first use a sparse point cloud network to generate sparse point cloud data for each image, and then input the sparse point cloud data into a dense model (Dense Module) to generate dense point cloud. The sparse point cloud network can include an encoder and a decoder. The encoder is composed of a convolutional network, and the decoder is composed of a deconvolutional network and a convolutional network. The dense model can process the sparse point cloud data through a feature extraction operation and a feature expansion operation. The dense point cloud data output by the dense model can be used as the point cloud data 320 for the target image. It can be understood that the point cloud data 320 can include point cloud data corresponding to each pixel point in the target image 310.

[0061] After the point cloud data 320 is obtained, the embodiment can determine first point cloud data 321 in the point cloud data according to the pixel points corresponding to the first object in the target image (for the sake of convenience, referred to as third pixel points 311). For example, the embodiment can use a target detection model to process the target image 310 to detect the position of the first object in the target image 310, which can be represented by the coordinate values of the bounding box of the first object. The embodiment can use the pixel points in the bounding box of the first object as the pixel points corresponding to the first object. Then, the embodiment can determine the first point cloud data according to the correspondence between the pixel points and the point cloud data.

[0062] Similarly, after the point cloud data 320 is obtained, the embodiment can also determine second point cloud data 322 in the point cloud data according to the pixel points corresponding to the second object in the target image (for the sake of convenience, referred to as second pixel points 312). It can be understood that the determination principle of the second point cloud data 322 is similar to that of the first point cloud data 321, which will not be described herein.

[0063] After obtaining the first point cloud data 321, a plane detection algorithm 330 can be employed to perform plane fitting on the first point cloud data 321 to obtain a plane model 340. The principle of obtaining the plane model 340 is similar to the operation S210 described above, and thus will not be repeated here.

[0064] After obtaining the plane model 340, the first point cloud data 321 can be corrected according to the plane model 340 to obtain corrected point cloud data 350. Subsequently, the embodiment can employ a principle similar to the operations S230-S240 described above to determine the depth value 360 corresponding to the second pixel point 312 according to the depth value included in the target point cloud data in the corrected point cloud data 350, and correct the second point cloud data 322 of the second object according to the depth value 360.

[0065] In an embodiment, when determining the first point cloud data 321 and the second point cloud data 322, a deep learning model can be first employed to process the target object 310 to obtain the pixel positions of the boundary lines between the first object and other objects in the target image. It can be understood that the boundary lines between the first object and other objects include the boundary line between the first object and the second object. The deep learning model can employ the HohoNet network framework described above, for example. For example, if the first object is a wall, the detected boundary lines can include the boundary line between the wall and the ceiling, the boundary line between the wall and the ground, etc.

[0066] Subsequently, the pixel points corresponding to the first object in the target image can be determined according to the pixel positions of the boundary lines. For example, the pixel points included in the pixel region between the boundary line between the wall and the ground and the boundary line between the wall and the ceiling can be taken as the pixel points corresponding to the first object. Meanwhile, the embodiment can also determine the pixel points corresponding to the second object in the target image 310 according to the pixel positions of the boundary line between the first object and the second object. For example, the embodiment can take the closed region surrounded by the boundary line between the first object and the second object and the edge line of the target image as the region where the second object is located, and then take the pixel points in the closed region as the pixel points corresponding to the second object.

[0067] The embodiment can improve the accuracy of the obtained pixel points by employing a deep learning model to determine the boundary lines and determining the pixel points according to the boundary lines. Furthermore, when employing the HohoNet network framework, the boundary lines can be detected simultaneously when obtaining the point cloud data for the target image 310, which can improve the implementation efficiency of the point cloud correction method and improve the correction effect.

[0068] The principle of obtaining the depth value in the operation S240 will be further expanded and limited below.

[0069] Figure 4 This is a schematic diagram illustrating the principle of determining the depth value corresponding to the second pixel point according to an embodiment of the present disclosure.

[0070] like Figure 4 As shown, in this embodiment 400, when determining the depth value corresponding to the second pixel, for example, the pixel in the target image 410 corresponding to the center of the virtual camera for the target image can be determined first, and that pixel can be used as the center pixel. For example, if the target image 410 is a panoramic image, the center point 411 of the lower edge of the image can be used as the center pixel, or the center point of the image can be used as the center pixel; this disclosure does not limit this. It can be understood that the pixel corresponding to the center of the virtual camera for the target image can be understood as the projection point of the optical center of the camera acquiring the target image on the phase plane of the target image. For example, in the target image 410, curve 420 is the boundary line between the first object and the second object, and each pixel where curve 420 is located is the first pixel described above.

[0071] After determining the center pixel 411, this embodiment can, for example, determine the lines connecting the center pixel to each pixel in the first pixel group to obtain the target connection. For example, for pixel 421 on curve 420, the line connecting pixel 421 and center pixel 411 is a straight line 430.

[0072] After determining the straight line 430, this embodiment can determine the target pixel point located on the target connection line among the second pixel points corresponding to the second object. For example, the second pixel point located on the connection line between pixel point 421 and the center pixel point 411 may include multiple pixels such as pixel point 441.

[0073] In this embodiment 400, the depth value of the target pixel can be determined, for example, based on the pixel distance between each pixel in the first pixel and the target pixel on the line connecting them. Alternatively, the depth value of the target pixel can be determined based on the depth values ​​included in the target point cloud data of each pixel.

[0074] For example, for pixel 441, the depth value of pixel 441 can be determined based on the pixel distance between pixel 441 and pixel 421, and the depth value included in the point cloud data corresponding to pixel 421. For instance, if the distance between pixel 441 and pixel 421 is set to D, the depth value included in the point cloud data corresponding to pixel 421 is set to d1, and the depth value corresponding to the center pixel is set to a predetermined value d0, then the depth value d2 corresponding to pixel 441 can be calculated using the following formula:

[0075] d2 = d1 + d1 * D / (d1 - d0).

[0076] Specifically, in an embodiment, the depth value corresponding to the center pixel point is set as a predetermined value d0, and the lethality can be determined according to the pixel distance and the depth value included in the target point cloud data corresponding to each pixel point by using a linear interpolation algorithm.

[0077] It can be understood that the principle of determining the depth value corresponding to the target pixel point is only an example for facilitating understanding of the present disclosure, and the present disclosure is not limited in this regard.

[0078] Based on the point cloud correction method provided in the present disclosure, the present disclosure further provides a three-dimensional model generation method, which will be described below in combination with Figures 5-6 The method will be described in detail.

[0079] Figure 5 is a flowchart of the three-dimensional model generation method according to an embodiment of the present disclosure.

[0080] As Figure 5 shown, the three-dimensional model generation method 500 of the embodiment can include operation S510 to operation S530.

[0081] In operation S510, initial point cloud data corresponding to a target image is generated according to the target image including a first object and a second object.

[0082] It can be understood that the initial point cloud data includes first point cloud data of the first object and second point cloud data of the second object. The principle of generating the initial point cloud data by the embodiment may, for example, be similar to the principle of generating the point cloud data for the target image described above, and will not be described herein again.

[0083] After obtaining the initial point cloud data, the embodiment can also determine the first point cloud data of the first object and the second point cloud data of the second object in the initial point cloud data by using a similar method as described above.

[0084] In operation S520, the first point cloud data and the second point cloud data are corrected to obtain corrected point cloud data.

[0085] In operation S530, a three-dimensional model is generated according to the corrected point cloud data.

[0086] The operation S520 can implement the correction of the first point cloud data and the second point cloud data by using the point cloud correction method described above, and will not be described herein again. The three-dimensional model of the scene described by the target image can be represented by the corrected point cloud data.

[0087] The generation method of the three-dimensional model according to the embodiments of the present disclosure can improve the authenticity of the obtained three-dimensional model by correcting the point cloud data first and then using the corrected point cloud data to represent the three-dimensional model. This is because the first point cloud data of the first object is corrected based on the plane model obtained by plane fitting, so that the three-dimensional model obtained can ensure that the first object is a plane structure, so that the three-dimensional model can more truly express the three-dimensional structure of the first object.

[0088] The following will be described in combination with Figure 6 The effect of the three-dimensional model generated according to the corrected point cloud data is described.

[0089] Figure 6 is an effect diagram of the three-dimensional model generated according to the generation method of the three-dimensional model according to the embodiments of the present disclosure.

[0090] As Figure 6 shown, in the embodiment 600, the first object is set as a wall, and the second object is set as a ground. If a three-dimensional model is generated according to the initial point cloud data, the three-dimensional model obtained is model 610. If the point cloud correction method provided by the present disclosure is used to correct the first point cloud data and the second point cloud data first, and then a three-dimensional model is generated according to the corrected point cloud data, the model 620 can be obtained.

[0091] It can be seen that the wall surface 611 in the model 610 and the junction line between the wall surface 611 and the ground 612 are distorted, which is not consistent with the real scene. By correcting the point cloud data of the wall surface, the wall surface 621 in the model 620 generated is a plane structure, which is consistent with the real scene. Furthermore, by correcting the point cloud data of the ground based on the corrected point cloud data of the wall surface, the joint between the ground 622 and the wall surface 621 is complete, and there is no gap, which is consistent with the real scene.

[0092] Based on the point cloud correction method provided by the present disclosure, the present disclosure further provides a point cloud correction device. The following will be described in combination with Figure 7 The device is described in detail.

[0093] Figure 7 is a structure block diagram of the point cloud correction device according to the embodiments of the present disclosure.

[0094] As Figure 7 shown, the point cloud correction device 700 of the embodiment can include a plane fitting module 710, a first correction module 720, a boundary point cloud determination module 730, a depth value determination module 740, and a second correction module 750.

[0095] The plane fitting module 710 is configured to perform plane fitting on the first point cloud data of the first object to obtain a plane model representing a plane where the first object is located. In an embodiment, the plane fitting module 710 can be configured to perform the operation S210 described above, and details are not repeated here.

[0096] The first correction module 720 is configured to correct the first point cloud data according to the plane model to obtain corrected point cloud data. In an embodiment, the first correction module 720 can be configured to perform the operation S220 described above, and details are not repeated here.

[0097] The boundary point cloud determination module 730 is configured to determine target point cloud data corresponding to a target boundary line in the target image from the corrected point cloud data, the target boundary line being a boundary line between the first object and the second object in the target image. In an embodiment, the boundary point cloud determination module 730 can be configured to perform the operation S230 described above, and details are not repeated here.

[0098] The depth value determination module 740 is configured to determine a depth value corresponding to a second pixel point of the second object according to the target point cloud data based on a positional relationship between the first pixel point corresponding to the target point cloud data in the target image and the second pixel point corresponding to the second object in the target image. In an embodiment, the depth value determination module 740 can be configured to perform the operation S240 described above, and details are not repeated here.

[0099] The second correction module 750 is configured to correct second point cloud data of the second object according to the depth value. In an embodiment, the second correction module 750 can be configured to perform the operation S250 described above, and details are not repeated here.

[0100] According to an embodiment of the present disclosure, the depth value determination module 740 described above can include: a center point determination sub-module configured to determine a pixel point corresponding to a virtual camera center of the target image as a center pixel point; a line determination sub-module configured to determine a line between the center pixel point and each of the first pixel points as a target line; a target point determination sub-module configured to determine a target pixel point of the second pixel points on the target line; and a depth determination sub-module configured to determine a depth value corresponding to the target pixel point according to the target point cloud data corresponding to each of the pixel points based on a pixel distance between each of the pixel points and the target pixel point.

[0101] According to an embodiment of the present disclosure, the depth determination sub-module is configured to determine the depth value corresponding to the target pixel point by using a linear interpolation algorithm according to the pixel distance and a depth value in the target point cloud data corresponding to each of the pixel points. The depth value corresponding to the center pixel point is a predetermined value.

[0102] According to an embodiment of the present disclosure, the apparatus 700 can further include a point cloud generation module configured to generate point cloud data corresponding to the target image according to the target image; a first point cloud determination module configured to determine first point cloud data in the point cloud data according to the pixel points corresponding to the first object in the target image; and a second point cloud determination module configured to determine second point cloud data in the point cloud data according to the pixel points corresponding to the second object in the target image.

[0103] According to an embodiment of the present disclosure, the apparatus 700 can further include a boundary line determination module configured to process the target image by using a deep learning model to obtain pixel positions of a boundary line between the first object and other objects except the first object in the target image; the boundary line between the first object and the other objects includes a boundary line between the first object and the second object; a first pixel point determination module configured to determine the pixel points corresponding to the first object in the target image according to the pixel positions; and a second pixel point determination module configured to determine the pixel points corresponding to the second object in the target image according to the pixel positions of the boundary line between the first object and the second object.

[0104] According to an embodiment of the present disclosure, one of the first object and the second object is a wall, and the other of the first object and the second object is a ground.

[0105] Based on the method for generating a three-dimensional model provided in the present disclosure, the present disclosure further provides a device for generating a three-dimensional model. The following will be described in detail in combination with Figure 8 The device will be described in detail.

[0106] Figure 8 is a structural block diagram of the device for generating a three-dimensional model according to an embodiment of the present disclosure.

[0107] As Figure 8 shown, the device 800 for generating a three-dimensional model according to the embodiment can include a point cloud generation module 810, a point cloud correction module 820, and a model generation module 830.

[0108] The point cloud generation module 810 is configured to generate initial point cloud data corresponding to a target image according to the target image including a first object and a second object; the initial point cloud data includes first point cloud data of the first object and second point cloud data of the second object. In an embodiment, the point cloud generation module 810 can be configured to perform the operation S510 described above, and thus no further description is given here.

[0109] The point cloud correction module 820 is configured to correct the first point cloud data and the second point cloud data by using the device for correcting point cloud provided in the present disclosure to obtain corrected point cloud data. In an embodiment, the point cloud correction module 820 can be configured to perform the operation S520 described above, and thus no further description is given here.

[0110] The model generation module 830 is used to generate a 3D model based on the corrected point cloud data. In one embodiment, the model generation module 830 can be used to perform the operation S530 described above, which will not be repeated here.

[0111] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information in this disclosed technical solution comply with relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals. In this disclosed technical solution, user authorization or consent has been obtained before acquiring or collecting user personal information.

[0112] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0113] Figure 9 A schematic block diagram of an example electronic device 900 is shown, which can be used to implement a point cloud correction method and / or a 3D model generation method for embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0114] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0115] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs various methods and processes described above, such as the correction method of point cloud and / or the generation method of three-dimensional model. For example, in some embodiments, the correction method of point cloud and / or the generation method of three-dimensional model can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the correction method of point cloud and / or the generation method of three-dimensional model described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the correction method of point cloud and / or the generation method of three-dimensional model by any other appropriate means, such as by means of firmware.

[0117] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0118] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0121] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between clients and servers arises by interplay between programs running on the respective computers and having a client-server relationship. Among other things, the server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short) services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0123] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0124] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method for correcting point cloud, comprising: performing plane fitting according to first point cloud data of a first object to obtain a plane model representing a plane where the first object is located; correcting the first point cloud data according to the plane model to obtain corrected point cloud data; determining target point cloud data corresponding to a target boundary line in a target image from the corrected point cloud data, the target boundary line being a boundary line between the first object and a second object in the target image; determining a depth value corresponding to a second pixel point of the second object according to the target point cloud data based on a positional relationship between a first pixel point corresponding to the target point cloud data and the second pixel point in the target image; and correcting second point cloud data of the second object according to the depth value. The determining of the depth value corresponding to the second pixel point according to the target point cloud data based on the positional relationship between the first pixel point and the second pixel point in the target image comprises: determining a center pixel point corresponding to a virtual camera center of the target image as a center pixel point; determining a target line connecting each of the first pixel point and the center pixel point as a target line; determining a target pixel point of the second pixel point on the target line; and determining the depth value corresponding to the target pixel point according to target point cloud data corresponding to each of the first pixel point and the center pixel point based on a pixel distance between each of the first pixel point and the center pixel point and the target pixel point. The determining of the depth value corresponding to the target pixel point according to the target point cloud data corresponding to each of the first pixel point and the center pixel point based on the pixel distance between each of the first pixel point and the center pixel point and the target pixel point comprises:

2. The method of claim 1, wherein, determining the depth value corresponding to the target pixel point using a linear interpolation algorithm according to the pixel distance and a depth value in the target point cloud data corresponding to each of the first pixel point and the center pixel point. 3.The method of claim 1, further comprising: generating point cloud data for the target image according to the target image; determining the first point cloud data in the point cloud data according to a pixel point corresponding to the first object in the target image; and determining the second point cloud data in the point cloud data according to a pixel point corresponding to the second object in the target image. 4.The method of claim 3, further comprising: processing the target image using a deep learning model to obtain a pixel position of a boundary line between the first object and other objects except the first object in the target image; the boundary line between the first object and the other objects comprises the boundary line between the first object and the second object; determining the pixel point corresponding to the first object in the target image according to the pixel position; and determining the pixel point corresponding to the second object in the target image according to the pixel position of the boundary line between the first object and the second object. ​ ​ ​ ​ 5. The method of any one of claims 1-4, wherein, One of the first object and the second object is a wall; the other of the first object and the second object is a ground.

6. A method for generating a three-dimensional model, comprising: generating initial point cloud data corresponding to a target image including a first object and a second object according to the target image; the initial point cloud data includes first point cloud data of the first object and second point cloud data of the second object; correcting the first point cloud data and the second point cloud data by using the point cloud correction method in any one of claims 1-5 to obtain corrected point cloud data; and generating the three-dimensional model according to the corrected point cloud data.

7. An apparatus for correcting point cloud data, comprising: a plane fitting module configured to perform plane fitting on first point cloud data of a first object to obtain a plane model representing a plane where the first object is located; a first correction module configured to correct the first point cloud data according to the plane model to obtain corrected point cloud data; a boundary point cloud determination module configured to determine target point cloud data in the corrected point cloud data corresponding to a target boundary line in a target image, the target boundary line being a boundary line between the first object and a second object in the target image; a depth value determination module configured to determine a depth value corresponding to a second pixel in the target image according to the target point cloud data based on a positional relationship between the second pixel and a first pixel in the target image corresponding to the target point cloud data; and a second correction module configured to correct second point cloud data of the second object according to the depth value; wherein the depth value determination module comprises: a center point determination submodule configured to determine a pixel in the target image corresponding to a virtual camera center for the target image as a center pixel; a line determination submodule configured to determine a line between the center pixel and each of the first pixels to obtain a target line; a target point determination submodule configured to determine a target pixel in the second pixels located on the target line; and a depth determination submodule configured to determine the depth value corresponding to the target pixel according to target point cloud data corresponding to the each of the pixels based on a pixel distance between the each of the pixels and the target pixel.

8. The apparatus of claim 7, wherein, The depth determination submodule is configured to: determine the depth value corresponding to the target pixel according to the pixel distance and a depth value in the target point cloud data corresponding to the each of the pixels by using a linear interpolation algorithm, wherein the depth value corresponding to the center pixel is a predetermined value.

9. The apparatus of claim 7, further comprising: a point cloud generation module configured to generate point cloud data for the target image according to the target image; a first point cloud determination module configured to determine the first point cloud data in the point cloud data according to pixels in the target image corresponding to the first object; and a second point cloud determination module configured to determine the second point cloud data in the point cloud data according to pixels in the target image corresponding to the second object. ​ 10. The apparatus of claim 9, further comprising: a boundary line determination module configured to process the target image using a deep learning model to obtain pixel positions of boundary lines between the first object and other objects than the first object in the target image; the boundary lines between the first object and the other objects include a boundary line between the first object and a second object; a first pixel point determination module configured to determine pixel points corresponding to the first object in the target image according to the pixel positions; and a second pixel point determination module configured to determine pixel points corresponding to the second object in the target image according to the pixel positions of the boundary line between the first object and the second object. One of the first object and the second object is a wall, and the other of the first object and the second object is a ground.

11. The apparatus of any one of claims 7-10, wherein, 12. An apparatus for generating a three-dimensional model, comprising: a point cloud generation module configured to generate initial point cloud data corresponding to a target image including a first object and a second object according to the target image; the initial point cloud data includes first point cloud data of the first object and second point cloud data of the second object; a point cloud correction module configured to correct the first point cloud data and the second point cloud data using the apparatus of any one of claims 7-11 to obtain corrected point cloud data; and a model generation module configured to generate the three-dimensional model according to the corrected point cloud data.

13. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6. The computer instructions are used to enable the computer to perform the method of any one of claims 1-6.

15. A computer program product comprising computer programs / instructions stored on at least one of a readable storage medium and an electronic device, the computer programs / instructions, when executed by a processor, implement the steps of the method of any one of claims 1-6.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, ​ ​

Images