Room point cloud data processing method and device, electronic device and storage medium

Abstract

The present application discloses a method and device for processing room point cloud data, an electronic device, and a storage medium. The method for processing room point cloud data includes: generating an initial spatial body model corresponding to the initial room point cloud data; determining a surface network model corresponding to the initial spatial body model; performing denoising processing on the initial room point cloud data according to the vertex data of the surface network model to obtain denoised point cloud data after denoising; determining the ground normal vector of the denoised point cloud data, performing coordinate rotation on the denoised point cloud data according to the ground normal vector to obtain rotated point cloud data after rotation; performing error registration processing on the rotated point cloud data to obtain processed target point cloud data. The embodiments of the present application can effectively reduce calibration errors, measurement noise, etc., improve point cloud accuracy and modeling correctness, make the room VR model generated according to the processed point cloud data more accurate, effectively reduce the steps of manual correction, and improve modeling efficiency.

Description

Technical Field

[0001] The present application belongs to the field of VR (Virtual Reality) technology, and specifically relates to a method and device for processing room point cloud data, an electronic device, and a storage medium. Background Art

[0002] In the related art, in order to improve the efficiency of house viewing, VR house viewing scenarios have emerged. Before VR house viewing, it is necessary to shoot the room with a camera to obtain a three-dimensional point cloud model, and then build a VR model of the room based on the point cloud model.

[0003] However, the point cloud data directly shot has calibration errors, measurement noise, etc., which leads to inaccurate VR model of the room and low modeling efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and device for processing room point cloud data, an electronic device and a storage medium, which can solve the problems of inaccurate room VR model and low modeling efficiency.

[0005] In a first aspect, an embodiment of the present application provides a method for processing room point cloud data, comprising:

[0006] Generate an initial space volume model corresponding to the initial room point cloud data;

[0007] Determine the surface network model corresponding to the initial space body model;

[0008] According to the vertex data of the surface network model, the initial room point cloud data is subjected to denoising to obtain denoised point cloud data after denoising;

[0009] Determine the ground normal vector of the de-noised point cloud data, and perform coordinate rotation on the de-noised point cloud data according to the ground normal vector to obtain rotated point cloud data;

[0010] The rotation point cloud data is processed by error registration to obtain the processed target point cloud data.

[0011] In a second aspect, an embodiment of the present application provides a device for processing room point cloud data, including:

[0012] A generation module, used to generate an initial space body model corresponding to the initial room point cloud data;

[0013] A determination module, used to determine the surface network model corresponding to the initial space body model;

[0014] A processing module, used for performing noise reduction processing on the initial room point cloud data according to the vertex data of the surface network model to obtain noise-reduced point cloud data after noise reduction;

[0015] A rotation module is used to determine the ground normal vector of the de-noised point cloud data, and to perform coordinate rotation on the de-noised point cloud data according to the ground normal vector to obtain rotated point cloud data;

[0016] The processing module is also used to perform error registration processing on the rotated point cloud data to obtain processed target point cloud data.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method of the first aspect.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0021] In the embodiment of the present application, a body model of the original point cloud data is generated, and the noise data contained in the original point cloud data is removed according to the vertex data of the surface network model of the body model. After the noise reduction is completed, the coordinates of the noise-reduced point cloud data are rotated according to the generated ground normal vector, so as to deal with the skew error of the point cloud data, and finally the room point cloud data that has been subjected to the skew error removal and noise reduction processing is further error registration processing, so as to effectively reduce the calibration error, measurement noise, etc., improve the accuracy of the point cloud and the correctness of modeling, so that the room VR model generated according to the processed point cloud data is more accurate, effectively reduce the steps of manual correction, and improve the modeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flow chart showing a method for processing room point cloud data according to an embodiment of the present application is shown;

[0023] Figure 2 A logical schematic diagram of processing point cloud depth noise according to an embodiment of the present application is shown;

[0024] Figure 3 A logical schematic diagram of processing point cloud point skew errors according to an embodiment of the present application is shown;

[0025] Figure 4 A logical schematic diagram of processing point cloud registration errors according to an embodiment of the present application is shown;

[0026] Figure 5 A structural block diagram of a device for processing room point cloud data according to an embodiment of the present application is shown;

[0027] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present application is shown;

[0028] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0030] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish between classes of objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0031] In the following, in combination with the accompanying drawings, the room point cloud data processing method and device, electronic device and storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0032] In some embodiments of the present application, a method for processing room point cloud data is provided. Figure 1 A flow chart of a method for processing room point cloud data according to an embodiment of the present application is shown. Figure 1 As shown, the method includes:

[0033] Step 102, generating an initial space body model corresponding to the initial room point cloud data;

[0034] Step 104, determining the surface network model corresponding to the initial space body model;

[0035] Step 106, performing denoising processing on the initial room point cloud data according to the vertex data of the surface network model to obtain denoised point cloud data after denoising;

[0036] Step 108, determining the ground normal vector of the de-noised point cloud data, and performing coordinate rotation on the de-noised point cloud data according to the ground normal vector to obtain rotated point cloud data;

[0037] Step 110 , performing error registration processing on the rotated point cloud data to obtain processed target point cloud data.

[0038] In the embodiment of the present application, when building a VR model, such as a room model for VR house viewing, it is generally necessary to use an RGB-D (Red Green Bled-Depth, three-channel color depth image) camera to shoot the house to obtain an initial room point cloud model. Due to the calibration error and measurement noise of the RGB-D camera, the collected point cloud data contains a lot of noise, resulting in low modeling efficiency.

[0039] Therefore, the present application first performs noise reduction processing on the initial room point cloud data. Specifically, by generating a primitive model of the initial room point cloud data, and removing the noise data contained in the original point cloud data according to the vertex data of the surface network model of the primitive model.

[0040] After the noise reduction is completed, the noise-reduced point cloud data can be obtained. However, due to problems with the shooting environment of the RGB-D camera, such as the tilt of the tripod or the deviation between the camera optical axis and the rotation axis, the overall point cloud data may be skewed, that is, there is an angle between the floor of the point cloud data and the actual ground. In response to this situation, this Shentong generates a ground normal vector from the noise-reduced point cloud data, and uses the ground normal vector to rotate the coordinates of the point cloud data as a whole, thereby straightening the "skewed" point cloud data.

[0041] After the alignment is completed, further error registration processing is performed, which can effectively reduce calibration errors, measurement noise, etc., improve point cloud accuracy and modeling correctness, and make the room VR model generated based on the processed point cloud data more accurate, effectively reducing the steps of manual correction and improving modeling efficiency.

[0042] In some embodiments of the present application, generating an initial spatial volume model corresponding to the initial room point cloud data includes:

[0043] Acquire camera images of at least two shooting angles and shooting data of the camera images, wherein the camera images include a color channel image and a depth information image, and the shooting data include pan / tilt angle information and camera calibration parameters;

[0044] According to the camera image, point cloud data corresponding to at least two shooting angles are respectively established;

[0045] Through the preset algorithm and shooting data, the point cloud data corresponding to at least two shooting angles are fused into an initial spatial volume model.

[0046] In the embodiment of the present application, the camera image is specifically an image taken by an RGB-D camera. Specifically, the camera image includes an RGB (Red Green Blue, red, green and blue three-channel color) image and a Depth image, wherein the depth image is specifically a depth image with noise.

[0047] The camera image and the shooting data of the camera that shot the camera image are combined to construct the camera images of at least two shooting angles into point cloud data at the current shooting angle, and obtain point cloud data at at least two shooting angles. The shooting data includes the pan / tilt angle data of the camera that shot the camera image, and also includes the calibration parameters of the camera.

[0048] After obtaining the point cloud data under at least two shooting angles, the point cloud data under these shooting angles are fused through the TSDF (Truncated Signed Distance Function) algorithm to finally obtain an initial spatial volume model, which is a model constructed using point cloud data containing noise information.

[0049] In some embodiments of the present application, determining a surface network model corresponding to an initial spatial volume model includes:

[0050] The surface network model of the initial space body model is extracted through the marching cube algorithm;

[0051] According to the vertex data of the surface network model, the initial room point cloud data is denoised, including:

[0052] Determines the vertex data as denoised point cloud data.

[0053] In an embodiment of the present application, after obtaining an initial spatial body model constructed according to point cloud data containing noise information, a surface network model of the initial spatial body model is extracted by a marching cube algorithm.

[0054] Specifically, Figure 2 FIG. 4 shows a logic diagram of processing point cloud depth noise according to an embodiment of the present application, as shown in FIG. Figure 2 As shown:

[0055] (1) Obtain the original RGB image and noisy depth map, and reconstruct the current view into a point cloud based on the camera intrinsic parameters.

[0056] (2) Using the Scalable TSDF algorithm, the point clouds captured from different perspectives are fused into a spatial voxel model according to the pan / tilt angle and calibration parameters.

[0057] (3) Use the Marching Cube algorithm to extract the surface mesh model.

[0058] (4) Retain the mesh vertices as the denoised point cloud data at that point.

[0059] The surface network model includes a plurality of grids, each of which corresponds to point data in at least one point cloud data. The vertices of these grids are used as the noise-reduced point cloud data after the noise data of the initial room point cloud data is removed. The noise data in the initial room power data, specifically the deep noise data, is removed by this method, thereby effectively improving the accuracy of the point cloud.

[0060] In some embodiments of the present application, determining a ground normal vector of denoised point cloud data includes:

[0061] Obtain the height information of each point data in the noise-reduced point cloud data;

[0062] The point data whose height information is less than the height threshold is determined as the target point data;

[0063] Fit the target point data into a ground model;

[0064] Determine the ground normal vector based on the ground model.

[0065] In the embodiment of the present application, when processing the point skew error in the point cloud data, first, the height information of each point in the noise-reduced point cloud data after noise reduction is determined. Then, based on the camera's prior knowledge or experience, the points whose heights are less than the threshold are screened.

[0066] Furthermore, the points whose heights are less than the threshold are fitted into a ground model using the RANSAC (Random Sample Consensus) algorithm, which is the model of the floor of the photographed room.

[0067] After fitting the ground model, the ground normal vector of the ground model is calculated. The ground normal vector indicates the angle between the ground normal line and the plumb line, thereby clearly indicating the skew error of the point cloud data, which is beneficial for correcting the point cloud data and improving the modeling accuracy.

[0068] In some embodiments of the present application, coordinate rotation of the noise-reduced point cloud data is performed according to the ground normal vector, including:

[0069] Get the standard coordinate system;

[0070] Determine the rotation matrix between the ground normal vector and the vertical coordinate axis of the standard coordinate system;

[0071] The coordinates of the denoised point cloud data are rotated using the rotation matrix.

[0072] In the embodiment of the present application, the standard coordinate system is the global coordinate system in which the point cloud data is located. Specifically, when generating point cloud data, a standard coordinate system needs to be determined first, and then, according to the position of each scanned point in the RGB-D image, the point is projected into the standard coordinate system, so that all the points captured correspond to a coordinate value (x, y, z) in the standard coordinate system.

[0073] According to the rotation matrix between the ground normal vector and the vertical direction in the standard coordinate system, the rotation matrix is ​​specifically used to correct the ground normal direction corresponding to the ground normal vector to the y-axis direction of the standard coordinate system, thereby completing the correction of the skew error of the point cloud data.

[0074] Figure 3 FIG. 4 shows a logic diagram of processing a point cloud point skew error according to an embodiment of the present application, as shown in FIG. Figure 3 As shown:

[0075] (1) Based on the camera prior knowledge or experience value, the point cloud whose point cloud height is less than the threshold is selected.

[0076] (2) Use the RANSAC algorithm to fit the ground and calculate the ground normal vector.

[0077] (3) Correct the ground normal to the global y-axis direction and calculate the rotation matrix.

[0078] (4) Rotate the point cloud in the rigid coordinate system according to the calculated rotation matrix.

[0079] The embodiment of the present application corrects the skew error of the point cloud point based on the ground normal, so that the ground direction of the generated room VR model is consistent with the real ground direction, without the need for manual correction, thereby improving the model generation efficiency.

[0080] In some embodiments of the present application, error registration processing is performed on the rotated point cloud data, including:

[0081] Get the preset registration algorithm;

[0082] The Jacobian matrix corresponding to the registration algorithm is processed to project the pose update obtained by the registration algorithm into a non-degenerate direction.

[0083] In the embodiment of the present application, when point cloud information is captured by an RGB-D camera, due to the limited visual range of the camera or the limitations of the scene itself, it is easy to produce a situation where the point cloud has less geometric feature information. The use of traditional point cloud registration methods often causes the optimization problem to degenerate, resulting in the two structures being misaligned, causing the point cloud to have arbitrary solutions in insufficiently constrained geometric directions, resulting in erroneous results.

[0084] The embodiment of the present application obtains the Jacobian matrix of the normal equation calculated in the registration algorithm, and processes the Jacobian matrix to project the pose update calculated by the registration algorithm to a non-degenerate direction, thereby improving the accuracy of point cloud data processing.

[0085] In some embodiments of the present application, the Jacobian matrix corresponding to the registration algorithm is processed, including:

[0086] Perform eigendecomposition on the Jacobian matrix to obtain the corresponding eigenvalue matrix and eigenvector matrix, where the eigenvalues ​​in the eigenvalue matrix correspond one-to-one to the eigenvectors in the eigenvector matrix;

[0087] In the eigenvalue matrix, determining a target eigenvalue whose eigenvalue is less than an eigenvalue threshold;

[0088] Set the eigenvector corresponding to the target eigenvalue to the 0 vector.

[0089] In an embodiment of the present application, when processing the Jacobian matrix, first, the Jacobian matrix is ​​eigendecomposed to obtain an eigenvalue matrix corresponding to the Jacobian matrix and an eigenvector matrix corresponding one-to-one to the eigenvalue matrix.

[0090] Then, all eigenvalues ​​in the eigenvalue matrix are traversed, and whenever an eigenvalue is determined to be less than a preset eigenvalue threshold, this eigenvalue is marked as a target eigenvalue, and the eigenvector corresponding to the target eigenvalue is set to a 0 vector.

[0091] By way of example, the feature value threshold may be set to 95 to 105.

[0092] After setting the eigenvectors corresponding to all target eigenvalues ​​to 0 vectors, the processed Jacobian matrix is ​​obtained.

[0093] Figure 4 FIG. 4 shows a logic diagram of processing point cloud registration errors according to an embodiment of the present application, as shown in FIG. Figure 4 As shown:

[0094] (1) Obtain the Jacobian matrix of the normal equation calculated in the registration algorithm.

[0095] (2) Perform eigendecomposition on the Jacobian matrix to obtain the eigenvalue matrix and eigenvector matrix.

[0096] (3) Traverse the eigenvalues, and if the eigenvalue is less than a threshold (such as 100), set the corresponding eigenvector to a 0 vector. Obtain the processed eigenvector matrix.

[0097] (4) Obtain the pose update calculated by the registration algorithm and project the update to the non-degenerate direction as the final result.

[0098] The present application obtains the Jacobian matrix of the normal equation calculated in the registration algorithm and processes the Jacobian matrix, thereby projecting the pose update calculated by the registration algorithm to a non-degenerate direction, thereby improving the accuracy of point cloud data processing.

[0099] In some embodiments of the present application, a device for processing room point cloud data is provided. Figure 5 FIG. 4 shows a structural block diagram of a device for processing room point cloud data according to an embodiment of the present application. Figure 5 As shown, the room point cloud data processing device 500 includes:

[0100] A generating module 502 is used to generate an initial space volume model corresponding to the initial room point cloud data;

[0101] A determination module 504 is used to determine a surface network model corresponding to the initial spatial volume model;

[0102] The processing module 506 is used to perform noise reduction processing on the initial room point cloud data according to the vertex data of the surface network model to obtain noise-reduced point cloud data after noise reduction;

[0103] A rotation module 508 is used to determine a ground normal vector of the de-noised point cloud data, and perform coordinate rotation on the de-noised point cloud data according to the ground normal vector to obtain rotated point cloud data;

[0104] The processing module 506 is further used to perform error registration processing on the rotated point cloud data to obtain processed target point cloud data.

[0105] In the embodiment of the present application, when building a VR model, such as a room model for VR house viewing, it is generally necessary to use an RGB-D (Red Green Bled-Depth, three-channel color depth image) camera to shoot the house to obtain an initial room point cloud model. Due to the calibration error and measurement noise of the RGB-D camera, the collected point cloud data contains a lot of noise, resulting in low modeling efficiency.

[0106] Therefore, the present application first performs noise reduction processing on the initial room point cloud data. Specifically, by generating a primitive model of the initial room point cloud data, and removing the noise data contained in the original point cloud data according to the vertex data of the surface network model of the primitive model.

[0107] After the noise reduction is completed, the noise-reduced point cloud data can be obtained. However, due to problems with the shooting environment of the RGB-D camera, such as the tilt of the tripod or the deviation between the camera optical axis and the rotation axis, the overall point cloud data may be skewed, that is, there is an angle between the floor of the point cloud data and the actual ground. In response to this situation, this Shentong generates a ground normal vector from the noise-reduced point cloud data, and uses the ground normal vector to rotate the coordinates of the point cloud data as a whole, thereby straightening the "skewed" point cloud data.

[0108] After the alignment is completed, further error registration processing is performed, which can effectively reduce calibration errors, measurement noise, etc., improve point cloud accuracy and modeling correctness, and make the room VR model generated based on the processed point cloud data more accurate, effectively reducing the steps of manual correction and improving modeling efficiency.

[0109] In some embodiments of the present application, the device for processing room point cloud data further includes:

[0110] An acquisition module, used to acquire camera images of at least two shooting angles and shooting data of the camera images, wherein the camera images include color channel images and depth information images, and the shooting data include pan / tilt angle information and camera calibration parameters;

[0111] An establishing module, used to establish point cloud data corresponding to at least two shooting angles according to the camera image;

[0112] The fusion module is used to fuse the point cloud data corresponding to at least two shooting angles into an initial space volume model through a preset algorithm and shooting data.

[0113] In the embodiment of the present application, the camera image is specifically an image taken by an RGB-D camera. Specifically, the camera image includes an RGB (Red Green Blue, red, green and blue three-channel color) image and a Depth image, wherein the depth image is specifically a depth image with noise.

[0114] The camera image and the shooting data of the camera that shoots the camera image are combined to construct the camera images at at least two shooting angles into point cloud data at the current shooting angle, thereby obtaining point cloud data at at least two shooting angles.

[0115] After obtaining the point cloud data under at least two shooting angles, the point cloud data under these shooting angles are fused through the TSDF (Truncated Signed Distance Function) algorithm to finally obtain an initial spatial volume model, which is a model constructed using point cloud data containing noise information.

[0116] In some embodiments of the present application, the device for processing room point cloud data further includes:

[0117] An extraction module, used to extract the surface network model of the initial space body model by using a marching cube algorithm;

[0118] The determination module is also used to determine the vertex data as denoised point cloud data.

[0119] In an embodiment of the present application, after obtaining an initial spatial body model constructed according to point cloud data containing noise information, a surface network model of the initial spatial body model is extracted by a marching cube algorithm.

[0120] The surface network model includes a plurality of grids, each of which corresponds to point data in at least one point cloud data. The vertices of these grids are used as the noise-reduced point cloud data after the noise data of the initial room point cloud data is removed. The noise data in the initial room power data, specifically the deep noise data, is removed by this method, thereby effectively improving the accuracy of the point cloud.

[0121] In some embodiments of the present application, the acquisition module is further used to obtain the height information of each point data in the noise-reduced point cloud data;

[0122] The determination module is further used to determine the point data whose height information is less than the height threshold as the target point data;

[0123] The room point cloud data processing device also includes:

[0124] The fitting module is used to fit the target point data into a ground model;

[0125] The determination module is also used to determine the ground normal vector according to the ground model.

[0126] In the embodiment of the present application, when processing the point skew error in the point cloud data, first, the height information of each point in the noise-reduced point cloud data after noise reduction is determined. Then, based on the camera's prior knowledge or experience, the points whose heights are less than the threshold are screened.

[0127] Furthermore, the points whose heights are less than the threshold are fitted into a ground model using the RANSAC (Random Sample Consensus) algorithm, which is the model of the floor of the photographed room.

[0128] After fitting the ground model, the ground normal vector of the ground model is calculated. The ground normal vector indicates the angle between the ground normal line and the plumb line, thereby clearly indicating the skew error of the point cloud data, which is beneficial for correcting the point cloud data and improving the modeling accuracy.

[0129] In some embodiments of the present application, the acquisition module is further used to acquire a standard coordinate system;

[0130] The determination module is also used to determine the rotation matrix between the ground normal vector and the vertical coordinate axis of the standard coordinate system;

[0131] The rotation module is also used to rotate the coordinates of the denoised point cloud data through the rotation matrix.

[0132] In the embodiment of the present application, the standard coordinate system is the global coordinate system in which the point cloud data is located. Specifically, when generating point cloud data, a standard coordinate system needs to be determined first, and then, according to the position of each scanned point in the RGB-D image, the point is projected into the standard coordinate system, so that all the points captured correspond to a coordinate value (x, y, z) in the standard coordinate system.

[0133] According to the rotation matrix between the ground normal vector and the vertical direction in the standard coordinate system, the rotation matrix is ​​specifically used to correct the ground normal direction corresponding to the ground normal vector to the y-axis direction of the standard coordinate system, thereby completing the correction of the skew error of the point cloud data.

[0134] The embodiment of the present application corrects the skew error of the point cloud point based on the ground normal, so that the ground direction of the generated room VR model is consistent with the real ground direction, without the need for manual correction, thereby improving the model generation efficiency.

[0135] In some embodiments of the present application, the acquisition module is further used to acquire a preset registration algorithm;

[0136] The processing module is also used to process the Jacobian matrix corresponding to the registration algorithm to project the posture update amount obtained by the registration algorithm to a non-degenerate direction.

[0137] In the embodiment of the present application, when point cloud information is captured by an RGB-D camera, due to the limited visual range of the camera or the limitations of the scene itself, it is easy to produce a situation where the point cloud has less geometric feature information. The use of traditional point cloud registration methods often causes the optimization problem to degenerate, resulting in the two structures being misaligned, causing the point cloud to have arbitrary solutions in insufficiently constrained geometric directions, resulting in erroneous results.

[0138] The embodiment of the present application obtains the Jacobian matrix of the normal equation calculated in the registration algorithm, and processes the Jacobian matrix to project the pose update calculated by the registration algorithm to a non-degenerate direction, thereby improving the accuracy of point cloud data processing.

[0139] In some embodiments of the present application, the device for processing room point cloud data further includes:

[0140] A decomposition module is used to perform eigendecomposition on the Jacobian matrix to obtain a corresponding eigenvalue matrix and an eigenvector matrix, wherein the eigenvalues ​​in the eigenvalue matrix correspond one-to-one to the eigenvectors in the eigenvector matrix;

[0141] The determination module is further used to determine, in the eigenvalue matrix, a target eigenvalue whose eigenvalue is less than an eigenvalue threshold;

[0142] The room point cloud data processing device also includes:

[0143] The setting module is used to set the eigenvector corresponding to the target eigenvalue to a 0 vector.

[0144] In an embodiment of the present application, when processing the Jacobian matrix, first, the Jacobian matrix is ​​eigendecomposed to obtain an eigenvalue matrix corresponding to the Jacobian matrix and an eigenvector matrix corresponding one-to-one to the eigenvalue matrix.

[0145] Then, all eigenvalues ​​in the eigenvalue matrix are traversed, and whenever an eigenvalue is determined to be less than a preset eigenvalue threshold, this eigenvalue is marked as a target eigenvalue, and the eigenvector corresponding to the target eigenvalue is set to a 0 vector.

[0146] By way of example, the feature value threshold may be set to 95 to 105.

[0147] After setting the eigenvectors corresponding to all target eigenvalues ​​to 0 vectors, the processed Jacobian matrix is ​​obtained.

[0148] The present application obtains the Jacobian matrix of the normal equation calculated in the registration algorithm and processes the Jacobian matrix, thereby projecting the pose update calculated by the registration algorithm to a non-degenerate direction, thereby improving the accuracy of point cloud data processing.

[0149] The processing device of room point cloud data in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0150] The room point cloud data processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0151] The room point cloud data processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0152] Optionally, an embodiment of the present application further provides an electronic device, Figure 6 A structural block diagram of an electronic device according to an embodiment of the present application is shown. Figure 6 As shown, the electronic device 600 includes a processor 602, a memory 604, and a program or instruction stored in the memory 604 and executable on the processor 602. When the program or instruction is executed by the processor 602, each process of the above-mentioned method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0153] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0154] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0155] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0156] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0157] The processor 710 is used for:

[0158] Generate an initial space volume model corresponding to the initial room point cloud data;

[0159] Determine the surface network model corresponding to the initial space body model;

[0160] According to the vertex data of the surface network model, the initial room point cloud data is subjected to denoising to obtain denoised point cloud data after denoising;

[0161] Determine the ground normal vector of the de-noised point cloud data, and perform coordinate rotation on the de-noised point cloud data according to the ground normal vector to obtain rotated point cloud data;

[0162] The rotation point cloud data is processed by error registration to obtain the processed target point cloud data.

[0163] In the embodiment of the present application, a body model of the original point cloud data is generated, and the noise data contained in the original point cloud data is removed according to the vertex data of the surface network model of the body model. After the noise reduction is completed, the coordinates of the noise-reduced point cloud data are rotated according to the generated ground normal vector, so as to deal with the skew error of the point cloud data, and finally the room point cloud data that has been subjected to the skew error removal and noise reduction processing is further error registration processing, so as to effectively reduce the calibration error, measurement noise, etc., improve the accuracy of the point cloud and the correctness of modeling, so that the room VR model generated according to the processed point cloud data is more accurate, effectively reduce the steps of manual correction, and improve the modeling efficiency.

[0164] Optionally, the processor 710 is further configured to:

[0165] Acquire camera images of at least two shooting angles and shooting data of the camera images, wherein the camera images include a color channel image and a depth information image, and the shooting data include pan / tilt angle information and camera calibration parameters;

[0166] According to the camera image, point cloud data corresponding to at least two shooting angles are respectively established;

[0167] Through the preset algorithm and shooting data, the point cloud data corresponding to at least two shooting angles are fused into an initial spatial volume model.

[0168] After obtaining point cloud data under at least two shooting angles, the embodiment of the present application fuses the point cloud data under these shooting angles through the TSDF (Truncated Signed Distance Function) algorithm, and finally obtains an initial spatial volume model, which is a model constructed by point cloud data containing noise information.

[0169] Optionally, the processor 710 is further configured to:

[0170] The surface network model of the initial space body model is extracted through the marching cube algorithm;

[0171] Determines the vertex data as denoised point cloud data.

[0172] The embodiment of the present application uses this method to remove noise data, specifically deep noise data, from the initial room power data, thereby effectively improving the accuracy of the point cloud.

[0173] Optionally, the processor 710 is further configured to:

[0174] Obtain the height information of each point data in the noise-reduced point cloud data;

[0175] The point data whose height information is less than the height threshold is determined as the target point data;

[0176] Fit the target point data into a ground model;

[0177] Determine the ground normal vector based on the ground model.

[0178] In the embodiment of the present application, after fitting the ground model, the ground normal vector of the ground model is calculated. The ground normal vector indicates the angle relationship between the ground normal line and the plumb line, thereby clearly indicating the skew error of the point cloud data, which is beneficial to correcting the point cloud data and improving the modeling accuracy.

[0179] Optionally, the processor 710 is further configured to:

[0180] Get the standard coordinate system;

[0181] Determine the rotation matrix between the ground normal vector and the vertical coordinate axis of the standard coordinate system;

[0182] The coordinates of the denoised point cloud data are rotated using the rotation matrix.

[0183] The embodiment of the present application corrects the skew error of the point cloud point based on the ground normal, so that the ground direction of the generated room VR model is consistent with the real ground direction, without the need for manual correction, thereby improving the model generation efficiency.

[0184] Optionally, the processor 710 is further configured to:

[0185] Get the preset registration algorithm;

[0186] The Jacobian matrix corresponding to the registration algorithm is processed to project the pose update obtained by the registration algorithm into a non-degenerate direction.

[0187] The embodiment of the present application obtains the Jacobian matrix of the normal equation calculated in the registration algorithm, and processes the Jacobian matrix to project the pose update calculated by the registration algorithm to a non-degenerate direction, thereby improving the accuracy of point cloud data processing.

[0188] Optionally, the processor 710 is further configured to:

[0189] Perform eigendecomposition on the Jacobian matrix to obtain the corresponding eigenvalue matrix and eigenvector matrix, where the eigenvalues ​​in the eigenvalue matrix correspond one-to-one to the eigenvectors in the eigenvector matrix;

[0190] In the eigenvalue matrix, determining a target eigenvalue whose eigenvalue is less than an eigenvalue threshold;

[0191] Set the eigenvector corresponding to the target eigenvalue to the 0 vector.

[0192] The present application obtains the Jacobian matrix of the normal equation calculated in the registration algorithm and processes the Jacobian matrix, thereby projecting the pose update calculated by the registration algorithm to a non-degenerate direction, thereby improving the accuracy of point cloud data processing.

[0193] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0194] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0195] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 710.

[0196] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0197] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0198] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0199] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0200] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0201] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0203] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for processing room point cloud data, characterized in that: include: Generate an initial space volume model corresponding to the initial room point cloud data; Determine a surface network model corresponding to the initial spatial volume model; According to the vertex data of the surface network model, the initial room point cloud data is subjected to denoising processing to obtain denoised point cloud data after denoising; Determine a ground normal vector of the de-noised point cloud data, and perform coordinate rotation on the de-noised point cloud data according to the ground normal vector to obtain rotated point cloud data; Performing error registration processing on the rotation point cloud data to obtain processed target point cloud data; The performing error registration processing on the rotation point cloud data comprises: Get the preset registration algorithm; Processing the Jacobian matrix corresponding to the registration algorithm to project the pose update obtained by the registration algorithm into a non-degenerate direction; The step of performing noise reduction processing on the initial room point cloud data according to the vertex data of the surface network model comprises: The vertex data is determined as the denoised point cloud data.

2. The processing method according to claim 1, characterized in that: The generating of the initial space volume model corresponding to the initial room point cloud data includes: Acquire camera images at at least two shooting angles and shooting data of the camera images, wherein the camera images include a color channel image and a depth information image; According to the camera image, respectively establish point cloud data corresponding to the at least two shooting angles; The point cloud data corresponding to the at least two shooting angles are fused into the initial space volume model through a preset algorithm and the shooting data.

3. The processing method according to claim 2, characterized in that: The determining of the surface network model corresponding to the initial spatial volume model includes: The surface network model of the initial spatial volume model is extracted by a marching cube algorithm.

4. The processing method according to claim 1, characterized in that: The determining of the ground normal vector of the noise-reduced point cloud data comprises: Obtaining height information of each point data in the noise-reduced point cloud data; Determine the point data whose height information is less than the height threshold as the target point data; Fitting the target point data into a ground model; The ground normal vector is determined according to the ground model.

5. The processing method according to claim 4, characterized in that: The coordinate rotation of the noise reduction point cloud data according to the ground normal vector includes: Get the standard coordinate system; Determine a rotation matrix between the ground normal vector and the vertical coordinate axis of the standard coordinate system; The denoised point cloud data is subjected to coordinate rotation using the rotation matrix.

6. The processing method according to claim 1, characterized in that: The processing of the Jacobian matrix corresponding to the registration algorithm includes: Performing eigendecomposition on the Jacobian matrix to obtain a corresponding eigenvalue matrix and an eigenvector matrix, wherein the eigenvalues ​​in the eigenvalue matrix correspond one-to-one to the eigenvectors in the eigenvector matrix; In the eigenvalue matrix, determining a target eigenvalue whose eigenvalue is less than an eigenvalue threshold; The eigenvector corresponding to the target eigenvalue is set to a 0 vector.

7. A room point cloud data processing device, characterized in that: include: A generation module, used to generate an initial space body model corresponding to the initial room point cloud data; A determination module, used to determine the surface network model corresponding to the initial spatial body model; A processing module, configured to perform denoising on the initial room point cloud data according to the vertex data of the surface network model to obtain denoised point cloud data after denoising; A rotation module, used to determine a ground normal vector of the de-noised point cloud data, and perform coordinate rotation on the de-noised point cloud data according to the ground normal vector to obtain rotated point cloud data; The processing module is further used to perform error registration processing on the rotation point cloud data to obtain processed target point cloud data; An acquisition module, used to acquire a preset registration algorithm; The processing module is further used to process the Jacobian matrix corresponding to the registration algorithm to project the pose update amount obtained by the registration algorithm to a non-degenerate direction; The determination module is further used to determine the vertex data as the noise reduction point cloud data.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the processing method according to any one of claims 1 to 6 are implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the processing method according to any one of claims 1 to 6 are implemented.

Images