An intelligent VR three-dimensional digital product display system and method

Abstract

The present invention relates to the technical field of three-dimensional models, and discloses an intelligent VR three-dimensional digital product display system and method, which includes a product contour extraction module, a point cloud registration module, a target point cloud coordinate point acquisition module, a surface support domain generation module and a three-dimensional model construction module, extracting the product contour through the product image of the target product; matching the corresponding point cloud coordinate points in the point cloud data set according to the product contour, performing point cloud registration on the point cloud coordinate points, and obtaining initial registration point cloud coordinate points; performing color matching and point cloud filtering on the initial registration point cloud coordinate points to obtain target point cloud coordinate points; calculating the surface center point according to the target point cloud coordinate point, and generating the surface support domain according to the surface center point; fitting the three-dimensional surface of the target product according to the surface support domain, and performing three-dimensional modeling on the three-dimensional surface to obtain the three-dimensional model of the target product. The present invention can improve the accuracy of three-dimensional display of products.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional model technology, and in particular to an intelligent VR three-dimensional digital product display system and method. Background Art

[0002] Virtual Reality (VR) technology is a high-tech in the field of graphics and images that has emerged in recent years. Virtual reality uses computers and other information processing equipment to simulate a three-dimensional virtual world, providing users with simulations of vision, hearing, touch and other senses, allowing users to observe things in the three-dimensional space in a timely and unlimited manner as if they were in the scene. The emergence of this technology provides a new dimension for product display, allowing users to understand products in a more intuitive and interactive way.

[0003] However, the VR three-dimensional digital product method in the prior art mainly uses visual simulation technology to simulate the visual scene of the three-dimensional animation display system, and mainly designs an information acquisition module, a database model, a 3D reconstruction module of the three-dimensional animation, a visual simulation module, etc. However, the product features extracted by this method are poor and the signal-to-noise ratio is low. The design method based on the dual discriminator GAN reconstructs the loss function and the objective function of the product image, outputs an image with optimal authenticity and diversity, and realizes the optimal design of product display. However, there is a problem of poor visual effect optimization ability, and the output three-dimensional model has a large background noise and a low signal-to-noise ratio. Therefore, how to improve the accuracy of product three-dimensional display has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides an intelligent VR three-dimensional digital product display system and method, the main purpose of which is to solve the problem of poor accuracy of three-dimensional product display.

[0005] To achieve the above-mentioned purpose, the present invention provides an intelligent VR three-dimensional digital product display system, which includes a product contour extraction module, a point cloud registration module, a target point cloud coordinate point acquisition module, a surface support domain generation module and a three-dimensional model construction module, wherein:

[0006] The product contour extraction module is used to obtain a product image and a point cloud data set of a target product, extract product corner points of the target product according to the product image, and extract the product contour of the target product according to the product corner points;

[0007] The point cloud registration module is used to match the point cloud coordinate points corresponding to the product outline in the point cloud data set according to the product outline, perform point cloud registration on the point cloud coordinate points, and obtain initial registration point cloud coordinate points;

[0008] The target point cloud coordinate point acquisition module is used to perform color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, and perform point cloud filtering on the matching point cloud coordinate points to obtain target point cloud coordinate points;

[0009] The curved surface support region generating module is used to calculate the curved surface center point according to the target point cloud coordinate point, and generate the curved surface support region according to the curved surface center point;

[0010] The three-dimensional model building module is used to fit the three-dimensional surface of the target product according to the surface support domain, perform three-dimensional modeling on the three-dimensional surface, and obtain the three-dimensional model of the target product.

[0011] Optionally, when extracting product corner points of the target product according to the product image, the product contour extraction module is specifically used to:

[0012] Using a pre-built sliding window to traverse each pixel in the product image, and obtaining window neighborhood pixels of the pixel;

[0013] Calculating the grayscale variance values ​​of the pixel points in multiple directions according to the window neighborhood pixel points;

[0014] When the minimum value of the grayscale variance value is greater than a preset variance threshold, taking the pixel point as a candidate corner point to obtain a candidate corner point set;

[0015] Local non-maximum suppression is performed on the candidate corner point set to obtain product corner points of the target product.

[0016] Optionally, when extracting the product contour of the target product according to the product corner points, the product contour extraction module is specifically used to:

[0017] Performing an inverse perspective transformation on the product image of the target product according to the product corner points to obtain a transformed image;

[0018] Performing image binarization processing and morphological dilation processing on the transformed image to obtain a morphologically processed image;

[0019] Edge detection is performed on the morphologically processed image to obtain a product outline of the target product.

[0020] Optionally, when performing point cloud registration on the point cloud coordinate points to obtain initial registration point cloud coordinate points, the point cloud registration module is specifically used to:

[0021] Calculate the point cloud centroids of the laser point cloud coordinate points and the three-dimensional point cloud coordinate points in the point cloud coordinate points respectively;

[0022] Calculate the laser matching point corresponding to the laser point cloud coordinate point and the three-dimensional matching point corresponding to the three-dimensional point cloud coordinate point according to the point cloud centroid;

[0023] Calculating a similarity transformation matrix based on the laser matching points and the three-dimensional matching points, and using the similarity transformation matrix to perform a rigid transformation on each of the three-dimensional matching points to obtain a transformed matching point;

[0024] Calculating the point cloud distance between the transformation matching point and the laser matching point, and determining a target transformation matching point set according to the point cloud distance;

[0025] Iteratively updating the similarity transformation matrix based on the number of point clouds in the target transformation matching point set to obtain an updated similarity transformation matrix, until the number of point clouds in the target transformation matching point set is greater than a preset number threshold, thereby obtaining a target similarity transformation matrix;

[0026] The target similarity transformation matrix is ​​used to perform point cloud registration on the three-dimensional point cloud coordinate points in the point cloud coordinate points to obtain initial registration point cloud coordinate points.

[0027] Optionally, when the point cloud registration module calculates the laser matching point corresponding to the laser point cloud coordinate point and the three-dimensional matching point corresponding to the three-dimensional point cloud coordinate point according to the point cloud centroid, it is specifically used to:

[0028] Decentralizing the laser point cloud coordinate points and the three-dimensional point cloud coordinate points using the point cloud centroid to obtain a target laser point cloud and a target three-dimensional point cloud;

[0029] Performing voxel filtering on the target laser point cloud and the target three-dimensional point cloud respectively to obtain a plurality of laser point cloud grids and a three-dimensional point cloud grid;

[0030] Calculating the laser point cloud grid centroid of each of the laser point cloud grids and the three-dimensional point cloud grid centroid of each of the three-dimensional point cloud grids;

[0031] Calculating the covariance matrix of each laser point cloud grid centroid and the three-dimensional point cloud grid centroid respectively;

[0032] The covariance matrix of each laser point cloud grid centroid and the three-dimensional point cloud grid centroid is calculated using the following formula: in, , Respectively represent Laser point cloud grid centroid The corresponding covariance matrix and Laser point cloud grid centroid The corresponding covariance matrix is, , Respectively represent Laser point cloud grid centroid The corresponding neighborhood point cloud and the 3D point cloud grid centroid The corresponding neighborhood point cloud, , Respectively represent The laser point cloud grid corresponding to the center of gravity of the laser point cloud grid and the The 3D point cloud grid corresponding to the centroid of the 3D point cloud grid, represents transpose, , Respectively represent the number of neighborhood point clouds in the laser point cloud grid and the 3D point cloud grid, Indicates the calculation of the second norm;

[0033] The laser matching points corresponding to the laser point cloud coordinate points and the three-dimensional matching points corresponding to the three-dimensional point cloud coordinate points are determined according to the covariance matrix.

[0034] Optionally, when the target point cloud coordinate point acquisition module performs color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, it is specifically used to:

[0035] Extracting registration point cloud pairs from the initial registration point cloud coordinate points;

[0036] Calculating the registration point cloud distance between the registration point cloud pairs;

[0037] When the distance between the registration point clouds is less than a preset threshold, color matching is performed on the registration point cloud pair to obtain matching point cloud coordinate points.

[0038] Optionally, when the target point cloud coordinate point acquisition module performs point cloud filtering on the matching point cloud coordinate point to obtain the target point cloud coordinate point, it is specifically used to:

[0039] Extracting the front connection point and the rear connection point of each matching point cloud coordinate point;

[0040] Calculate the connection distance of the matching point cloud coordinate point according to the front connection point and the rear connection point;

[0041] The connection distance of the matching point cloud coordinate points is calculated using the following formula: in, Indicates the connection distance, Represents the coordinate point of the matching point cloud, represents the front connection point, Represents the post-connection point;

[0042] The matching point cloud coordinate points whose connection distance is greater than a preset connection distance threshold are filtered to obtain the target point cloud coordinate points.

[0043] Optionally, when calculating the center point of the surface according to the target point cloud coordinate point, the surface support region generating module is specifically used to:

[0044] Clustering the target point cloud coordinate points to obtain a clustered point cloud set;

[0045] Calculate the mass point of the target point cloud coordinate point in each of the clustered point clouds;

[0046] The mass point is used as the center point of the surface in the target point cloud coordinate point.

[0047] Optionally, when the curved surface support region generating module generates the curved surface support region according to the curved surface center point, it is specifically used to:

[0048] Obtain a preset candidate support domain model set, and calculate the parameter encoding length of each support domain model in the candidate support domain model set;

[0049] Calculate the point cloud encoding length corresponding to the center point of the surface under the support domain model;

[0050] Determine a target support domain model according to the parameter encoding length and the point cloud encoding length;

[0051] The surface support domain of the center point of the surface is generated by using the support domain size of the target support domain model.

[0052] In order to solve the above problems, the present invention also provides an intelligent VR three-dimensional digital product display method, the method comprising:

[0053] Acquire a product image and a point cloud data set of a target product, extract product corner points of the target product according to the product image, and extract a product contour of the target product according to the product corner points;

[0054] Matching the point cloud coordinate points corresponding to the product outline in the point cloud data set according to the product outline, performing point cloud registration on the point cloud coordinate points to obtain initial registration point cloud coordinate points;

[0055] Performing color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, and performing point cloud filtering on the matching point cloud coordinate points to obtain target point cloud coordinate points;

[0056] Calculate the center point of the surface according to the coordinate point of the target point cloud, and generate the surface support domain according to the center point of the surface;

[0057] The three-dimensional surface of the target product is fitted according to the surface support domain, and three-dimensional modeling is performed on the three-dimensional surface to obtain a three-dimensional model of the target product.

[0058] The embodiment of the present invention extracts product corner points through the product image of the target product, and can extract the product outline according to the product corner points; and determines the point cloud coordinate points corresponding to the product outline in the point cloud data set, performs point cloud registration on the point cloud coordinate points, and obtains the initial registration point cloud coordinate points, which can effectively reduce the complexity of the point cloud coordinate points and remove the noise of the point cloud data; performs color matching and point cloud filtering on the initial registration point cloud coordinate points, which can make the real features in the point cloud data more obvious, and obtain high-quality target point cloud coordinate points, which is conducive to improving the accuracy of subsequent three-dimensional model construction; calculates the center point of the surface according to the target point cloud coordinate point, and generates the surface support domain according to the center point of the surface, and can decompose the point cloud data to generate a three-dimensional surface close to the target product; performs three-dimensional modeling on the three-dimensional surface, and can more accurately obtain the entity of the target product in the three-dimensional space, and realize the high-precision three-dimensional display of the target product. Therefore, the intelligent VR three-dimensional digital product display system and method proposed by the present invention can improve the motion interaction effect of the treadmill panoramic somatosensory motion system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A system architecture diagram of an intelligent VR three-dimensional digital product display system provided by an embodiment of the present invention;

[0060] Figure 2 A system architecture diagram of a point cloud registration module provided in an embodiment of the present invention when performing point cloud registration on point cloud coordinate points to obtain initial registration point cloud coordinate points;

[0061] Figure 3 A system architecture diagram of a target point cloud coordinate point acquisition module provided in an embodiment of the present invention when performing point cloud filtering on matching point cloud coordinate points;

[0062] Figure 4 A flowchart of an intelligent VR three-dimensional digital product display method provided by an embodiment of the present invention;

[0063] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] like Figure 1, which is a system architecture diagram of an intelligent VR three-dimensional digital product display system provided by an embodiment of the present invention.

[0066] The intelligent VR three-dimensional digital product display system 100 of the present invention can be set in a cloud server. In terms of implementation, it can be used as one or more service devices, or it can be installed as an application on the cloud (such as a server or server cluster of a mobile service operator), or it can be developed as a website. According to the functions implemented, the intelligent VR three-dimensional digital product display system 100 can include a product contour extraction module 101, a point cloud registration module 102, a target point cloud coordinate point acquisition module 103, a surface support domain generation module 104 and a three-dimensional model construction module 105. The module of the present invention can also be called a unit, which is a series of computer program segments that can complete fixed functions and are stored in the memory of an electronic device.

[0067] The following is a description of each component and specific workflow of the intelligent VR three-dimensional digital product display system in combination with a specific embodiment:

[0068] The product contour extraction module 101 is used to obtain a product image and a point cloud data set of a target product, extract product corner points of the target product according to the product image, and extract the product contour of the target product according to the product corner points.

[0069] In an embodiment of the present invention, the target product is a product that needs to be displayed in three dimensions, and the product image is an image taken from multiple angles by a camera device, wherein the product image covers a panoramic view of the target product, that is, a 360-degree image of the target product; the point cloud data set is point cloud data obtained by three-dimensional laser scanning of the target product and three-dimensional data of the target product extracted by the SFM (Structure from Motion) method, and the point cloud data of the three-dimensional laser scanning and the three-dimensional data extracted by the SFM method are used as the point cloud data set.

[0070] In an embodiment of the present invention, a product corner point is a place in a product image that looks like a corner or an edge intersection, usually with two or more edges converging from different directions. A product corner point is a notable feature in a product image and can provide important spatial relationship information between different parts of the product image.

[0071] In the embodiment of the present invention, when extracting the product corner points of the target product according to the product image, the product contour extraction module 101 is specifically used to:

[0072] Using a pre-built sliding window to traverse each pixel in the product image, and obtaining window neighborhood pixels of the pixel;

[0073] Calculating the grayscale variance values ​​of the pixel points in multiple directions according to the window neighborhood pixel points;

[0074] When the minimum value of the grayscale variance value is greater than a preset variance threshold, taking the pixel point as a candidate corner point to obtain a candidate corner point set;

[0075] Local non-maximum suppression is performed on the candidate corner point set to obtain product corner points of the target product.

[0076] In an embodiment of the present invention, a window of a preset size is selected, for example, a window of 3x3, 5x5 or 7x7, and the window is slid in the product image so that the center point of the sliding window traverses each pixel point, and the window neighborhood pixel points with each pixel point as the center are obtained, and then the sum of the squares of the grayscale value differences of the adjacent pixel pairs of the central pixel point in the sliding window in the horizontal, vertical, diagonal and anti-diagonal directions is calculated based on the window neighborhood pixel points to obtain the grayscale variance value.

[0077] Furthermore, local non-maximum suppression (NMS) can remove candidate corner points that are not local maxima, thereby improving the accuracy of product corner point extraction. Specifically, local non-maximum suppression takes each candidate corner point as the center point, creates a suppression window centered on the point, calculates the minimum value of the grayscale variance value of each pixel point in the suppression window as the interest value, finds the point with the largest interest value, and if the interest value of the center point is equal to the maximum interest value in the window, retains the center point as the product corner point. If the interest value of the center point is less than the maximum interest value in the window, the candidate corner point corresponding to the center point is not a product corner point.

[0078] In the embodiment of the present invention, the product contour extraction module 101 extracts the product contour of the target product according to the product corner points, including:

[0079] Performing an inverse perspective transformation on the product image of the target product according to the product corner points to obtain a transformed image;

[0080] Performing image binarization processing and morphological dilation processing on the transformed image to obtain a morphologically processed image;

[0081] Edge detection is performed on the morphologically processed image to obtain a product outline of the target product.

[0082] In an embodiment of the present invention, the inverse perspective transformation generally refers to reversing the perspective distortion effect in the product image so that the product image looks like an effect of being viewed directly from the front. The inverse perspective transformation can be performed using the corner points corresponding to the four corners of the product image rectangle in the product corner points.

[0083] Furthermore, the transformed image is binarized using a preset threshold and the image is expanded using a preset structural element, so that the foreground (usually white or high grayscale value area) in the binarized image can be expanded outward until it contacts the adjacent foreground, so that the boundary in the morphologically processed image becomes more obvious and the product contour is extracted more accurately. Among them, edge detection can be performed by an edge operator, and the contour can be smoothed to obtain the product contour.

[0084] The point cloud registration module 102 is used to match the point cloud coordinate points corresponding to the product outline in the point cloud data set according to the product outline, perform point cloud registration on the point cloud coordinate points, and obtain initial registration point cloud coordinate points.

[0085] In an embodiment of the present invention, the point cloud coordinate points are the coordinate points corresponding to the product contour in the point cloud data set, which can be extracted based on the coordinates of the product contour at the target product, including the laser point cloud coordinate points corresponding to the laser point cloud coordinate points of three-dimensional laser scanning and the three-dimensional point cloud coordinate points corresponding to the SFM method. The laser point cloud coordinate points and the three-dimensional point cloud coordinate points are point cloud registered to obtain the initial registered point cloud coordinate points.

[0086] In the embodiment of the present invention, point cloud registration is the process of aligning point cloud data from different sources or perspectives to the same coordinate system, that is, roughly aligning the coordinate points of the laser point cloud to the coordinate system corresponding to the coordinate points of the three-dimensional point cloud. Specifically, a similarity transformation matrix Perform registration, where Can be a 3x3 rotation matrix, represents the scaling factor, can be a 3x1 translation vector, It is a 1x3 zero vector. By multiplying the laser point cloud coordinate point with the similarity transformation matrix, the three-dimensional point cloud coordinate point is converted to the coordinate system corresponding to the three-dimensional point cloud coordinate point to obtain the initial registration point cloud coordinate point.

[0087] In an embodiment of the present invention, the point cloud registration module 102 includes a point cloud centroid calculation module 201, a three-dimensional matching point calculation module 202, a rigid transformation module 203, a target transformation matching point set determination module 204, a target similarity transformation matrix calculation module 205 and an initial registration point cloud coordinate point registration module 206.

[0088] In the embodiment of the present invention, refer to Figure 2 As shown, when the point cloud registration module 102 performs point cloud registration on the point cloud coordinate points to obtain the initial registration point cloud coordinate points, it is specifically used to:

[0089] The point cloud centroid calculation module 201 is used to calculate the point cloud centroids of the laser point cloud coordinate points and the three-dimensional point cloud coordinate points in the point cloud coordinate points respectively;

[0090] The three-dimensional matching point calculation module 202 is used to calculate the laser matching point corresponding to the laser point cloud coordinate point and the three-dimensional matching point corresponding to the three-dimensional point cloud coordinate point according to the point cloud centroid;

[0091] The rigid transformation module 203 is used to calculate a similarity transformation matrix according to the laser matching points and the three-dimensional matching points, and use the similarity transformation matrix to perform a rigid transformation on each of the three-dimensional matching points to obtain a transformed matching point;

[0092] The target transformation matching point set determination module 204 is used to calculate the point cloud distance between the transformation matching point and the laser matching point, and determine the target transformation matching point set according to the point cloud distance;

[0093] The target similarity transformation matrix calculation module 205 is used to iteratively update the similarity transformation matrix according to the number of point clouds in the target transformation matching point set to obtain an updated similarity transformation matrix, until the number of point clouds in the target transformation matching point set is greater than a preset number threshold, thereby obtaining a target similarity transformation matrix;

[0094] The initial registration point cloud coordinate point registration module 206 is used to perform point cloud registration on the three-dimensional point cloud coordinate points in the point cloud coordinate points using the target similarity transformation matrix to obtain initial registration point cloud coordinate points.

[0095] Specifically, the point cloud centroid is to calculate the mean coordinates of the laser point cloud coordinate points and the three-dimensional point cloud coordinate points, and use the mean coordinates as the point cloud centroid.

[0096] Furthermore, the point cloud registration module 102 calculates the laser matching point corresponding to the laser point cloud coordinate point and the three-dimensional matching point corresponding to the three-dimensional point cloud coordinate point according to the point cloud centroid, specifically for:

[0097] Decentralizing the laser point cloud coordinate points and the three-dimensional point cloud coordinate points using the point cloud centroid to obtain a target laser point cloud and a target three-dimensional point cloud;

[0098] Performing voxel filtering on the target laser point cloud and the target three-dimensional point cloud respectively to obtain a plurality of laser point cloud grids and a three-dimensional point cloud grid;

[0099] Calculating the laser point cloud grid centroid of each of the laser point cloud grids and the three-dimensional point cloud grid centroid of each of the three-dimensional point cloud grids;

[0100] Calculating the covariance matrix of each laser point cloud grid centroid and the three-dimensional point cloud grid centroid respectively;

[0101] The laser matching points corresponding to the laser point cloud coordinate points and the three-dimensional matching points corresponding to the three-dimensional point cloud coordinate points are determined according to the covariance matrix.

[0102] In an embodiment of the present invention, the centering processing is to remove the center of gravity of each laser point cloud coordinate point and three-dimensional point cloud coordinate point to obtain a target laser point cloud and a target three-dimensional point cloud, and then the point cloud data can be divided into multiple small cubes (voxels) in the three-dimensional space through voxel filtering. Each laser point cloud grid and three-dimensional point cloud grid includes multiple laser point cloud coordinate points and three-dimensional point cloud coordinate points. When the number of laser point cloud coordinate points and three-dimensional point cloud coordinate points in the point cloud grid is less than a preset threshold, the corresponding laser point cloud coordinate points and three-dimensional point cloud coordinate points in the laser point cloud grid and the three-dimensional point cloud grid can be regarded as noise point clouds and removed, so as to effectively reduce the complexity of the point cloud coordinate points, remove noise, and improve the accuracy of subsequent point cloud registration.

[0103] Furthermore, the above method for calculating the point cloud centroid can be used to calculate the laser point cloud grid centroid of each laser point cloud grid and the three-dimensional point cloud grid centroid of the three-dimensional point cloud grid, and the point cloud coordinate point of the laser point cloud grid where the laser point cloud grid centroid is located is used as the neighborhood point cloud to calculate the covariance matrix of each laser point cloud grid centroid. Similarly, the covariance matrix of each three-dimensional point cloud grid centroid is calculated.

[0104] For example, the covariance matrix of each laser point cloud grid centroid and the three-dimensional point cloud grid centroid is calculated using the following formula: in, , Respectively represent Laser point cloud grid centroid The corresponding covariance matrix and Laser point cloud grid centroid The corresponding covariance matrix is, , Respectively represent Laser point cloud grid centroid The corresponding neighborhood point cloud and the 3D point cloud grid centroid The corresponding neighborhood point cloud, , Respectively represent The laser point cloud grid corresponding to the center of gravity of the laser point cloud grid and the The 3D point cloud grid corresponding to the centroid of the 3D point cloud grid, represents transpose, , Respectively represent the number of neighborhood point clouds in the laser point cloud grid and the 3D point cloud grid, Indicates that the second norm calculation is performed.

[0105] Further, the eigenvalues ​​of the covariance matrix are calculated. Since the point cloud data is a three-dimensional space, there are three eigenvalues, which correspond to the deviation values ​​in the direction of the two plane eigenvectors and the deviation values ​​in the direction of the normal vector. The eigenvalues ​​are sorted from large to small to obtain the first eigenvalue, the second eigenvalue and the third eigenvalue. When the eigenvalues ​​satisfy the preset matching formula, the centroid of the laser point cloud grid and the centroid of the three-dimensional point cloud grid are taken as a pair of matching points, that is, the laser matching point corresponding to the laser point cloud coordinate point and the three-dimensional matching point corresponding to the three-dimensional point cloud coordinate point are a pair of matching points. The matching formula is as follows: in, , , Respectively represent The third eigenvalue, the second eigenvalue, and the first eigenvalue of the covariance matrix, Represents the preset empirical parameters, represents a mathematical constant, , , Respectively represent The third eigenvalue, the second eigenvalue, and the first eigenvalue of the covariance matrix, Indicates the preset matching parameters.

[0106] When the above formula is satisfied, The laser point cloud grid centroid corresponding to the covariance matrix and the The covariance matrices corresponding to the centroids of the three-dimensional point cloud grid are taken as a pair of matching points, which are respectively regarded as the laser matching points corresponding to the laser point cloud coordinate points and the three-dimensional matching points corresponding to the three-dimensional point cloud coordinate points.

[0107] In an embodiment of the present invention, three groups of matching points can be selected to perform matrix solution on the pre-constructed initial similarity transformation matrix to obtain the scaling coefficient, rotation matrix and translation vector, and then obtain the similarity transformation matrix. The similarity transformation matrix is ​​used to perform a rigid transformation on each three-dimensional matching point to obtain the transformed transformation matching point.

[0108] Furthermore, the three-dimensional matching points and the laser matching points are one-to-one corresponding. Therefore, the point cloud distance between the transformed matching point and the corresponding laser matching point is calculated. When the point cloud distance is less than a preset distance threshold, the transformed matching point is used as the target transformed matching point, indicating that the transformed matching point is the initial registration point cloud coordinate point after point cloud registration, and the number of target transformed matching points is counted. When the number of target transformed matching points, that is, the number of point clouds in the target transformed matching point set is not greater than the preset threshold, it indicates that the similarity transformation matrix is ​​not universal and cannot perform point cloud registration for all three-dimensional point cloud coordinate points. It is necessary to use the least squares method to update the similarity transformation matrix until the number of point clouds in the target transformed matching point set is greater than the preset number threshold to obtain the target similarity transformation matrix.

[0109] Specifically, the point cloud distance between the transformed matching point and the laser matching point is calculated using the following formula: in, represents the point cloud distance, Represents the transformed matching point The corresponding laser matching point, represents the rotation matrix in the similarity transformation matrix, Represents the translation vector in the similarity transformation matrix.

[0110] The target point cloud coordinate point acquisition module 103 is used to perform color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, and perform point cloud filtering on the matching point cloud coordinate points to obtain target point cloud coordinate points.

[0111] In the embodiment of the present invention, color matching is based on the color of each point cloud in the initial registration point cloud coordinate points, for example, the grayscale color in the laser point cloud coordinate points and the point cloud color (R, G, B) value of the three-dimensional point cloud coordinate points, and the point set with color in the initial registration point cloud coordinate points is extracted through color matching to improve the accuracy and robustness of the point cloud registration.

[0112] In the embodiment of the present invention, when the target point cloud coordinate point acquisition module 103 performs color matching on the initial registration point cloud coordinate points to obtain the matching point cloud coordinate points, it is specifically used to:

[0113] Extracting registration point cloud pairs from the initial registration point cloud coordinate points;

[0114] Calculating the registration point cloud distance between the registration point cloud pairs;

[0115] When the distance between the registration point clouds is less than a preset threshold, color matching is performed on the registration point cloud pair to obtain matching point cloud coordinate points.

[0116] In an embodiment of the present invention, the registration point cloud pair is the laser point cloud coordinate points and the three-dimensional point cloud coordinate points after point cloud registration. The data distance between the laser point cloud coordinate points and the three-dimensional point cloud coordinate points after point cloud registration is calculated to obtain the registration point cloud distance. When the registration point cloud distance is less than a preset threshold, the point cloud color of the three-dimensional point cloud coordinate points in the initial registration point cloud coordinate points can be matched to the laser point cloud coordinate points after point cloud registration to obtain the matching point cloud coordinate points after color matching.

[0117] Specifically, color matching can improve the accuracy and robustness of point cloud registration, so that the matching point cloud coordinates have more feature information, meet the requirements of high-precision fusion of point cloud data, and obtain more accurate matching point cloud coordinates.

[0118] In the embodiment of the present invention, the target point cloud coordinate point acquisition module 103 includes a connection point extraction module 301 , a connection distance calculation module 302 and a target point cloud coordinate point generation module 303 .

[0119] Furthermore, when the target point cloud coordinate point acquisition module 103 performs point cloud filtering on the matching point cloud coordinate point to obtain the target point cloud coordinate point, it is specifically used to:

[0120] The connection point extraction module 301 is used to extract the front connection point and the rear connection point of each matching point cloud coordinate point;

[0121] The connection distance calculation module 302 is used to calculate the connection distance of the matching point cloud coordinate point according to the front connection point and the rear connection point;

[0122] The target point cloud coordinate point generating module 303 is used to filter the matching point cloud coordinate points whose connection distance is greater than a preset connection distance threshold to obtain the target point cloud coordinate points.

[0123] In detail, the number of matching point cloud coordinate points is much greater than one, and the points adjacent to and before each matching point cloud coordinate point are used as the front connection point and the rear connection point, and the connection distance is calculated based on the front connection point and the rear connection point.

[0124] Specifically, the connection distance of the matching point cloud coordinate points is calculated using the following formula: in, Indicates the connection distance, Represents the coordinate point of the matching point cloud, represents the front connection point, Represents a post-join point.

[0125] In an embodiment of the present invention, matching point cloud coordinate points whose connection distance is greater than a preset connection distance threshold are removed to filter the matching point cloud coordinates, thereby removing noise in the matching point cloud coordinate points, thereby improving the quality of the target point cloud coordinate points, and making the real features in the point cloud data more obvious, which is beneficial to improving the accuracy of subsequent three-dimensional model construction.

[0126] The surface support region generating module 104 is used to calculate the surface center point according to the target point cloud coordinate point, and generate the surface support region according to the surface center point.

[0127] In an embodiment of the present invention, the surface center point refers to a point found in a set of target point cloud coordinate points, which can represent the center or average position of the set of target point cloud coordinate points, and the point is used as the surface center point of the set of target point cloud coordinate points.

[0128] In the embodiment of the present invention, when the curved surface support region generating module 104 calculates the curved surface center point according to the target point cloud coordinate point, it is specifically used to:

[0129] Clustering the target point cloud coordinate points to obtain a clustered point cloud set;

[0130] Calculate the mass point of the target point cloud coordinate point in each of the clustered point clouds;

[0131] The mass point is used as the center point of the surface in the target point cloud coordinate point.

[0132] In detail, the target point cloud coordinate points can be grouped by clustering algorithms (such as K-means clustering, hierarchical clustering, DBSCAN, etc.), and the distance or density between the target point cloud coordinate point groups can be used to group similar point clouds into the same group to obtain a clustered point cloud set, and then the center point of the surface can be obtained by calculating the centroid of the clustered point cloud set (the average value of all target point cloud coordinate points in the clustered point cloud set).

[0133] In an embodiment of the present invention, a surface support region (Local Support Region) is a concept defined in point cloud data or spatial data sets, which refers to a set of neighboring points associated with a specific point, and can represent the local neighborhood or environment of the center point of the surface through surface support.

[0134] Furthermore, when the curved surface support region generating module 104 generates the curved surface support region according to the curved surface center point, it is specifically used to:

[0135] Obtain a preset candidate support domain model set, and calculate the parameter encoding length of each support domain model in the candidate support domain model set;

[0136] Calculate the point cloud encoding length corresponding to the center point of the surface under the support domain model;

[0137] Determine a target support domain model according to the parameter encoding length and the point cloud encoding length;

[0138] The surface support domain of the center point of the surface is generated by using the support domain size of the target support domain model.

[0139] In an embodiment of the present invention, a series of possible support domain sizes are defined as candidate support domain models, for example, different radii or numbers of neighborhood points, different geometric shapes (such as planes, spheres, ellipsoids, etc.). For each candidate support domain model, this is usually related to the number of candidate support domain model parameters. The model is parameterized. For example, for a plane, the parameter ax+by+cz+d=0 may be required. The number of bits of the model parameters is calculated to determine the parameter coding length required to encode the support domain model parameters.

[0140] Furthermore, the point cloud coding length is the data length required for encoding the target point cloud coordinates of the surface center point under each candidate support domain. The point cloud coding data can be obtained by transforming or fitting the coordinates of the target point cloud coordinate points in the candidate support domain using model parameters, and then calculating the number of bits of the encoded data. For example, the target point cloud coordinate points are converted into an encoded form using a lossless compression algorithm, entropy coding (such as Huffman coding), etc., and the total number of bits of the encoded data is calculated to obtain the point cloud coding data.

[0141] Furthermore, the support domain model corresponding to the minimum value of the sum of the parameter coding length and the point cloud coding length is used as the target support domain model, and the size of the target support domain model is used as the size of the surface support domain. Then, a group of adjacent target point cloud coordinate points associated with each surface center point is calculated according to the size of the surface support domain. The three-dimensional space formed by the surface center point and the associated group of adjacent target point cloud coordinate points is called the surface support domain.

[0142] In the embodiment of the present invention, the point cloud data can be decomposed through the surface support domain to generate a three-dimensional surface that approximates the target product, thereby improving the accuracy of three-dimensional model generation.

[0143] The three-dimensional model building module 105 is used to fit the three-dimensional surface of the target product according to the surface support domain, perform three-dimensional modeling on the three-dimensional surface, and obtain a three-dimensional model of the target product.

[0144] In the embodiment of the present invention, the three-dimensional surface is a fitting surface of the target product. The shape, structure and characteristics of the target product can be digitally represented in three-dimensional space through the three-dimensional surface, so that users can understand the product in a more intuitive and interactive way.

[0145] In the embodiment of the present invention, when fitting the three-dimensional surface of the target product according to the surface support domain, the three-dimensional model building module 105 is specifically used to:

[0146] Performing surface fitting according to the surface support domain to obtain a local fitting surface;

[0147] Performing weighted fusion on the local fitting surfaces to obtain a global fitting surface;

[0148] The global fitting surface is subjected to boundary smoothing processing to obtain a three-dimensional surface of the target product.

[0149] In an embodiment of the present invention, surface fitting is performed using the target point cloud coordinate points contained in the surface support domain. For example, a curve formula corresponding to the local fitting surface is fitted by a polynomial to determine the overlapping parts between different local fitting surfaces, so as to fuse the local fitting surfaces and obtain a global fitting surface.

[0150] Furthermore, the boundary areas where the local fitting surfaces intersect can be identified, and the boundary areas can be aligned and fused, so that the fused global fitting surface is smoother and will not be affected by the boundary areas. Among them, the pre-built 3D modeling software can be used to perform weighted fusion and boundary smoothing on the local fitting surfaces to obtain a more accurate 3D surface.

[0151] In detail, 3D modeling is the process of creating entities or shapes in three-dimensional space by using three-dimensional surfaces as building blocks. Modeling can be performed using professional 3D modeling software with preset modeling parameters, such as Rhino, Maya, 3ds Max, Blender, etc. For example, by using 3D modeling software to subdivide the three-dimensional surface, add materials and textures, perform lighting and rendering, etc., the entity of the target product in three-dimensional space can be obtained. The target product can be displayed from multiple angles and in detail, thereby improving the display effect of the target product and achieving high-precision three-dimensional display of the target product.

[0152] Reference Figure 4 FIG. 1 is a flow chart of an intelligent VR three-dimensional digital product display method provided by an embodiment of the present invention. In this embodiment, the intelligent VR three-dimensional digital product display method includes:

[0153] S1. Obtain a product image and a point cloud dataset of a target product, extract product corner points of the target product according to the product image, and extract a product contour of the target product according to the product corner points;

[0154] S2, matching the point cloud coordinate points corresponding to the product outline in the point cloud data set according to the product outline, performing point cloud registration on the point cloud coordinate points, and obtaining initial registration point cloud coordinate points;

[0155] S3, performing color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, and performing point cloud filtering on the matching point cloud coordinate points to obtain target point cloud coordinate points;

[0156] S4, calculating the center point of the surface according to the coordinate point of the target point cloud, and generating the surface support domain according to the center point of the surface;

[0157] S5. Fit the three-dimensional surface of the target product according to the surface support domain, perform three-dimensional modeling on the three-dimensional surface, and obtain a three-dimensional model of the target product.

[0158] An embodiment of the present invention also provides an electronic device of an intelligent VR three-dimensional digital product display system.

[0159] The electronic device may include a processor, a memory, a communication bus, and a communication interface, and may also include a computer program stored in the memory and executable on the processor, such as an intelligent VR three-dimensional digital product display program.

[0160] In some embodiments, the processor may be composed of an integrated circuit, for example, it may be composed of a single packaged integrated circuit, or it may be composed of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips.

[0161] The memory includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device.

[0162] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory and at least one processor, etc.

[0163] The communication interface is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices.

[0164] In the several embodiments provided by the present invention, it should be understood that the disclosed electronic device, apparatus and method can be implemented in other ways. For example, the apparatus embodiment described above is only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0165] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0167] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. An intelligent VR three-dimensional digital product display system, characterized in that: The system includes a product contour extraction module, a point cloud registration module, a target point cloud coordinate point acquisition module, a surface support domain generation module and a three-dimensional model construction module, wherein: The product contour extraction module is used to obtain a product image and a point cloud data set of a target product, extract product corner points of the target product according to the product image, and extract the product contour of the target product according to the product corner points; The point cloud registration module is used to match the point cloud coordinate points corresponding to the product outline in the point cloud data set according to the product outline, perform point cloud registration on the point cloud coordinate points, and obtain initial registration point cloud coordinate points; The target point cloud coordinate point acquisition module is used to perform color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, and perform point cloud filtering on the matching point cloud coordinate points to obtain target point cloud coordinate points; wherein, when the target point cloud coordinate point acquisition module performs point cloud filtering on the matching point cloud coordinate points to obtain the target point cloud coordinate points, it is specifically used to: extract the front connection point and the rear connection point of each matching point cloud coordinate point; calculate the connection distance of the matching point cloud coordinate point according to the front connection point and the rear connection point; The connection distance of the matching point cloud coordinate points is calculated using the following formula: Among them, H represents the connection distance, F represents the coordinate point of the matching point cloud, and F -1 Indicates the front connection point, F +1 Represents the post-connection point; Filtering the matching point cloud coordinate points whose connection distance is greater than a preset connection distance threshold to obtain the target point cloud coordinate points; The surface support domain generation module is used to calculate the surface center point according to the target point cloud coordinate point, and generate the surface support domain according to the surface center point; wherein, when the surface support domain generation module generates the surface support domain according to the surface center point, it is specifically used to: obtain a preset candidate support domain model set, calculate the parameter coding length of each support domain model in the candidate support domain model set; calculate the point cloud coding length corresponding to the surface center point under the support domain model; determine the target support domain model according to the parameter coding length and the point cloud coding length; and generate the surface support domain of the surface center point using the support domain size of the target support domain model; The three-dimensional model building module is used to fit the three-dimensional surface of the target product according to the surface support domain, perform three-dimensional modeling on the three-dimensional surface, and obtain the three-dimensional model of the target product.

2. The intelligent VR three-dimensional digital product display system according to claim 1, characterized in that: When the product contour extraction module extracts the product corner points of the target product according to the product image, it is specifically used to: Using a pre-built sliding window to traverse each pixel in the product image, and obtaining window neighborhood pixels of the pixel; Calculating the grayscale variance values ​​of the pixel points in multiple directions according to the window neighborhood pixel points; When the minimum value of the grayscale variance value is greater than a preset variance threshold, taking the pixel point as a candidate corner point to obtain a candidate corner point set; Local non-maximum suppression is performed on the candidate corner point set to obtain product corner points of the target product.

3. The intelligent VR three-dimensional digital product display system according to claim 1, characterized in that: When the product contour extraction module extracts the product contour of the target product according to the product corner points, it is specifically used to: Performing an inverse perspective transformation on the product image of the target product according to the product corner points to obtain a transformed image; Performing image binarization processing and morphological dilation processing on the transformed image to obtain a morphologically processed image; Edge detection is performed on the morphologically processed image to obtain a product outline of the target product.

4. The intelligent VR three-dimensional digital product display system according to claim 1, characterized in that: When the point cloud registration module performs point cloud registration on the point cloud coordinate points to obtain initial registration point cloud coordinate points, it is specifically used to: Respectively calculating the point cloud centroids of the laser point cloud coordinate points and the three-dimensional point cloud coordinate points in the point cloud coordinate points; Calculate the laser matching point corresponding to the laser point cloud coordinate point and the three-dimensional matching point corresponding to the three-dimensional point cloud coordinate point according to the point cloud centroid; Calculating a similarity transformation matrix based on the laser matching points and the three-dimensional matching points, and using the similarity transformation matrix to perform a rigid transformation on each of the three-dimensional matching points to obtain a transformed matching point; Calculating the point cloud distance between the transformation matching point and the laser matching point, and determining a target transformation matching point set according to the point cloud distance; Iteratively updating the similarity transformation matrix based on the number of point clouds in the target transformation matching point set to obtain an updated similarity transformation matrix, until the number of point clouds in the target transformation matching point set is greater than a preset number threshold, thereby obtaining a target similarity transformation matrix; The target similarity transformation matrix is ​​used to perform point cloud registration on the three-dimensional point cloud coordinate points in the point cloud coordinate points to obtain initial registration point cloud coordinate points.

5. The intelligent VR three-dimensional digital product display system according to claim 4, characterized in that: When the point cloud registration module calculates the laser matching point corresponding to the laser point cloud coordinate point and the three-dimensional matching point corresponding to the three-dimensional point cloud coordinate point according to the point cloud centroid, it is specifically used to: Decentralizing the laser point cloud coordinate points and the three-dimensional point cloud coordinate points using the point cloud gravity center to obtain a target laser point cloud and a target three-dimensional point cloud; Performing voxel filtering on the target laser point cloud and the target three-dimensional point cloud respectively to obtain a plurality of laser point cloud grids and a three-dimensional point cloud grid; Calculating the laser point cloud grid centroid of each of the laser point cloud grids and the three-dimensional point cloud grid centroid of each of the three-dimensional point cloud grids; Calculating the covariance matrix of each laser point cloud grid centroid and the three-dimensional point cloud grid centroid respectively; The covariance matrix of each laser point cloud grid centroid and the three-dimensional point cloud grid centroid is calculated using the following formula: Among them, Cov(P oi )、Cov(P oj ) represent the centroid P of the i-th laser point cloud grid oi The corresponding covariance matrix and the j-th laser point cloud grid centroid P oj The corresponding covariance matrix, p i 、p j Respectively represent the centroid P of the i-th laser point cloud grid oi The corresponding neighborhood point cloud and the j-th 3D point cloud grid centroid P oj The corresponding neighborhood point cloud, N i 、N j They represent the laser point cloud grid corresponding to the centroid of the i-th laser point cloud grid and the 3D point cloud grid corresponding to the centroid of the j-th 3D point cloud grid, T represents transposition, m i 、m j Respectively represent the number of neighborhood point clouds in the laser point cloud grid and the three-dimensional point cloud grid, ||·|| represents the calculation of the two-norm; The laser matching points corresponding to the laser point cloud coordinate points and the three-dimensional matching points corresponding to the three-dimensional point cloud coordinate points are determined according to the covariance matrix.

6. The intelligent VR three-dimensional digital product display system according to claim 1, characterized in that: When the target point cloud coordinate point acquisition module performs color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, it is specifically used to: Extracting registration point cloud pairs from the initial registration point cloud coordinate points; Calculating the registration point cloud distance between the registration point cloud pairs; When the distance between the registration point clouds is less than a preset threshold, color matching is performed on the registration point cloud pair to obtain matching point cloud coordinate points.

7. The intelligent VR three-dimensional digital product display system according to claim 1, characterized in that: When the curved surface support region generation module calculates the curved surface center point according to the target point cloud coordinate point, it is specifically used to: Clustering the target point cloud coordinate points to obtain a clustered point cloud set; Calculate the mass point of the target point cloud coordinate point in each of the clustered point clouds; The mass point is used as the center point of the surface in the target point cloud coordinate point.

8. An intelligent VR three-dimensional digital product display method, characterized in that: The method comprises: Acquire a product image and a point cloud data set of a target product, extract product corner points of the target product according to the product image, and extract a product contour of the target product according to the product corner points; Matching the point cloud coordinate points corresponding to the product outline in the point cloud data set according to the product outline, performing point cloud registration on the point cloud coordinate points to obtain initial registration point cloud coordinate points; Performing color matching on the initial registration point cloud coordinate points to obtain matching point cloud coordinate points, and performing point cloud filtering on the matching point cloud coordinate points to obtain target point cloud coordinate points; wherein, performing point cloud filtering on the matching point cloud coordinate points to obtain target point cloud coordinate points includes: extracting the front connection point and the rear connection point of each matching point cloud coordinate point; calculating the connection distance of the matching point cloud coordinate points according to the front connection point and the rear connection point; The connection distance of the matching point cloud coordinate points is calculated using the following formula: Among them, H represents the connection distance, F represents the coordinate point of the matching point cloud, and F -1 Indicates the front connection point, F +1 Represents the post-connection point; Filtering the matching point cloud coordinate points whose connection distance is greater than a preset connection distance threshold to obtain the target point cloud coordinate points; Calculate the center point of the surface according to the target point cloud coordinate point, and generate the surface support domain according to the center point of the surface; wherein, generating the surface support domain according to the center point of the surface includes: obtaining a preset candidate support domain model set, calculating the parameter coding length of each support domain model in the candidate support domain model set; calculating the point cloud coding length corresponding to the center point of the surface under the support domain model; determining the target support domain model according to the parameter coding length and the point cloud coding length; and generating the surface support domain of the center point of the surface using the support domain size of the target support domain model; The three-dimensional surface of the target product is fitted according to the surface support domain, and three-dimensional modeling is performed on the three-dimensional surface to obtain a three-dimensional model of the target product.

Images