A method and device for intelligent three-dimensional modeling based on machine learning
Through the methods of image splitting, gamma correction and feature point screening, the image and point cloud data are combined to solve the problem of low accuracy and adaptability of three-dimensional modeling, and achieve higher model accuracy and adaptability.
Patent Information
- Application Number
- CN202411073886.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-08-07
AI Technical Summary
The accuracy and adaptability of three-dimensional modeling in the prior art are mainly due to the lack of accuracy due to the fact that the model parameter confirmation is only relied on image data or single data in point cloud data.
By obtaining the image and point cloud data of the object to be modeled, split the image into sub-images and performing gamma correction, filtering feature points, building feature pair sets, filtering feature pairs using relative image quality and point cloud accuracy indicators, and building coordinate systems Transform matrix, converting image pixel points to point cloud three-dimensional coordinates, and combining machine learning models for three-dimensional modeling.
It improves the accuracy and adaptability of three-dimensional modeling, ensures the combination of image quality and point cloud data, and enhances the accuracy and adaptability of model parameters.
Smart Images
Figure CN118608702B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of three-dimensional modeling technology, and more specifically, to an intelligent three-dimensional modeling method and device based on machine learning. Background Art
[0002] The background technology of intelligent 3D modeling is a new 3D modeling method that has emerged in recent years with the development of artificial intelligence and deep learning technology. Compared with traditional 3D modeling, intelligent 3D modeling uses deep learning algorithms to automatically and intelligently improve the efficiency and accuracy of modeling. The core of the background technology is to use deep learning algorithms to automatically extract data features or process data, and perform complex pattern recognition and prediction.
[0003] In the existing technology, the three-dimensional model parameters are often confirmed by relying on only a single type of data in image data or point cloud data. The three-dimensional model parameters include geometric parameters (vertices, edges, faces, etc.) and texture parameters (texture images, material properties, colors, etc.). Image data and point cloud data both have their own characteristics. Only considering a single data situation leads to low precision of the three-dimensional model parameters, which makes the accuracy and adaptability of three-dimensional modeling poor.
[0004] Therefore, how to combine image data and point cloud data to improve the accuracy and adaptability of 3D modeling is a technical problem that needs to be solved. Summary of the invention
[0005] The present invention provides an intelligent three-dimensional modeling method based on machine learning to solve the technical problems of low accuracy and adaptability of three-dimensional modeling in the prior art. The method comprises:
[0006] Acquire an image and point cloud data of the object to be modeled, split the image of the object to be modeled into multiple sub-images according to illumination information of the image of the object to be modeled, evaluate illumination intensity of each sub-image according to the illumination information, determine a target gamma value of each sub-image based on the illumination intensity, and perform gamma correction on each sub-image according to the target gamma value, so as to obtain an image of the object to be modeled after gamma correction;
[0007] Selecting first-class feature points and second-class feature points from the image and point cloud data of the object to be modeled respectively, judging whether the first-class feature points and the second-class feature points match by a preset matching algorithm, recording the first-class feature points and the second-class feature points that match each other as feature pairs, marking the feature pairs in the image, and obtaining the adjacent area of the feature pairs;
[0008] Defining a relative image quality index of an adjacent region of a feature pair according to an image of an adjacent region of the feature pair and an object to be modeled, defining a point cloud accuracy index according to point cloud data and the feature pair, and selecting feature pairs according to the relative image quality index and the point cloud accuracy index, respectively, to construct a first feature pair set and a second feature pair set;
[0009] Determine the quality feature pairs through the first feature pair set and the second feature pair set, construct a coordinate system transformation matrix based on the quality feature pairs, the coordinate system transformation matrix transforms the two-dimensional coordinates of the image pixel points of the object to be modeled into the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data;
[0010] The gamma-corrected image of the object to be modeled and the merged point cloud coordinate data are input into the preset machine learning model, the model parameters are output, and three-dimensional modeling is performed based on the model parameters.
[0011] In some embodiments of the present application, the image of the object to be modeled is split into multiple sub-images according to the illumination information of the image of the object to be modeled, including:
[0012] Identify the shadow area and non-shadow area of the image of the object to be modeled, determine the shadow size in the shadow area and the brightness of the non-shadow area, the lighting information includes the shadow size and the brightness of the non-shadow area;
[0013] The non-shadow area is split according to its brightness, and different labels are configured according to the different intervals to which the brightness of the non-shadow area belongs;
[0014] The shadow area is split by shadow size, and different labels are configured according to the different intervals to which the shadow size belongs;
[0015] The image of the object to be modeled is split into multiple sub-images according to the labels of the shadow area and the non-shadow area, and the shadow area or the non-shadow area under the same label is taken as a single sub-image.
[0016] In some embodiments of the present application, the illumination intensity of each sub-image is evaluated by illumination information, and the target gamma value of each sub-image is determined based on the illumination intensity, including:
[0017] The lighting information also includes shadow conditions. The light source position is analyzed according to the shadow conditions and shadow size, and the light source position is mapped to obtain a surrounding distance. The ratio of the brightness of the shadow area to the brightness of the adjacent non-shadow area within the surrounding distance is defined as the density of the shadow area.
[0018] For a sub-image in a shadow area, a light intensity is determined according to a density of the shadow area of the sub-image, and a target gamma value of the sub-image is determined according to a first corresponding relationship;
[0019] For the sub-image in the non-shadow area, a light intensity is determined according to the brightness of the non-shadow area of the sub-image, and a target gamma value of the sub-image is determined through the second corresponding relationship.
[0020] In some embodiments of the present application, the first type of feature points and the second type of feature points are respectively screened out from the image and point cloud data of the object to be modeled, including:
[0021] Extracting feature points in the image of the object to be modeled by using a feature extraction algorithm, and using the extracted feature points as a class of feature points;
[0022] In the point cloud data of the object to be modeled, the density of the point cloud is determined by calculating the average spacing between the points in the point cloud, the local detail requirements of the object to be modeled are obtained, and two neighborhood intervals are determined according to the density of the point cloud and the local detail requirements respectively;
[0023] The intersection interval of two neighborhood intervals is taken as the neighborhood of the point cloud, and the curvature is calculated in the neighborhood. The point whose curvature change exceeds the curvature threshold is taken as a feature point, and the feature point is taken as a second-class feature point.
[0024] In some embodiments of the present application, the relative image quality index of the feature pair adjacent region is defined according to the image of the feature pair adjacent region and the object to be modeled, including:
[0025] ;
[0026] in, is the relative image quality index of the feature to the adjacent area, 3 is the three indicators of resolution, noise level and contrast in the image, For the image The quality weight of the indicator, For the image The size of the indicator, is the first The quality weight of the indicator, is the first The size of the index, n is the number of shadow areas in the vicinity of the feature pair in the image, The first The reflection weight of the shadow area is The first The comprehensive shadow characteristic index of the shadow area.
[0027] In some embodiments of the present application, point cloud accuracy indicators are defined based on point cloud data and feature pairs, including:
[0028] Determine the location of the feature pair in the point cloud data and the neighborhood of the location;
[0029] ;
[0030] in, is the point cloud accuracy index of the feature pair, , are the weights corresponding to the point cloud density and noise level in the neighborhood of the feature pair, , are the point cloud density and noise level in the neighborhood of the feature pair, respectively. , are the weights corresponding to the point cloud density and noise level of the entire point cloud data, respectively. , They are the point cloud density and noise level of the entire point cloud data respectively.
[0031] In some embodiments of the present application, the feature pairs are selected according to the relative image quality index and the point cloud accuracy index respectively to construct a first feature pair set and a second feature pair set, including:
[0032] Determine the relative image quality index and point cloud accuracy index of each feature pair, filter out the feature pairs that meet the relative image quality index threshold, construct a first feature pair set, filter out the feature pairs that meet the point cloud accuracy index threshold, and construct a second feature pair set.
[0033] In some embodiments of the present application, determining the quality feature pair by using the first feature pair set and the second feature pair set includes:
[0034] Select the feature pairs that intersect the first feature pair set and the second feature pair set as quality feature pairs;
[0035] If there are no overlapping or common feature pairs between the first feature pair set and the second feature pair set, a comprehensive quality index is defined by the relative image quality index and the point cloud accuracy index. Each feature pair in the first feature pair set and the second feature pair set is screened one by one based on the comprehensive quality index, and the feature pairs that meet the requirements are taken as quality feature pairs.
[0036] Correspondingly, the present application also provides an intelligent three-dimensional modeling device based on machine learning, comprising:
[0037] a correction module, used to obtain an image and point cloud data of the object to be modeled, split the image of the object to be modeled into multiple sub-images according to illumination information of the image of the object to be modeled, evaluate illumination intensity of each sub-image according to the illumination information, determine a target gamma value of each sub-image based on the illumination intensity, and perform gamma correction on each sub-image according to the target gamma value, so as to obtain an image of the object to be modeled after gamma correction;
[0038] A matching module is used to select the first-class feature points and the second-class feature points from the image and point cloud data of the object to be modeled, respectively, and determine whether the first-class feature points and the second-class feature points match by using a preset matching algorithm, record the first-class feature points and the second-class feature points that match each other as feature pairs, and mark the feature pairs in the image to obtain the adjacent area of the feature pairs;
[0039] A definition module, used to define a relative image quality index of an adjacent area of a feature pair according to an image of an adjacent area of the feature pair and an object to be modeled, define a point cloud accuracy index according to point cloud data and the feature pair, and select feature pairs according to the relative image quality index and the point cloud accuracy index, respectively, to construct a first feature pair set and a second feature pair set;
[0040] A merging module, used to determine the quality feature pairs through the first feature pair set and the second feature pair set, and to construct a coordinate system transformation matrix based on the quality feature pairs, wherein the coordinate system transformation matrix transforms the two-dimensional coordinates of the image pixel points of the object to be modeled into the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data;
[0041] The modeling module is used to input the gamma-corrected image of the object to be modeled and the merged point cloud coordinate data into a preset machine learning model, output model parameters, and perform three-dimensional modeling based on the model parameters.
[0042] By applying the above technical scheme, the image and point cloud data of the object to be modeled are obtained, the image of the object to be modeled is split into multiple sub-images according to the illumination information of the image of the object to be modeled, the illumination intensity of each sub-image is evaluated according to the illumination information, the target gamma value of each sub-image is determined based on the illumination intensity, and gamma correction is performed on each sub-image according to the target gamma value, so as to obtain the image of the object to be modeled after gamma correction; the first category of feature points and the second category of feature points are respectively screened out from the image and point cloud data of the object to be modeled, and it is determined whether the first category of feature points and the second category of feature points match by a preset matching algorithm, the first category of feature points and the second category of feature points that match each other are recorded as feature pairs, and the feature pairs are marked in the image to obtain the adjacent area of the feature pairs; according to the feature pairs, the adjacent area of the feature pairs is obtained. The image of the near area and the object to be modeled is used to define the relative image quality index of the feature pair in the adjacent area, the point cloud accuracy index is defined according to the point cloud data and the feature pair, and the feature pairs are screened according to the relative image quality index and the point cloud accuracy index respectively to construct a first feature pair set and a second feature pair set; the quality feature pairs are determined by the first feature pair set and the second feature pair set, and the coordinate system transformation matrix is constructed based on the quality feature pairs. The coordinate system transformation matrix converts the two-dimensional coordinates of the image pixel points of the object to be modeled to the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data; the gamma-corrected image of the object to be modeled and the merged point cloud coordinate data are input into a preset machine learning model, the model parameters are output, and three-dimensional modeling is performed according to the model parameters. The present application performs gamma correction on each sub-image by using the target gamma value to ensure that the contrast of the image is within a reasonable range and improve the image quality. Feature pairs are selected based on relative image quality indicators and point cloud accuracy indicators, and a coordinate system transformation matrix is constructed based on quality feature pairs, so as to convert the position coordinates of the object to be modeled on the image to the three-dimensional coordinates on the point cloud, merge the point cloud data, and combine the position coordinates of the object to be modeled in the image and the point cloud, thereby improving the accuracy of the model parameters of the appearance structure of the object to be modeled and ensuring the accuracy of three-dimensional modeling. The gamma-corrected image of the object to be modeled and the merged point cloud coordinate data are input into the preset machine learning model, and the model parameters are output. Other features of the image can be combined with the point cloud coordinate position, such as mapping the texture features of the image to the corresponding part of the three-dimensional model composed of the point cloud, thereby improving the adaptability of three-dimensional modeling. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0044] Figure 1A schematic diagram of a process of an intelligent three-dimensional modeling method based on machine learning proposed in an embodiment of the present invention is shown;
[0045] Figure 2 A schematic structural diagram of an intelligent three-dimensional modeling device based on machine learning proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0047] The present application embodiment provides an intelligent three-dimensional modeling method based on machine learning, such as Figure 1 As shown, the method comprises the following steps:
[0048] Step S101, obtaining an image and point cloud data of the object to be modeled, splitting the image of the object to be modeled into multiple sub-images according to the illumination information of the image of the object to be modeled, evaluating the illumination intensity of each sub-image through the illumination information, determining a target gamma value of each sub-image based on the illumination intensity, and performing gamma correction on each sub-image according to the target gamma value, so as to obtain an image of the object to be modeled after gamma correction.
[0049] In this embodiment, the image of the object to be modeled is a multi-angle related image obtained by a shooting device, and the point cloud data is the three-dimensional point cloud data of the target object obtained by a three-dimensional scanning device (such as a laser scanner, a structured light scanner, etc.). The object to be modeled here can be an object, a person, an animal or a scene, etc. An image is split into multiple sub-images according to the lighting conditions of the image (shadow and non-shadow), and the target gamma value is determined by the illumination intensity of each sub-image. Gamma correction is a nonlinear transformation of the brightness of an image, which can improve the contrast of the image. By adjusting the gamma value, it can adapt to different lighting conditions, but excessive gamma correction may cause image distortion, so the appropriateness of gamma correction must be ensured.
[0050] It should be noted that both images and point cloud data provide spatial information about scenes or objects, although they are represented in different forms. Images provide spatial information from a two-dimensional perspective, while point clouds provide coordinate data in three-dimensional space. The lighting conditions within an image can significantly affect the extraction of texture and color features of the image. For example, the unevenness of lighting may lead to a reduction in the contrast of texture features, making texture details difficult to identify and affecting the texture mapping and color mapping of subsequent three-dimensional modeling.
[0051] In some embodiments of the present application, the image of the object to be modeled is split into multiple sub-images according to the illumination information of the image of the object to be modeled, including:
[0052] Identify the shadow area and non-shadow area of the image of the object to be modeled, determine the shadow size in the shadow area and the brightness of the non-shadow area, the lighting information includes the shadow size and the brightness of the non-shadow area;
[0053] The non-shadow area is split according to its brightness, and different labels are configured according to the different intervals to which the brightness of the non-shadow area belongs;
[0054] The shadow area is split by shadow size, and different labels are configured according to the different intervals to which the shadow size belongs;
[0055] The image of the object to be modeled is split into multiple sub-images according to the labels of the shadow area and the non-shadow area, and the shadow area or the non-shadow area under the same label is taken as a single sub-image.
[0056] In this embodiment, a shadow detection algorithm (such as a shadow detection method based on color, brightness or texture) is used to identify shadow areas in an image. Shadows are usually darker or smoother areas that contrast with the surrounding environment. Non-shadow areas are classified by their brightness, and shadow areas are classified by their shadow size, thereby integrating them into a single sub-image of multiple categories.
[0057] In some embodiments of the present application, the illumination intensity of each sub-image is evaluated by illumination information, and the target gamma value of each sub-image is determined based on the illumination intensity, including:
[0058] The lighting information also includes shadow conditions. The light source position is analyzed according to the shadow conditions and shadow size, and the light source position is mapped to obtain a surrounding distance. The ratio of the brightness of the shadow area to the brightness of the adjacent non-shadow area within the surrounding distance is defined as the density of the shadow area.
[0059] For a sub-image in a shadow area, a light intensity is determined according to a density of the shadow area of the sub-image, and a target gamma value of the sub-image is determined according to a first corresponding relationship;
[0060] For the sub-image in the non-shadow area, a light intensity is determined according to the brightness of the non-shadow area of the sub-image, and a target gamma value of the sub-image is determined through the second corresponding relationship.
[0061] In this embodiment, the shadow condition includes the shape, edge and distribution of the shadow, so as to estimate the direction and position of the light source. For example, a long strip of shadow may indicate that the light source comes from the side, while a circular shadow may indicate that the light source comes from above. The edge of the shadow is determined by the position of the light source, and the preset distance outside the edge is set as the range of the surrounding distance. Generally speaking, the shadow density refers to the ratio of the brightness of the shadow area to the brightness of the surrounding non-shadow area. The definition of this surrounding non-shadow area is specifically limited to improve the accuracy of the shadow density.
[0062] In this embodiment, different shadow densities correspond to different light intensities, and different brightnesses of non-shadow areas correspond to different light intensities. The target gamma value of the sub-image is determined by the first corresponding relationship or the second corresponding relationship, the first corresponding relationship is the corresponding relationship of the shadow area, the second corresponding relationship is the corresponding relationship of the non-shadow area, and different light intensities correspond to different target gamma values.
[0063] Step S102, respectively select the first category feature points and the second category feature points from the image and point cloud data of the object to be modeled, determine whether the first category feature points and the second category feature points match by using a preset matching algorithm, record the first category feature points and the second category feature points that match each other as feature pairs, and mark the feature pairs in the image to obtain the feature pair adjacent area.
[0064] In this embodiment, both the image and point cloud data provide spatial information about the scene or object, although they are represented in different forms. The image provides spatial information from a two-dimensional perspective, while the point cloud provides coordinate data in a three-dimensional space. Image data usually contains rich texture information but lacks depth information; point cloud data provides precise geometric information but usually does not contain color or texture information. Preset matching algorithms include FLANN, Brute-Force, etc., which match feature points in the image with feature points in the point cloud. The matching algorithm will try to find feature point pairs with similar geometric properties in the image and point cloud. The feature pairs are marked in the image, and the preset range where the feature points are located is used as the adjacent area of the feature pairs. What we are talking about here is the feature pairs, which are essentially feature points.
[0065] In some embodiments of the present application, the first type of feature points and the second type of feature points are respectively screened out from the image and point cloud data of the object to be modeled, including:
[0066] Extracting feature points in the image of the object to be modeled by using a feature extraction algorithm, and using the extracted feature points as a class of feature points;
[0067] In the point cloud data of the object to be modeled, the density of the point cloud is determined by calculating the average spacing between the points in the point cloud, the local detail requirements of the object to be modeled are obtained, and two neighborhood intervals are determined according to the density of the point cloud and the local detail requirements respectively;
[0068] The intersection interval of two neighborhood intervals is taken as the neighborhood of the point cloud, and the curvature is calculated in the neighborhood. The point whose curvature change exceeds the curvature threshold is taken as a feature point, and the feature point is taken as a second-class feature point.
[0069] In this embodiment, for images, feature extraction algorithms include SIFT, SURF, ORB and other feature extraction algorithms to extract feature points in the image. These algorithms detect key points in the image and calculate the direction, scale and other information of each key point. For point clouds, the local curvature of each point is calculated to identify points with large curvature changes, which may correspond to the edges, corners or areas with large curvature changes of objects.
[0070] In this embodiment, local curvature refers to the local curvature of the surface shape in the neighborhood around a specific point in three-dimensional space. In point cloud data, local curvature is usually used to describe the local geometric characteristics of each point in the point cloud, that is, the curvature change of the area near the point. The size of the neighborhood depends on the density of the point cloud and the required level of local detail. For example, the higher the density of the point cloud, the more points there are in each unit space, so the size of the neighborhood can be relatively small. The lower the density of the point cloud, the fewer points there are in each unit space, so the size of the neighborhood needs to be appropriately increased. Two neighborhood intervals are determined based on the density of the point cloud and the local detail requirements, and the intersection interval of the two neighborhood intervals is used as the neighborhood of the point cloud, thereby taking into account the density of the point cloud and the required level of local detail.
[0071] Step S103, defining a relative image quality index of the adjacent area of the feature pair based on the image of the adjacent area of the feature pair and the object to be modeled, defining a point cloud accuracy index based on the point cloud data and the feature pair, and filtering the feature pairs based on the relative image quality index and the point cloud accuracy index, respectively, to construct a first feature pair set and a second feature pair set.
[0072] In this embodiment, the matched feature points are feature pairs. In order to improve the accuracy of the coordinate system transformation matrix in step S104, the feature pairs need to be screened and high-quality and high-precision feature pairs are retained as quality feature pairs. The relative image quality index of the feature pair's adjacent area is defined by comparing the feature pair's adjacent area with the overall image, and the point cloud accuracy index is defined by comparing the feature pair's neighborhood with the overall point cloud data. The larger the two types of indicators are, the better the quality of the feature pair is.
[0073] It should be noted that the relative image quality index is calculated for the feature points on the image of the feature pair, and the point cloud accuracy index is calculated for the feature points on the point cloud of the feature pair, thereby constructing the first feature pair set and the second feature pair set. For example, the feature pairs include (s1, s2), (s3, s4), (s5, s6), s1, s3, s5 are all feature points on the image, s2, s4, s6 are all feature points on the point cloud, and each of these three feature pairs corresponds to a relative image quality index and a point cloud accuracy index. The relative image quality index is calculated for the three points s1, s3, and s5, and the point cloud accuracy index is calculated for the three points s2, s4, and s6.
[0074] In some embodiments of the present application, the relative image quality index of the feature pair adjacent region is defined according to the image of the feature pair adjacent region and the object to be modeled, including:
[0075] ;
[0076] in, is the relative image quality index of the feature to the adjacent area, 3 is the three indicators of resolution, noise level and contrast in the image, For the image The quality weight of the indicator, For the image The size of the indicator, is the first The quality weight of the indicator, is the first The size of the index, n is the number of shadow areas in the vicinity of the feature pair in the image, The first The reflection weight of the shadow area is The first The comprehensive shadow characteristic index of the shadow area.
[0077] In this embodiment, the relative image quality index is determined by three indicators: resolution, noise level and contrast. The class average of the comprehensive shadow feature index representing multiple shadow areas corrects the image quality in the adjacent area, because the shadow area usually reflects the low contrast area in the image, which can characterize the image quality to a certain extent. The comprehensive shadow feature index is the weighted sum of multiple shadow features to represent the situation of the shadow area.
[0078] In some embodiments of the present application, point cloud accuracy indicators are defined based on point cloud data and feature pairs, including:
[0079] Determine the location of the feature pair in the point cloud data and the neighborhood of the location;
[0080] ;
[0081] in, is the point cloud accuracy index of the feature pair, , are the weights corresponding to the point cloud density and noise level in the neighborhood of the feature pair, , are the point cloud density and noise level in the neighborhood of the feature pair, respectively. , are the weights corresponding to the point cloud density and noise level of the entire point cloud data, respectively. , They are the point cloud density and noise level of the entire point cloud data respectively.
[0082] In this embodiment, the point cloud accuracy index is determined by the ratio of the point cloud density and noise level in the neighborhood of the feature point of the feature pair point cloud to the point cloud density and noise level of the overall point cloud data.
[0083] In some embodiments of the present application, the feature pairs are selected according to the relative image quality index and the point cloud accuracy index respectively to construct a first feature pair set and a second feature pair set, including:
[0084] Determine the relative image quality index and point cloud accuracy index of each feature pair, filter out the feature pairs that meet the relative image quality index threshold, construct a first feature pair set, filter out the feature pairs that meet the point cloud accuracy index threshold, and construct a second feature pair set.
[0085] In this embodiment, for example, the feature pairs include (s1, s2), (s3, s4), (s5, s6), s1, s3, s5 are all feature points on the image, s2, s4, s6 are all feature points on the point cloud, and as long as one feature point in the feature pair meets the requirements, the feature pair can be divided into the first feature pair set or the second feature pair set.
[0086] Step S104, determine the quality feature pairs through the first feature pair set and the second feature pair set, and construct a coordinate system transformation matrix based on the quality feature pairs. The coordinate system transformation matrix converts the two-dimensional coordinates of the image pixel points of the object to be modeled into the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data.
[0087] In this embodiment, the first feature pair set and the second feature pair set are used to screen out feature pairs with high relative image quality index and point cloud accuracy index, so as to construct a coordinate system transformation matrix. The successfully matched feature point pairs record their image coordinates (two-dimensional) and point cloud coordinates (three-dimensional) to obtain the rotation and translation conditions, and the coordinate system transformation matrix is combined according to the rotation matrix and the translation vector. The matrix can convert the two-dimensional coordinates of the image to the three-dimensional coordinates of the point cloud. Because the point cloud is three-dimensional data and the spatial coordinates provided are more accurate, the image coordinates are transferred to the point cloud. Merge the point cloud coordinate data and merge the converted image data (usually in the form of points) into the original point cloud. This may involve creating new points or updating the positions of existing points. Thereby, the position data of the image is combined with the position data of the point cloud to obtain a more accurate spatial position structure of the object to be modeled.
[0088] It should be noted that the method of calculating the coordinate system transformation matrix that converts the two-dimensional coordinates of the image to the three-dimensional coordinates of the point cloud is not the focus of protection of this scheme. As long as the scheme can calculate the coordinate system transformation matrix based on the quality characteristics, it falls within the scope of protection of this application.
[0089] In some embodiments of the present application, determining the quality feature pair by using the first feature pair set and the second feature pair set includes:
[0090] Select the feature pairs that intersect the first feature pair set and the second feature pair set as quality feature pairs;
[0091] If there are no overlapping or common feature pairs between the first feature pair set and the second feature pair set, a comprehensive quality index is defined by the relative image quality index and the point cloud accuracy index. Each feature pair in the first feature pair set and the second feature pair set is screened one by one based on the comprehensive quality index, and the feature pairs that meet the requirements are taken as quality feature pairs.
[0092] In this embodiment, the intersection here means that there are common feature pairs in the two sets, indicating that the relative image quality index and the point cloud accuracy index are both high, and such feature pairs are recorded as quality feature pairs. If there is no intersection in the two sets, the comprehensive quality index is defined by the relative image quality index and the point cloud accuracy index (weighted summation, weighted average, or any method that can take into account both qualities).
[0093] Step S105, inputting the gamma-corrected image of the object to be modeled and the merged point cloud coordinate data into a preset machine learning model, outputting model parameters, and performing three-dimensional modeling according to the model parameters.
[0094] In this embodiment, the appearance shape structure of the object to be modeled is constructed by the merged point cloud coordinate data, and each part of the appearance shape structure is matched with each part of the image, and the color features and texture features of the image are mapped to each part of the corresponding appearance shape structure. Thus, model parameters are generated and three-dimensional modeling is performed.
[0095] It is understandable that not only the color features and texture features of the image can be used here, but also some other features can be used to help model building.
[0096] By applying the above technical scheme, the image and point cloud data of the object to be modeled are obtained, the image of the object to be modeled is split into multiple sub-images according to the illumination information of the image of the object to be modeled, the illumination intensity of each sub-image is evaluated according to the illumination information, the target gamma value of each sub-image is determined based on the illumination intensity, and gamma correction is performed on each sub-image according to the target gamma value, so as to obtain the image of the object to be modeled after gamma correction; the first category of feature points and the second category of feature points are respectively screened out from the image and point cloud data of the object to be modeled, and it is determined whether the first category of feature points and the second category of feature points match by a preset matching algorithm, the first category of feature points and the second category of feature points that match each other are recorded as feature pairs, and the feature pairs are marked in the image to obtain the adjacent area of the feature pairs; according to the feature pairs, the adjacent area of the feature pairs is obtained. The image of the near area and the object to be modeled is used to define the relative image quality index of the feature pair in the adjacent area, the point cloud accuracy index is defined according to the point cloud data and the feature pair, and the feature pairs are screened according to the relative image quality index and the point cloud accuracy index respectively to construct a first feature pair set and a second feature pair set; the quality feature pairs are determined by the first feature pair set and the second feature pair set, and the coordinate system transformation matrix is constructed based on the quality feature pairs. The coordinate system transformation matrix converts the two-dimensional coordinates of the image pixel points of the object to be modeled to the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data; the gamma-corrected image of the object to be modeled and the merged point cloud coordinate data are input into a preset machine learning model, the model parameters are output, and three-dimensional modeling is performed according to the model parameters. The present application performs gamma correction on each sub-image by using the target gamma value to ensure that the contrast of the image is within a reasonable range and improve the image quality. Feature pairs are selected based on relative image quality indicators and point cloud accuracy indicators, and a coordinate system transformation matrix is constructed based on quality feature pairs, so as to convert the position coordinates of the object to be modeled on the image to the three-dimensional coordinates on the point cloud, merge the point cloud data, and combine the position coordinates of the object to be modeled in the image and the point cloud, thereby improving the accuracy of the model parameters of the appearance structure of the object to be modeled and ensuring the accuracy of three-dimensional modeling. The gamma-corrected image of the object to be modeled and the merged point cloud coordinate data are input into the preset machine learning model, and the model parameters are output. Other features of the image can be combined with the point cloud coordinate position, such as mapping the texture features of the image to the corresponding part of the three-dimensional model composed of the point cloud, thereby improving the adaptability of three-dimensional modeling.
[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or by software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present invention.
[0098] In order to further explain the technical idea of the present invention, the technical solution of the present invention is now described in combination with specific application scenarios.
[0099] Correspondingly, the present application also provides an intelligent three-dimensional modeling device based on machine learning, such as Figure 2 As shown, including:
[0100] a correction module, used to obtain an image and point cloud data of the object to be modeled, split the image of the object to be modeled into multiple sub-images according to illumination information of the image of the object to be modeled, evaluate illumination intensity of each sub-image according to the illumination information, determine a target gamma value of each sub-image based on the illumination intensity, and perform gamma correction on each sub-image according to the target gamma value, so as to obtain an image of the object to be modeled after gamma correction;
[0101] A matching module is used to select the first-class feature points and the second-class feature points from the image and point cloud data of the object to be modeled, respectively, and determine whether the first-class feature points and the second-class feature points match by using a preset matching algorithm, record the first-class feature points and the second-class feature points that match each other as feature pairs, and mark the feature pairs in the image to obtain the adjacent area of the feature pairs;
[0102] A definition module, used to define a relative image quality index of an adjacent area of a feature pair according to an image of an adjacent area of the feature pair and an object to be modeled, define a point cloud accuracy index according to point cloud data and the feature pair, and select feature pairs according to the relative image quality index and the point cloud accuracy index, respectively, to construct a first feature pair set and a second feature pair set;
[0103] A merging module, used to determine the quality feature pairs through the first feature pair set and the second feature pair set, and to construct a coordinate system transformation matrix based on the quality feature pairs, wherein the coordinate system transformation matrix transforms the two-dimensional coordinates of the image pixel points of the object to be modeled into the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data;
[0104] The modeling module is used to input the gamma-corrected image of the object to be modeled and the merged point cloud coordinate data into a preset machine learning model, output model parameters, and perform three-dimensional modeling based on the model parameters.
[0105] Those skilled in the art will appreciate that the modules in the device in the implementation scenario can be distributed in the device in the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more devices different from the implementation scenario. The modules in the above implementation scenario can be combined into one module, or can be further split into multiple sub-modules.
[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An intelligent three-dimensional modeling method based on machine learning, characterized in that: include: Acquire an image and point cloud data of the object to be modeled, split the image of the object to be modeled into multiple sub-images according to illumination information of the image of the object to be modeled, evaluate illumination intensity of each sub-image according to the illumination information, determine a target gamma value of each sub-image based on the illumination intensity, and perform gamma correction on each sub-image according to the target gamma value, so as to obtain an image of the object to be modeled after gamma correction; Selecting first-class feature points and second-class feature points from the image and point cloud data of the object to be modeled respectively, judging whether the first-class feature points and the second-class feature points match by a preset matching algorithm, recording the first-class feature points and the second-class feature points that match each other as feature pairs, marking the feature pairs in the image, and obtaining the adjacent area of the feature pairs; Defining a relative image quality index of an adjacent region of a feature pair according to an image of an adjacent region of the feature pair and an object to be modeled, defining a point cloud accuracy index according to point cloud data and the feature pair, and selecting feature pairs according to the relative image quality index and the point cloud accuracy index, respectively, to construct a first feature pair set and a second feature pair set; Determine the quality feature pairs through the first feature pair set and the second feature pair set, construct a coordinate system transformation matrix based on the quality feature pairs, the coordinate system transformation matrix transforms the two-dimensional coordinates of the image pixel points of the object to be modeled into the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data; Input the gamma-corrected image of the object to be modeled and the merged point cloud coordinate data into a preset machine learning model, output the model parameters, and perform three-dimensional modeling according to the model parameters; The image of the object to be modeled is split into multiple sub-images according to the illumination information of the image of the object to be modeled, including: Identify the shadow area and non-shadow area of the image of the object to be modeled, determine the shadow size in the shadow area and the brightness of the non-shadow area, the lighting information includes the shadow size and the brightness of the non-shadow area; The non-shadow area is split according to its brightness, and different labels are configured according to the different intervals to which the brightness of the non-shadow area belongs; The shadow area is split by shadow size, and different labels are configured according to the different intervals to which the shadow size belongs; The image of the object to be modeled is split into multiple sub-images according to the labels of the shadow area and the non-shadow area, and the shadow area or the non-shadow area under the same label is taken as a single sub-image; the illumination intensity of each sub-image is evaluated through illumination information, and the target gamma value of each sub-image is determined based on the illumination intensity, including: The lighting information also includes shadow conditions. The light source position is analyzed according to the shadow conditions and shadow size, and the light source position is mapped to obtain a surrounding distance. The ratio of the brightness of the shadow area to the brightness of the adjacent non-shadow area within the surrounding distance is defined as the density of the shadow area. For a sub-image in a shadow area, a light intensity is determined according to a density of the shadow area of the sub-image, and a target gamma value of the sub-image is determined according to a first corresponding relationship; For a sub-image of a non-shadow area, a light intensity is determined according to the brightness of the non-shadow area of the sub-image, and a target gamma value of the sub-image is determined through a second corresponding relationship; a first-class feature point and a second-class feature point are respectively selected from the image and point cloud data of the object to be modeled, including: Extracting feature points in the image of the object to be modeled by using a feature extraction algorithm, and using the extracted feature points as a class of feature points; In the point cloud data of the object to be modeled, the density of the point cloud is determined by calculating the average spacing between the points in the point cloud, the local detail requirements of the object to be modeled are obtained, and two neighborhood intervals are determined according to the density of the point cloud and the local detail requirements respectively; The intersection interval of two neighborhood intervals is taken as the neighborhood of the point cloud, and the curvature is calculated in the neighborhood. The point where the curvature change exceeds the curvature threshold is taken as a feature point, and the feature point is taken as a second-class feature point; the relative image quality index of the feature pair adjacent area is defined according to the image of the feature pair adjacent area and the object to be modeled, including: ； in, is the relative image quality index of the feature to the adjacent area, 3 is the three indicators of resolution, noise level and contrast in the image, For the image The quality weight of the indicator, For the image The size of the indicator, is the first The quality weight of the indicator, is the first The size of the index, n is the number of shadow areas in the vicinity of the feature pair in the image, The first The reflection weight of the shadow area is The first The comprehensive shadow feature index of the shadow area; the point cloud accuracy index is defined according to the point cloud data and feature pairs, including: Determine the location of the feature pair in the point cloud data and the neighborhood of the location; ； in, is the point cloud accuracy index of the feature pair, , are the weights corresponding to the point cloud density and noise level in the neighborhood of the feature pair, , are the point cloud density and noise level in the neighborhood of the feature pair, respectively. , are the weights corresponding to the point cloud density and noise level of the entire point cloud data, respectively. , are the point cloud density and noise level of the entire point cloud data respectively; and feature pairs are selected according to the relative image quality index and the point cloud accuracy index respectively, and a first feature pair set and a second feature pair set are constructed, including: Determine the relative image quality index and point cloud accuracy index of each feature pair, screen out the feature pairs that meet the relative image quality index threshold, construct a first feature pair set, screen out the feature pairs that meet the point cloud accuracy index threshold, and construct a second feature pair set; determine the quality feature pairs through the first feature pair set and the second feature pair set, including: Select the feature pairs that intersect the first feature pair set and the second feature pair set as quality feature pairs; If there are no overlapping or common feature pairs between the first feature pair set and the second feature pair set, a comprehensive quality index is defined by the relative image quality index and the point cloud accuracy index. Each feature pair in the first feature pair set and the second feature pair set is screened one by one based on the comprehensive quality index, and the feature pairs that meet the requirements are taken as quality feature pairs.
2. An intelligent three-dimensional modeling device based on machine learning, characterized in that: As used in claim 1, comprising: a correction module, used to obtain an image and point cloud data of the object to be modeled, split the image of the object to be modeled into multiple sub-images according to illumination information of the image of the object to be modeled, evaluate illumination intensity of each sub-image according to the illumination information, determine a target gamma value of each sub-image based on the illumination intensity, and perform gamma correction on each sub-image according to the target gamma value, so as to obtain an image of the object to be modeled after gamma correction; A matching module is used to select the first-class feature points and the second-class feature points from the image and point cloud data of the object to be modeled, respectively, and determine whether the first-class feature points and the second-class feature points match by using a preset matching algorithm, record the first-class feature points and the second-class feature points that match each other as feature pairs, and mark the feature pairs in the image to obtain the adjacent area of the feature pairs; A definition module, used to define a relative image quality index of an adjacent area of a feature pair according to an image of an adjacent area of the feature pair and an object to be modeled, define a point cloud accuracy index according to point cloud data and the feature pair, and select feature pairs according to the relative image quality index and the point cloud accuracy index, respectively, to construct a first feature pair set and a second feature pair set; A merging module, used to determine the quality feature pairs through the first feature pair set and the second feature pair set, and to construct a coordinate system transformation matrix based on the quality feature pairs, wherein the coordinate system transformation matrix transforms the two-dimensional coordinates of the image pixel points of the object to be modeled into the three-dimensional coordinates of the point cloud of the object to be modeled, and merges the point cloud coordinate data; The modeling module is used to input the gamma-corrected image of the object to be modeled and the merged point cloud coordinate data into a preset machine learning model, output model parameters, and perform three-dimensional modeling based on the model parameters.