Texture mapping method, storage medium and intraoral scanning system

Abstract

The application discloses a texture mapping method, a storage medium and an intraoral scanning system. The method comprises the following steps: acquiring a 2D image sequence, a 3D point cloud grid model and scanning parameters in three-dimensional scanning; selecting key frames according to the 2D image sequence, the 3D point cloud grid model and the scanning parameters to obtain a key frame sequence; adjusting the brightness of all the key frames to obtain a first image sequence; calculating the gradient amplitude and the corresponding shooting angle of all the triangular mesh facets on the corresponding first image according to the corresponding first image, the 3D point cloud grid model and the scanning parameters of all the triangular mesh facets; selecting the optimal first image according to the gradient amplitude and the shooting angle by using a Markov random field function to obtain a second image sequence and the corresponding relationship between the triangular mesh facets and the texture triangles in the second image; and performing texture mapping on the 3D point cloud grid model by using the second image sequence and the corresponding relationship. The method can efficiently and high-quality complete the texture rendering of the 3D point cloud grid model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a texture mapping method, storage medium, and intraoral scanning system. Background Technology

[0002] Currently, there are two main methods for texture mapping of 3D scanned point cloud mesh models. One method is to obtain a texture atlas by fusing keyframes. The texture blocks after image fusion have good consistency to meet the needs of texture rendering of 3D point cloud mesh models. The other method is to select an optimal viewpoint for each facet on the 3D point cloud mesh model obtained by scanning and reconstruction on its corresponding texture image, and transform the optimal viewpoint selection problem into solving an optimization problem based on Markov random fields. Finally, subsequent optimization is used to reduce texture seams.

[0003] However, methods for selecting the optimal viewpoint for each facet in a 3D point cloud mesh model are limited by the size of the point cloud and the number of image frames involved in the scan reconstruction. Generally, when the scanned point cloud size exceeds several million, the number of frames involved in the scan reconstruction reaches several thousand. At this point, traversing all images facet by face is time-consuming and cannot meet the high time requirements for 3D scan reconstruction in medical clinical dentistry, where texture rendering of point clouds is critical. Using keyframe selection followed by image fusion can lead to incomplete alignment of texture blocks at texture boundaries if there are slight errors in camera parameter calibration, resulting in ghosting and obvious texture seams. Furthermore, due to the resolution inconsistency of the corresponding facet on different images during 3D scanning acquisition, directly fusing these images will result in blurred textures in the final product. Weighted image fusion, on the other hand, causes image resampling, leading to texture distortion. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in related technologies. Therefore, one object of this invention is to provide a texture mapping method, storage medium, and intraoral scanning system to efficiently and effectively perform texture mapping on 3D point cloud mesh models.

[0005] In a first aspect, the present invention proposes a texture mapping method, the method comprising: acquiring a 2D image sequence, a 3D point cloud mesh model, and scanning parameters in a 3D scan; selecting keyframes in the 2D image sequence according to the 2D image sequence, the 3D point cloud mesh model, and the scanning parameters to obtain a keyframe sequence; adjusting the brightness of all keyframes in the keyframe sequence to obtain a first image sequence; and calculating the first gradient magnitude and the gradient value of all first triangular mesh faces on the corresponding first images according to the first images corresponding to all first triangular mesh faces, the 3D point cloud mesh model, and the scanning parameters. The first shooting angle corresponding to the first image is described, wherein all first triangular mesh patches are patches on the 3D point cloud mesh model; using a Markov random field function, based on the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, the optimal first image is selected to obtain a second image sequence, and the correspondence between the first triangular mesh patches and the texture triangles in the second image; using the second image sequence and the correspondence between the first triangular mesh patches and the texture triangles in the second image, texture mapping is performed on the 3D point cloud mesh model.

[0006] Secondly, the present invention proposes a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the above-described texture mapping method is implemented.

[0007] Thirdly, the present invention proposes an intraoral scanning system, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, it implements the texture mapping method described above.

[0008] The texture mapping method, storage medium, and intraoral scanning system of this invention can efficiently and with high quality complete the texture rendering of 3D point cloud mesh models (such as 3D point cloud mesh models of tooth surfaces in the oral cavity), which can meet the clinical application requirements of true color oral digital impression instrument for texture rendering of 3D point cloud mesh models and improve the doctor-patient diagnosis and treatment experience.

[0009] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0010] Figure 1 This is a flowchart of a texture mapping method according to one embodiment of the present invention;

[0011] Figure 2 This is a 2D image sequence of a plaster dental membrane, as an example of the present invention;

[0012] Figure 3This is an image after keyframe selection of a 2D image sequence of a plaster dental membrane, as an example of the present invention.

[0013] Figure 4 This is a schematic diagram of a 3D point cloud mesh model of a plaster dental membrane, as an example of the present invention.

[0014] Figure 5 This is a 2D image sequence of a natural tooth, as an example of the present invention;

[0015] Figure 6 This is an image of a 2D image sequence of a real tooth after keyframe selection, as an example of the present invention.

[0016] Figure 7 This is a schematic diagram of a 3D point cloud mesh model of a natural tooth, as an example of the present invention.

[0017] Figure 8 This is a schematic diagram of a triangular mesh facet in a 3D point cloud mesh model of a plaster dental membrane, as an example of the present invention.

[0018] Figure 9 This is a schematic diagram illustrating the distribution of 2D images in a target area according to an example of the present invention;

[0019] Figure 10 This is a flowchart of step S12 of an embodiment of the present invention;

[0020] Figure 11 This is a schematic diagram illustrating color adjustment of a first region, as an example of the present invention.

[0021] Figure 12 This is a graph showing the change of the objective function value of a Markov random field as a function of the number of iterations, according to an example of the present invention.

[0022] Figure 13 This is a schematic diagram of the texture mapping result of an example of the present invention;

[0023] Figure 14 Yes Figure 13 The diagram shows the texture mapping result after global color adjustment.

[0024] Figure 15 This is a schematic diagram of a constraint box for a Poisson fusion algorithm, as an example of the present invention;

[0025] Figure 16 This is a texture image, which is an example of the present invention. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0027] The following is a reference appendix. Figure 1-16 This invention describes a texture mapping method, storage medium, and intraoral scanning system according to embodiments of the present invention.

[0028] Figure 1 This is a flowchart of a texture mapping method according to an embodiment of the present invention.

[0029] like Figure 1 As shown, texture mapping methods include:

[0030] S11, acquire the 2D image sequence, 3D point cloud mesh model and scanning parameters in the 3D scan.

[0031] Specifically, a three-dimensional scan of teeth, plaster dental models, etc., can be performed using an intraoral scanning system to obtain 2D rendered textures. These 2D rendered textures are sequences of 2D images, which can be time-series images containing thousands or even tens of thousands of 2D images. Figure 2 , Figure 4 As shown, only a few hundred 2D images are displayed. Figure 8 As shown, the 3D point cloud mesh model contains the vertex coordinate data of all triangular mesh faces. The scanning head of the intraoral scanning system can include a camera and a projector. The camera and projector, as the data acquisition module and image processing module for 3D scanning, together constitute the hardware and software components of the intraoral scanning system. Therefore, the scanning process of the intraoral scanning system can be achieved through the combined action of the camera and projector. Scanning parameters can include camera parameters (including internal and external camera parameters), projector external parameters, etc. Based on the scanning parameters, the rendering mapping relationship between the 3D point cloud mesh model and the 2D images in the 2D image sequence can be obtained.

[0032] S12, based on the 2D image sequence, 3D point cloud mesh model and scanning parameters, key frames are selected from the 2D image sequence to obtain the key frame sequence.

[0033] As an example, such as Figure 10 As shown, step S12 may include:

[0034] S121, downsample the 3D point cloud mesh model to obtain a 3D point cloud mesh downsampled model.

[0035] Specifically, after obtaining the 3D point cloud mesh model, the scanned model boundary of the 3D point cloud mesh model can be identified; then, the 3D point cloud mesh model is downsampled based on the scanned model boundary. This downsampling based on the scanned model boundary can include: downsampling the mesh vertices in the 3D point cloud mesh model excluding the scanned model boundary, for example, after fixing the scanned model boundary, using a mean sampling method, random sampling method, etc., to downsample the mesh vertices excluding the scanned model boundary, obtaining sampled mesh vertices; and reconstructing the 3D point cloud mesh model based on the sampled mesh vertices and the mesh vertices above the scanned model boundary, obtaining the downsampled 3D point cloud mesh model.

[0036] Specifically, the boundaries of the scanned model can be identified when acquiring a 3D point cloud mesh model. See also Figure 3 , Figure 5 The scanned model boundary refers to the model boundary, shown as a boundary line. During downsampling, only the mesh vertices outside the scanned model boundary can be downsampled, while retaining all mesh vertices on the scanned model boundary. Thus, by downsampling the 3D point cloud mesh model with fixed boundaries, a 3D point cloud mesh downsampled model with reduced data volume and preserved original scanned model boundaries is obtained, facilitating faster selection of keyframes from 2D image sequences.

[0037] It should be noted that after selecting the keyframes, the 3D model used for texture mapping is still the 3D point cloud mesh model before downsampling, so as to ensure that the texture mapping does not lose the input 3D point cloud mesh model.

[0038] S122, based on the 2D images, 3D point cloud mesh downsampling models and scanning parameters corresponding to all second triangular mesh patches, calculate the second gradient magnitude of all second triangular mesh patches on the corresponding 2D images and the second shooting angle of the corresponding 2D images.

[0039] Among them, all second triangular mesh patches are patches on the 3D point cloud mesh downsampling model, and the 2D images corresponding to all second triangular mesh patches are images in the 2D image sequence.

[0040] Specifically, before calculating the second gradient magnitude and the second shooting angle, the correspondence between all second triangular mesh patches in the 3D point cloud mesh downsampling model and the 2D images in the 2D image sequence can be determined based on the 2D image sequence, the 3D point cloud mesh model, and the scanning parameters. This correspondence can be one-to-one, meaning one second triangular mesh patch corresponds to one 2D image, or one-to-many, meaning one second triangular mesh patch corresponds to multiple 2D images. In other words, each 2D image in the 2D image sequence has a corresponding triangular mesh patch. Before determining the correspondence between all second triangular mesh patches in the 3D point cloud mesh downsampling model and the 2D images in the 2D image sequence, the correspondence between three-dimensional spatial points in the 3D point cloud mesh model and their corresponding two-dimensional pixels in the 2D images can be calculated. The calculation process is as follows:

[0041] Taking a monocular camera as an example, Table 1 below shows the mathematical model parameters of a monocular camera:

[0042] Table 1

[0043]

[0044] In the table above, f x f y For parameters related to the camera focal length, u0 and v0 are the coordinates of the principal point of the image, c is the non-perpendicularity factor of the coordinate axes, which is usually approximated as 0; k1, k2, p1, p2 are the radial and tangential distortion coefficients of the camera lens; R and T are the transformation matrices between the world coordinate system and the camera coordinate system.

[0045] Let p 1j =[x w1j y w1j z w1j 1] T Let p2 be the homogeneous world coordinates of point 1, and set it as the global world coordinate system on this virtual plane. w2j y w2j z w2j 1] T Let q be the homogeneous world coordinates of point 2. 1j =[u 1j v 1j 1] T q 2j =[u 2j v 2j 1] T According to the affine transformation of the camera, we have:

[0046]

[0047] Considering the influence of camera lens distortion, the distorted coordinate vector (x) d ,yd ) T With respect to the ideal coordinate vector (x, y) T The relationship is as follows:

[0048]

[0049] In the above formula, r 2 =x 2 +y 2 Substituting the normal position coordinates into the above formula yields the distorted position coordinates, which in turn provides a relatively accurate coordinate correspondence between the three-dimensional spatial points of the 3D point cloud mesh model and their corresponding two-dimensional pixels on the 2D image.

[0050] Based on the coordinate correspondence between the 3D spatial points of the 3D point cloud mesh model and their corresponding 2D pixels in the 2D image, and the relationship between the 3D point cloud mesh model and the 3D point cloud mesh downsampling model, the correspondence between all second triangular mesh patches in the 3D point cloud mesh downsampling model and the 2D images in the 2D image sequence can be obtained. Then, the second gradient magnitude of the three points of each second triangular mesh patch in the 3D point cloud mesh downsampling model on the corresponding 2D image can be calculated. Simultaneously, the patch normals of all second triangular mesh patches in the 3D point cloud mesh downsampling model can be calculated. Based on these patch normals and scanning parameters, the second shooting angle, i.e., the angle between the normal direction and the camera's line of sight, can be obtained. The calculation process for the patch normals may include: calculating the cross product of the vectors corresponding to any two edges of the second triangular mesh patch, normalizing the cross product, and taking the direction upwards (i.e., away from the scanned object such as teeth).

[0051] It should be noted that since the acquired 2D images may be blurred due to excessive camera movement or camera shake causing defocusing, after obtaining the 2D image sequence, the correspondence between the blurred image and the second triangular mesh patch can be removed first.

[0052] As an example, before calculating the gradient magnitude and shooting angle, the blur of each 2D image in the 2D image sequence can be calculated first. Then, the correspondence between all second triangular mesh patches and 2D images in the 2D image sequence can be filtered based on the blur. Then, based on the filtered correspondence, the 3D point cloud mesh downsampling model and the scanning parameters, the second gradient magnitude of all second triangular mesh patches in the 3D point cloud mesh downsampling model on the corresponding 2D image and the second shooting angle of the corresponding 2D image can be calculated.

[0053] Specifically, a low-pass filter can be used to blur each 2D image in the 2D image sequence; based on each 2D image in the 2D image sequence and its corresponding blurred image, the blur degree of each 2D image is calculated; a preset blur degree standard is obtained; based on the blur degree and the preset blur degree standard, the blurred image in the 2D image sequence is obtained; the correspondence between the blurred image and its corresponding second triangular mesh patch is removed. Finally, combining the 3D point cloud mesh downsampling model and scanning parameters, the second gradient magnitude of all second triangular mesh patches in the 3D point cloud mesh downsampling model on the corresponding 2D image and the second shooting angle of the corresponding 2D image are calculated.

[0054] Specifically, image blurriness is an indicator of image sharpness. Generally, blurred images have indistinct edges, while sharp images have sharper edges. Filtering a 2D image using a low-pass filter, i.e., blurring the 2D image, can be achieved by averaging the values ​​of surrounding pixels for each pixel in the 2D image. During blurring, a larger low-pass filter kernel (i.e., a larger range of surrounding pixels) results in more similar pixel values ​​among neighboring pixels, leading to a stronger blurring effect. Since the blurred image after low-pass filtering is very close to the blurred image obtained through scanning, blurriness can be calculated by comparing the original 2D image with its corresponding blurred image. Blurriness can be the difference in pixel grayscale values ​​between the original 2D image and the corresponding blurred image. Based on the calculated blurriness and a pre-set blurriness standard, the more blurred 2D image containing repetitive texture information in the 2D image sequence (i.e., the blurred image) can be identified, and the correspondence between the blurred image and its corresponding second triangular mesh facet can be removed. Finally, the subsequent calculations are performed based on the correspondence between the second triangular mesh facets after removing the blurry image and the corresponding second triangular mesh facets, and the 2D image and its correspondence.

[0055] Optionally, after calculating the ambiguity, the images can be sorted according to their ambiguity magnitude. 2D images with preset values ​​before sorting are then classified as blurred images, and the correspondence between the blurred images and their corresponding second triangular mesh faces is removed. The preset data can be determined based on the total number of images in the 2D image sequence and is positively correlated with that total number.

[0056] S123, using the Markov random field function, based on the second gradient magnitude of all second triangular mesh patches on the corresponding 2D image and the second shooting angle of the corresponding 2D image, the optimal 2D image is selected to obtain the target image sequence and the correspondence between the second triangular mesh patches and the target images in the target image sequence.

[0057] It should be noted that the optimal 2D image here refers to the unique target image selected for the triangular mesh using the Markov random field function. This target image has higher texture quality and clarity compared to other 2D images corresponding to the triangular mesh.

[0058] The correspondence between the triangular mesh facet and the target image in the target image sequence can be the correspondence between the vertex coordinates of the second triangular mesh facet and the pixel points on the target image.

[0059] Specifically, step S123 may include:

[0060] S1231, Based on the gradient magnitude and shooting angle, establish the following Markov random field function E. l :

[0061]

[0062] in, It considers not only the area and sharpness of the triangular texture image, but also the camera's shooting angle, aiming to select a 2D image with relatively high texture quality (or high sharpness) for each second triangular mesh facet. In the 3D point cloud mesh downsampling model, the second triangular mesh face F represents... k In the corresponding 2D image l k The gradient magnitude φ at the projection point p in the middle lk (F k ,l k ) represents the second triangular mesh face F k In the corresponding 2D image l k The projection onto the image, θ represents the 2D image l. k The shooting angle, F k ,F j This represents two adjacent second triangular mesh faces with a common edge in a 3D point cloud mesh downsampling model. k ,l j These represent the second triangular mesh face F respectively. k ,F j The corresponding 2D image, edges represents the set of common edges of adjacent faces, E smooth The smoothing term, designed to minimize the seam error between the texture images selected for two adjacent second triangular mesh faces, is shown in the following formula:

[0063]

[0064] That is, if two adjacent patches come from the same 2D image, the smoothing term is set to 0; otherwise, the smoothing term is set to 1. By selecting textures from the same image as much as possible, texture seams and texture discretization can be avoided.

[0065] Alternatively, the smoothing term can also be obtained by the following formula:

[0066]

[0067] Where, δ (k,j) This refers to the ratio of the actual length of the common side of two adjacent second triangles to the number of pixels in their corresponding 2D image. This refers to p in the selected 2D image l k The projection on the surface.

[0068] S1232, solve the Markov random field function to obtain the optimal 2D image corresponding to all second triangular mesh patches in the 3D point cloud mesh downsampling model, i.e., the target image.

[0069] Specifically, the problem of selecting the optimal 2D image for a triangular facet can be viewed as a process of optimal texture stitching, which can be solved using a Markov random field-based method according to equation (1) above. The data term E, which is one of the inputs to the Markov random field function, is... data (F k ,l k This can be achieved using CUDA (Compute Unified Device Architecture) for parallel computation. The reason is that the product of the gradient magnitude of each second triangular mesh facet on its corresponding 2D image and the sine of the shooting angle of the corresponding 2D image are independent and parallel, thus allowing for parallel computation using CUDA. Simultaneously, a binary function, using a smoothing term to determine whether the 2D images corresponding to two adjacent second triangular mesh faces are the same image, calls a Markov random field-based function to perform viewpoint selection calculations for the faces, outputting the selected result—the unique, highest-quality 2D image corresponding to all second triangular mesh faces in the 3D point cloud mesh downsampling model. In other words, after the Markov random field-based computation, each second triangular mesh facet corresponds one-to-one with the selected optimal 2D image; however, before the Markov random field-based computation, one second triangular mesh facet might correspond to multiple 2D images or only one 2D image.

[0070] It should be noted that removing the correspondence between the blurred image and the corresponding second triangular mesh facet may result in the 3D point cloud mesh model containing second triangular mesh facets corresponding to the default 2D image. This means that the texture of these second triangular mesh facets is missing, or they are not covered by the image texture, thus preventing texture mapping and ensuring texture coverage of all areas of the 3D point cloud mesh model. In this case, a default 2D image must be selected from the removed modular images to ensure that all second triangular mesh facets are covered by the 2D image.

[0071] In one example, after removing the correspondence between the blurred image and the corresponding second triangular mesh patch, and before calculating the second gradient magnitude and the second shooting angle, it can be determined whether there is a second triangular mesh patch corresponding to the default 2D image in the 3D point cloud mesh downsampling model based on the filtered 2D image sequence, the 3D point cloud mesh downsampling model, and the scanning parameters; if there is, at least one of the blurred images can be selected to replace the default 2D image so as to correspond to the second triangular mesh patch.

[0072] In this example, when using the Markov random field function for calculation, all second triangular mesh patches in the 3D point cloud mesh downsampling model have corresponding 2D images. Therefore, the second gradient magnitude and the second shooting angle can be calculated for all second triangular mesh patches for Markov random field function calculation.

[0073] To facilitate 2D image processing, after obtaining the 2D image sequence, the 2D images in the sequence can be numbered sequentially to obtain the first image label value sequence. After filtering the correspondence between blurred images and their corresponding triangular mesh patches, the first image label value sequence needs to be updated simultaneously. At least one blurred image is selected to replace the default 2D image to correspond with the second triangular mesh patch, and then the first image label value sequence can be updated again to obtain the second image label value sequence. Therefore, the correspondence between the second triangular mesh patch and the 2D image can be determined by the image label value corresponding to the second triangular mesh patch, i.e., which 2D image the patch corresponds to. The blurred image is also a texture of the second triangular mesh patch on the 3D point cloud mesh downsampling model. If, after removing the correspondence between the blurred image and its corresponding triangular mesh patch, the second triangular mesh patch has no corresponding texture, it means that the blurred image represents all the corresponding textures of that part of the second triangular mesh patch and cannot be discarded entirely. Therefore, it is necessary to reassign these second triangular mesh patches using image label value calculation. In other words, after the correspondence between the blurred image and the corresponding triangular mesh patch is removed, the second triangular mesh patch corresponding to the blurred image is not assigned an image label value. That is, the second triangular mesh patch has a default image label value. At this time, the image label value corresponding to these second triangular mesh patches that do not have an overwritten image label value needs to be recalculated.

[0074] For example, a 2D image sequence contains 10 2D images, which can be numbered 1, 2, 3, ..., 9, 10, serving as image label values ​​in the first image label value sequence. These 10 2D images can cover all the second triangular mesh faces in the 3D point cloud mesh downsampling model. After blurring the images, the image label values ​​assigned to the second triangular mesh faces are 1, 3, 5, 6, 7, 8, 9, 10. This results in one second triangular mesh face in the 3D point cloud mesh downsampling model lacking a corresponding image label value. In other words, a 2D image corresponding to one second triangular mesh face is missing, requiring a new 2D image to be selected as the texture for that second triangular mesh face. At this point, we can first update 1, 3, 5, 6, 7, 8, 9, 10 to 1, 2, 3, 4, 5, 6, 7, 8, and then use the default image label values ​​to replace the default 2D image placeholders. Then, we select the 2D image corresponding to the default second triangular mesh facet from the removed blurred images. If the new selection from the blurred image is 4, we can update the image label value sequence again to obtain the corresponding second image label value sequence: 1, 2, 3, 4, 5, 6, 7, 8, 9, where the original 4 is encoded as 9.

[0075] In this example, when using the Markov random field function for calculation, there is a second triangular mesh facet in the 3D point cloud mesh downsampling model that corresponds to the default 2D image. Therefore, when performing the Markov random field function calculation, the second gradient magnitude and the second shooting angle corresponding to the second triangular mesh facet of the default 2D image can be set to 0 for use in the Markov random field function calculation.

[0076] In another example, after removing the correspondence between the blurred image and the corresponding second triangular mesh patch, the optimal 2D image selection can be performed using the Markov random field function based on the second gradient magnitude of all second triangular mesh patches on the corresponding 2D image and the second shooting angle of the corresponding 2D image, to obtain the target image sequence and the correspondence between the second triangular mesh patch and the target image in the target image sequence. Then, default compensation is performed.

[0077] Specifically, based on the correspondence between the second triangular mesh patch and the target image in the target image sequence, it is determined whether there is a default 2D image corresponding to the second triangular mesh patch.

[0078] If it exists, select one from the blurred images to replace the default 2D image, so that it corresponds to the second triangular mesh facet, and use it as the target image corresponding to the second triangular mesh facet of the default 2D image.

[0079] Alternatively, if present, re-execute the following steps: based on the 2D images corresponding to all second triangular mesh patches in the 3D point cloud mesh downsampling model, the 3D point cloud mesh downsampling model, and the scanning parameters, calculate the second gradient magnitude of all second triangular mesh patches on the corresponding 2D images and the second shooting angle of the corresponding 2D images. Then, using the Markov random field function, select the optimal 2D image based on the second gradient magnitude of all second triangular mesh patches on the corresponding 2D images and the second shooting angle of the corresponding 2D images to obtain the target image sequence and the correspondence between the second triangular mesh patches and the target images in the target image sequence.

[0080] From the blurred image, the image sequence corresponding to the second triangular mesh patch (i.e., the part of the 2D image sequence corresponding to the second triangular mesh patch) is selected. The second gradient magnitude and the second shooting angle of all second triangular mesh patches in the 3D point cloud mesh downsampling model on the corresponding 2D image are recalculated. Since the overall 2D image corresponding to the second triangular mesh patch has changed, this step yields a new second gradient magnitude and second shooting angle. Then, based on the obtained new second gradient magnitude and second shooting angle, the optimal 2D image is reselected, resulting in the target image sequence and the correspondence between the new second triangular mesh patch and the target image in the target image sequence.

[0081] S124: Calculate the distribution of the number of second triangular mesh patches in the target image, and select keyframes from all target images based on the distribution using the 3σ criterion.

[0082] Specifically, step S124 may include:

[0083] S1241, count the number of second triangular mesh patches corresponding to each target image, and calculate the standard deviation σ based on the sequence of the numbers;

[0084] S1242 uses the target image corresponding to the second triangular mesh facet that satisfies the 3σ criterion as a keyframe.

[0085] Specifically, the Markov random field computes a unique image label value for each second triangular mesh facet.

[0086] In one embodiment, the 3D point cloud mesh downsampling model can be divided into target regions based on the image label values ​​of the target image. For example, second triangular mesh patches corresponding to the same or similar image label values ​​can be divided into the same target region. Simultaneously, the number of second triangular mesh patches in the divided target region must conform to a normal distribution. After dividing the target region, the number of second triangular mesh patches in each target region is counted to form a sequence, and the standard deviation σ is calculated for the sequence of patch counts. The image label values ​​corresponding to the number of patches between [-3σ, 3σ] can be determined as the selected label values, and the optimal 2D images corresponding to these label values ​​are the keyframes. Thus, the selected 2D images can be optimized, further reducing the number of selected 2D images.

[0087] It should be noted that, to ensure full coverage of the second triangular mesh, the image label values ​​corresponding to the number of patches outside [-3σ, 3σ] are first discarded, and then the image label values ​​of the patches distributed outside [-3σ, 3σ] are set to the same as the image label values ​​distributed within [-3σ, 3σ]. For example, as... Figure 9 As shown, target region 1 with image label value 014 is surrounded by target region with image label value 003, and target region 2 with image label value 014 is surrounded by target region with image label value 004. At this time, the image label value of target region 1 can be reset to image label value 003, and the image label value of target region 2 can be reset to image label 004.

[0088] like Figure 2 , Figure 5 The image shown is a plaster dental model and a true color image of a real tooth, which were acquired during the 3D scanning and reconstruction process as the source of texture mapping. Figure 3 , Figure 6 The image shown is a 2D image selected using the keyframe selection method of this invention. The final selected 2D image has good quality and can completely cover the scanned and reconstructed area. Figure 4 , 7 The surface of the 3D point cloud mesh model shown can be used as a texture source for subsequent point cloud rendering.

[0089] Furthermore, it's important to note that Markov random fields (MRFs) characterize the local interactions between pixels in an image and their neighboring pixels' grayscale values. Specifically, MRFs apply local functional relationships to a set of points, reflecting the interactions of pixels at grayscale levels. The global functional mapping can be described using probability theory. MRFs can reflect both the randomness and the underlying structure of an image, thus effectively describing its properties.

[0090] This invention requires projecting a downsampled 3D point cloud mesh model onto its corresponding 2D image to calculate the sharpness evaluation index of the image texture corresponding to each second triangular mesh facet. This part of the calculation can be performed in parallel using CUDA in a GPU (Graphics Processing Unit). A multi-threaded control module is designed in the GPU. By setting the CUDA kernel function and setting an appropriate number of thread blocks and threads, the program automatically allocates threads for calculation. Each thread is designed to process one second triangular mesh facet, calculating its quality evaluation index in parallel for hundreds of thousands of triangular facests. This enables high-speed switching between threads to cope with large-scale parallel computing and hidden latency, achieving the goal of concurrent throughput of hundreds of millions of data points. It can more efficiently complete the calculation of the comprehensive quality evaluation index of all facests on keyframes.

[0091] The specific steps for setting up the CUDA kernel function are as follows: First, set the number of two-dimensional threads in the thread block according to the data scale of the facets and the configuration of the graphics card. Then, set the number of two-dimensional thread blocks in the thread grid according to the number of faces of the second triangle, the number of images, and the number of threads in the thread block.

[0092] Therefore, by selecting the keyframes mentioned above, we can reduce the amount of computation and improve the efficiency of the program by covering the model surface with as few images as possible while ensuring the quality of the generated texture.

[0093] S13, adjust the brightness of all keyframes in the keyframe sequence to obtain the first image sequence.

[0094] Specifically, considering the internal environment of the oral cavity, the acquired 2D images are prone to forming highly reflective local areas. Therefore, the overall brightness of the directly acquired images is relatively dark. It is necessary to perform non-linear adaptive brightness adjustment on the selected keyframes to make the brightness of the keyframes more consistent. This can be achieved by following the steps S131 to S133:

[0095] S131, calculate the brightness value of all keyframes in the keyframe sequence.

[0096] Specifically, the brightness value can be calculated based on the average color value of each channel (such as R, G, B channels) in the key frame.

[0097] S132, determine a first brightness threshold and a second brightness threshold based on the brightness values, wherein the first brightness threshold is less than the second brightness threshold, the first brightness threshold is greater than the minimum value among the brightness values, and the second brightness value is greater than or equal to the maximum value among the brightness values.

[0098] S133, adjust the brightness values ​​of keyframes whose brightness values ​​are less than the first brightness threshold to the first brightness threshold, and adjust the brightness values ​​of other keyframes to the second brightness threshold.

[0099] Specifically, through the brightness adjustment described above, the brightness enhancement ratio of each keyframe can be obtained. The ratio is equal to the brightness threshold divided by the brightness value of the corresponding keyframe before adjustment. Since each ratio is greater than or equal to 1, the overall brightness of each keyframe is improved.

[0100] It should be noted that three, four, or even more brightness thresholds can be set for segmented brightness enhancement adjustments. The minimum threshold among these multiple brightness thresholds should be greater than the minimum brightness value of all keyframes, and the maximum threshold should be less than or equal to the maximum brightness value of all keyframes. If, after overall brightness enhancement of the keyframes, darker areas still appear in certain keyframes, further non-linear brightness enhancement of the image is required.

[0101] Therefore, by performing brightness processing on the selected keyframes, the texture image can have better brightness consistency, thereby reducing redundant information, motion blur or image defocus, avoiding texture seams, and improving the efficiency of texture mapping and the quality of generated textures.

[0102] S14. Based on the first image, 3D point cloud mesh model and scanning parameters corresponding to all first triangular mesh patches, calculate the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image.

[0103] Among them, all the first triangular mesh patches are patches on the 3D point cloud mesh model.

[0104] It should be noted that the first triangular mesh patch and the second triangular mesh patch are actually both triangular mesh patches. The difference between the first and second is to distinguish the origin of the triangular mesh patches. That is, the first triangular mesh patch is a triangular mesh patch in the 3D point cloud mesh model, while the second triangular mesh patch is a triangular mesh patch in the 3D point cloud mesh downsampling model.

[0105] As an example, the first image in the first image sequence is an intraoral image. Before calculating the first gradient magnitude and the first shooting angle, the method further includes: identifying teeth and gums in the first image; and color-adjusting a first region outside the teeth and gums in the first image.

[0106] Specifically, AI (Artificial Intelligence) algorithms can be used to identify teeth and gums in the first image. After identifying teeth and gums, the pixel values ​​of the pixels in the first region can be set to a first preset value (e.g., 0, see [link]). Figure 11(a)); Determine the first boundary between the first region and the gingival region; Fill the pixels in the first region with pixel values ​​(i.e., gray values) based on the pixel values ​​(i.e., gray values) of the pixels on the first boundary.

[0107] The process of filling pixels in the first region with pixel values ​​based on the pixel values ​​of pixels on the first boundary may include: assigning the pixel values ​​of pixels on the first boundary to target pixels in the same row and / or column, wherein the target pixels are pixels whose pixel values ​​are equal to a first preset value. Figure 11 (a) After filling, see [link / reference] Figure 11 (b)).

[0108] Specifically, if the boundary of the real tooth image after AI soft tissue removal is not filled and texture mapping is performed directly, black stripe areas may appear at the boundary of the final generated texture due to two reasons: slight errors in system parameters or the selection of textures from the same image when choosing textures for the first triangular mesh facet. This reduces the quality of the generated texture. Filling the boundary pixel values ​​can solve the black edge problem.

[0109] S15, using the Markov random field function, based on the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, the optimal first image is selected to obtain the second image sequence and the correspondence between the first triangular mesh patches and the texture triangles in the second image.

[0110] The correspondence between the first triangular mesh and the texture triangle in the second image can be the correspondence between the three vertices of the first triangular mesh and the three vertices of the texture triangle in the second image.

[0111] It should be noted that the method for selecting the optimal first image is similar to the method for selecting the optimal 2D image described above. Similar to the method for selecting the optimal 2D image, CUDA is used to perform an optimization problem based on Markov random fields to select the best viewpoint for all first triangular mesh faces in the first image sequence. The relationship between the objective function value and the number of iterations is as follows: Figure 12 As shown, the minimum value of the objective function depends on the following four factors:

[0112] (1) The gradient magnitude on the first image corresponding to the first triangular mesh patch. Generally, the larger the gradient magnitude on the image corresponding to the patch, the clearer the texture, the richer the texture details, and the higher the quality of the final generated texture.

[0113] (2) The angle between the normal direction of the first triangular mesh and the camera's line of sight. The smaller the angle, the more accurate the shooting angle, and the clearer the texture details of the captured image.

[0114] (3) The average color of the facets on the 2D image corresponding to the first triangular mesh facet is used as a penalty factor. The smaller the value, the easier it is to select the black area, resulting in poor texture quality in the final generated texture and reducing the texture rendering quality of the point cloud.

[0115] (4) Try to select textures from the same first image to avoid texture seams and texture discretization.

[0116] As an example, using a Markov random field function to select the optimal first image based on the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, a second image sequence is obtained, along with the correspondence between the first triangular mesh patches and the texture triangles in the second image. This can include: performing image range detection on the first triangular mesh patches to determine whether the projections of the vertices of the first triangular mesh patches onto the corresponding first image are all within the corresponding first image; if they are, performing geometric visibility detection on the first triangular mesh patches to determine whether the vertices of the first triangular mesh patches are visible in the scanning viewpoint corresponding to the first image; otherwise, removing the corresponding first triangular mesh patches and their corresponding texture triangles. The correspondence between the first image and the first image is determined. If the first image is visible, color anomaly detection is performed on the first image corresponding to the first triangular mesh to determine whether there is a color anomaly in the corresponding first image. Otherwise, the correspondence between the corresponding first triangular mesh and the first image is removed. If there is a color anomaly, the color anomaly value is corrected for the corresponding first image. Otherwise, the correspondence between the corresponding first triangular mesh and the first image is retained. The optimal first image is selected using the Markov random field function based on the first gradient magnitude of all adjusted first triangular meshes on the corresponding first image and the first shooting angle of the corresponding first image to obtain the second image sequence and the correspondence between the first triangular mesh and the texture triangle in the second image.

[0117] It should be noted that image range detection, geometric visibility detection, color anomaly detection, and color anomaly correction are performed first to remove some correspondences. Compared with the above keyframe selection and the deletion of the correspondence between the blurred image and the second triangular mesh facet, this is the adjustment of the local image (i.e., texture triangle), while the keyframe selection and blurred image processing are the adjustment of the correspondences of the overall image.

[0118] Specifically, image range detection refers to determining whether the vertex coordinates of the first triangular mesh facet in the 3D point cloud mesh model, after being transformed by system parameters and back-projected onto the first image, fall within the size range of the first image. If the vertex exceeds the range, it indicates that the vertex is not within the first image.

[0119] Because 3D point cloud mesh models have issues such as corners and occlusion, vertices and triangular mesh faces that meet the image range detection criteria are not necessarily visible in their corresponding first images. Therefore, further detection of the correspondence between vertices and their corresponding first images is required. Geometric visibility detection can be based on BVH (Bounding Volume Hierarchies) trees, including: building a BVH tree for the coordinates of the first triangular mesh face and its corresponding vertex; and performing bounding box collision detection on the vertices using the BVH tree to determine whether the vertex is visible in the camera view of the corresponding first image.

[0120] To remove color anomalies caused by occlusion in local areas of the first image, it is also necessary to perform color anomaly correction on the image textures in the first image corresponding to each first triangular mesh facet that exhibits color anomalies. This color anomaly correction can be achieved using a color anomaly removal algorithm based on mean shift. For each first triangular mesh facet, the correspondences with occluded first images are removed from the set of correspondences with the first image corresponding to it. The optimal viewpoint is then selected from the remaining set of first image correspondences. Assuming that erroneous and correct colors are completely independent, views with inconsistent colors are removed using color mean shift. The specific steps are as follows:

[0121] A. For each first triangular facet, calculate the average color of the first image projection triangle region corresponding to its first image correspondence set;

[0122] The average color can be the average value of a pixel across all color channels (such as R, G, and B channels).

[0123] B. Make the set of correspondences between the first images of each first triangular facet a valid set;

[0124] C. Calculate the mean and covariance matrix of the average color of the first image corresponding to the effective set;

[0125] D. Calculate the multivariable Gaussian function for the first image corresponding to the effective set;

[0126] E. Retain the first image correspondence in the effective set whose Gaussian function value is greater than the set reasonable threshold;

[0127] F. Iterate through steps C to E multiple times (e.g., 10 times). If at this point, every term in the covariance matrix is ​​less than 1*e... -5 If the transpose of the covariance becomes unstable or the number of first images in the effective set is less than 4, the calculation is stopped. In this case, the correspondence of the first images in the effective set is the correspondence of the first images after removing color anomalies.

[0128] Therefore, by performing image range detection, geometric visibility detection, and color outlier correction as described above, severe black borders and dark areas can be avoided during texture mapping.

[0129] S16, using the second image sequence and the correspondence between the first triangular mesh patch and the texture triangle in the second image, texture mapping is performed on the 3D point cloud mesh model.

[0130] Specifically, using the second image sequence and the correspondence between the first triangular mesh facets and the texture triangles in the second image, texture mapping is performed on the 3D point cloud mesh model. The resulting texture mapping can be as follows: Figure 13 As shown.

[0131] It should be noted that after the above image range detection, geometric visibility detection or color anomaly detection, the correspondence between the first triangular mesh and the first image will be removed, which may result in the first triangular mesh corresponding to the default first image. This will cause the texture map to be missing or have black edges.

[0132] Therefore, as an example, before applying texture mapping to the 3D point cloud mesh model using the second image sequence, the method may further include: determining whether the second image in the second image sequence covers all the first triangular mesh faces; if not, then selecting at least one of the image range detection parameters, geometric visibility detection parameters, color anomaly detection parameters, and color anomaly correction parameters, and re-executing the optimal first image selection using a Markov random field function based on the first gradient magnitude of all the first triangular mesh faces on the corresponding first image and the first shooting angle of the corresponding first image, until all the first triangular mesh faces are covered by the second image in the newly obtained second image sequence.

[0133] Among them, the image range detection parameter can be the projection angle of the vertex of the first triangular mesh facet on the first image, such as adjusting it from the optimal projection angle of 90° to 85°, 80°, etc.; the geometric visibility detection parameter can be the parameter in the BVH tree; the color anomaly detection parameter can be the anomaly judgment threshold parameter, and the color anomaly correction parameter can be the reasonable threshold set above, the iteration number 10, 1*e -5 Number of items: 4, etc.

[0134] To ensure the texture mapping effect, before applying texture mapping to the 3D point cloud mesh model using the second image sequence and the correspondence between the first triangular mesh facets and the texture triangles in the second image, the method may further include: calculating the color gradient of the texture triangles corresponding to all the first triangular mesh facets to obtain the color gradient of the texture triangles; and adjusting the global color of the texture triangles corresponding to all the first triangular mesh facets based on the color gradient.

[0135] Specifically, this invention selects the optimal texture source for each first triangular mesh facet on a 3D point cloud mesh model, and tries to select textures from the same image to minimize texture gaps. However, the final generated texture may still have obvious texture seams due to significant differences in brightness between different texture images. To reduce this impact, it is necessary to use color gradients to adjust the global color of the texture for consistency, and to calculate a brightness compensation value for each first triangular mesh facet. If the textures of adjacent first triangular mesh facest come from different images, the brightness difference between them can be made to be consistent through appropriate brightness compensation, thereby avoiding the noticeable texture seams caused by inconsistent global brightness. Figure 14 As shown.

[0136] Before applying global color consistency adjustment based on color gradient, it is determined that each first triangular mesh facet has a corresponding texture block (i.e., texture triangle) in the second image. At the boundary of the texture block, the vertices of the gaps are split into vertices belonging to the left texture block. left And the vertex v belonging to the right texture block right Each vertex has a unique color value f. v Correspondingly, to ensure the uniqueness of color, a correction term g is constructed. v This ensures that every vertex globally has a unique color value, which can be achieved by minimizing the following expression:

[0137]

[0138] Where μ is a preset parameter, such as 0.2; seam patch represents adjacent facets; the first summation term in the above formula is used to calculate the color adjustment value of the 3D points on the texture boundary. After adjustment, it can ensure that the colors on both sides of the texture boundary are as similar as possible; the second summation term in the above formula is used to calculate the color adjustment value of the 3D points on the 3D point cloud mesh model, which can ensure the gradual color change; the above formula can be transformed into calculation using the gradient descent iterative method.

[0139] To further ensure the texture mapping effect, after global color adjustment, the Poisson fusion algorithm can be used to adjust the local color of the adjacent regions of the texture triangles corresponding to two adjacent first triangular mesh faces.

[0140] Specifically, if texture seams still exist after the aforementioned global optimization, a Poisson fusion algorithm can be used for local color adjustment. Poisson fusion seamlessly merges two images, using the gradient of the original image as a guide to smoothly diffuse the differences between the target scene and the original image at the boundary into the fused image. This not only preserves information from the original image but also introduces information from the fused image, allowing the fused image to seamlessly integrate into the target scene while maintaining consistent hue and luminance, resulting in a high-quality final texture.

[0141] For example, Poisson blending can be limited to a boundary of 20 pixels wide for a texture block, such as... Figure 15 As shown, the outer and inner boundaries are used as constraints for the Poisson equation. The average color of the pixels from both texture images is assigned to the pixels on the outer boundary, while the pixels on the inner boundary retain their original color values. If the texture image is too small, the inner boundary is ignored, and the Poisson equation can be derived using the linear Laplacian operator. It is important to note that the Laplacian operators of the two textures cannot be mixed for calculation.

[0142] Since only a portion of the continuous area on each image is a texture region, and the rest is unused, we can crop all the texture regions from the second image, place them on the same image, and then load the texture once.

[0143] Therefore, as an example, using the second image sequence and the correspondence between the first triangular mesh facets and the texture triangles in the second image to perform texture mapping on the 3D point cloud mesh model may include: according to the second image sequence and the correspondence between the first triangular mesh facets and the texture triangles in the second image, recombining all texture triangles into a preset number of texture images, and determining the correspondence between the first triangular mesh facets and the texture triangles in the texture images; and performing texture mapping on the 3D point cloud mesh model according to the correspondence between the first triangular mesh facets and the texture triangles in the texture images.

[0144] The preset number is set to reduce the number of images stored. Optionally, all texture triangles can be recombine into a single texture image.

[0145] Specifically, after local adjustments to the texture boundaries are completed, a 2D texture atlas is created. For each image in the texture atlas, the coordinates of the four leftmost, rightmost, topmost, and bottommost points on the texture region are calculated. Based on these coordinates, the maximum bounding box of the texture region is constructed. This same process is then applied to all images to find the bounding boxes within each image. Next, the BinPacking algorithm is used to calculate the texture size from these bounding box information and arrange them to obtain a texture image, such as... Figure 16As shown, the texture coordinates of the first triangular mesh facet can be recalculated based on the bounding box's coordinates on the new texture image. This allows for the simplification of texture data in large-scale images, resulting in higher computational efficiency.

[0146] The above process can be used to encapsulate input / output interfaces, generate DLL (Library Files) files, and integrate them into other programs, such as the main program of an intraoral scanning system for scanning.

[0147] The operational effects of this invention are as follows:

[0148] Processor: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz

[0149] RAM: 8.00GB

[0150] Graphics card / VRAM: NVIDIA GeForce GTX 1660 / 6.00GB

[0151] System type: 64-bit operating system

[0152] The time required for texture mapping is shown in Table 2 below:

[0153] Table 2

[0154]

[0155]

[0156] Referring to Table 2, with a fixed total number of 2D images, the larger the total number of first triangular mesh faces, the longer the program runs. Furthermore, even for 3D point cloud mesh models containing millions of first triangular mesh faces, and thousands of 2D images, high-quality texture mapping can be achieved in a relatively short time.

[0157] In summary, the texture mapping method of this invention can efficiently and with high quality complete the texture rendering of 3D point cloud mesh models (such as 3D point cloud mesh models of tooth surfaces in the oral cavity), which can meet the clinical application requirements of the true color dental digital impression instrument for texture rendering of 3D point cloud mesh models and improve the doctor-patient diagnosis and treatment experience.

[0158] Based on the texture mapping method described above, this invention also proposes a computer-readable storage medium.

[0159] In an embodiment of the present invention, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, it implements the texture mapping method described above.

[0160] The present invention also proposes an electronic device.

[0161] In this embodiment, the electronic device includes a memory, a processor, and a computer program stored in the memory. The computer program, when executed by the processor, implements the texture mapping method described above. This electronic device can be an intraoral scanning system.

[0162] In one embodiment, the intraoral scanning system may include a handheld image acquisition device and an image processing device. The handheld image acquisition device is used to scan and acquire intraoral image data, and the image processing device is used to process the intraoral image data.

[0163] The aforementioned memory, processor, and computer program stored in the memory can be distributed within the handheld image acquisition device and / or image processing device. The handheld image acquisition device and image processing device can be integrated or separate units.

[0164] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0165] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0166] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0167] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0168] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0169] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0170] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0171] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A texture mapping method, characterized in that, The method includes: Acquire 2D image sequences, 3D point cloud mesh models, and scanning parameters from 3D scanning; Based on the 2D image sequence, the 3D point cloud mesh model, and the scanning parameters, keyframes are selected from the 2D image sequence to obtain a keyframe sequence, including: The 3D point cloud mesh model is downsampled to obtain a 3D point cloud mesh downsampled model. Based on the 2D images corresponding to all second triangular mesh patches, the 3D point cloud mesh downsampled model, and the scanning parameters, the second gradient magnitude of all second triangular mesh patches on the corresponding 2D images and the second shooting angle of the corresponding 2D images are calculated, wherein all second triangular mesh patches are patches on the 3D point cloud mesh downsampled model. Using a Markov random field function, the optimal 2D image is selected for all second triangular mesh patches based on the second gradient magnitude and the second shooting angle. The keyframe is selected from all optimal 2D images. Brightness adjustment is performed on all keyframes in the keyframe sequence to obtain the first image sequence; Based on the first image corresponding to all first triangular mesh patches, the 3D point cloud mesh model, and the scanning parameters, calculate the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, wherein all first triangular mesh patches are patches on the 3D point cloud mesh model. Using the Markov random field function, based on the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, the optimal first image is selected to obtain the second image sequence and the correspondence between the first triangular mesh patches and the texture triangles in the second image; Using the second image sequence and the correspondence between the first triangular mesh patch and the texture triangle in the second image, texture mapping is performed on the 3D point cloud mesh model.

2. The texture mapping method according to claim 1, characterized in that, The selection of the keyframe from all optimal 2D images includes: The distribution of the number of the second triangular mesh facets in relation to the optimal 2D image is statistically analyzed, and then 3D is used. The criteria select the keyframe from all the best 2D images based on the distribution.

3. The texture mapping method according to claim 1, characterized in that, The step of adjusting the brightness of keyframes in the keyframe sequence includes: Calculate the brightness value of all key frames in the keyframe sequence; A first brightness threshold and a second brightness threshold are determined based on the brightness values, wherein the first brightness threshold is less than the second brightness threshold, the first brightness threshold is greater than the minimum value among the brightness values, and the second brightness threshold is greater than or equal to the maximum value among the brightness values. Adjust the brightness values ​​of keyframes whose brightness values ​​are less than the first brightness threshold to the first brightness threshold, and adjust the brightness values ​​of other keyframes to the second brightness threshold.

4. The texture mapping method according to claim 1, characterized in that, The first image in the first image sequence is an intraoral image. Before calculating the first gradient magnitude and the first shooting angle, the method further includes: Identify teeth and gums in the first image; The color of the first region outside the teeth and gums in the first image is adjusted.

5. The texture mapping method according to claim 4, characterized in that, The step of color adjustment for the first region outside the teeth and gums in the first image includes: Set the pixel value of the pixel in the first region to the first preset value; Determine the first boundary between the first region and the gingival region; The pixels in the first region are filled with pixel values ​​based on the pixel values ​​of the pixels on the first boundary.

6. The texture mapping method according to claim 5, characterized in that, The step of filling pixels in the first region with pixel values ​​based on the pixel values ​​of pixels on the first boundary includes: The pixel value of the pixel on the first boundary is assigned to the target pixel in the same row and / or the same column, wherein the target pixel is the pixel whose pixel value is equal to the first preset value.

7. The texture mapping method according to claim 1, characterized in that, The step of using a Markov random field function to select the optimal first image based on the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, to obtain a second image sequence, and the correspondence between the first triangular mesh patches and the texture triangles in the second image, includes: Image range detection is performed on the first triangular mesh to determine whether the projections of the vertices of the first triangular mesh onto the corresponding first image are all within the corresponding first image; If both are present, then perform geometric visibility detection on the first triangular mesh to determine whether the vertices of the first triangular mesh are visible in the scanning view corresponding to the first image; otherwise, remove the corresponding relationship between the first triangular mesh and the first image. If visible, then perform color anomaly detection on the first image corresponding to the first triangular mesh to determine whether there is a color anomaly in the corresponding first image; otherwise, remove the corresponding relationship between the first triangular mesh and the first image. If color anomalies are present, the corresponding first image is corrected for color anomaly values; otherwise, the correspondence between the corresponding first triangular mesh facet and the first image is preserved. Using a Markov random field function, based on the first gradient magnitude of all adjusted first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, the optimal first image is selected to obtain a second image sequence, as well as the correspondence between the first triangular mesh patches and the texture triangles in the second image.

8. The texture mapping method according to claim 7, characterized in that, Before performing texture mapping on the 3D point cloud mesh model using the second image sequence, the method further includes: Determine whether the second image in the second image sequence covers all the first triangular mesh faces; If not, adjust at least one of the image range detection parameters, geometric visibility detection parameters, color anomaly detection parameters, and color anomaly correction parameters, and re-execute the optimal first image selection using the Markov random field function based on the first gradient magnitude of all first triangular mesh patches on the corresponding first image and the first shooting angle of the corresponding first image, until all first triangular mesh patches are covered by the second image in the newly obtained second image sequence.

9. The texture mapping method according to claim 1, characterized in that, Before performing texture mapping on the 3D point cloud mesh model using the second image sequence and the correspondence between the first triangular mesh facet and the texture triangle in the second image, the method further includes: Calculate the color gradient of the texture triangles corresponding to all first triangular mesh faces to obtain the color gradient of the texture triangles; Global color adjustment is performed on the texture triangles corresponding to all first triangular mesh faces based on the color gradient; The Poisson fusion algorithm is used to adjust the local color of the adjacent regions of the texture triangles corresponding to two adjacent first triangular mesh faces.

10. The texture mapping method according to any one of claims 1-9, characterized in that, The step of applying texture mapping to the 3D point cloud mesh model using the second image sequence and the correspondence between the first triangular mesh facet and the texture triangles in the second image includes: Based on the second image sequence and the correspondence between the first triangular mesh and the texture triangles in the second image, all texture triangles are recombined into a preset number of texture images, and the correspondence between the first triangular mesh and the texture triangles in the texture images is determined. Based on the correspondence between the first triangular mesh facet and the texture triangle in the texture image, texture mapping is performed on the 3D point cloud mesh model.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the texture mapping method as described in any one of claims 1-10.

12. An intraoral scanning system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is executed by the processor, it implements the texture mapping method as described in any one of claims 1-10.

Images