Keyframe selection method, storage medium, and intraoral scanning system

Abstract

The application discloses a key frame selection method, a storage medium and an intraoral scanning system. The method comprises the following steps: acquiring a 2D image sequence, a 3D point cloud grid model and scanning parameters in three-dimensional scanning; performing down-sampling on the 3D point cloud grid model to obtain a 3D point cloud grid down-sampling model; calculating the gradient amplitude of all triangular mesh facets on the corresponding 2D image and the shooting angle of the corresponding 2D image according to the 2D image corresponding to all triangular mesh facets, the 3D point cloud grid down-sampling model and the scanning parameters; selecting the optimal 2D image according to the gradient amplitude and the shooting angle by using a Markov random field function, obtaining a target image sequence and the corresponding relationship between the triangular mesh facets and the target image; and counting the distribution of the number of triangular mesh facets to the target image, and selecting a key frame from all target images according to the distribution by using a 3sigma criterion. According to the method, the key frame with clearer overall texture and capable of covering the entire 3D point cloud grid model can be selected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a key frame selection method, a storage medium and an intraoral scanning system. BACKGROUND

[0002] Intraoral scanning generally needs to obtain a three-dimensional point cloud of the whole dental arch surface, and the scanning mode is a video type continuous scanning reconstruction. The input images participating in the scanning can be several hundred to several thousand, and these image sequences with a certain scale contain a large amount of redundant information. Most of the input images have no effect on the texture rendering of the final scanning point cloud. Moreover, the images collected by the handheld camera are inevitably blurred or out of focus due to excessive motion or shaking. Applying clear images for texture mapping will inevitably bring better visual effects. In order to more efficiently complete the texture mapping and reduce the waiting time of doctors and patients to obtain a better diagnosis and treatment experience, key frame selection needs to be performed on the collected image sequence.

[0003] There are mainly two kinds of related key frame selection techniques: one is to select the image key frame for texture mapping according to the time sequence characteristics of the input image sequence by reasonably setting the time interval; the other is to calculate the spatial uniqueness of the image, that is, when the 3D information corresponding to each pixel in the image does not coincide with the 3D information corresponding to each pixel in other images, the image has spatial uniqueness. By calculating this information, the images with less uniqueness are removed, the image sequence is updated, and until the images have enough unique information, the image key frame for subsequent texture rendering of the point cloud can be selected.

[0004] However, the method of selecting image key frames by setting a certain time interval is essentially a random sampling of images, which may cause the texture of a certain part of the 3D point cloud of the model surface to be missing, and may skip some clearer texture images; the method of calculating the spatial uniqueness of the image needs to calculate all the corresponding points of the 3D points and the 2D image points through the projection transformation, and the calculation of these corresponding points is very time-consuming. By reducing the image pixels, the processing speed can be improved, but the accuracy of the spatial uniqueness will be affected. If the accuracy is reduced, the texture of the selected image may not completely cover the 3D model surface obtained by scanning. Only considering the spatial uniqueness is not enough to measure some images with higher quality, which leads to a decrease in the quality of subsequent texture rendering.

[0005] To this end, for the key frame selection problem of texture mapping in the related art, a method is proposed to increase the field of view of scanning imaging, so that the texture information contained in a single image corresponds to a larger area of 3D point cloud, and considering the image blur and the interval on the image time sequence, the entire 3D scanning model surface can be covered with fewer texture images. This method can to some extent avoid the texture seams caused by the brightness inconsistency between different images, but the random sampling of images on the time sequence is likely to cause the image texture to not completely cover the 3D scanning model surface during texture mapping, resulting in a decrease in the texture rendering quality of the point cloud and affecting the doctor-patient diagnosis and treatment experience. SUMMARY

[0006] The present application aims to at least partially solve one of the technical problems in the related art. To this end, one purpose of the present application is to propose a key frame selection method, a storage medium and an intraoral scanning system to accurately and efficiently select 2D images for texture mapping of a 3D scanning model surface.

[0007] In a first aspect, the present application proposes a key frame selection method, comprising: obtaining a sequence of 2D images in three-dimensional scanning, a 3D point cloud grid model and scanning parameters; downsampling the 3D point cloud grid model to obtain a 3D point cloud grid downsampling model; calculating the gradient amplitude of all triangular mesh patches in the 3D point cloud grid downsampling model on the corresponding 2D image and the shooting angle of the corresponding 2D image according to the corresponding 2D image of all triangular mesh patches, the 3D point cloud grid downsampling model and the scanning parameters; using a Markov random field function to select the optimal 2D image according to the gradient amplitude of all triangular mesh patches on the corresponding 2D image and the shooting angle of the corresponding 2D image, to obtain a target image sequence and the corresponding relationship between the triangular mesh patches and the target images in the target image sequence; and counting the distribution of the number of triangular mesh patches with respect to the target images, and selecting key frames from all target images according to the distribution using the 3σ criterion.

[0008] In a second aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the key frame selection method described above.

[0009] In a third aspect, the present application proposes an intraoral scanning system, comprising a memory, a processor and a computer program stored on the memory, wherein the computer program is executed by the processor to implement the key frame selection method described above.

[0010] The key frame selection method and the medium and the device of the embodiment of the application first down-sample the 3D point cloud grid model, and then select the key frame in combination with Markov random field and 3σ criterion in statistics and probability theory, so that the overall texture is clearer and the key frame covering the surface of the entire 3D point cloud grid model can be accurately and efficiently selected, so as to facilitate subsequent texture mapping of the 3D point cloud grid model.

[0011] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a flow chart of the key frame selection method of the embodiment of the application;

[0013] Figure 2 is a 2D image sequence of a gypsum dental film of one example of the application;

[0014] Figure 3 is an image after key frame selection of a 2D image sequence of a gypsum dental film of one example of the application;

[0015] Figure 4 is a schematic diagram of a 3D point cloud grid model of a gypsum dental film of one example of the application;

[0016] Figure 5 is a 2D image sequence of a real tooth of one example of the application;

[0017] Figure 6 is an image after key frame selection of a 2D image sequence of a real tooth of one example of the application;

[0018] Figure 7 is a schematic diagram of a 3D point cloud grid model of a real tooth of one example of the application;

[0019] Figure 8 is a schematic diagram of a triangular mesh patch in a 3D point cloud grid model of a gypsum dental film of one example of the application;

[0020] Figure 9 is a schematic diagram of the distribution of 2D images in a target area of one example of the application. DETAILED DESCRIPTION

[0021] Embodiments of the application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the application, and cannot be understood as a limitation of the application.

[0022] The following description refers to the accompanying drawings, which are meant to be exemplary and not limiting.Figures 1-9 The key frame selection method and medium and device of the embodiments of the present application are described.

[0023] Figure 1 The flowchart of the key frame selection method of the embodiments of the present application.

[0024] As shown in Figure 1 , the key frame selection method comprises:

[0025] S101, obtaining a 2D image sequence, a 3D point cloud grid model and scanning parameters in three-dimensional scanning.

[0026] Specifically, the 2D rendering map, i.e., the 2D image sequence, can be obtained by three-dimensional scanning of teeth, plaster teeth, etc. by an intraoral scanning system. The sequence can be a time sequence and can contain thousands or even tens of thousands of 2D images. As shown in Figure 2 、 Figure 4 , only hundreds of 2D images are shown. As shown in Figure 8 , the 3D point cloud grid model contains vertex coordinate data of all triangular mesh patches. The scanning head of the intraoral scanning system can contain a camera and a projector, which are the data acquisition module and image processing module of three-dimensional scanning and jointly constitute the hardware and software part of the intraoral scanning system. Therefore, the scanning process of the intraoral scanning system can be realized by the joint action of the camera and the projector, and the scanning parameters can include camera parameters (including camera internal and external parameters), projector external parameters, etc. According to the scanning parameters, the rendering mapping relationship of the 3D point cloud grid model and the 2D images in the 2D image sequence, i.e., the corresponding relationship of the triangular mesh patches and the 2D images, can be obtained.

[0027] S102, down-sampling the 3D point cloud grid model to obtain a 3D point cloud grid down-sampling model.

[0028] Specifically, after obtaining the 3D point cloud grid model, the scanning model boundary of the 3D point cloud grid model can be identified; and then the 3D point cloud grid model is down-sampled according to the scanning model boundary. The down-sampling of the 3D point cloud grid model according to the scanning model boundary can include: down-sampling the grid vertices in the 3D point cloud grid model except the scanning model boundary, such as using the mean sampling method, the random sampling method, etc. to down-sample the grid vertices except the scanning model boundary after fixing the scanning model boundary to obtain the sampled grid vertices; and reconstructing the 3D point cloud grid model according to the sampled grid vertices and the grid vertices above the scanning model boundary to obtain the 3D point cloud grid down-sampling model.

[0029] Specifically, the scanning model boundary can be identified when the 3D point cloud grid model is obtained. Referring to Figure 3 、 Figure 5The scanning model boundary refers to a model boundary, which is shown by a boundary line. When downsampling is performed, only the grid vertices other than the scanning model boundary can be downsampled, and all the grid vertices on the scanning model boundary are retained. Thus, by performing fixed-boundary downsampling on the 3D point cloud grid model, a 3D point cloud grid downsampling model with reduced data quantity and retained original scanning model boundary is obtained, which facilitates subsequent faster selection of key frames from the 2D image sequence.

[0030] It should be noted that after the key frames are selected, the 3D model used for texture mapping is still the 3D point cloud grid model before downsampling, so as to ensure that the texture mapping does not lose the input 3D point cloud grid model.

[0031] S103, according to all the triangular mesh patches corresponding to the 2D images, the 3D point cloud grid downsampling model and the scanning parameters, the gradient amplitudes of all the triangular mesh patches on the corresponding 2D images and the shooting angles of the corresponding 2D images are calculated.

[0032] All the triangular mesh patches are the patches on the 3D point cloud grid downsampling model.

[0033] Specifically, before the gradient amplitudes and the shooting angles are calculated, the correspondence between all the triangular mesh patches in the 3D point cloud grid downsampling model and the 2D images in the 2D image sequence can be determined according to the 2D image sequence, the 3D point cloud grid model and the scanning parameters. The correspondence can be one-to-one, i.e., one triangular mesh patch corresponds to one 2D image, or one-to-many, i.e., one triangular mesh patch corresponds to multiple 2D images. That is to say, each 2D image in the 2D image sequence has a corresponding triangular mesh patch. Before the correspondence between all the triangular mesh patches in the 3D point cloud grid downsampling model and the 2D images in the 2D image sequence is determined, the correspondence between the three-dimensional space points in the 3D point cloud grid model and the two-dimensional pixel points on the corresponding 2D images can be calculated first, and the calculation process is as follows:

[0034] Taking a monocular camera as an example, Table 1 below shows the mathematical model parameters of the monocular camera:

[0035] Table 1

[0036]

[0037] In the above table, f x , f y are parameters related to the focal length of the camera, u0 and v0 are the coordinates of the principal point of the image, c is the coordinate axis non-perpendicular factor, which is usually approximately 0; k1, k2, p1 and p2 are the radial and tangential distortion coefficients of the camera lens; R and T are the transformation matrices of the world coordinate system and the camera coordinate system.

[0038] Let p 1j= [x w1j y w1j z w1j 1] T is the homogeneous world coordinate of point 1, and is set as the global world coordinate system on the virtual plane, p2 = [x w2j y w2j z w2j 1] T is the homogeneous world coordinate of point 2, q 1j = [u 1j v 1j 1] T , q 2j = [u 2j v 2j 1] T , according to the affine transformation of the camera, we have:

[0039]

[0040] Considering the influence of camera lens distortion, the relationship between the coordinate vector (x d ,y d ) T with distortion and the ideal coordinate vector (x,y) T is as follows:

[0041]

[0042] In the above formula, r 2 = x 2 +y 2 , the normal position coordinates are substituted into the above formula to obtain the distortion position coordinates, and thus the coordinate correspondence relationship between the three-dimensional space points of the relatively accurate 3D point cloud grid model and the corresponding two-dimensional pixel points on the 2D image can be obtained.

[0043] According to the coordinate correspondence relationship between the three-dimensional space points of the 3D point cloud grid model and the corresponding two-dimensional pixel points on the 2D image, and the relationship between the 3D point cloud grid model and the 3D point cloud grid downsampling model, the corresponding relationship between all triangular mesh patches in the 3D point cloud grid downsampling model and 2D images in the 2D image sequence can be obtained. Then, the gradient amplitudes of the three points of each triangular mesh patch in the 3D point cloud grid downsampling model on the corresponding 2D image can be calculated. Meanwhile, the patch normal of all triangular mesh patches in the 3D point cloud grid downsampling model can also be calculated, and the shooting angle, i.e., the included angle between the normal direction and the camera sight direction, can be obtained according to the patch normal and the scanning parameters. The calculation process of the patch normal can include: calculating the cross product of the vectors corresponding to any two edges of the triangular mesh patch, unitizing the cross product, and taking the direction upward (i.e., the direction away from the scanned object such as teeth).

[0044] It should be noted that since the 2D images collected may be blurred due to excessive camera motion or camera shake, the correspondence between the blurred images and the triangular mesh patches can be removed after obtaining the 2D image sequence.

[0045] As an example, before calculating the gradient magnitude and the shooting angle, the blurriness of each 2D image in the 2D image sequence can be calculated first, and then the correspondence between all triangular mesh patches and 2D images in the 2D image sequence is filtered according to the blurriness, and then the gradient magnitude of all triangular mesh patches in the 3D point cloud mesh downsampling model on the corresponding 2D images and the shooting angle of the corresponding 2D images are calculated according to the filtered correspondence, the 3D point cloud mesh downsampling model and the scanning parameters.

[0046] Specifically, the low-pass filter can be used to blur each 2D image in the 2D image sequence; the blurriness of each 2D image can be calculated according to each 2D image in the 2D image sequence and the corresponding blurred image; a preset blurriness standard can be obtained; the blurred image in the 2D image sequence can be obtained according to the blurriness and the preset blurriness standard; and the correspondence between the blurred image and the corresponding triangular mesh patch can be removed. Finally, the gradient magnitude of all triangular mesh patches in the 3D point cloud mesh downsampling model on the corresponding 2D images and the shooting angle of the corresponding 2D images are calculated in combination with the 3D point cloud mesh downsampling model and the scanning parameters.

[0047] Specifically, the blurriness of the image is an index for measuring the sharpness of the image. Generally, the edge of the blurred image is not obvious, and the edge of the clear image is relatively sharp. The 2D image is filtered by using the low-pass filter, that is, the 2D image is blurred, and the blurring processing can be to take the average value of the surrounding pixels in the 2D image. When blurring, the larger the kernel of the low-pass filter, that is, the larger the range of the surrounding, the more the pixel values of the neighboring pixels tend to be similar, and the stronger the blurring effect. Since the blurred image filtered by the low-pass filter is very close to the blurred image obtained by scanning and collecting, the blurriness can be calculated according to the 2D image in the original 2D image sequence and the corresponding blurred image, such as the difference between the pixel gray values of the original 2D image and the corresponding blurred image, and then the 2D image with repeated texture information in the 2D image sequence that is relatively blurred, that is, the blurred image, can be determined according to the calculated blurriness and the preset blurriness standard, and the correspondence between the blurred image and the corresponding triangular mesh patch can be removed.

[0048] Optionally, after the blurriness is calculated, the blurriness can be sorted according to the size, and the 2D image before sorting the preset number of values is determined as the blurred image, and the corresponding relationship is removed. The preset data can be determined according to the total number of the 2D image sequence and is positively correlated with the total number.

[0049] S104, performing optimal 2D image selection according to the gradient amplitude of all triangular mesh patches on the corresponding 2D images and the shooting angles of the corresponding 2D images by using the Markov random field function, to obtain a target image sequence and a corresponding relationship between the triangular mesh patches and the target images in the target image sequence.

[0050] It should be noted that the optimal 2D image here refers to the unique corresponding target image selected for the triangular mesh patch by using the Markov random field function, and the target image has higher texture quality and clarity compared to other 2D images corresponding to the corresponding triangular mesh patch.

[0051] The corresponding relationship between the triangular mesh patch and the target image in the target image sequence can be a corresponding relationship between the vertex coordinates of the triangular mesh patch and the pixel points on the target image.

[0052] Specifically, step S104 can include:

[0053] S1041, establishing a Markov random field function E l :

[0054]

[0055] wherein, not only considers the area and clarity of the triangular texture image, but also considers the shooting angle of the camera, aiming to select a 2D image with relatively high texture quality (or relatively high clarity) for each triangular mesh patch, denotes a triangular mesh patch F k in a 3D point cloud mesh downsampling model k , φ lk (F k , l k ) denotes the projection of the triangular mesh patch F k on the corresponding 2D image l k , and θ denotes the shooting angle of the 2D image l k , F k , F j denote two adjacent triangular mesh patches with a common edge in the 3D point cloud mesh downsampling model, l k , l j denote the corresponding 2D images of the triangular mesh patches F k , F j , edges denote a set of common edges of adjacent patches, and E smooth denotes a smoothing term, aiming to minimize the seam error between the selected texture images of the two adjacent triangular mesh patches, as shown in the following formula:

[0056]

[0057] That is, if two adjacent patches are from the same 2D image, the smooth term is set to 0, otherwise the smooth term is set to 1. By trying to select the texture from the same image, the texture seam and texture discretization can be avoided.

[0058] Alternatively, the smooth term can also be obtained by the following formula:

[0059]

[0060] where δ (k,j) denotes the ratio of the actual length of the common edge of two adjacent second triangles and the number of pixels on the corresponding 2D image, denotes the projection of p on the selected 2D image l k .

[0061] In S1042, the Markov random field function is solved to obtain the optimal 2D image corresponding to each triangular mesh patch in the 3D point cloud grid downsampling model, i.e., the target image.

[0062] Specifically, the problem of selecting the optimal 2D image for a triangular patch can be regarded as a texture optimal splicing process, and the above formula (1) can be used to solve this texture splicing problem by using a Markov random field-based method. The data term, i.e., E data (F k , l k ), which is one of the inputs of the Markov random field function, can be calculated in parallel using CUDA (Compute Unified Device Architecture). The reason is that the product of the gradient amplitude of each triangular mesh patch on the corresponding 2D image and the sine value of the shooting angle of the corresponding 2D image is independent of each other, which has parallelism, so it can be calculated in parallel using CUDA. At the same time, the binary function of the smooth term judges whether the corresponding 2D images of two adjacent triangular mesh patches are the same image, and calls the Markov random field-based function to calculate the view selection of the patch, and outputs the selected result, i.e., the unique optimal 2D image corresponding to all triangular mesh patches in the 3D point cloud grid downsampling model. That is, after the calculation based on the Markov random field, the triangular mesh patch and the selected optimal 2D image are one-to-one corresponding, i.e., one patch corresponds to one optimal 2D image; but before the calculation based on the Markov random field, one triangular mesh patch may correspond to multiple 2D images, or may correspond to one 2D image.

[0063] It should be noted that after removing the correspondence between the blurred image and the corresponding triangular mesh patch, the 3D point cloud mesh model may have triangular mesh patches corresponding to default 2D images, i.e. the texture of these triangular mesh patches is missing or not covered by the image texture, so that texture mapping cannot be completed, i.e. texture coverage cannot be achieved for all regions of the 3D point cloud mesh model. At this time, a default 2D image needs to be selected from the blurred image to ensure that all triangular mesh patches can be covered by a 2D image.

[0064] In one example, after removing the correspondence between the blurred image and the corresponding triangular mesh patch, the gradient amplitude and the shooting angle can be calculated before the filtered 2D image sequence, the 3D point cloud mesh down-sampling model and the scanning parameters are used to determine whether there are triangular mesh patches corresponding to default 2D images in the 3D point cloud mesh down-sampling model; if so, at least one default 2D image is selected from the blurred image to replace the default 2D image corresponding to the triangular mesh patch.

[0065] In this example, when the Markov random field function is used for calculation, all triangular mesh patches in the 3D point cloud mesh down-sampling model have corresponding 2D images, and thus the gradient amplitude and the shooting angle of all triangular mesh patches can be calculated for the Markov random field function calculation.

[0066] In order to facilitate the processing of 2D images, after obtaining the 2D image sequence, the 2D images in the 2D image sequence can be sequentially numbered to obtain a first image label value sequence; after the 2D image sequence is filtered, the first image label value sequence needs to be updated; after the default 2D image is selected from the blurred image, the first image label value sequence can be updated again to obtain a second image label value sequence. Thus, the correspondence between the triangular mesh patch and the 2D image can be determined by the image label value corresponding to the triangular mesh patch, i.e. the corresponding 2D image of the patch. The blurred image is also a map corresponding to the triangular mesh patch on the 3D point cloud mesh down-sampling model. If the correspondence between the blurred image and the corresponding triangular mesh patch is removed, the triangular mesh patch has no corresponding map, which means that the blurred image is all the corresponding maps of this part of the triangular mesh patch and cannot be completely discarded. Therefore, the image label value needs to be calculated and assigned to these triangular mesh patches again. That is, after the correspondence between the blurred image and the corresponding triangular mesh patch is removed, the triangular mesh patch corresponding to the blurred image is not assigned an image label value, i.e. the triangular mesh patch has a default default image label value. At this time, the image label value of the triangular mesh patch which is not covered by the image label value needs to be recalculated.

[0067] For example, there are 10 2D images in a 2D image sequence, which can be numbered 1, 2, 3, …, 9, 10 respectively as image label values in the first image label value sequence, and the 10 2D images can cover all triangular mesh patches in the 3D point cloud grid downsampling model. After the corresponding relationship between the blurred images and the corresponding triangular mesh patches is removed, the image label values assigned to the triangular mesh patches are 1, 3, 5, 6, 7, 8, 9, 10, so that there is a triangular mesh patch in the 3D point cloud grid downsampling model without a corresponding image label value, that is, a default 2D image corresponding to the triangular mesh patch needs to be selected to determine the texture of the triangular mesh patch. At this time, 1, 3, 5, 6, 7, 8, 9, 10 can be updated to 1, 2, 3, 4, 5, 6, 7, 8, and then the default image label value is used to replace the default 2D image, and then the 2D image corresponding to the triangular mesh patch of the default 2D image is selected from the blurred images. Suppose that the newly selected 2D image from the blurred images is 4, then the image label value sequence can be updated again to obtain the corresponding second image label value sequence: 1, 2, 3, 4, 5, 6, 7, 8, 9, and the original 4 is encoded as 9.

[0068] In this example, when the Markov random field function is calculated, there is a triangular mesh patch in the 3D point cloud grid downsampling model corresponding to a default 2D image, so when the Markov random field function is calculated, the gradient amplitude and the shooting angle corresponding to the triangular mesh patch corresponding to the default 2D image can be set to 0 for Markov random field function calculation.

[0069] In another example, after the corresponding relationship between the blurred images and the corresponding triangular mesh patches is removed, the optimal 2D image selection can be performed according to the gradient amplitude of all triangular mesh patches on the corresponding 2D image and the shooting angle of the corresponding 2D image using the Markov random field function to obtain a target image sequence and the corresponding relationship between the triangular mesh patches and the target images in the target image sequence, and then the default compensation is performed.

[0070] Specifically, according to the corresponding relationship between the triangular mesh patches and the target images in the target image sequence, it is determined whether there is a triangular mesh patch corresponding to a default 2D image.

[0071] If there is, at least one replacement default 2D image is selected from the blurred images to correspond to the triangular mesh patch, and is used as the target image corresponding to the triangular mesh patch corresponding to the default 2D image.

[0072] Or if there is, re-perform according to all the triangular mesh patches corresponding to the 2D image, 3D point cloud grid downsampling model and scanning parameters, calculate the gradient amplitude of all triangular mesh patches in the 3D point cloud grid downsampling model on the corresponding 2D image and the shooting angle of the corresponding 2D image, and use the Markov random field function to select the optimal 2D image according to the gradient amplitude of all triangular mesh patches on the corresponding 2D image and the shooting angle of the corresponding 2D image. Get the target image sequence and the corresponding relationship between the triangular mesh patch and the target image in the target image sequence.

[0073] From the fuzzy image, select the re-joined triangular mesh patch corresponding image sequence (i.e. the part corresponding to the triangular mesh patch in the 2D image sequence) to re-calculate the gradient amplitude of all triangular mesh patches in the 3D point cloud grid downsampling model on the corresponding 2D image and the shooting angle of the corresponding 2D image. The corresponding 2D image of the triangular mesh patch has changed as a whole, so through the calculation of this step, new gradient amplitude and shooting angle will be obtained. Then based on the obtained new gradient amplitude and shooting angle, the optimal 2D image selection is re-performed to obtain the target image sequence and the corresponding relationship between the new triangular mesh patch and the target image in the target image sequence.

[0074] S105, count the number of triangular mesh patches and the distribution of target images, and use the 3σ criterion to select key frames from all target images according to the distribution.

[0075] Specifically, step S105 can include:

[0076] S1051, count the number of triangular mesh patches corresponding to each target image, and calculate the standard deviation σ according to the number sequence;

[0077] S1052, the target image corresponding to the triangular mesh patch meeting the 3σ criterion is taken as the key frame.

[0078] Specifically, the Markov random field will calculate a unique image label value for each triangular mesh patch.

[0079] In an embodiment, the target region division of the 3D point cloud grid model can be performed based on the image label value of the target image, for example, the same corresponding triangular mesh patches can be divided into the same target region, and the number of the triangular mesh patches in the divided target region also needs to conform to the normal distribution. After the target region division is completed, the number of the triangular mesh patches in each target region is counted to form a sequence, and the standard deviation σ of the sequence formed by the number of the patches is calculated. It can be determined that the image label value corresponding to the number of the patches located between [-3σ, 3σ] is the selected label value, and the target image corresponding to these label values is the key frame. In this way, the optimization of the selected 2D image can be realized, and the number of the selected 2D image is further reduced.

[0080] It should be noted that, in order to ensure the full coverage of the triangular mesh patches, the image label values corresponding to the number of the patches outside [-3σ, 3σ] are discarded first, and then the image label values corresponding to the number of the patches distributed outside [-3σ, 3σ] are set to be the same as the image label values distributed within [-3σ, 3σ]. For example, as shown in Figure 9 , the target region 1 with the image label value of 014 is surrounded by the target region with the image label value of 003, and the target region 2 with the image label value of 014 is surrounded by the target region with the image label value of 004. At this time, the image label value of the target region 1 can be reset to the image label value of 003, and the image label value of the target region 2 can be reset to the image label value of 004.

[0081] Optionally, in order to make the subsequent texture mapping clearer, after the key frame selection is completed, the selected key frame can be subjected to brightness preprocessing, so that the texture image has better brightness consistency, so as to reduce the redundant information, motion blur or image defocus, avoid texture joints, and improve the efficiency of texture mapping and the quality of generated texture.

[0082] As shown in Figure 2 , Figure 5 , the gypsum dental model and the real tooth true color image collected as the source of texture mapping in the 3D scanning reconstruction process are shown, Figure 3 , Figure 6 The 2D images selected after the key frame selection method of the application is screened are shown, the finally selected 2D images have good quality and can completely cover the Figure 4 , 7 The surface of the 3D point cloud grid model shown can be used as the texture source for subsequent point cloud rendering.

[0083] In addition, it should be noted that the Markov random field can represent the local interaction between the gray value of each pixel in the image and its adjacent pixels. Specifically, the Markov random field reflects the interaction of the pixel set in the gray level through the application of some local function relationship in the point set, and the global function mapping relationship can be described by probability theory. The Markov random field can reflect the randomness and potential structure of the image, thereby effectively describing the properties of the image.

[0084] The present application needs to project the 3D point cloud grid model after downsampling to the corresponding 2D image to calculate the clarity evaluation index of the image texture corresponding to the triangular mesh patch. This part of the calculation can use CUDA in the GPU (Graphics Processing Unit, graphics processor) for parallel calculation. A multi-thread control module is designed in the GPU. By setting the CUDA kernel function, setting a reasonable number of thread blocks and threads, and then automatically allocating threads for calculation. Each thread is designed to process one triangular mesh patch. The quality evaluation index of hundreds of thousands of triangular patches is calculated in parallel, and then the high-speed switching between threads is realized to cope with large-scale parallel calculation and hidden latency, achieving the purpose of concurrent throughput of hundreds of millions of data, and more efficiently completing the calculation of the quality comprehensive evaluation index of all patches on the key frame.

[0085] The specific setting steps of the CUDA kernel function are as follows: first, set the number of two-dimensional threads in the thread block according to the size of the patch data and the configuration of the graphics card, and then set the number of two-dimensional thread blocks in the thread grid according to the number of triangular patches, the number of images and the number of threads in the thread block.

[0086] In summary, the key frame selection method can first downsample the 3D point cloud grid model obtained by scanning and deblur the 2D image sequence, then use CUDA to complete the texture clarity evaluation index evaluation of the 2D image corresponding to all triangular mesh patches in the downsampled 3D point cloud grid model. Based on the evaluation result, the image quality evaluation result can be obtained by using the Markov random field, that is, the optimal 2D image uniquely corresponding to each triangular mesh patch is obtained, and finally the optimal 2D image is further filtered according to the 3σ criterion to obtain the key frame for subsequent texture mapping. Therefore, under the premise of ensuring the quality of the generated texture, as few images as possible are used to cover the model surface, so as to reduce the amount of calculation and improve the efficiency of the program. At the same time, in order to make the subsequent texture mapping clearer, the image can be preprocessed after the key frame selection is completed, so that the input texture image has better brightness consistency, so as to reduce redundant information, motion blur or image defocus, avoid texture seams, and improve the efficiency and quality of texture mapping.

[0087] The application further provides a computer readable storage medium.

[0088] In this embodiment, the computer storage medium stores a computer program, and the computer program is executed by the processor to implement the key frame selection method.

[0089] The application further provides an electronic device.

[0090] In this embodiment, the electronic device includes a memory, a processor and a computer program stored in the memory, and the computer program is executed by the processor to implement the key frame selection method.

[0091] The electronic device can be an intraoral scanning system.

[0092] In one embodiment, the intraoral scanning system can include a handheld image acquisition device and an image processing device, the handheld image acquisition device is used to scan and acquire intraoral image data, and the image processing device is used to process the intraoral image data.

[0093] The memory, the processor and the computer program stored in the memory can be distributed in the handheld image acquisition device and / or the image processing device. The handheld image acquisition device and the image processing device can be integrated or separated.

[0094] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logical functions and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, processor- based or other system that can fetch the instructions from a instruction execution system, apparatus or device and execute the instructions, or in conjunction with such an instruction execution system, apparatus or device. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device or in conjunction with such an instruction execution system, apparatus or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber (optical apparatus), and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic conversion of the optical scanning into a form that can be edited, interpreted or otherwise processed as appropriate, and then stored in a computer memory if necessary.

[0095] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, in part, or in whole, in software, or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the techniques described herein can be implemented with or without the use of any one or more of the following technologies: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals; an application specific integrated circuit having appropriate combinational logic gates; a programmable gate array (PGA), such as a field programmable gate array (FPGA), having a suitable configuration; a computer having a suitable architecture for executing software or firmware that controls the operation of such computer; or a hybrid of the above which can include one or more of the above technologies.

[0096] In the description of the present application, the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The appearances of the above terms in various places in the specification are not necessarily intended to refer to the same embodiment or example. Furthermore, describing features, structures, materials, or characteristics as being in "some embodiments" or "one embodiment" is intended to indicate that the features, structures, materials, or characteristics are included in at least one embodiment of the application. Thus, the appearances of the above phrases in various places in the specification are not necessarily intended to refer to the same embodiment or example.

[0097] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", and the like are intended to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are merely for the convenience of describing the present application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0098] In addition, the terms "first", "second", "third", etc. are used only for descriptive purposes and are not to be construed as indicating or implying relative importance or a specific number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can include at least one of the features, explicitly or implicitly. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0099] In the present application, unless specifically defined otherwise, the terms "mounting", "connected", "connecting", "fixed", and the like should be construed broadly, for example, can be fixed connection, can also be detachable connection, or integral; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the internal communication of two elements or the interaction relationship of two elements, unless specifically defined otherwise. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0100] In the present application, unless specifically defined otherwise, the first feature is "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "above", "over" and "on" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "under" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0101] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A keyframe selection method, characterized by, The method comprises: obtaining a 2D image sequence, a 3D point cloud grid model and scanning parameters in a three-dimensional scan; down-sampling the 3D point cloud grid model to obtain a 3D point cloud grid down-sampled model; calculating gradient amplitudes of all triangular mesh patches on corresponding 2D images and shooting angles of the corresponding 2D images according to the 2D images corresponding to all the triangular mesh patches, the 3D point cloud grid down-sampled model and the scanning parameters, wherein all the triangular mesh patches are mesh patches on the 3D point cloud grid down-sampled model; performing optimal 2D image selection according to the gradient amplitudes of all the triangular mesh patches on the corresponding 2D images and the shooting angles of the corresponding 2D images by using a Markov random field function to obtain a target image sequence and a corresponding relationship between the triangular mesh patches and target images in the target image sequence; statistically analyzing a distribution of the number of the triangular mesh patches with respect to the target images, and selecting key frames from all the target images according to the distribution by using a 3σ criterion.

2. The keyframe selection method of claim 1, wherein, The down-sampling of the 3D point cloud grid model comprises: identifying a scanning model boundary of the 3D point cloud grid model; down-sampling grid vertices of the 3D point cloud grid model except the scanning model boundary to obtain sampled grid vertices; reconstructing the 3D point cloud grid model according to the sampled grid vertices and grid vertices above the scanning model boundary.

3. The keyframe selection method of claim 1, wherein, Before the gradient amplitudes and the shooting angles are calculated, the method further comprises: calculating blurriness of each 2D image in the 2D image sequence; filtering the corresponding relationship between all the triangular mesh patches and 2D images in the 2D image sequence according to the blurriness.

4. The keyframe selection method of claim 3, wherein, The calculation of the blurriness of each 2D image in the 2D image sequence comprises: performing blurring processing on each 2D image in the 2D image sequence by using a low-pass filter; calculating the blurriness of each 2D image according to each 2D image in the 2D image sequence and a corresponding blurred image.

5. The keyframe selection method of claim 4, wherein, The filtering of the corresponding relationship between all the triangular mesh patches and 2D images in the 2D image sequence according to the blurriness comprises: obtaining a preset blurriness standard; obtaining a blurred image in the 2D image sequence according to the blurriness and the preset blurriness standard; removing the corresponding relationship between the blurred image and the corresponding triangular mesh patch.

6. The keyframe selection method of claim 5, wherein, Before the gradient amplitudes and the shooting angles are calculated, the method further comprises: judging whether there are triangular mesh patches corresponding to a default 2D image in the 3D point cloud grid down-sampled model; if there are, selecting at least one blurred image to replace the default 2D image to correspond to the triangular mesh patch.

7. The keyframe selection method of claim 5, wherein, After the optimal 2D image selection according to the gradient amplitudes of all the triangular mesh patches on the corresponding 2D images and the shooting angles of the corresponding 2D images by using the Markov random field function is performed, the method further comprises: determining whether there is a triangle mesh patch corresponding to a default 2D image in the 3D point cloud mesh downsampling model according to the correspondence between the triangle mesh patch and the target image in the target image sequence; if there is, selecting one from the blurred images to replace the default 2D image to correspond to the triangle mesh patch, and taking the target image corresponding to the triangle mesh patch as the target image corresponding to the triangle mesh patch of the default 2D image.

8. The keyframe selection method of claim 5, wherein, After the optimal 2D image selection is performed by using the Markov random field function according to the gradient amplitude of all triangle mesh patches on the corresponding 2D image and the shooting angle of the corresponding 2D image, the target image sequence is obtained, and the correspondence between the triangle mesh patch and the target image in the target image sequence, the method further comprises: determining whether there is a triangle mesh patch corresponding to a default 2D image in the 3D point cloud mesh downsampling model according to the correspondence between the triangle mesh patch and the target image in the target image sequence; if there is, selecting at least one from the blurred images to replace the default 2D image to correspond to the triangle mesh patch; calculating the gradient amplitude of all triangle mesh patches on the corresponding 2D image and the shooting angle of the corresponding 2D image in the 3D point cloud mesh downsampling model according to all 2D images corresponding to the triangle mesh patch, the 3D point cloud mesh downsampling model and the scanning parameter; performing optimal 2D image selection by using the Markov random field function according to the gradient amplitude of all triangle mesh patches on the corresponding 2D image and the shooting angle of the corresponding 2D image, obtaining the target image sequence, and the correspondence between the triangle mesh patch and the target image in the target image sequence.

9. The keyframe selection method of claim 1, wherein, The statistics of the number of the triangle mesh patches for the distribution of the target image, and the selection of the key frame from all target images according to the distribution by using the 3σ criterion, comprises: statistics of the number of the triangle mesh patches corresponding to each target image, and calculation of the standard deviation σ according to the number series composed of the number; taking the target image corresponding to the triangle mesh patch satisfying the 3σ criterion as the key frame.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the key frame selection method according to any one of claims 1-9.

11. An intraoral scanning system comprising a memory, a processor, and a computer program stored on the memory, wherein, The computer program is executed by the processor to realize the key frame selection method according to any one of claims 1-9.

Images