Oral cavity image processing method and system based on multi-modal enhancement

Abstract

The invention discloses an oral cavity image processing method and system based on multi-mode enhancement. The method comprises the following steps: acquiring oral cavity images of multiple modes; performing multi-scale down-sampling and matching processing on each oral cavity image, and performing multiple edge detection to obtain the feature intensity and local structure density of each feature image; according to a preset multi-modal fusion rule and the feature intensity and the local structure density of each feature image, the fusion confidence coefficient of each feature image of each modal is determined, the feature images of each modal are superposed and fused, and a multi-modal enhanced oral cavity fusion image is obtained; calculating a saliency value of each pixel point in the oral fusion image and performing saliency adjustment; and filtering the adjusted pixel points, and constructing an oral cavity three-dimensional model by using the oral cavity fusion image. According to the invention, multi-modal oral cavity image fusion can be realized, and pixel points with rich textures are reserved, so that the precision of the oral cavity fusion image is improved, and the precision and trueness of an oral cavity three-dimensional model are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for processing oral images based on multimodal enhancement. Background Technology

[0002] The digital dental impression instrument collects morphological data of teeth, gums and other tissues in the oral cavity through optical scanning, laser ranging and other means. It converts traditional plaster models, silicone rubber impressions or intraoral dentition morphology into three-dimensional visual digital models in a computer. The digital dental impression instrument has high accuracy and operability and is an important tool in modern prosthodontics and implantology.

[0003] However, current digital impression instruments for dentistry process oral images by reconstructing 3D information of the oral cavity from 2D images. Due to the low contrast of 2D images, tooth edges (such as interproximal gaps and occlusal fissures) are easily blurred, resulting in large errors in 3D reconstruction. Moreover, the current method only relies on grayscale images for acquisition and analysis, and cannot fuse (e.g., gingivitis) or depth data (e.g., laser point clouds), resulting in low accuracy of the acquired and processed image data. Consequently, the accuracy of the constructed 3D oral model is poor and cannot meet the needs of complex cases, such as implant restoration and early caries diagnosis. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for oral image processing based on multimodal enhancement, which can realize multimodal oral image fusion and retain pixels with rich textures, so as to improve the accuracy of oral fusion images, thereby improving the accuracy and realism of oral 3D models.

[0005] To address the aforementioned technical problems, this invention provides a multimodal enhancement-based oral cavity image processing method, comprising: acquiring oral cavity images of multiple modalities, wherein the oral cavity images include visible light images, near-infrared images, and laser depth images from different angles, and performing image preprocessing on the visible light images; performing multi-scale downsampling and matching processing on the visible light images, near-infrared images, and laser depth images respectively, and performing multiple edge detection on the feature images of each modality at different scales after matching to obtain the feature intensity and local structure density of each feature image; determining the fusion confidence of each feature image of each modality through a preset multimodal fusion rule and the feature intensity and local structure density of each feature image, and superimposing and fusing the feature images of each modality according to the fusion confidence of all feature images to obtain a multimodal enhanced oral cavity fusion image; calculating the saliency value of each pixel in the oral cavity fusion image, and adjusting the saliency value of each pixel according to a saliency value adjustment rule; filtering the saliency values ​​of all adjusted pixels to retain the required pixels in the oral cavity fusion image, and constructing a three-dimensional oral cavity model based on the optimized oral cavity fusion image.

[0006] As an improvement to the above scheme, the step of performing multi-scale downsampling and matching processing on the visible light image, near-infrared image, and laser depth image includes: performing multi-scale downsampling processing on the matched and aligned visible light image, near-infrared image, and laser depth image according to multiple preset scale factors to obtain scale images at different scales for each modality. The multiple scale factors include scale factors of 3, 5, 7, and 9, as well as an ultra-micro scale factor, where the ultra-micro scale factor includes scale factors of 1.2 and 1.5; performing denoising processing on each scale image using a GAN-based semantic denoising algorithm; performing grayscale transformation processing on each scale image using a grayscale level division method to generate two grayscale transformed images for each scale image as feature images; extracting feature points from the feature images and performing spatial registration processing on the extracted matching feature points to match and align feature images of different angles or modalities at the pixel level.

[0007] As an improvement to the above scheme, the step of performing multiple edge detection on the feature images of each modality at different scales after matching to obtain the feature intensity and local structure density of each feature image includes: performing Canny edge detection on the feature images at different scales to obtain the feature intensity and local structure density of each feature image; wherein, when calculating the gradient magnitude and direction of the feature image, a preset arctangent function value table is directly called to replace floating-point operations with table lookup.

[0008] As an improvement to the above scheme, the step of determining the fusion confidence of each feature image of each modality through preset multimodal fusion rules and the feature intensity and local structure density of each feature image includes: calculating the multimodal fusion confidence of each feature image to obtain the fusion confidence of each feature image, and the calculation formula is as follows: Among them, Q i f is the fusion confidence of the i-th feature image. i L represents the feature intensity of the i-th feature image. i Let be the local structure density of the i-th feature image, where i ranges from [1, M] and M is the total number of feature images.

[0009] As an improvement to the above scheme, the step of superimposing and fusing the feature images of each modality based on the fusion confidence of all feature images to obtain a multimodal enhanced oral cavity fusion image includes: performing superimposing and fusion calculations on the feature images of each modality to obtain a multimodal enhanced oral cavity fusion image, the calculation formula of which is as follows: Where F represents the multimodal enhanced oral cavity fusion image, Q i Let T be the fusion confidence of the i-th feature image. i Let i be the i-th feature image, where i ranges from [1, M] and M is the total number of feature images.

[0010] As an improvement to the above scheme, the step of calculating the saliency value of each pixel in the oral cavity fusion image includes: the formula for calculating the saliency value of each pixel is: Where S(x,y) represents the saliency value of the pixel at coordinate (x,y), and F(x,y) represents the feature data of the pixel at coordinate (x,y). This represents the grayscale gradient of the pixel at coordinates (x, y). Let F_N represent the gradient in the color space, and let σ represent the average eigenvector of a predefined local neighborhood. 2 (x,y) represents the gray-level variance of the preset local neighborhood of the pixel with coordinates (x,y), where w1, w2, and w3 are all adjustable weight coefficients, and w1+w2+w3=1.

[0011] As an improvement to the above scheme, the step of adjusting the saliency value of each pixel according to the saliency value adjustment rule includes: for each pixel, selecting a preset local neighborhood centered on it, and calculating the local neighborhood variance, the calculation formula is as follows: σ 2 (x,y)=(1 / |N|)×∑_{(i,j)∈N}[I(i,j)-μ_N] 2 ; where σ 2(x, y) represents the local neighborhood variance of the pixel point with coordinates (x, y), N represents a preset local neighborhood centered at the coordinates (x, y) and is a set, |N| represents the size of the preset local neighborhood set N, which includes all pixel points within the preset local neighborhood, I(i, j) represents the pixel intensity or gray value of the pixel point with coordinates (x, y), and μ_N is the gray mean value of the preset local neighborhood;

[0012] Construct an inhibition factor according to the local neighborhood variance, and its construction formula is as follows: ψ(x, y) = σ 2 (x, y) / [σ 2 (x, y) + ε]; where, the ψ(x, y) represents the inhibition factor of the pixel point with coordinates (x, y), and ε is a constant;

[0013] The adjustment formula for adjusting the significance value of each pixel point is:

[0014] S'(x, y) = S(x, y) × ψ(x, y); where, S'(x, y) represents the adjusted significance value of the pixel point with coordinates (x, y), and S(x, y) represents the original significance value of the pixel point with coordinates (x, y).

[0015] As an improvement to the above solution, the step of filtering the significance values of all the pixel points after adjustment includes: arranging all pixel points in ascending order according to the size of their adjusted significance values to obtain the first quartile Q1 and the third quartile Q3; filtering all the pixel points, when S'(x, y) > Q3 + (1.5·IQR) or S'(x, y) < Q1 - (1.5·IQR), then filter the abnormal pixel point with the current coordinates (x, y), where, S'(x, y) represents the adjusted significance value of the pixel point with coordinates (x, y), and IQR = Q3 - Q1.

[0016] As an improvement to the above solution, the step of performing gray-level transformation processing on each scale image by using the gray-level division method to generate two gray-level transformation images for each scale image includes: performing equidistant division and central offset division on each scale image respectively, so that each scale image is transformed into two gray-level transformation images.

[0017] This invention provides a multimodal enhancement-based oral cavity image processing system, comprising: an acquisition module for acquiring oral cavity images of multiple modalities, wherein the oral cavity images include visible light images, near-infrared images, and laser depth images at different angles, and performing image preprocessing on the visible light images; a multi-scale image processing module for performing multi-scale downsampling and matching processing on the visible light images, near-infrared images, and laser depth images respectively, and performing multiple edge detection on the feature images of each modality at different scales after matching to obtain the feature intensity and local structure density of each feature image; and a multimodal fusion module for using a preset multimodal fusion rule. The system uses the feature intensity and local structure density of each feature image to determine the fusion confidence of each feature image in each modality. Based on the fusion confidence of all feature images, the feature images of each modality are superimposed and fused to obtain a multimodal enhanced oral cavity fusion image. A saliency processing module calculates the saliency value of each pixel in the oral cavity fusion image and adjusts the saliency value of each pixel according to a saliency value adjustment rule. An optimization module filters the adjusted saliency values ​​of all pixels to retain the required pixels in the oral cavity fusion image and constructs a three-dimensional oral cavity model based on the optimized oral cavity fusion image.

[0018] Implementing this invention has the following beneficial effects:

[0019] This invention extracts macroscopic to ultra-microscopic features from multimodal oral images at multiple scales, achieving full-range feature extraction from macroscopic to ultra-microscopic features, avoiding missed detection of ultra-microscopic structures, improving detection accuracy, and employing multiple edge detection to enhance the comprehensiveness of feature capture. By superimposing and fusing multimodal oral images, noisy edges can be effectively suppressed and real edges can be enhanced and preserved, improving edge localization accuracy, thereby improving the recognition accuracy of oral images. Furthermore, by adjusting the saliency value and filtering the fused oral image, abrupt bright spots, reflection errors, and isolated abnormal pixels can be removed while retaining pixels with rich textures, thus outputting a high-precision fused oral image to construct a three-dimensional oral model, improving the accuracy and realism of the three-dimensional oral model. Attached Figure Description

[0020] Figure 1 This is a flowchart of the oral image processing method based on multimodal enhancement according to the present invention;

[0021] Figure 2 This is a schematic diagram of the oral image processing system based on multimodal enhancement according to the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It is hereby declared that the directional terms such as up, down, left, right, front, back, inside, and outside used in this text are based solely on the accompanying drawings and are not intended to specifically limit the invention.

[0023] like Figure 1 As shown in the figure, a specific embodiment of the present invention provides a method for oral cavity image processing based on multimodal enhancement, including:

[0024] S101. Acquire oral cavity images of multiple modalities, wherein the oral cavity images include visible light images, near-infrared images, and laser depth images from different angles, and perform image preprocessing on the visible light images;

[0025] It should be noted that for visible light images (i.e., RGB images), grayscale transformation is performed using the formula Y = 0.299R + 0.587G + 0.114B to convert the visible light image into a grayscale image, in order to eliminate color interference, highlight brightness changes, and display the essential features of the edges.

[0026] S102. Perform multi-scale downsampling and matching processing on the visible light image, near-infrared image and laser depth image respectively, and perform multiple edge detection on the feature images of each modality at different scales after matching to obtain the feature intensity and local structure density of each feature image.

[0027] Specifically, the steps of performing multi-scale downsampling and matching processing on the visible light image, near-infrared image, and laser depth image respectively include:

[0028] Step 1: Perform multi-scale downsampling processing on the matched and aligned visible light image, near-infrared image, and laser depth image according to multiple preset scale factors to obtain scale images of each modality at different scales. The multiple scale factors include scale factors of 3, 5, 7, and 9, as well as an ultra-micro scale factor.

[0029] It should be noted that setting ultrascale factors can effectively identify the ultrastructures of the oral cavity, improving the accuracy and scope of feature extraction. These ultrascale factors include, but are not limited to, scale factors of 1.2 and 1.5. Multiple scale factors can form a six-level scale pyramid (1.2, 1.5, 3, 5, 7, 9), covering the full range of features from 0.1mm micro-intervals to macroscopic occlusal surfaces, solving the problem of missed detection of ultrastructures (such as implant margins and early caries) in existing technologies and improving detection accuracy.

[0030] Step 2: Denoise the image at each scale using a GAN-based semantic denoising algorithm;

[0031] It should be noted that by employing the semantic denoising algorithm of GAN, noise interference in images at each scale can be eliminated and structural edge details can be enhanced, thereby improving image quality and readability.

[0032] Step 3: Perform grayscale transformation on each scale image using a grayscale level division method, so that each scale image generates two grayscale transformed images and serves as feature images;

[0033] It should be noted that each scale image is divided into equal intervals and center offsets, so that each scale image is transformed into two grayscale images, thereby achieving diversified feature extraction.

[0034] Step 4: Extract feature points from the feature image and perform spatial registration on the extracted matching feature points to match and align feature images of different angles or modalities at the pixel level.

[0035] It should be noted that algorithms such as SURF, SIFT, or ORB can be used to extract feature points for each feature image and compare the feature points between images to perform spatial location matching. By calculating the affine / perspective matrix between images, spatial registration is performed on data from different viewpoints and modalities to ensure that data from different angles or modalities can be accurately aligned, laying a spatial foundation for subsequent multimodal image fusion.

[0036] Furthermore, by performing Canny edge detection on the feature images at different scales after the above processing, the feature intensity and local structure density of each feature image can be obtained, providing a fusion basis for subsequent multimodal image fusion; moreover, by adopting dual edge detection, the comprehensiveness of feature capture can be improved, thereby improving the accuracy of image data recognition.

[0037] In dual edge detection, when calculating the gradient magnitude and direction of the feature image, a pre-defined arctangent function value table is directly invoked to replace floating-point operations, thereby improving data processing efficiency. For example, a pre-calculated arctangent function value table for 2048×2048 pixels (precision 0.001) can reduce floating-point operations by 90% by using an index instead of real-time calculation. Simultaneously, the weight coefficients are magnified by 1024 times and stored as fixed-point integers (e.g., 0.25 is stored as 256), and then divided by 1024 during calculation, which can increase the calculation speed by 6 times.

[0038] Secondly, image data of different modalities and scales can be computed in parallel using GPUs. Multiple parallel threads independently calculate the gradient magnitude of corresponding pixel regions, eliminating the "scale waiting" bottleneck of serial processing. This increases computational efficiency to 85%, far exceeding the 20% efficiency of existing technologies, thus significantly improving image data processing efficiency. Simultaneously, preprocessing templates for implant restoration, orthodontics, and pediatric dentistry can be developed for different application scenarios. By one-click access to pre-adjusted edge detection weights (e.g., bone edge weight +30% for implant scenarios) and noise thresholds (β = 0.01 for pediatric scenarios), intelligent and rapid adaptation between "scenario and algorithm" is achieved. This reduces doctors' manual parameter tuning time by 90% and improves the detection efficiency of key features (such as bracket edges) by 3 times.

[0039] In addition, the above-mentioned method based on the present invention can improve the detection accuracy of ultra-micro features by 66.7% and reduce the edge error to ±0.1mm: the edge positioning error of ultra-micro structures such as 0.1mm micro gaps and 0.5mm pits and fissures is reduced from ±15 pixels (about 0.3mm) in the prior art to within ±5 pixels (<0.1mm), and the edge clarity of the final constructed oral cavity three-dimensional model can be significantly improved by about 3 times.

[0040] S103. By using the preset multimodal fusion rules and the feature intensity and local structure density of each feature image, the fusion confidence of each feature image of each modality is determined, and the feature images of each modality are superimposed and fused according to the fusion confidence of all feature images to obtain the multimodal enhanced oral cavity fusion image.

[0041] Specifically, as an improvement to the above scheme, the step of determining the fusion confidence of each feature image of each modality through preset multimodal fusion rules and the feature intensity and local structure density of each feature image includes: calculating the multimodal fusion confidence of each feature image to obtain the fusion confidence of each feature image, and the calculation formula is as follows:

[0042]

[0043] Among them, Q i f is the fusion confidence of the i-th feature image. i L represents the feature intensity of the i-th feature image. i Let be the local structure density of the i-th feature image, where i ranges from [1, M] and M is the total number of feature images.

[0044] It should be noted that the molecule f i* L i The confidence score of the i-th feature image is also used as a contribution to the validity of the final stitching result; the denominator is... To perform a weighted sum of all feature images involved in the fusion, representing the sum of the total information content or confidence level, this function is used to standardize the confidence level of each part to the [0,1] interval, achieving normalization and improving data processing efficiency. i This represents the fusion confidence (i.e., fusion contribution) of the i-th feature image in the weighted sum of all feature images involved in the fusion. This allows for subsequent fusion of each feature image based on its fusion confidence, thereby improving the accuracy of the fused data.

[0045] The step of superimposing and fusing the feature images of each modality based on the fusion confidence of all feature images to obtain a multimodal enhanced oral cavity fusion image includes: performing superimposing and fusion calculations on the feature images of each modality to obtain a multimodal enhanced oral cavity fusion image, the calculation formula being as follows:

[0046]

[0047] Where F represents the multimodal enhanced oral cavity fusion image, Q i Let T be the fusion confidence of the i-th feature image. i Let i be the i-th feature image, where i ranges from [1, M] and M is the total number of feature images.

[0048] It should be noted that, through the multimodal fusion formula described above, the corresponding feature images can be weighted and fused according to the fusion confidence of each feature image, thereby fusing the feature images of multiple modalities together to generate a unified, detailed, clear edge, and low-error high-quality oral fusion image. This can effectively suppress noisy edges and enhance the preservation of real edges, and improve edge positioning accuracy, thereby improving the recognition accuracy of oral images. In this way, it can solve the problem of missed detection of internal tooth lesions (such as early caries demineralization) by single modality. By simultaneously acquiring tooth surface texture and internal structure information, the diagnostic accuracy can be improved by more than 20%.

[0049] S104. Calculate the saliency value of each pixel in the oral cavity fusion image, and adjust the saliency value of each pixel according to the saliency value adjustment rule;

[0050] Specifically, the step of calculating the saliency value of each pixel in the oral cavity fusion image includes:

[0051] The formula for calculating the saliency value of each pixel is:

[0052]

[0053] Where S(x,y) represents the saliency value of the pixel at coordinate (x,y), and F(x,y) represents the feature data of the pixel at coordinate (x,y). This represents the grayscale gradient of the pixel at coordinates (x, y). Let F_N represent the gradient in the color space, and let σ represent the average eigenvector of a predefined local neighborhood. 2 (x,y) represents the gray-level variance of the preset local neighborhood of the pixel with coordinates (x,y), where w1, w2, and w3 are all adjustable weight coefficients, and w1+w2+w3=1.

[0054] It should be noted that the above method, through multimodal fusion of the image, can accurately calculate the saliency score by combining the fusion gradient, texture contrast and color changes with local statistical models and structural difference measures, thereby improving the fidelity of edge and texture structure.

[0055] Further, the step of adjusting the saliency value of each pixel according to the saliency value adjustment rule includes:

[0056] Step 1: For each pixel, select a preset local neighborhood centered on it, and calculate the local neighborhood variance using the following formula:

[0057] σ 2 (x,y)=(1 / |N|)×∑_{(i,j)∈N}[I(i,j)-μ_N] 2 ;

[0058] Where, σ 2 (x,y) represents the local neighborhood variance of the pixel at coordinates (x,y), N represents the preset local neighborhood centered at coordinates (x,y) and is a set, |N| represents the size of the preset local neighborhood set N, which includes all pixels in the preset local neighborhood, I(i,j) represents the pixel intensity or gray value of the pixel at coordinates (x,y), and μ_N is the gray mean of the preset local neighborhood;

[0059] It should be noted that in this embodiment, a fixed-size window (such as 5×5 or 7×7) is constructed with each pixel (x,y) as the center and used as a preset local neighborhood N, so as to calculate the gray-level variance of the preset local neighborhood N of each pixel (x,y).

[0060] Step 2: Construct the suppression factor based on the local neighborhood variance. The construction formula is as follows:

[0061] ψ(x,y)=σ 2 (x,y) / [σ 2 [(x,y)+ε];

[0062] It should be noted that the ψ(x,y) represents the suppression factor of the pixel point with coordinates (x,y), where ε is a constant. To prevent division by zero, ε is preferably 0.01 - 1, but not limited thereto.

[0063] Step 3: The adjustment formula for adjusting the significance value of each pixel point is:

[0064] S'(x,y) = S(x,y) × ψ(x,y);

[0065] Where, S'(x,y) represents the adjusted significance value of the pixel point with coordinates (x,y), and S(x,y) represents the original significance value of the pixel point with coordinates (x,y).

[0066] It should be noted that if the texture of the S'(x,y) pixel region is rich, that is, the value of σ2(x,y) is high, the corresponding ψ(x,y) ≈ 1, and the S'(x,y) pixel region shows that the significance value is retained; if the S'(x,y) pixel region is smooth or flat, that is, the value of σ2(x,y) is low, the corresponding ψ(x,y) ≈ 0 - 0.2, and the S'(x,y) pixel region shows that the significance value is suppressed. Thus, by multiplying the original significance score by the suppression factor ψ(x,y) constructed by the local variance, the significance value of the smooth or flat region (such as the reflective tooth surface) is reduced, realizing the automatic weakening process of the smooth or flat region, which can significantly reduce the misjudgment rate of the high-reflection region and improve the accuracy and stability of edge positioning.

[0067] S105: Filter the significance values of all the adjusted pixel points to retain the required pixel points in the oral fusion image, and construct an oral three-dimensional model based on the optimized oral fusion image.

[0068] Specifically, the step of filtering the significance values of all the adjusted pixel points includes:

[0069] Step 1: Arrange all pixel points in ascending order according to the magnitude of their adjusted significance values to obtain the first quartile Q1 and the third quartile Q3;

[0070] Step 2: Filter all the pixel points. When S'(x,y) > Q3 + (1.5·IQR) or S'(x,y) < Q1 - (1.5·IQR), filter the abnormal pixel point with the current coordinates (x,y), where S'(x,y) represents the adjusted significance value of the pixel point with coordinates (x,y), and IQR = Q3 - Q1.

[0071] It should be noted that the above filtering method can eliminate extreme value interference and improve the robustness of data analysis. This can remove abrupt bright spots, reflection errors, and isolated abnormal pixels while retaining pixels with rich textures, thereby outputting high-precision oral cavity fusion images to construct oral cavity 3D models and improving the accuracy and realism of oral cavity 3D models.

[0072] like Figure 2 As shown in the figure, a specific embodiment of the present invention also provides an oral cavity image processing system based on multimodal enhancement. The oral cavity image processing system 1 includes:

[0073] The acquisition module 11 is used to acquire oral cavity images of multiple modalities, wherein the oral cavity images include visible light images, near-infrared images and laser depth images from different angles, and to perform image preprocessing on the visible light images;

[0074] The multi-scale image processing module 12 is used to perform multi-scale downsampling and matching processing on the visible light image, near-infrared image and laser depth image respectively, and to perform multiple edge detection on the feature images of each modality at different scales after matching, so as to obtain the feature intensity and local structure density of each feature image.

[0075] The multimodal fusion module 13 is used to determine the fusion confidence of each feature image of each modality by using preset multimodal fusion rules and the feature intensity and local structure density of each feature image, and to superimpose and fuse the feature images of each modality according to the fusion confidence of all feature images to obtain a multimodal enhanced oral cavity fusion image.

[0076] The saliency processing module 14 is used to calculate the saliency value of each pixel in the oral cavity fusion image and adjust the saliency value of each pixel according to the saliency value adjustment rule;

[0077] The optimization module 15 filters the saliency values ​​of all the adjusted pixels to retain the required pixels in the oral cavity fusion image and constructs a three-dimensional oral cavity model based on the optimized oral cavity fusion image.

[0078] It should be noted that the oral image processing system 1 of the present invention performs the specific steps of the oral image processing method described above. Therefore, the working principle of the oral image processing system 1 will not be described in detail.

[0079] In summary, this invention achieves full-range feature extraction from macroscopic to ultramicroscopic features by performing multi-scale macroscopic to ultramicroscopic feature extraction on multimodal oral images, avoiding missed detection of ultramicroscopic structures and improving detection accuracy. Multiple edge detection is employed to enhance the comprehensiveness of feature capture. By superimposing and fusing multimodal oral images, noisy edges are effectively suppressed, while realistic edges are enhanced and preserved, improving edge localization accuracy, thereby increasing the recognition accuracy of oral images. Furthermore, by adjusting and filtering the saliency of the fused oral image, abrupt bright spots, reflection errors, and isolated abnormal pixels can be removed while retaining pixels with rich textures. This results in a high-precision fused oral image for constructing a three-dimensional oral model, improving the accuracy and realism of the three-dimensional oral model.

[0080] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for processing oral images based on multimodal enhancement, characterized in that, include: Multiple modal images of the oral cavity are acquired, including visible light images, near-infrared images, and laser depth images from different angles, and the visible light images are preprocessed. The visible light image, near-infrared image, and laser depth image are subjected to multi-scale downsampling and matching processing, and multiple edge detection is performed on the feature images of each modality at different scales after matching to obtain the feature intensity and local structure density of each feature image. By using preset multimodal fusion rules and the feature intensity and local structure density of each feature image, the fusion confidence of each feature image of each modality is determined, and the feature images of each modality are superimposed and fused according to the fusion confidence of all feature images to obtain a multimodal enhanced oral cavity fusion image. The saliency value of each pixel in the oral cavity fusion image is calculated, and the saliency value of each pixel is adjusted according to the saliency value adjustment rule; The saliency values ​​of all the adjusted pixels are filtered to retain the desired pixels in the oral cavity fusion image, and a three-dimensional oral cavity model is constructed based on the optimized oral cavity fusion image.

2. The oral cavity image processing method based on multimodal enhancement according to claim 1, characterized in that, The steps of performing multi-scale downsampling and matching processing on the visible light image, near-infrared image, and laser depth image respectively include: The matched and aligned visible light image, near-infrared image, and laser depth image are subjected to multi-scale downsampling processing according to multiple preset scale factors to obtain scale images of each modality at different scales. The multiple scale factors include scale factors of 3, 5, 7, and 9, as well as an ultra-micro scale factor, wherein the ultra-micro scale factor includes scale factors of 1.2 and 1.

5. Denoising is performed on images at each scale using a GAN-based semantic denoising algorithm; The grayscale transformation process is performed on each scale image by using a grayscale level division method, so that each scale image generates two grayscale transformed images and serves as feature images. Feature points are extracted from the feature image, and spatial registration is performed on the extracted matching feature points to match and align feature images of different angles or modalities at the pixel level.

3. The oral cavity image processing method based on multimodal enhancement according to claim 1, characterized in that, The step of performing multiple edge detection on the feature images of each modality at different scales after matching, and obtaining the feature intensity and local structure density of each feature image, includes: Canny edge detection is performed on the feature images at different scales to obtain the feature intensity and local structure density of each feature image: When calculating the gradient magnitude and direction of the feature image, a preset arctangent function value table is directly called to replace floating-point operations with table lookup.

4. The oral cavity image processing method based on multimodal enhancement according to claim 1, characterized in that, The step of determining the fusion confidence of each feature image of each modality using preset multimodal fusion rules and the feature intensity and local structure density of each feature image includes: Multimodal fusion confidence is calculated for each feature image to obtain the fusion confidence of each feature image. The calculation formula is as follows: Among them, Q i f is the fusion confidence of the i-th feature image. i L represents the feature intensity of the i-th feature image. i Let be the local structure density of the i-th feature image, where i ranges from [1, M] and M is the total number of feature images.

5. The oral cavity image processing method based on multimodal enhancement according to claim 1 or 4, characterized in that, The step of superimposing and fusing the feature images of each modality based on the fusion confidence of all feature images to obtain a multimodal enhanced oral cavity fusion image includes: Superimpose and fuse the feature images of each modality to obtain an oral fusion image enhanced by multi-modalities. The calculation formula is as follows: Where F represents the multimodal enhanced oral cavity fusion image, Q i Let T be the fusion confidence of the i-th feature image. i Let i be the i-th feature image, where i ranges from [1, M] and M is the total number of feature images.

6. The oral cavity image processing method based on multimodal enhancement according to claim 1, characterized in that, The steps of calculating the saliency value of each pixel in the oral fusion image include: The calculation formula for the saliency value of each pixel is: Where S(x,y) represents the saliency value of the pixel at coordinate (x,y), and F(x,y) represents the feature data of the pixel at coordinate (x,y). This represents the grayscale gradient of the pixel at coordinates (x, y). Let F_N represent the gradient in the color space, and let σ represent the average eigenvector of a predefined local neighborhood. 2 (x,y) represents the gray-level variance of the preset local neighborhood of the pixel with coordinates (x,y), where w1, w2, and w3 are all adjustable weight coefficients, and w1+w2+w3=1.

7. The oral cavity image processing method based on multimodal enhancement according to claim 1, characterized in that, The steps of adjusting the saliency value of each pixel according to the saliency value adjustment rule include: For each pixel, select a preset local neighborhood centered on it, and calculate the local neighborhood variance. The calculation formula is as follows: σ 2 (x,y)=(1 / |N|)×∑_{(i,j)∈N}[I(i,j)-μ_N] 2 ; Where, σ 2 (x,y) represents the local neighborhood variance of the pixel at coordinates (x,y), N represents the preset local neighborhood centered at coordinates (x,y) and is a set, |N| represents the size of the preset local neighborhood set N, which includes all pixels in the preset local neighborhood, I(i,j) represents the pixel intensity or gray value of the pixel at coordinates (x,y), and μ_N is the gray mean of the preset local neighborhood; Construct an inhibition factor based on the local neighborhood variance. The construction formula is as follows: ψ(x,y)=σ 2 (x,y) / [σ 2 (x,y)+ε]; Where, ψ(x,y) represents the inhibition factor of the pixel with coordinates (x,y), and ε is a constant; The adjustment formula for adjusting the saliency value of each pixel is: S'(x,y) = S(x,y) × ψ(x,y); Where, S'(x,y) represents the adjusted saliency value of the pixel with coordinates (x,y), and S(x,y) represents the original saliency value of the pixel with coordinates (x,y).

8. The oral cavity image processing method based on multimodal enhancement according to claim 1, characterized in that, The steps of filtering the saliency values of all the adjusted pixels include: Arrange all pixels in ascending order according to the magnitude of their adjusted saliency values to obtain the first quartile Q1 and the third quartile Q3; Filter all the pixels. When S'(x,y) > Q3 + (1.5·IQR) or S'(x,y) < Q1 - (1.5·IQR), filter the abnormal pixel with the current coordinates (x,y). Where, S'(x,y) represents the adjusted saliency value of the pixel with coordinates (x,y), and IQR = Q3 - Q1.

9. The oral cavity image processing method based on multimodal enhancement according to claim 2, characterized in that, The steps of performing gray-level transformation on each scale image by using the gray-level division method to generate two gray-level transformed images for each scale image include: Perform equidistant division and central offset division on each scale image respectively, so that each scale image is transformed into two gray-level transformed images.

10. A dental image processing system based on multimodal enhancement, characterized in that, Include: An acquisition module, used to acquire oral images of multiple modalities. Among them, the oral images include visible light images, near-infrared images, and laser depth images at different angles, and perform image preprocessing on the visible light images; A multi-scale image processing module, used to perform multi-scale downsampling and matching processing on the visible light images, near-infrared images, and laser depth images respectively, and perform multiple edge detections on the feature images of different scales after matching to obtain the feature intensity and local structure density of each feature image; A multi-modal fusion module, used to determine the fusion confidence of each feature image of each modality through a preset multi-modal fusion rule and the feature intensity and local structure density of each feature image, and perform superimposed fusion on the feature images of each modality according to the fusion confidence of all feature images to obtain an oral fusion image enhanced by multi-modalities; A saliency processing module, used to calculate the saliency value of each pixel in the oral fusion image, and adjust the saliency value of each pixel according to the saliency value adjustment rule; The optimization module filters the saliency values ​​of all the adjusted pixels to retain the required pixels in the oral cavity fusion image and constructs a three-dimensional oral cavity model based on the optimized oral cavity fusion image.

Images