An efficient and lightweight method and device for calculus segmentation
By constructing an efficient and lightweight dental calculus segmentation model based on attention mechanism bilateral residuals and attention pyramid pooling modules, the problem of balancing accuracy and sensitivity with lightweight design in dental calculus image segmentation is solved, making it suitable for rapid diagnosis on mobile platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF GEOSCIENCES (WUHAN)
- Filing Date
- 2023-10-30
- Publication Date
- 2026-07-21
AI Technical Summary
Existing dental calculus image segmentation techniques struggle to achieve both accuracy and sensitivity while remaining lightweight and practical. Furthermore, dental calculus image datasets exhibit diverse shapes and varying image quality, making them prone to misjudgment.
We designed a bilateral residual module and an attention pyramid pooling module based on the attention mechanism to build an efficient and lightweight calculus segmentation model. Through a multi-level downsampling and decoder structure, combined with a feature aggregation module, we optimized the model parameters and training process to adapt to mobile platform deployment.
It achieves high-precision calculus segmentation with extremely low parameter count, is suitable for low-memory mobile platforms, assists doctors in completing fast and accurate medical diagnoses, and solves the problem of limited storage space on mobile devices.
Smart Images
Figure CN117522880B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image processing, and in particular to an efficient and lightweight method and device for dental calculus segmentation. Background Technology
[0002] Dental calculus is a common oral disease, consisting of calcified or calcifying plaque or soft deposits on teeth. If not removed promptly, it continuously irritates periodontal tissues and compresses the gums, leading to periodontitis, gingivitis, and other oral diseases. Therefore, a comprehensive understanding of dental calculus is crucial for clinicians to develop treatment plans. However, relying on dentists to analyze patient oral images is a tedious and time-consuming task. Furthermore, the varied morphology of dental calculus and visual fatigue increase the potential risk of missing small lesions. Therefore, using automated methods for detecting and segmenting dental calculus to assist treatment and improve screening accuracy has significant research value and practical application prospects. However, accurately segmenting dental calculus from oral images for diagnosis and evaluation remains a challenging problem.
[0003] In recent years, medical image segmentation technology has been widely used to assist doctors in diagnosis. Traditional segmentation methods mainly include thresholding, edge detection, clustering, and region growing. These methods can segment dental calculus to a certain extent, but they have some limitations, such as the need for manual threshold selection and sensitivity to noise and texture. With the development of deep learning, convolutional neural networks have become an important tool for image segmentation, achieving more accurate and efficient segmentation compared to traditional techniques. However, most current neural network models suffer from a large number of parameters, often employing a strategy of trading parameters for accuracy. Furthermore, we note that dental calculus image datasets are inherently varied in shape and quality, with blurred boundaries between the images and surrounding tissues. These tissue structures resemble the texture of dental calculus images, easily leading to misjudgments, resulting in current research on dental calculus image segmentation being neither stable nor reliable.
[0004] Therefore, improving the accuracy and sensitivity of calculus segmentation while ensuring lightweight design and practicality is an urgent problem to be solved. Summary of the Invention
[0005] The purpose of this application is to solve the technical problem that conventional medical image segmentation techniques, while ensuring the accuracy and sensitivity of tartar segmentation, are difficult to balance lightweight and practicality, and to provide an efficient and lightweight method and device for tartar segmentation.
[0006] The above-mentioned objective of this application is achieved through the following technical solution:
[0007] S1: Based on an attention mechanism strategy, a bilateral residual module and an attention pyramid pooling module are designed to construct an efficient and lightweight model for calculus segmentation, specifically including:
[0008] S11: The efficient and lightweight model includes: multiple downsampling modules, a bilateral residual module, an attention pyramid pooling module, and a feature aggregation module; the encoder of the efficient and lightweight model is constructed through the multiple downsampling modules and the bilateral residual module;
[0009] S12: Construct the decoder of the efficient and lightweight model through the attention pyramid pooling module and the feature aggregation module;
[0010] S13: Combine the encoder and the decoder to construct the efficient and lightweight model;
[0011] S2: Acquire dental calculus image data; train the efficient and lightweight model based on the training set of the dental calculus image data;
[0012] S3: Validate the model weights of the trained high-efficiency lightweight model based on the validation set of the dental calculus image data, and generate an evaluation result; adjust the model weight parameters of the high-efficiency lightweight model based on the evaluation result;
[0013] S4: Test the validated efficient and lightweight model using the test set of the tartar image data, and deploy the model weight file of the tested efficient and lightweight model to a mobile platform for tartar segmentation and detection.
[0014] Optionally, step S11 includes:
[0015] The encoder's network structure is constructed by combining multiple downsampling modules and bilateral residual modules, and connecting and aggregating them across multiple levels. The input data to the encoder is processed sequentially through each level, as follows:
[0016] In the first level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 29 convolutional kernels of size 3×3 with a stride of 2; the Conv module obtains data with features of 512×256×29; the Pooling module uses max pooling with a stride of 2 to obtain data with features of 512×256×3; the outputs of the Conv module and the Pooling module are combined to obtain data with features of 512×256×32;
[0017] In the second level: each side of the bilateral residual module is configured with: a spatial attention module, 32 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 512×256×32 data;
[0018] In the third level: two bilateral residual modules are stacked repeatedly, each side configured with: a spatial attention module, 32 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 512×256×32 data;
[0019] In the fourth level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 32 convolutional kernels of size 3×3 with a stride of 2, and outputs data with a feature of 256×128×32; the Pooling module uses max pooling with a stride of 2, and outputs data with a feature of 256×128×32; the outputs of the Conv module and the Pooling module are combined to obtain data with a feature of 256×128×64;
[0020] In the fifth level: both sides of the bilateral residual module are configured with: a spatial attention module and a channel attention module, 64 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 256×128×64 data.
[0021] In the sixth level: both sides of the bilateral residual module are configured with: a spatial attention module and a channel attention module, 64 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 256×128×64 data.
[0022] In the seventh level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 64 convolutional kernels of size 3×3 with a stride of 2, outputting data with a feature size of 128×64×64; the Pooling module uses max pooling with a stride of 2, outputting data with a feature size of 128×64×64; the outputs of the Conv module and the Pooling module are merged to obtain data with a feature size of 128×64×128;
[0023] In the eighth level: each side of the bilateral residual module is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilatation rate of (2, 1), and 128 convolutional kernels of size 1×3 with a dilatation rate of (1, 2); the output feature is 128×64×128 data.
[0024] In the ninth level: Each side of the bilateral residual module in the ninth level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilation rate of (5, 1), and 128 convolutional kernels of size 1×3 with a dilation rate of (1, 5); the output feature is 128×64×128 data.
[0025] In the tenth level: Each side of the bilateral residual module in the tenth level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilation rate of (9, 1), and 128 convolutional kernels of size 1×3 with a dilation rate of (1, 9); the output feature is 128×64×128 data.
[0026] In the eleventh level: Each side of the bilateral residual module in the eleventh level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilatation rate of (17, 1), and 128 convolutional kernels of size 1×3 with a dilatation rate of (1, 17); the output features are 128×64×128 data.
[0027] Optionally, step S12 includes:
[0028] The data input to the decoder is processed sequentially through each layer. The network structure of the decoder is as follows:
[0029] In the first level: the Pooling module is set to max pooling with a step size of 4. After processing by the Pooling module, the output data has a feature of 128×64×32.
[0030] In the second level: the spatial attention module processes the data and outputs data with features of 128×64×32;
[0031] In the third level: the Pooling module sets the step size to max pooling of 2. After processing by the Pooling module, the output data has a feature of 128×64×64.
[0032] In the fourth level: a convolutional attention module is constructed, and through the convolutional attention module, data with features of 128×64×64 is output;
[0033] In the fifth level: a merging module is constructed to merge the output of the second level in the decoder, the output of the fourth level in the decoder, and the output of the eleventh level in the encoder, with the output feature being 128×64×224 data;
[0034] In the sixth level: through the adaptive pooling submodule in the attention pyramid pooling module, four data with sizes of 1×1, 2×2, 3×3, and 6×6 are output in parallel. Then, through the spatial attention module, the channel attention module, and the Conv module with 56 convolutional kernels of 1×1, the four outputs are finally merged and added to the residual to obtain data of 128×64×280.
[0035] In the seventh level: the feature map size of the input data is doubled by bilinear interpolation, resulting in data with features of 256×128×280;
[0036] In the eighth level: a feature aggregation module is constructed, which merges the outputs of the encoder's third and sixth levels into a 256×128×96 data set. This data set, along with the 256×128×280 data set from the decoder's seventh level, is then input into the feature aggregation module to generate a 256×128×128 feature map, as detailed below:
[0037] The 256×128×96 data in the encoder and the 256×128×280 data in the decoder are merged, and then passed through a Conv module with 128 convolutional kernels of 1×1 to obtain a 256×128×128 feature map. Then, a global pooling operation is performed to realize feature mapping and obtain the first guidance information.
[0038] The first guidance information is multiplied with the 256×128×128 feature map to generate a multiplication result, thereby capturing important information;
[0039] The multiplication result is added to the 256×128×128 feature map, and the final output is data with a feature of 256×128×128;
[0040] In the ninth level: the feature map size of the input data in the eighth level is doubled by bilinear interpolation, and the output feature is 512×256×128 data.
[0041] In the tenth level: the 512×256×32 output of the encoder's third level and the 512×256×128 output of the decoder's ninth level are input together into the feature aggregation module, ultimately generating a 512×256×64 feature map, as follows:
[0042] The 512×256×32 data in the encoder and the 512×256×128 data in the decoder are merged, and then passed through a Conv module with 64 convolutional kernels of 1×1 to obtain a 512×256×64 feature map. After passing through global pooling feature mapping, the second guidance information is obtained.
[0043] The second guidance information is multiplied by the 512×256×64 feature map to capture important information;
[0044] The result of the multiplication is added to the 512×256×64 feature map, and the final output is data with a feature of 512×256×64.
[0045] In the eleventh level: the data input from the tenth level is processed by two Conv modules with a kernel size of 1×1 and a stride of 1, and the output features are 512×256×2 data.
[0046] In the twelfth level: the feature map size of the input data from the eleventh level is doubled by bilinear interpolation to obtain data with features of 1024×512×2; the 1024×512×2 data is the final prediction result.
[0047] Optionally, step S2 includes:
[0048] S21: Use stochastic gradient descent as the optimizer and set the initial parameters of the optimizer;
[0049] S22: Set the number of training iterations required for training, obtain the loss function, and train the efficient and lightweight model using the loss function and dental calculus image data;
[0050] S23: Update and calculate the network parameters affecting the efficient lightweight model through the optimizer, and adjust the value of the loss function according to the network parameters;
[0051] S24: The training process is guided by a weight decay strategy and a learning rate reduction method using cosine annealing to ensure that the efficient and lightweight model reaches a better convergence state during training.
[0052] Optionally, step S3 includes:
[0053] During the training process of the efficient and lightweight model, the average intersection-over-union ratio and loss value of the efficient and lightweight model are verified using the validation set.
[0054] Obtain the loss curve of the loss function graph of the efficient and lightweight model after one training cycle, and determine whether the loss curve has converged.
[0055] If the loss curve converges, it is determined that the training of the efficient lightweight model has stabilized, and the training of the efficient lightweight model is stopped.
[0056] If the loss curve does not converge, it is determined that the training of the efficient lightweight model is not yet stable, and the training of the efficient lightweight model continues.
[0057] Optionally, step S4 includes:
[0058] The validated efficient and lightweight model was tested using a test set of the dental calculus image data.
[0059] If the test accuracy is less than the preset accuracy threshold, then continue training the efficient and lightweight model.
[0060] If the test accuracy is not less than the preset accuracy threshold, then training is stopped and the model weight file is output.
[0061] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform an efficient and lightweight method for calculus separation.
[0062] A computer-readable storage medium storing instructions that, when executed, perform an efficient and lightweight method for calculus segmentation.
[0063] The beneficial effects of the technical solution provided in this application are:
[0064] Based on the attention mechanism strategy, we design an efficient and lightweight model with a bilateral residual module and an attention pyramid pooling module, and configure two attention mechanisms to capture feature information and improve segmentation performance.
[0065] A neural network model was trained using a dataset of dental calculus images collected by dentists. Various parameters were set, and the model was trained until the error converged. The receptive field was expanded by using asymmetric convolutions (1×3 and 3×1 kernels) and dilated convolutions with different dilation rates in the bilateral residual module, and the parameters of the efficient, lightweight model were reduced to improve its sensitivity. The trained model weights were validated using a partitioned validation set, and the parameters were adjusted based on the evaluation results. The model was tested using a partitioned test set. Finally, the model was deployed to mobile devices, and the actual segmentation prediction results were used to assist dentists in making diagnoses.
[0066] Compared with existing methods of the same type, this application achieves a better balance between segmentation performance and model lightweighting, improves the accuracy and sensitivity of calculus segmentation tasks, is more suitable for use in clinical settings, and has broad prospects for practical application.
[0067] The efficient and lightweight model of this application achieves high-precision calculus segmentation with extremely low parameter count, striking a good balance between segmentation accuracy and model lightweight design. It is suitable for deployment on low-memory mobile platforms to assist doctors in completing medical diagnoses, solving the problem of limited storage space on mobile deployment devices and providing medical professionals with a fast and accurate diagnostic tool. Attached Figure Description
[0068] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:
[0069] Figure 1 This is a flowchart illustrating the steps of an efficient and lightweight method for calculus segmentation in the embodiments of this application;
[0070] Figure 2 This is a schematic diagram of the efficient and lightweight model structure of the efficient and lightweight method for calculus segmentation in the embodiments of this application;
[0071] Figure 3 This is a schematic diagram of an electronic device structure for an efficient and lightweight method for tartar separation in the embodiments of this application. Detailed Implementation
[0072] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0073] The embodiments of this application provide an efficient and lightweight method and apparatus for calculus segmentation.
[0074] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of an efficient and lightweight method for calculus segmentation in an embodiment of this application, specifically including the following steps:
[0075] S1: Based on an attention mechanism strategy, a bilateral residual module and an attention pyramid pooling module are designed to construct an efficient and lightweight model for calculus segmentation, specifically including:
[0076] S11: The efficient and lightweight model includes: multiple downsampling modules, a bilateral residual module, an attention pyramid pooling module, and a feature aggregation module; the encoder of the efficient and lightweight model is constructed through the multiple downsampling modules and the bilateral residual module;
[0077] S12: Construct the decoder of the efficient and lightweight model through the attention pyramid pooling module and the feature aggregation module;
[0078] S13: Combine the encoder and the decoder to construct the efficient and lightweight model;
[0079] S2: Acquire dental calculus image data; train the efficient and lightweight model based on the training set of the dental calculus image data;
[0080] S3: Validate the model weights of the trained high-efficiency lightweight model based on the validation set of the dental calculus image data, and generate an evaluation result; adjust the model weight parameters of the high-efficiency lightweight model based on the evaluation result;
[0081] S4: Test the validated efficient and lightweight model using the test set of the tartar image data, and deploy the model weight file of the tested efficient and lightweight model to a mobile platform for tartar segmentation and detection.
[0082] Specifically, dentists use an endoscope to take images of the patient's mouth and upload the photos to mobile platforms such as mobile phones and tablets. The images are then processed by the efficient and lightweight model to obtain prediction results, which help medical personnel make judgments about the patient's health status.
[0083] Specifically, the downsampling module cropped the size of the tartar image data; the bilateral residual module extracted the feature information of the tartar image data; the attention pyramid pooling module processed the global information and detailed information of the tartar image data; and the feature aggregation module aggregated feature maps at different levels and enhanced the interaction between the encoder's feature map and the decoder's feature map.
[0084] Step S11 includes: combining multiple downsampling modules and the bilateral residual module, and connecting and aggregating them at multiple levels to construct the network structure of the encoder. The data input to the encoder is processed sequentially through each level, as follows:
[0085] In the first level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 29 convolutional kernels of size 3×3 with a stride of 2; the Conv module obtains data with features of 512×256×29; the Pooling module uses max pooling with a stride of 2 to obtain data with features of 512×256×3; the outputs of the Conv module and the Pooling module are combined to obtain data with features of 512×256×32;
[0086] In the second level: each side of the bilateral residual module is configured with: a spatial attention module, 32 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 512×256×32 data;
[0087] In the third level: two bilateral residual modules are stacked repeatedly, each side configured with: a spatial attention module, 32 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 512×256×32 data;
[0088] In the fourth level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 32 convolutional kernels of size 3×3 with a stride of 2, and outputs data with a feature of 256×128×32; the Pooling module uses max pooling with a stride of 2, and outputs data with a feature of 256×128×32; the outputs of the Conv module and the Pooling module are combined to obtain data with a feature of 256×128×64;
[0089] In the fifth level: both sides of the bilateral residual module are configured with: a spatial attention module and a channel attention module, 64 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 256×128×64 data.
[0090] In the sixth level: both sides of the bilateral residual module are configured with: a spatial attention module and a channel attention module, 64 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 256×128×64 data.
[0091] In the seventh level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 64 convolutional kernels of size 3×3 with a stride of 2, outputting data with a feature size of 128×64×64; the Pooling module uses max pooling with a stride of 2, outputting data with a feature size of 128×64×64; the outputs of the Conv module and the Pooling module are merged to obtain data with a feature size of 128×64×128;
[0092] In the eighth level: each side of the bilateral residual module is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilatation rate of (2, 1), and 128 convolutional kernels of size 1×3 with a dilatation rate of (1, 2); the output feature is 128×64×128 data.
[0093] In the ninth level: Each side of the bilateral residual module in the ninth level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilation rate of (5, 1), and 128 convolutional kernels of size 1×3 with a dilation rate of (1, 5); the output feature is 128×64×128 data.
[0094] In the tenth level: Each side of the bilateral residual module in the tenth level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilation rate of (9, 1), and 128 convolutional kernels of size 1×3 with a dilation rate of (1, 9); the output feature is 128×64×128 data.
[0095] In the eleventh level: Each side of the bilateral residual module in the eleventh level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilatation rate of (17, 1), and 128 convolutional kernels of size 1×3 with a dilatation rate of (1, 17); the output features are 128×64×128 data.
[0096] Specifically, in the efficient and lightweight model for calculus segmentation, the encoder is used to extract features from the calculus image; the downsampling module is used to reduce the problem of information loss in the feature map of the calculus image during downsampling. This module retains important detailed feature information of the calculus image through parallel convolution and pooling modules; the bilateral residual module is the main structure for feature extraction. It separates the input image information into two branches through channels. The two branches are each passed through an attention mechanism. Then, asymmetric dilated convolution is used to divide the 3×3 dilated convolution into a 1×3 and a 3×1 dilated convolution, which expands the receptive field of the model with a lower number of parameters. Finally, the outputs of the two branches are merged; the bilateral residual module first divides the input into two paths through channel separation, and then performs channel shuffling in its last step to achieve information sharing and obtain the output result.
[0097] The constructed encoder can fully acquire information from the input image and strictly control the size and number of channels of the input image during model processing, ensuring that the number of model parameters is within a low range, which can meet the needs of efficient and lightweight calculus segmentation in actual use.
[0098] Step S12 includes:
[0099] The data input to the decoder is processed sequentially through each layer. The network structure of the decoder is as follows:
[0100] In the first level: the Pooling module is set to max pooling with a step size of 4. After processing by the Pooling module, the output data has a feature of 128×64×32.
[0101] In the second level: the spatial attention module processes the data and outputs data with features of 128×64×32;
[0102] In the third level: the Pooling module sets the step size to max pooling of 2. After processing by the Pooling module, the output data has a feature of 128×64×64.
[0103] In the fourth level: a convolutional attention module is constructed, and through the convolutional attention module, data with features of 128×64×64 is output;
[0104] In the fifth level: a merging module is constructed to merge the output of the second level in the decoder, the output of the fourth level in the decoder, and the output of the eleventh level in the encoder, with the output feature being 128×64×224 data;
[0105] In the sixth level: through the adaptive pooling submodule in the attention pyramid pooling module, four data with sizes of 1×1, 2×2, 3×3, and 6×6 are output in parallel. Then, through the spatial attention module, the channel attention module, and the Conv module with 56 convolutional kernels of 1×1, the four outputs are finally merged and added to the residual to obtain data of 128×64×280.
[0106] In the seventh level: the feature map size of the input data is doubled by bilinear interpolation, resulting in data with features of 256×128×280;
[0107] In the eighth level: a feature aggregation module is constructed, which merges the outputs of the encoder's third and sixth levels into a 256×128×96 data set. This data set, along with the 256×128×280 data set from the decoder's seventh level, is then input into the feature aggregation module to generate a 256×128×128 feature map, as detailed below:
[0108] The 256×128×96 data in the encoder and the 256×128×280 data in the decoder are merged, and then passed through a Conv module with 128 convolutional kernels of 1×1 to obtain a 256×128×128 feature map. Then, a global pooling operation is performed to realize feature mapping and obtain the first guidance information.
[0109] The first guidance information is multiplied with the 256×128×128 feature map to generate a multiplication result, thereby capturing important information;
[0110] The multiplication result is added to the 256×128×128 feature map, and the final output is data with a feature of 256×128×128;
[0111] In the ninth level: the feature map size of the input data in the eighth level is doubled by bilinear interpolation, and the output feature is 512×256×128 data.
[0112] In the tenth level: the 512×256×32 output of the encoder's third level and the 512×256×128 output of the decoder's ninth level are input together into the feature aggregation module, ultimately generating a 512×256×64 feature map, as follows:
[0113] The 512×256×32 data in the encoder and the 512×256×128 data in the decoder are merged, and then passed through a Conv module with 64 convolutional kernels of 1×1 to obtain a 512×256×64 feature map. After passing through global pooling feature mapping, the second guidance information is obtained.
[0114] The second guidance information is multiplied by the 512×256×64 feature map to capture important information;
[0115] The result of the multiplication is added to the 512×256×64 feature map, and the final output is data with a feature of 512×256×64.
[0116] In the eleventh level: the data input from the tenth level is processed by two Conv modules with a kernel size of 1×1 and a stride of 1, and the output features are 512×256×2 data.
[0117] In the twelfth level: the feature map size of the input data from the eleventh level is doubled by bilinear interpolation to obtain data with features of 1024×512×2; the 1024×512×2 data is the final prediction result.
[0118] Specifically, the encoder outputs semantically rich features, which are then merged at different scales before being input into the decoder to obtain a feature map with global contextual information. The specific steps are: scaling the pooling results of information at different scales, then performing feature aggregation using an attention mechanism, and finally merging the output of the encoder with the merged feature map before inputting it into the attention pyramid pooling module of the decoder. The decoder has excellent feature interpretation capabilities, so the attention pyramid pooling module processes four branches in parallel to obtain information from different spatial locations of the feature map. Each branch then uses an attention module to suppress unimportant information and highlight effective information. Finally, an upsampling module merges the results of the four branches. Common feature aggregation operations can lose some feature information; therefore, a feature aggregation module is used to guide global information. It utilizes global pooling to help the model focus on specific features, then performs element-wise multiplication with the merged result to achieve this guidance. The feature aggregation module improves the model's segmentation accuracy without adding additional computational burden.
[0119] Step S2 includes:
[0120] S21: Use stochastic gradient descent as the optimizer and set the initial parameters of the optimizer;
[0121] S22: Set the number of training iterations required for training, obtain the loss function, and train the efficient and lightweight model using the loss function and tartar image data;
[0122] S23: Update and calculate the network parameters affecting the efficient lightweight model through the optimizer, and adjust the value of the loss function according to the network parameters;
[0123] S24: The training process is guided by a weight decay strategy and a learning rate reduction method using cosine annealing to ensure that the efficient and lightweight model reaches a better convergence state during training.
[0124] Step S3 includes:
[0125] During the training process of the efficient and lightweight model, the average intersection-over-union ratio and loss value of the efficient and lightweight model are verified using the validation set.
[0126] Obtain the loss curve of the loss function graph of the efficient and lightweight model after one training cycle, and determine whether the loss curve has converged.
[0127] If the loss curve converges, it is determined that the training of the efficient lightweight model has stabilized, and the training of the efficient lightweight model is stopped.
[0128] If the loss curve does not converge, it is determined that the training of the efficient lightweight model is not yet stable, and the training of the efficient lightweight model continues.
[0129] Specifically, an efficient and lightweight model for calculus segmentation is trained based on the training set. Generally, the BCE loss function is chosen for training, and stochastic gradient descent (SGD) is used as the optimizer with an initial learning rate of 0.01 and a momentum parameter of 0.9. A weight decay strategy with a decay coefficient of 0.0001 is applied, and cosine annealing is used to further guide the training process and ensure the model reaches a better convergence state. The training run is 1000 epochs to fully explore the model's potential.
[0130] Specifically, after each round of training and validation, the average crossover ratio and loss value of the efficient and lightweight model are validated using a validation set to obtain the loss function graph of the model after training, and then determine whether it has converged. The final average crossover ratio is the optimal average crossover ratio of the efficient and lightweight model for tartar segmentation on the validation set.
[0131] Step S4 includes: testing the validated efficient lightweight model using the test set of the dental calculus image data; if the test accuracy is less than a preset accuracy threshold, continuing to train the efficient lightweight model; if the test accuracy is not less than the preset accuracy threshold, stopping training and outputting the model weight file.
[0132] Specifically, the preset accuracy uses the average intersection-over-union ratio (IoU) as the metric, which can be set to 80%.
[0133] In another embodiment, based on 1033 images collected by dentists in the hospital, images containing tartar are used as positive samples and images without tartar are used as negative samples, thus forming a sample dataset containing 1033 images.
[0134] Specifically, the collected tartar images are labeled as tartar and background under the guidance of dentists, and the location and corresponding category of tartar in the images are marked using LabelMe software; the tartar images and corresponding annotation files are divided into training set, validation set and test set according to a ratio of 6:2:2.
[0135] Specifically, the efficient and lightweight model is built on a deep learning framework, which includes, but is not limited to, the PyTorch framework.
[0136] The method described in this application embodiment is executed by a terminal. The terminal can be any type of terminal; for example, the terminal can be, but is not limited to, at least one of the following: a server, a computer, a tablet computer, or other electronic devices.
[0137] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0138] This application also discloses an electronic device. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.
[0139] The communication bus 502 is used to enable communication between these components.
[0140] The user interface 503 may include a display screen and a camera. Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.
[0141] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0142] The processor 501 may include one or more processing cores. The processor 501 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 505, and by calling data stored in memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.
[0143] The memory 505 may include random access memory (RAM) or read-only memory.
[0144] Optionally, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. The memory 505 may also optionally include, but is not limited to, at least one storage device located remotely from the aforementioned processor 501. (Refer to...) Figure 3 The memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for an efficient and lightweight method of calculus segmentation.
[0145] exist Figure 3 In the illustrated electronic device 500, the user interface 503 is mainly used to provide an input interface for the user and acquire user input data; while the processor 501 can be used to call an application program stored in the memory 505 for an efficient and lightweight method for calculus segmentation. When executed by one or more processors 501, the electronic device 500 performs one or more methods as described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0146] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0147] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed include, but are not limited to, indirect couplings or communication connections between apparatuses or units through some service interfaces, including but not limited to electrical or other forms.
[0148] The units described as separate components include, but are not limited to, physically separate units. The components shown as units include, but are not limited to, physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of these units can be selected to achieve the purpose of this embodiment according to actual needs.
[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or may exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.
[0151] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.
[0152] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.
Claims
1. A highly efficient and lightweight method for calculus segmentation, characterized in that, The method includes the following steps: S1: Based on an attention mechanism strategy, a bilateral residual module and an attention pyramid pooling module are designed to construct an efficient and lightweight model for calculus segmentation, specifically including: S11: The efficient and lightweight model includes: multiple downsampling modules, a bilateral residual module, an attention pyramid pooling module, and a feature aggregation module; the encoder of the efficient and lightweight model is constructed through the multiple downsampling modules and the bilateral residual module; Step S11 includes: The encoder's network structure is constructed by combining multiple downsampling modules and bilateral residual modules, and connecting and aggregating them across multiple levels. The input data to the encoder is processed sequentially through each level, as follows: In the first level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 29 convolutional kernels of size 3×3 with a stride of 2; the Conv module obtains data with features of 512×256×29; the Pooling module uses max pooling with a stride of 2 to obtain data with features of 512×256×3; the outputs of the Conv module and the Pooling module are combined to obtain data with features of 512×256×32; In the second level: each side of the bilateral residual module is configured with: a spatial attention module, 32 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 512×256×32 data; In the third level: two bilateral residual modules are stacked repeatedly, each side configured with: a spatial attention module, 32 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 512×256×32 data; In the fourth level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 32 convolutional kernels of size 3×3 with a stride of 2, and outputs data with a feature of 256×128×32; the Pooling module uses max pooling with a stride of 2, and outputs data with a feature of 256×128×32; the outputs of the Conv module and the Pooling module are combined to obtain data with a feature of 256×128×64; In the fifth level: both sides of the bilateral residual module are configured with: a spatial attention module and a channel attention module, 64 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 256×128×64 data. In the sixth level: both sides of the bilateral residual module are configured with: a spatial attention module and a channel attention module, 64 convolutional kernels of size 3×1 and 32 convolutional kernels of size 1×3; the output features are 256×128×64 data. In the seventh level: the downsampling module consists of a Conv module and a Pooling module in parallel; the Conv module uses 64 convolutional kernels of size 3×3 with a stride of 2, outputting data with a feature size of 128×64×64; the Pooling module uses max pooling with a stride of 2, outputting data with a feature size of 128×64×64; the outputs of the Conv module and the Pooling module are merged to obtain data with a feature size of 128×64×128; In the eighth level: each side of the bilateral residual module is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilatation rate of (2, 1), and 128 convolutional kernels of size 1×3 with a dilatation rate of (1, 2); the output feature is 128×64×128 data. In the ninth level: Each side of the bilateral residual module in the ninth level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilation rate of (5, 1), and 128 convolutional kernels of size 1×3 with a dilation rate of (1, 5); the output feature is 128×64×128 data. In the tenth level: Each side of the bilateral residual module in the tenth level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilation rate of (9, 1), and 128 convolutional kernels of size 1×3 with a dilation rate of (1, 9); the output feature is 128×64×128 data. In the eleventh level: Each side of the bilateral residual module in the eleventh level is configured with: a channel attention module, 128 convolutional kernels of size 3×1 with a dilatation rate of (17, 1), and 128 convolutional kernels of size 1×3 with a dilatation rate of (1, 17); the output feature is 128×64×128 data. S12: Construct the decoder of the efficient and lightweight model through the attention pyramid pooling module and the feature aggregation module; Step S12 includes: The data input to the decoder is processed sequentially through each layer. The network structure of the decoder is as follows: In the first level: the Pooling module is set to max pooling with a step size of 4. After processing by the Pooling module, the output data has a feature of 128×64×32. In the second level: the spatial attention module processes the data and outputs data with features of 128×64×32; In the third level: the Pooling module sets the step size to max pooling of 2. After processing by the Pooling module, the output data has a feature of 128×64×64. In the fourth level: a convolutional attention module is constructed, and through the convolutional attention module, data with features of 128×64×64 is output; In the fifth level: a merging module is constructed to merge the output of the second level in the decoder, the output of the fourth level in the decoder, and the output of the eleventh level in the encoder, with the output feature being 128×64×224 data; In the sixth level: through the adaptive pooling submodule in the attention pyramid pooling module, four data with sizes of 1×1, 2×2, 3×3, and 6×6 are output in parallel. Then, through the spatial attention module, the channel attention module, and the Conv module with 56 convolutional kernels of 1×1, the four outputs are finally merged and added to the residual to obtain data of 128×64×280. In the seventh level: the feature map size of the input data is doubled by bilinear interpolation, resulting in data with features of 256×128×280; In the eighth level: a feature aggregation module is constructed, which merges the outputs of the encoder's third and sixth levels into a 256×128×96 data set. This data set, along with the 256×128×280 data set from the decoder's seventh level, is then input into the feature aggregation module to generate a 256×128×128 feature map, as detailed below: The 256×128×96 data in the encoder and the 256×128×280 data in the decoder are merged, and then passed through a Conv module with 128 convolutional kernels of 1×1 to obtain a 256×128×128 feature map. Then, a global pooling operation is performed to realize feature mapping and obtain the first guidance information. The first guidance information is multiplied with the 256×128×128 feature map to generate a multiplication result, thereby capturing important information; The multiplication result is added to the 256×128×128 feature map, and the final output is data with a feature of 256×128×128; In the ninth level: the feature map size of the input data in the eighth level is doubled by bilinear interpolation, and the output feature is 512×256×128 data. In the tenth level: The 512×256×32 output from the third level of the encoder and the 512×256×128 output from the ninth level of the decoder are input together into the feature aggregation module to finally generate a 512×256×64 feature map, as follows: The 512×256×32 data in the encoder and the 512×256×128 data in the decoder are merged, and then passed through a Conv module with 64 convolutional kernels of 1×1 to obtain a 512×256×64 feature map. After passing through global pooling feature mapping, the second guidance information is obtained. The second guidance information is multiplied by the 512×256×64 feature map to capture important information; The result of the multiplication is added to the 512×256×64 feature map, and the final output is data with a feature of 512×256×64. In the eleventh level: the data input from the tenth level is processed by two Conv modules with a kernel size of 1×1 and a stride of 1, and the output features are 512×256×2 data. In the twelfth level: the feature map size of the input data from the eleventh level is doubled by bilinear interpolation to obtain data with a feature size of 1024×512×2; the 1024×512×2 data is the final prediction result; S13: Combine the encoder and the decoder to construct the efficient and lightweight model; The bilateral residual module is used to separate the input image information into two branches through the channel. The two branches are respectively passed through an attention mechanism and asymmetric dilated convolution is used to divide the 3×3 dilated convolution into a 1×3 and a 3×1 dilated convolution. Then the outputs of the two branches are merged. S2: Acquire dental calculus image data; train the efficient and lightweight model based on the training set of the dental calculus image data; S3: Validate the model weights of the trained high-efficiency lightweight model based on the validation set of the dental calculus image data, and generate an evaluation result; adjust the model weight parameters of the high-efficiency lightweight model based on the evaluation result; S4: Test the validated efficient and lightweight model using the test set of the tartar image data, and deploy the model weight file of the tested efficient and lightweight model to a mobile platform for tartar segmentation and detection.
2. The efficient and lightweight method for calculus segmentation as described in claim 1, characterized in that, Step S2 includes: S21: Use stochastic gradient descent as the optimizer and set the initial parameters of the optimizer; S22: Set the number of training iterations required for training, obtain the loss function, and train the efficient and lightweight model using the loss function and dental calculus image data; S23: Update and calculate the network parameters affecting the efficient lightweight model through the optimizer, and adjust the value of the loss function according to the network parameters; S24: The training process is guided by a weight decay strategy and a learning rate reduction method using cosine annealing to ensure that the efficient and lightweight model reaches a better convergence state during training.
3. The efficient and lightweight method for calculus segmentation as described in claim 1, characterized in that, Step S3 includes: During the training process of the efficient and lightweight model, the average intersection-over-union ratio and loss value of the efficient and lightweight model are verified using the validation set. Obtain the loss curve of the loss function graph of the efficient and lightweight model after one training cycle, and determine whether the loss curve has converged. If the loss curve converges, it is determined that the training of the efficient lightweight model has stabilized, and the training of the efficient lightweight model is stopped. If the loss curve does not converge, it is determined that the training of the efficient lightweight model is not yet stable, and the training of the efficient lightweight model continues.
4. The efficient and lightweight method for calculus segmentation as described in claim 1, characterized in that, Step S4 includes: The validated efficient and lightweight model was tested using a test set of the dental calculus image data. If the test accuracy is less than the preset accuracy threshold, then continue training the efficient and lightweight model. If the test accuracy is not less than the preset accuracy threshold, then training is stopped and the model weight file is output.
5. An electronic device, characterized in that, The device includes a processor (501), a memory (505), a user interface (503), and a network interface (504). The memory (505) is used to store instructions. The user interface (503) and the network interface (504) are used to communicate with other devices. The processor (501) is used to execute the instructions stored in the memory (505) to cause the electronic device (500) to perform the method as described in any one of claims 1-4.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the steps of the method as described in any one of claims 1-4.