Head shadow measurement method based on global attention and multi-scale feature fusion
By adopting the method of integrating global attention and multi -scale characteristics in the diverse measurement measurement, the problem of inaccurate and inconsistent measurement caused by manual recognition signs in the existing technology, and achieve higher measurement accuracy and consistency.
Patent Information
- Application Number
- CN202411406902.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-28
AI Technical Summary
Existing cephalogram measurement methods rely on manual identification of marks, which are time-consuming, labor-intensive and error-prone, affecting the accuracy and consistency of measurements.
The measurement method based on global attention and multi -scale characteristics is adopted. By obtaining and preparing X -ray images with a large number of skull side positions, it is built and divided into training, testing and verification sets. Use local feature extraction modules, global feature extraction modules, multi -scale feature fusion modules, and predictive output modules to create headhadling measurement models, training and verification, and eventually used for the leading measurement.
Effectively improve the accuracy and consistency of the measurement measurement, reduce human errors, and improve measurement efficiency.
Smart Images
Figure CN119380370A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cephalometric measurement, and in particular to a cephalometric measurement method based on global attention and multi-scale feature fusion. Background Art
[0002] Cephalometric analysis was first proposed by Broadbent and Hofrath in 1931 and has become an essential step in the orthodontic diagnostic process. It involves the identification and location of anatomical landmarks on a lateral skull radiograph. These landmarks are then used to measure distances, angles, and ratios, providing insight into the structural features and interrelationships of the craniofacial soft and hard tissues.
[0003] However, landmark detection on lateral skull radiographs is a significant challenge. Conventional cephalometric measurements rely on manual identification and marking of landmarks, a process that is not only time-consuming and labor-intensive, but also prone to errors due to subjective visual interpretation and differences in clinician experience levels, leading to inconsistent measurements during repeated assessments.
[0004] To mitigate these challenges, some clinicians have opted for computer-assisted semiautomated cephalometric methods, where the clinician manually identifies relevant landmarks and software performs distance, angle, and ratio measurements. However, manual landmark identification is necessary for both traditional and semiautomated methods and can be subject to inaccuracies, particularly among less experienced clinicians.
[0005] Therefore, how to provide a cephalometric measurement method based on global attention and multi-scale feature fusion to improve the accuracy and consistency of cephalometric measurement has become a technical problem that needs to be solved urgently. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a cephalometric measurement method based on global attention and multi-scale feature fusion, so as to improve the accuracy and consistency of cephalometric measurement.
[0007] The present invention is implemented as follows: a cephalometric method based on global attention and multi-scale feature fusion comprises the following steps:
[0008] Step S1, obtaining a large number of historical X-ray images of the lateral position of the skull, and constructing a data set after preprocessing and annotating each of the historical X-ray images;
[0009] Step S2: dividing the data set into a training set, a test set and a validation set based on a preset segmentation ratio;
[0010] Step S3, creating a cephalometric model based on the local feature extraction module, the global feature extraction module, the multi-scale feature fusion module and the prediction output module;
[0011] Step S4, training the cephalometric model using the training set, testing the trained cephalometric model using the test set, and verifying the cephalometric model that passes the test using the verification set;
[0012] Step S5: performing cephalometric measurement using the verified cephalometric model.
[0013] Furthermore, the step S1 is specifically as follows:
[0014] A large number of historical X-ray images of the lateral position of the skull are obtained, each of the historical X-ray images is preprocessed including at least image enhancement and sample expansion, key points of each of the historical X-ray images for cephalometric measurement are annotated and a data set is constructed.
[0015] Furthermore, the number of key points is 19.
[0016] Furthermore, the step S2 is specifically as follows:
[0017] The dataset was randomly divided into a training set, a test set, and a validation set based on a split ratio of 6:2:2.
[0018] Furthermore, in step S3, the local feature extraction module is composed of four convolutional layers with step sizes of 2, 1, 2, and 1 and a size of 3×3, which are used to extract the local feature from the input historical X-ray image I∈R C×H×W Local features are extracted from the historical X-ray image, and the image size of the historical X-ray image is reduced, and the local features and the reduced-size historical X-ray image are output to the global feature extraction module; wherein C represents the number of channels of the historical X-ray image; H represents the height of the historical X-ray image; and W represents the width of the historical X-ray image.
[0019] Furthermore, in step S3, the global feature extraction module includes two first global attention units, which are used to extract global features from historical X-ray images, and output the extracted global features and the input local features to a multi-scale feature fusion module.
[0020] Further, in step S3, the multi-scale feature fusion module includes three second global attention units and three first attention guiding units, and the input end of each of the first attention guiding units is respectively connected to the output end of each of the second global attention units;
[0021] The multi-scale feature fusion module is used to perform multi-scale feature fusion on the input global features and local features to obtain fused features, and output the fused features to the prediction output module.
[0022] Further, in step S3, the prediction output module includes four third global attention units and four second attention guiding units, and the input end of each second attention guiding unit is respectively connected to the output end of each third global attention unit;
[0023] The prediction output module is used to output a heat map for cephalometric measurement based on the input fusion features, and the number of the heat maps is the same as the number of the marked key points.
[0024] Furthermore, the step S4 is specifically as follows:
[0025] Training the cephalometric model using the training set, and continuously adjusting the loss function, optimizer function, and learnable hyperparameters of the cephalometric model during the training process until a hyperparameter combination that enables the cephalometric model to perform best on the training set is found;
[0026] The trained cephalometric model is tested using the test set to determine whether the measurement accuracy is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, the test passes, and:
[0027] The cephalometric model that has passed the test is verified using the verification set to determine whether the confidence is greater than a preset confidence threshold. If not, the training set is expanded to continue training; if so, the verification is passed and the training ends.
[0028] Furthermore, the step S5 is specifically as follows:
[0029] The latest X-ray image to be measured is obtained, and the latest X-ray image is pre-processed for image enhancement and then input into a verified cephalometric model, and the cephalometric model outputs a corresponding thermal map in real time for cephalometric measurement.
[0030] The advantages of the present invention are:
[0031] A large number of historical X-ray images of the lateral head are obtained, and a data set is constructed after preprocessing and annotating each historical X-ray image, and the data set is divided into a training set, a test set and a validation set; then a cephalometric model is created based on a local feature extraction module, a global feature extraction module, a multi-scale feature fusion module and a prediction output module, the cephalometric model is trained with the training set, the trained cephalometric model is tested with the test set, the cephalometric model that passes the test is verified with the validation set, and finally cephalometric measurement is performed with the verified cephalometric model; that is, the cephalometric model is trained with historical X-ray images with key points (landmarks) annotated, and the cephalometric model consists of a local feature extraction module, a global feature extraction module, a multi-scale feature fusion module and a prediction output module, which can effectively learn the local and global features of the historical X-ray images, and fuse the local and global features of different scales to output a heat map (the location information of key points), which ultimately greatly improves the accuracy and consistency of cephalometric measurement. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.
[0033] Figure 1 It is a flow chart of a cephalometric method based on global attention and multi-scale feature fusion of the present invention.
[0034] Figure 2 It is a structural schematic diagram of the cephalometric model of the present invention.
[0035] Figure 3 It is a structural schematic diagram of the global feature extraction module of the present invention.
[0036] Figure 4 It is a schematic diagram of multi-scale feature fusion of the present invention. DETAILED DESCRIPTION
[0037] The technical solution in the embodiments of the present application has the following overall idea: the cephalometric model is trained by using historical X-ray images with key points annotated, and the cephalometric model consists of a local feature extraction module, a global feature extraction module, a multi-scale feature fusion module and a prediction output module, which can effectively learn the local features and global features of historical X-ray images, and fuse local features and global features of different scales to improve the accuracy and consistency of cephalometric measurement.
[0038] Please refer to Figures 1 to 4 As shown, a preferred embodiment of a cephalometric method based on global attention and multi-scale feature fusion of the present invention comprises the following steps:
[0039] Step S1, obtaining a large number of historical X-ray images of the lateral position of the skull, and constructing a data set after preprocessing and annotating each of the historical X-ray images;
[0040] Step S2: dividing the data set into a training set, a test set and a validation set based on a preset segmentation ratio;
[0041] Step S3, creating a cephalometric model based on a local feature extraction module (CS), a global feature extraction module, a multi-scale feature fusion module and a prediction output module;
[0042] Step S4, training the cephalometric model using the training set, testing the trained cephalometric model using the test set, and verifying the cephalometric model that passes the test using the verification set;
[0043] Step S5: performing cephalometric measurement using the verified cephalometric model.
[0044] Traditional automated detection methods rely on cascaded or multi-stage convolutional neural networks, which have limitations when processing high-resolution X-ray images; in addition, multi-stage detection usually requires additional network structures and data annotations, resulting in increased parameters and manpower input. The cephalometric model (GAHRNet) of the present invention is a single-stage detection that integrates a multi-scale architecture that always maintains high-resolution output, uses more efficient global attention to enhance learning capabilities, and introduces a global context mechanism to further refine the generated features.
[0045] The step S1 is specifically as follows:
[0046] A large number of historical X-ray images of the lateral position of the skull are obtained, each of the historical X-ray images is preprocessed including at least image enhancement and sample expansion, key points of each of the historical X-ray images for cephalometric measurement are annotated and a data set is constructed.
[0047] In the specific implementation, the public cephalometric IEEE ISSBI2015Grand Challenges dataset containing 400 lateral skull X-ray images can be selected. Each image has 19 key points manually annotated by two experienced doctors. The image resolution is 1935×2400 pixels, and the pixel spacing is 0.1 mm.
[0048] The number of key points is 19.
[0049] The step S2 is specifically as follows:
[0050] The dataset was randomly divided into a training set, a test set, and a validation set based on a split ratio of 6:2:2.
[0051] In step S3, the local feature extraction module is composed of four convolutional layers with step sizes of 2, 1, 2, and 1 and a size of 3×3, which are used to extract the local feature from the input historical X-ray image I∈R C×H×W Local features are extracted from the historical X-ray image, and the image size of the historical X-ray image is reduced to reduce the computational cost, and the local features and the reduced-size historical X-ray image are output to the global feature extraction module; wherein C represents the number of channels of the historical X-ray image; H represents the height of the historical X-ray image; and W represents the width of the historical X-ray image.
[0052] In step S3, the global feature extraction module includes two first global attention units (EGAs) for extracting global features from historical X-ray images, and outputting the extracted global features and the input local features to a multi-scale feature fusion module.
[0053] In step S3, the multi-scale feature fusion module includes three second global attention units (EGA) and three first attention guiding units (AGB), and the input end of each of the first attention guiding units is respectively connected to the output end of each of the second global attention units;
[0054] The multi-scale feature fusion module is used to perform multi-scale feature fusion on the input global features and local features to obtain fused features, and output the fused features to the prediction output module.
[0055] In step S3, the prediction output module includes four third global attention units (EGA) and four second attention guiding units (AGB), and the input end of each of the second attention guiding units is respectively connected to the output end of each of the third global attention units;
[0056] The prediction output module is used to output a heat map for cephalometric measurement based on the input fusion features, and the number of the heat maps is the same as the number of the marked key points.
[0057] The global attention unit consists of positional encoding based on convolution (PEC), local aggregation global attention (LAGA) and multi-scale feature fusion (MSFF), and the formula is:
[0058] PEC(I)=I+DWC(I);
[0059]
[0060] MSFF(C t )=C t +F3(F2(F1(C t )));
[0061] Where I represents an X-ray image; PEC(I) represents the position encoding of the X-ray image I; DWC(I) represents the deep convolution operation on the X-ray image I; S i represents the output of convolution; A(S i ) represents the output of the GA module (genetic algorithm) in the global feature extraction module; Q(S i ) represents the linear projection of the weight parameter W_q; K(S i ) represents the linear projection of the weight parameter K_q; T represents transposition; d represents the number of channels; V(S i ) represents the linear projection of the weight parameter V_q; C i represents the output of cross attention; M represents the query vector; C t Represents the output after feature concatenation; MSFF(C t ) indicates that C t The fused features are obtained by multi-scale feature fusion; F1() represents a 1×1 regular convolution with uC channels, where u is the expansion factor; F2() represents a 3×3 separable convolution; and F3() represents a 1×1 regular convolution with C channels.
[0062] PEC is a dynamic positional encoding implemented by using a 2D convolution with a kernel size of k and (k-1) / 2 zero padding. A 3×3 deep convolution (DWC) is used to save computational cost, and residual connections are used to alleviate the gradient vanishing problem. First, multi-scale convolutions (MSCs) are used to locally aggregate input features. MSCs are used to reduce the size of input features to reduce the computational cost of global attention and enhance the model's perception and adaptation to local features. To achieve these goals, MSCs with larger kernel sizes (>3) are set with a step size of half the kernel size. In order to further limit the number of parameters of the model, deep convolution combined with point convolution (PWC) is used to replace conventional convolutions, and the number of output channels of point convolution is only one-fourth of the number of input feature channels. Then global attention is calculated for the output features, and the output of each cross attention (CA) and M are concatenated along the channel dimension and then input to MSFF. The structure of MSFF is similar to the building block in MobileNetV2, with the only difference that the present invention uses GELU activation function instead of ReLU activation function.
[0063] MSFF integrates various types of global features to prevent the model from over-converging on a specific type of global features, thereby enhancing its inductive preference.
[0064] To promote information interaction between features of different resolutions, a dense connection (DC) scheme is designed at the end of each stage of GAHRNet. Since the DC operation also brings more redundancy to the output of each branch, which may affect the feature extraction of the next stage, AGB is used to refine the fused features. Since the purpose of AGB is to evaluate the importance of each position on the fused feature map, it no longer needs to focus on local feature extraction. The global context mechanism (GCM) proposed in the FCVi T model can achieve this goal with a lower computational cost. The AGB of the present invention follows the process of GCM and is defined as follows:
[0065]
[0066] Y=X A +FFN(LN(X A ));
[0067] Among them, X F Indicates the fused features; X Q Indicates LN standardization; Indicates GAP standardization; X V represents the fully connected layer; Y represents the output of AGB; FFN() represents the feedforward neural network; LN() represents the natural logarithm function;
[0068] AGB can be considered as a complement to EGA because it captures global features at a coarser level. However, GCM is only effective at a deeper level, which indicates that its applicability is lower compared to its EGA counterpart.
[0069] In order to extract the features and structural information of historical X-ray images and locate the key points, the present invention adopts a multi-scale network structure that always maintains high resolution in the model structure, which not only reduces the information loss of high-resolution images, but also extracts image features in three stages, and adds multi-scale feature information in each stage, that is, a global feature extraction module (EGA, Effective Global Attention) is designed to fully extract the global features of a large-resolution image, and a multi-scale feature fusion module (AGB, Attention Guided Block) is designed to fully fuse feature information of different scales, and finally 19 heat maps (thermal maps) corresponding to the position information of 19 key points are output.
[0070] The step S4 is specifically as follows:
[0071] Training the cephalometric model using the training set, and continuously adjusting the loss function, optimizer function, and learnable hyperparameters of the cephalometric model during the training process until a hyperparameter combination that enables the cephalometric model to perform best on the training set is found;
[0072] The trained cephalometric model is tested using the test set to determine whether the measurement accuracy is greater than a preset accuracy threshold. If not, the training set is expanded to continue training; if yes, the test passes, and:
[0073] The cephalometric model that has passed the test is verified using the verification set to determine whether the confidence is greater than a preset confidence threshold. If not, the training set is expanded to continue training; if so, the verification is passed and the training ends.
[0074] In a specific implementation, the cephalometric model may be compared and analyzed with some benchmark models based on set evaluation indicators to verify and evaluate the performance of the cephalometric model.
[0075] The step S5 is specifically as follows:
[0076] The latest X-ray image to be measured is obtained, and the latest X-ray image is pre-processed for image enhancement and then input into a verified cephalometric model, and the cephalometric model outputs a corresponding thermal map in real time for cephalometric measurement.
[0077] In summary, the advantages of the present invention are:
[0078] A large number of historical X-ray images of the lateral head are obtained, and a data set is constructed after preprocessing and annotating each historical X-ray image, and the data set is divided into a training set, a test set and a validation set; then a cephalometric model is created based on a local feature extraction module, a global feature extraction module, a multi-scale feature fusion module and a prediction output module, the cephalometric model is trained with the training set, the trained cephalometric model is tested with the test set, the cephalometric model that passes the test is verified with the validation set, and finally cephalometric measurement is performed with the verified cephalometric model; that is, the cephalometric model is trained with historical X-ray images with key points (landmarks) annotated, and the cephalometric model consists of a local feature extraction module, a global feature extraction module, a multi-scale feature fusion module and a prediction output module, which can effectively learn the local and global features of the historical X-ray images, and fuse the local and global features of different scales to output a heat map (the location information of key points), which ultimately greatly improves the accuracy and consistency of cephalometric measurement.
[0079] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. A cephalometric method based on global attention and multi-scale feature fusion, characterized by: The steps include: Step S1, obtaining a large number of historical X-ray images of the lateral position of the skull, and constructing a data set after preprocessing and annotating each of the historical X-ray images; Step S2: dividing the data set into a training set, a test set and a validation set based on a preset segmentation ratio; Step S3, creating a cephalometric model based on the local feature extraction module, the global feature extraction module, the multi-scale feature fusion module and the prediction output module; Step S4, training the cephalometric model using the training set, testing the trained cephalometric model using the test set, and verifying the cephalometric model that passes the test using the verification set; Step S5: performing cephalometric measurement using the verified cephalometric model.
2. The cephalometric method based on global attention and multi-scale feature fusion as claimed in claim 1, characterized in that: The step S1 is specifically as follows: A large number of historical X-ray images of the lateral position of the skull are obtained, each of the historical X-ray images is preprocessed including at least image enhancement and sample expansion, key points of each of the historical X-ray images for cephalometric measurement are annotated and a data set is constructed.
3. The cephalometric method based on global attention and multi-scale feature fusion as claimed in claim 2, characterized in that: The number of key points is 19.
4. The cephalometric method based on global attention and multi-scale feature fusion as claimed in claim 1, characterized in that: The step S2 is specifically as follows: The dataset was randomly divided into a training set, a test set, and a validation set based on a split ratio of 6:2:
2.
5. The cephalometric method based on global attention and multi-scale feature fusion as claimed in claim 1, characterized in that: In step S3, the local feature extraction module is composed of four convolutional layers with step sizes of 2, 1, 2, and 1 and a size of 3×3, which are used to extract the local feature from the input historical X-ray image I∈R C×H×W Local features are extracted from the historical X-ray image, and the image size of the historical X-ray image is reduced, and the local features and the reduced-size historical X-ray image are output to the global feature extraction module; wherein C represents the number of channels of the historical X-ray image; H represents the height of the historical X-ray image; and W represents the width of the historical X-ray image.
6. The cephalometric method based on global attention and multi-scale feature fusion according to claim 1, characterized in that: In step S3, the global feature extraction module includes two first global attention units, which are used to extract global features from historical X-ray images, and output the extracted global features and the input local features to the multi-scale feature fusion module.
7. The cephalometric method based on global attention and multi-scale feature fusion according to claim 1, characterized in that: In step S3, the multi-scale feature fusion module includes three second global attention units and three first attention guiding units, and the input end of each of the first attention guiding units is respectively connected to the output end of each of the second global attention units; The multi-scale feature fusion module is used to perform multi-scale feature fusion on the input global features and local features to obtain fused features, and output the fused features to the prediction output module.
8. The cephalometric method based on global attention and multi-scale feature fusion as claimed in claim 1, characterized in that: In step S3, the prediction output module includes four third global attention units and four second attention guiding units, and the input end of each second attention guiding unit is respectively connected to the output end of each third global attention unit; The prediction output module is used to output a heat map for cephalometric measurement based on the input fusion features, and the number of the heat maps is the same as the number of the marked key points.
9. The cephalometric method based on global attention and multi-scale feature fusion as claimed in claim 1, characterized in that: The step S4 is specifically as follows: Training the cephalometric model using the training set, and continuously adjusting the loss function, optimizer function, and learnable hyperparameters of the cephalometric model during the training process until a hyperparameter combination that enables the cephalometric model to perform best on the training set is found; Using the test set to test the trained cephalometric model, determining whether the measurement accuracy is greater than a preset accuracy threshold, and if not, expanding the training set to continue training; If yes, the test passes and: The cephalometric model that has passed the test is verified using the verification set to determine whether the confidence is greater than a preset confidence threshold. If not, the training set is expanded to continue training; if so, the verification is passed and the training ends.
10. The cephalometric method based on global attention and multi-scale feature fusion according to claim 1, characterized in that: The step S5 is specifically as follows: The latest X-ray image to be measured is obtained, and the latest X-ray image is pre-processed for image enhancement and then input into a verified cephalometric model, and the cephalometric model outputs a corresponding thermal map in real time for cephalometric measurement.