Skull side position film key point detection method based on feature fusion neural network

Abstract

The invention provides a skull side position film key point detection method based on a feature fusion neural network, and relates to the technical field of medical image processing, and the method comprises the steps: obtaining a skull side position film image and a key point label calibrated by a doctor; segmenting the image and the label into a training set, a verification set and a test set, and preprocessing the training set and the verification set; inputting the training set into a key point detection model, and training the model; inputting the verification set into the model after each period of training is finished, verifying the performance of the model, and obtaining a pre-training model with the optimal performance after all training is finished; inputting the test set into a pre-training model to obtain a key point Gaussian heat map on the skull side film predicted by the model; the Gaussian heat map represents the probability value that each pixel point in the skull side position film is a key point, and the position of the key point on each skull side position film is obtained according to the Gaussian heat map. The detection accuracy of the key points on the skull side position film can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for detecting key points of a lateral skull film based on a feature fusion neural network. Background Art

[0002] Cephalometric analysis is of vital importance in the field of orthodontics. Cephalometric analysis involves the precise identification of key points on the skull on lateral skull radiographs, which provides doctors with important information about the patient's craniofacial condition and significantly affects treatment planning decisions. However, due to the inherent differences in the imaging quality of lateral skull radiographs and the complex anatomical differences between individual patients, it is a huge challenge to obtain reliable key point annotations on lateral skull radiographs. This usually requires the expertise of experienced medical professionals, and even for experienced orthodontists, manually identifying these key points is still a labor-intensive and time-consuming task.

[0003] Fully automatic and accurate methods for lateral cephalogram keypoint detection have been a long-standing and in-demand area. Existing methods for lateral cephalogram keypoint detection often lack standardized processes and well-designed bias reduction processes, which seriously affects their performance. Due to the high cost of image acquisition and the need for extensive manual annotation by professional doctors, the amount of available training data is very limited, resulting in these methods using very shallow networks, which limits the performance of the methods. In addition, the network output resolution of existing methods is usually very small, further introducing quantization errors into the prediction of lateral cephalogram keypoints. Therefore, there is a need to develop feature fusion neural networks that can generate high-resolution prediction heat maps to achieve accurate lateral cephalogram keypoint detection. Summary of the invention

[0004] In view of the above problems in the prior art, the purpose of the present invention is to provide a method for detecting key points on lateral skull films based on a feature fusion neural network, which can train the feature fusion neural network on a small data set to improve the accuracy of key point positioning on lateral skull films.

[0005] In order to achieve the above object, the present invention provides a method for detecting key points of a lateral cranial radiograph based on a feature fusion neural network, the method comprising the following steps:

[0006] Step 1: Obtain the lateral head radiograph image and the key point labels calibrated by the doctor;

[0007] Step 2: Split the images and labels into training sets, validation sets, and test sets, and preprocess the dataset;

[0008] Step 3: Input the training set into the key point detection model to train the model;

[0009] Step 4: After each training cycle, input the validation set into the model to verify the model performance. After all training is completed, the pre-trained model with the best performance is obtained;

[0010] Step 5: Input the test set into the pre-trained model to obtain the Gaussian heat map of key points on the lateral head film predicted by the model;

[0011] Step 6: The Gaussian heat map represents the probability value of each pixel in the lateral skull film being a key point, and the key point position on each lateral skull film is obtained based on the Gaussian heat map.

[0012] In one embodiment, in step 2, preprocessing the data set includes:

[0013] Step 2-1: The lateral head radiograph data set obtained in step 1 is divided into a training set Train with 150 images and labels, a test set Test1 with 150 images and labels, and a test set Test2 with 100 images and labels. The training set Train is used for training, the test set Test1 is used for verification, and the test set Test2 is used for testing;

[0014] Step 2-2: Perform data enhancement on the training set obtained in step 2-1 to obtain an enhanced training set. The data enhancement operations include random flipping, random cropping, and random bounding box transformation.

[0015] Step 2-3: Fill the validation set images and test set images obtained in step 2-1 and the images in the enhanced training set obtained in step 2-2 with zeros to a uniform size scale (1024, 1024, 3) to obtain preprocessed training set images, validation set images, and test set images;

[0016] Step 2-4: Convert the key point labels in the validation set and test set obtained in step 2-1 and the key point labels in the enhanced training set obtained in step 2-2 into the JSON file format required by the model framework input, and obtain the training set JSON label file, the validation set JSON label file, and the test set JSON label file;

[0017] Step 2-5: Read the training set JSON label file obtained in step 2-4, obtain the key point coordinates of each image in the preprocessed training set image obtained in step 2-3, and for each key point on each training set image, create a blank heat map with the same size as the image. The initial values ​​of the blank heat map are all zero, and generate a Gaussian distribution for each key point on the blank heat map. The Gaussian distribution can be expressed as a two-dimensional normal distribution. The formula is as follows:

[0018] ;

[0019] Where G(x, y) is the Gaussian heat map representing the output, (x0, y0) is the coordinate of the key point, (x, y) represents the coordinate of any pixel on the blank heat map, and σ is the standard deviation of the Gaussian distribution. For each key point on each training set image, an independent Gaussian heat map is generated. The Gaussian heat maps generated for all key points on each training set image are superimposed to form a multi-channel heat map as the target of model training, and the target Gaussian heat map of each training set image is obtained.

[0020] In one embodiment, in step 2-5, the value of the standard deviation σ of the Gaussian distribution is 6.

[0021] In one embodiment, in step 3, the key point detection model includes modules such as a backbone network, a neck, a prediction head, and a codec.

[0022] In one embodiment, the backbone network module uses a pre-trained HRNet model as a feature extraction module.

[0023] In one embodiment, the neck module adopts a feature fusion neural network with a feature pyramid network structure, including two feature fusion paths from small-scale feature maps to large-scale feature maps and one feature fusion path from large-scale feature maps to small-scale feature maps; point-by-point convolution and depth-wise convolution are used to fuse feature maps from different scales in the feature fusion process.

[0024] In one embodiment, the prediction head module receives the feature fused feature map from the neck module and maps it into a high-resolution heat map, including encoding the feature fused feature map using point-by-point convolution and pixel rearrangement and mapping it into the heat map.

[0025] In one embodiment, the codec module includes an encoder and a decoder. The encoder encodes the coordinate values ​​of the key point labels in the training set JSON label file obtained in step 2-4 into a 2D Gaussian distribution heat map to generate a target Gaussian heat map as the target of model training; the decoder decodes the predicted Gaussian heat map output by the model into the key point coordinate values ​​on the model input picture. The formula for generating the key point coordinate values ​​by the decoder is:

[0026] ;

[0027] Where H(x, y) represents the value of the heat map at the pixel point (x, y). max ,y max ) is the point with the maximum value in the heat map, which is also the key point position. The heat map H is a two-dimensional matrix with a size of W×H. Each channel represents the probability distribution of a key point. The point with the largest value in the heat map, that is, the maximum response point, is the most likely key point position in the heat map.

[0028] In one embodiment, in step 3, the loss function calculation formula used for training the model is:

[0029] ;

[0030] Where L is the total loss for each training image, H j represents the predicted heat map of the jth key point on each training image, G j represents the corresponding target Gaussian heatmap of the j-th keypoint, N is the total number of keypoints, and ||·||2 represents the L2 norm between the predicted heatmap and the target heatmap.

[0031] In one embodiment, in step 4, the pre-trained model with the best performance is obtained by:

[0032] Step 4-1: After each training cycle, the preprocessed validation set images obtained in step 2-3 are input into the lateral cranial radiograph key point detection model to obtain a predicted Gaussian heat map output by the model with the same scale as the input image;

[0033] Step 4-2: Input the predicted Gaussian heat map obtained in step 4-1 into the decoder part of the codec module to obtain the predicted key point coordinate values ​​on the input image;

[0034] Step 4-3: Compare the predicted key point coordinate values ​​on the input image obtained in step 4-2 with the target key point coordinate values ​​in the validation set JSON label file obtained in step 2-4, and calculate the average distance MRE between the predicted key point coordinate values ​​and the target key point coordinate values; calculate the percentage of predicted key points in the target key points within the range of MRE less than or equal to 2.0 mm, and obtain the successful detection rate SDR2.0 within 2.0 mm;

[0035] Step 4-4: For the value of the successful detection rate SDR2.0 within 2.0 mm on the validation set of each cycle obtained in step 4-3, it will be compared with the value of SDR2.0 obtained in each previous cycle. If the value of SDR2.0 in the current cycle is higher than the previous one, it means that the current model has the best performance. The current optimal model is saved. After the model training is completed, the pre-trained model with the best performance is obtained.

[0036] In one embodiment, step 5 includes: inputting the preprocessed test set images obtained in step 2-3 into the pre-trained model with the best performance obtained in step 4-4, and the model outputs a predicted Gaussian heat map with the same scale as the input image.

[0037] In one embodiment, in step 6, the key point positions on each lateral head film are obtained based on the Gaussian heat map, and the specific method includes: inputting the predicted Gaussian heat map obtained in step 5 into the decoder part of the codec module to generate the predicted key point coordinate values ​​on the test set image.

[0038] The method for detecting key points of lateral cranial radiographs based on feature fusion neural network provided by the present invention has the following beneficial effects:

[0039] 1. The entire lateral cranial radiograph key point detection model is subdivided into modules such as backbone network, neck, prediction head, and codec, realizing a standardized design process and a fully automatic and accurate key point positioning method;

[0040] 2. The neck module uses a feature fusion neural network similar to the feature pyramid network structure, which contains three feature fusion paths and achieves a good balance between the network parameter quantity and key point detection efficiency. Compared with the existing method that uses a very shallow network and limits the performance of the method, the present invention achieves accurate key point detection of the lateral cranial radiograph;

[0041] 3. The present invention can generate a high-resolution heat map with the same scale as the model input image, and predict the coordinate values ​​of key points based on the high-resolution heat map, which further alleviates the quantization error problem and has high computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to better illustrate the technical solution of the present invention, the following provides relevant drawings to assist in the description. The embodiments of the present invention can be more intuitively understood and grasped through these drawings. It should be noted that these drawings are only used for illustration and example, and are not intended to be a restrictive description of the present invention.

[0043] Figure 1 It is a flow chart of a method for detecting key points of a lateral cranial radiograph based on a feature fusion neural network in an embodiment of the present invention;

[0044] Figure 2 Schematic diagram of a key point detection model for a lateral cranial radiograph according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of a feature fusion module in an embodiment of the present invention;

[0046] Figure 4 Schematic diagram of a codec in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to more clearly illustrate the technical content and advantages of the present invention, the present invention is described in detail in conjunction with the following embodiments. The multiple embodiments of the present invention can be understood in conjunction with the relevant drawings, and the drawings are only schematically shown and should not be understood as limiting the present invention.

[0048] This embodiment constructs a standardized lateral head film key point detection process and a carefully designed quantitative deviation reduction process, realizing a fully automatic and accurate lateral head film key point detection method. This embodiment uses a feature fusion neural network to achieve a good balance between the network parameter quantity and key point detection efficiency.

[0049] refer to Figure 1 The basic process of the method for detecting key points of lateral cranial radiographs based on feature fusion neural network provided in this embodiment is as follows:

[0050] Step 1: Obtain the lateral head radiograph image and the key point labels calibrated by the doctor, including: Using the lateral head radiograph image and the calibrated key point label dataset, the dataset collects 400 lateral head radiographs and is divided into 150 training sets, 150 Test1 datasets, and 100 Test2 datasets. The pixel size of each image in the dataset is 1935×2400, and the size of each pixel is 0.1×0.1mm 2 , contains the marking information of 19 key points of the lateral skull radiograph. Each landmark point is manually marked and checked by two doctors, and the average of the two doctors' marking results is taken as the true key point label.

[0051] Step 2: Split the images and labels into training, validation, and test sets, and preprocess the dataset, including:

[0052] Step 2-1: The lateral head radiograph data set obtained in step 1 is divided into a training set Train of 150 images and labels, a test set Test1 of 150 images and labels, and a test set Test2 of 100 images and labels. The training set Train is used for training, the test set Test1 is used for verification, and the test set Test2 is used for testing.

[0053] Step 2-2: Perform data enhancement on the training set obtained in step 2-1 to obtain an enhanced training set. The data enhancement operations include random flipping, random cropping, and random bounding box transformation.

[0054] Step 2-3: The validation set images and test set images obtained in step 2-1 and the images in the enhanced training set obtained in step 2-2 are filled with zeros to a uniform size scale (1024, 1024, 3) to obtain preprocessed training set images, validation set images, and test set images.

[0055] Step 2-4: Convert the key point labels in the validation set and test set obtained in step 2-1 and the key point labels in the enhanced training set obtained in step 2-2 into the JSON file format required by the model framework input to obtain the training set JSON label file, validation set JSON label file, and test set JSON label file.

[0056] Step 2-5: Read the training set JSON label file obtained in step 2-4, obtain the key point coordinates of each image in the preprocessed training set image obtained in step 2-3, and for each key point on each training set image, create a blank heat map with the same size as the image. The initial values ​​of the blank heat map are all zero, and generate a Gaussian distribution for each key point on the blank heat map. The Gaussian distribution can be expressed as a two-dimensional normal distribution. The formula is as follows:

[0057] ;

[0058] Where G(x, y) is the Gaussian heat map representing the output, (x0, y0) is the coordinate of the key point, (x, y) represents the coordinate of any pixel on the blank heat map, and σ is the standard deviation of the Gaussian distribution. For each key point on each training set image, an independent Gaussian heat map is generated. The Gaussian heat maps generated for all key points on each training set image are superimposed to form a multi-channel heat map as the target of model training, and the target Gaussian heat map (1024*1024*19) of each training set image is obtained.

[0059] Step 3: Input the training set into the key point detection model and train the model, including:

[0060] Step 3-1: The preprocessed training set images obtained in step 2-3 are input into the lateral cranial radiograph key point detection model, and after each training cycle, the predicted Gaussian heat map of each training set image at multiple scales output by the model is obtained.

[0061] Step 3-2: Compare the predicted Gaussian heat map of each training set image obtained in step 3-1 with the target Gaussian heat map of each training set image obtained in step 2-5 to calculate the loss, and obtain the loss of each training image in each cycle. The average of the losses on all training set images is used to obtain the total loss of each cycle model training. The loss function calculation formula is:

[0062] ;

[0063] Where L is the total loss for each training image, H j represents the predicted heat map of the jth key point on each training image, G jrepresents the corresponding target Gaussian heatmap of the j-th keypoint, N is the total number of keypoints, and ||·||2 represents the L2 norm between the predicted heatmap and the target heatmap.

[0064] refer to Figure 2 In step 3-1, the head lateral view key point detection model provided in this embodiment includes a backbone network, a neck, a prediction head, a codec and other modules.

[0065] The backbone network module uses the pre-trained HRNet model as the feature extraction module. The HRNet model is divided into HRNet-W32 and HRNet-W48, where 32 and 48 represent the width of the high-resolution sub-networks in the last three stages of the model. We use the HRNet-W48 pre-trained model. The backbone network HRNet contains four parallel sub-networks. The resolution of the feature map output by the first sub-network is one-quarter of the input image. From top to bottom, the resolution of the sub-network gradually decreases to half of the previous sub-network, and the width (number of channels) increases by two times accordingly. These multi-resolution sub-networks are connected in parallel and repeated multi-scale fusion is performed at the same time. Each resolution to low-resolution expression will share information from other parallel high-resolution to low-resolution expressions, resulting in a richer resolution expression. After the training set image (1024*1024*3) is input into the key point detection model of the lateral head film, the features will be extracted through the backbone network HRNet first to obtain the feature maps of F2 to F5. Fi represents a feature map with a resolution of H×W divided by the power of 2. The resolution of feature map F2 is one-fourth of the input image resolution, and the resolution of feature map F5 is one-thirty-second of the input image. The number of channels of feature maps F2 to F5 extracted by the backbone network HRNet is 48, 96, 192, and 384, which are then fused in the neck module.

[0066] The neck module adopts a feature fusion neural network with a similar feature pyramid network structure, which includes two feature fusion paths from small-scale feature maps to large-scale feature maps (M5 to M2 and G5 to G2) and one feature fusion path from large-scale feature maps to small-scale feature maps (W2 to W5).

[0067] refer to Figure 3 In order to efficiently fuse features from multiple levels, the feature fusion module uses a series of separable convolutions to optimize model parameters. The feature fusion module first connects the feature maps F and M by passing them through convolution blocks and upsampling operations, and then passes the connected feature maps through point-by-point convolution blocks. Then, two consecutive modules are applied, which include depthwise convolution, batch normalization BN, point convolution, and GELU activation function. Finally, a new feature map M after feature fusion is generated through point convolution operations.

[0068] In this embodiment, the feature map F5 (32*32*384) extracted by the backbone network HRNet is first processed by convolution to obtain the feature map M5 (32*32*192). Then, the feature fusion operation is performed. First, the feature map M5 is upsampled and interpolated to the resolution of the previous layer (F4) to ensure size matching. At this time, the feature map M5 (64*64*192) is convolved on the feature map F4 (64*64*192) of the previous scale, and the number of channels remains unchanged. Then, the interpolated feature map M5 is connected with the convolved feature map F4 by channel to obtain the connected feature map (64*64*384). Subsequently, the connected feature map is subjected to a point convolution operation to reduce the number of channels to half (64*64*192), and then two consecutive fusion modules are applied. Finally, a new feature map M4 (64*64*192) after the fusion of M5 and F4 features is generated through a point convolution operation.

[0069] In this embodiment, the feature fusion process of the subsequent feature maps M4 and F3, M3 and F2 is the same as that of M5 and F4, and finally the fused feature map M2 (256*256*48) is output, and then the feature fusion path from the large-scale feature map to the small-scale feature map (W2 to W5) is performed. First, the feature map M2 is convolved to obtain the feature map W2 (256*256*96), and then the feature fusion operation is performed, the feature map W2 is downsampled, and the feature map is interpolated, and the size is reduced to the resolution of the next layer (M3), at this time the feature map W2 (128*128*96). The next scale feature map M3 (128*128*96) is convolved, and the number of channels remains unchanged, and then the interpolated feature map W2 is connected with the convolved feature map M3 by channel to obtain the connected feature map (128*128*192). Subsequently, a point convolution operation is performed on the connected feature map to reduce the number of channels to half (128*128*96), and then two consecutive fusion modules are applied. Finally, a point convolution operation is performed to generate a new feature map W3 (128*128*96) after the fusion of W2 and M3 features.

[0070] The feature fusion process from large-scale feature maps to small-scale feature maps (W2 to W5) is similar to that from M5 to M2, except that the resolution is decreasing and the number of channels is gradually increasing, while in the feature fusion process from small-scale feature maps to large-scale feature maps (M5 to M2), the resolution of the feature map after each feature fusion is increasing and the number of channels is gradually decreasing. Finally, the second feature fusion process from small-scale feature maps to large-scale feature maps (G5 to G2) is performed. This process is the same as the method from M5 to M2, and finally the feature map G2 of multi-scale feature fusion is obtained.

[0071] refer to Figure 4, the prediction head module receives the feature map after feature fusion from the neck module and maps it to a high-resolution heat map, including encoding the feature map after feature fusion and mapping it to the heat map using point convolution and pixel rearrangement. First, the feature map G2 obtained by the neck module is point convolved, the number of channels is adjusted to the number of embedding channels, the key point embedding representation is generated, and then the GELU activation function is performed. Finally, the feature map H is generated through the convolution layer, and H is upsampled using the pixel rearrangement operation to improve the spatial resolution, and the final heat map H2 (1024*1024*19) with the same resolution as the input image is obtained.

[0072] Step 4: After each training cycle, input the validation set into the model to verify the model performance. After all the training is completed, the pre-trained model with the best performance is obtained, which includes:

[0073] Step 4-1: After each training cycle, the preprocessed validation set images obtained in step 2-3 are input into the lateral cranial radiograph key point detection model to obtain a predicted Gaussian heat map output by the model with the same scale as the input image;

[0074] Step 4-2: Input the predicted Gaussian heat map obtained in step 4-1 into the decoder part of the codec module to obtain the predicted key point coordinate values ​​on the input image;

[0075] Step 4-3: Compare the predicted key point coordinate values ​​on the input image obtained in step 4-2 with the target key point coordinate values ​​in the validation set JSON label file obtained in step 2-4, and calculate the average distance MRE between the predicted key point coordinate values ​​and the target key point coordinate values; calculate the percentage of predicted key points in the target key points within the range of MRE less than or equal to 2.0 mm, and obtain the successful detection rate SDR2.0 within 2.0 mm;

[0076] Step 4-4: For the value of the successful detection rate SDR2.0 within 2.0 mm on the validation set of each cycle obtained in step 4-3, it will be compared with the value of SDR2.0 obtained in each previous cycle. If the value of SDR2.0 in the current cycle is higher than the previous one, it means that the current model has the best performance. The current optimal model is saved. After the model training is completed, the pre-trained model with the best performance is obtained.

[0077] In this embodiment, the codec module used in step 4-2 includes an encoder and a decoder. The encoder preprocesses the training set JSON label file obtained in step 2-4, encodes the coordinate values ​​of the key point labels in the training set into a 2D Gaussian distribution heat map, and generates a target Gaussian heat map as the target of model training; the decoder decodes the predicted Gaussian heat map output by the model into the key point coordinate values ​​on the model input picture. The formula for generating the key point coordinate values ​​by the decoder is:

[0078] ;

[0079] Where H(x, y) represents the value of the heat map at the pixel point (x, y). max ,y max ) is the point with the maximum value in the heat map, which is also the key point position. The heat map H is a two-dimensional matrix with a size of W×H. Each channel represents the probability distribution of a key point. The point with the largest value in the heat map, that is, the maximum response point, is the most likely key point position in the heat map.

[0080] Step 5: Input the test set into the pre-trained model to obtain the Gaussian heat map of key points on the lateral skull film predicted by the model, specifically including: inputting the pre-processed test set images obtained in steps 2-3 into the pre-trained model with the best performance obtained in step 4-4, and the model outputs a predicted Gaussian heat map with the same scale as the input image.

[0081] Step 6: The Gaussian heat map represents the probability value of each pixel in the lateral skull film being a key point. The key point position on each lateral skull film is obtained according to the Gaussian heat map, which specifically includes: inputting the predicted Gaussian heat map obtained in step 5 into the decoder part of the codec module to generate the predicted key point coordinate value on the test set image.

[0082] The method for detecting key points of lateral cranial radiographs based on feature fusion neural network provided in this embodiment focuses on solving the problem of quantization deviation and designing a unified process for detecting key points of lateral cranial radiographs. The present invention adopts a design for generating high-resolution heat maps, which further alleviates the deviation problem, and performs heat map prediction based on high-resolution heat maps, thereby improving the accuracy of key point detection.

[0083] The embodiments of the present invention are described in detail above to help understand the technical solutions and effects of the present invention. However, these specific embodiments should not be construed as limiting the scope of the present invention. For those of ordinary skill in the art, various forms of modification, deformation or equivalent replacement can be made without departing from the spirit and technical solutions of the present invention. These changes should all be deemed to be included in the protection scope of the present invention.

Claims

1. A method for detecting key points of lateral cranial radiographs based on feature fusion neural network, characterized in that: include: Step 1: Obtain the lateral head radiograph image and the key point labels calibrated by the doctor; Step 2: Split the images and labels into training sets, validation sets, and test sets, and preprocess the dataset; Step 3: Input the training set into the key point detection model to train the model; Step 4: After each training cycle, input the validation set into the model to verify the model performance. After all training is completed, the pre-trained model with the best performance is obtained; Step 5: Input the test set into the pre-trained model to obtain the Gaussian heat map of key points on the lateral head film predicted by the model; Step 6: The Gaussian heat map represents the probability value of each pixel in the lateral skull film being a key point, and the key point position on each lateral skull film is obtained based on the Gaussian heat map.

2. The method for detecting key points of lateral cranial radiographs based on feature fusion neural network according to claim 1, characterized in that: In step 1, the step of obtaining the lateral skull radiograph image and the key point labels calibrated by the doctor includes: using the lateral skull radiograph image and the calibrated key point label data set.

3. The method for detecting key points of lateral cranial radiographs based on feature fusion neural network according to claim 1, characterized in that: In step 2, the images and labels are divided into a training set, a validation set, and a test set, and the data set is preprocessed, including: Step 2-1: The lateral head radiograph data set obtained in step 1 is divided into a training set Train with 150 images and labels, a test set Test1 with 150 images and labels, and a test set Test2 with 100 images and labels. The training set Train is used for training, the test set Test1 is used for verification, and the test set Test2 is used for testing; Step 2-2: Perform data enhancement on the training set obtained in step 2-1 to obtain an enhanced training set. The data enhancement operations include random flipping, random cropping, and random bounding box transformation. Step 2-3: Fill the validation set images and test set images obtained in step 2-1 and the images in the enhanced training set obtained in step 2-2 with zeros to a uniform size scale (1024, 1024, 3) to obtain preprocessed training set images, validation set images, and test set images; Step 2-4: Convert the key point labels in the validation set and test set obtained in step 2-1 and the key point labels in the enhanced training set obtained in step 2-2 into the JSON file format required by the model framework input, and obtain the training set JSON label file, the validation set JSON label file, and the test set JSON label file; Step 2-5: Read the training set JSON label file obtained in step 2-4, obtain the key point coordinates of each image in the preprocessed training set image obtained in step 2-3, and for each key point on each training set image, create a blank heat map with the same size as the image. The initial values ​​of the blank heat map are all zero, and generate a Gaussian distribution for each key point on the blank heat map. The Gaussian distribution can be expressed as a two-dimensional normal distribution. The formula is as follows: ; Where G(x, y) is the Gaussian heat map representing the output, (x0, y0) is the coordinate of the key point, (x, y) represents the coordinate of any pixel on the blank heat map, and σ is the standard deviation of the Gaussian distribution. For each key point on each training set image, an independent Gaussian heat map is generated. The Gaussian heat maps generated for all key points on each training set image are superimposed to form a multi-channel heat map as the target of model training, and the target Gaussian heat map of each training set image is obtained.

4. The method for detecting key points of lateral cranial radiographs based on feature fusion neural network according to claim 1, characterized in that: In step 3, the training set is input into the key point detection model to train the model, including: Step 3-1: Input the preprocessed training set images obtained in step 2-3 into the lateral cranial radiograph key point detection model, and obtain the predicted Gaussian heat map of each training set image at multiple scales output by the model after each training cycle. Step 3-2: Compare the predicted Gaussian heat map of each training set image obtained in step 3-1 with the target Gaussian heat map of each training set image obtained in step 2-5 to calculate the loss, and obtain the loss of each training image in each cycle. The average of the losses on all training set images is used to obtain the total loss of each cycle model training. The loss function calculation formula is: ; Where L is the total loss for each training image, H j represents the predicted heat map of the jth key point on each training image, G j represents the corresponding target Gaussian heatmap of the j-th keypoint, N is the total number of keypoints, and ||·||2 represents the L2 norm between the predicted heatmap and the target heatmap.

5. The method for detecting key points of lateral cranial radiographs based on feature fusion neural network according to claim 4, characterized in that: The key point detection model for the lateral cranial film in step 3-1 includes: the model is composed of a backbone network, a neck, a prediction head, a codec and other modules; the backbone network module adopts a pre-trained HRNet model as a feature extraction module to extract a multi-scale feature map of the model input image; the neck module adopts a feature fusion network with a similar feature pyramid network structure, including two feature fusion paths from a small-scale feature map to a large-scale feature map and a feature fusion path from a large-scale feature map to a small-scale feature map; the prediction head module receives the feature map after feature fusion from the neck module and maps it to a high-resolution heat map; the codec module includes an encoder and a decoder part, the encoder encodes the coordinate values ​​of the key point labels in the training set into a 2D Gaussian distribution heat map, generates a target Gaussian heat map as the target of model training; the decoder decodes the predicted Gaussian heat map output by the model into the coordinate values ​​of the key points on the model input image.

6. The method for detecting key points of lateral cranial radiographs based on feature fusion neural network according to claim 1, characterized in that: In step 4, after each training cycle, the validation set is input into the model to verify the model performance. After all the training is completed, the pre-trained model with the best performance is obtained, including: Step 4-1: After each training cycle, the preprocessed validation set images obtained in step 2-3 are input into the lateral cranial radiograph key point detection model to obtain a predicted Gaussian heat map output by the model with the same scale as the input image; Step 4-2: Input the predicted Gaussian heat map obtained in step 4-1 into the decoder part of the codec module to obtain the predicted key point coordinate values ​​on the input image; Step 4-3: Compare the predicted key point coordinate values ​​on the input image obtained in step 4-2 with the target key point coordinate values ​​in the validation set JSON label file obtained in step 2-4, and calculate the average distance MRE between the predicted key point coordinate values ​​and the target key point coordinate values; calculate the percentage of predicted key points in the target key points within the range of MRE less than or equal to 2.0 mm, and obtain the successful detection rate SDR2.0 within 2.0 mm; Step 4-4: For the value of the successful detection rate SDR2.0 within 2.0 mm on the validation set of each cycle obtained in step 4-3, it will be compared with the value of SDR2.0 obtained in each previous cycle. If the value of SDR2.0 in the current cycle is higher than the previous one, it means that the current model has the best performance. The current optimal model is saved. After the model training is completed, the pre-trained model with the best performance is obtained.

7. The method for detecting key points of lateral cranial radiographs based on feature fusion neural network according to claim 1, characterized in that: In step 5, the test set is input into the pre-trained model to obtain a Gaussian heat map of key points on the lateral skull film predicted by the model, including: for the pre-trained model with the best performance obtained in step 4-4, the pre-processed test set image obtained in step 2-3 is input into the pre-trained model for testing, and the model outputs a predicted Gaussian heat map with the same scale as the input image.

8. The method for detecting key points of lateral cranial radiographs based on feature fusion neural network according to claim 1, characterized in that: In step 6, the Gaussian heat map represents the probability value of each pixel in the lateral skull film being a key point, and the key point position on each lateral skull film is obtained according to the Gaussian heat map, including: inputting the predicted Gaussian heat map obtained in step 5 into the decoder part of the codec module to generate the predicted key point coordinate value on the test set image.

Images