A method for automatic positioning of cephalometric landmarks

Abstract

The present application provides a method for automatically locating landmark points in cephalometric measurement, including: inputting a cephalometric image to be located into a first positioning network model to extract features and obtain a first positioning heat map; inputting the cephalometric image to be located into a digital adjacent point coupling transformation overall optimization network (i.e., a second positioning network model), focusing on the connection between digital adjacent landmark points, and extracting features to obtain a second positioning heat map; inputting the cephalometric image into a medical geometry association point coupling transformation targeted optimization network (i.e., a third positioning network model), focusing on the medical relationship and geometric relationship between landmark points, and extracting features to obtain a third positioning heat map, decoding the first positioning heat map, the second positioning heat map, and the third positioning heat map to obtain an overall optimized landmark point positioning result, and obtaining a final landmark point precise positioning result. The present application realizes automatic annotation of cephalometric images, reduces the annotation burden of doctors, and improves the speed and accuracy of cephalometric measurement.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a method for automatically locating cephalometric landmarks. Background Art

[0002] In clinical diagnosis and treatment, most of the landmarks of cephalometric measurement are still manually positioned by professional doctors based on medical theories and personal clinical diagnostic experience. The positioning process is cumbersome and time-consuming, requiring a lot of human resources and time. Moreover, manual positioning is highly subjective and random. Due to the different training and personal clinical experience backgrounds of different doctors, the results of landmark positioning by doctors vary. Due to the limitations of the human eye and annotation tools, it is difficult to ensure that the results of the same doctor annotating the same X-ray twice are completely consistent. And because subsequent measurement analysis and treatment planning must be carried out on the basis of the precise positioning of each landmark, the human errors caused by the subjectivity and randomness of manual positioning may have a serious impact on the results of cephalometric analysis. Summary of the invention

[0003] In view of this, the purpose of the present application is to propose a method for automatically locating cephalometric landmarks to solve the problem that the human errors caused by the subjectivity and randomness of manual cephalometric positioning may have a serious impact on the cephalometric analysis results.

[0004] Based on the above purpose, the present application provides a method for automatically locating cephalometric landmarks, comprising:

[0005] Inputting the head shadow image to be positioned into a first preset positioning network model to obtain a first positioning heat map;

[0006] Inputting the head shadow image to be positioned into a second preset positioning network model to obtain a second positioning heat map;

[0007] Inputting the head shadow image to be positioned into a third preset positioning network model to obtain a third positioning heat map;

[0008] The first positioning heat map, the second positioning heat map and the third positioning heat map are merged to form a target positioning heat map; the target positioning heat map is decoded to obtain coordinates corresponding to the target positioning heat map.

[0009] Optionally, the training process of the first preset positioning network model includes:

[0010] Input each head shadow image in the training set into the neural network model, iteratively train the neural network model based on a supervised learning training method until the number of iterations reaches an iteration threshold, input each head shadow image in the validation set into the neural network model in each round of training, and select the neural network model with the best positioning performance as the first preset positioning network model; wherein the neural network model is formed by a feature extraction module and a resolution preservation module;

[0011] For each round of iterative training, the following operations are performed:

[0012] Inputting the current head shadow image in the training set into the neural network model to obtain a first predicted heat map of the current head shadow image;

[0013] A loss function is used to calculate a first error value between a real heat map corresponding to the current image and the first predicted heat map, parameters of the neural network model are adjusted according to the first error value, and the neural network model after the adjusted parameters is used as the neural network model for the next round.

[0014] Optionally, the training process of the second preset positioning network model includes:

[0015] Input each head shadow image in the training set into the neural network model, and iteratively train the neural network model using a first coupled transformation algorithm based on a supervised learning training method until the number of iterations reaches an iteration threshold, and input each head shadow image in the validation set into the neural network model in each round of training, and select the neural network model with the best positioning performance as the second training model; wherein the neural network model is formed by a feature extraction module and a resolution preservation module;

[0016] For each round of iterative training, the following operations are performed:

[0017] Acquire all landmarks of the current head shadow image, wherein the landmarks include landmark numbers and landmark positions;

[0018] The current head shadow image is horizontally mirrored and flipped, and the adjacent landmark numbers are numbered to form a data pair, and the landmark positions in the data pair are coupled and transformed to form a first training head shadow image and a real heat map corresponding to the first training head shadow image;

[0019] Inputting the first training head shadow image into the neural network model to obtain a second predicted heat map of the current head shadow image;

[0020] A second error value between a real heat map corresponding to the first training head shadow image and the second predicted heat map is calculated using a loss function, parameters of the neural network model are adjusted according to the second error value, and the neural network model after the adjusted parameters is used as the neural network model for the next round.

[0021] Optionally, the training process of the third preset positioning network model includes:

[0022] Input each head shadow image in the training set into the neural network model, and iteratively train the neural network model using a second coupled transformation algorithm based on a supervised learning training method until the number of iterations reaches an iteration threshold, and input each head shadow image in the validation set into the neural network model in each round of training, and select the neural network model with the best positioning performance as the third preset positioning network model; wherein the neural network model is formed by a feature extraction module and a resolution preservation module;

[0023] For each round of iterative training, the following operations are performed:

[0024] Get all the landmarks of the current head shadow image;

[0025] The current cephalogram image is horizontally mirrored and flipped to determine a set of landmark points that have a medical relationship and a geometric position relationship with the landmark points to be optimized;

[0026] Couple transforming the positions of the marker points in the marker point set according to the medical relationship and the geometric position relationship to form a second training cephalogram image and a real heat map corresponding to the second training cephalogram image;

[0027] Inputting the second training head shadow image into the neural network model to obtain a third predicted heat map of the current head shadow image;

[0028] A third error value between the real thermal map corresponding to the second training head shadow image and the third predicted thermal map is calculated using the loss function, parameters of the neural network model are adjusted according to the third error value, and the neural network model with adjusted parameters is used as the neural network model for the next round.

[0029] Optionally, the process of acquiring the training set includes:

[0030] Training data is obtained, and the training data is preprocessed to obtain a training set.

[0031] Optionally, the training data includes a plurality of initial head shadow images with coordinate marking points;

[0032] The preprocessing of the training data to obtain a training set includes:

[0033] Cutting the portion of the initial head shadow image without the coordinate marking points to obtain a first head shadow image;

[0034] Processing the first head shadow image by using a limited contrast adaptive histogram equalization processing method to obtain a second head shadow image;

[0035] performing augmentation processing on the second head shadow image to obtain a target head shadow image;

[0036] Generate a real heat map according to the coordinate marking points of the target head shadow image;

[0037] The sizes of all the target head shadow images and the real heat maps are adjusted to form a training set.

[0038] Optionally, generating a real heat map according to the coordinate marking points of the target head shadow image includes:

[0039] Obtain pixel position information of the coordinate marking point;

[0040] Based on the pixel position information of the coordinate marking points, a true heat map is generated using an unnormalized Gaussian function.

[0041] Optionally, the unnormalized Gaussian function is:

[0042] Among them, H k (i, j) is the confidence that the pixel position (i, j) of the head shadow image is the kth landmark point, (m k ,n k ) represents the true coordinates of the kth landmark point.

[0043] Optionally, decoding the target positioning heat map includes:

[0044] The target positioning heat map is decoded according to the heat map decoding formula; the heat map decoding formula is:

[0045] in, is the positioning heat map output by the model, Represents the coordinates of the head shadow landmark points obtained by decoding the heat map.

[0046] Optionally, the step of fusing the first positioning heat map, the second positioning heat map and the third positioning heat map according to the preset landmark points to form a target positioning heat map includes:

[0047] According to the preset mark point, the heat map with the clearest positioning corresponding to the preset mark point is selected from the first positioning heat map, the second positioning heat map and the third positioning heat map for fusion to form the target positioning heat map.

[0048] From the above, it can be seen that the present application provides a method for automatically positioning cephalometric landmark points, which inputs the cephalometric image to be positioned into the first preset positioning network model to obtain a first positioning heat map; performs preliminary positioning in the cephalometric image to be positioned through the first preset positioning network model to obtain an initial heat map (first positioning heat map); inputs the cephalometric image to be positioned into the second preset positioning network model to obtain a second positioning heat map; locates some landmark points in the cephalometric image to be positioned through the second preset positioning network model to obtain a second positioning heat map, which can be further used to optimize unclear landmark points in the initial heat map; inputs the cephalometric image of the landmark points to be positioned into the third preset positioning network model to obtain a third positioning heat map; and locates the landmark points in the cephalometric image to be positioned through the third preset positioning network model. Some landmark points in the cephalogram image are located to obtain a third positioning heat map, which can be further used to optimize unclear landmark points in the initial heat map; the first positioning heat map, the second positioning heat map and the third positioning heat map are merged to form a target positioning heat map; by fusing the first positioning heat map, the second positioning heat map and the third positioning heat map, that is, selecting the best positioning point in each positioning heat map as the final landmark point, a target positioning heat map with the highest accuracy is formed, so that the coordinates of the positioning heat map of the cephalogram image are obtained according to the target positioning heat map, thereby realizing automatic annotation of the cephalogram image, reducing the annotation burden of the doctor, improving the speed of cephalogram measurement, solving the problem of the doctor's personal subjectivity or the error caused by the annotation tool in the manual annotation method, and improving the accuracy of cephalogram landmark point positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A schematic diagram of a flow chart of a method for automatically locating cephalometric landmarks according to an embodiment of the present application;

[0051] Figure 2 A flowchart of a training set formation process according to an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the process of training a first preset positioning network model according to an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the architecture of a neural network model according to an embodiment of the present application;

[0054] Figure 5A schematic diagram of the process of training a second preset positioning network model according to an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the process flow of the third preset positioning network model training process of an embodiment of the present application;

[0056] Figure 7 This is a schematic structural diagram of an automatic cephalometric landmark positioning device according to an embodiment of the present application;

[0057] Figure 8 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0059] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0060] In related technologies, X-ray cephalometric analysis refers to the measurement and analysis of the line angles drawn at the dental and maxillary and craniofacial landmarks in the lateral skull X-ray, so as to gain a deeper understanding of the structure of the dental and maxillary and craniofacial soft and hard tissues, and to achieve the inspection and diagnosis of the dental and maxillary and craniofacial structures at the skeletal structure level. It has been widely used in medical tasks such as craniofacial growth and development research, deformity diagnosis, orthodontic treatment, and maxillofacial surgery planning. Cephalometric analysis is the measurement and analysis of lines and angles based on the located landmarks, so accurately locating the landmarks in the lateral skull X-ray is a key step in cephalometric analysis.

[0061] Although X-ray cephalometric technology is quite mature and has been widely used in medicine, in the current domestic clinical diagnosis and treatment, the landmarks of cephalometric measurement still rely on professional doctors to manually locate them based on medical theory and personal clinical diagnosis experience. The positioning process is cumbersome and time-consuming, requiring a lot of human resources and time. In addition, manual positioning is highly subjective and random. Due to the different training and personal clinical experience backgrounds of different doctors, the results of doctors' landmark positioning are different. Due to the limitations of the human eye and annotation tools, it is difficult to ensure that the results of the same doctor's annotation of the same X-ray film twice are completely consistent. Since subsequent measurement analysis and treatment planning must be carried out on the basis of the precise positioning of each landmark, the human errors caused by the subjectivity and randomness of manual positioning may have a serious impact on the results of cephalometric analysis. In view of the above problems of manual positioning of landmarks, it is of great significance to study and develop methods for automatic and precise positioning of X-ray cephalometric landmarks.

[0062] With the continuous development and maturity of artificial intelligence, in the current environment of serious population aging and shortage of medical staff resources, artificial intelligence is gradually being applied in the medical field to reduce the workload of medical staff, improve the work efficiency of the medical system, and alleviate the contradiction between supply and demand of medical resources to a certain extent.

[0063] In response to the above problems, the embodiment of the present application obtains a first positioning heat map by inputting the head shadow image to be positioned into a first preset positioning network model; the landmark points in the head shadow image to be positioned are preliminarily positioned through the first preset positioning network model to obtain an initial heat map (first positioning heat map); the head shadow image to be positioned is input into a second preset positioning network model to obtain a second positioning heat map; some landmark points in the head shadow image to be positioned are positioned through the second preset positioning network model to obtain a second positioning heat map, which can be further used to optimize the unclear landmark points in the initial heat map; the head shadow image of the landmark points to be positioned is input into a third preset positioning network model to obtain a third positioning heat map; the middle part of the head shadow image to be positioned is positioned through the third preset positioning network model The landmark points are positioned to obtain a third positioning heat map, which can be further used to optimize unclear landmark points in the initial heat map; the first positioning heat map, the second positioning heat map and the third positioning heat map are fused to form a target positioning heat map; the first positioning heat map, the second positioning heat map and the third positioning heat map are fused, that is, the best positioning point in each positioning heat map is selected as the final landmark point, so as to form a target positioning heat map with the highest accuracy, and then the coordinates of the landmark points of the cephalogram are obtained according to the target positioning heat map, so as to realize automatic labeling of the cephalogram image, reduce the labeling burden of the doctor, improve the speed of cephalogram measurement, solve the problem of the doctor's personal subjectivity or the error caused by the labeling tool in the manual labeling method, and improve the accuracy of the positioning of the cephalogram landmark points.

[0064] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0065] Figure 1 A schematic flow chart of a method for automatically locating cephalometric landmark points according to an embodiment of the present application is shown.

[0066] Reference Figure 1 The present application embodiment provides a method for automatically locating cephalometric landmarks, comprising the following steps:

[0067] Step S100: input the head shadow image to be positioned into a first preset positioning network model to obtain a first positioning heat map.

[0068] In this step, the first preset positioning network model is formed by iteratively training the neural network model using a supervised learning algorithm through a training set. The landmark points in the head shadow image to be positioned are preliminarily positioned by the first preset positioning network model to obtain a first positioning heat map, that is, an initial positioning heat map, that is, in this step, the head shadow image is first preliminarily positioned, so that the head shadow image to be positioned can be selected according to the initial positioning heat map to select a clearer positioning heat map for the positioning landmark points, so as to obtain a more accurate positioning result of the landmark points.

[0069] Figure 2 Schematic diagram of the training set formation process flow of the embodiment of the present application.

[0070] In some embodiments, reference Figure 2 , the process of obtaining the training set includes:

[0071] Training data is obtained, and the training data is preprocessed to obtain a training set.

[0072] Furthermore, the training data includes a plurality of initial head shadow images with coordinate marking points;

[0073] The preprocessing of the training data to obtain a training set comprises the following steps:

[0074] Step S101 : cropping the portion of the initial head shadow image without coordinate marking points to obtain a first head shadow image.

[0075] In this step, the training data is an initial head image with marked points that has been annotated by medical experts. For example, there are 19 marked points, and the positions of the marked points are: 1. Sella point: the center point of the sella image; 2. Nasal root point: the frontmost point of the frontal suture; 3. Orbital point: the lowest point of the lower orbital margin; 4. Ear point: the mechanical ear point, the uppermost point of the external auditory canal; 5. Upper alveolar seat point: the most concave point of the bone between the anterior nasal spine and the upper alveolar margin; 6. Lower alveolar seat point: the most concave point of the bone between the lower alveolar margin and the prepalpebral point; 7. Anterior chin point: the frontmost point of the bony chin; 8. Submental point: the lowest point of the lower jaw joint contour; 9. Chin vertex: Anterior chin point 1. The midpoint between the lower jaw point and the submental point; 2. The intersection of the angle line of the intersection of the lower jaw support plane and the lower jaw angle plane and the lower jaw angle; 3. The most anterior point of the lower central incisor incisal edge; 4. The most anterior point of the upper central incisor incisal edge; 5. The most anterior point of the upper lip protrusion; 6. The most anterior point of the lower lip protrusion; 7. The most anterior point of the lower lip protrusion; 8. The most anterior point of the lower lip protrusion; 9. The most anterior point of the lower lip protrusion; 10. The most anterior point of the lower jaw angle; 11. The most anterior point of the lower jaw angle; 12. The most anterior point of the upper central incisor incisal edge; 13. The most anterior point of the upper lip protrusion; 14. The most anterior point of the lower lip protrusion; 15. The most anterior point of the nose: the connection point between the columella and the upper lip; 16. The most anterior point of the soft tissue: the most anterior point of the facial soft tissue; 17. The most anterior nasal spine: the tip of the posterior nasal spine; 18. The most anterior nasal spine: the tip of the anterior nasal spine; 19. The most anterior point of the lateral side of the skull base and the most anterior edge of the lower jaw protrusion. The above adjacent numbers are related, and their adjacent points are called medical related points. The operculum area without marker points in the upper half of the cephalogram is cropped, that is, the redundant part of the cephalogram without marker points is cropped, so that the marker points are relatively evenly distributed in the cephalogram, and the sizes of all the cropped cephalograms are kept consistent. After cropping, the cropped images are checked to ensure the integrity of the remaining structures, thereby obtaining a complete first cephalogram.

[0076] Step S102: Process the first head shadow image using a limited contrast adaptive histogram equalization processing method to obtain a second head shadow image.

[0077] In this step, the first head shadow image is processed for image detail by using a contrast-limited adaptive histogram equalization method, thereby balancing the grayscale of different areas in the head shadow image, enhancing the detail structure and features of the head shadow image, enhancing the details of the head shadow image while suppressing noise, which is beneficial to the learning of the neural network.

[0078] Step S103: perform augmentation processing on the second head shadow image to obtain a target head shadow image.

[0079] In this step, since medical experts have annotated the training data, there are fewer samples. Therefore, by performing operations such as random rotation, random scaling, and random removal of landmarks on the second head shadow image, the diversity of the head shadow image samples is increased, which is convenient for the neural network to learn better and improve the robustness and generalization ability of the neural network. Exemplarily, the second head shadow image is randomly rotated, and the rotation angle range is: -45°-45°; the second head shadow image is randomly scaled, and the scaling ratio range is: 0.65-1.35; the second head shadow image is randomly removed from the landmarks, and the landmarks are partially removed. By processing the second head shadow image, a target head shadow image is obtained, and the target head shadow image has diversity, which is convenient for the neural network to learn better.

[0080] Step S104: Generate a real heat map according to the coordinate marking points of the target head shadow image.

[0081] In this step, the pixel position information of the coordinate marking point is obtained; based on the pixel position information of the coordinate marking point, a real heat map is generated using an unnormalized Gaussian function.

[0082] Furthermore, the unnormalized Gaussian function is:

[0083] Among them, H k (i, j) is the confidence that the pixel position (i, j) of the head shadow image is the kth landmark point, (m k ,n k ) represents the true coordinates of the kth landmark point.

[0084] Specifically, based on the pixel position information of the coordinate marker points of the target cephalogram, a real heat map is generated using a Gaussian function with the coordinate points as the center. The real heat map represents the pseudo-probability or confidence that the marker point is located at a certain pixel position, which is used to accurately annotate the cephalogram image, thereby improving the accuracy of locating the cephalogram marker points and providing reliable assistance and reference for assisting doctors in diagnosis and formulating treatment plans.

[0085] Step S105: adjusting the sizes of all the target head shadow images and the real heat maps to form a training set.

[0086] In this step, the target head shadow image and the real heat map have the same size. By adjusting the sizes of the target head shadow image and the real heat map to adapt to the size required by the input of the neural network model, it is more helpful to learn the neural network model.

[0087] Figure 3 A schematic diagram of the process of training the first preset positioning network model is shown.

[0088] In some embodiments, reference Figure 3 , the training process of the first preset positioning network model includes:

[0089] Input each head shadow image in the training set into the neural network model, iteratively train the neural network model based on a supervised learning training method until the number of iterations reaches an iteration threshold, input each head shadow image in the validation set into the neural network model in each round of training, and select the neural network model with the best positioning performance as the first preset positioning network model; wherein the neural network model is formed by a feature extraction module and a resolution preservation module;

[0090] For each round of iterative training, the following operations are performed:

[0091] Step S110, inputting the current head image in the training set into the neural network model to obtain a first predicted heat map of the current head image;

[0092] Step S112: use the loss function to calculate a first error value between the real heat map corresponding to the current image and the first predicted heat map, adjust the parameters of the neural network model according to the first error value, and use the neural network model with adjusted parameters as the neural network model for the next round.

[0093] Specifically, in this embodiment, since the location of the cephalometric landmarks is a delicate task, it requires detailed position information and high-resolution representation; at the same time, the X-ray film for landmark location has high resolution and rich detail information. Therefore, a neural network model is built through a feature extraction model and a resolution preservation model, and while always maintaining high-resolution features, the multi-scale features extracted by the neural network are integrated to complete the location of X-ray cephalometric landmarks.

[0094] Figure 4 A schematic diagram of the architecture of the neural network model is shown.

[0095] Reference Figure 4 The feature extraction model is the feature extraction backbone network module (i.e., module 1). The classic feature extraction network DenseNet121 pre-trained on the ImageNet dataset is used to extract features of different scales of head images. The feature map output by the last layer of each convolution stage of the feature extraction backbone network is consistent with the features of the corresponding size extracted in the high-resolution feature preservation module. Figure 1It serves as the input of the next convolution stage of the high-resolution feature preservation module. The resolution preservation model is the high-resolution feature preservation module (i.e., module 2). While the high-resolution feature preservation module fuses the features of different sizes extracted by the feature extraction backbone network module, it uses a structure similar to HRNet to always maintain the high-resolution features of the image, avoiding the impact of the high-resolution features lost in the downsampling process on the final feature extraction. The high-resolution preservation module does not adopt the serial structure of upsampling to restore high resolution adopted by most high-resolution networks, but adopts a parallel structure. Starting from a high-resolution subnet, it continuously adds subnets from high to low resolution to form multiple stages, and connects multiple subnets in parallel. While always maintaining the high-resolution features of the head shadow image, it uses low-resolution features of the same depth and similar level to enhance the representation of high-resolution features, ensuring that the high-resolution information of the head shadow image is not lost. The parallel multi-resolution subnets repeatedly exchange information and fuse multi-scale features, which is suitable for processing high-resolution X-ray images. According to the characteristics of the landmark location task and the high resolution of the head shadow X-ray, a suitable high-resolution head shadow landmark location neural network model is built. The model can always maintain the high-resolution features of the image, while fusing the extracted multi-scale features to ensure the full extraction and utilization of image features, avoiding the influence of the high-resolution features lost during the downsampling process on the X-ray cephalogram landmark positioning results.

[0096] Furthermore, in this embodiment, in order to help the neural network model to fully learn, a threshold for the number of training iterations and a loss function are preset. The neural network model is trained using supervised training. The high-resolution model extracts features of different scales in the head shadow image, and the output of the last layer of convolution of the neural network model is the heat map predicted by the neural network model. The loss function is calculated to measure the gap between the predicted heat map and the real heat map, and the parameters of the model are continuously updated according to the loss function until the pre-training iteration threshold is reached. Each head shadow image in the validation set is input into the neural network model in each round of training, and the neural network model with the best positioning performance is selected as the first preset positioning network model. Each pixel point is a piece of supervision information, and many supervision samples are added, which is conducive to better training the model and increasing the spatial generalization ability of the model.

[0097] Since different head shadow landmarks have different difficulties in positioning, the loss function of this step adopts the commonly used mean square error (MSE) loss function. In order to better locate landmarks with different positioning difficulties, a weight matrix is ​​set in the loss function to increase the model's attention to points with high positioning difficulties, which is conducive to better training the model and improving the positioning accuracy of the model. The loss function formula is as follows:

[0098]

[0099] Among them, K represents the number of landmarks in the head shadow image, W represents the length of the real heat map, and H represents the width of the real heat map. represents the mean square error loss, h i,j,k Represents the value of the true heat map at position (i, j), Represents the value of the predicted heat map at position (i, j), a k is the weight of the kth head image landmark point.

[0100] Different cephalometric tasks require different numbers of marked landmarks. In order to meet the needs of different medical landmark positioning tasks, the number of landmarks to be positioned and the loss function in this embodiment can be set separately, which is convenient for dynamic adjustment, conducive to better learning of the model, and improves the performance of the X-ray cephalometric positioning network model.

[0101] The real heat map is also a label, for example, a label marked at the corresponding coordinates in the image.

[0102] Step S200: input the head shadow image to be positioned into a second preset positioning network model to obtain a second positioning heat map.

[0103] In this step, the second preset positioning network model is formed by iteratively training the neural network model using the first coupled transformation algorithm through a training set and a training method based on supervised learning. The second preset positioning network model locates the head shadow image to be located to obtain a second positioning heat map. The landmark points of the head shadow image to be located in the second preset positioning network model are different from the landmark points of the head shadow image to be located in the first preset positioning network model. Through the two positioning network models, the landmark points of the head shadow image to be located can be located more accurately. Exemplarily, the second preset positioning network model can be used to optimize the positioning results of the landmark points with poor initial positioning effects. By inputting the head shadow image to be calibrated into the second preset positioning network, according to the marking results of the landmark points of the initial positioning, the calibration results of the head shadow image to be calibrated are optimized by the second preset positioning network to obtain a second positioning heat map, thereby further improving the accuracy of the positioning of the head shadow landmark points.

[0104] Figure 5 A schematic diagram of the process of training the second preset positioning network model is shown.

[0105] In some embodiments, reference Figure 5 , the training process of the second preset positioning network includes:

[0106] Input each head shadow image in the training set into the neural network model, and iteratively train the neural network model using a first coupled transformation algorithm based on a supervised learning training method until the number of iterations reaches an iteration threshold, and input each head shadow image in the validation set into the neural network model in each round of training, and select the neural network model with the best positioning performance as the second training model; wherein the neural network model is formed by a feature extraction module and a resolution preservation module;

[0107] For each round of iterative training, the following operations are performed:

[0108] Step S201, obtaining all the landmarks of the current head image, wherein the landmarks include landmark numbers and landmark positions;

[0109] Step S202, horizontally mirror-flipping the current head shadow image, and composing data pairs with the adjacent landmark numbers, and performing coupling transformation on the landmark positions in the data pairs to form a first training head shadow image and a real heat map corresponding to the first training head shadow image;

[0110] Step S203, inputting the first training head image into the neural network model to obtain a second predicted heat map of the current head image;

[0111] Step S204: use the loss function to calculate the second error value between the real heat map corresponding to the current image and the second predicted heat map, adjust the parameters of the neural network model according to the second error value, and use the neural network model with adjusted parameters as the neural network model for the next round.

[0112] Specifically, in this embodiment, the selection and definition of each marker point is based on a certain medical background and has a specific medical meaning. For example, the marker points with adjacent numbers have certain connections and similarities in terms of medical meaning, and most of the marker points with adjacent numbers have anatomical symmetry in terms of geometric position relationship. Therefore, the image texture structure features near these marker points also have certain connections. It can be seen that at the numbered marker points in the above embodiment, the adjacent numbered marker points have certain connections and similarities.

[0113] Exemplarily, the associated head shadow landmarks are determined according to the proximity of the digital numbers of the landmarks. The head shadow landmark positioning task includes a total of 19 landmarks (see the 19 landmarks exemplified in the above embodiment). The head shadow image and the landmarks are randomly flipped horizontally and symmetrically with a probability of 50%. At the same time, the position of the 19th landmark is kept unchanged, and the remaining 18 landmarks are swapped in pairs according to the principle of numerical proximity. The high-resolution model extracts features of different scales in the head shadow image, and the output of the last convolution layer of the neural network model is the predicted heat map of the neural network model. The loss function is calculated to measure the gap between the predicted heat map and the real heat map, and the parameters of the neural network model are continuously updated according to the loss function until the pre-training iteration number threshold is reached. Each head shadow image in the validation set is input into the neural network model of each round of training, and the neural network model with the best positioning performance is selected as the second preset positioning network model; each pixel point is a supervisory information, and many supervisory samples are added, which is conducive to better training the model and increasing the spatial generalization ability of the model; and the targeted enhancement of the positioning accuracy of the head shadow landmarks is achieved; computing resources are saved and the speed of positioning the head shadow landmarks is improved. Among them, the loss function can adopt the loss function of the first pre-training model; and the preset number of iterative training rounds in this embodiment can adopt the preset number of iterative training rounds set by the first pre-training model.

[0114] Step S300: input the head shadow image to be positioned into a third preset positioning network model to obtain a third positioning heat map.

[0115] In this step, the third preset positioning network model is formed by iteratively training the neural network model using the second coupled transformation algorithm through the training set and the supervised learning-based training method. The third preset positioning network model locates the head shadow image to be located to obtain a third positioning heat map. The landmark points of the head shadow image to be located in the third preset positioning network model are different from the landmark points of the head shadow image to be located in the first preset positioning network model and the second preset positioning network model. Through the three positioning network models, the landmark points of the head shadow image to be located can be located more accurately.

[0116] Exemplarily, the third preset positioning network model can also be used to optimize and screen out the projected landmark points whose positioning effect does not reach the threshold in the second preset positioning network model, wherein the projected landmark points whose positioning effect does not reach the threshold in the second preset positioning network are called landmark points to be optimized. By inputting the corresponding head shadow image to be calibrated into the third preset positioning network model, according to the landmark point positioning result of the second positioning heat map, the landmark point positioning corresponding to the heat map in the second preset positioning network is optimized through the third preset positioning network, a new heat map is generated, and the accuracy of the landmark point positioning is further improved.

[0117] Figure 6A schematic diagram of the training process flow of the third pre-training model is shown.

[0118] In some embodiments, reference Figure 6 , the training process of the third preset positioning network model includes:

[0119] Input each head shadow image in the training set into the neural network model, and iteratively train the neural network model using a second coupled transformation algorithm based on a supervised learning training method until the number of iterations reaches an iteration threshold, and input each head shadow image in the validation set into the neural network model in each round of training, and select the neural network model with the best positioning performance as the third preset positioning network model; wherein the neural network model is formed by a feature extraction model and a resolution preservation model;

[0120] For each round of iterative training, the following operations are performed:

[0121] Step S301, obtaining all landmark points of the current head image;

[0122] Step S302, horizontally mirror-flipping the current cephalogram image, and determining a set of landmark points that have a medical relationship and a geometric position relationship with the landmark points to be optimized;

[0123] Step S303, performing coupling transformation on the positions of the marker points in the marker point set according to the medical relationship and the geometric position relationship, so as to form a second training cephalogram image and a real heat map corresponding to the second training cephalogram image;

[0124] Step S304, inputting the current head image in the training set into the neural network model to obtain a third predicted heat map of the current head image;

[0125] Step S305: Calculate a third error value between the real thermal map corresponding to the second training head shadow image and the third predicted thermal map using a loss function, adjust the parameters of the neural network model according to the third error value, and use the neural network model after adjusting the parameters as the neural network model for the next round.

[0126] Specifically, in this embodiment, based on the medical meaning of the selected head shadow landmark points to be optimized, closely related landmark points are found; at the same time, the geometric position relationship between the landmark points to be optimized and other landmark points is observed to check whether there are landmark points with special position relationships such as equilateral triangles, horizontal symmetry or constant relative position distance; based on the medical relationship and the geometric position relationship, a set of landmark points for coupling training is determined for each landmark point to be optimized. The medical relationship is illustrated by an example, such as: the definition of landmark point No. 10 in the head shadow in medicine is the mandibular angle point, that is, the intersection of the angle dividing line of the intersection of the mandibular ramus plane and the mandibular angle plane and the mandibular angle. The line connecting point No. 10 and point No. 19 is approximately the mandibular ramus plane described in the definition, and the line connecting point No. 10 and point No. 9 is approximately the mandibular angle plane described in the definition. According to the medical definition of point No. 10, point No. 9 and point No. 19 have a certain medical relationship with point No. 10. Exemplarily, the head shadow landmark positioning task includes 19 landmarks in total. The head shadow image and the landmarks are randomly flipped horizontally and symmetrically with a probability of 50%. The landmarks to be tested are coupled and transformed with the landmarks in the corresponding landmark set. The high-resolution model extracts features of different scales in the head shadow image. The output of the last convolution layer of the neural network model is the predicted heat map of the neural network model. The loss function is calculated to measure the gap between the predicted heat map and the real heat map. The parameters of the model are continuously updated according to the loss function until the threshold of the number of pre-training iterations is reached. Each head shadow image in the validation set is input into the neural network model of each round of training, and the neural network model with the best positioning performance is selected as the third preset positioning network model. Each pixel point is a supervisory information, which adds many supervisory samples, is conducive to better training the model, and increases the spatial generalization ability of the model; and realizes the targeted enhancement of the positioning accuracy of the head shadow landmarks; saves computing resources, and improves the speed of positioning the head shadow landmarks. Among them, the loss function can adopt the loss function of the first pre-training model; and the preset number of iterative training rounds in this embodiment can adopt the preset number of iterative training rounds set by the first pre-training model.

[0127] Step S400: According to preset landmarks, the first positioning heat map, the second positioning heat map and the third positioning heat map are merged to form a target positioning heat map.

[0128] In this step, the first positioning heat map, the second positioning heat map and the third positioning heat map are fused according to the preset landmark points to form a target positioning heat map, including:

[0129] According to the preset mark point, the heat map with the clearest positioning corresponding to the preset mark point is selected from the first positioning heat map, the second positioning heat map and the third positioning heat map for fusion to form the target positioning heat map.

[0130] Specifically, it includes selecting the clearest positioning mark point corresponding to the preset mark point in the first positioning heat map, the second positioning heat map and the third positioning heat map as the target positioning mark point according to the preset mark point; forming the target positioning heat map according to the target positioning mark point, so as to decode the target positioning heat map and obtain the coordinates corresponding to the head shadow image, so as to automatically locate the mark point of the head shadow image.

[0131] Step S500: Decode the target positioning heat map to obtain coordinates corresponding to the target positioning heat map.

[0132] In this step, decoding the target positioning heat map includes:

[0133] The target positioning heat map is decoded according to the heat map decoding formula; the heat map decoding formula is:

[0134] in, is the positioning heat map output by the model, Represents the coordinates of the head shadow landmark points obtained by decoding the heat map.

[0135] Specifically, by decoding the target positioning heat map, the accurate positioning coordinates of the preset marker points can be obtained, and the cephalometric image to be tested can be marked according to the coordinates of the cephalometric marker points to obtain a marked cephalometric image.

[0136] Furthermore, according to the coordinates of the cephalometric landmark points, they are restored to the original X-ray cephalogram to complete the positioning and depiction of the cephalometric landmark points, realize the visualization of the positioning of the cephalometric landmark points, enhance readability, and help assist doctors in diagnosis and formulating treatment plans.

[0137] In the present application, automatic and accurate positioning of cephalometric landmarks is achieved based on neural networks and relevant knowledge of cephalometric anatomy. The application of the neural network model can realize batch processing of cephalometric X-rays, reduce the annotation burden of doctors, and improve the speed of cephalometric measurement. At the same time, training the model to achieve automatic annotation can avoid the personal subjectivity of doctors in manual annotation methods or the errors caused by annotation tools, and improve the accuracy of cephalometric landmark positioning.

[0138] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0139] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a device for automatically positioning cephalometric landmark points.

[0141] Figure 7 A schematic structural diagram of a device for automatically locating cephalometric landmark points according to an embodiment of the present application is shown.

[0142] refer to Figure 7 The automatic positioning device for cephalometric landmarks comprises:

[0143] A first training module 401 is used to input the head shadow image to be positioned into a first preset positioning network model to obtain a first positioning heat map;

[0144] The second training module 402 is used to input the head shadow image to be positioned into a second preset positioning network model to obtain a second positioning heat map;

[0145] The third training module 403 is used to input the head shadow image to be positioned into a third preset positioning network model to obtain a third positioning heat map;

[0146] A generating module 404 is used to fuse the first positioning heat map, the second positioning heat map and the third positioning heat map to form a target positioning heat map;

[0147] The decoding module 405 is used to decode the target positioning heat map to obtain coordinates corresponding to the target positioning heat map.

[0148] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0149] The device of the above embodiment is used to implement a corresponding method for automatically locating cephalometric landmarks in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail herein.

[0150] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for automatically locating cephalometric landmark points as described in any of the above-mentioned embodiments is implemented.

[0151] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0152] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0153] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0154] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0155] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0156] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0157] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0158] The electronic device of the above embodiment is used to implement a corresponding method for automatically locating cephalometric landmarks in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail herein.

[0159] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute a method for automatically positioning cephalometric landmark points as described in any of the above embodiments.

[0160] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0161] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute a method for automatically locating cephalometric landmarks as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0162] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0163] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0164] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0165] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for automatically locating cephalometric landmarks, characterized in that: include: Inputting the head image to be positioned into a first preset positioning network model to obtain a first positioning heat map; wherein the first preset positioning network model is formed by a feature extraction module and a resolution preservation module, and the resolution preservation module adopts a structure based on HRNet; Input the head shadow image to be positioned into the second preset positioning network model to obtain a second positioning heat map; wherein the second preset positioning network model is formed by a feature extraction module and a resolution preservation module, and the resolution preservation module adopts a structure based on HRNet; the training process of the second preset positioning network model includes: performing the following operations for each round of iterative training: obtaining all landmark points of the current head shadow image, the landmark points including landmark point numbers and landmark point positions; horizontally mirroring the current head shadow image, and forming data pairs with adjacent landmark point numbers, coupling and transforming the landmark point positions in the data pairs to form the first training head shadow image and the real heat map corresponding to the first training head shadow image; training the second preset positioning network model according to the first training head shadow image and the real heat map corresponding to the first training head shadow image; Input the head shadow image to be positioned into the third preset positioning network model to obtain a third positioning heat map; wherein the third preset positioning network model is formed by a feature extraction module and a resolution preservation module, and the resolution preservation module adopts a structure based on HRNet; the training process of the third preset positioning network model includes: performing the following operations for each round of iterative training: obtaining all landmark points of the current head shadow image; horizontally mirroring the current head shadow image to determine a landmark point set that has a medical relationship and a geometric position relationship with the landmark point to be optimized; coupling and transforming the positions of the landmark points in the landmark point set according to the medical relationship and the geometric position relationship to form a second training head shadow image and a real heat map corresponding to the second training head shadow image; training the third preset positioning network model according to the second training head shadow image and the real heat map corresponding to the second training head shadow image; According to the preset landmarks, the first positioning heat map, the second positioning heat map and the third positioning heat map are merged to form a target positioning heat map; The target positioning heat map is decoded to obtain coordinates corresponding to the target positioning heat map.

2. The method according to claim 1, characterized in that The training process of the first preset positioning network model includes: Input each head shadow image in the training set into the neural network model, iteratively train the neural network model based on a supervised learning training method until the number of iterations reaches an iteration threshold, input each head shadow image in the validation set into the neural network model in each round of training, and select the neural network model as the first preset positioning network model; For each round of iterative training, the following operations are performed: Inputting the current head shadow image in the training set into the neural network model to obtain a first predicted heat map of the current head shadow image; A loss function is used to calculate a first error value between a real heat map corresponding to the current image and the first predicted heat map, parameters of the neural network model are adjusted according to the first error value, and the neural network model after the adjusted parameters is used as the neural network model for the next round.

3. The method according to claim 1, characterized in that The training process of the second preset positioning network model includes: Input each head image in the training set into the neural network model, and iteratively train the neural network model using a first coupled transformation algorithm based on a supervised learning training method until the number of iterations reaches an iteration threshold, input each head image in the validation set into the neural network model in each round of training, and select the neural network model as the second preset positioning network model; The step of training the second preset positioning network model according to the first training head shadow image and the real heat map corresponding to the first training head shadow image includes: For each round of iterative training, the following operations are performed: Inputting the first training head shadow image into the neural network model to obtain a second predicted heat map of the current head shadow image; A second error value between a real heat map corresponding to the first training head shadow image and the second predicted heat map is calculated using a loss function, parameters of the neural network model are adjusted according to the second error value, and the neural network model after the adjusted parameters is used as the neural network model for the next round.

4. The method according to claim 1, characterized in that: The training process of the third preset positioning network model includes: Input each head image in the training set into the neural network model, and iteratively train the neural network model using the second coupled transformation algorithm based on the supervised learning training method until the number of iterations reaches the iteration threshold, and input each head image in the validation set into the neural network model in each round of training, and select the neural network model as the third preset positioning network model; The step of training the third preset positioning network model according to the second training head shadow image and the real heat map corresponding to the second training head shadow image includes: For each round of iterative training, the following operations are performed: Inputting the second training head shadow image into the neural network model to obtain a third predicted heat map of the current head shadow image; A third error value between the real thermal map corresponding to the second training head shadow image and the third predicted thermal map is calculated using the loss function, parameters of the neural network model are adjusted according to the third error value, and the neural network model with adjusted parameters is used as the neural network model for the next round.

5. The method according to claim 1, characterized in that The process of obtaining the training set includes: Training data is obtained, and the training data is preprocessed to obtain a training set.

6. The method according to claim 5, characterized in that The training data includes a plurality of initial head shadow images with coordinate marking points; The preprocessing of the training data to obtain a training set includes: Cutting the portion of the initial head shadow image without the coordinate marking points to obtain a first head shadow image; Processing the first head shadow image by using a limited contrast adaptive histogram equalization processing method to obtain a second head shadow image; performing augmentation processing on the second head shadow image to obtain a target head shadow image; Generate a real heat map according to the coordinate marking points of the target head shadow image; The sizes of all the target head shadow images and the real heat maps are adjusted to form a training set.

7. The method according to claim 6, characterized in that Generating a real heat map according to the coordinate marking points of the target head shadow image includes: Obtain pixel position information of the coordinate marking point; Based on the pixel position information of the coordinate marking points, a true heat map is generated using an unnormalized Gaussian function.

8. The method according to claim 7, characterized in that The unnormalized Gaussian function is: in, H k (i,j) is the head shadow image ( i,j ) The confidence level of the pixel position being the kth landmark point, (m k ,n k ) represents the true coordinates of the kth landmark point.

9. The method according to claim 1, characterized in that: Decoding the target positioning heat map includes: The target positioning heat map is decoded according to the heat map decoding formula; the heat map decoding formula is: in, is the positioning heat map output by the model, Represents the coordinates of the head shadow landmark points obtained by decoding the heat map.

10. The method according to claim 1, characterized in that The step of fusing the first positioning heat map, the second positioning heat map and the third positioning heat map according to the preset landmark points to form a target positioning heat map includes: According to the preset mark point, the heat map with the clearest positioning corresponding to the preset mark point is selected from the first positioning heat map, the second positioning heat map and the third positioning heat map for fusion to form the target positioning heat map.

Images