Method and system for locating key points of hand bone x-ray medical image

Abstract

This invention belongs to the field of intelligent medical image recognition technology, specifically relating to a method and system for locating key points in hand bone X-ray medical images. The method includes the following steps: S1, collecting and preprocessing hand bone X-ray images and corresponding key point annotation data; S2, constructing a key point localization model for hand bone X-ray medical images; S3, training and validating the model using an auxiliary loss function; S4, performing end-to-end prediction using the validated model. This invention features high localization accuracy and the ability to locate a sufficient number of hand bone key points at once.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical image recognition technology, specifically relating to a method and system for locating key points in medical images of hand bone X-ray films. Background Technology

[0002] Automatic detection of key points in medical images is fundamental to medical image analysis applications and has been widely used in tasks such as medical image registration, tissue segmentation, and parameter measurement.

[0003] In bone age assessment tasks, current techniques such as the TW method and GP method require processing of left-hand X-ray images: the first step is to accurately segment the ossification center image from the original image, and the second step is to detect bone age using the segmented ossification center image. The accuracy of the segmented ossification center image in the first step and the presence of hand bone misalignment in the image have a significant impact on the bone age detection in the second step.

[0004] To obtain an image of the ossification center, one approach is to directly acquire the bounding box. Current techniques use the Faster-RCNN object detection algorithm to directly locate the bounding box from the original X-ray image. However, even after post-processing filtering, this direct localization method cannot accurately distinguish between different fingers (for example, the bounding box on the little finger might be located on the ring finger). This segmentation method cannot effectively control the size of the bounding box. Furthermore, because the bounding boxes obtained by the Faster-RCNN algorithm are always vertical, the hand bones within the bounding boxes are often skewed. In addition, some existing techniques employ a two-step approach: first, obtain a hand contour image from the original X-ray image, then extract four images of the 1st, 3rd, and 5th fingers (where the ossification center is located) and the wrist; then, train four Faster-RCNN models using these four images to locate the bounding boxes. The drawback of this method is that it uses multiple cascaded models, resulting in complex and slow localization calculations, and it can only locate specific fingers, not directly the finger joints.

[0005] The second method involves first locating key points of the hand bones in the X-ray image, from which the bounding box can be obtained. For example, existing fast finger bone key point localization methods can only provide nine peak and valley points in the binary image of the hand shape, and the number and accuracy of key points are insufficient for locating the bounding box. Existing technologies have also proposed near-contour point localization methods for hand bones, achieving rapid segmentation of multiple RUS bone blocks. Combined with key point localization techniques such as Active Shape Model (ASM), Active Appearance Model (AAM), and Local Constraint Model (CLM), this method achieves the localization of multiple key points of the hand bones. The number and quality of the located key points are significantly improved compared to the method of directly locating bounding boxes from the original X-ray image using the Faster-RCNN object detection algorithm, but the model's localization accuracy remains insufficient.

[0006] Therefore, to obtain accurate and unbiased images of the ossification center, it is necessary to design a method with sufficient accuracy that can locate a sufficient number of key points of the hand bones at once.

[0007] For example, Chinese patent document CN201710975940.6 describes an automatic extraction method for hand bone regions of interest (ROI) from X-ray images based on deep neural networks. This method involves: removing text embedded in the black background on both sides of the original hand bone X-ray image; uniformly brightening and denoising the original hand bone X-ray image; sampling and training model M1 to obtain a hand bone X-ray image Output2 without text; normalizing the size of Output2 to obtain Output3; sampling and training model M2 to judge the hand bones, background, and the intersection of hand bones and background in Output3; using model M2 to perform image sliding window judgment in Output3, and obtaining a hand bone marker mapping map Output4 based on the judgment value; obtaining an image Output5 containing only hand bones based on Output3 and Output4; and optimizing Output5 to obtain the final hand bone ROI. While this method can automatically extract hand bone ROI from X-ray images, its shortcomings include the inability to locate a sufficient number of key hand bone points and limitations in positioning accuracy. Summary of the Invention

[0008] This invention aims to overcome the problems of existing bone age detection technologies, such as the inability to directly locate finger joints and insufficient positioning accuracy. It provides a method and system for locating key points in hand bone X-ray medical images with high positioning accuracy and the ability to locate a sufficient number of key hand bone points at once.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0010] The method for locating key points in medical images of hand bones on X-ray films includes the following steps:

[0011] S1, Collection and preprocessing of hand bone X-ray images and corresponding key point annotation data:

[0012] Collect X-ray images of hand bones and corresponding key point annotation data, and randomly divide them into training and validation sets;

[0013] S2, Construct a key point localization model for medical images of hand bone X-ray films;

[0014] S3, Train the model using the auxiliary loss function, and validate the model:

[0015] Using an auxiliary loss function, and based on the training and validation sets obtained in step S1, the key point localization model for hand bone X-ray medical images is trained and validated.

[0016] S4, using a validated model for end-to-end prediction:

[0017] The trained and validated key point localization model for hand bone X-ray medical images is used to predict key points in hand bone X-ray medical images.

[0018] Preferably, step S1 includes the following steps:

[0019] S11, normalize the pixel values ​​of the collected hand bone X-ray images to [0,1]; fill the square with 0 to the left or down, and denot the size of the i-th image after filling as r. 0i ×r 0i ; Scale to r×r size using bilinear interpolation;

[0020] S12, assuming each image has m key points, represented as... Wherein, the coordinates of the m-th key point (x m ,y m R is the real number field; the nth key point P n =[x n y n ], where x n The x-axis represents the pixel distance of the keypoint relative to the top-left corner of the image, and the y-axis represents the distance from the keypoint in the x-axis relative to the top-left corner of the image. n This represents the pixel distance of the keypoint in the y-direction relative to the top-left corner of the image;

[0021] Normalize the key points into

[0022] As a preferred embodiment, the key point localization model for hand bone X-ray medical images includes a bone feature extraction network, a Transformer encoding layer, and a progressive key point localization Transformer decoding layer. The model construction process includes the following steps:

[0023] S21, Construct the backbone feature extraction network:

[0024] The backbone feature extraction network adopts a basic feature extraction network, specifically ResNet, DenseNet, or Swin Transformer, to convert the input hand bone X-ray medical image into a level-one feature vector F1.

[0025] S22, Construct the Transformer encoding layer:

[0026] The Transformer encoding layer includes a feature vector position encoding layer and stacked N layers. EnThe first layer of Transformer encoders; the feature vector position encoding layer is used to perform sinusoidal position encoding on the first-level feature vector and add it to the first-level feature vector. The result of the addition is used as the input of the first-level Transformer encoder. The inputs of the remaining Transformer encoders are the outputs of the previous Transformer encoder. After passing through the Transformer encoding layer, the first-level feature vector F1 is transformed into the second-level feature vector F2.

[0027] The formula for self-attention in a Transformer encoder:

[0028]

[0029] Q = K = V = F1.

[0030] Where D is the feature dimension of the level-one feature vector F1;

[0031] S23, Constructing a progressive keypoint localization Transformer decoding layer:

[0032] Progressive keypoint localization Transformer decoding layer, consisting of N DL The system consists of N progressive keypoint correction layers, with each progressive keypoint correction layer containing N layers. DD The Transformer decoder has the same structure for all layers. Its inputs are level 2 feature vectors, decoder embeddings, and keypoints to be corrected; the output is the updated decoder embedding. The level 2 feature vectors are output from the Transformer encoding layer in step S22 and remain unchanged in the progressive keypoint localization Transformer decoding layer. The initial decoder embedding is initialized to 0 and is not trainable. The initial keypoints to be corrected are initialized a priori from the training set and are not trainable. At the end of each progressive keypoint correction layer, the updated decoder embedding is used to calculate the proposed offset of a keypoint to be corrected using a multilayer perceptron to update the keypoint. After N... DL After the progressive key point correction layer is applied, the key points to be corrected are finally corrected into high-precision key point output.

[0033] P n =(x n ,y n ) represents the nth key point. The DropOut and residual join are omitted in the following formulas;

[0034] For any Transformer decoder layer, PE: Sine wave position encoding:

[0035]

[0036] PE(P n ) = PE(x n ,y n ) = CAT(PE(x n ),PE(y n ));

[0037] Where CAT represents the Concatenate function; subscripts 2i and 2i+1 represent the positions in the encoding vector; T is an introduced hyperparameter, T>1, the larger T is, the flatter the positional encoding; T 2i / D This represents the exponentiation of T;

[0038] The kernel of the Transformer decoder consists of a self-attention computation and a cross-attention computation, with the same formula for the attention computation as that in the Transformer encoder.

[0039] Parameters for self-attention calculation:

[0040] Q = K = C + MLP (s) (PE(P)), V=C;

[0041] Parameters for cross-attention calculation:

[0042] Q = C′ + MLP (c) (PE(P)),

[0043] K = F² + PE(P), V = F²;

[0044] Among them, MLP: In a multilayer perceptron, the two MLPs in self-attention computation and cross-attention computation share weights with all Transformer decoders respectively. These are the decoder embeddings before and after self-attention computation, respectively;

[0045] For the j-th level progressive keypoint correction layer, the formula for correcting keypoints is:

[0046]

[0047] Among them, P j-in For the j-th level input key points to be corrected, P j-out The key points to be corrected in the output of level j; MLP (fix) : For multilayer perceptron, it is responsible for embedding the decoder into the correction of the key points to be corrected; t is an introduced hyperparameter, t>1, which is used to weaken the influence of higher-level correction layers on key points, improve model stability, and achieve progressive correction.

[0048] Final output key point P final .

[0049] Preferably, step S3 includes the following steps:

[0050] S31: Input the training data into the model, and the model directly outputs the key points. With tag value Calculate loss = MAE(P) final ,P label ); Represents a 2xM dimensional matrix in the real number field;

[0051] S32, enabling the auxiliary loss function improves model convergence speed and final performance:

[0052] Record the key points after each progressive key point correction layer. This represents the keypoints after correction by the i-th progressive keypoint correction layer, calculated as follows: Backpropagation is performed based on the calculated loss function to update the parameters;

[0053] S33: After each round of training on all training data, the model performance is validated.

[0054] Input the validation data into the model, and the model will directly output the key points. With tag value Calculate the average error criterion = MAE(P) for all keypoints in a single image. final ,P label The average error of all validation data is calculated and used as the evaluation index, and the model parameters are saved.

[0055] S34. After training for k rounds, the model parameters with the smallest average error index are used as the final parameters of the validated model.

[0056] Preferably, step S4 includes the following steps:

[0057] S41, normalize the pixel values ​​of the hand bone X-ray image to be predicted to [0,1]; fill the image with 0 to the left or down to form a square, and record the size of the filled image as r0×r0; use bilinear interpolation to scale it to size r×r;

[0058] S42, Input the preprocessed image into the validated hand bone X-ray medical image key point localization model, and the model outputs key points. The pixel coordinates P of the desired key point relative to the top-left corner of the image are obtained. pix =P final ×r0.

[0059] This invention also provides a key point localization system for medical images of hand bone X-ray films, comprising:

[0060] The collection and preprocessing module is used to collect hand bone X-ray images and corresponding key point annotation data, and randomly divide them into training and validation sets;

[0061] The model building module is used to build a key point localization model for medical images of hand bone X-ray films;

[0062] The training and validation module is used to train and validate the key point localization model of hand bone X-ray medical images using an auxiliary loss function and based on the obtained training and validation sets.

[0063] The prediction output module is used to generate a trained and validated key point localization model for hand bone X-ray medical images, which is then used to predict key points in hand bone X-ray medical images.

[0064] Preferably, the collection and preprocessing module is as follows:

[0065] The pixel values ​​of the collected hand bone X-ray images are normalized to [0,1]; they are then filled with 0s to the left or down to form a square, and the size of the i-th image after filling is denoted as r. 0i ×r 0i ; Scale to r×r size using bilinear interpolation;

[0066] Let each image have m key points, represented as... Wherein, the coordinates of the m-th key point (x m ,y m R is the real number field; the nth key point P n =[x n y n ], where x n The x-axis represents the pixel distance of the keypoint relative to the top-left corner of the image, and the y-axis represents the distance from the keypoint in the x-axis relative to the top-left corner of the image. n This represents the pixel distance of the keypoint in the y-direction relative to the top-left corner of the image;

[0067] Normalize the key points into

[0068] Preferably, the model building module is as follows:

[0069] Constructing a backbone feature extraction network:

[0070] The backbone feature extraction network adopts a basic feature extraction network, specifically ResNet, DenseNet, or Swin Transformer, to convert the input hand bone X-ray medical image into a level-one feature vector F1.

[0071] Building the Transformer encoding layer:

[0072] The Transformer encoding layer includes a feature vector position encoding layer and stacked N layers. En The first layer of Transformer encoders; the feature vector position encoding layer is used to perform sinusoidal position encoding on the first-level feature vector and add it to the first-level feature vector. The result of the addition is used as the input of the first-level Transformer encoder. The inputs of the remaining Transformer encoders are the outputs of the previous Transformer encoder. After passing through the Transformer encoding layer, the first-level feature vector F1 is transformed into the second-level feature vector F2.

[0073] The formula for self-attention in a Transformer encoder:

[0074]

[0075] Q = K = V = F1.

[0076] Where D is the feature dimension of the level-one feature vector F1;

[0077] Constructing a progressive keypoint localization Transformer decoding layer:

[0078] Progressive keypoint localization Transformer decoding layer, consisting of N DL The system consists of N progressive keypoint correction layers, with each progressive keypoint correction layer containing N layers. DD The Transformer decoder has the same structure for all layers. The inputs are level 2 feature vectors, decoder embeddings, and keypoints to be corrected; the output is the updated decoder embedding. The level 2 feature vectors are output from the Transformer encoding layer and remain unchanged in the progressive keypoint localization Transformer decoding layer. The initial decoder embedding is initialized to 0 and is not trainable. The initial keypoints to be corrected are initialized a priori from the training set and are not trainable. At the end of each progressive keypoint correction layer, the updated decoder embedding is used to calculate the proposed offset of a keypoint to be corrected using a multilayer perceptron to update the keypoint. After N... DL After the progressive key point correction layer is applied, the key points to be corrected are finally corrected into high-precision key point output.

[0079] P n =(x n ,y n ) represents the nth key point. The DropOut and residual join are omitted in the following formulas;

[0080] For any Transformer decoder layer, PE: Sine wave position encoding:

[0081]

[0082] PE(P n ) = PE(x n ,y n ) = CAT(PE(x n ),PE(y n ));

[0083] Where CAT represents the Concatenate function; subscripts 2i and 2i+1 represent the positions in the encoding vector; T is an introduced hyperparameter, T>1, the larger T is, the flatter the positional encoding; T 2i / D This represents the exponentiation of T;

[0084] The kernel of the Transformer decoder consists of a self-attention computation and a cross-attention computation, with the same formula for the attention computation as that in the Transformer encoder.

[0085] Parameters for self-attention calculation:

[0086] Q = K = C + MLP (s) (PE(P)), V=C;

[0087] Parameters for cross-attention calculation:

[0088] Q = C′ + MLP (c) (PE(P)),

[0089] K = F² + PE(P), V = F²;

[0090] Among them, MLP: In a multilayer perceptron, the two MLPs in self-attention computation and cross-attention computation share weights with all Transformer decoders respectively. These are the decoder embeddings before and after self-attention computation, respectively;

[0091] For the j-th level progressive keypoint correction layer, the formula for correcting keypoints is:

[0092]

[0093] Among them, P j-in For the j-th level input key points to be corrected, P j-out The key points to be corrected in the output of level j; MLP (fix) : For multilayer perceptron, it is responsible for embedding the decoder into the correction of the key points to be corrected; t is an introduced hyperparameter, t>1, which is used to weaken the influence of higher-level correction layers on key points, improve model stability, and achieve progressive correction.

[0094] Final output key point P final .

[0095] Preferably, the training and verification module is as follows:

[0096] The training data is input into the model, and the model directly outputs the key points. With tag value Calculate loss = MAE(P) final ,P label ); Represents a 2xM dimensional matrix in the real number field;

[0097] Enabling an auxiliary loss function can improve model convergence speed and final performance.

[0098] Record the key points after each progressive key point correction layer. This represents the keypoints after correction by the i-th progressive keypoint correction layer, calculated as follows: Backpropagation is performed based on the calculated loss function to update the parameters;

[0099] After each round of training on all the training data, the model performance is validated.

[0100] Input the validation data into the model, and the model will directly output the key points. With tag value Calculate the average error criterion = MAE(P) for all keypoints in a single image. final ,P label The average error of all validation data is calculated and used as the evaluation index, and the model parameters are saved.

[0101] After training for k rounds, the model parameters with the smallest average error index are used as the final parameters of the validated model.

[0102] Preferably, the prediction output module is as follows:

[0103] The pixel values ​​of the hand bone X-ray image to be predicted are normalized to [0,1]; the image is filled with 0 to the left or down to form a square, and the size of the filled image is denoted as r0×r0; the image is scaled to r×r using bilinear interpolation.

[0104] The preprocessed image is input into a validated medical image key point localization model for hand bone X-rays, and the model outputs key points. The pixel coordinates P of the desired key point relative to the top-left corner of the image are obtained. pix =P final ×r0.

[0105] Compared with the prior art, the beneficial effects of this invention are: (1) This invention extracts the deep language information of the feature extraction network through the Transformer encoder. The Transformer decoder realizes the localization of any number of key points. The self-attention calculation between key points allows each key point to "see" other key points, effectively solving the problems of incorrect finger prediction, low accuracy and skewness in traditional methods. The progressive decoder structure design also greatly improves the accuracy of the model. Attached Figure Description

[0106] Figure 1 A flowchart of a method for locating key points in medical images of hand bone X-ray films provided in an embodiment of the present invention;

[0107] Figure 2 A schematic diagram of an architecture for a key point localization model for hand bone X-ray medical images provided in an embodiment of the present invention;

[0108] Figure 3 This is a schematic diagram illustrating the location and numbering of 59 key points marked on a medical image of a hand bone X-ray, as provided in an embodiment of the present invention.

[0109] Figure 4 A schematic diagram of a Transformer encoder provided in an embodiment of the present invention;

[0110] Figure 5 A schematic diagram of a progressive keypoint correction layer provided in an embodiment of the present invention;

[0111] Figure 6 This is a schematic diagram of a Transformer decoder provided in an embodiment of the present invention. Detailed Implementation

[0112] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0113] Example:

[0114] like Figure 1 and Figure 2 As shown, this invention provides a method for locating key points in medical images of hand bone X-ray films, comprising the following steps:

[0115] S1, Collection and preprocessing of hand bone X-ray images and corresponding key point annotation data:

[0116] Collect X-ray images of hand bones and corresponding key point annotation data, and randomly divide them into training and validation sets;

[0117] S2, Construct a key point localization model for medical images of hand bone X-ray films;

[0118] S3, Train the model using the auxiliary loss function, and validate the model:

[0119] Using an auxiliary loss function, and based on the training and validation sets obtained in step S1, the key point localization model for hand bone X-ray medical images is trained and validated.

[0120] S4, using a validated model for end-to-end prediction:

[0121] The trained and validated key point localization model for hand bone X-ray medical images is used to predict key points in hand bone X-ray medical images.

[0122] The specific implementation of step S1 is as follows:

[0123] S1, Collection and preprocessing of hand bone X-ray images and corresponding key point annotation data.

[0124] We collected hand bone X-ray images from the RSNA dataset, labeled the corresponding key points, and used images of the left hand that included the complete hand and radius. The images ranged in width and height from 2000 to 3000 pixels, totaling 6460 images. 5814 images were randomly selected as the training set, and the remaining 646 images were used as the test set.

[0125] The pixel values ​​of the hand bone X-ray image are normalized to [0,1]; a square is filled with 0 to the left or down, and the size after filling is recorded as r0×r0; the image is scaled to 512×512 using bilinear interpolation.

[0126] like Figure 3 As shown, each image is labeled with 59 key points, represented as follows: Normalize the key points into

[0127] The specific implementation of step S2 is as follows:

[0128] S2, Construct a key point localization model for medical images of hand bones using X-ray films.

[0129] The key point localization model for hand bone X-ray medical images consists of three parts: a bone feature extraction network, a Transformer encoding layer, and a progressive key point localization Transformer decoding layer.

[0130] S21, Construct the backbone feature extraction network.

[0131] Construct a Swin-Small feature extraction network with 1 input channel to convert the input hand bone X-ray medical image into a level-one feature vector.

[0132] S22, Construct the Transformer encoding layer.

[0133] The Transformer encoding layer includes a feature vector position encoding layer and stacked N layers. En Layer Transformer encoder, the structure of the Transformer encoder is as follows Figure 4 As shown. The feature vector position encoding layer is used to perform sinusoidal position encoding on the level one feature vector and add it to the level one feature vector. The result of the addition is used as the input of the first layer Transformer encoder. The inputs of the remaining Transformer encoders are the outputs of the previous layer Transformer encoder. After passing through the Transformer encoding layer, the level one feature vector F1 is transformed into a level two feature vector.

[0134] The formula for self-attention in a Transformer encoder:

[0135]

[0136] Q = K = V = F1.

[0137] S23, Construct a progressive keypoint localization Transformer decoding layer.

[0138] The progressive keypoint localization Transformer decoding layer consists of three progressive keypoint correction layers, the structure of which is as follows: Figure 5 As shown; each progressive keypoint correction layer has 4 Transformer decoders, and the structure of the Transformer decoder is as follows. Figure 6As shown, all Transformer decoders have the same structure: input is a level 2 feature vector, decoder embedding, and keypoints to be corrected; output is the updated decoder embedding. The level 2 feature vector is output from the Transformer encoding layer in S2.2 and remains unchanged in the progressive keypoint localization Transformer decoding layer. The initial decoder embedding is initialized to 0 and is not trainable. The initial keypoints to be corrected are initialized using prior training data and are not trainable: the mean of the keypoint labels in the training set is calculated to initialize the initial decoder embedding, and its gradient is turned off. At the end of each progressive keypoint correction layer, the updated decoder embedding output from that layer is used to calculate a proposed offset for a keypoint to be corrected using a multilayer perceptron to update the keypoint. After three progressive keypoint correction layers, the keypoints to be corrected are finally corrected to high-precision keypoint outputs.

[0139] P n =(x n ,y n ) represents the nth key point. The DropOut and residual join are omitted in the following formulas.

[0140] For any Transformer decoder layer, PE: Sine wave position encoding:

[0141]

[0142] PE(P n ) = PE(x n ,y n ) = CAT(PE(x n ),PE(y n )).

[0143] Where CAT represents the Concatenate function; the subscripts 2i and 2i+1 are the positions in the encoding vector.

[0144] The core of the Transformer decoder consists of a self-attention computation and a cross-attention computation. The formula for the attention computation is as follows:

[0145]

[0146] Parameters for self-attention calculation:

[0147] Q = K = C + MLP (s) (PE(P)), V=C;

[0148] Parameters for cross-attention calculation:

[0149] Q = C′ + MLP(c) (PE(P)),

[0150] K = F² + PE(P), V = F²;

[0151] MLP: In a multilayer perceptron, the two MLPs in self-attention computation and cross-attention computation share weights with all Transformer decoders respectively. These are the decoder embeddings before and after self-attention computation, respectively.

[0152] For the j-th level progressive keypoint correction layer, the formula for correcting keypoints is:

[0153]

[0154] MLP (fix) : It is a multilayer perceptron, responsible for embedding the decoder back into the correction of the key points to be corrected; t is an introduced hyperparameter, t>1, which can weaken the influence of higher-level correction layers on key points, improve model stability, and achieve progressive correction.

[0155] Final output key point P final .

[0156] The specific implementation of step S3 is as follows:

[0157] S3 uses an auxiliary loss function to train the model.

[0158] Enable the auxiliary loss function, input the training data into the model, and record the key points after correction at each progressive keypoint correction layer. calculate Backpropagation is performed based on the calculated loss function to update the parameters.

[0159] The specific implementation of step 4 is as follows:

[0160] S4 uses a trained model to make end-to-end predictions.

[0161] The pixel values ​​of the hand bone X-ray image to be predicted are normalized to [0,1]; the image is filled with 0 to the left or down to form a square, and the size of the filled image is recorded as r0×r0; the image is scaled to 256×256 using bilinear interpolation.

[0162] The preprocessed image is input into a validated medical image key point localization model for hand bone X-rays, and the model can then output the key points. The pixel coordinates P of the desired key point relative to the top-left corner of the image are obtained. pix =P final ×r0.

[0163] This invention also provides an adaptive system for chest radiograph anomaly recognition domain that combines Wasserstein distance and difference metric, comprising:

[0164] The data preparation and preprocessing module is used to acquire CT images from chest X-rays and to perform data preprocessing on the CT images.

[0165] The feature extraction module is used to input CT images into the Patch partitioning module for block division. Each Patch consists of 4×4=16 adjacent pixels and is flattened in the channel direction. The channel data of each pixel is linearly transformed through a linear embedding layer, and feature maps of different sizes are constructed in four stages.

[0166] The loss minimization module is used to select the source domain sample that is closest to the target domain sample and calculate the Wasserstein distance; it performs cross-domain processing on the target domain samples to obtain the contrast domain difference; it constructs the overall objective function based on the obtained Wasserstein distance and contrast domain difference; and it optimizes and updates the parameters of the Swing Transformer network based on the overall objective function.

[0167] The model validation module is used to validate the optimized and parameter-updated Swin Transformer network and perform chest X-ray classification and prediction tasks.

[0168] This invention also provides a key point localization system for medical images of hand bone X-ray films, comprising:

[0169] The collection and preprocessing module is used to collect hand bone X-ray images and corresponding key point annotation data, and randomly divide them into training and validation sets;

[0170] The model building module is used to build a key point localization model for medical images of hand bone X-ray films;

[0171] The training and validation module is used to train and validate the key point localization model of hand bone X-ray medical images using an auxiliary loss function and based on the obtained training and validation sets.

[0172] The prediction output module is used to generate a trained and validated key point localization model for hand bone X-ray medical images, which is then used to predict key points in hand bone X-ray medical images.

[0173] The collection and preprocessing module is as follows:

[0174] The pixel values ​​of the collected hand bone X-ray images are normalized to [0,1]; they are then filled with 0s to the left or down to form a square, and the size of the i-th image after filling is denoted as r. 0i ×r 0i; Scale to r×r size using bilinear interpolation;

[0175] Let each image have m key points, represented as... The nth key point P n =[x n y n ], where x n The x-axis represents the pixel distance of the keypoint relative to the top-left corner of the image, and the y-axis represents the distance from the keypoint in the x-axis relative to the top-left corner of the image. n This represents the pixel distance of the keypoint in the y-direction relative to the top-left corner of the image;

[0176] Normalize the key points into

[0177] The model building module is as follows:

[0178] Constructing a backbone feature extraction network:

[0179] The backbone feature extraction network adopts a basic feature extraction network, specifically ResNet, DenseNet, or Swin Transformer, to convert the input hand bone X-ray medical image into a level-one feature vector F1.

[0180] Building the Transformer encoding layer:

[0181] The Transformer encoding layer includes a feature vector position encoding layer and stacked N layers. En The first layer of Transformer encoders; the feature vector position encoding layer is used to perform sinusoidal position encoding on the first-level feature vector and add it to the first-level feature vector. The result of the addition is used as the input of the first-level Transformer encoder. The inputs of the remaining Transformer encoders are the outputs of the previous Transformer encoder. After passing through the Transformer encoding layer, the first-level feature vector F1 is transformed into the second-level feature vector F2.

[0182] The formula for self-attention in a Transformer encoder:

[0183]

[0184] Q = K = V = F1.

[0185] Where D is the feature dimension of the level-one feature vector F1;

[0186] Constructing a progressive keypoint localization Transformer decoding layer:

[0187] Progressive keypoint localization Transformer decoding layer, consisting of NDL The system consists of N progressive keypoint correction layers, with each progressive keypoint correction layer containing N layers. DD The Transformer decoder has the same structure for all layers. Its inputs are level 2 feature vectors, decoder embeddings, and keypoints to be corrected; the output is the updated decoder embedding. The level 2 feature vectors are output from the Transformer encoding layer in step S22 and remain unchanged in the progressive keypoint localization Transformer decoding layer. The initial decoder embedding is initialized to 0 and is not trainable. The initial keypoints to be corrected are initialized a priori from the training set and are not trainable. At the end of each progressive keypoint correction layer, the updated decoder embedding is used to calculate the proposed offset of a keypoint to be corrected using a multilayer perceptron to update the keypoint. After N... DL After the progressive key point correction layer is applied, the key points to be corrected are finally corrected into high-precision key point output.

[0188] P n =(x n ,y n ) represents the nth key point. The DropOut and residual join are omitted in the following formulas;

[0189] For any Transformer decoder layer, PE: Sine wave position encoding:

[0190]

[0191] PE(P n ) = PE(x n ,y n ) = CAT(PE(x n ),PE(y n ));

[0192] Where CAT represents the Concatenate function; the subscripts 2i and 2i+1 are the positions in the encoding vector; T is an introduced hyperparameter, T>1, the larger T is, the flatter the position encoding;

[0193] The kernel of the Transformer decoder consists of a self-attention computation and a cross-attention computation, with the same formula for the attention computation as that in the Transformer encoder.

[0194] Parameters for self-attention calculation:

[0195] Q = K = C + MLP (s) (PE(P)), V=C;

[0196] Parameters for cross-attention calculation:

[0197] Q = C′ + MLP (c) (PE(P)),

[0198] K = F² + PE(P), V = F²;

[0199] Among them, MLP: In a multilayer perceptron, the two MLPs in self-attention computation and cross-attention computation share weights with all Transformer decoders respectively. These are the decoder embeddings before and after self-attention computation, respectively;

[0200] For the j-th level progressive keypoint correction layer, the formula for correcting keypoints is:

[0201]

[0202] MLP (fix) : For multilayer perceptron, it is responsible for embedding the decoder into the correction of the key points to be corrected; t is an introduced hyperparameter, t>1, which is used to weaken the influence of higher-level correction layers on key points, improve model stability, and achieve progressive correction.

[0203] Final output key point P final .

[0204] The training and validation modules are as follows:

[0205] The training data is input into the model, and the model directly outputs the key points. With tag value Calculate loss = MAE(P) final ,P label );

[0206] Enabling an auxiliary loss function can improve model convergence speed and final performance.

[0207] Record the key points after each progressive key point correction layer. calculate Backpropagation is performed based on the calculated loss function to update the parameters;

[0208] After each round of training on all the training data, the model performance is validated.

[0209] Input the validation data into the model, and the model will directly output the key points. With tag value Calculate the average error criterion = MAE(P) for all keypoints in a single image.final ,P label The average error of all validation data is calculated and used as the evaluation index, and the model parameters are saved.

[0210] After training for k rounds, the model parameters with the smallest average error index are used as the final parameters of the validated model.

[0211] The prediction output module is as follows:

[0212] The pixel values ​​of the hand bone X-ray image to be predicted are normalized to [0,1]; the image is filled with 0 to the left or down to form a square, and the size of the filled image is denoted as r0×r0; the image is scaled to r×r using bilinear interpolation.

[0213] The preprocessed image is input into a validated medical image key point localization model for hand bone X-rays, and the model outputs key points. The pixel coordinates P of the desired key point relative to the top-left corner of the image are obtained. pix =P final ×r0.

[0214] This invention extracts deep language information from a feature extraction network using a Transformer encoder. The Transformer decoder enables the localization of any number of keypoints. Self-attention computation between keypoints allows each keypoint to "see" other keypoints, effectively solving problems such as mispredicting fingers, low accuracy, and skewness in traditional methods. The progressive decoder structure design also greatly improves the accuracy of the model.

[0215] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for locating key points in medical images of X-ray images of hand bones, characterized in that, Comprising the following steps: S1, collection and preprocessing of hand bone X-ray image and corresponding key point annotation data: Collect hand bone X-ray image and corresponding key point annotation data, and randomly divide the training set and the validation set; S2, constructing a hand bone X-ray image key point positioning model; S3, training the model using an auxiliary loss function and verifying the model: Using an auxiliary loss function and according to the training set and the validation set obtained in step S1, the hand bone X-ray image key point positioning model is trained and verified; S4, end-to-end prediction using the verified model: The hand bone X-ray image key point positioning model after training and verification is used to predict the hand bone X-ray image key point; The hand bone X-ray image key point positioning model comprises a backbone feature extraction network, a Transformer encoding layer and a progressive key point positioning Transformer decoding layer, and the model construction process comprises the following steps: S21, constructing a backbone feature extraction network: The backbone feature extraction network adopts a basic feature extraction network, specifically ResNet or DenseNet or SwinTransformer, for converting the input hand bone X-ray image into a first-level feature vector F1; S22, constructing a Transformer encoding layer: The transformer encoding layer comprises a feature vector position encoding layer and N En layer transformer encoder; the feature vector position encoding layer is used for performing sinusoidal position encoding on the level one feature vector and adding the level one feature vector, and an addition result is used as an input of the first layer transformer encoder, and an input of a remaining transformer encoder is an output of a previous layer transformer encoder, and the level one feature vector F1 is converted into a level two feature vector F2 through the transformer encoding layer. The formula of self-attention in the Transformer encoder: Q=K=V=F1. Wherein, D is the feature dimension of the first-level feature vector F1; S23, constructing a progressive key point positioning Transformer decoding layer: an incremental keypoint positioning Transformer decoding layer composed of N DL incremental keypoint refinement layers, each incremental keypoint refinement layer has a Transformer decoder with N DD layers; all the Transformer decoders have the same structure, the input is the level two feature vector, the decoder embedding, the keypoint to be refined, and the output is the updated decoder embedding; the level two feature vector is output by the Transformer encoding layer in step S22 and does not change in the incremental keypoint positioning Transformer decoding layer; the initial decoder embedding is initialized by 0 and cannot be trained; the initial keypoint to be refined is initialized by the training set prior and cannot be trained; at the end of each incremental keypoint refinement layer, the updated decoder embedding is used to calculate a suggested offset of the keypoint to be refined by using a multi-layer perceptron to update the keypoint to be refined; after being refined by N DL incremental keypoint refinement layers, the keypoint to be refined is finally refined into a high-precision keypoint output; P n = (x n ,y n ) represents the nth key point, DropOut and residual connections are omitted in the following equations; For any certain layer Transformer decoder, denotes the sinusoidal position encoding: PE(P n ) = PE(x n ,y n ) = CAT(PE(x n ), PE(y n )); where CAT represents a Concatenate function; subscripts 2i and 2i+1 are positions in the encoding vector; T is an introduced hyperparameter, T > 1, the larger T is, the flatter the position encoding is; T 2i / D represents a power operation of T; The kernel of the Transformer decoder is a self-attention calculation and a cross-attention calculation, and the formula of the attention calculation is the same as that in the Transformer encoder; Parameters of self-attention calculation: Q = K = C + MLP (s) (PE(P)), V = C; Parameters of cross-attention calculation: Q = C' + MLP (c) (PE(P)), K=F2+PE(P), V=F2; wherein, denotes a multi-layer perceptron, two MLPs in the self-attention computation and the cross-attention computation respectively share weights across all Transformer decoders; are the decoder embeddings before and after the self-attention computation, respectively; For the jth progressive key point correction layer, the formula for correcting the key point is: wherein P j-in is the input key point to be corrected at the jth level, P j-out is the output key point to be corrected at the jth level; is a multi-layer perception responsible for mapping the decoder embedding back to the correction of the key point to be corrected; t is a hyperparameter introduced, t > 1, used to weaken the influence of the correction layer at the high level on the key point, improve the stability of the model, and realize progressive correction. Final output key points P final .

2. The method of claim 1, wherein, Step S1 comprises the following steps: S11, normalize the pixel values of the collected hand bone X-ray images to [0, 1]; pad to the left or down with 0 to a square, and the size of the i-th image after padding is denoted as r 0i ×r 0i ; scale to r x r size using a bilinear interpolation method; S12, set m key points for each image, denoted as wherein the mth key point coordinate (x m ,y m ), R is a real number field; the nth key point P n =[x n y n ], wherein x n represents the pixel distance of the key point in the x direction relative to the upper left corner of the image, and y n represents the pixel distance of the key point in the y direction relative to the upper left corner of the image; normalizing the key points to 3. The method of claim 2, wherein, Step S3 comprises the following steps: S31, input the training data into the model, and the model directly outputs the key points with the label value Calculate loss = MAE(P final ,P label ); denotes a real number field 2xM matrix; S32, enabling the auxiliary loss function to improve the convergence speed and final performance of the model: record the modified key points of each progressive key point modification layer denote the modified key points of the i-th progressive key point modification layer, calculate perform back propagation according to the loss function obtained by calculation, update parameters; S33, after training all the training data for one round each time, the model performance is verified: The verification data is input into the model, and the model directly outputs the key points With the label value The average error criterion of all key points in a single image is calculated MAE(P final ,P label ), the average error of all verification data is calculated to obtain the final average error as an evaluation index, and the model parameters are saved; S34, after training for k rounds, the model parameters with the smallest average error index are taken as the final parameters of the verified model.

4. The method of claim 3, wherein, Step S4 comprises the following steps: S41, normalize the pixel value of the hand bone X-ray image to be predicted to [0, 1]; pad to the left or down to a square, and the size of the image after padding is denoted as r0x r0; use the bilinear interpolation method to scale to r x r size; S42, input the preprocessed image into the verified hand bone X-ray medical image key point positioning model, and the model outputs the key points Get the pixel coordinates P of the key points relative to the upper left corner of the image pix = P final × r0.

5. A hand bone X-ray medical image keypoint localization system, characterized by, Comprise: The collection and preprocessing module is used to collect hand bone X-ray image and corresponding key point annotation data, and randomly divide the training set and the validation set; The model construction module is used to construct a hand bone X-ray image key point positioning model; The training and verification module is used to use an auxiliary loss function and according to the obtained training set and the validation set, the hand bone X-ray image key point positioning model is trained and verified; The prediction output module is configured to use the trained and verified hand bone X-ray medical image key point positioning model to predict hand bone X-ray medical image key points. The model construction module is specifically configured as follows: A backbone feature extraction network is constructed. The backbone feature extraction network adopts a basic feature extraction network, specifically ResNet or DenseNet or SwinTransformer, and is configured to convert an input hand bone X-ray medical image into a first-level feature vector F1. A Transformer encoding layer is constructed. The Transformer encoding layer includes a feature vector position encoding layer and N En The feature vector position encoding layer is used for sinusoidal position encoding of the level one feature vector and adding the level one feature vector, and the addition result is used as the input of the first layer Transformer encoder, the input of the remaining Transformer encoders is the output of the previous layer Transformer encoder, and the level one feature vector F1 is converted into the level two feature vector F2 through the Transformer encoding layer. The formula of self-attention in the Transformer encoder is as follows: Q=K=V=F1. D is the feature dimension of the first-level feature vector F1. A progressive key point positioning Transformer decoding layer is constructed. an incremental keypoint positioning Transformer decoding layer, composed of N DL incremental keypoint refinement layers, each incremental keypoint refinement layer has a Transformer decoder with N DD layers; all the Transformer decoders have the same structure, the input is the level two feature vector, the decoder embedding, the keypoint to be refined, and the output is the updated decoder embedding; the level two feature vector is output by the Transformer encoding layer and does not change in the incremental keypoint positioning Transformer decoding layer; the initial decoder embedding is initialized by 0 and cannot be trained; the initial keypoint to be refined is initialized by the training set prior and cannot be trained; at the end of each incremental keypoint refinement layer, the updated decoder embedding is used to calculate a suggested offset of the keypoint to be refined by using a multi-layer perceptron to update the keypoint to be refined; after being refined by N DL incremental keypoint refinement layers, the keypoint to be refined is finally refined into a high-precision keypoint output; P n = (x n ,y n ) represents the nth key point, DropOut and residual connections are omitted in the following equations; For any certain layer Transformer decoder, denotes the sinusoidal position encoding: PE(P n ) = PE(x n ,y n ) = CAT(PE(x n ), PE(y n )) where CAT represents a Concatenate function; subscripts 2i and 2i+1 are positions in the encoding vector; T is an introduced hyperparameter, T > 1, the larger T is, the flatter the position encoding is; T 2i / D represents a power operation of T; The kernel of the Transformer decoder is a self-attention calculation and a cross-attention calculation, and the formula of the attention calculation is the same as that in the Transformer encoder. The parameters of the self-attention calculation are as follows: Q = K = C + MLP (s) (PE(P)), V = C; The parameters of the cross-attention calculation are as follows: Q = C' + MLP (c) (PE(P)), K=F2+PE(P), V=F2. wherein, denotes a multi-layer perceptron, two MLPs in the self-attention computation and the cross-attention computation respectively share weights across all Transformer decoders; are the decoder embeddings before and after the self-attention computation, respectively; For the jth progressive key point correction layer, the formula for correcting the key points is as follows: wherein P j-in is the input key point to be corrected at the jth level, P j-out is the output key point to be corrected at the jth level; is a multi-layer perception responsible for mapping the decoder embedding back to the correction of the key point to be corrected; t is a hyperparameter introduced, t > 1, for weakening the influence of the correction layer at the high level on the key point, improving the stability of the model, and realizing progressive correction. Final output key points P final .

6. The hand X-ray medical image keypoint localization system of claim 5, wherein, The collection and preprocessing module is specifically configured as follows: The pixel values of the collected hand bone X-ray images are normalized to [0, 1]; padded to the left or down with 0 to a square, and the size of the i-th image after padding is denoted as r 0i ×r 0i ; scaled to r x r size using a bilinear interpolation method; It is set that each image has m key points, denoted as Wherein, the mth key point coordinate (x m ,y m ), R is a real number field; the nth key point P n =[x n y n ], wherein x n indicates the pixel distance of the key point in the x direction relative to the upper left corner of the image, and y n indicates the pixel distance of the key point in the y direction relative to the upper left corner of the image; normalizing the key points to 7. The hand X-ray medical image keypoint localization system of claim 6, wherein, The training and verification module is specifically configured as follows: The training data is input into the model, and the model directly outputs the key points with the label value Calculate loss = MAE(P final ,P label ); denotes a real-valued 2xM matrix; An auxiliary loss function is enabled to improve the model convergence speed and the final performance. record the modified key points of each progressive key point modification layer denote the modified key points of the i-th progressive key point modification layer, calculate perform back propagation according to the loss function obtained by calculation, update parameters; After each round of training of all the training data, the model performance is verified. The verification data is input into the model, and the model directly outputs the key points With the label value The average error criterion of all key points in a single image is calculated MAE(P final ,P label ), the average error of all verification data is calculated to obtain the final average error as an evaluation index, and the model parameters are saved; After k rounds of training, the model parameters with the minimum average error index are taken as the final parameters of the verified model.

8. The hand X-ray medical image keypoint localization system of claim 7, wherein, The prediction output module is specifically configured as follows: The pixel values of the hand bone X-ray image to be predicted are normalized to [0, 1]; 0 is filled to the left or the bottom to form a square, and the size of the filled image is denoted as r0xr0; the bilinear interpolation method is used to scale the image to the size of rxr; and the key points are predicted. input the preprocessed image into the verified hand bone X-ray medical image key point positioning model, and the model outputs the key points get the pixel coordinates P of the key points relative to the upper left corner of the image pix = P final x r0.

Images